Distributed system deterministic data recording and playback method and device based on event causal relationship and medium

Abstract

The invention discloses a distributed system deterministic data recording and playback method and device based on an event causal relationship and a medium, and the method comprises the steps: compiling a configuration file which comprises a unique identifier label of each calculation node; reading the label attribute of the configuration file when the computing node is started, and initializing a logic clock; when an event occurs in the node, adding 1 to a logic clock value, and recording event pair to a buffer area; when the computing node outputs data to the outside, recording the event pair in the buffer area and the output data as metadata, and then emptying the buffer area; metadata is recorded through rosbag2, and an event is output to a database; performing offline processing on the database, determining the causal relationship of the output event according to the metadata, and sorting and screening the messages according to the causal relationship of the output event to generate a new database for playback; and controlling a playback process according to the new database for playback, and realizing deterministic playback of the distributed system.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically to a distributed system deterministic data recording and playback method, device and medium based on event causality. Background Art

[0002] In a distributed system, multiple computing nodes work together to complete complex tasks. These nodes communicate through the network, generating a large number of events and data streams. In order to debug and analyze the operation of these systems, it is usually necessary to record and replay the system's behavior.

[0003] As a lightweight tool built into ros2, rosbag2 is only used to implement the most basic message recording and playback functions, and cannot guarantee that the content of the message during playback is the same as that received by the node in the actual application. In addition, the order of messages during playback is also determined by the order in which rosbag2 records and stores these messages. In the ros2 distributed system, different nodes may not receive the same messages in the same order, so the order of messages during playback cannot be guaranteed to be the same as that received by the node in the actual application.

[0004] Therefore, how to provide a distributed system deterministic data recording and playback method based on event causality is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of this, the present invention provides a method, device and medium for recording and replaying deterministic data in a distributed system based on event causality, which ensures the causality and sequence of events and realizes deterministic playback of the distributed system.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A distributed system deterministic data recording and playback method based on event causality includes:

[0008] Write a configuration file that contains the unique label of each computing node;

[0009] When the computing node starts, it reads the label attribute of the configuration file and initializes the logical clock;

[0010] When an event occurs inside a computing node, the logical clock value is incremented by 1 and the event pair is recorded in the buffer.

[0011] When a computing node outputs data to the outside, the event pair in the buffer is recorded together with the output data as metadata, and then the buffer is cleared;

[0012] Record metadata and output events to the database through rosbag2;

[0013] Process the database offline, determine the causal relationship of the output events based on the metadata, sort and filter the messages based on the causal relationship of the output events, and generate a new database for playback;

[0014] The replay executor controls the replay process according to the new database used for replay, realizing deterministic replay of distributed systems.

[0015] Preferably, each computing node corresponds to a logical clock, and the logical clock starts counting from 0. When an event occurs inside the computing node, the logical clock value is increased by 1, and each event is represented by an event pair.

[0016] Preferably, the internal events of the computing node include the following events:

[0017] (a) Receive data sent by other nodes as a subscriber;

[0018] (b) Publish data as a publisher node itself;

[0019] (c) Sending requests to the server as a client;

[0020] (d) Receive the client's request as a server;

[0021] (e) Periodic events within the node.

[0022] Preferably, the database is processed offline, the causal relationship of the output events is determined according to the metadata, and the messages are sorted and screened according to the causal relationship of the output events to generate a new database for playback, which specifically includes:

[0023] Extract the output data and metadata of each computing node from the database;

[0024] Extract event pairs from metadata to form a set of trigger events;

[0025] Match the trigger event sets of all computing nodes to determine the causal relationship of output events;

[0026] Messages are sorted and filtered according to the causal relationship of output events to generate a new database for playback.

[0027] Preferably, the playback process is controlled by a playback executor to achieve deterministic playback of a distributed system, specifically including:

[0028] The playback executor controls the playback process according to the new database for playback. Events in the new database generated after offline processing are played back in a determined order.

[0029] After the event data is played back, the computing node that receives the data notifies the executor, and the executor notifies the playback system that it can play back the next event data;

[0030] If the next event is a periodic event, the executor notifies the corresponding computing node to execute the corresponding periodic task;

[0031] If the next event is a data transmission type event, the executor notifies the playback system to publish the event data to the system;

[0032] If the next event is a request type event, the executor requests the corresponding computing node to execute the corresponding task;

[0033] All event data recorded in the database are executed in the above logical order to achieve deterministic playback of the distributed system.

[0034] A computer device comprises: a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, a distributed system deterministic data recording and playback method based on event causality is implemented.

[0035] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a distributed system deterministic data recording and playback method based on event causality.

[0036] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method, device and medium for recording and replaying deterministic data in a distributed system based on event causality, which has the following effects:

[0037] The introduction of a logical clock solution in the designed distributed system is a lightweight implementation method that can uniquely determine events.

[0038] The designed implementation strategy for recording causal relationships between events can be used to determine the relationship between events in any distributed system.

[0039] The designed causal-based database offline processing specific strategy can realize the preprocessing requirements of deterministic playback.

[0040] The designed deterministic replay strategy based on replay executor can meet the deterministic replay requirements in any multi-node complex computing scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0042] Figure 1 This is a flow chart of the distributed system deterministic data recording and playback method based on event causality provided by the present invention.

[0043] Figure 2 This is a principle block diagram of the distributed system deterministic data recording and playback method based on event causality provided by the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The embodiment of the present invention discloses a distributed system deterministic data recording and playback method based on event causality, such as Figure 1 As shown, including:

[0046] Write a configuration file that contains the unique label of each computing node;

[0047] When the computing node starts, it reads the label attribute of the configuration file and initializes the logical clock;

[0048] When an event occurs inside a computing node, the logical clock value is incremented by 1 and the event pair is recorded in the buffer.

[0049] When a computing node outputs data to the outside, the event pair in the buffer is recorded together with the output data as metadata, and then the buffer is cleared;

[0050] Record metadata and output events to the database through rosbag2;

[0051] Process the database offline, determine the causal relationship of the output events based on the metadata, sort and filter the messages based on the causal relationship of the output events, and generate a new database for playback;

[0052] The replay executor controls the replay process according to the new database used for replay, realizing deterministic replay of distributed systems.

[0053] In this embodiment, due to the inconsistency of the systems between nodes in the distributed system, the present invention adopts the method of introducing a logical clock to record events, and each event corresponds to a separate logical timestamp.

[0054] The most commonly used logical clock algorithms in distributed systems are the Lamport clock algorithm and the Vector clock algorithm. The Lamport clock is simple to implement but concurrent events are difficult to distinguish, while the vector clock can distinguish concurrent events but the storage and overhead are huge.

[0055] The present invention is based on practical applications and refers to the principles of both to realize a lightweight logical clock system capable of providing a unique identifier for each event.

[0056] Each node corresponds to a label, which stores a logical clock counting from 0. When an event occurs in the node, the logical clock value is +1. In this way, each event can be represented by an event pair (label, logical timestamp). This not only achieves the uniqueness of event labels in concurrent situations, but also ensures system performance.

[0057] In this embodiment, the specific strategy for event recording is:

[0058] In a distributed system, taking ROS2 as an example, without considering failures and lifecycles, the behavior of a node with a certain initial state will be affected by the following events:

[0059] (a) Receive data sent by other nodes as a subscriber;

[0060] (b) Publish data as a publisher node itself;

[0061] (c) Sending requests to the server as a client;

[0062] (d) Receive the client's request as a server;

[0063] (e) Periodic events within the node.

[0064] Therefore, when the above event occurs in a node, the logical clock value is increased by 1, so that each event that affects the node behavior is given a unique identifier.

[0065] In this embodiment, the causal relationship between events is recorded to achieve the following strategy:

[0066] The behavior of a node is reflected through its external output data. Therefore, to achieve deterministic playback of the system, it is necessary to record all events that affect the node's output data and their order during recording.

[0067] The present invention allows each node to store a buffer inside. When an event that affects the output occurs, the event pair (label, logical timestamp) corresponding to the event is stored in the buffer. When the output event occurs, the content recorded in the buffer is recorded as the metadata of the output data. After that, the buffer is cleared, so that the next event will be recorded as the triggering cause of the next output event.

[0068] When all data is recorded, its event pair will also be recorded as metadata. In this way, in the database generated after the data is recorded, we can know which data corresponds to which events, and at the same time determine the causal relationship between these events.

[0069] In this embodiment, the specific strategy of database offline processing based on causality is as follows:

[0070] In order to realize the determined playback based on causal relationship, the database needs to be preprocessed in an offline state. The present invention implements the relevant processing logic by designing a script tool to screen out and sort the data that needs to be played back.

[0071] Deterministic replay preprocessing logic:

[0072] For a single node, the data corresponding to its output event is determined, that is, the output data and metadata of each computing node are extracted from the database.

[0073] According to the metadata of the output event, the event pair is extracted from the metadata to form its triggering event set.

[0074] Match the trigger event sets of all computing nodes to determine the causal relationship of the output events.

[0075] For a distributed system, the same logic is used to determine the input and output boundaries and then process them.

[0076] Messages are sorted and filtered according to the causal relationship of output events to generate a new database for playback.

[0077] In this embodiment, based on the deterministic playback strategy of the playback executor:

[0078] Since the timing of each node in a distributed system is inconsistent, in order to replay these events in a certain order, the present invention proposes and designs a replay executor to ensure the timing of all events in the replay. Specifically, it includes:

[0079] The playback executor controls the playback process according to the new database for playback. Events in the new database generated after offline processing are played back in a determined order.

[0080] After an event data is played back, the node that receives the data needs to notify the executor, and the executor notifies the playback system that it can play back the next event data.

[0081] If the next event is a periodic event of a node, the executor will notify the corresponding node to perform the corresponding periodic task. If the next event is a data transmission type event, the executor will notify the playback system to publish the event data to the system. If the next event is a request type event, the executor will request the corresponding node to perform the corresponding task.

[0082] After executing all the event data recorded in the database in this logical order, the deterministic playback of the distributed system can be achieved.

[0083] In this embodiment, the computing nodes involved in the distributed system deterministic recording and playback method can support the setting of this function in the configuration file, so as to facilitate flexible use and modification by system and application developers.

[0084] In addition to the logic of the application function itself, the content of the configuration file must also include different labels for each node to meet the design principle that the events corresponding to different data can be uniquely determined.

[0085] Based on the above invention example, a schematic diagram of introducing a deterministic data recording and playback method based on event causality in the ros2 distributed system framework is given as follows: Figure 2 As shown, Node is the basic computing node in ros2, and the underlying communication between nodes is realized through DDS. Causality_tracer is a C++ class that contains the logical clock mentioned in the above invention content, the flag of each node and the buffer for saving trigger events. When writing a node, add Causality_tracer as an attribute to the node to record the causal relationship between events.

[0086] Take an autonomous driving computing application based on ROS2 as an example:

[0087] Write the corresponding configuration file and read the label attribute in the configuration file as the unique identification bit of each node during the startup and loading process of the computing node.

[0088] As a publisher in ROS2, the sensor publishes messages to the specified topic. These messages have their own metadata describing the corresponding events according to the "strategy for recording the causal relationship between events". As a subscriber in ROS2, the perception computing processing unit receives these messages and records the events corresponding to these messages in the buffer according to the "specific strategy for event recording in the system" and "strategy for recording the causal relationship between events". After processing the received sensor messages, the node will publish messages to the fusion computing and intelligent driving algorithm nodes as a publisher, and will also send the metadata of the triggering events of these messages. Subsequent nodes also process data and events according to the same logic.

[0089] In this use case, metadata can be recorded in the message header of the ROS2 message, so in this process, the recording function of rosbag2 can be used to record and store messages and their corresponding metadata in the database.

[0090] According to the database processing logic designed in "Specific strategies for offline database processing based on causal relationships", the messages in the system are sorted and filtered according to the causal relationships between events. It should be noted that the metadata content is saved in the serialized message, and the data needs to be deserialized first when post-processing the database based on the metadata.

[0091] The replay system completes deterministic replay of the processed database under the control of the executor according to the "deterministic replay strategy based on replay executor".

[0092] This embodiment provides a computer device, including: a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, a distributed system deterministic data recording and playback method based on event causality is implemented.

[0093] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, a distributed system deterministic data recording and playback method based on event causality is implemented.

[0094] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc. Various media that can store program codes.

[0095] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0096] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A distributed system deterministic data recording and playback method based on event causality, characterized in that: include: Write a configuration file that contains the unique label of each computing node; When the computing node starts, it reads the label attribute of the configuration file and initializes the logical clock; When an event occurs inside a computing node, the logical clock value is incremented by 1 and the event pair is recorded in the buffer. When a computing node outputs data to the outside, the event pair in the buffer is recorded together with the output data as metadata, and then the buffer is cleared; Record metadata and output events to the database through rosbag2; Process the database offline, determine the causal relationship of the output events based on the metadata, sort and filter the messages based on the causal relationship of the output events, and generate a new database for playback; The replay executor controls the replay process according to the new database used for replay, realizing deterministic replay of distributed systems.

2. The method for recording and replaying deterministic data in a distributed system based on event causality according to claim 1, characterized in that: Each computing node corresponds to a logical clock. The logical clock starts counting from 0. When an event occurs inside the computing node, the logical clock value increases by 1. Each event is represented by an event pair.

3. The method for recording and replaying deterministic data in a distributed system based on event causality according to claim 1, characterized in that: Compute node internal events include the following: (a) Receive data sent by other nodes as a subscriber; (b) Publish data as a publisher node itself; (c) Sending requests to the server as a client; (d) Receive the client's request as a server; (e) Periodic events within the node.

4. The method for recording and replaying deterministic data in a distributed system based on event causality according to claim 1, characterized in that: The database is processed offline, the causal relationship of the output events is determined based on the metadata, and the messages are sorted and filtered based on the causal relationship of the output events to generate a new database for playback, including: Extract the output data and metadata of each computing node from the database; Extract event pairs from metadata to form a set of trigger events; Match the trigger event sets of all computing nodes to determine the causal relationship of output events; Messages are sorted and filtered according to the causal relationship of output events to generate a new database for playback.

5. The method for recording and replaying deterministic data in a distributed system based on event causality according to claim 1, characterized in that: The playback process is controlled by the playback executor to achieve deterministic playback of distributed systems, including: The playback executor controls the playback process according to the new database for playback. Events in the new database generated after offline processing are played back in a determined order. After the event data is played back, the computing node that receives the data notifies the executor, and the executor notifies the playback system that it can play back the next event data; If the next event is a periodic event, the executor notifies the corresponding computing node to execute the corresponding periodic task; If the next event is a data transmission type event, the executor notifies the playback system to publish the event data to the system; If the next event is a request type event, the executor requests the corresponding computing node to execute the corresponding task; All event data recorded in the database are executed in the above logical order to achieve deterministic playback of the distributed system.

6. A computer device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

7. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 5.

Images