Distributed system deterministic data recording playback method and device based on event causality, and medium

Abstract

The application discloses a kind of based on event causality distributed system deterministic data recording playback method, equipment and medium, method includes writing a configuration file, configuration file includes the unique identification label of each computing node;Computing node reads the label attribute of configuration file when starting, initializes logical clock;When event occurs inside computing node, logical clock value is added 1, and event pair is recorded to buffer zone;When computing node exports data externally, event pair in buffer zone and output data are recorded as metadata, and then buffer zone is emptied;Metadata and output event are recorded into database by rosbag2;Offline processing is carried out to database, the causality of output event is determined according to metadata, and message is sorted and screened according to the causality of output event, to generate new database for playback;Playback process is controlled according to new database for playback, to realize the deterministic playback of distributed system.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to a method, device, and medium for deterministic data recording and playback in a distributed system based on event causality. Background Technology

[0002] In distributed systems, multiple computing nodes collaborate to complete complex tasks. These nodes communicate over a network, generating a large number of events and data streams. To debug and analyze the operation of these systems, it is often necessary to record and replay their behavior.

[0003] As a lightweight tool built into ROS2, rosbag2 is only used to implement the most basic message recording and playback functions. It cannot guarantee that the content of the messages during playback is the same as that received by the nodes in the actual application. Furthermore, the order of the messages during playback depends on the order in which rosbag2 records and stores these messages. However, in the distributed system of ROS2, different nodes may receive the same few messages in different orders. Therefore, the order of the messages during playback cannot be guaranteed to be the same as that received by the nodes in the actual application.

[0004] Therefore, how to provide a deterministic data recording and playback method for distributed systems based on event causality is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a method, device and medium for deterministic data recording and playback of distributed systems based on event causality, ensuring the causal relationship and sequence of events, and realizing deterministic playback of distributed systems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] Deterministic data recording and playback methods for distributed systems based on event causality include:

[0008] Write a configuration file containing a unique identifier label for each compute node;

[0009] The compute node reads the label attribute from the configuration file and initializes the logical clock upon startup;

[0010] When an event occurs inside a compute node, the logical clock value is incremented by 1, and the event pair is recorded in the buffer.

[0011] When a compute node outputs data to an external system, it records the event pair in the buffer along with the output data as metadata, and then clears the buffer.

[0012] Record metadata and output events into database by rosbag2;

[0013] Offline process the database, determine the causality of output events according to metadata, and sort and filter messages according to the causality of output events to generate a new database for playback;

[0014] The playback executor excutor controls the playback process according to the new database for playback, and realizes the deterministic playback of the distributed system.

[0015] Preferably, each computing node corresponds to a logical clock, and the logical clock counts from 0. When an event occurs inside the computing node, the logical clock value is incremented by 1, and each event is represented by an event pair.

[0016] Preferably, the internal events of the computing node include the following events:

[0017] (a) receiving data sent by other nodes as a subscriber;

[0018] (b) publishing data outside the node itself as a publisher node;

[0019] (c) sending a request to the server as a client;

[0020] (d) receiving a request from a client as a server;

[0021] (e) periodic events inside the node.

[0022] Preferably, the database is offline processed, the causality of output events is determined according to the metadata, and the messages are sorted and filtered according to the causality of output events to generate a new database for playback, which specifically includes:

[0023] Extract the output data and its metadata of each computing node from the database;

[0024] Extract the event pair from the metadata to form a set of trigger events;

[0025] Match the trigger event sets of all computing nodes to determine the causality of output events;

[0026] Sort and filter messages according to the causality of output events to generate a new database for playback.

[0027] Preferably, the playback process is controlled by the playback executor excutor to realize the deterministic playback of the distributed system, which specifically includes:

[0028] The playback excutor controls the playback process according to the new database for playback, and the events in the new database generated after offline processing are played back in a determined order;

[0029] After the event data is played back, the computing node receiving the data notifies the excutor, and the excutor notifies the playback system that the next event data can be played back;

[0030] If the next event is a periodic event, the excutor notifies the corresponding computing node to perform the corresponding periodic task;

[0031] If the next event is a data transmission type event, the excutor notifies the playback system to publish the event data to the system;

[0032] If the next event is a request type event, the excutor requests the corresponding computing node to perform the corresponding task;

[0033] All event data recorded in the database are executed in the above logical order, and the deterministic playback of the distributed system is realized.

[0034] A computer device, comprising: a memory and a processor, the memory stores a computer program executable on the processor, and the processor executes the computer program to realize the event causal relationship-based deterministic data recording playback method of the distributed system.

[0035] A computer readable storage medium, the storage medium stores a computer program, and the computer program is executed by a processor to realize the event causal relationship-based deterministic data recording playback method of the distributed system.

[0036] According to the above technical solution, compared with the prior art, the present disclosure provides an event causal relationship-based deterministic data recording playback method of a distributed system, a device and a medium, which has the following effects:

[0037] The logical clock scheme introduced in the designed distributed system is a lightweight method that can uniquely determine events.

[0038] The designed implementation strategy of the causal relationship between recorded events can be used to determine the relationship between events in any distributed system.

[0039] The designed offline processing specific strategy of the causal relationship-based database can meet the preprocessing requirements of deterministic playback.

[0040] The designed deterministic playback strategy based on the playback excutor can meet the deterministic playback requirements in any multi-node complex computing scenario. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the accompanying drawings in the following description only only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0042] Figure 1 The event-causal relationship-based distributed system deterministic data recording playback method flowchart provided by the present application.

[0043] Figure 2 The event-causal relationship-based distributed system deterministic data recording playback method principle block diagram provided by the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0045] The embodiments of the present application disclose an event-causal relationship-based distributed system deterministic data recording playback method, as shown in Figure 1 The method comprises the following steps:

[0046] A configuration file is written, and the configuration file contains a unique identifier label of each computing node;

[0047] The computing node reads the label attribute of the configuration file when starting, and initializes the logical clock;

[0048] When an event occurs inside the computing node, the logical clock value is incremented by 1, and the event pair is recorded to the buffer;

[0049] When the computing node outputs data externally, the event pair in the buffer and the output data are recorded as metadata, and then the buffer is emptied;

[0050] The rosbag2 records the metadata and output events into the database;

[0051] The database is processed offline, the causal relationship of the output events is determined according to the metadata, and the messages are sorted and filtered according to the causal relationship of the output events, to generate a new database for playback;

[0052] The playback excutor controls the playback process according to the new database for playback, and realizes the deterministic playback of the distributed system.

[0053] In the embodiment, the application records events by introducing a logical clock due to the inconsistency of the system between nodes in the distributed system, and each event corresponds to a single logical timestamp.

[0054] The most commonly used logical clock algorithm in the distributed system is the Lamport clock algorithm and the Vector clock algorithm.

[0055] The application is based on practical application and refers to the principles of both, and realizes a lightweight logical clock system capable of providing a unique identifier for each event.

[0056] Each node corresponds to a label, and a logical clock counted from 0 is saved.

[0057] In the embodiment, the specific strategy for event recording is:

[0058] In the distributed system, for example, ROS2, without considering faults and life cycles, the behavior of the node in the initial state will be affected by the following events:

[0059] (a) receiving data sent by other nodes as a subscriber;

[0060] (b) publishing data to the outside as a publisher node itself;

[0061] (c) sending a request to the server as a client;

[0062] (d) receiving a request from a client as a server;

[0063] (e) periodic events inside the node.

[0064] Therefore, when the above events occur in a node, the logical clock value is incremented by 1, so that each event that has an impact on the behavior of the node is given a unique identifier.

[0065] In the embodiment, the implementation strategy of recording the causal relationship between events is:

[0066] The behavior of a node is reflected by its output data to the outside, so to achieve deterministic playback of the system, all events that affect the output data of a node and their order need to be recorded at the time of recording.

[0067] The present application is to let each node internally save a buffer, when an event that affects the output occurs, the event pair (label, logical timestamp) corresponding to the event is saved in the buffer, and when the output event occurs, the recorded content in the buffer is recorded as the metadata of the output data. Then empty the buffer, so that the next event will be recorded as the trigger of the next output event.

[0068] And all data will record their event pair as metadata when being recorded, so in the database generated after the data is recorded, it can be known which data corresponds to which event, and the causal relationship between these events can be determined.

[0069] In this embodiment, the specific strategy of offline processing of the causal relationship-based database is:

[0070] In order to achieve deterministic playback based on causal relationship, a preprocessing of the database in an offline state is needed. The present application realizes the related processing logic by designing a script tool to filter out and sort the data to be played back.

[0071] Deterministic playback preprocessing logic:

[0072] For a single node, determine its output event corresponding data, that is, extract the output data and its metadata of each computing node from the database.

[0073] According to the metadata of the output event, extract the event pair from the metadata to form its trigger event set.

[0074] Match all computing node trigger event sets to determine the causal relationship of the output event.

[0075] For a distributed system, the same logic determines the input and output boundaries, and the same logic is used for processing.

[0076] According to the causal relationship of the output event, sort and filter the messages to generate a new database for playback.

[0077] In this embodiment, the deterministic playback strategy based on the playback executor is:

[0078] In order to replay these events in a determined order due to the time inconsistency of each node in the distributed system, the application proposes and designs a replay excutor to ensure the time sequence of all events in the replay. Specifically, it includes:

[0079] The replay excutor controls the replay process according to the new database for replay, and the events in the new database generated after offline processing are replayed in a determined order.

[0080] After the replay of an event data, the node receiving the data needs to notify the excutor, which notifies the replay system that the next event data can be replayed.

[0081] If the next event is a periodic event of a node, the excutor will notify the corresponding node to perform the corresponding periodic task, if the next event is a data transmission type event, the excutor will notify the replay system to publish the event data to the system, and if the next event is a request type event, the excutor will request the corresponding node to perform the corresponding task.

[0082] After the execution of all event data recorded in the database in the logical order, the deterministic replay of the distributed system can be realized.

[0083] In this embodiment, the computing nodes involved in the deterministic recording and replay method of the distributed system can support this function in the configuration file, which is convenient for system and application developers to use and modify flexibly.

[0084] In addition to including the logic of the application function itself, the content in the configuration file also needs to include different labels of each node to meet the design principle that the events corresponding to different data can be uniquely determined.

[0085] Based on the above application example, a simple diagram of introducing a deterministic data recording and replay method based on event causality in the ros2 distributed system framework is given as follows Figure 2 The Node is the basic computing node in ros2, and the underlying communication between nodes is realized through DDS. Causality_tracer is a c++ class that includes the logical clock mentioned in the above application content, the label of each node and the buffer for saving triggered events. When writing the node, Causality_tracer is added as an attribute to the node to record the causality between events.

[0086] Taking an automatic driving computing application based on ROS2 as an example:

[0087] Write the corresponding configuration file, and read the label attribute in the configuration file as the unique identifier of each node during the startup loading process of the computing node.

[0088] The sensor acts as a publisher in ROS2, publishing messages to a specified topic, which have their own metadata describing the corresponding event according to the "implementation strategy of recording the causal relationship between events". The perception computing processing unit acts as a subscriber in ROS2, receiving these messages, and recording the corresponding events in the buffer according to the "specific strategy of recording event records in the system" and "implementation strategy of recording the causal relationship between events" while receiving the messages. After processing the received sensor messages, the node will act as a publisher to publish messages to the fusion computing and intelligent driving algorithm node, and also send the metadata containing the triggering event of these messages. The subsequent nodes also process data and events according to the same logic.

[0089] In this use case, metadata can be recorded in the message header of ROS2 message, so in this process, the rosbag2 recording function can be used to record and store messages and their corresponding metadata into the database.

[0090] According to the database processing logic designed in the "specific strategy of offline processing of databases based on causal relationships", the messages in the system are sorted and filtered according to the causal relationship between events. It should be noted that the metadata content is saved in the serialized message, and the data needs to be deserialized before processing the database according to the metadata.

[0091] The playback system completes the deterministic playback of the processed database under the control of the excutor according to the "deterministic playback strategy based on playback executors".

[0092] The embodiment provides a computer device, including a memory and a processor, the memory stores a computer program capable of running on the processor, and when the processor executes the computer program, a deterministic data recording playback method of a distributed system based on event causal relationship is realized.

[0093] The embodiment provides a computer readable storage medium, and the storage medium stores a computer program, and when the processor executes the computer program, a deterministic data recording playback method of a distributed system based on event causal relationship is realized.

[0094] Those skilled in the art can understand that all or part of the steps of the foregoing method embodiments can be completed by relevant hardware of program instructions, and the foregoing program can be stored in a computer readable storage medium. When the program is executed, the program executes the steps of the foregoing method embodiments. The foregoing storage medium includes a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0095] The various embodiments are described in the specification by way of progression, each building on the last to facilitate ease of understanding. The embodiments are illustrative of the application but not limiting thereof. The same reference numerals in different drawings represent elements with similar or analogous functions.

[0096] The above description of disclosed embodiments enables a person skilled in the art to implement or use the application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for deterministic data logging and playback of distributed systems based on event causality, characterized in that, Comprise: Write a configuration file, which contains the unique label of each computing node; Computing nodes read the label attribute of the configuration file at startup, and initialize the logical clock; When an event occurs inside the computing node, the logical clock value is incremented by 1, and the event pair is recorded to the buffer; When the computing node outputs data externally, the event pair in the buffer and the output data are recorded as metadata, and then the buffer is emptied; Record the metadata and output events into the database through rosbag2; Offline processing of the database, determining the causal relationship of the output events according to the metadata, and sorting and filtering the messages according to the causal relationship of the output events to generate a new database for playback; The playback excutor controls the playback process according to the new database for playback, realizing the deterministic playback of the distributed system; Each computing node corresponds to a logical clock, which starts counting from 0. When an event occurs inside the computing node, the logical clock value is incremented by 1. Each event is represented by an event pair in the form of (label, logical timestamp); Offline processing of the database, determining the causal relationship of the output events according to the metadata, and sorting and filtering the messages according to the causal relationship of the output events to generate a new database for playback, specifically including: Extract the output data and its metadata of each computing node from the database; Extract the event pair from the metadata to form a set of trigger events; Match the trigger event sets of all computing nodes to determine the causal relationship of the output events; Sort and filter the messages according to the causal relationship of the output events to generate a new database for playback.

2. The event-causality-based distributed system deterministic data logging playback method of claim 1, wherein, Internal events of computing nodes include the following events: (a) As a subscriber, receive data sent by other nodes; (b) As a publisher, publish data externally; (c) As a client, send a request to the server; (d) As a server, receive a request from a client; (e) Periodic events inside the node.

3. The event-causality-based distributed system deterministic data recording playback method of claim 1, wherein, Control the playback process through the playback excutor to realize the deterministic playback of the distributed system, specifically including: The playback excutor controls the playback process according to the new database for playback. The events in the new database generated after offline processing are played back in a determined order; After the event data is played back, the computing node that receives the data notifies the excutor, which notifies the playback system that the next event data can be played back; If the next event is a periodic event, the excutor notifies the corresponding computing node to perform the corresponding periodic task; If the next event is a data transmission type event, the excutor notifies the playback system to publish the event data to the system; If the next event is a request type event, the excutor requests the corresponding computing node to perform the corresponding task; Execute all event data recorded in the database in the above logical order to realize the deterministic playback of the distributed system.

4. A computer device, comprising: Comprise: A memory and a processor, the memory having stored therein a computer program executable on the processor, the processor implementing the method of any one of claims 1 to 3 when executing the computer program.

5. A computer readable storage medium, characterized in that, The storage medium has stored thereon a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 3.

Images