Data recording method and system, electronic device and computer readable storage medium
By adopting a co-node filtering strategy and a ring buffer persistent storage mechanism in the autonomous driving system, the network load pressure problem caused by the centralized recording scheme is solved, and the real-time performance and reliability of data recording are improved.
Patent Information
- Application Number
- CN202511555534.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-29
- Publication Date
- 2026-01-23
AI Technical Summary
In the field of autonomous driving, centralized data recording schemes lead to a surge in cross-node communication traffic, consuming network bandwidth and affecting the real-time performance of data recording.
A same-machine node filtering strategy is adopted to control the recording physical nodes to subscribe to topic data on the same physical host, and to achieve real-time synchronization and reliable persistence of data through the collaborative mechanism of the ring buffer and persistent storage area.
It effectively reduces cross-node communication traffic, improves the real-time performance of data recording and system efficiency, and ensures the integrity and reliability of critical data.
Smart Images
Figure CN121397003A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data recording method, system, electronic device, and computer-readable storage medium. Background Technology
[0002] Due to its modular and distributed characteristics, the Robot Operating System (ROS) has become the mainstream development framework in fields such as robotics and autonomous driving. In ROS-based systems, reliable recording of multi-source topic data generated by distributed nodes is crucial for system debugging, behavior analysis, and problem backtracking.
[0003] In related technologies, the autonomous driving field typically adopts a centralized architecture-based data recording scheme to record multi-source topic data. When the system scales up and the nodes are distributed across different physical hosts, the centralized recording mode can easily lead to a surge in cross-node communication traffic, which consumes a large amount of network bandwidth, resulting in communication delays and affecting the real-time performance of data recording. Summary of the Invention
[0004] To address or partially address the problems existing in related technologies, this application provides a data recording method, system, electronic device, and computer-readable storage medium, which can effectively reduce cross-node communication traffic, avoid network load pressure caused by centralized recording, ensure the reliability of data recording in emergency scenarios, and effectively improve the overall efficiency of the recording system and the real-time performance of data recording.
[0005] The first aspect of this application provides a data recording method, including: Obtain a configuration file; wherein the configuration file defines at least one data recording scenario; The data recording rules for recording physical nodes are determined based on the configuration file; the data recording rules include at least: recording trigger conditions, target topic set, and backtracking duration; Based on the same-machine node filtering strategy, the recording physical node is controlled to subscribe to topics in the target topic set published by other ROS nodes located on the same physical host as its local node; The data from each of the subscribed topics is continuously fed into a preset circular buffer, and all recording trigger conditions are monitored. When any of the recording trigger conditions is triggered, the target topic to be recorded and the rewind duration are determined according to the data recording rules corresponding to the triggered recording trigger condition. Based on the backtracking duration, historical data of the target topic corresponding to it is extracted from the circular cache and synchronized with the real-time data of the target topic.
[0006] In some implementations, the data recording scenario includes at least one of the following: an external command trigger scenario, a scenario event trigger scenario, and a system status trigger scenario.
[0007] In some implementations, before retrieving historical data of the target topic from the circular buffer according to the backtracking duration and performing a data synchronization operation together with the real-time data of the target topic, the method further includes: Obtain the liveness status information of other ROS nodes located on the same physical host as the recording physical node, and update the target topic to be recorded based on the liveness status information.
[0008] In some implementations, the step of retrieving historical data of the target topic from the circular cache according to the backtracking duration and performing data synchronization operations together with the real-time data of the target topic includes: Based on the backtracking duration, extract the corresponding historical data of the target topic from the circular cache and obtain the real-time data of the target topic; The historical data and the real-time data are input into a preset persistent storage area and divided into blocks according to a preset data block size value. After obtaining each data block, the data is written to disk.
[0009] In some implementations, before performing data write-to-disk after each data block is obtained, the method further includes: Monitor I / O load status; The step of writing data to disk after obtaining each data block includes: After obtaining each data block, the data is written to disk based on the I / O load status and the preset write speed threshold.
[0010] In some implementations, the method further includes: Listen for update events of the configuration file; In response to the detection of the update event, the changes to the configuration file are obtained, and the data recording rules are adjusted based on the changes.
[0011] In some implementations, before retrieving historical data of the target topic from the circular buffer according to the backtracking duration and performing a data synchronization operation together with the real-time data acquired in real time for the target topic, the method further includes: Monitor hardware resource status; the hardware resource status includes at least one of: disk space, CPU load, and memory utilization. Determine whether the preset recording start conditions are met based on the hardware resource status; If the conditions are met, the subsequent response process for each of the recording trigger conditions will continue; if the conditions are not met, the execution of the response process will be suppressed or delayed.
[0012] A second aspect of this application provides a data recording system, comprising: A scene configuration module is used to obtain a configuration file; wherein the configuration file defines at least one data recording scene; The rule configuration module is used to determine the data recording rules for recording physical nodes based on the configuration file; the data recording rules include at least: recording trigger conditions, target topic set, and backtracking duration; The topic subscription control module is used to control the recording physical node to subscribe to topics in the target topic set published by other ROS nodes located on the same physical host as its local node, based on the same node filtering strategy. The recording task execution module is used to continuously input the data of each subscribed topic into a preset circular buffer and monitor all recording trigger conditions; in response to any recording trigger condition being triggered, it determines the target topic to be recorded and the backtracking duration according to the data recording rules corresponding to the triggered recording trigger condition; according to the backtracking duration, it extracts the historical data of the corresponding target topic from the circular buffer and performs a data synchronization operation together with the real-time data of the target topic.
[0013] A third aspect of this application provides an electronic device, comprising: Processor; and A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.
[0014] A fourth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.
[0015] The technical solution provided in this application may include the following beneficial effects: The technical solution of this application, by setting a same-machine node filtering strategy, divides the data collection range of the recording physical nodes, and limits the communication traffic generated by communication between the same nodes to within a single machine, thereby effectively reducing the network bandwidth occupation caused by cross-host communication in a distributed system; and by configuring rules based on scenarios, when recording is triggered, historical data of the corresponding topic is extracted from the circular buffer according to the backtracking duration and saved synchronously with real-time data, thereby improving the integrity of key data capture and achieving reliable persistence in a distributed system, avoiding the network load pressure caused by centralized recording, and ensuring the reliability of data in sudden scenarios, effectively improving the overall efficiency of the recording system and the real-time performance of data recording.
[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0017] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.
[0018] Figure 1 This is a schematic flowchart illustrating the data recording method in an embodiment of this application; Figure 2 This is another schematic flowchart illustrating the data recording method in an embodiment of this application; Figure 3 This is another schematic flowchart illustrating the data recording method in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of the data recording system shown in the embodiments of this application; Figure 5 This is another schematic diagram of the data recording system shown in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation
[0019] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.
[0020] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.
[0021] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.
[0022] In related technologies, the autonomous driving field typically adopts a centralized architecture-based data recording scheme to record multi-source topic data. When the system scales up and the nodes are distributed across different physical hosts, the centralized recording mode can easily lead to a surge in cross-node communication traffic, which consumes a large amount of network bandwidth, resulting in communication delays and affecting the real-time performance of data recording.
[0023] To address the aforementioned issues, this application provides a data recording method that effectively reduces cross-node communication traffic, avoids network load pressure caused by centralized recording, ensures the reliability of data recording in emergency scenarios, and effectively improves the overall efficiency and real-time performance of the recording system.
[0024] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.
[0025] The data recording method of this application can be applied to multi-source data recording systems, which can be used in fields including but not limited to autonomous driving and robotics.
[0026] Figure 1 This is a schematic flowchart illustrating the data recording method in an embodiment of this application.
[0027] See Figure 1 The data recording method of this application includes: S101, Obtain the configuration file; wherein the configuration file defines at least one data recording scenario.
[0028] In this step, pre-built configuration files are obtained. Each configuration file defines at least one data recording scenario, and each data recording scenario corresponds to a data recording rule for a recording physical node.
[0029] It should be understood that each configuration file can define different data recording scenarios for multiple recording physical nodes, thereby enabling control of the overall data recording process of the system through a unified configuration file.
[0030] Data recording scenarios can include at least one of the following types: external command triggered scenarios, scenario event triggered scenarios, and system status triggered scenarios. It should be understood that data recording scenarios correspond to actual working conditions encountered by the system during operation that require data recording.
[0031] Among them, external command triggering scenarios can refer to application scenarios that start the data recording process by receiving instructions from the user. For example, during debugging, recording can be manually started to capture data corresponding to a specific operation. By defining external command triggering scenarios, the active control needs in manual intervention scenarios can be met.
[0032] Among them, scene event triggering scenarios can refer to application scenarios that automatically start the data recording process according to preset business logic or environmental change conditions. For example, when an emergency braking signal is detected in an autonomous driving system, the recording process is actively triggered to record the environmental perception data before and after the event. By defining scene event triggering scenarios, specific events that occur during system operation can be responded to in a timely manner.
[0033] Among them, system status triggering scenarios can refer to application scenarios that start the data recording process based on abnormal hardware or software resource states. For example, when the memory usage exceeds the safety threshold, recording can be automatically started to retain the running status data before the system crashes. By defining system status triggering scenarios, the system can respond and collect data even when a fault occurs, for use in system anomaly analysis.
[0034] The configuration file can be in a structured text format, such as a YAML or JSON text file. It declares the data recording rules for different scenarios. Specifically, the data recording rules should include at least the recording trigger conditions, the target topic set, and the playback duration corresponding to different data collection scenarios.
[0035] It is understandable that by setting the contents of the configuration file, the data recording scenarios of different practical applications can be expanded so that the data recording process can flexibly adapt to the recording needs of different application scenarios, thereby effectively reducing the problem of response lag or omission in the data recording process in complex scenarios.
[0036] S102, determine the data recording rules for recording physical nodes according to the configuration file; the data recording rules include at least: recording trigger conditions, target topic set, and backtracking duration.
[0037] In this step, based on the contents declared in the configuration file, the data recording rules for recording physical nodes are determined, including at least the recording trigger conditions, the target topic set, and the backtracking duration.
[0038] It is understandable that each recording physical node corresponds to at least one exclusive data recording rule.
[0039] In this application, the number of recording physical nodes is at least one. That is, the method of this application can be applied to data recording control of a single recording physical node or multiple recording physical nodes.
[0040] S103, based on the same-machine node filtering strategy, controls the recording physical node to subscribe to topics in the target topic set published by other ROS nodes located on the same physical host as its local node.
[0041] In this step, based on a pre-set same-machine node filtering strategy, the recording physical node is controlled to subscribe only to topics published by other ROS nodes located on the same physical host as its local node.
[0042] It can be understood that each recording physical node subscribes to topics from other ROS nodes located on the same physical host, and the topics subscribed to by each recording physical node belong to the target topic set defined by the configuration file.
[0043] The co-host node filtering strategy can involve pre-analyzing the physical hosts of all recording physical nodes and other ROS nodes, matching each recording physical node with other ROS nodes that share the same physical host. This controls each recording physical node to only record topics published by other ROS nodes on its own machine, effectively avoiding cross-node communication. It should be understood that each physical host has a unique identifier; the physical host of the corresponding node can be obtained by querying the host identifier field in the node registration information.
[0044] S104 continuously inputs the data of each subscribed topic into the preset circular buffer and monitors all recording trigger conditions.
[0045] In this step, real-time data of each topic subscribed to by the recording physical node is continuously input into a preset circular buffer, and all recording trigger conditions are continuously detected.
[0046] It should be understood that when there are multiple recording physical nodes, the topic data subscribed to by all recording physical nodes are stored in the same or different circular caches in real time.
[0047] In this context, a circular buffer can refer to a storage structure with a fixed capacity and a circular overlay mechanism. For example, a circular buffer can be configured to store messages with a 30-second time threshold. After receiving a message from a subscribed topic, the received message is placed in the circular buffer for storage. When the time difference between the message at the head of the circular buffer queue and the current message reaches 30 seconds, the message at the head of the queue is removed and the currently received message is stored.
[0048] S105, in response to any recording trigger condition being triggered, determine the target topic to be recorded and the rewind duration according to the data recording rules corresponding to the triggered recording trigger condition.
[0049] In this step, after any recording trigger condition is triggered, the set of target topics to be recorded and the rewind duration corresponding to the triggered recording trigger condition are determined according to the pre-set data recording rules.
[0050] The target topic to be recorded can be determined from a set of corresponding target topics. It can be understood that the target topic to be recorded for each recording trigger condition can be the same or different. Furthermore, there can be one or more target topics to be recorded.
[0051] S106. Based on the backtracking duration, extract the historical data of the corresponding target topic from the circular cache and perform a data synchronization operation together with the real-time data of the target topic.
[0052] In this step, based on the determined backtracking duration, historical data corresponding to the target topic and backtracking duration are extracted from the circular cache and synchronized with the real-time data currently acquired for the target topic to achieve data recording.
[0053] The collection duration of real-time data for the target topic can be the same as or different from the backtracking duration. For example, if the backtracking duration is five seconds, the historical data of the target topic for the most recent five seconds can be extracted from the cache and merged with the real-time data of the target topic for the next three seconds for recording.
[0054] In this case, the historical data and real-time data of the target topic can be synchronized together by first integrating them according to the time series and then performing block storage operations.
[0055] In some implementations, before retrieving historical data of the corresponding target topic from the circular buffer based on the backtracking duration and performing data synchronization operations together with the real-time data acquired in real time for the target topic, the liveness status information of other ROS nodes located on the same physical host as the recording physical node can also be obtained, and the target topic to be recorded can be updated based on the liveness status information. Here, liveness status information can refer to the running status data of the ROS nodes on the physical host.
[0056] For example, the heartbeat status of a node can be obtained through the node status query interface provided by the ROS node manager, thereby determining whether the node is actively running or has terminated abnormally.
[0057] By using the liveness information of ROS nodes, we can identify ROS nodes that have stopped running. By removing the topics published by these stopped nodes from the target topics, we can avoid retrieving historical data of invalid topics from the ring buffer, thereby ensuring that the recorded data comes only from currently active ROS nodes, and further reducing the resource consumption of writing invalid data to disk. In this embodiment, the data recording method of this application, by setting a peer node filtering strategy, divides the data collection range of the recording physical nodes, and restricts the communication traffic generated by communication between the same nodes to within a single machine, thereby effectively reducing the network bandwidth occupation caused by cross-host communication in a distributed system; and by configuring scenario-based rules, when recording is triggered, historical data of the corresponding topic is extracted from the circular buffer according to the backtracking duration and saved synchronously with real-time data, thereby improving the integrity of key data capture and achieving reliable persistence in a distributed system, avoiding the network load pressure caused by centralized recording, and ensuring the reliability of data recording in sudden scenarios, effectively improving the overall efficiency of the recording system and the real-time performance of data recording.
[0058] Figure 2 This is another schematic flowchart illustrating the data recording method in an embodiment of this application. The embodiments of this application... Figure 1 The following is a further explanation based on the illustrated embodiment.
[0059] See Figure 2 The data recording method of this application includes: S201, Obtain the configuration file; wherein the configuration file defines at least one data recording scenario.
[0060] In this step, the pre-built configuration file is obtained.
[0061] S202, determine the data recording rules for recording physical nodes according to the configuration file; the data recording rules include at least: recording trigger conditions, target topic set, and backtracking duration.
[0062] In this step, based on the contents declared in the configuration file, the data recording rules for recording physical nodes are determined, including at least the recording trigger conditions, the target topic set, and the backtracking duration.
[0063] S203, based on the same-machine node filtering strategy, controls the recording physical node to subscribe to topics in the target topic set published by other ROS nodes located on the same physical host as its local node.
[0064] In this step, based on a pre-set same-machine node filtering strategy, the recording physical node is controlled to subscribe only to topics published by other ROS nodes located on the same physical host as its local node.
[0065] S204 continuously inputs the data from each subscribed topic into a preset circular buffer and monitors all recording trigger conditions.
[0066] In this step, real-time data of each topic subscribed to by the recording physical node is continuously input into a preset circular buffer, and all recording trigger conditions are continuously detected.
[0067] S205, in response to any recording trigger condition being triggered, determine the target topic to be recorded and the rewind duration according to the data recording rules corresponding to the triggered recording trigger condition.
[0068] In this step, based on the determined backtracking duration, historical data corresponding to the target topic and backtracking duration are extracted from the circular cache and synchronized with the real-time data currently acquired for the target topic to achieve data recording.
[0069] S206, based on the backtracking duration, extract the historical data of the target topic from the circular cache and obtain the real-time data of the target topic.
[0070] In this step, based on the determined backtracking duration, historical data corresponding to the target topic and backtracking duration are extracted from the circular cache, and real-time data of the target topic is collected.
[0071] The historical data for the target topic corresponds to the playback duration, while the real-time data for the target topic corresponds to the pre-set recording duration. It should be understood that the pre-set recording duration may be the same as or different from the playback duration.
[0072] S207: Input historical data and real-time data into the preset persistent storage area, divide it into blocks according to the preset data block size value, and execute data write to disk after obtaining each data block.
[0073] In this step, the acquired historical data and real-time data are input into the preset persistent storage area. In the persistent storage area, the data to be recorded is first sorted and arranged in chronological order, and then divided into blocks according to the preset data block size value. Once a data block of the corresponding size is obtained, the disk write operation is performed.
[0074] The preset data block size can be a fixed number of bytes or a dynamic value obtained according to preset rules. For example, the preset data block size can be a fixed 10MB. Alternatively, the preset data block size can be a dynamic value that is dynamically associated with the I / O load rate.
[0075] In this process, after obtaining a data block of the appropriate size in the persistent storage area, a forced disk write operation can be performed by calling the file synchronization system interface, thereby ensuring the storage integrity of each data block.
[0076] It is understood that the data recording method of this application uses a circular buffer to perform overlay circular caching of historical data, and sets up a persistent storage area to organize and divide the continuous data to be recorded into blocks before writing it to the disk. The circular buffer and the persistent storage area form a dual-buffering mechanism, which can effectively reduce the system space occupied and reduce the occurrence of I / O blocking. At the same time, it can reduce the data loss rate when the system crashes and improve the reliability of the recorded data.
[0077] In some implementations, the I / O load status can be monitored before each data block is written to disk; after each data block is obtained, the data is written to disk based on the I / O load status and a preset write speed threshold.
[0078] Among them, I / O load status can refer to the current data read and write pressure status of the disk subsystem. For example, I / O load status can be the disk I / O load rate, and the current disk data writing capacity can be judged by I / O load status; while the preset write speed threshold can refer to the preset maximum safe write rate of the disk.
[0079] It is understandable that during the data synchronization process, before each data block is written to the disk after being processed in the persistent storage area, the current disk I / O load rate is obtained first, and the data block is written according to the corresponding preset write speed threshold based on the I / O load rate.
[0080] It should be understood that when the I / O load status indicates that the current disk data write load exceeds a preset safety value, the amount of data written per block can be reduced to below a preset write speed threshold. Conversely, when the load is below the preset safety value, the disk write operation is allowed to proceed at the maximum safe rate (i.e., the preset write speed threshold). This method achieves a dynamic adjustment mechanism for data writes to the disk, thus preventing performance overload under high load and fully utilizing idle resources to improve write efficiency under low load.
[0081] In some embodiments, the data recording method of this application may further include, before executing the response process in response to any recording trigger condition being triggered: Monitor hardware resource status; hardware resource status includes at least one of disk space, CPU load, and memory usage; determine whether the preset recording start conditions are met based on the hardware resource status; if met, continue to execute the response process when any recording trigger condition is triggered; if not met, suppress or delay the execution of the response process.
[0082] In other words, before the recording trigger condition is triggered, key indicators such as disk space, CPU load, and memory usage are continuously collected to assess the current hardware resource status of the system. When the recording trigger condition is detected, the current hardware resource status is compared with the preset recording start conditions. If the conditions are met, the system's hardware resources are deemed sufficient, and subsequent response processes are allowed to proceed. If the conditions are not met, the system's hardware resources are deemed insufficient, and subsequent response processes are either blocked or added to a delay queue. Through this system resource monitoring and task suppression mechanism, the execution strategy of the response process can be proactively adjusted when the system load is too high, thereby avoiding system performance degradation caused by resource contention and ensuring the reliability of the critical data recording process. Furthermore, the data recording rules in the data recording method of this application may also include task priority. Task priority can refer to a level parameter used to identify the importance of recording trigger conditions. For example, it can be a numerical value used to classify different recording trigger conditions, such as assigning task priorities from 1 to 10, with higher values indicating higher priority.
[0083] Upon determining that the current system's hardware resources are insufficient, the priority of the currently triggered recording condition can be compared with the corresponding recording trigger conditions of existing data recording tasks. The execution order is then determined based on priority. If the currently triggered recording condition has a higher priority, the response process corresponding to the currently triggered recording condition is executed after pausing existing low-priority tasks to release resources. If the currently triggered recording condition has a lower priority, the response process corresponding to the currently triggered recording condition is either blocked from execution or added to a delay queue to postpone execution.
[0084] Figure 3 This is another schematic flowchart illustrating the data recording method in an embodiment of this application.
[0085] See also Figure 3 In some embodiments, the data recording method of this application may further include: S208 listens for configuration file update events. In response to detecting an update event, it obtains the changed content of the configuration file and adjusts the data recording rules based on the changed content.
[0086] In this step, by monitoring configuration file update events in real time, when the configuration file is modified, the system retrieves the changed content and readjusts the data recording rules accordingly. It should be understood that after the data recording rules are readjusted, the system performs relevant data extraction and recording operations based on the revised rules.
[0087] This involves loading the latest configuration file and performing a difference analysis with the older version to identify changes. Essentially, when a configuration file is modified, the older version is retained but not updated. The changes are revealed by performing a difference analysis between the latest and previous versions. These changes can include, but are not limited to, added, deleted, or modified configuration entries.
[0088] One key feature is the ability to adjust data recording rules based on changes made through hot updates. This means that changes can be made to data recording rules in real time without requiring a service restart, effectively preventing data loss or redundant storage due to configuration lag.
[0089] For example, if a new data recording scenario definition is detected in the configuration file, the corresponding data recording rule for the new data recording scenario will be automatically created; as another example, if the backtracking duration parameter of an existing data recording rule is detected to be modified, the corresponding backtracking duration in the existing data recording rule will be dynamically adjusted.
[0090] In this embodiment, the data recording method of this application divides the historical and real-time data to be stored into blocks of a preset size and performs disk write operations block by block. This breaks down the traditional single large-scale I / O into multiple small-scale I / Os, which can effectively distribute the system I / O load and avoid I / O blocking caused by concentrated disk flushing. At the same time, by writing each data block to disk in real time, the residence time of dirty pages in the system cache can be shortened, thereby effectively reducing the data loss rate caused by unexpected system failures. Furthermore, the dual-cache architecture, which uses a persistent storage area and a ring cache area in collaboration, can ensure data backtracking capability and alleviate the sudden pressure of data flow, enabling the system to have both high-throughput real-time data reception and high-reliability data storage capabilities.
[0091] Corresponding to the aforementioned application function implementation method embodiments, this application also provides a data recording system, an electronic device, and corresponding embodiments.
[0092] Figure 4 This is a schematic diagram of the structure of a data recording system shown in an embodiment of this application.
[0093] See Figure 4The data recording system 300 of this application includes: a scene configuration module 310, a rule configuration module 320, a topic subscription control module 330, and a recording task execution module 340.
[0094] The scene configuration module 310 is used to obtain a configuration file; wherein the configuration file defines at least one data recording scene.
[0095] In some implementations, the data recording scenario includes at least one of the following: external command triggered scenario, scenario event triggered scenario, and system status triggered scenario.
[0096] The rule configuration module 320 is used to determine the data recording rules for recording physical nodes based on the configuration file; the data recording rules include at least: recording trigger conditions, target topic set, and backtracking duration.
[0097] The topic subscription control module 330 is used to control the recording physical node to subscribe to topics in the target topic set published by other ROS nodes located on the same physical host as its local node, based on the same node filtering strategy.
[0098] The recording task execution module 340 is used to continuously input the data of each subscribed topic into a preset circular buffer and monitor all recording trigger conditions. When any recording trigger condition is triggered, the target topic to be recorded and the backtracking duration are determined according to the data recording rules corresponding to the triggered recording trigger condition. Based on the backtracking duration, the historical data of the corresponding target topic is extracted from the circular buffer and a data synchronization operation is performed together with the real-time data of the target topic.
[0099] In some implementations, the recording task execution module 340 may also obtain the survival status information of other ROS nodes located on the same physical host as the recording physical node before extracting the historical data of the corresponding target topic from the circular buffer according to the backtracking duration and performing data synchronization operation together with the real-time data of the target topic, and update the target topic to be recorded according to the survival status information.
[0100] In some implementations, the recording task execution module 340 can also extract historical data of the target topic from the circular cache and obtain real-time data of the target topic according to the backtracking duration; input the historical data and real-time data into the preset persistent storage area and divide them into blocks according to the preset data block size value, and execute data write to disk after obtaining each data block.
[0101] In some implementations, the recording task execution module 340 may also monitor the I / O load status before writing the data to disk after obtaining each data block; and then, after obtaining each data block, write the data to disk according to the I / O load status and a preset write speed threshold.
[0102] In some implementations, the recording task execution module 340 may also monitor the hardware resource status before extracting the historical data of the corresponding target topic from the circular buffer according to the backtracking duration and performing a data synchronization operation together with the real-time data obtained in real time for the target topic; the hardware resource status includes at least one of disk space, CPU load, and memory occupancy; the hardware resource status is used to determine whether the preset recording start conditions are met; if met, the subsequent response process when any recording trigger condition is triggered continues to be executed; if not met, the execution of the response process is suppressed or delayed.
[0103] Figure 5 This is another schematic diagram of the data recording system shown in the embodiments of this application.
[0104] See also Figure 5 The data recording system 300 of this application includes: a scene configuration module 310, a rule configuration module 320, a topic subscription control module 330, a recording task execution module 340, and a rule update module 350.
[0105] The rule update module 350 is used to listen for update events of the configuration file; in response to the detection of an update event, it obtains the changed content of the configuration file and adjusts the data recording rules based on the changed content.
[0106] In this embodiment, the data recording system of this application, by setting a same-machine node filtering strategy, divides the data collection range of the recording physical nodes, and restricts the communication traffic generated by communication between the same nodes to within a single machine, thereby effectively reducing the network bandwidth occupation caused by cross-host communication in a distributed system; and by configuring scenario-based rules, when recording is triggered, historical data of the corresponding topic is extracted from the circular buffer according to the backtracking duration and saved synchronously with real-time data, thereby improving the integrity of key data capture and achieving reliable persistence in a distributed system, avoiding the network load pressure caused by centralized recording, and ensuring the reliability of data in sudden scenarios, effectively improving the overall efficiency of the recording system and the real-time performance of data recording.
[0107] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.
[0108] Figure 6 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application.
[0109] See Figure 6 The electronic device 1000 includes a memory 1010 and a processor 1020.
[0110] The processor 1020 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.
[0111] Memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by processor 1020 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 1010 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.
[0112] The memory 1010 stores executable code, which, when processed by the processor 1020, can cause the processor 1020 to execute part or all of the methods described above.
[0113] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.
[0114] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon, which, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.
[0115] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.
Claims
1. A data recording method, characterized by, The method comprises: obtaining a configuration file; wherein the configuration file defines at least one data recording scene; determining a data recording rule of a recording physical node according to the configuration file; the data recording rule at least comprises a recording trigger condition, a target topic set and a rollback time length; controlling the recording physical node to subscribe to topics in the target topic set published by other ROS nodes located on the same physical host as the local node of the recording physical node based on a same-machine node screening strategy; continuously inputting data of each subscribed topic to a preset ring buffer area, and monitoring all recording trigger conditions; in response to any recording trigger condition being triggered, determining a target topic to be recorded and a rollback time length according to the data recording rule corresponding to the triggered recording trigger condition; extracting historical data of the target topic from the ring buffer area according to the rollback time length, and performing a data synchronization operation together with real-time data of the target topic.
2. The method of claim 1, wherein, The data recording scene comprises at least one of the following: an external command triggering scene, a scene event triggering scene and a system state triggering scene.
3. The method of claim 1, wherein, Before the step of extracting historical data of the target topic from the ring buffer area according to the rollback time length, and performing a data synchronization operation together with real-time data of the target topic, the method further comprises: obtaining survival state information of other ROS nodes located on the same physical host as the recording physical node, and updating the target topic to be recorded according to the survival state information.
4. The method of claim 1, wherein, The step of extracting historical data of the target topic from the ring buffer area according to the rollback time length, and performing a data synchronization operation together with real-time data of the target topic, comprises: extracting historical data of the target topic from the ring buffer area according to the rollback time length, and obtaining real-time data of the target topic; inputting the historical data and the real-time data to a preset to-be-persistently-stored area to sequentially block according to a preset data block size value, and performing data landing after obtaining each data block.
5. The method of claim 4, wherein, Before the step of performing data landing after obtaining each data block, the method further comprises: monitoring an I / O load state; The step of performing data landing after obtaining each data block, comprises: after obtaining each data block, performing data landing according to the I / O load state and a preset write speed threshold.
6. The method of claim 1, wherein, The method further comprises: listening to an update event of the configuration file; in response to detecting the update event, obtaining changed content of the configuration file, and adjusting the data recording rule based on the changed content.
7. The method according to any one of claims 1 to 6, characterized in that, Before the step of extracting historical data of the target topic from the ring buffer area according to the rollback time length, and performing a data synchronization operation together with real-time data of the target topic, the method further comprises: monitoring a hardware resource state; the hardware resource state comprises at least one of the following: disk space, CPU load and memory occupancy rate; determining whether a preset recording start condition is met according to the hardware resource state; If yes, continue to execute a subsequent response process when any of the recording trigger conditions is triggered; if no, suppress or delay the execution of the response process.
8. A data recording system characterized by comprising: The method comprises the following steps: A scene configuration module is configured to obtain a configuration file; wherein the configuration file defines at least one data recording scene; A rule configuration module is configured to determine a data recording rule of a recording physical node according to the configuration file; the data recording rule at least comprises a recording trigger condition, a target topic set and a backtracking time length; A topic subscription control module is configured to control the recording physical node to subscribe to topics in the target topic set published by other ROS nodes on the same physical host based on an in-machine node screening strategy; A recording task execution module is configured to continuously input data of each subscribed topic to a preset ring buffer area, and monitor all recording trigger conditions; in response to any recording trigger condition being triggered, determine a target topic to be recorded and a backtracking time length according to the data recording rule corresponding to the triggered recording trigger condition; according to the backtracking time length, extract historical data of the corresponding target topic from the ring buffer area, and perform a data synchronization operation together with real-time data of the target topic.
9. An electronic device, comprising: The method comprises the following steps: A processor; And A memory having executable code stored thereon, wherein when the executable code is executed by the processor, the processor executes the method of any one of claims 1-7. 10.A computer readable storage medium having executable code stored thereon, wherein when the executable code is executed by a processor of an electronic device, the processor executes the method of any one of claims 1-7.