Data recording method and device, storage medium and electronic equipment

Abstract

The invention provides a data recording method and device, a storage medium and electronic equipment, and the method comprises the steps: continuously recording and caching a group of data frames of a specified data stream; continuously monitoring each specified data stream to judge whether a preset triggering condition corresponding to each specified data stream is met or not based on the data frame of each specified data stream; the preset trigger condition corresponding to each specified data stream indicates that each specified data stream contains an abnormal data event; under the condition that a preset triggering condition corresponding to a target data stream is determined to be satisfied based on a first data frame of the target data stream in the group of specified data streams, determining a target time interval in which the target data stream is about to fall into a disk; the data frames of the target data stream in the target time interval are extracted from the cached data frames of the set of specified data streams, the to-be-stored data are obtained, the to-be-stored data are stored in the database, and the problem that the IO resource overhead cannot be reduced while the data integrity cannot be guaranteed in the prior art is solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of autonomous driving technology, and more specifically, to a data recording method, device, storage medium, and electronic device. Background Art

[0002] In autonomous driving, especially high-level autonomous driving, data recording is a common method for problem discovery. During real vehicle testing, some specific scenarios or problem scenarios are difficult to reproduce. Data recording can save all kinds of data recordings involved in the algorithm operation at that time, and reproduce and simulate them in the local development environment, making it easier for developers to locate problems and optimize algorithms.

[0003] Currently available data recording methods mainly include recording by manually inputting commands, recording at the time when a problem occurs, or continuously recording data.

[0004] However, manual recording may miss data before the problem occurs, making it difficult to analyze the problem; although continuous recording can ensure data integrity, a large amount of redundant data also causes great trouble for problem analysis, and continuous recording occupies more IO resources, which will also affect the normal operation of the autonomous driving algorithm. Therefore, the existing solutions cannot reduce IO resource overhead while ensuring data integrity. Summary of the invention

[0005] The embodiments of the present application provide a data recording method, device, storage medium and electronic device to at least solve the problem that the data recording method in the related art cannot reduce IO resource overhead while ensuring data integrity.

[0006] According to one aspect of an embodiment of the present application, a data recording method is provided, including: continuously recording and caching data frames of a group of specified data streams; each specified data stream in the group of specified data streams corresponds to a subscribed topic; continuously monitoring each specified data stream to determine whether a preset trigger condition corresponding to each specified data stream is satisfied based on the data frames of each specified data stream; the preset trigger condition corresponding to each specified data stream refers to that each specified data stream contains an abnormal data event; when it is determined based on the first data frame of the target data stream in the group of specified data streams that the preset trigger condition corresponding to the target data stream is satisfied, determining a target time interval for the target data stream to be written to disk; the target time interval includes the recording time of the first data frame; extracting the data frames of the target data stream within the target time interval from the cached data frames of the group of specified data streams to obtain the data to be written to disk, and storing the data to be written to disk in a database.

[0007] According to another aspect of an embodiment of the present application, a data recording device is provided, including: a recording module, used to continuously record and cache data frames of a group of specified data streams; each specified data stream in the group of specified data streams corresponds to a subscribed topic; a monitoring module, used to continuously monitor each specified data stream to determine whether a preset trigger condition corresponding to each specified data stream is satisfied based on the data frames of each specified data stream; the preset trigger condition corresponding to each specified data stream refers to that each specified data stream contains an abnormal data event; the recording module is also used to determine a target time interval for the target data stream to be written to disk when it is determined based on the first data frame of the target data stream in the group of specified data streams that the preset trigger condition corresponding to the target data stream is satisfied; the target time interval includes the recording time of the first data frame; the data frames of the target data stream within the target time interval are extracted from the cached data frames of the group of specified data streams to obtain the data to be written to disk; the writing module is used to store the data to be written to disk in a database.

[0008] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned data recording method when running.

[0009] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the data recording method through the computer program.

[0010] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in any of the above data recording method embodiments are implemented.

[0011] Through the embodiments of the present application, by continuously recording and caching a group of data frames of a specified data stream, it can be ensured that when the trigger condition is met, complete data before and after the problem occurs can be obtained to ensure data integrity, and each specified data stream is continuously monitored to ensure that the specified data stream can be immediately monitored when an abnormal data event occurs. At the same time, only when it is determined based on the first data frame of the target data stream in a group of specified data streams that the preset trigger condition corresponding to the target data stream is met, the data frames of the target data stream in the target time interval are extracted from the cached data frames to obtain the data to be written to the disk, and the data to be written to the disk is stored in the database. Targeted writing to the disk according to the preset trigger condition can avoid continuously and indiscriminately writing all data frames to the disk, and effectively reduce redundant data. That is, the embodiments of the present application adopt a triggered method to automatically write to the disk, which can effectively reduce redundant data, reduce IO resource occupation and waste of storage space while ensuring data integrity, thereby solving the problem in the related art that it is impossible to reduce IO resource overhead while ensuring data integrity. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a hardware structure block diagram of an optional data recording method of an embodiment of the present application;

[0013] Figure 2 is a flowchart of an optional data recording method according to an embodiment of the present application;

[0014] Figure 3 is an optional data recording architecture diagram according to an embodiment of the present application;

[0015] Figure 4 is a flow chart of an optional data recording according to an embodiment of the present application;

[0016] Figure 5 is a flow chart of an optional monitoring module performing monitoring according to an embodiment of the present application;

[0017] Figure 6 is a flowchart of an optional recording module performing recording according to an embodiment of the present application;

[0018] Figure 7 is a flow chart of performing disk placement by an optional disk placement module according to an embodiment of the present application;

[0019] Figure 8 is a structural block diagram of an optional data recording device according to an embodiment of the present application;

[0020] Fig. 9 It is a structural block diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] According to one aspect of an embodiment of the present application, a data recording method is provided. Optionally, in this embodiment, the data recording method can be applied to Figure 1 In the hardware environment composed of the vehicle terminal 102 and the server 104 shown in FIG. Figure 1 As shown, the server 104 is connected to the vehicle-mounted terminal 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or a client installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 104.

[0024] The above network may include but is not limited to at least one of the following: wired network, wireless network. The above wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above wireless network may include but is not limited to at least one of the following: WIFI, Bluetooth.

[0025] The data recording method of the embodiment of the present application can be executed by the vehicle terminal 102, or can be executed by the server 104 and the vehicle terminal 102 together. Taking the vehicle terminal 102 executing the data recording method of the embodiment as an example, Figure 2 is a flow chart of an optional data recording method according to an embodiment of the present application, such as Figure 2As shown, the process of the method may include the following steps:

[0026] Step S202, continuously recording and caching data frames of a group of specified data streams; each specified data stream in the group of specified data streams corresponds to a subscribed topic.

[0027] The designated data stream refers to the information stream continuously output from components such as sensors, actuators, and vehicle controllers in the autonomous driving system, including but not limited to images, radar data, vehicle status, algorithm output, etc. The designated data stream is identified by the subscribed topic to ensure that the designated data stream can be accurately identified and received during the recording process.

[0028] Optionally, Figure 3 is a diagram of the architecture of data recording in one embodiment, such as Figure 3 As shown, the vehicle terminal includes a monitoring module, a recording module, and a disk storage module. Figure 4 is a flow chart of data recording in one embodiment, such as Figure 4 As shown, after the recording module is started, different data streams are identified by topics in advance, and the topics corresponding to the specified data streams are subscribed. During the autonomous driving process, the recording module obtains a set of data frames of the specified data streams corresponding to the subscribed topics, and caches a set of data frames of the specified data streams into the memory.

[0029] Step S204, continuously monitor each designated data stream to determine whether a preset trigger condition corresponding to each designated data stream is satisfied based on the data frame of each designated data stream; the preset trigger condition corresponding to each designated data stream means that each designated data stream contains an abnormal data event.

[0030] The preset trigger conditions refer to a set of rules or conditions set in advance in the autonomous driving data recording system to trigger the data recording operation. The preset trigger conditions are set based on the monitoring and analysis of each data stream (i.e., topic data) during the autonomous driving process, and are used to identify anomalies or important events that may occur in a specific data stream. When a data frame of a specified data stream is monitored to meet a preset trigger condition, the data recording operation will be triggered to save the relevant data for subsequent analysis and optimization.

[0031] Abnormal data events refer to data that does not conform to normal conditions or exceeds the expected range in a specified data stream during the actual autonomous driving process. Abnormal data events are defined based on preset trigger conditions, and are used to describe the data definition of each specified data stream in a set of specified data streams and the trigger conditions that are met. Among them, the trigger conditions may be specific field anomalies, frame rate anomalies, data delays, etc. in the data frames of a set of specified data streams. For example, the trigger condition may be an abnormal frame rate data for the reception and transmission of a certain topic data, with a value that is too large or too small; or a field data in a certain topic data frame at a certain moment exceeds a specified threshold.

[0032] Optionally, Figure 5 FIG. 1 is a flow chart of a monitoring module performing monitoring in an embodiment. Figure 5 As shown, the monitoring module loads the preset trigger conditions from the configuration and continuously monitors each specified data stream, traverses all the preset trigger conditions according to the monitored specified data stream to determine whether there are abnormal data events in each specified data stream, and in the case of abnormal data events, determines that the preset trigger conditions corresponding to the specified data stream are met; otherwise, determines that the preset trigger conditions corresponding to the specified data stream are not met.

[0033] Step S206, when it is determined based on the first data frame of the target data stream in a group of specified data streams that the preset trigger condition corresponding to the target data stream is met, determine the target time interval for the target data stream to be recorded on the disk; the target time interval includes the recording time of the first data frame.

[0034] The target data refers to a specified data stream in a group of specified data streams that meets the corresponding preset trigger condition.

[0035] The first data frame refers to a data frame in which an abnormal data event exists in the target data flow, that is, a data frame that causes the target data flow to meet a corresponding preset trigger condition.

[0036] During the autonomous driving data recording process, when a data frame (i.e., the first data frame) of a specified data stream (i.e., the target data stream) meets the preset trigger condition, it is necessary to intercept a section of data containing abnormal data events starting from the first data frame as the data to be stored on disk. The intercepted time period is the target time interval to be stored on disk.

[0037] Alternatively, if Figure 4As shown, when the monitoring module detects that a data frame (i.e., the first data frame) in a data stream (i.e., the target data stream) in a set of specified data streams meets the corresponding preset trigger condition, the trigger logic is executed, i.e., an instruction is sent to the recording module to notify the recording module to write the data to the disk. After receiving the instruction for instructing the data to be written to the disk, the recording module determines the data to be written to the disk including the first data frame, and determines the time interval corresponding to the data to be written to the disk as the target time interval for the target data stream to be written to the disk.

[0038] Step S208, extracting data frames of the target data stream within the target time interval from the cached data frames of a group of designated data streams, obtaining the data to be written to the disk, and storing the data to be written to the disk in the database.

[0039] The data to be stored on disk refers to a set of data frames of a target data stream within a target time interval extracted from a set of data frames of a specified data stream currently cached during the autonomous driving data recording process when the vehicle terminal detects that a preset trigger condition has been met (i.e. an abnormal data event has occurred).

[0040] Alternatively, if Figure 4 As shown, the recording module extracts the target data stream from the data frames of a set of specified data streams cached in the memory according to the subscription topic corresponding to the target data stream, and extracts the data frames of the target data stream within the target time interval from the data frames of a set of specified data streams currently cached, obtains the data to be written to the disk, and sends the data to be written to the disk writing module. The disk writing module writes the data to be written to the database.

[0041] By continuously recording and caching a group of data frames of a specified data stream through the above steps S202 to S206, it can be ensured that when the trigger condition is met, complete data before and after the problem occurs can be obtained to ensure data integrity, and each specified data stream is continuously monitored to ensure that the specified data stream can be immediately monitored when an abnormal data event occurs. At the same time, only when it is determined based on the first data frame of the target data stream in a group of specified data streams that the preset trigger condition corresponding to the target data stream is met, the data frame of the target data stream in the target time interval is extracted from the cached data frame to obtain the data to be written to the disk, and the data to be written to the disk is stored in the database. Targeted writing to the disk according to the preset trigger condition can avoid continuously and indiscriminately writing all data frames to the disk, and effectively reduce redundant data. That is, the embodiment of the present application adopts a triggered method to automatically write to the disk, which can effectively reduce redundant data, reduce IO resource occupation and waste of storage space while ensuring data integrity, thereby solving the problem in the related art that it is impossible to reduce IO resource overhead while ensuring data integrity.

[0042] In an exemplary embodiment, extracting data frames of a target data stream within a target time interval from a group of cached data frames of a specified data stream to obtain data to be written to a disk includes:

[0043] After waiting to continue recording for the first preset time length, data frames of the target data stream within the target time interval are extracted from the data frames of a cached set of specified data streams to obtain the data to be stored on the disk; the target time interval includes a first time interval before the recording time of the first data frame and a second time interval after the recording time of the first data frame, and the length of the second time interval is less than or equal to the first preset time length.

[0044] The first preset duration refers to the length of time set for waiting to continue recording after a specified data stream is monitored to meet the corresponding preset trigger condition (i.e., contains an abnormal data event) in order to ensure that sufficient data before and after the abnormal event can be captured for analysis. The first preset duration is pre-configured and is used to continue recording data during the waiting period so that a complete data set containing the abnormal event and related information before and after it can be extracted from the cache later. For example, the first preset duration can be 10 seconds.

[0045] The first time interval refers to the time period starting from before the recording time of the first data frame (i.e., the data frame corresponding to when the trigger condition is met) to the end of the recording time of the first data frame. The first time interval is selected to obtain data before the abnormal data event occurs, so as to analyze the cause and background of the abnormality.

[0046] The second time interval refers to a time period starting from the recording time of the first data frame and ending at a certain time after the recording time of the first data frame (the time is less than or equal to the end time of the first preset time length). The second time interval is selected to obtain data after the abnormal data event occurs, so as to observe the impact of the abnormal data event and subsequent changes.

[0047] Optionally, Figure 6 FIG. 1 is a flowchart of a recording module performing recording in one embodiment, such as Figure 6 As shown, the recording module starts after loading the configuration, accepts a set of data frames of a specified data stream corresponding to the subscribed topic, and caches a set of data frames of a specified data stream into the memory. After receiving the instruction sent by the monitoring module to instruct the data to be dropped to the disk, the recording module waits for the first preset time (such as 10 seconds) to continue recording, and then extracts the data frames of the target data stream within the target time interval from the cached set of data frames of the specified data stream to obtain the data to be dropped to the disk.

[0048] Through this embodiment, waiting to continue recording for the first preset duration, and the target time interval includes a first time interval before the recording time of the first data frame and a second time interval after the recording time of the first data frame, and the duration of the second time interval is less than or equal to the first preset duration. It can ensure that after an abnormal data event (i.e., the first data frame) is detected, not only the data of the abnormal data event itself is recorded, but also the data within a period of time before and after the event is included, so that the complete context of the occurrence of the abnormal data event can be captured, thereby ensuring data integrity and accuracy while minimizing the loss of redundant data to the disk, further improving resource utilization efficiency.

[0049] In an exemplary embodiment, extracting data frames of a target data stream within a target time interval from a group of cached data frames of a specified data stream to obtain data to be written to a disk includes:

[0050] Data frames of a target data stream are extracted from a cached set of data frames of a specified data stream; the extracted data frames of the target data stream are cached in a first cache queue; the first cache queue and a second cache queue that caches a set of data frames of a specified data stream are different cache queues; data frames within a target time interval are intercepted from the extracted data frames of the target data stream to obtain data to be stored on disk.

[0051] The first cache queue is a dedicated queue for caching data frames of the extracted target data stream.

[0052] The second cache queue is a general queue for caching data frames of a set of specified data streams being monitored. The second cache queue is created when the recording module is started and continuously receives data frames from various topics.

[0053] Alternatively, if Figure 6 As shown, when the monitoring module detects that a preset trigger condition is met, the recording module is notified to extract data. The recording module then extracts the data frames of the target data stream related to the preset trigger condition from the data frames of a set of specified data streams being cached in the second cache queue, and moves the data frames of the target data stream to the first cache queue. The recording module intercepts the data frames within the target time interval from the data frames of the target data stream extracted from the first cache queue to obtain the data to be stored on the disk.

[0054] Through this embodiment, the first cache queue and the second cache queue are used to cache data frames in a collaborative manner, and the data frames are stored in the memory without occupying disk consumption. The first cache queue is focused on processing data frames related to the trigger conditions, while the second cache queue is responsible for capturing and caching data frames of all specified data streams. This collaborative caching method of the first cache queue and the second cache queue can effectively reduce IO resource overhead while ensuring data integrity, thereby reducing the performance impact on algorithm operation on the actual vehicle.

[0055] In an exemplary embodiment, the data recording method further includes:

[0056] When it is determined based on the first data frame of the target data stream that the preset trigger condition corresponding to the target data stream is met, monitoring of a group of designated data streams is suspended until the duration of monitoring suspension reaches a second preset duration, and each designated data stream is continuously monitored.

[0057] The second preset duration is a specific time length, which is used to define the time length for suspending monitoring of a set of specified data streams after confirming that a preset trigger condition is met. The second preset duration is usually determined based on system requirements, characteristics of the data stream, and the specific meaning of the trigger condition. For example, the second preset duration can be 10 seconds.

[0058] Optionally, the monitoring module suspends monitoring of a group of specified data streams when it determines that a preset trigger condition corresponding to the target data stream is met based on the first data frame of the target data stream. After suspending monitoring, the monitoring module starts timing until the duration of the suspension of monitoring reaches a preset time length, i.e., a second preset time length, and the monitoring module resumes continuous monitoring of each specified data stream that was previously suspended.

[0059] Through this embodiment, when it is determined based on the first data frame of the target data stream that the preset trigger condition corresponding to the target data stream is met, monitoring is suspended for the first preset time period to prevent repeated triggering, avoid excessive reaction of the monitoring module due to instantaneous abnormalities or false alarms, and improve overall stability.

[0060] In an exemplary embodiment, a group of data frames of a specified data flow are buffered in a pre-established second buffer queue, and the second buffer queue is configured with a maximum buffer duration.

[0061] The maximum cache duration is a parameter in the second cache queue configuration, which refers to the maximum time that data is allowed to be retained in the second cache queue. When a data frame enters the second cache queue, the timing starts. If the data frame stays in the second cache queue for longer than the maximum cache duration, the data frame will be removed from the second cache queue to free up the second cache space for subsequent data frames. This configuration helps ensure that the second cache queue does not grow indefinitely, thereby avoiding excessive consumption of system resources.

[0062] In some embodiments, during the process of continuously recording and caching a set of data frames of a specified data stream, the data recording method further includes:

[0063] When the second data frame is to be written to the second cache queue, determine the interval duration between the timestamp of the first data frame in the second cache queue and the timestamp of the second data frame; when the interval duration between the timestamp of the first data frame and the timestamp of the second data frame is greater than or equal to the maximum cache duration, remove the first data frame.

[0064] The second data frame refers to a data frame that is received and added to the second cache queue, that is, the second data frame refers to a group of data frames of a specified data stream that are recorded and cached.

[0065] The head data frame in the second cache queue refers to the data frame at the front of the second cache queue in the second cache queue. The head data frame is the earliest data frame to enter the second cache queue and wait to be processed.

[0066] The timestamp of the head data frame indicates the time when the head data frame is received by the second cache queue.

[0067] The timestamp of the second data frame indicates the time when the second data frame is received by the second cache queue.

[0068] Optionally, the recording module obtains the timestamps of the first data frame and the second data frame, calculates the interval between the timestamp of the first data frame and the timestamp of the second data frame, and compares the calculated interval with the maximum cache duration configured for the second cache queue. If the interval duration is greater than or equal to the maximum cache duration, the recording module performs a removal operation, deletes the first data frame from the second cache queue, and updates the current cache memory capacity of the second cache queue. After completing the above steps, the second data frame is written into the second cache queue.

[0069] Through this embodiment, in the process of continuously recording and caching data frames of a group of specified data streams, when the interval between the timestamp of the head data frame of the second cache queue and the timestamp of the second data frame is greater than or equal to the maximum cache duration, the head data frame is removed. This can effectively manage the data in the second cache queue, ensure the timeliness and integrity of the data, and avoid outdated data from occupying too many resources.

[0070] In an exemplary embodiment, a group of data frames of a specified data flow are buffered in a second buffer queue, and the second buffer queue is configured with a maximum buffer memory capacity.

[0071] The maximum cache memory capacity configured for the second cache queue refers to the maximum memory limit of the data that can be stored in the second cache queue. When the amount of data in the second cache queue reaches or exceeds the maximum cache memory capacity, the cache is managed according to a specific strategy (such as removing the head of the queue data) to ensure the continuity and stability of data recording.

[0072] In some embodiments, during the process of continuously recording and caching a set of data frames of a specified data stream, the data recording method further includes:

[0073] When the current cache memory capacity of the second cache queue is greater than the maximum cache memory capacity, the head data frame in the second cache queue is removed, and the current cache memory capacity of the second cache queue is updated.

[0074] Optionally, the recording module obtains the current cache memory capacity of the second cache queue and compares the current cache memory capacity with the configured maximum cache memory capacity. If the current cache memory capacity is greater than the configured maximum cache memory capacity, the recording module performs a removal operation to delete the head data frame from the second cache queue and updates the current cache memory capacity of the second cache queue.

[0075] Through this embodiment, in the process of continuously recording and caching data frames of a set of specified data streams, when the current cache memory capacity of the second cache queue is greater than the maximum cache memory capacity, the head data frame in the second cache queue is removed, so as to effectively manage the data in the second cache queue, ensure the timeliness and integrity of the data, and avoid outdated data from occupying too many resources.

[0076] In an exemplary embodiment, each disc data is stored in the database in the form of a recording package; a meta information table and a data frame table are set in the database; the meta information table is used to record the metadata of the recording package stored in the database; the data frame table is used to record the information of the data frame in the recording package stored in the database.

[0077] Among them, the recording package refers to each disk data being organized and stored in the database in a specific format (for example, embedded sqlite database format). The recording package contains the specified topic data extracted from the first cache queue and the information within the time interval of the extracted specified topic data when a certain preset trigger condition is met. The embodiment of the present application adopts a dynamic recording method of trigger conditions, which can effectively reduce the size of the final recording package dropped to the disk, avoid interference from other redundant information, and facilitate subsequent algorithm problem location.

[0078] The common data recording package format is a binary file format, and both writing and reading require a specific software development kit and operating environment to support it, which is not convenient for later data processing. The recording package in the embodiment of the present application uses an embedded sqlite database as a storage format. As a lightweight relational database, sqlite occupies less resources, has good compatibility across platforms, and is easy to integrate.

[0079] The metadata of a recording package refers to the information about the recording package recorded in the meta information table. It includes the name of the recording package (used to identify and distinguish different recording packages), the total size (indicating the total amount of data in the recording package), the start timestamp and the end timestamp (recording the start and end time of the recording package data respectively), and the tag information of the trigger condition (used to indicate what the condition is that triggers the recording of the recording package). The metadata of the recording package provides an important reference and basis for the subsequent processing and analysis of the data.

[0080] The meta information table is a table in the database used to record the metadata of the recording package. Metadata is data about data, which provides basic information about the recording package. The meta information table records key information such as the name, total size, start timestamp, end timestamp, and tag information of the trigger condition of the recording package. For example, Table 1 is an example of a meta information table:

[0081] Table 1

[0082] Field Name type Occupied Bytes illustrate id int 4 Sequence number, default is 0 name char[] Indefinite length Recording package name size int 4 All dataframe sizes start_timestamp long 8 Start timestamp end_timestamp long 8 End timestamp trigger char[] Indefinite length Trigger condition corresponding label

[0083] The data frame table is a table in the database used to record the data frame information in the recording package. In the data recording of autonomous driving, the data frame is the basic unit of data. It contains data information from different topics and key information such as the timestamp of these data. The data frame table will record in detail the serial number of each data frame (used to identify and distinguish different data frames), the topic to which it belongs (indicating which topic the data frame comes from), the data size (indicating the amount of data in the data frame), the data timestamp (recording the reception time of the data frame), and the binary data (i.e. the original data content of the data frame). These data frame information provides important data support for subsequent algorithm problem location, data analysis, and data secondary processing. For example, Table 2 is an example of a data frame table:

[0084] Table 2

[0085] Field Name type Occupied Bytes illustrate id int 4 Data frame number topic int 4 Topic number size int 4 Number of data bytes data byte[] Indefinite length Binary Data timestamp long 8 Timestamp

[0086] In some embodiments, a topic data table is also provided in the database, which is an important component of the database for recording all topic information in the cached data. The topic data table includes the name of the topic, the number of data frames included in the topic, and the timestamps of these data frames. Table 3 is an example of a topic data table:

[0087] Table 3

[0088] Field Name type Occupied Bytes illustrate id int 4 Topic number name char[] Indefinite length Topic name size int 4 Number of data frames timestamp long 8 Timestamp

[0089] In some embodiments, storing the data to be stored in the database includes:

[0090] 1. Add metadata of the target recording package in the meta information table; the target recording package is a recording package of data to be written to the disk, and the metadata of the target recording package includes at least one of the following information of the target recording package: sequence number, recording package name, size of all data frames, start recording timestamp, end recording timestamp, and label of preset trigger condition corresponding to the target data stream.

[0091] The target recording package refers to the form in which the data to be written to the disk is stored in the database. The metadata of the target recording package contains detailed information about the recording package, such as the sequence number (used to identify and sort different recording packages), the recording package name (used to identify and distinguish different recording packages), the size of all data frames (provides an overview of the amount of data in the recording package), the start recording timestamp and the end recording timestamp (used to determine the time range covered by the recording package), and the label of the preset trigger condition corresponding to the target data stream (used to identify the specific condition that triggers the recording of the recording package).

[0092] The label of the preset trigger condition is an identifier used to identify and classify different trigger conditions.

[0093] Optionally, Figure 7 FIG. 1 is a schematic diagram of a process of performing disk placement by a disk placement module in an embodiment. Figure 7 As shown in the figure, the disk placement module starts after loading the configuration. The configuration information of the disk placement module mainly includes the disk storage directory. After the disk placement module is started, it will continue to monitor the data to be placed on the disk sent by the recording module; when the disk placement module receives the data to be placed on the disk from the recording module, it will create a database file, extract the metadata of the target recording package to which the data to be placed belongs, and add the metadata of the target recording package in the metadata table.

[0094] 2. Add the information of the data frame in the target recording package to the data frame table; the information of the data frame in the target recording package includes at least one of the following information of the data frame in the target recording package: data frame sequence number, subject sequence number, number of data bytes, binary data, and timestamp.

[0095] Alternatively, if Figure 7 As shown, when the disk placement module receives the data to be placed on disk from the recording module, the disk placement module will create a database file, extract the information of each data frame in the data to be placed on disk, and add the information of the data frame in the target recording package to the data frame table.

[0096] Through this embodiment, a universal lightweight database format is adopted. Compared with the current common data recording package format which is a binary file format, this embodiment does not require an additional development environment and supports cross-platform development, which is convenient for subsequent data secondary processing, effectively improves the efficiency of subsequent problem analysis and repair, and also provides strong support for subsequent data secondary processing.

[0097] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0098] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware server, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0099] According to another aspect of the embodiments of the present application, a data recording device is also provided, which is used to implement the data recording method provided in the above embodiments, and the description thereof has been made without further explanation. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and contemplated.

[0100] Figure 8 is a structural block diagram of an optional data recording device according to an embodiment of the present application, such as Figure 8 As shown, the device may include:

[0101] A recording module 801 is used to continuously record and cache data frames of a set of specified data streams; each specified data stream in the set of specified data streams corresponds to a subscribed topic;

[0102] The monitoring module 802 is used to continuously monitor each designated data stream to determine whether a preset trigger condition corresponding to each designated data stream is satisfied based on the data frame of each designated data stream; the preset trigger condition corresponding to each designated data stream means that each designated data stream contains an abnormal data event;

[0103] The recording module 801 is further configured to determine a target time interval for the target data stream to be written to the disk when it is determined based on the first data frame of the target data stream in the set of specified data streams that a preset trigger condition corresponding to the target data stream is satisfied; the target time interval includes the recording time of the first data frame; and extract the data frames of the target data stream within the target time interval from the cached data frames of the set of specified data streams to obtain the data to be written to the disk;

[0104] The disk storage module 803 is used to store the data to be stored in the database.

[0105] In an exemplary embodiment, the recording module 801 is also used to extract data frames of the target data stream within a target time interval from a cached set of data frames of a specified data stream after waiting to continue recording for a first preset time length, so as to obtain data to be stored on the disk; the target time interval includes a first time interval before the recording time of the first data frame and a second time interval after the recording time of the first data frame, and the duration of the second time interval is less than or equal to the first preset time length.

[0106] In an exemplary embodiment, the recording module 801 is also used to extract data frames of a target data stream from a cached set of data frames of a specified data stream; the extracted data frames of the target data stream are cached in a first cache queue; the first cache queue and the second cache queue that caches a set of data frames of a specified data stream are different cache queues; data frames within a target time interval are intercepted from the extracted data frames of the target data stream to obtain data to be stored on disk.

[0107] In an exemplary embodiment, the monitoring module 802 is also used to suspend monitoring of a group of specified data streams when it is determined based on the first data frame of the target data stream that a preset trigger condition corresponding to the target data stream is met, until the duration of the suspension of monitoring reaches a second preset duration, and then continue to monitor each specified data stream.

[0108] In an exemplary embodiment, a group of data frames of a specified data stream are cached in a pre-established second cache queue, and the second cache queue is configured with a maximum cache duration; the recording module 801 is also used to determine the interval duration between the timestamp of the first data frame in the second cache queue and the timestamp of the second data frame when the second data frame is to be written to the second cache queue; when the interval duration between the timestamp of the first data frame and the timestamp of the second data frame is greater than or equal to the maximum cache duration, remove the first data frame.

[0109] In an exemplary embodiment, a group of data frames of a specified data stream are cached in a second cache queue, and the second cache queue is configured with a maximum cache memory capacity; the recording module 801 is also used to remove the head data frame in the second cache queue and update the current cache memory capacity of the second cache queue when the current cache memory capacity of the second cache queue is greater than the maximum cache memory capacity.

[0110] In an exemplary embodiment, each time the data is dropped to the disk, it is stored in the database in the form of a recording package; a meta information table and a data frame table are set in the database; the meta information table is used to record the metadata of the recording package stored in the database; the data frame table is used to record the information of the data frames in the recording package stored in the database; the drop module 803 is also used to add the metadata of the target recording package in the meta information table; the target recording package is the recording package of the data to be dropped to the disk, and the metadata of the target recording package includes at least one of the following information: sequence number, recording package name, size of all data frames, start recording timestamp, end recording timestamp, and label of the preset trigger condition corresponding to the target data stream; the information of the data frame in the target recording package is added to the data frame table; the information of the data frame in the target recording package includes at least one of the following information: data frame sequence number, subject sequence number, number of data bytes, binary data, and timestamp.

[0111] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules as part of the device can be run in Figure 1 In the hardware environment shown, it can be implemented by software or by hardware, wherein the hardware environment includes a network environment.

[0112] According to another aspect of the embodiment of the present application, a storage medium is further provided. Optionally, in this embodiment, the storage medium can be used to execute the program code of any of the above data recording methods in the embodiment of the present application.

[0113] Optionally, in this embodiment, the storage medium may be located on at least one network device among a plurality of network devices in the network shown in the above embodiment.

[0114] Optionally, in this embodiment, the storage medium is configured to store a program code for executing any of the above data recording methods.

[0115] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, which will not be described in detail in this embodiment.

[0116] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disk.

[0117] According to another aspect of the embodiments of the present application, an electronic device for implementing the above data recording method is also provided. The electronic device may be a smart device, a server, a terminal, or a combination thereof.

[0118] Fig. 9is a structural block diagram of an optional electronic device according to an embodiment of the present application, such as Fig. 9 As shown, it includes a processor 902, a communication interface 904, a memory 906 and a communication bus 908, wherein the processor 902, the communication interface 904 and the memory 906 communicate with each other through the communication bus 908, wherein,

[0119] Memory 906, used to store computer programs;

[0120] The processor 902 is used to implement the program code of any of the above-mentioned data recording methods when executing the computer program stored in the memory 906.

[0121] Optionally, the communication bus may be a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 The communication interface is used for communication between the above electronic device and other devices.

[0122] The memory may include RAM, or may include non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0123] As an example, the memory 906 may include but is not limited to each module in the data recording device. In addition, it may also include but is not limited to other module units in the data recording device, which will not be described in detail in this example.

[0124] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0125] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0126] It can be understood by those skilled in the art that Fig. 9 The structure shown is for illustration only. The device for implementing the above data recording method may be a terminal device, which may be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Fig. 9 The structure of the electronic device is not limited. Fig. 9 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig. 9 Different configurations shown.

[0127] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0128] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0129] The embodiments of the present application also provide a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in any one of the above method embodiments.

[0130] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which can include: a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0131] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0132] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0133] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0134] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0135] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution provided in this embodiment.

[0136] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist physically separately, or at least two units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0137] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data recording method, characterized in that: include: Continuously record and cache data frames of a set of specified data streams; Each designated data stream in the set of designated data streams corresponds to a subscribed topic; Continuously monitoring each designated data stream to determine whether a preset trigger condition corresponding to each designated data stream is satisfied based on a data frame of each designated data stream; the preset trigger condition corresponding to each designated data stream refers to that each designated data stream contains an abnormal data event; When it is determined based on the first data frame of the target data stream in the set of specified data streams that a preset trigger condition corresponding to the target data stream is satisfied, determining a target time interval for the target data stream to be stored on a disk; The target time interval includes the recording time of the first data frame; The data frames of the target data stream within the target time interval are extracted from the cached data frames of the group of designated data streams to obtain the data to be written to the disk, and the data to be written to the disk is stored in the database.

2. The method according to claim 1, characterized in that The step of extracting data frames of the target data stream within the target time interval from the cached data frames of the group of designated data streams to obtain the data to be written to the disk includes: After waiting to continue recording for a first preset time length, extract the data frames of the target data stream within the target time interval from the cached data frames of the set of specified data streams to obtain the data to be written to the disk; the target time interval includes a first time interval before the recording time of the first data frame and a second time interval after the recording time of the first data frame, and the duration of the second time interval is less than or equal to the first preset time length.

3. The method according to claim 1, characterized in that The step of extracting data frames of the target data stream within the target time interval from the cached data frames of the group of designated data streams to obtain the data to be written to the disk includes: Extracting data frames of the target data stream from the cached data frames of the group of designated data streams; caching the extracted data frames of the target data stream into a first cache queue; the first cache queue and the second cache queue for caching the data frames of the group of designated data streams are different cache queues; Data frames within the target time interval are intercepted from the extracted data frames of the target data stream to obtain the data to be written to the disk.

4. The method according to claim 1, characterized in that The method further comprises: When it is determined based on the first data frame of the target data stream that a preset trigger condition corresponding to the target data stream is met, monitoring of the group of designated data streams is suspended until the duration of the suspension of monitoring reaches a second preset duration, and then each designated data stream is continuously monitored.

5. The method according to claim 1, characterized in that The data frames of the group of designated data flows are cached in a pre-established second cache queue, and the second cache queue is configured with a maximum cache duration; In the process of continuously recording and caching a set of data frames of a specified data stream, the method further includes: When a second data frame is to be written to the second cache queue, determining an interval between a timestamp of a head data frame in the second cache queue and a timestamp of the second data frame; When the interval between the timestamp of the first data frame and the timestamp of the second data frame is greater than or equal to the maximum cache duration, the first data frame is removed.

6. The method according to claim 1, characterized in that The data frames of the group of designated data flows are cached in a second cache queue, and the second cache queue is configured with a maximum cache memory capacity; In the process of continuously recording and caching a set of data frames of a specified data stream, the method further includes: When the current cache memory capacity of the second cache queue is greater than the maximum cache memory capacity, the head data frame in the second cache queue is removed, and the current cache memory capacity of the second cache queue is updated.

7. The method according to any one of claims 1 to 6, characterized in that Each time the data is dropped onto the disk, it is stored in the database in the form of a recording package; a meta information table and a data frame table are set in the database; the meta information table is used to record the metadata of the recording package stored in the database; the data frame table is used to record the information of the data frame in the recording package stored in the database; The storing the data to be stored in the database includes: Adding metadata of a target recording package in the meta information table; the target recording package is a recording package of the data to be written to the disk, and the metadata of the target recording package includes at least one of the following information: a sequence number, a recording package name, a size of all data frames, a start recording timestamp, an end recording timestamp, and a label of a preset trigger condition corresponding to the target data stream; The information of the data frame in the target recording package is added to the data frame table; the information of the data frame in the target recording package includes at least one of the following information: data frame sequence number, subject sequence number, number of data bytes, binary data, and timestamp.

8. A data recording device, characterized in that: include: A recording module, used to continuously record and cache data frames of a set of specified data streams; Each designated data stream in the set of designated data streams corresponds to a subscribed topic; A monitoring module, used for continuously monitoring each designated data stream, to determine whether a preset trigger condition corresponding to each designated data stream is satisfied based on a data frame of each designated data stream; the preset trigger condition corresponding to each designated data stream means that each designated data stream contains an abnormal data event; The recording module is further configured to determine a target time interval for the target data stream to be recorded on a disk when it is determined based on the first data frame of the target data stream in the group of specified data streams that a preset trigger condition corresponding to the target data stream is satisfied; The target time interval includes the recording time of the first data frame; Extracting data frames of the target data stream within the target time interval from the cached data frames of the group of designated data streams to obtain data to be written to the disk; The disk storage module is used to store the data to be stored in the database.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of any one of the methods of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images