File writing method and device, file parsing method and device, equipment and medium

Abstract

The invention discloses a file writing method, an analysis method, a device, equipment and a medium, and is applied to an automatic driving domain controller of a vehicle, the controller is configured to perform file writing by adopting a preset file format, the preset file format comprises a file header, a special frame area and a plurality of data frame areas, and the method comprises the following steps: when the writing of a new file is started, writing the new file into the special frame area; writing file format version information into a file header of the new file, and filling inherent information of the vehicle into a data structure defined by a special frame area of the new file to form a special frame; collecting automatic driving data of the vehicle according to a preset storage period, and sequentially writing the automatic driving data obtained in each storage period into a data frame area of the new file as a data frame; and when the storage space of the new file is full or the writing duration of the new file is reached, ending the writing of the new file. According to the writing method, efficient storage and flexible expansion of the automatic driving data can be realized.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a file writing method, parsing method, apparatus, device, and medium. Background Technology

[0002] In the rapid development of autonomous driving technology, various sensors on vehicles, such as cameras, radar, and lidar, continuously generate massive amounts of data. Simultaneously, the algorithms also produce a large amount of intermediate result data during operation. The persistent storage and efficient reuse of this data are crucial for optimizing autonomous driving algorithms, troubleshooting, and iterative system upgrades.

[0003] Currently, the main open-source solutions for persistent autonomous driving data include ROS bag and Apollo CyberRecorder. Related patent technologies, such as "A method for storing closed-loop files of autonomous driving data" with publication number CN118132519A, use binary encoding to organize data, including variable information data and location information data, sorted by timestamp and saved as a single file, thereby achieving coupling between variable data and location information.

[0004] However, existing technologies have many shortcomings: ROS bag has moderate serialization efficiency, poor random access performance, and insufficient scalability; Apollo Cyber ​​Recorder has high serialization efficiency and storage density, but its random access performance and scalability are only at a good level. Similar to ROS bag, it stores data separately according to specific dimensions (topic or channel), which makes management and maintenance inconvenient; some solutions have the problem of separate file storage, which is not convenient for unified management and maintenance; at the same time, existing data persistence technologies generally face problems such as low storage efficiency, poor flexibility, difficulty in adapting to changes in data structure, and insufficient version compatibility, which cannot fully meet the requirements of autonomous driving systems for efficient, scalable, consistent, and convenient data storage.

[0005] Therefore, there is an urgent need for a persistent file processing solution for autonomous driving data that can overcome the above-mentioned technical deficiencies and achieve efficient storage, flexible expansion, rapid positioning, strong consistency, and good version compatibility. Summary of the Invention

[0006] This application provides a file writing method, parsing method, apparatus, device, and medium that enables efficient storage and flexible expansion of autonomous driving data, effectively solving problems such as low storage efficiency, poor flexibility, difficulty in adapting to changes in data structure, and insufficient version compatibility in existing autonomous driving data persistence technologies.

[0007] In a first aspect, the present invention provides the following technical solution through an embodiment of the present invention: A file writing method is applied to an autonomous driving domain controller of a vehicle. The controller is configured to write files using a preset file format, which includes a file header, a special frame area, and multiple data frame areas. The method includes: when writing a new file begins, writing file format version information to the file header of the new file, and filling the data structure defined by the special frame area of ​​the new file with the inherent information of the vehicle to form a special frame; collecting autonomous driving data of the vehicle according to a preset storage cycle, and sequentially writing the autonomous driving data acquired in each storage cycle into a data frame area of ​​the new file as a data frame; and ending the writing of the new file when the storage space of the new file is full or the writing time of the new file is reached.

[0008] Preferably, the data frame area includes a metadata storage area and a business data storage area. The step of sequentially writing the autonomous driving data acquired in each storage cycle into a data frame area of ​​the new file includes: for each storage cycle, traversing the autonomous driving data acquired in that storage cycle to extract key information corresponding to different data sources from the autonomous driving data, and filling the key information into the data structure defined in the metadata storage area to obtain the metadata of a single data frame. The metadata contains a data description array composed of multiple data descriptions; serializing the autonomous driving data acquired in the storage cycle to obtain... The serialized byte streams of autonomous driving data from different data sources are stored in a byte stream array; the total length of the serialized byte streams of autonomous driving data from different data sources within the storage period is calculated; the metadata of the single data frame area is serialized, and the length of the serialized metadata and the byte stream are written into the metadata field of the metadata storage area, and the total length of the byte stream is written into the business data field of the business data storage area; and the lengths and byte streams of the serialized autonomous driving data from different data sources are written sequentially according to the byte stream array, wherein the autonomous driving data from different data sources are matched one-to-one with the data descriptions in the data description array.

[0009] Preferably, calculating the total length of the serialized byte stream of autonomous driving data from different data sources within the storage period includes: multiplying the sum of the number of bytes occupied by the business data field and the length of the serialized autonomous driving data from different data sources by the number of data sources of the autonomous driving data acquired within the storage period to obtain the total length of the serialized byte stream within the storage period.

[0010] Preferably, the step of filling the inherent information of the vehicle into the data structure defined by the special frame area of ​​the new file to form a special frame includes: filling the inherent information of the vehicle into the data structure defined by the special frame area of ​​the new file; extracting key information from the inherent information and filling the key information into the data structure corresponding to the metadata of the special frame area to construct the metadata of the special frame; serializing the metadata of the special frame area and writing the serialized length and byte stream of the metadata into the metadata field of the special frame area; serializing the inherent information and calculating the total length of the serialized byte stream of the inherent information, writing the total length of the byte stream, the serialized length of the inherent information, and the byte stream into the business data field of the special frame area.

[0011] Preferably, a single data frame area includes a metadata storage area and a business data storage area; the metadata storage area includes a 4-byte metadata length field and a byte stream of metadata serialization, and the business data storage area includes a 4-byte metadata total length field and at least one set of business data segments, the business data segments including a 4-byte business data length field and a byte stream of business data serialization.

[0012] Secondly, through an embodiment of the present invention, the present invention provides the following technical solution: A file parsing method is applied to an autonomous driving domain controller of a vehicle. The controller is configured to parse files stored in a preset file format. The preset file format includes a file header, a special frame area, and multiple data frame areas. The file header records file format version information, the special frame area is filled with vehicle-specific information, and the multiple data frame areas include multiple frames of autonomous driving data, with each data frame area storing one frame of autonomous driving data. The parsing method includes: initializing a pre-established key-value pair data structure for storing the parsed data; reading the file header and the inherent information in the special frame area; and parsing each data frame area sequentially according to its storage time order. The parsing steps include: deserializing the autonomous driving data stored in the data frame area, and storing the parsed data in the key-value pair data structure after parsing one piece of data in each data frame area; completing the parsing of all data frames and outputting the file.

[0013] Preferably, the data frame area includes a metadata storage area and a business data storage area. The metadata storage area stores the length and byte stream of the serialized metadata, and the business data storage area stores the total length of the byte stream of business data, the length and byte stream of the serialized autonomous driving data from different data sources. The deserialization of the autonomous driving data stored in the data frame area includes: reading the length of the byte stream after the metadata serialization, and reading the byte stream data corresponding to the length of the byte stream; deserializing the byte stream data according to the metadata definition; parsing out the data description; traversing the data description; storing data with a true flag in the data description into the current frame data description array; the data descriptions in the data description array are matched one-to-one with the autonomous driving data from different data sources; reading the total length of the business data in the current data frame area, and performing the following matching and parsing steps sequentially according to the storage order of a single data source: obtaining the length of the autonomous driving data from a single data source and the byte stream data under that length; obtaining the business data definition according to the data description matching the data source; deserializing the byte stream data according to the business data definition, until the matching and parsing of all data is completed.

[0014] Thirdly, through one embodiment of the present invention, the following technical solution is provided: A computing device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in any of the first aspects above.

[0015] Fourthly, through one embodiment of the present invention, the following technical solution is provided: A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the first aspects above.

[0016] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: The file writing method provided in this invention, when starting to write a new file, writes file format version information to the header of the new file, and fills the data structure defined by the special frame area of ​​the new file with the inherent information of the vehicle to form a special frame; collects autonomous driving data of the vehicle according to a preset storage cycle, and writes the autonomous driving data acquired in each storage cycle into a data frame area of ​​the new file as a data frame; when the storage space of the new file is full or the writing time of the new file is reached, the writing of the new file ends. It adopts a version area design of "fixed header + extended description". The fixed header stores core version information, which can be recognized by parsers of all versions. The extended description stores version-related extended information. Older version parsers can ignore the extended description and only parse the core content, while newer version parsers can parse the entire content, achieving forward and backward compatibility and solving the compatibility problem between different version data and parsing tools. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the file writing method in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a preset file format in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a special frame in an embodiment of the present invention; Figure 4 This is a schematic diagram of the data frame structure in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of the mete part in an embodiment of the present invention; Figure 6 This is a schematic diagram of the data section in an embodiment of the present invention; Figure 7 This is a schematic diagram of the file parsing method in an embodiment of the present invention; Figure 8 This is a schematic diagram of the file writing device in an embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of the computing device in an embodiment of the present invention. Detailed Implementation

[0019] This application provides a file writing method, parsing method, apparatus, device, and medium that enable efficient storage and flexible expansion of autonomous driving data.

[0020] The technical solution of this application embodiment is to solve the above-mentioned technical problems, and the general idea is as follows: A file writing method is applied to an autonomous driving domain controller of a vehicle. The controller is configured to write files using a preset file format, which includes a file header, a special frame area, and multiple data frame areas. The method includes: when writing a new file begins, writing file format version information to the file header of the new file, and filling the data structure defined by the special frame area of ​​the new file with the vehicle's inherent information to form a special frame; collecting autonomous driving data of the vehicle according to a preset storage cycle, and sequentially writing the autonomous driving data acquired in each storage cycle into a data frame area of ​​the new file as a data frame; and ending the writing of the new file when the storage space of the new file is full or the writing time of the new file is reached.

[0021] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0022] Firstly, the present invention provides a file writing method applied to a vehicle autonomous driving domain controller. The controller is configured to write files using a preset file format. The preset file format includes: a file header, a special frame area, and multiple data frame areas. Specifically, as follows: Figure 1 As shown, the method includes the following steps S101 to S103: Step S101: When writing a new file begins, file format version information is written to the header of the new file, and the inherent information of the vehicle is filled into the data structure defined by the special frame area of ​​the new file to form a special frame. Step S102: Collect the autonomous driving data of the vehicle according to the preset storage period, and write the autonomous driving data acquired in each storage period into a data frame area of ​​the new file as a data frame. Step S103: When the storage space of the new file is full or the writing time of the new file is reached, the writing of the new file ends.

[0023] This file writing method is suitable for real-time recording, persistent storage, and efficient parsing and playback of various types of autonomous driving data, such as multi-source heterogeneous sensor data and algorithm intermediate result data.

[0024] As an example, a single data frame area includes a metadata storage area and a business data storage area; such as Figure 2As shown, the metadata storage area may include a 4-byte metadata length field and a byte stream of serialized metadata. The business data storage area includes a 4-byte metadata total length field and at least one set of business data segments. The business data segment includes a 4-byte business data length field and a byte stream of serialized business data.

[0025] It should be noted that the autonomous driving data from different sources here can refer to data generated by different components, such as sensor A collecting the first data, sensor B collecting the second data, and so on.

[0026] For example, during the data acquisition phase, autonomous vehicles are equipped with various sensors, such as cameras, radar, and lidar. These sensors continuously generate a large amount of raw data. Once the data is generated, the system generates corresponding structured metadata (the serialized part of the meta-database) based on information such as the data type (images, point clouds, etc.) and the acquisition time. For instance, the metadata for image data acquired by a camera would include information such as the acquisition timestamp, image resolution, and camera ID.

[0027] As an example, to distinguish the file format for disk persistence, four bytes can be reserved in the file header to differentiate the data persistence file version. The first four bytes contain version information of the data persistence file format, such as... Figure 2 As shown, Figure 3 This is an illustration of a special frame. Figure 4 This is a diagram of a data frame.

[0028] The file consists of two sections: a special frame section (located at the beginning) and a data frame section. The special frame section stores data that remains constant over a short period, such as software version and calibration parameters. The data frame section stores real-time changing data, such as sensor outputs and algorithm module outputs. Multiple data frames follow the special frame, and both the special and data frames use the same frame format.

[0029] It should be noted that the data of special frames is stored in the current file only once and remains unchanged.

[0030] The following uses a data frame as an example to explain the preset file format: like Figure 4 As shown, each data frame written to disk consists of structured meta (metadata) and unstructured data (business data). The structured meta part describes key information about the data, such as timestamps, frame IDs, and data types. The structured meta is defined as follows: 1. 4 bytes meta_len, defines the length of the meta after serialization; 2. Variable-length byte stream (meta) is a byte stream after meta serialization; 3. Meta data descriptions are defined using protobuferated, which is analogous to an array; a single meta tag contains multiple data items. For example... Figure 5 The image shows the metadata portion of a data frame.

[0031] For example, the protobuf definition corresponding to metadata (meta) can be: syntax="proto3"; package Meta; option go_package = "proto / auto_msg / proto_interface / meta"; / / Data descriptor, describing the characteristics of a single data block message DataDescriptor { int64 ts = 1; / / Data generation timestamp int32 frame_id = 2; / / Data frame ID string type = 3; / / Data type bool with_data_field = 4; / / Check if there is corresponding business data. int64 sync_ts = 5; / / Synchronization timestamp int64 log_ts = 6; / / Log timestamp } / / Metadata message structure, containing global metadata information for the current frame and multiple data descriptors. message Meta { int64 ts = 1; / / Metadata generation timestamp repeated DataDescriptor data_descriptor = 2; / / List of data descriptors, supporting multiple data block descriptions. }

[0032] The Meta message structure, defined using protobuf, contains timestamps and a list of data descriptors. Each DataDescriptor describes the characteristics of a data block (timestamp, frame ID, data type, etc.).

[0033] Meta is defined using protobuf IDL, and the Meta message structure includes a timestamp (ts) and a list of data descriptors (data_descriptor).

[0034] The unstructured date portion stores serialized byte data, defined as follows: 1. data_len occupies 4 bytes and stores the serialized length of data_pb; 2. Data_pb serialization stores the serialized bytes of data; 3. `total_len` occupies 4 bytes and represents the total length of unstructured data in a single frame, with a size of n × (4 + data_len1 + data_len2 + data_len3 ... data_lenN), where n is the number of different data sources. For example... Figure 6 As shown, this is the data portion of the business data in the data frame.

[0035] In autonomous driving, the definition of data depends on the business, and Google Protobuf IDL is generally used. Regardless of the form, serialization and deserialization capabilities are required.

[0036] This approach combines variable-length byte storage technology with Protocol Buffers serialization. A "fixed-length + variable-length data" structure dynamically adapts the frame length. The fixed header stores core version information in a fixed format (e.g., version=1, occupying 1-2 bytes), ensuring basic compatibility for all parsers. The extended description stores version-related extended information (such as descriptions of newly added fields, format change records, and version number) in a variable-length field, with its length marked by `version_len`. Older version parsers can ignore the extended description (only parsing the fixed header), while newer version parsers can fully parse the extended information. This significantly improves storage density and I / O efficiency.

[0037] At the same time, integrating special frames and multiple data frames into a single file avoids the management inconvenience caused by separate file storage and further improves the utilization rate of storage resources.

[0038] The following describes the process of writing a new file: Specifically, the step of filling the inherent information of the vehicle into the data structure defined by the special frame area of ​​the new file to form a special frame may include: The inherent information of the vehicle is filled into the data structure defined in the special frame area of ​​the new file; key information is extracted from the inherent information and filled into the data structure corresponding to the metadata of the special frame area to construct the metadata of the special frame; the metadata of the special frame area is serialized, and the length and byte stream of the serialized metadata are written into the metadata field of the special frame area; the inherent information is serialized, and the total length of the byte stream after serialization of the inherent information is calculated. The total length of the byte stream, as well as the length and byte stream after serialization of the inherent information, are written into the business data field of the special frame area.

[0039] Example, Step 1: New file initialization and special frame writing When writing a new file begins, 4 bytes of file format version information are written to the file header to identify the format specification version followed by the data storage, providing a version basis for subsequent data parsing. Simultaneously, the vehicle's inherent information is filled into the data structure defined in the special frame area of ​​the new file, forming a special frame. The specific process is as follows: Collect inherent information about the vehicle, which may include data that does not change in a short period of time, such as the version of the autonomous driving software and the vehicle calibration parameters. Extract key information from the inherent information. Key information may include timestamps, data types, frame IDs, etc. Fill the key information into the data structure corresponding to the metadata of the special frame area to construct the metadata of the special frame. Then, perform serialization processing on the metadata of the special frame. First, write 4 bytes of metadata serialization length (meta_len) into the metadata field of the special frame area, and then write the byte stream after metadata serialization (meta pb serialization). The intrinsic information is serialized, and the total length of the serialized byte stream (total_len) is calculated. The total length is n×(4+data_len), where n is the number of intrinsic information data and data_len is the length of a single intrinsic information after serialization. First, write the total length of 4 bytes (total_len) into the business data field of the special frame area, and then write the 4-byte data length (data_len) corresponding to each inherent information and the byte stream after data serialization (data pb serialization) in sequence.

[0040] The metadata area centrally stores key index information such as timestamps and data types, enabling time range queries with O(1) time complexity. Combined with the length prefix design, it can quickly locate data positions, greatly improve random access performance and data retrieval efficiency, and solve the problem of poor random access performance in existing solutions.

[0041] In a specific embodiment, the data frame area includes a metadata storage area and a business data storage area. The step of sequentially writing the autonomous driving data acquired in each storage cycle into a data frame area of ​​a new file may include: For each storage cycle, the autonomous driving data acquired within the storage cycle is traversed to extract key information corresponding to different data sources. This key information is then filled into the data structure defined in the metadata storage area to obtain the metadata of a single data frame. The metadata contains a data description array composed of multiple data descriptions. The autonomous driving data acquired within the storage cycle is serialized to obtain the serialized byte streams of autonomous driving data from different data sources, which are stored in a byte stream array. The total length of the serialized byte streams of autonomous driving data from different data sources within the storage cycle is calculated. The metadata of a single data frame area is serialized, and the length and byte stream of the serialized metadata are written to the metadata field in the metadata storage area. The total length of the byte stream is also written to the business data field in the business data storage area. Furthermore, the length and byte stream of the serialized autonomous driving data from different data sources are written sequentially according to the byte stream array. The autonomous driving data from different data sources are matched one-to-one with the data descriptions in the data description array.

[0042] Among them, different business data and metadata are stored separately. The key information in the metadata is a general description of autonomous driving data from different data sources in the business data.

[0043] Specifically, the throttling array is calculated to obtain the total length of the serialized byte stream of autonomous driving data from different data sources within the storage period. This can include: multiplying the sum of the number of bytes occupied by the business data field and the length of the serialized autonomous driving data from different data sources by the number of data sources of the autonomous driving data obtained within the storage period to obtain the total length of the serialized byte stream within the storage period.

[0044] For example, the calculation formula 1 is n×(4+data_len1+data_len2+data_len3……data_lenN).

[0045] As an example, autonomous driving data of the vehicle is collected according to a preset storage cycle. The autonomous driving data includes multi-source heterogeneous data such as sensor output data, algorithm module output data, and vehicle body data. The autonomous driving data acquired in each storage cycle is sequentially written into a data frame area of ​​a new file as a data frame. The specific process is as follows: Iterate through the batch of autonomous driving data acquired within the current storage cycle, extract key information corresponding to different data sources from the autonomous driving data, including timestamp (ts), frame ID (frame_id), data type (type), whether it carries actual business data (with_data_field), synchronization timestamp (sync_ts), log timestamp (log_ts), etc., and fill the key information into the data structure defined in the metadata storage area of ​​the data frame area to obtain the metadata of a single data frame. The metadata contains an array of data descriptions composed of multiple data descriptors. The autonomous driving data acquired during the storage period is serialized using Google ProtobufIDL to obtain the serialized byte streams of autonomous driving data from different data sources, and these byte streams are stored in a byte stream array. Calculate the total length (total_len) of the serialized byte stream of autonomous driving data from different data sources within the storage period. The total length is n×(4+data_len1+data_len2+data_len3……data_lenN), where n is the number of data sources for autonomous driving data within the current storage period, data_len is the length of the serialized autonomous driving data corresponding to a single data source, data_len1 is the length of the serialized data in the first data block, data_len2 is the length of the serialized data in the second data block, and data_lenN is the length of the serialized data in the Nth data block. The metadata of a single data frame is serialized. First, 4 bytes of metadata serialization length (meta_len) are written into the metadata storage area of ​​the data frame area, and then the byte stream after metadata serialization is written (metapb serialization). First, write a total length of 4 bytes (total_len) into the business data storage area of ​​the data frame area. Then, traverse the byte stream array and write the 4-byte data length (data_len) and the serialized byte stream (datapb serialization) corresponding to each data source in turn. The data descriptors in the structured data correspond one-to-one with the data segments in the unstructured data.

[0046] It adopts a storage format defined in a single file using storage frames. The outer Meta layer serves as the overall metadata container, while the inner DataDescriptor acts as the description unit for individual data blocks. The `repeated` keyword (in Protobuf, representing a repeatable field, similar to a list) allows a Meta layer to contain multiple DataDescriptors, forming a relationship of "one global metadata layer corresponding to N data block descriptions," adapting to scenarios with multiple data blocks. Through hierarchical metadata (global + individual), it comprehensively records the temporal relationships, identification information, and structural characteristics of the data block set, facilitating both rapid understanding of the overall data outline by the receiver and precise location and parsing of individual data blocks.

[0047] Based on storage size and storage period, the storage is divided into segments. When the storage space for a new file is full or the writing time for a new file is reached, the writing of the new file ends, and the data persistence of the current file is completed.

[0048] The data frame area includes a metadata storage area and a business data storage area. The metadata storage area includes a 4-byte metadata length field and a byte stream of serialized metadata. The business data storage area includes a 4-byte total length field and at least one set of business data segments. The business data segments include a 4-byte business data length field and a byte stream of serialized business data.

[0049] Data serialization is implemented using Google Protobuf IDL. Data changes comply with the protobuf incremental principle, allowing new data definitions to be added, but existing definitions cannot be modified or deleted, ensuring compatibility during data structure evolution.

[0050] In simple terms, the process of writing a new file includes: 1. Whenever a new file is written (the first file or when a slice occurs), a 4-byte file format version is written first; II. Next, a special frame is constructed: 1. Collect information such as the self-driving software version and vehicle calibration parameters, and populate the data in the special frame definition; 2. Construct a special frame meta tag, extract key information from the special frame such as time, ID, type, etc., and populate the meta tag; 3. Meta serialization; 4. The length of the serialized meta tag is written, occupying 4 bytes; 5. Write the meta-serialized byte stream; 6. Special frame serialization; 7. Calculate the total length, write the total length, which occupies 4 bytes. The total length is 4 + the length of the special frame serialization byte stream. 8. Write the length of the special frame after serialization, which occupies 4 bytes; 9. Write the byte stream after special frame serialization.

[0051] Third, the landing application continuously collects data generated by intelligent driving: It should be noted that intelligent driving data generally includes data from sensors, the vehicle body, and intermediate algorithm results. The disk writing process uses batch writing, writing multiple different types and sources of data at once. Each batch write is a disk writing data frame. The process of each batch write is described as follows: Step 1: Traverse the batch of data; Step 2: Construct meta tags, extract key information for each data item, such as time, ID, type, etc., and populate the meta tags; Step 3: Data serialization, storing the byte stream after each data sequence into an array; Step 4: Calculate the data length, increasing it by 4 times the length of the byte stream after each data sequence; Step 5: After the batch data traversal is complete, serialize the meta data; Step 6: Write the length of the serialized meta tag, which occupies 4 bytes; Step 7: Write the meta-serialized byte stream; Step 8: Write the total length, which occupies 4 bytes. The total length is the length calculated in step 4. Step 9: Traverse the byte stream array following the data sequence in Step 3; Step 10: Write the serialized length, which occupies 4 bytes, sequentially into the serialized byte stream.

[0052] Compared with traditional technologies, the present invention has at least the following significant advantages: 1. High storage efficiency By combining variable-length byte storage technology with Protocol Buffers serialization, a "fixed-length + variable-length data" structure is used to dynamically adapt the frame length, significantly improving storage density and I / O efficiency. Simultaneously, special frames and multiple data frames are integrated into a single file, avoiding the management inconveniences of separate file storage and further enhancing storage resource utilization.

[0053] 2. Highly flexible and scalable By decoupling metadata from business data, and utilizing DataDescriptor to dynamically describe business data types, it supports the expansion of new data types and fields without modifying the storage format, perfectly adapting to the evolving data structure needs of autonomous driving systems. The layered metadata structure (global metadata + individual data block descriptions) facilitates quick understanding of the overall data outline while accurately locating individual data blocks, adapting to the storage requirements of multi-source heterogeneous data.

[0054] In summary, the file parsing method provided by this invention solves the problems of low storage efficiency, poor flexibility, difficulty in adapting to changes in data structure, and insufficient version compatibility in existing autonomous driving data persistence technologies, thereby achieving efficient storage and flexible expansion of autonomous driving data.

[0055] Secondly, based on the same inventive concept, this embodiment provides a file parsing method applied to an autonomous driving domain controller of a vehicle. The controller is configured to parse files stored in a preset file format. The preset file format includes a file header, a special frame area, and multiple data frame areas. The file header records file format version information. The special frame area is filled with inherent vehicle information. The multiple data frame areas include multiple frames of autonomous driving data, with each data frame area storing one frame of autonomous driving data. Figure 7 As shown, the parsing method is as follows: Step S201: Initialize the pre-established key-value pair data structure, which is used to store the parsed data; Step S202: Read the file header and the inherent information in the special frame area; Step S203: Parse each data frame area in sequence according to the storage time order. The parsing steps include: deserializing the autonomous driving data stored in the data frame area, and storing the parsed data into the key-value pair data structure after parsing one piece of data in each data frame area. Step S204: Complete the parsing of all data frames and output the file.

[0056] Specifically, the data frame area includes a metadata storage area and a business data storage area. The metadata storage area stores the length and byte stream of the serialized metadata, while the business data storage area stores the total length of the business data byte stream, the length and byte stream of the serialized autonomous driving data from different data sources. Deserialization of the autonomous driving data stored in the data frame area can include: The process reads the length of the serialized byte stream from the metadata and the corresponding byte stream data. It then deserializes the byte stream data according to the metadata definition, parses out the data description, iterates through the data descriptions, and stores data with true flags in the data descriptions into the current frame's data description array. The data descriptions in the array are matched one-to-one with autonomous driving data from different data sources. Next, the process reads the total length of the business data in the current data frame area and, according to the storage order of individual data sources, sequentially performs the following matching and parsing steps for different data: obtains the length of the autonomous driving data from a single data source and the byte stream data at that length; obtains the business data definition based on the data description matching the data source; and deserializes the byte stream data according to the business data definition until the matching and parsing of all data is completed.

[0057] First, a pre-established key-value pair data structure is initialized. This structure uses time as the key to store the parsed data, facilitating subsequent retrieval and use of the data by time. Next, the 4-byte file format version information in the file header is read to determine the current file's data storage format version. Then, the inherent information in the special frame area is read. By parsing the metadata and business data of the special frames, key inherent information such as the autonomous driving software version and vehicle calibration parameters is obtained, providing basic background information for subsequent data parsing and applications.

[0058] Next, each data frame area is parsed sequentially according to its storage time. The specific steps are as follows: Read the 4-byte metadata length (meta_len) of the current data frame area metadata storage area, and read the corresponding metadata byte stream (meta pb serialization) based on this length. The byte stream data is deserialized according to the protobuf definition of the metadata, and the data description array (data_descriptor) is parsed out. The data description array is traversed, and the data in the data description with the flag bit (with_data_field) being true is stored in the current frame data description array. The data description in the data description array is matched one by one with the autonomous driving data from different data sources. Read the 4-byte total length (total_len) of the current data frame's service data storage area to determine the total number of bytes of the current data frame's service data. According to the storage order of a single data source, the following matching and parsing steps are executed sequentially: read the 4-byte data length (data_len), read the corresponding business data byte stream (datapb serialization) according to the length, determine the business data definition according to the data type (type) in the data description that matches the data source, and deserialize the byte stream data according to the business data definition to obtain the parsed business data; After parsing one piece of data within each data frame area, the parsed data is stored in the key-value pair data structure.

[0059] After parsing all data frames, the system outputs a key-value pair data structure that stores the parsed data, which can be used for scenarios such as algorithm optimization, data playback, and fault analysis in autonomous driving systems.

[0060] Specifically, first, a data container is created to store the parsing results. This container uses a key-value pair (map) structure, where the key is a timestamp (such as the specific time the data was generated), and the "value" stores all the parsed data corresponding to that time point. The file header contains four consecutive bytes as a file version identifier. The data in the file is stored in data blocks, and each frame contains metadata and business data (actual sensor, algorithm data, etc.).

[0061] First, read the first 4 bytes of the current frame to obtain the metadata length. Simultaneously, create a data description array (similar to a table) for the current frame to temporarily store the valid metadata information that will be filtered out later, ensuring that each piece of business data can find its corresponding description.

[0062] Based on the aforementioned metadata length (e.g., 48 bytes), a byte stream of the corresponding length is read from the file. Then, using a preset metadata structure definition (e.g., protobuf format), the byte stream is deserialized into understandable structured data. After deserialization, global information of the current frame (e.g., the generation time of the entire frame data) and basic descriptions of each business data (e.g., data A is a camera image) can be obtained.

[0063] The metadata contains multiple data descriptions, each corresponding to a piece of business data. Each description has a flag "with_data_field" (a boolean value): if "true", it indicates that the business data corresponding to that description has actual content (requires parsing); if "false", it indicates that the description is merely a placeholder (no actual data). All data descriptions are iterated through, and only those with a boolean value of "true" are stored in a pre-created data description array. For example, if the metadata has 5 descriptions, and 3 of them are marked "true", then the data description array will ultimately contain 3 items, corresponding to 3 pieces of business data that need to be parsed.

[0064] In the business data storage area, 4 bytes are used to record the total length of all business data in the current frame, for example, 1800 bytes. This total length is the verification standard for subsequent parsing; the sum of the lengths of all business data in this data frame must equal this value, otherwise it indicates that the data may be corrupted or parsing has failed. Parsing begins with the length of the business data and the byte stream. Using the length of a single data block (e.g., 1024 bytes), a binary byte stream of the corresponding length is read from the file. Since the business data and the data description array are sequentially corresponding (e.g., the first item in the array corresponds to the first piece of business data, and the second item corresponds to the second piece of business data), the specific format definition of the data (e.g., a camera image includes fields such as resolution) can be determined through the "type" field of the corresponding item in the array. Finally, the byte stream is deserialized into understandable specific data (e.g., the pixel matrix of an image) according to the format definition.

[0065] The parsed data is then stored in a previously created map according to its corresponding timestamp (which can be obtained from metadata or data description items). For example, if the timestamp of the data is "1625097610000", the key is found in the map, and the data is added to the corresponding value. If other data (such as radar data at the same moment) already exists at the same timestamp, they are stored together to form a complete record of multiple types of data corresponding to a single time point.

[0066] Repeat the steps of reading the length of a single data item, parsing the data, and storing it in a map, parsing one piece of business data at a time. Simultaneously, calculate the total length of the parsed data. When the accumulated length equals the total length of the acquired business data (e.g., 1800 bytes), it indicates that all business data in the current frame has been parsed; if not, data loss or parsing errors may have occurred, requiring an error report.

[0067] After one data frame is parsed, the next data frame is parsed: read the metadata length of the next data frame, and repeat the subsequent operations until all data frames in the file have been parsed.

[0068] Ultimately, the map will store all the data in the file completely according to the timestamp, which can be directly used for subsequent operations such as autonomous driving scenario playback, algorithm verification, and fault analysis.

[0069] In short, since special frames and data frames use the same frame format, they can be parsed in the same way. The parsing process for a new file is described below (taking data frames as an example): 1. Initialize the data parsing and storage map, with time as the key; 2. Read 4 bytes to parse the persistent file version; 3. Read 4 bytes to obtain the length of the structured meta, and construct a data description array for each data frame written to disk; 4. Read byte stream data of length meta, use the meta data to define deserialization, and decrypt the meta; 5. Iterate through the meta data descriptions. When a data description with_data_field is true, add the description to the current frame's data description array. 6. Read 4 bytes to obtain the total length of one frame of unstructured data written to disk; 7. Read 4 bytes to get the length of a data item; 8. Read a byte stream of data of a certain length. Since the data and the current frame data description array are matched in order, obtain the data definition based on the type in the data description, and deserialize the data. 9. Fill the parsed data into the data parsing map from step 1; 10. Repeat steps 7 and 9 until n×(4+data_len1+data_len2+data_len3……data_lenN) is equal to the total length of the unstructured data in a frame written to disk in step 6, where n is the number of times steps 7 and 9 are repeated, i.e. the number of data items; 11. Repeat steps 3 and 10 until the file is finished reading; 12. Data parsing map, which is the final parsed data.

[0070] During the process of parsing persistent files, data is sorted by time based on the description information in the meta tags, and data is parsed according to the type. Tools for data storage, viewing, playback, splitting, and monitoring are designed and developed.

[0071] This application adopts variable-length byte storage technology, which achieves dynamic adaptation of frame length through fixed-length + variable-length data, enabling flexible storage. Furthermore, the combination of variable-length bytes and protocol buffering significantly improves storage density and IO (read and write data) efficiency. It employs a decoupling approach between metadata and business data: DataDescriptor dynamically describes business data types, supporting the expansion of new data types without modifying the storage format. Separation of metadata and business data supports diverse business scenarios. The metadata area centrally stores key index information such as timestamps, enabling O(1) time range queries and achieving efficient indexing. Each new file uses a version area design with a "fixed header + extended description," supporting forward and backward compatibility. The layered design supports data structure evolution without affecting existing data reading, allowing for flexible expansion. During data parsing, rapid data location is achieved through length prefixes, improving random read performance. The frame-to-disk mechanism ensures data persistence and integrity.

[0072] Compared with traditional technologies, the present invention has at least the following significant advantages: 1. Excellent data access performance The metadata area centrally stores key index information such as timestamps and data types, enabling time range queries with O(1) time complexity. Combined with the length prefix design, it can quickly locate data positions, greatly improving random access performance and data retrieval efficiency, and solving the problem of poor random access performance in existing solutions.

[0073] 2. Strong data consistency By adopting a frame-to-disk mechanism, each data frame fully contains metadata and corresponding business data, ensuring data integrity during storage, avoiding data loss or corruption, guaranteeing strong consistency of data persistence, and providing a reliable data foundation for subsequent data playback, algorithm verification and other scenarios.

[0074] In summary, the file parsing method provided by this invention solves the problem of poor random access performance in existing autonomous driving data persistence technologies, and enables rapid location and reliable parsing of autonomous driving data.

[0075] Thirdly, based on the same inventive concept, this embodiment provides a file writing device, such as... Figure 8 As shown, it includes: The first writing module 401 is used to write file format version information to the header of the new file when writing a new file begins, and to fill the inherent information of the vehicle into the data structure defined by the special frame area of ​​the new file to form a special frame. The second writing module 402 is used to collect the autonomous driving data of the vehicle according to a preset storage cycle, and write the autonomous driving data acquired in each storage cycle into a data frame area of ​​the new file as a data frame. The control module 403 is used to terminate the writing of the new file when the storage space of the new file is full or the writing time of the new file is reached.

[0076] As an optional embodiment, the data frame area includes a metadata storage area and a business data storage area, and the second writing module 402 is used for: For each storage cycle, the autonomous driving data acquired within the storage cycle is traversed to extract key information corresponding to different data sources. This key information is then filled into the data structure defined in the metadata storage area to obtain the metadata of a single data frame. The metadata contains a data description array composed of multiple data descriptions. The autonomous driving data acquired within the storage cycle is serialized to obtain the serialized byte streams of autonomous driving data from different data sources, which are stored in a byte stream array. The total length of the serialized byte streams of autonomous driving data from different data sources within the storage cycle is calculated. The metadata of a single data frame area is serialized, and the length and byte stream of the serialized metadata are written to the metadata field in the metadata storage area. The total length of the byte stream is also written to the business data field in the business data storage area. Furthermore, the length and byte stream of the serialized autonomous driving data from different data sources are written sequentially according to the byte stream array. The autonomous driving data from different data sources are matched one-to-one with the data descriptions in the data description array.

[0077] As an optional embodiment, calculating the total length of the serialized byte stream of autonomous driving data from different data sources within the storage period includes: multiplying the sum of the number of bytes occupied by the business data field and the length of the serialized autonomous driving data from different data sources by the number of data sources of the autonomous driving data obtained within the storage period to obtain the total length of the serialized byte stream within the storage period.

[0078] As an optional embodiment, the first writing module 401 includes: filling the inherent information of the vehicle into the data structure defined by the special frame area of ​​the new file; extracting key information from the inherent information and filling the key information into the data structure corresponding to the metadata of the special frame area to construct the metadata of the special frame; serializing the metadata of the special frame area and writing the length and byte stream of the serialized metadata into the metadata field of the special frame area; serializing the inherent information and calculating the total length of the byte stream after serialization of the inherent information, writing the total length of the byte stream into the business data field of the special frame area, and writing the length and byte stream of the serialized inherent information.

[0079] As an optional embodiment, a single data frame area includes a metadata storage area and a business data storage area; the metadata storage area includes a 4-byte metadata length field and a byte stream of metadata serialization, and the business data storage area includes a 4-byte metadata total length field and at least one set of business data segments, the business data segments including a 4-byte business data length field and a byte stream of business data serialization.

[0080] Each of the above modules can be implemented using software code, in which case they can be stored in the memory of the control device. Alternatively, each of the above modules can be implemented using hardware, such as integrated circuit chips.

[0081] The file writing device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0082] Fourthly, based on the same inventive concept, this embodiment provides a computing device 500, such as... Figure 9 As shown, it includes: a memory 501, a processor 502, and a computer program 503 stored on the memory 501 and executable on the processor 502. When the processor 502 executes the program, it implements the steps of the file writing method described in the first aspect above.

[0083] Since the computing device described in this embodiment is the computing device used to implement the file writing method in the embodiments of this application, those skilled in the art can understand the specific implementation methods and various variations of the computing device in this embodiment based on the file writing method described in the embodiments of this application. Therefore, how the computing device implements the method in the embodiments of this application will not be described in detail here. As long as those skilled in the art implement the computing device used in the file writing method in the embodiments of this application, it falls within the scope of protection of this application.

[0084] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A module that specifies the function in one or more boxes.

[0086] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction modules implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0088] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0089] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims and their equivalents, this invention also intends to include these modifications and variations in the description.

Claims

1. A file writing method, characterized in that, An autonomous driving domain controller for vehicles, configured to write files using a preset file format, the preset file format including: a file header, a special frame area, and multiple data frame areas, the method including: When writing a new file begins, file format version information is written to the header of the new file, and the vehicle's inherent information is filled into the data structure defined in the special frame area of ​​the new file to form a special frame. The autonomous driving data of the vehicle is collected according to a preset storage period, and the autonomous driving data acquired in each storage period is sequentially written into a data frame area of ​​the new file as a data frame. The writing of the new file ends when the storage space for the new file is full or the writing time for the new file is reached.

2. The file writing method as described in claim 1, characterized in that, The data frame area includes a metadata storage area and a business data storage area. The step of sequentially writing the autonomous driving data acquired in each storage cycle into a data frame area of ​​the new file includes: For each storage cycle, the autonomous driving data acquired within the storage cycle is traversed to extract key information corresponding to different data sources from the autonomous driving data, and the key information is filled into the data structure defined in the metadata storage area to obtain the metadata of a single data frame. The metadata contains a data description array composed of multiple data descriptions. The autonomous driving data acquired within the storage period is serialized to obtain byte streams of autonomous driving data from different data sources, and then stored in a byte stream array. Calculate the total length of the serialized byte stream of autonomous driving data from different data sources within the storage period; The metadata of the single data frame area is serialized, and the length and byte stream of the serialized metadata are written into the metadata field of the metadata storage area. The total length of the byte stream is written into the business data field of the business data storage area. The length and byte stream of the serialized autonomous driving data from different data sources are written sequentially according to the byte stream array. The autonomous driving data from different data sources are matched one-to-one with the data descriptions in the data description array.

3. The file writing method as described in claim 2, characterized in that, The calculation of the total length of the serialized byte stream of autonomous driving data from different data sources within the storage period includes: The total length of the serialized byte stream within the storage period is obtained by multiplying the sum of the number of bytes occupied by the business data field and the length of the serialized autonomous driving data from different data sources by the number of data sources of the autonomous driving data acquired within the storage period.

4. The file writing method as described in claim 1, characterized in that, The step of filling the inherent information of the vehicle into the data structure defined by the special frame area of ​​the new file to form a special frame includes: The inherent information of the vehicle is filled into the data structure defined by the special frame area of ​​the new file; Key information is extracted from the inherent information and filled into the data structure corresponding to the metadata of the special frame area to construct the metadata of the special frame. The metadata of the special frame area is serialized, and the length and byte stream of the serialized metadata are written into the metadata field of the special frame area. The inherent information is serialized, and the total length of the serialized byte stream is calculated. The total length of the byte stream, as well as the length and byte stream of the serialized inherent information, are written into the service data field of the special frame area.

5. The file writing method as described in claim 1, characterized in that, A single data frame area includes a metadata storage area and a business data storage area; The metadata storage area includes a 4-byte metadata length field and a byte stream of serialized metadata. The business data storage area includes a 4-byte metadata total length field and at least one set of business data segments. The business data segment includes a 4-byte business data length field and a byte stream of serialized business data.

6. A file parsing method, characterized in that, An autonomous driving domain controller for vehicles is configured to parse files stored in a preset file format. The preset file format includes a file header, a special frame area, and multiple data frame areas. The file header records file format version information, the special frame area is filled with vehicle-specific information, and the multiple data frame areas include multiple frames of autonomous driving data, with each data frame area storing one frame of autonomous driving data. The parsing method includes: Initialize a pre-established key-value pair data structure, which is used to store the parsed data; Read the file header and the inherent information in the special frame region; According to the storage time order, each data frame area is parsed sequentially. The parsing steps include: deserializing the autonomous driving data stored in the data frame area, and after parsing one piece of data in each data frame area, storing the parsed data into the key-value pair data structure. Complete the parsing of all data frames and output the file.

7. The file parsing method as described in claim 6, characterized in that, The data frame area includes a metadata storage area and a business data storage area. The metadata storage area stores the length and byte stream of the serialized metadata. The business data storage area stores the total length of the byte stream of business data, the length and byte stream of the serialized autonomous driving data from different data sources. The deserialization of the autonomous driving data stored in the data frame area includes: The system reads the length of the byte stream after metadata serialization and the byte stream data corresponding to the byte stream length. It then deserializes the byte stream data according to the metadata definition, parses out the data description, iterates through the data description, and stores the data with the flag bit set to true in the data description into the current frame data description array. The data description in the data description array is matched one-to-one with the autonomous driving data from different data sources. Read the total length of the business data in the current data frame area, and perform the following matching and parsing steps on different data in the order of storage of individual data sources: obtain the length of the autonomous driving data of a single data source and the byte stream data under that length, obtain the business data definition according to the data description that matches the data source, and deserialize the byte stream data according to the business data definition until the matching and parsing of all data is completed.

8. A file writing device, characterized in that, include: The first writing module is used to write file format version information to the header of the new file when writing a new file begins, and to fill the inherent information of the vehicle into the data structure defined by the special frame area of ​​the new file to form a special frame. The second writing module is used to collect the autonomous driving data of the vehicle according to a preset storage cycle, and write the autonomous driving data acquired in each storage cycle into a data frame area of ​​the new file as a data frame. The control module is used to terminate the writing of the new file when the storage space of the new file is full or the writing time of the new file is reached.

9. A computing device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 5.

Images