Data storage method, data reading method, data acquisition system, medium and equipment

Abstract

The invention provides a data storage method, a data reading method, a data acquisition system, a medium and equipment, and relates to the technical field of automatic driving. The method comprises the steps of firstly obtaining target data of a vehicle end and recording the target data according to a first preset format to obtain a plurality of meta record data; writing the version information of the data protocol into the head of the original blank file; based on the record type of the meta record data, writing the multiple meta record data groups into multiple data groups of the data part of the original blank file; writing index information of a plurality of meta-record data in each data group into an abstract part of the original blank file; writing the index information of each data group into the abstract index part of the original blank file; and writing index information of the abstract part and the abstract index part into the tail of the original blank file to obtain a target data file. Therefore, through a standardized data structure and an efficient data storage strategy, complex and diversified data storage and access requirements in intelligent driving system application are met.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a data storage method, a reading method, a collection system, a medium and a device. Background Art

[0002] With the rapid development of intelligent driving technology, unmanned driving systems have gradually moved from experiments to applications, bringing great convenience to people's travel. In intelligent driving systems, data collection, processing, storage and transmission are crucial, and they are the core technical framework of intelligent driving systems. Among them, data storage is an indispensable part of intelligent driving systems, and its performance directly affects the stability and reliability of the system.

[0003] In the current intelligent driving system, the existing data storage protocols often have problems such as poor compatibility, insufficient scalability, and weak real-time performance, which makes it difficult to meet the actual needs of the intelligent driving system. Therefore, a new intelligent driving data storage protocol is urgently needed to achieve effective storage of intelligent driving data. Summary of the invention

[0004] The present application provides a data storage method, a reading method, a collection system, a medium and a device, which can improve data processing efficiency and realize fast and effective storage of intelligent driving data.

[0005] A first aspect of an embodiment of the present application provides a data storage method, the method comprising:

[0006] Acquire target data on the vehicle side, and record the target data in a first preset format to obtain a plurality of meta-record data; the first preset format includes a record type, a record length, and a record content;

[0007] Write the version information of the data protocol into the header of the original blank file;

[0008] Based on the record type of the meta record data, write a plurality of the meta record data groups into a plurality of data groups in the data portion of the original blank file;

[0009] Writing index information of the plurality of meta record data in each of the data groups into a summary part of the original blank file;

[0010] Writing the index information of each data group into the summary index part of the original blank file;

[0011] The index information of the summary part and the summary index part is written into the tail of the original blank file to obtain a target data file.

[0012] Optionally, the summary part includes statistical records and index records;

[0013] Writing index information of the plurality of meta record data in each data group into the summary part of the original blank file comprises:

[0014] Determine statistical information of a plurality of meta record data in each of the data groups, and write the statistical information into the statistical record of the summary part according to a first preset format;

[0015] The index information of the plurality of meta record data in each of the data groups is written into the index record of the summary part according to a first preset format.

[0016] Optionally, the data group includes a message group, an attachment group and a block group;

[0017] The statistical information includes: the number of the metadata record data of the message group, the number of the metadata record data of the attachment group, the number of the metadata record data of the block group, the minimum time and the maximum time of all the metadata record data, the information table of the message group, the information table of the attachment group and the information table of the block group.

[0018] Optionally, the index record includes a message index for the message group, an attachment index for the attachment group, and a block index for the block group;

[0019] Writing index information of the plurality of meta record data in each data group into the index record of the summary part according to a first preset format includes:

[0020] Writing index information of a plurality of the meta record data in the message group into the message index according to a first preset format;

[0021] Writing index information of the plurality of metadata record data in the attachment group into the attachment index according to a first preset format;

[0022] The index information of the plurality of meta record data in the block group is written into the block index according to a first preset format.

[0023] Optionally, the message index, the attachment index and the block index are index tables formed based on names and / or times of a plurality of the meta record data.

[0024] Optionally, based on the type of the meta record data, after writing a plurality of the meta record data groups into corresponding data groups in the data portion of the original blank file, the method further comprises:

[0025] When the meta record data written into the data portion reaches a preset packetization condition, the writing of the meta record data is stopped.

[0026] Based on the same inventive concept, a second aspect of an embodiment of the present application provides a data reading method, the method comprising:

[0027] Acquire a target data file; the target data file is obtained according to the data storage method described in the first aspect;

[0028] Read the header of the target data file to obtain version information of the storage data protocol;

[0029] Read the tail of the target data file to obtain index information of the summary part and the summary index part;

[0030] Based on the index information of the summary index part, reading the index information of each data group from the summary index part;

[0031] Based on the index information of the summary part and the index information of the data group, reading the index information of the plurality of meta record data in the data group from the summary part;

[0032] Based on the index information of the meta record data, the target data is read.

[0033] Based on the same inventive concept, a third aspect of the embodiment of the present application provides a data acquisition system, the system comprising a vehicle end and an industrial computer;

[0034] The vehicle end is used to collect raw data and convert the format of the raw data to obtain target data, and send the target data to the industrial computer;

[0035] The industrial computer is used to store the target data according to the data storage method described in the first aspect, obtain a target data file, and upload the target data file to the cloud.

[0036] Based on the same inventive concept, a fourth aspect of an embodiment of the present application provides a data storage device, the device comprising:

[0037] A recording module, used to obtain target data of the vehicle end, and record the target data in a first preset format to obtain a plurality of meta-record data; the first preset format includes a record type, a record length and a record content;

[0038] A header writing module is used to write the version information of the data protocol into the header of the original blank file;

[0039] A data writing module, configured to write a plurality of the meta record data groups into a plurality of data groups in the data portion of the original blank file based on the record type of the meta record data;

[0040] A summary writing module, used for writing index information of the plurality of meta record data in each of the data groups into the summary part of the original blank file;

[0041] A summary index writing module, used for writing the index information of each data group into the summary index part of the original blank file;

[0042] The tail writing module is used to write the index information of the summary part and the summary index part into the tail of the original blank file to obtain a target data file.

[0043] Based on the same inventive concept, a fifth aspect of an embodiment of the present application provides a data reading device, the device comprising:

[0044] A file acquisition module, used to acquire a target data file; the target data file is obtained according to the data storage method described in the first aspect;

[0045] A header reading module, used for reading the header of the target data file to obtain version information of the storage data protocol;

[0046] A tail reading module, used for reading the tail of the target data file to obtain index information of the summary part and the summary index part;

[0047] A summary index reading module, configured to read index information of each data group from the summary index part based on the index information of the summary index part;

[0048] A summary reading module, configured to read index information of a plurality of meta record data in the data group from the summary part based on the index information of the summary part and the index information of the data group;

[0049] The data reading module is used to read the target data based on the index information of the meta record data.

[0050] Based on the same inventive concept, the sixth aspect of an embodiment of the present application provides a storage medium, which stores machine executable instructions. When the machine executable instructions are executed by a processor, they implement the data storage method proposed in the first aspect of the present application or the data reading method proposed in the second aspect.

[0051] A seventh aspect of an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when executed, the processor implements the data storage method proposed in the first aspect of the present application or the data reading method proposed in the second aspect.

[0052] Compared with the prior art, this application has the following advantages:

[0053] The data storage method provided by the embodiment of the present application first obtains the target data of the vehicle side, and records the target data according to the first preset format to obtain multiple meta-record data; then writes the version information of the data protocol into the header of the original blank file; based on the record type of the meta-record data, writes the multiple meta-record data into multiple data groups of the data part of the original blank file; writes the index information of the multiple meta-record data in each data group into the summary part of the original blank file; writes the index information of each data group into the summary index part of the original blank file; writes the index information of the summary part and the summary index part into the tail of the original blank file to obtain the target data file. Thus, after obtaining various different data of the vehicle side, the data are abstracted into a series of identical meta-record data, and are uniformly stored in a data file with a standardized structure according to the corresponding data storage strategy, so as to meet the complex and diverse data storage and access requirements in the application of intelligent driving system through a standardized data structure and an efficient and flexible data storage strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0055] Figure 1 is a flow chart of a data storage method in one embodiment of the present application;

[0056] Figure 2 It is a schematic diagram of the structure of a meta record data in one embodiment of the present application;

[0057] Figure 3 is a schematic diagram of the structure of a target data file in one embodiment of the present application;

[0058] Figure 4 is a schematic diagram of the structure of the data portion of the target data file in one embodiment of the present application;

[0059] Figure 5 is a schematic diagram of the structure of the summary part of the target data file in one embodiment of the present application;

[0060] Figure 6 is a schematic diagram of the structure of the summary index part of the target data file in one embodiment of the present application;

[0061] Figure 7 is a schematic structural diagram of the tail of a target data file in one embodiment of the present application;

[0062] Figure 8This is a schematic diagram of the structure of message data in an embodiment of the present application;

[0063] Fig. 9 It is a schematic diagram of the structure of the statistical record of the summary part of the target data file in one embodiment of the present application;

[0064] Fig.10 It is a schematic diagram of the structure of the index record of the summary part of the target data file in one embodiment of the present application;

[0065] Fig.11 is a flow chart of a data reading method in one embodiment of the present application;

[0066] Fig.12 This is an application schematic diagram of a data acquisition system in one embodiment of the present application;

[0067] Fig.13 This is a functional module diagram of a data storage device in one embodiment of the present application;

[0068] Fig.14 This is a functional module diagram of a data reading device in one embodiment of the present application;

[0069] Fig.15 It is a structural schematic diagram of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0071] With the rapid development of intelligent driving technology, unmanned driving systems have gradually moved from experiments to applications, bringing great convenience to people's travel. In intelligent driving systems, data collection, processing, storage and transmission are all crucial, and they are the core technical framework of intelligent driving systems. Among them, data storage is an indispensable part of intelligent driving systems, and its performance directly affects the stability and reliability of the system. In current intelligent driving systems, data storage mainly faces the following problems:

[0072] 1. Diversity of data formats: Intelligent driving systems need to process many types of data, including sensor data, map data, vehicle status data, etc. These data often have different formats and characteristics, and a unified data storage protocol is required to be compatible with different data types.

[0073] 2. Large amount of data: With the continuous advancement of intelligent driving technology, the amount of data generated by the system is also growing rapidly. How to effectively store this data and ensure its integrity and availability has become a problem that needs to be solved urgently.

[0074] 3. Poor data security: The intelligent driving system involves the safety of the vehicle, so the security of data storage is very important. Therefore, a safe and reliable storage protocol needs to be designed to prevent data from being illegally obtained or tampered with.

[0075] 4. Low reading and writing efficiency: The intelligent driving system has high requirements for the real-time and efficiency of data, and requires a data storage protocol that can respond quickly and process efficiently to support the real-time operation of the system.

[0076] However, existing data storage protocols often have problems such as poor compatibility, insufficient scalability, and weak real-time performance, which make it difficult to meet the actual needs of intelligent driving systems. Therefore, a new type of intelligent driving data storage protocol is urgently needed to achieve effective storage of intelligent driving data.

[0077] In view of this, this application proposes a data storage method, which, after acquiring various different data on the vehicle side, abstracts all the data into a series of identical meta-record data, and stores them uniformly in a data file with a standardized structure according to the corresponding data storage strategy. Thus, by designing a standardized data structure and an efficient and flexible data storage strategy, the complex and diverse data storage and access requirements in the application of intelligent driving systems can be met.

[0078] Please refer to Figure 1 , Figure 1 1 is a flow chart of a data storage method proposed in one embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0079] S101: Acquire target data on the vehicle side, and record the target data according to a first preset format to obtain a plurality of meta-record data.

[0080] In this embodiment, the target data refers to data related to the intelligent driving system on the vehicle side, including image data collected by cameras, point cloud data collected by lidar, sensor data collected by complex sensors such as IMU (Inertial Measurement Unit), etc. Since these target data usually have different formats and characteristics, there will often be format incompatibility problems if they are stored in ordinary data storage formats. In this regard, the present application first designs a standardized data structure, namely the first preset format, to record the target data, so as to abstract the target data into a series of meta-record data (Record) with the same format, thereby improving the efficiency of subsequent data processing and analysis.

[0081] For details, please refer to Figure 2 , Figure 2 Schematic diagram of the structure of a metadata record data in one embodiment of the present application. Figure 2 As shown, the first preset format includes a record type (record type), a record length (record content length) and a record content (record content). Among them, the record type represents the type of the target data recorded, including message data, attachment data and block data, etc. It is distinguished by the Op character code (operation code, character encoding), stored in the Uint8 type, and occupies 1 byte. The record length represents the length of the target data recorded, stored in the Uint64 type, and occupies 8 bytes. The record content represents the data value (value) of the target data recorded, including the target data itself (data), the pointer (char*) and the length (data len Byte) of the target data.

[0082] It is easy to understand that these numerous meta-record data may be data of different sensor types recorded on the vehicle side, or data of different parameters of the same sensor, or different types of data such as vehicle status information and CAN data sent by the domain controller on the vehicle side. Thus, the data of different formats on the vehicle side are unified into the same first preset format to obtain multiple meta-record data.

[0083] S102: Writing the version information of the data protocol into the header of the original blank file.

[0084] In this embodiment, the structure of the original blank file (.dat) is as follows Figure 3 As shown, it includes a header (Ver), a data section (Data Section), a summary section (Summary Section), a summary index section (Summary Offset Section), and a footer section (Footer Section). Among them, the header is mainly used to store the version information of the data protocol, that is, the version of the DAT file.

[0085] In the actual storage process, the version information can be encoded using ASCII encoding, and after obtaining the corresponding ASCII code, it is written into the header of the original blank file. When reading, the corresponding characters are read from the header and then parsed according to the ASCII table.

[0086] S103: Based on the record type of the meta record data, write the multiple meta record data groups into multiple data groups in the data part of the original blank file.

[0087] In this embodiment, the data part is used to store the above-mentioned multiple metadata records. Figure 4 , Figure 4 Schematic diagram of the structure of the data part of the target data file in one embodiment of the present application. Figure 4 As shown, the data part includes multiple data groups, such as message group (topic), attachment group (attachment) and block group (chunk), etc., so as to group and store many different types of meta record data, so as to facilitate the subsequent rapid reading of the desired target data.

[0088] Among them, the message group contains all the data related to a specific topic. For example, in the intelligent driving database, there may be multiple Topics such as "map", "user", "fusion information", etc., and each Topic contains all the data related to it. The data of the attachment group includes serialized data, table data, and MP4 audio data. The data of the block group is a compressed package compressed from multiple message-type data or multiple attachment-type data, that is, multiple message-type data form a block or multiple attachment-type data form a block to reduce the use of storage space. For example, it can be a block composed of different sensor data of a module of the intelligent driving system.

[0089] In the actual storage process, the metadata record data with the encoding type of 0x02 is written into the message group, the metadata record data with the encoding type of 0x03 is written into the attachment group, and the metadata record data with the encoding type of 0x0A is written into the block group to realize the storage of data. It should be noted that in addition to the above three data groups, other types of data groups can also be set, which can be expanded according to the needs of data storage.

[0090] Furthermore, Figure 4 The structure of the date part (data part) of the message group (topic) in is as follows Figure 8 As shown, it includes the message header, message name length, message timestamp, message identifier, message metadata length, message name pointer and message metadata pointer. The structure of the date part (data part) of the attachment group is similar to that of the attachment group, including the attachment header, attachment name length, attachment timestamp, attachment type, attachment data length, attachment name pointer and attachment data pointer. Among them, the attachment data includes proto (short for Protocol Buffers, a data description language) serialized data, table data and MP4 audio data. The structure of the date part (data part) of the chunk group is similar to that of the attachment group, so it will not be repeated here.

[0091] S104: Write the index information of the plurality of meta record data in each data group into the summary part of the original blank file.

[0092] In this implementation, the summary part is used to group and store the index information of the metadata record data. Figure 5 , Figure 5 Schematic diagram of the structure of the summary part of the target data file in one embodiment of the present application. Figure 5 As shown, the summary part mainly includes two types of records: statistical records (coding type is 0x04) and index records. Among them, there is only one statistical record, which is the first and fixed record in the summary part. It marks the end of the data part and the beginning of the summary part. The statistical record is mainly used to store the overall overview of all meta-record data, such as the number, start time, end time, etc. of all meta-record data. Secondly, the index record is used to store the index information of the meta-record data (similar to the table of contents of a book). There may be multiple index records, and one index record corresponds to one data group. For example, Topic Index (message index, coding type is 0x05) corresponds to the message group, attachment Index (attachment index, coding type is 0x07) corresponds to the attachment group, and Chunk Index (block index, coding type is 0x0B) corresponds to the block group, so as to record the index information of multiple meta-record data in each data group respectively.

[0093] In the actual storage process, the index information of multiple meta-record data in each data group is sequentially written into the index record corresponding to the data group to achieve the storage of the data index. When reading the data later, the address of a meta-record data can be found from the index record in the summary part, and then the meta-record data can be read. Thus, it helps users to find the required data more conveniently and improves the efficiency of data reading.

[0094] S105: Write the index information of each data group into the summary index part of the original blank file.

[0095] In this embodiment, the summary index part is used to store the index information of the data group. Figure 6 , Figure 6 Schematic diagram of the structure of the summary index part of the target data file in one embodiment of the present application. Figure 6 As shown, the summary index part may include multiple summary records (encoding type is 0x08), and the date part (data part) of any summary record stores the index information of a data group, including the encoding type (opcode), starting address (groupstart), and length (grouplenth) of the data group.

[0096] The relationship between the summary part and the summary index part can be understood as a multi-level directory. The summary index part is equivalent to the first-level directory, which stores the address and other information of each data group, and the summary part is equivalent to the second-level directory under the first-level directory, which stores the address and other information of each meta record data in each data group. Therefore, through this multi-level index form, users can find the required data more conveniently, and it is also convenient for system administrators to manage and maintain data content.

[0097] In the actual storage process, by writing the index information of each data group into the summary record Record, the user can quickly locate (retrieve) and access (read) the data stored in the DAT file when reading the data.

[0098] S106: Write the index information of the summary part and the summary index part into the tail of the original blank file to obtain the target data file.

[0099] In this implementation, the tail is used to store the index information of the summary part and the summary index part. Figure 7 , Figure 7 Schematic diagram of the structure of the tail of the target data file in one embodiment of the present application. Figure 7 As shown, the tail contains only one record Record, and its date part (data part) stores the offset (offset) from the starting position of the DAT file to the summary part and the offset (offset) from the starting position of the DAT file to the summary index part, that is, the starting address of the summary part and the starting address of the summary index part.

[0100] In the actual storage process, by writing the index information of the summary part and the summary index part into the tail record Record, the user can first locate the location of the summary index part and the summary part when reading data, and then read the desired target data step by step according to the multi-level index.

[0101] The present application designs a standardized data structure and a corresponding data storage strategy, which abstracts various different data on the vehicle side into a series of identical meta-record data, and uniformly stores them in a data file with a standardized structure according to the corresponding data storage strategy, so that structured and unstructured data of multiple streams can be recorded in one DAT file, with good data reading, writing and positioning performance, so that users can quickly access and retrieve the data stored in the file.

[0102] like Figure 6 As shown, the summary part includes statistical records and index records. The above step of writing the index information of the plurality of meta-record data in each data group into the summary part of the original blank file specifically includes:

[0103] S201: Determine statistical information of a plurality of meta record data in each data group, and write the statistical information into a statistical record of a summary part according to a first preset format.

[0104] In this embodiment, if Figure 6 As shown in the figure, the structure of the statistical record is also in the form of a Record. The statistical record is the first and fixed record in the summary part, marking the end of the data part and the beginning of the summary part. The date part of the statistical record stores the statistical information (statistics) of all the meta-record data of the entire DAT file. Users can understand the overall overview of all the meta-record data through statistics.

[0105] For details, please refer to Fig. 9 , Fig. 9 FIG. 1 is a schematic diagram of the structure of the statistical record of the summary part of the target data file in one embodiment of the present application. Fig. 9 As shown, the statistical information includes: the number of meta-record data of the message group, the number of meta-record data of the attachment group, the number of meta-record data of the block group, the minimum time and maximum time of all meta-record data, the information table of the message group, the information table of the attachment group and the information table of the block group. Among them, the information table of the message group records the name of the message group (topic_name), the size of the meta-record data in the message group (topic_datasize), the number (topic_counts), the start time (start_time), the end time (end_time) and the maximum interval time (max_gap) between two meta-record data. The information table of the attachment group records the name of the attachment group (attachment_name), the size of the meta-record data in the attachment group (data_size) and other contents. The information table of the block group records similar contents.

[0106] In the actual storage process, by writing the statistical information of each data group into a statistical record in a standardized format in sequence, the user can more quickly and intuitively understand the relevant information of the data stored in the DAT file during the reading process.

[0107] S202: writing index information of a plurality of meta record data in each data group into an index record of a summary part according to a first preset format.

[0108] In this embodiment, if Figure 6 As shown, the structure of the index record is also in the form of a Record. There may be multiple index records, and one index record corresponds to one data group. For example, the index record may include a message index for a message group, an attachment index for an attachment group, and a block index for a block group.

[0109] In the case of three data groups including a message group, an attachment group and a block group, when writing the index information of the meta-record data, the index information of the multiple meta-record data in the message group is written into the message index according to a first preset format, the index information of the multiple meta-record data in the attachment group is written into the attachment index according to the first preset format, and the index information of the multiple meta-record data in the block group is written into the block index according to the first preset format, so as to realize the storage of the index information of all the meta-record data.

[0110] Furthermore, the message index, attachment index and block index are index tables composed of the names and / or times of multiple metadata records. Fig.10 , Fig.10 It is a schematic diagram of the structure of the index record of the summary part of the target data file in one embodiment of the present application.

[0111] like Fig.10 As shown, it is assumed that there are multiple meta-record data in the current message group and one meta-record data in the attachment group. When storing the index information of these data, this embodiment constructs two index records based on the names and times of the multiple meta-record data for the multiple meta-record data in the message group, including a message index (TopicIndex, encoding type is 0x05) and a message timestamp index (TopicPeriodIndex, encoding type is 0x06), and the data part of the message index and the message timestamp index specifically records the index information of the multiple meta-record data in the form of an index table, namely a name index table and a time index table. Among them, the name index table uses the name (topic_name) as the header, and records the offset (offset) and length (len) corresponding to the meta-record data of different names. The time index table uses the storage time (time_period) as the header, and records the offset and length corresponding to the meta-record data of different times. At the same time, for a meta record data in the attachment group, only one index record (attachment index) is constructed, and the data part of the attachment index (AttachmentIndex, encoding type 0x07) directly records the offset and length of the only meta record data in the group.

[0112] It is easy to understand that when there are multiple meta record data in the attachment group, the index information of the multiple meta record data can be recorded based on the names and times of the multiple meta record data in the attachment group according to the construction method of the message index and the message timestamp index. In addition, the block group is similar, which will not be repeated here.

[0113] In this embodiment, a corresponding index table is constructed based on the name and time of the meta-record data, so that the index information of the meta-record data is recorded according to the name and time of the meta-record data respectively. Therefore, when reading data, the user can directly read the desired data quickly according to the name index or the time index, without having to read all the data to extract the desired data as in the prior art, thereby achieving the purpose of improving data reading efficiency.

[0114] Optionally, after writing the plurality of metadata record data groups into corresponding data groups in the data portion of the original blank file based on the type of the metadata record data, the method further comprises:

[0115] When the meta record data of the written data part reaches the preset packetization condition, the writing of the meta record data is stopped.

[0116] In this embodiment, the subpackaging condition can be set based on time or the size of the meta record data. For example, when the time for continuously writing data to the data part exceeds a preset time threshold, it is considered that the preset subpackaging condition is met. Alternatively, when the amount of data written to the data part exceeds a preset data amount threshold, it is considered that the preset subpackaging condition is met. Exemplarily, the time threshold can be set to 30 seconds, 1 minute, 5 minutes, etc., and the data amount threshold can be set to 1GB, 2GB, 5GB, etc.

[0117] When the preset subpackaging conditions are met, the writing of the meta-record data is stopped. Then, the summary part, summary index part and tail part of the data are written in sequence according to the above steps, and finally a target data file, namely, a DAT file, is formed. Thus, users can flexibly adjust the storage strategy and subpack by size and time to meet the needs of specific scenarios.

[0118] Based on the same inventive concept, a second aspect of an embodiment of the present application provides a data reading method, the method comprising:

[0119] S301: Acquire a target data file; the target data file is obtained according to the data storage method in the first aspect;

[0120] S302: Read the header of the target data file to obtain version information of the storage data protocol;

[0121] S303: Read the tail of the target data file to obtain index information of the summary part and the summary index part;

[0122] S304: Based on the index information of the summary index part, read the index information of each data group from the summary index part;

[0123] S305: Based on the index information of the summary part and the index information of the data group, read the index information of the plurality of meta record data in the data group from the summary part;

[0124] S306: Read the target data based on the index information of the meta record data.

[0125] In this embodiment, the index information includes at least a memory address and a length. The data reading process can be abstractly understood as a dictionary search process, first opening the dictionary homepage, then searching for the first letter, then searching for the syllable, and finally finding the word or phrase.

[0126] For details, please refer to Fig.11 , Fig.11 1 is a flow chart of a data reading method in one embodiment of the present application. Fig.11 As shown, when reading data, first open the .dat file (i.e., the target data file) and read the version information of the fixed bytes in the file header (equivalent to opening the first page of the dictionary). Then, read the tail record to obtain the offset of the first record in the summary part, which is the statistical record (Summary Start) and the offset of the first summary record in the summary index part (SummaryOffset Start), based on which the summary index part and the summary part can be located (equivalent to checking the first letter). Next, read a set of index records from the summary index part to obtain the memory address and length of each index record (i.e., each data group) in the summary part (equivalent to checking syllables). After that, read the memory address and length of multiple meta-record data in each data group from each index record in the summary part, so that a certain record Record or a certain period of time Records can be read (equivalent to finding a word or a term).

[0127] Based on the same inventive concept, a third aspect of the embodiment of the present application provides a data acquisition system, the system includes a vehicle end and an industrial computer;

[0128] The vehicle end is used to collect raw data and convert the format of the raw data to obtain target data, and then send the target data to the industrial computer;

[0129] The industrial computer is used to store the target data according to the data storage method in the first aspect, obtain the target data file, and upload the target data file to the cloud.

[0130] Please refer to Fig.12 , Fig.12 Schematic diagram of an application of a data acquisition system in one embodiment of the present application. Fig.12As shown, after the vehicle end collects raw data of different formats and types from domain controllers, CAN devices, sensors, etc., it first converts the format of the raw data to obtain target data, and then forwards the target data to the industrial computer through the switch. After the data acquisition software in the industrial computer receives the target data, it stores the target data according to the data storage method in the first aspect to obtain the target data file, and uploads the target data file to the cloud server to provide a basis for rapid data reading.

[0131] Among them, the vehicle side can use the Proto serialization method when converting the format of the original data. In the specific implementation, it is necessary to first define the message structure, that is, the organization method and field type of the data, which can be defined using the syntax of Protocol Buffers, including message name, field name, field type, etc. After defining the good message structure, use the protocolBuffers compiler to compile the .proto file into the corresponding code file. The compiler will generate code in the corresponding language according to the definition in the .proto file, including message class, serialization and deserialization methods, etc. proto uses binary format to serialize and deserialize data. Serialization is the process of converting message objects into binary data, while deserialization is the process of converting binary data into message objects. This can effectively reduce the size of data and improve data transmission efficiency. At the same time, its efficient encoding encryption mechanism can prevent data from being illegally obtained or tampered with during transmission.

[0132] At the same time, when the industrial computer stores the target data, it can adopt a distributed storage method to disperse the data on the vehicle side on multiple physical nodes (different nodes have different names). During data transmission, data can be sent through Topic. A topic should contain information such as topic header length, topic header content, topic name address, and metadata address. Furthermore, for each functional module on the vehicle side that needs to record data, the corresponding node is used to receive data, and different nodes can be transmitted through different channels, so that even if an error occurs in one node, it will not affect the recording and storage of data on other nodes.

[0133] Please refer to Fig.13 Based on the same inventive concept, a fourth aspect of the embodiment of the present application provides a data storage device, the data storage device 13 comprising:

[0134] The recording module 131 is used to obtain the target data of the vehicle end, and record the target data according to a first preset format to obtain a plurality of meta-record data; the first preset format includes a record type, a record length and a record content;

[0135] A header writing module 132, used to write the version information of the data protocol into the header of the original blank file;

[0136] A data writing module 133, configured to write the plurality of meta-record data groups into the plurality of data groups of the data part of the original blank file based on the record type of the meta-record data;

[0137] Summary writing module 134, used to write index information of multiple meta record data in each data group into the summary part of the original blank file;

[0138] A summary index writing module 135, used to write the index information of each data group into the summary index part of the original blank file;

[0139] The tail writing module 136 is used to write the index information of the summary part and the summary index part into the tail of the original blank file to obtain the target data file.

[0140] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts may refer to the partial description of the method embodiment in the first aspect above.

[0141] Please refer to Fig.14 Based on the same inventive concept, the fifth aspect of the embodiment of the present application provides a data reading device, the data reading device 14 includes:

[0142] A file acquisition module 141 is used to acquire a target data file; the target data file is obtained according to the data storage method in the first aspect;

[0143] The header reading module 142 is used to read the header of the target data file and obtain the version information of the storage data protocol;

[0144] The tail reading module 143 is used to read the tail of the target data file to obtain the index information of the summary part and the summary index part;

[0145] A summary index reading module 144, configured to read index information of each data group from the summary index part based on the index information of the summary index part;

[0146] A summary reading module 145, configured to read index information of a plurality of meta record data in a data group from the summary part based on the index information of the summary part and the index information of the data group;

[0147] The data reading module 146 is used to read the target data based on the index information of the meta record data.

[0148] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts may refer to the partial description of the method embodiment in the second aspect above.

[0149] Based on the same inventive concept, the sixth aspect of an embodiment of the present application provides a storage medium, in which machine executable instructions are stored. When the machine executable instructions are executed by a processor, the data storage method proposed in the first aspect of the present application or the data reading method proposed in the second aspect is implemented.

[0150] It should be noted that the specific implementation of the storage medium in the embodiment of the present application refers to the specific implementation of the data storage method proposed in the first aspect of the embodiment of the present application or the data reading method proposed in the second aspect, and will not be repeated here.

[0151] Please refer to Fig.15 Based on the same inventive concept, the seventh aspect of the embodiment of the present application provides an electronic device 15, including a processor 151 and a memory 152; the memory 152 stores machine executable instructions that can be executed by the processor 151, and the processor 151 is used to execute the machine executable instructions to implement the data storage method proposed in the first aspect of the present application or the data reading method proposed in the second aspect.

[0152] It should be noted that the specific implementation of the electronic device 15 in the embodiment of the present application refers to the specific implementation of the data storage method proposed in the first aspect of the embodiment of the present application or the data reading method proposed in the second aspect, and will not be repeated here.

[0153] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0154] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0156] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0158] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0159] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0160] The above is a detailed introduction to a data storage method, reading method, acquisition system, medium and equipment provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data storage method, characterized in that: The method comprises: Acquire target data on the vehicle side, and record the target data in a first preset format to obtain a plurality of meta-record data; the first preset format includes a record type, a record length, and a record content; Write the version information of the data protocol into the header of the original blank file; Based on the record type of the meta record data, write a plurality of the meta record data groups into a plurality of data groups in the data portion of the original blank file; Writing index information of the plurality of meta record data in each of the data groups into a summary part of the original blank file; Writing the index information of each data group into the summary index part of the original blank file; The index information of the summary part and the summary index part is written into the tail of the original blank file to obtain a target data file.

2. The method according to claim 1, characterized in that: The summary part includes statistical records and index records; Writing index information of the plurality of meta record data in each data group into the summary part of the original blank file comprises: Determine statistical information of a plurality of meta record data in each of the data groups, and write the statistical information into the statistical record of the summary part according to a first preset format; The index information of the plurality of meta record data in each of the data groups is written into the index record of the summary part according to a first preset format.

3. The method according to claim 2, characterized in that The data group includes a message group, an attachment group and a block group; The statistical information includes: the number of the metadata record data of the message group, the number of the metadata record data of the attachment group, the number of the metadata record data of the block group, the minimum time and the maximum time of all the metadata record data, the information table of the message group, the information table of the attachment group and the information table of the block group.

4. The method according to claim 3, characterized in that: The index record includes a message index for the message group, an attachment index for the attachment group, and a block index for the block group; Writing index information of the plurality of meta record data in each data group into the index record of the summary part according to a first preset format includes: Writing index information of the plurality of meta record data in the message group into the message index according to a first preset format; Writing index information of the plurality of metadata record data in the attachment group into the attachment index according to a first preset format; The index information of the plurality of meta record data in the block group is written into the block index according to a first preset format.

5. The method according to claim 4, characterized in that The message index, the attachment index and the block index are index tables formed based on the names and / or times of the plurality of metadata record data.

6. The method according to claim 1, characterized in that After writing a plurality of the metadata record data groups into corresponding data groups in the data portion of the original blank file based on the type of the metadata record data, the method further comprises: When the meta record data written into the data portion reaches a preset packetization condition, the writing of the meta record data is stopped.

7. A data reading method, characterized in that: The method comprises: Acquire a target data file; the target data file is obtained according to the data storage method described in any one of claims 1 to 6; Read the header of the target data file to obtain version information of the storage data protocol; Read the tail of the target data file to obtain index information of the summary part and the summary index part; Based on the index information of the summary index part, reading the index information of each data group from the summary index part; Based on the index information of the summary part and the index information of the data group, reading the index information of the plurality of meta record data in the data group from the summary part; Based on the index information of the meta record data, the target data is read.

8. A data acquisition system, characterized in that: The system includes a vehicle end and an industrial computer; The vehicle end is used to collect raw data and convert the format of the raw data to obtain target data, and send the target data to the industrial computer; The industrial computer is used to store the target data according to the data storage method described in any one of claims 1 to 6, obtain a target data file, and upload the target data file to the cloud.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data storage method according to any one of claims 1 to 6 or the data reading method according to claim 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data storage method according to any one of claims 1 to 6 or the data reading method according to claim 7 is implemented.

Images