Data recording or reading method, device, acquisition equipment, vehicle and medium

Abstract

The present application relates to the technical field of vehicle data processing, and particularly relates to a data recording or reading method, device, acquisition device, vehicle, and medium, including: identifying the actual capacity of target data to be recorded; if the actual capacity is less than the remaining storage capacity of a preset cache, writing the target data into the preset cache and updating the write cache index of the preset cache; if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, writing partial data of the target data into the preset cache, when the preset cache is full, writing the data in the preset cache into a record file of a storage device through an operation interface of an operating system, copying the remaining data of the target data to the head address of the preset cache, and updating the write cache index of the preset cache. Thus, the problems in the related art that high-frequency and large-volume data recording easily causes system cache cleaning, resulting in lags and reducing the safety of vehicle autonomous driving are solved.

Description

Technical Field

[0001] The present application relates to the technical field of vehicle data processing, and particularly relates to a data recording or reading method, device, acquisition device, vehicle, and medium. Background Art

[0002] With the development of technology, autonomous driving technology has been widely applied. Autonomous driving technology refers to the technology of simulating the behavior of a driver through an autonomous driving system on a vehicle to control the normal driving of the vehicle. Through autonomous driving technology, the operations that a driver needs to perform when driving a vehicle can be reduced, improving the driving experience of the driver. The autonomous driving system not only needs to obtain the operations required by the driver when driving a car, but also needs to obtain and collect relevant data information around the vehicle, such as lane lines, crowds, distances between front and rear vehicles, and obstacles, etc., and perform corresponding operations according to the relevant data information to improve safety.

[0003] A vehicle can record relevant data based on a data recording system of a distributed server network of a network management server and a data recording server, thereby effectively reducing the centralized load of the server and facilitating grouped management; or, complete the data recording of a data recorder through RS422 and RS232 interfaces; or, reduce the write processing to a data recording unit and increase the amount of data written each time to reduce the write overhead, and use high-speed sampling to ensure the data recording of sampling data with a large number of summary recording points, such as a data recording device.

[0004] In related technologies, in terms of data recording and unloading of a data recording device, a distributed data recording system, and a method for realizing high-speed sampling, it is found through actual tests that when the system cache is too high, the entire system will experience occasional lags, seriously affecting the safety of the autonomous driving system. Summary of the Invention

[0005] The present application provides a data recording or reading method, device, acquisition device, vehicle, and medium to solve the problems in related technologies that when recording a large amount of high-frequency data, it is easy to cause lags due to system cache cleaning, reducing the safety of vehicle autonomous driving.

[0006] An embodiment of the first aspect of the present application provides a data recording method, including the following steps: identifying the actual capacity of target data to be recorded; if the actual capacity is less than the remaining storage capacity of a preset cache, writing the target data into the preset cache and updating the write cache index of the preset cache; if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, writing partial data of the target data into the preset cache, and when the preset cache is full, writing the data in the preset cache into a record file of a storage device through an operation interface of an operating system, copying the remaining data of the target data to the head address of the preset cache, and updating the write cache index of the preset cache, where when receiving data for the first time after the record file is created, the kernel cache of the operating system is prohibited using a buffer-free flag.

[0007] According to the above technical means, the embodiment of the present application identifies the actual capacity of target data to be recorded, and performs different processing for different actual capacities of target data. When the actual capacity is less than the remaining storage capacity of the preset cache, the target data is written into the preset cache and the written cache index is updated; when the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, partial data is written into the preset cache and when the preset cache is full, the cached data is written into the record file of the storage device through the operating system interface, the remaining data is copied to the head address of the cache and the write cache index of the cache is updated, which can effectively solve the problem of jamming caused by system cache cleaning during high-frequency and large-volume data recording, and ensure the safety of vehicle autonomous driving.

[0008] Optionally, the head address of the preset cache is a multiple of a data block in the storage device, and the maximum cache capacity is a multiple of a data page in the storage device.

[0009] It can be understood that when the head address of the preset cache in the embodiment of the present application is a multiple of the data block in the storage device and the maximum cache capacity is a multiple of the data page in the storage device, the storage device can be directly written without passing through the operating system cache, thereby solving the occasional jamming problem of the operating system when the operating system cache is too high.

[0010] Optionally, writing the data in the preset cache into the record file of the storage device through the operation interface of the operating system includes: identifying whether the file capacity of the record file is greater than a preset capacity; if the file capacity is less than or equal to the preset capacity, writing the preset cache into the storage device and entering the next write cycle; if the file capacity is greater than the preset capacity, or an abnormal signal is received, calling an abnormal signal processing function to write the data in the preset cache into the storage device, closing the current record file after writing, and generating a new record file, and starting to record data again under the new record file.

[0011] According to the above technical means, in the embodiment of the present application, it is determined whether the file capacity of the record file is greater than the set capacity, and then the next operation is carried out. If it is less than or equal to, the cache is written into the storage device and the next cycle is entered; if it is greater than the preset capacity or an abnormal information is received, the abnormal signal processing function is called to write the cache data into the storage device, and after the writing, the record file is closed and a new file record is generated, and data recording is restarted under the new file. Therefore, data caching and packetization are required when recording data, so as to avoid losses caused by data loss and ensure data security.

[0012] Optionally, the calling the abnormal signal processing function to write the data in the preset cache into the storage device includes: determining whether the current cache byte number of the preset cache is an integer multiple of the data page in the storage device; if the current cache byte number is an integer multiple of the data page, writing the data in the preset cache into the storage device; if the current cache byte number is not an integer multiple of the data page, calculating the extra writing byte number according to the current cache byte number and the preset multiple of the data page, writing the extra writing byte number and the current cache byte number into the storage device together, and deleting the extra writing byte number after the writing.

[0013] According to the above technical means, in the embodiment of the present application, it is determined whether the current cache byte number of the preset cache is an integer multiple of the data page. If it is an integer multiple, the cache data is written into the storage device. If it is not an integer multiple, the extra writing byte number is calculated according to the current cache byte number and a certain multiple of the data page, and they are written into the storage device together and the extra bytes are deleted after the writing. Since the size of the data written into the storage device must be a multiple of the memory page size, it can be written directly into the storage device without passing through the operating system cache, thus solving the occasional freezing problem of the operating system when the operating system cache is too high.

[0014] Optionally, before identifying the remaining storage capacity of the preset cache defined in advance, it further includes: defining a data recording interface based on the serialized data recording protocol; calling the data recording interface to perform serialization processing on the received target data to obtain serialized data, and writing the serialized data into the preset cache.

[0015] According to the above technical means, in the embodiment of the present application, a data recording interface is defined based on the serialized data recording protocol, and the interface is called to perform serialization processing on the received target data to obtain serialized data and write it into the preset cache. The data can be converted into a specific format through automatic and manual data recording and other methods, so as to facilitate subsequent writing and storage.

[0016] Optionally, the serialized data recording protocol includes a file header structure, file header content, data header structure, and data header content. Defining a data recording interface based on the serialized data recording protocol includes: reading a configuration file to obtain a list of data type names; traversing the list of data type names to obtain message descriptors; if the message descriptor is empty, determining that the recorded information is incorrect, otherwise obtaining a dependent file and performing a recursive operation on the dependent file until the dependent file is cleared; initializing the file header structure, writing the interface content length of the file header structure and the file header content to a record file, and when recording each frame of data, defining a data header structure, serializing the data using a protocol buffer interface, and writing the updated data header content to the record file.

[0017] According to the above technical means, in the embodiment of the present application, all data type names that need to be recorded are predefined through a configuration file. When the program is initialized, all interface text contents of all interfaces are obtained at one time based on the configuration file, and the interface text contents are filled into the file header content part of the data, so that when playing back the data subsequently, the corresponding text contents of the interface files stored in the data file header can be read, and the interface text contents are processed and loaded into the data pool, thereby further supporting the subsequent deserialization of serialized data and realizing the interpretability of the record file itself.

[0018] Optionally, the serialized data recording protocol includes a file header structure, file header content, data header structure, and data header content. Defining a data recording interface based on the serialized data recording protocol includes: creating a list of data type names; defining a data header structure, serializing the data using a protocol buffer interface, and writing the updated data header content to the record file; before stopping recording data, defining the file header structure, arranging the interface content length of the file header structure and the file header content continuously, and inserting them into the record file.

[0019] According to the above technical means, in the embodiment of the present application, without a configuration file, it is determined whether the interface text content of the type variable has been processed and searched each time data is recorded. If not, a list of data type names is created to record the variable types for which the interface corresponding text content has been processed and searched, and a data header structure is defined, the data is serialized using a protocol buffer interface, and the updated data header content is written to the record file. Before stopping recording, the file header structure is defined, and its interface content length and file header content are arranged and inserted into the record file, so that when playing back the data subsequently, the corresponding text contents of the interface files stored in the data file header can be read, and the interface text contents are processed and loaded into the data pool, thereby further supporting the subsequent deserialization of serialized data and realizing the interpretability of the record file itself.

[0020] Optionally, while prohibiting the kernel cache of the operating system by using the unbuffered flag, it further includes: opening a file descriptor by using the open function of the operating system; opening the record file by using the file descriptor to create a target file.

[0021] According to the above technical means, the embodiment of the present application can open a file descriptor by using the open function of the operating system, open a record file by using the file descriptor to create a target file. Once the target file is created, it indicates that the data has been received, which can more intuitively prompt the user.

[0022] The embodiment of the second aspect of the present application provides a data reading method, including the following steps: obtaining a data reading request; opening any record file storing serialized data in a storage device according to the data reading request, reading the file header of the record file, filling the file header content corresponding to the file header into the corresponding file header structure, and filling the interface content length into the interface content variable of the file header structure; generating a data pool based on the filled file header structure, reading the data header in the data pool, filling the data content corresponding to the data header into a string, obtaining the data type of the data content, matching a preset data template according to the data type name, and deserializing the serialized data by using the preset data template and the string to obtain target data.

[0023] The embodiment of the third aspect of the present application provides a data recording device, including: an identification module, configured to identify the actual capacity of the target data to be recorded; a first update module, configured to write the target data into the preset cache and update the write cache index of the preset cache if the actual capacity is less than the remaining storage capacity of the preset cache; a second update module, configured to write a part of the target data into the preset cache if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache. When the preset cache is full, write the data in the preset cache into the record file of the storage device through the operation interface of the operating system, copy the remaining data of the target data to the start address of the preset cache, and update the write cache index of the preset cache. Among them, when receiving data for the first time after the record file is created, the kernel cache of the operating system is prohibited by using the unbuffered flag.

[0024] The fourth aspect of the embodiments of the present application provides a data reading device, including: an acquisition module, configured to acquire a data reading request; a filling module, configured to open, according to the data reading request, a record file storing serialized data in any record of a storage device, read a file header of the record file, fill file header content corresponding to the file header into a corresponding file header structure, and fill an interface content length into an interface content variable of the file header structure; a deserialization module, configured to generate a data pool based on the filled file header structure, read a data header in the data pool, fill data content corresponding to the data header into a string, obtain a data type of the data content, match a preset data template according to the data type name, and use the preset data template and the string to deserialize the serialized data to obtain target data.

[0025] The fifth aspect of the embodiments of the present application provides a data acquisition device, which is used for: identifying an actual capacity of target data to be recorded; if the actual capacity is less than a remaining storage capacity of a preset cache, writing the target data into the preset cache and updating a write cache index of the preset cache; if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, writing partial data of the target data into the preset cache, when the preset cache is full, writing data in the preset cache into a record file of the storage device through an operation interface of an operating system, copying the remaining data of the target data to a start address of the preset cache, and updating the write cache index of the preset cache, where, when receiving data for the first time after the record file is created, a kernel cache of the operating system is prohibited by using a bufferless flag.

[0026] The sixth aspect of the embodiments of the present application provides a data reading device, which is used for: acquiring a data reading request; opening, according to the data reading request, a record file storing serialized data in any record of a storage device, reading a file header of the record file, filling file header content corresponding to the file header into a corresponding file header structure, and filling an interface content length into an interface content variable of the file header structure; generating a data pool based on the filled file header structure, reading a data header in the data pool, filling data content corresponding to the data header into a string, obtaining a data type of the data content, matching a preset data template according to the data type name, and using the preset data template and the string to deserialize the serialized data to obtain target data.

[0027] An embodiment of the seventh aspect of the present application provides a vehicle, including: a storage device, wherein one or more record files are created in the storage device; a data acquisition device, configured to identify the actual capacity of target data to be recorded; if the actual capacity is less than the remaining storage capacity of a preset cache, write the target data into the preset cache and update the write cache index of the preset cache; if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, write a part of the target data into the preset cache, and when the preset cache is full, write the data in the preset cache into the record file of the storage device through the operation interface of the operating system, copy the remaining data of the target data to the head address of the preset cache, and update the write cache index of the preset cache, wherein when receiving data for the first time after the record file is created, the kernel cache of the operating system is prohibited by using a bufferless flag.

[0028] Optionally, it further includes: a reading device, configured to obtain a data reading request; open any record file storing serialized data in the storage device according to the data reading request, read the file header of the record file, fill the file header content corresponding to the file header into the corresponding file header structure, and fill the interface content length into the interface content variable of the file header structure; generate a data pool based on the filled file header structure, read the data header in the data pool, fill the data content corresponding to the data header into a string, obtain the data type of the data content, match a preset data template according to the data type name, and use the preset data template and the string to deserialize the serialized data to obtain target data.

[0029] An embodiment of the eighth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored, and the program is executed by a processor to implement the data recording method or the data reading method as described in the above embodiments.

[0030] Therefore, the present application has at least the following beneficial effects:

[0031] (1) The embodiment of the present application identifies the actual capacity of the target data to be recorded, and performs different processing for different actual capacities of the target data. If the actual capacity is less than the remaining storage capacity of the preset cache, the target data is written into the preset cache and the written cache index is updated; if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, a part of the data is written into the preset cache and when the preset cache is full, the cached data is written into the record file of the storage device through the operating system interface, the remaining data is copied to the head address of the cache and the write cache index of the cache is updated, which can effectively solve the problem of lag caused by system cache cleaning during high-frequency and large-volume data recording, and ensure the safety of vehicle autonomous driving.

[0032] (2) When the starting address of the preset cache in the embodiment of the present application is a multiple of the data block in the storage device and the maximum cache capacity is a multiple of the data page in the storage device, it can be directly written to the storage device without passing through the operating system cache, thus solving the occasional freezing problem of the operating system when the operating system cache is too high.

[0033] (3) In the embodiment of the present application, by judging whether the file capacity of the record file is greater than the set capacity, the next operation is carried out. If it is less than or equal to, the cache is written to the storage device and the next cycle is entered; if it is greater than the preset capacity or an abnormal message is received, the abnormal signal processing function is called to write the cache data to the storage device, and after writing, the record file is closed and a new file record is generated, and data recording is restarted under the new file. Therefore, data caching and packetization are required when recording data, so as to avoid losses caused by data loss and ensure data security.

[0034] (4) In the embodiment of the present application, it is judged whether the current cache byte number of the preset cache is an integer multiple of the data page. If it is an integer multiple, the cache data is written to the storage device. If it is not an integer multiple, the current cache byte number and a certain multiple of the data page are used to calculate the extra write byte number, and they are jointly written to the storage device and the extra byte number is deleted after writing. Since the size of the data written to the storage device must be a multiple of the memory page size, it can be directly written to the storage device without passing through the operating system cache, thus solving the occasional freezing problem of the operating system when the operating system cache is too high.

[0035] (5) In the embodiment of the present application, a data record interface is defined based on the serialized data record protocol, and the interface is called to perform serialization processing on the received target data to obtain serialized data, and it is written to the preset cache. The data can be converted into a specific format through automatic and manual recording of data, etc., so as to facilitate subsequent writing and storage.

[0036] (6) In the embodiment of the present application, all the data type names that need to be recorded are predefined through a configuration file. When the program is initialized, all the text contents of the interfaces are obtained at one time based on the configuration file, and the interface text contents are filled into the file header content part of the data, so as to facilitate reading the corresponding text contents of the interface files stored in the data file header during subsequent data playback, and processing the interface text contents and loading them into the data pool, thereby further supporting the deserialization of the serialized data subsequently and realizing the interpretability of the record file itself.

[0037] (7) In the embodiment of the present application, no configuration file is required. When recording data each time, it is judged whether the interface text content of this type of variable has been processed and searched. If not, a data type name list is created to record the variable types whose corresponding interface text content has been processed and searched, and a data header structure is defined. The protocol buffer interface is used to serialize the data, and the updated data header content is written into the record file. Before stopping the recording, a file header structure is defined, and its interface content length and file header content are arranged and inserted into the record file, so that when playing back the data subsequently, the corresponding text content of the interface file stored in the data file header can be read, and the interface text content is processed and loaded into the data pool, thereby further supporting the deserialization of the serialized data and realizing the interpretability of the record file itself.

[0038] (8) In the embodiment of the present application, the open function of the operating system can be used to open a file descriptor, and the record file can be opened using the file descriptor to create a target file. Once the target file is created, it indicates that the data has been received, which can more intuitively prompt the user.

[0039] Additional aspects and advantages of the present application will be given in part in the following description, will become apparent in part from the following description, or will be understood through the practice of the present application. Description of the Drawings

[0040] The above-mentioned and / or additional aspects and advantages of the present application will become apparent and easy to understand from the following description of the embodiments in conjunction with the drawings, where:

[0041] Figure 1 is a flowchart of a data recording method according to an embodiment of the present application;

[0042] Figure 2 is a flowchart of a data reading method according to an embodiment of the present application;

[0043] Figure 3 is an overall processing flowchart according to an embodiment of the present application;

[0044] Figure 4 is an exception handling flowchart according to an embodiment of the present application;

[0045] Figure 5 is a schematic diagram of a data structure according to an embodiment of the present application;

[0046] Figure 6 is a schematic diagram of data packet splitting according to an embodiment of the present application;

[0047] Figure 7 is a protocol schematic diagram according to an embodiment of the present application;

[0048] Figure 8A flowchart of a data recording solution according to an embodiment of the present application;

[0049] Figure 9 A flowchart of a second data recording solution according to an embodiment of the present application;

[0050] Figure 10 A flowchart of data parsing and re-injection according to an embodiment of the present application;

[0051] Figure 11 A block diagram of a data recording device according to an embodiment of the present application;

[0052] Figure 12 A block diagram of a data reading device according to an embodiment of the present application;

[0053] Figure 13 A structural diagram of a vehicle according to an embodiment of the present application. Detailed implementation manners

[0054] The embodiments of the present application will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, but should not be construed as limiting the present application.

[0055] Different methods can be adopted for data recording, and each method has its corresponding effect. The following embodiments innovate in terms of data recording and unloading of data recording devices, distributed data recording systems, and high-speed sampling methods, but none of them can effectively solve the problem of lag caused by system cache cleaning during high-frequency and large-volume data recording.

[0056] In related technology (1), a data recording and unloading method for a micro airborne data recording device. The data recording device includes a data recorder and a data transfer board. The data recorder is connected to the airborne system through a micro interface board, the data recorder is connected to the data transfer board through a micro interface board, and the data transfer board is connected to an external computer through a standard interface. It includes a data recording step and a data unloading step. The data recording step includes two sub-steps: a default working mode and an interrupt cache mode. The data unloading step includes three sub-steps: external computer control data unloading, airborne system control data unloading, and data unloading. The structure of the present application is simple and small in size. The data recording of the data recorder can be completed through the RS422 interface and the RS232 interface, and the high-speed unloading of the data recorder can be realized through the USB3.0 interface.

[0057] In the related art (2), a network data recording system includes: several remote measurement and control units, an aggregation routing device connected to the several remote measurement and control units and providing remote network access functions for them, a user application for monitoring and managing the remote measurement and control units and the aggregation routing device, as well as a network control server and a data recording server. The network control server includes a remote measurement and control unit network control server, an aggregation routing device network control server, and a user application network control server. The remote measurement and control unit data recording server, the aggregation routing device data recording server, and the user application data recording server. This application proposes a data recording system for a distributed server network based on a network management server and a data recording server, which can effectively reduce the centralized load of the server and facilitate grouped management; and proposes to use dynamic key technology to improve the data security of the system.

[0058] In the related art (3), through a data recording device, an inspection device with the data recording device, and a control device, a data recording device that reduces the write processing to the data recording unit, increases the amount of write data each time to reduce the write overhead, and uses high-speed sampling to ensure the sampling data of a large number of summary recording points is realized.

[0059] Due to the characteristics of the Linux operating system itself, data must pass through the operating system kernel cache from the user space and then reach the hard disk. In daily use, this cache can effectively improve the IO performance. However, in the scenario of autonomous driving data collection, due to the high-frequency writing of a large amount of data to the hard disk, the operating system cache will quickly rise and remain high. It is found through actual testing that when the system cache is too high, the entire system will experience occasional stuttering, seriously affecting the safety of the autonomous driving system.

[0060] Therefore, the embodiment of this application proposes a zero-copy data recording method, which can directly write to the hard disk without passing through the system buffer, effectively solving the problem of stuttering.

[0061] For the storage and reading of data, in related technology (4), a method, system, and vehicle for storing and reading multi-source heterogeneous sensor data include: Data storage: Serialize sensor data; Assign the serialized byte size, the enumeration values of the data source and data type, the current time of obtaining the data, and a predefined verification code to the corresponding variables in the data header structure; Write the data header; Write the serialized data; Data reading: Read the number of bytes of the data header size to generate a data header variable, and obtain the corresponding variables stored in the data header; Verify the verification code. If it fails, an exception is thrown. If it succeeds, read the consecutive bytes with the specified data size length in the data header; According to the data source and data type in the data header, combined with the predefined deserialization method, deserialize the read serialized data to obtain the original real sensor data. The invention provides a data recording protocol that supports efficient multi-source sensor data recording, indexing, and re-injection.

[0062] The above application provides a method for storing serialized data, which can accurately obtain the serialized data of the corresponding data according to the time sequence during data recording. However, the deserialization part needs to be combined with a predefined deserialization method, and this predefined part is not reflected in the recorded data and requires an additional file for description. Therefore, the separately recorded data cannot be self-explanatory. If it is lost due to improper management or the deserialization method or description file of the specified version cannot be corresponded, the data is equivalent to being unable to be parsed and used, resulting in an increase in the management and maintenance cost of the data and the risk of data unavailability.

[0063] Another solution to this problem is to replace serialized data with structured data. The recorded binary file contains the name and value of each field of the data. However, this solution usually leads to an increase in the CPU load when recording data and an increase in the size of the recorded file.

[0064] Therefore, the embodiment of this application proposes a self-explanatory serialized binary file recording and re-injection method, which can achieve data self-explanation while adopting an efficient serialization recording scheme.

[0065] The following describes the data recording or reading method, device, acquisition device, vehicle, and medium of the embodiment of this application with reference to the accompanying drawings. Specifically, Figure 1 It is a schematic flowchart of a data recording method provided by an embodiment of this application.

[0066] As Figure 1 shown, the data recording method includes the following steps:

[0067] In step S101, identify the actual capacity of the target data to be recorded.

[0068] Among them, the target data can be the data to be recorded, and no specific limitation is made here.

[0069] It is understandable that the embodiment of the present application identifies the actual capacity of the data to be recorded by the data recording device, so as to facilitate subsequent corresponding operations according to different capacities.

[0070] In step S102, if the actual capacity is less than the remaining storage capacity of the preset cache, the target data is written into the preset cache, and the write cache index of the preset cache is updated.

[0071] Among them, the preset cache can be raw_data_buf (cache space), which can be selected according to actual needs; and the cache size is defined as KRawDataBufferMaxSize (the maximum cache amount of the cache space), which is not specifically limited here.

[0072] Among them, the cache index can record the position of each data in the cache and provide a pointer pointing to the data value stored in the specified column of the table according to the index to find the relevant data record; and the cache index rawdata_buf_cur_size (the size of the unit being operated in the matrix buffer) written during initialization = 0, which is not specifically limited here.

[0073] It is understandable that in the embodiment of the present application, when the actual capacity of the target data to be recorded is less than the remaining storage capacity of the preset cache, the target data is written into the cache, and the written cache index is updated, so as to quickly locate the relevant data position when obtaining data subsequently.

[0074] In step S103, if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, a part of the target data is written into the preset cache. When the preset cache is full, the data in the preset cache is written into the record file of the storage device through the operation interface of the operating system, the remaining data of the target data is copied to the starting address of the preset cache, and the write cache index of the preset cache is updated. Among them, when receiving data for the first time after the record file is created, the kernel cache of the operating system is prohibited by using the unbuffered flag.

[0075] Among them, the starting address is also called the initial address, and the address of the first byte in the storage area is used as the starting address of the storage area, which is not specifically limited here.

[0076] Among them, the starting address of the preset cache is a multiple of the data block in the storage device, and the maximum cache capacity is a multiple of the data page in the storage device.

[0077] Among them, unbuffered can be that the standard I / O (Input / Output) does not buffer characters, which is not specifically limited here.

[0078] It can be understood that when the actual capacity of the target data to be recorded is greater than or equal to the remaining storage capacity of the preset cache in the embodiments of the present application, part of the data is written into the preset cache. When the preset cache is full, the cached data is written into the record file of the storage device through the operating system interface, and the remaining data is copied to the cache start address and the write cache index of the cache is updated, which can effectively solve the lag problem caused by system cache cleaning during high-frequency and large-volume data recording and ensure the safety of vehicle autonomous driving.

[0079] In the embodiments of the present application, before identifying the remaining storage capacity of the predefined preset cache, it further includes: defining a data recording interface based on the serialized data recording protocol; calling the data recording interface to perform serialization processing on the received target data to obtain serialized data, and writing the serialized data into the preset cache.

[0080] Among them, the serialized data recording protocol includes a file header structure, file header content, data header structure, and data header content, which are not specifically limited herein.

[0081] It can be understood that in the embodiments of the present application, before identifying the remaining storage capacity of the preset cache, it is necessary to define a data recording interface based on the serialized data recording protocol, call the interface to perform serialization processing on the received data, and then write it into the preset cache. The data can be converted into a specific format through automatic and manual recording of data, etc., for subsequent writing and storage.

[0082] It should be noted that in the embodiments of the present application, defining the data recording interface based on the serialized data recording protocol can not only pre-define all the data type names to be recorded through a configuration file, and obtain the text content of all interfaces at one time based on the configuration file during program initialization; it can also, without a configuration file, determine whether the interface text content of this type of variable has been processed and searched each time data is recorded, and perform the next operation, which is not specifically limited herein.

[0083] As a possible implementation manner, defining the data recording interface based on the serialized data recording protocol includes: reading the configuration file to obtain a list of data type names; traversing the list of data type names to obtain a message descriptor; if the message descriptor is empty, it is determined that the record information is incorrect, otherwise, obtain the dependent file and perform a recursive operation on the dependent file until the dependent file is cleared; initialize the file header structure, write the interface content length and file header content of the file header structure into the record file, and when recording each frame of data, define the data header structure, perform serialization processing on the data using the protocol buffer interface, and write the updated data header content into the record file.

[0084] Among them, the configuration file can be some computer programs or system configuration parameters and initialization settings, and different operations can be performed according to different situations, which are not specifically limited herein.

[0085] Among them, the message descriptor can describe the characteristics of the message, such as the priority, life cycle, name, etc. of the message, and specific descriptions are made according to the actual situation, which are not specifically limited herein.

[0086] Among them, the dependent file can be that each interface file depends on other interface files, which are not specifically limited herein.

[0087] Among them, the recursive operation can define an infinite set of objects with a finite number of statements, transform large and complex problems into small-scale problems similar to the original problems step by step to solve, and perform repeated calculations multiple times, reducing the workload, which are not specifically limited herein.

[0088] Among them, the protocol buffer provides a language-neutral, platform-neutral, and extensible mechanism for serializing structured data in a forward-compatible and backward-compatible manner, which are not specifically limited herein.

[0089] It can be understood that in the embodiments of the present application, all the data type names that need to be recorded are predefined through the configuration file. When the program is initialized, the text content of all interfaces is obtained at one time based on the configuration file, and the interface text content is filled into the file header content part of the data, so that when the data is played back subsequently, the corresponding text content of the interface file stored in the data file header can be read, and the interface text content is processed and loaded into the data pool, thereby further supporting the deserialization of the serialized data subsequently and realizing the interpretability of the record file itself.

[0090] As another possible implementation manner, a data recording interface is defined based on the serialized data recording protocol, including: creating a list of data type names; defining a data header structure, serializing the data using the protocol buffer interface, and writing the updated data header content into the record file; before stopping recording data, defining a file header structure, arranging the interface content length and the file header content of the file header structure continuously, and inserting them into the record file.

[0091] It can be understood that in the embodiments of the present application, no configuration file is required. When recording data each time, it is determined whether the interface text content of the variable of this type has been processed and searched. If not, a data type name list is created to record the variable types whose corresponding interface text content has been processed and searched, and a data header structure is defined. The protocol buffer interface is used to serialize the data, and the updated data header content is written into the record file. Before stopping the recording, a file header structure is defined, and its interface content length and file header content are arranged and inserted into the record file, so that when playing back the data subsequently, the corresponding text content of the interface file stored in the data file header can be read, and the interface text content is processed and loaded into the data pool, thereby further supporting the deserialization of the serialized data and realizing the interpretability of the record file itself.

[0092] In the embodiments of the present application, writing the preset cached data into the record file of the storage device through the operation interface of the operating system includes: identifying whether the file capacity of the record file is greater than the preset capacity; if the file capacity is less than or equal to the preset capacity, writing the preset cache into the storage device and entering the next writing cycle; if the file capacity is greater than the preset capacity, or an exception signal is received, calling the exception signal processing function to write the data in the preset cache into the storage device, closing the current record file after writing, and generating a new record file, and starting to record data again under the new record file.

[0093] Among them, the preset capacity can be the capacity set by the user and can be specifically set according to the actual situation, and no specific limitation is made here.

[0094] Among them, the exception signal processing function can be Fulsh And Close File (flush and close the file), and no specific limitation is made here.

[0095] It can be understood that in the embodiments of the present application, the next operation is determined by identifying whether the file capacity of the record file is greater than the set capacity. If it is less than or equal to the set capacity, the preset cache is written into the storage device and the next writing cycle is entered; if it is greater than the set capacity or an exception signal is received, the exception signal processing function is called to write the cached data into the storage device, and the record file is closed and a new file record is generated after writing, and data recording starts again under the new file. Therefore, data caching and packet splitting are required when recording data, so as to avoid losses caused by data loss and ensure data security.

[0096] In an embodiment of the present application, calling an exception signal processing function to write the preset cached data into a storage device includes: determining whether the current cached byte count of the preset cache is an integer multiple of a data page in the storage device; if the current cached byte count is an integer multiple of the data page, writing the data in the preset cache into the storage device; if the current cached byte count is not an integer multiple of the data page, calculating the extra write byte count according to the current cached byte count and a preset multiple of the data page, writing the extra write byte count and the current cached byte count into the storage device together, and deleting the extra write byte count after writing.

[0097] Among them, the data page includes a file header, a page header, maximum and minimum records, user records, free space, a page directory, and a file tail, which are not specifically limited herein.

[0098] Among them, the preset multiple can be a multiple set by the user and can be set according to actual situations, which is not specifically limited herein.

[0099] It can be understood that in the embodiment of the present application, it is determined whether the current cached byte count of the preset cache is an integer multiple of the data page. If it is an integer multiple, the cached data is written into the storage device. If it is not an integer multiple, the extra write byte count is calculated according to a certain multiple of the current cached byte count and the data page, and they are written into the storage device together and the extra bytes are deleted after writing. Since the size of the data written into the storage device must be a multiple of the memory page size, it can be written directly into the storage device without passing through the operating system cache, thereby solving the occasional lag problem that occurs in the operating system when the operating system cache is too high.

[0100] In an embodiment of the present application, while prohibiting the kernel cache of the operating system by using a bufferless flag, it further includes: opening a file descriptor by using the open function of the operating system; opening a record file by using the file descriptor to create a target file.

[0101] It can be understood that in the embodiment of the present application, the open function of the operating system can be used to open a file descriptor, and the record file can be opened by using the file descriptor to create a target file. Once the target file is created, it indicates that the data has been received, which can more intuitively prompt the user.

[0102] According to the data recording method provided by the embodiments of the present application, the actual capacity of the target data to be recorded is identified, and different processing is performed for different actual capacities of the target data. When the actual capacity is less than the remaining storage capacity of the preset cache, the target data is written into the preset cache, and the written cache index is updated; when the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, part of the data is written into the preset cache, and when the preset cache is full, the cached data is written into the record file of the storage device through the operating system interface, the remaining data is copied to the cache start address, and the write cache index of the cache is updated, which can effectively solve the problem of system cache cleaning causing lag during high-frequency and large-volume data recording and ensure the safety of vehicle autonomous driving.

[0103] Next, the data reading method provided by the embodiments of the present application will be described with reference to the accompanying drawings.

[0104] Figure 2 It is a flowchart of the data reading method provided by the embodiments of the present application.

[0105] As Figure 2 shown, the data reading method includes the following steps:

[0106] In step S201, a data reading request is obtained.

[0107] It can be understood that in the embodiments of the present application, by obtaining the data reading request, it is convenient to open the record file according to the reading request subsequently.

[0108] In step S202, according to the data reading request, a record file storing serialized data in the storage device is opened, the file header of the record file is read, the file header content corresponding to the file header is filled into the corresponding file header structure, and the interface content length is filled into the interface content variable of the file header structure.

[0109] It can be understood that in the embodiments of the present application, according to the data reading request, a record file storing serialized data in the storage device is opened, its file header is read, the file header content corresponding to the file header is filled into the corresponding file header structure, and the interface content length is filled into the interface content variable of the file header structure, so as to generate a data pool based on the filled file header structure subsequently.

[0110] In step S203, a data pool is generated based on the filled file header structure, the data header in the data pool is read, the data content corresponding to the data header is filled into a string, the data type of the data content is obtained, a preset data template is matched according to the data type name, and the serialized data is deserialized by using the preset data template and the string to obtain the target data.

[0111] Among them, the data pool can be a user-defined data pool, which can be set according to the actual needs of the user and will not be specifically limited here.

[0112] Among them, the preset data template can be a data template set by the user and can be selected according to the actual needs of the user, and no specific limitation is made here.

[0113] It can be understood that the embodiment of the present application generates a data pool based on the filled file header structure, reads the data header therein, fills the corresponding data content into a string, and deserializes the serialized data by using the set data template and the string to obtain the target data, realizing the interpretability of the record file itself. And because the serialized binary data is recorded, and the acquisition and loading of the interface are both performed at runtime, no matter any interface is newly added or changed later, the programs for data recording and playback do not need to be modified at all.

[0114] According to the data reading method proposed by the embodiment of the present application, by obtaining a data reading request, opening any record file storing serialized data in a storage device according to the data reading request, and reading its file header, filling the file header content corresponding to the file header into the corresponding file header structure, and filling the interface content length into the interface content variable of the file header structure, generating a data pool based on the filled file header structure, reading the data header therein, filling the corresponding data content into a string, and deserializing the serialized data by using the set data template and the string to obtain the target data, realizing the interpretability of the record file itself. And because the serialized binary data is recorded, and the acquisition and loading of the interface are both performed at runtime, no matter any interface is newly added or changed later, the programs for data recording and playback do not need to be modified at all.

[0115] Next, Figures 3 to 10 The data recording or reading method proposed by the present application will be elaborated in detail, mainly divided into a data recording part and a data reinjection part, specifically as follows:

[0116] 1. Data recording part

[0117] Step 1, initialization. Define a data writing cache raw_data_buf (cache space), define the cache size as KRawDataBufferMaxSize (the maximum cache volume of the cache space), and the current writing cache index raw_data_buf_cur_size (the size of the unit being operated on in the matrix buffer) = 0. In order to write directly to the hard disk without passing through the operating system cache, two points must be ensured here: one is to ensure that the operating system allocates the starting address of raw_data_buf (cache space) as a multiple of the hard disk block size, and the other is to ensure that KRawDataBufferMaxSize (the maximum cache volume of the cache space) is a multiple of the memory page size. When recording data, first save the data in the writing cache raw_data_buf (cache space).

[0118] Step 2, Initialization: Define a variable file_name (variable file name), and complete the initialization using the current system time. Use the system time at the time of data recording as the file name, such as 2022_06_01-12_31_23.dat, where the.dat file is the data file; Define a variable max_file_size (maximum size of the file) for data packet splitting. Each time data is written to the hard disk, check whether the size of the recorded binary file exceeds max_file_size (maximum size of the file). If it exceeds, close the current recording file and create a new recording file with the current system time as the file name.

[0119] Step 3, Initialization: Create a data receiving socket and complete the initialization connection.

[0120] Step 4, Initialization: Create a working thread and run it periodically.

[0121] Step 5, Initialization: Bind an exception signal handling function FulshAndCloseFile (flush and close the file). When the program receives an exception interruption signal, perform the operation of safely writing the cache to the hard disk and close the file descriptor. As Figure 4 shown, to improve the recording efficiency, when data is received, it will first be saved in the cache raw_data_buf in memory, and the hard disk write instruction will only be executed when the cache capacity is greater than KRawDataBufferMaxSize. Since the zero-copy mechanism requires that the size of the data written to the hard disk each time is a multiple of the memory page, and we cannot require that all the received data be a multiple of the memory page size, so the received data must be split. In this way, in most cases, the data written to the hard disk each time is finally incomplete, and there are still some bytes belonging to a certain packet of data in the memory cache that have not been written to the hard disk; so without an exception signal handling function, the data in the cache will be lost, further resulting in the recorded data being incomplete and unusable.

[0122] Step 6, Receive data: When receiving data for the first time, use the open function to open the file descriptor fd in combination with the O_DIRECT (unbuffered input and output) parameter, indicating to the operating system kernel that direct I / O operations are adopted and no caching is required.

[0123] Opening the file when receiving data for the first time is to avoid creating an empty.dat file when no data is received; in this way, once the file is created, it clearly indicates that data has been received, which can more intuitively prompt the user; at the same time, since the initialization of file_name has been completed in Step 2, when recording data, the name of the file is still named after the time when the data recording module is started.

[0124] Step 7: Format the received data and add necessary auxiliary information. Each frame of data is defined as rdata, and the data length is defined as len. The necessary auxiliary information here can be understood as the data header, which can further describe information such as the source, name, batch, and length of the data. As shown in Figure 5 the following. The data header is a user-defined structure DataHead with a fixed size and continuous memory.

[0125] Step 8: Perform the following steps on the data header DataHead and the data rdata respectively.

[0126] Step 9: As shown in Figure 6 the following, determine whether there is enough remaining space in the current write buffer to store all the received data, rawdata_buf_cur_size + len <= KRawDataBufferMaxSize. If True, copy the data to the data write buffer raw_data_buf and update rawdata_buf_cur_size += len; if False, calculate the size of the bytes that can be copied copy_len = KRawDataBufferMaxSize - rawdata_buf_sur_size, and the remaining byte size eft_len = len - copy_len. Copy the bytes that can be copied to raw_data_buf, write the entire raw_data_buf to the hard disk, copy the remaining bytes to the starting position of raw_data_buf, and update rawdata_buf_cur_size = left_len.

[0127] Step 10: Determine whether the current file size exceeds max_file_size. If not, enter the next cycle and start executing from Step 6 to receive data. If so, execute the following steps.

[0128] Step 11: Calculate the quotient and remainder of the division of the current buffer byte count by the hard disk page size, and determine whether the remainder is 0. If True, write the data in the current raw_data_buf to the hard disk; if False, calculate the size of the bytes to be written write_byte = (quotient + 1) * hard disk page size, calculate the size of the extra bytes written more_byte = write_byte - rawdata_buf_cur_size, write the size of the bytes to be written in raw_data_buf to the hard disk, and then delete the extra more_byte size bytes from the file and close the current file descriptor. Adding the extra bytes to the hard disk and then deleting the extra bytes is also due to the reason that the amount of data written to the hard disk must be a multiple of the memory page size.

[0129] Step 12, update file_name and open a new file

[0130] Step 13, start executing from receiving data periodically in Step 6.

[0131] Step 14, the execution scheme of the FulshAndCloseFile exception handling function is the same as that in Step 10.

[0132] In the above description, the data recording module is regarded as an independent program, continuously obtains data through the communication interface, and records the received data.

[0133] And this recording method still supports the user to call relevant interfaces to achieve manual recording of data; the implementation steps are as follows:

[0134] Step 15, execute the initialization step, which is the same as Steps 1 and 2.

[0135] Step 16, fill the data header DataHead, complete the serialization of the recorded data, call the data recording interface, and pass in the data header and serialized data.

[0136] Step 17, repeat Step 15 as needed. Inside Step 15, the execution process is the same as Steps 8 to 11.

[0137] Step 18, when stopping recording data, call the FulshAndCloseFile function to write the data in the memory buffer to the hard disk and close the file descriptor.

[0138] 2. Data reinjection part

[0139] The data interfaces recorded by this method must be defined using protobuf. And there are two specific implementation methods: One is to pre-define all the data type names to be recorded through a configuration file. When the program is initialized, obtain the text content of all interfaces at once based on the configuration file; the other is without a configuration file. When recording data each time, judge whether the interface text content of this type of variable has been processed and searched. If it has been processed, continue to the next step. If not, obtain the interface text content of the corresponding variable type from the proto data pool.

[0140] The first solution is as Figure 8 shown:

[0141] Step 1, read the configuration file and obtain the list of all data type names to be recorded, topic Lists.

[0142] Step 2: Traverse the list topic Lists. For each data type name, call the google::protobuf::DescriptorPool::generated_pool()->FindMessageTypeByName() interface to obtain the message descriptor msg_descriptor. If msg_descriptor is empty, record the error information indicating that the message type symbol was not loaded during the compilation of this program. If msg_descriptor is not empty, call the text content content through msg_descriptor->file()->DebugString() to obtain and add it to the variable protoContents for recording.

[0143] Since each interface file may depend on other interface files, call msg_descriptor->file()->dependency_count() to obtain the number of file dependencies n. Establish a loop based on the dependency file data n, and file_descriptor = msg_descriptor->file()->dependency(n). Recursively execute the above steps in the loop body until the dependency data is 0. Thus, the text content of the interface definitions where all the data variable types to be recorded are located and the text content of the interface definitions they depend on are completed.

[0144] Step 3: Initialize the file header structure FileHead. FileHead is a predefined structure with 1-byte alignment and continuity, and it contains at least the version number version and the length of the proto interface content proto Content Size. Other fields can be dynamically extended according to actual needs, and update the proto Content Size variable to the length of proto Contents.

[0145] Step 4: Create a file descriptor and write the File Head content to the record file.

[0146] Step 5: Write proto Contents to the record file.

[0147] Step 6: Thus, the writing of the file header part is completed.

[0148] Step 7, when each frame of data needs to be recorded, first define the data header structure Raw Data Head. This structure is a self - defined structure aligned to 1 byte, and it contains at least an array of type char to represent the message name, a variable size representing the data length, and a time stamp representing the data timestamp. Other fields can be extended according to actual needs.

[0149] Step 8, based on the recorded message, use the Serialize To String interface of proto buf to serialize the data, update the content of Raw Data Head and write it to the file, and then write the serialized data to the file.

[0150] Step 9, repeat Step 7 to Step 8 to continuously record each frame of data.

[0151] Step 10, close the file descriptor, and the data recording is completed.

[0152] The second method is as Figure 9 shown:

[0153] Step 1, initialize, create a list of searched data type names handled Type List, which is used to record the variable types of the interface corresponding text content that has been processed and searched.

[0154] Step 2, when each frame of data needs to be recorded, first define the data header structure Raw Data Head. Based on the recorded message, use the Serialize To String interface of proto buf to serialize the data, update the content of Raw DataHead and write it to the file, and then write the serialized data to the file.

[0155] Step 3, search through the handled Type List by the data type name. If found, skip it. If not found, it means that this data type is recorded for the first time, and the interface text needs to be obtained according to the following method.

[0156] Among them, the following interface is called to obtain the message descriptor `msg_descriptor`, `google::protobuf::DescriptorPool::generated_pool()->FindMessageTypeByName()`. If `msg_descriptor` is empty, an error message is recorded, indicating that the message type symbol is not loaded during the compilation of this program. If `msg_descriptor` is not empty, the content of the interface call text `content` is obtained through `msg_descriptor->file()->DebugString()` and added to the variable `protoContents` for recording. Since each interface file may depend on other interface files, `msg_descriptor->file()->dependency_count()` is called to obtain the number of file dependencies `n`. A loop is established based on the dependency file data `n`, and `file_descriptor = msg_descriptor->file()->dependency(n)`. The above steps are recursively executed in the loop body until the dependency data is 0.

[0157] Step 4, repeat Step 2 to Step 3 to record all the data that needs to be recorded.

[0158] Step 5, before stopping data recording, define the file header structure `FileHead`, arrange the content of `FileHead` and the content of `protoContents` continuously, and insert them at the beginning of the record file.

[0159] Step 6, close the file descriptor, and the data recording is completed.

[0160] The specific implementation of the data read-back injection part is as Figure 10 shown as follows:

[0161] Step 1, open the binary record file

[0162] Step 2, read the byte stream of the size of the `FileHead` structure and fill it into the `FileHead` structure, and read the size of the `protoContentSize` variable

[0163] Step 3, read the byte stream of the size of `protoContentSize` and fill it into the `protoContents` variable.

[0164] Step 4: Write the content in protoContents to the interface.proto file according to the text encoding method, and create a google::protobuf::compiler::DiskSourceTree object sourceTree. Load the interface.proto file through the MapPath interface of sourceTree. Then load sourceTree through google::protobuf::compiloer::Importer to complete the creation of the interface description file.

[0165] Step 5: Read the size length bytes of the RawDataHead structure and fill them into the RawDataHead structure to obtain the data length size and the message type name.

[0166] Step 6: Read a byte stream of size and fill it into the string string.

[0167] Step 7: Mainly obtain the message file descriptor msg_descriptor through the following interface: google::protobuf::DescriptorPool::generated_pool()->FindMessageTypeByName().

[0168] Among them, if the msg_descriptor message file descriptor is empty, record the error information; if the msg_descriptor message file descriptor is not empty, the parent class pointer prototype of the protobuf variable structure corresponding to the message name can be obtained through google::protobuf::MessageFactory::generated_factory()->GetPrototype(msg_descriptor).

[0169] Step 8: Create an empty message copy msg through the new method of the parent class pointer protoypte.

[0170] Step 9: Complete the deserialization of the message through msg->ParseFromString(string).

[0171] Step 10: Through the above steps, the reading and deserialization of the binary serialized data can be completed.

[0172] In summary, the embodiment of the present application proposes a data recording method applicable to high-frequency and large amounts of data, which can directly write data to the hard disk from the user space without passing through the operating system kernel cache under the Linux operating system, thereby solving the occasional freezing problem of the operating system when the operating system cache is too high; at the same time, based on the serialization, deserialization, and reflection functions provided by protobuf, a self-explanatory serialized data recording protocol ( Figure 7 ) is implemented, and corresponding data recording and re-injection methods are proposed.

[0173] Next, a data recording device according to an embodiment of the present application is described with reference to the accompanying drawings.

[0174] Figure 11 It is a block diagram of the data recording device according to the embodiment of the present application.

[0175] As Figure 11 shown, the data recording device 10 includes: an identification module 110, a first update module 120, and a second update module 130.

[0176] Among them, the identification module 110 is used to identify the actual capacity of the target data to be recorded; the first update module 120 is used to write the target data into the preset cache and update the write cache index of the preset cache if the actual capacity is less than the remaining storage capacity of the preset cache; the second update module 130 is used to write a part of the target data into the preset cache if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, and when the preset cache is full, write the data in the preset cache into the record file of the storage device through the operation interface of the operating system, copy the remaining data of the target data to the head address of the preset cache, and update the write cache index of the preset cache, where when receiving data for the first time after the record file is created, the kernel cache of the operating system is prohibited by using the unbuffered flag.

[0177] It should be noted that the foregoing explanation of the data recording method embodiment also applies to the data recording device of this embodiment, and will not be elaborated here.

[0178] The data recording device according to the embodiment of the present application identifies the actual capacity of the target data to be recorded, performs different processing for different actual capacities of the target data. If the actual capacity is less than the remaining storage capacity of the preset cache, the target data is written into the preset cache and the written cache index is updated; if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, part of the data is written into the preset cache and when the preset cache is full, the cached data is written into the record file of the storage device through the operating system interface, the remaining data is copied to the cache head address and the write cache index of the cache is updated, which can effectively solve the freezing problem caused by system cache cleaning during high-frequency large data recording and ensure the safety of vehicle autonomous driving.

[0179] Next, a data reading device according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0180] Figure 12 It is a block diagram of the data reading device according to the embodiment of the present application.

[0181] As Figure 12 shown, the data reading device 20 includes: an acquisition module 210, a filling module 220, and a deserialization module 230.

[0182] Among them, the acquisition module 210 is used to acquire a data reading request; the filling module 220 is used to open a record file storing serialized data in any record in the storage device according to the data reading request, read the file header of the record file, fill the file header content corresponding to the file header into the corresponding file header structure, and fill the interface content length into the interface content variable of the file header structure; the deserialization module 230 is used to generate a data pool based on the filled file header structure, read the data header in the data pool, fill the data content corresponding to the data header into a string, obtain the data type of the data content, match a preset data template according to the data type name, and use the preset data template and the string to deserialize the serialized data to obtain the target data.

[0183] It should be noted that the foregoing explanation of the embodiment of the data reading method also applies to the data reading device of this embodiment, and will not be elaborated here.

[0184] The data reading device according to the embodiment of the present application acquires a data reading request, opens a record file storing serialized data in any record in the storage device according to the data reading request, reads its file header, fills the file header content corresponding to the file header into the corresponding file header structure, fills the interface content length into the interface content variable of the file header structure, generates a data pool based on the filled file header structure, reads the data header therein, fills the corresponding data content into a string, and uses the set data template and the string to deserialize the serialized data to obtain the target data, realizing the interpretability of the record file itself. Moreover, since serialized binary data is recorded, and the acquisition and loading of the interface are both performed at runtime, no matter any interface is newly added or changed later, the programs for data recording and playback do not need to be modified at all.

[0185] The embodiment of the present application further provides a data acquisition device, which is used for: identifying the actual capacity of the target data to be recorded; if the actual capacity is less than the remaining storage capacity of the preset cache, writing the target data into the preset cache and updating the write cache index of the preset cache; if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, writing a part of the target data into the preset cache, and when the preset cache is full, writing the data in the preset cache into the record file of the storage device through the operation interface of the operating system, copying the remaining data of the target data to the head address of the preset cache, and updating the write cache index of the preset cache, wherein when receiving data for the first time after the record file is created, the kernel cache of the operating system is prohibited by using the unbuffered flag.

[0186] The embodiment of the present application further provides a data reading device, which is used for: obtaining a data reading request; opening, according to the data reading request, any record file in the storage device that stores serialized data, reading the file header of the record file, filling the file header content corresponding to the file header into the corresponding file header structure, and filling the interface content length into the interface content variable of the file header structure; generating a data pool based on the filled file header structure, reading the data header in the data pool, filling the data content corresponding to the data header into a string, obtaining the data type of the data content, matching a preset data template according to the data type name, and deserializing the serialized data by using the preset data template and the string to obtain the target data.

[0187] Figure 13 It is a schematic structural diagram of a vehicle provided by an embodiment of the present application. The vehicle 30 may include: a storage device 310 and a data acquisition device 320.

[0188] Among them, one or more record files are created in the storage device 310; the data acquisition device 320 is used for identifying the actual capacity of the target data to be recorded; if the actual capacity is less than the remaining storage capacity of the preset cache, writing the target data into the preset cache and updating the write cache index of the preset cache; if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, writing a part of the target data into the preset cache, and when the preset cache is full, writing the data in the preset cache into the record file of the storage device through the operation interface of the operating system, copying the remaining data of the target data to the head address of the preset cache, and updating the write cache index of the preset cache, wherein when receiving data for the first time after the record file is created, the kernel cache of the operating system is prohibited by using the unbuffered flag.

[0189] In the embodiment of the present application, the vehicle 30 further includes: a reading device.

[0190] Among them, the reading device is used to obtain a data reading request; open any record file storing serialized data in the storage device according to the data reading request, read the file header of the record file, fill the file header content corresponding to the file header into the corresponding file header structure, and fill the interface content length into the interface content variable of the file header structure; generate a data pool based on the filled file header structure, read the data header in the data pool, fill the data content corresponding to the data header into a string, obtain the data type of the data content, match a preset data template according to the data type name, and use the preset data template and the string to deserialize the serialized data to obtain the target data.

[0191] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the above data recording method or data reading method is implemented.

[0192] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. mean that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0193] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0194] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment or part of code including one or more executable instructions for implementing a customized logic function or process. The scope of the preferred embodiments of the present application includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in a reverse order according to the involved functions, rather than in the order shown or discussed, which should be understood by those skilled in the art of the embodiments of the present application.

[0195] It should be understood that each part of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays, field programmable gate arrays, and the like.

[0196] Those of ordinary skill in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0197] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A data recording method, characterized in that, It includes the following steps: Identify the actual capacity of the target data to be recorded; If the actual capacity is less than the remaining storage capacity of the preset cache, write the target data into the preset cache and update the write cache index of the preset cache; If the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, write a part of the target data into the preset cache. When the preset cache is full, write the data in the preset cache into the record file of the storage device through the operation interface of the operating system, copy the remaining data of the target data to the head address of the preset cache, and update the write cache index of the preset cache. Among them, when receiving data for the first time after the record file is created, the kernel cache of the operating system is prohibited using the unbuffered flag; The head address of the preset cache is a multiple of the data block in the storage device, and the maximum capacity of the preset cache is a multiple of the data page in the storage device.

2. The method according to claim 1, characterized in that, The writing of the data in the preset cache into the record file of the storage device through the operation interface of the operating system includes: Identify whether the file capacity of the record file is greater than the preset capacity; If the file capacity is less than or equal to the preset capacity, write the preset cache into the storage device and enter the next writing cycle; If the file capacity is greater than the preset capacity, or an exception signal is received, call the exception signal processing function to write the data in the preset cache into the storage device, close the current record file after writing, and generate a new record file, and start recording data again under the new record file.

3. The method according to claim 2, wherein The calling of the exception signal processing function to write the data in the preset cache into the storage device includes: Judge whether the current cache byte number of the preset cache is an integer multiple of the data page in the storage device; If the current cache byte number is an integer multiple of the data page, write the data in the preset cache into the storage device; If the current cache byte number is a non-integer multiple of the data page, calculate the extra writing byte number according to the current cache byte number and the preset multiple of the data page, write the extra writing byte number and the current cache byte number into the storage device together, and delete the extra writing byte number after writing.

4. The method according to claim 1, characterized in that, Before identifying the remaining storage capacity of the predefined preset cache, it also includes: Define a data recording interface based on the serialized data recording protocol; Call the data recording interface to serialize the received target data to obtain serialized data, and write the serialized data into the preset cache.

5. The method according to claim 4, wherein The serialized data recording protocol includes a file header structure, file header content, data header structure and data header content. Defining the data recording interface based on the serialized data recording protocol includes: Read the configuration file to obtain the data type name list; Traverse the data type name list to obtain the message descriptor; If the message descriptor is empty, determine that the record information is incorrect, otherwise obtain the dependent file and perform a recursive operation on the dependent file until the dependent file is cleared; Initialize the file header structure, write the interface content length of the file header structure and the file header content into a record file, and when recording each frame of data, define a data header structure, perform serialization processing on the data using the protocol buffer interface, and write the updated data header content into the record file.

6. The method according to claim 4, characterized in that The serialized data recording protocol includes a file header structure, file header content, data header structure, and data header content. Defining a data recording interface based on the serialized data recording protocol includes: Create a list of data type names; Define a data header structure, perform serialization processing on the data using the protocol buffer interface, and write the updated data header content into the record file; Before stopping data recording, define the file header structure, concatenate the interface content length of the file header structure and the file header content, and insert them into the record file.

7. The method according to claim 1, wherein While disabling the kernel cache of the operating system using the unbuffered flag, it also includes: Open a file descriptor using the open function of the operating system; Open the record file using the file descriptor to create a target file.

8. A data reading method, characterized in that, Include the following steps: Obtain a data read request; Open a record file storing serialized data in the storage device according to the data read request, read the file header of the record file, fill the file header content corresponding to the file header into the corresponding file header structure, and fill the interface content length into the interface content variable of the file header structure; Generate a data pool based on the filled file header structure, read the data header in the data pool, fill the data content corresponding to the data header into a string, obtain the data type of the data content, match a preset data template according to the data type name, and use the preset data template and the string to deserialize the serialized data to obtain the target data.

9. A data recording device, characterized in that, Include: An identification module for identifying the actual capacity of the target data to be recorded; A first update module for writing the target data into the preset cache and updating the write cache index of the preset cache if the actual capacity is less than the remaining storage capacity of the preset cache; A second update module for writing a partial data of the target data into the preset cache if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache. When the preset cache is full, write the data in the preset cache into the record file of the storage device through the operation interface of the operating system, copy the remaining data of the target data to the start address of the preset cache, and update the write cache index of the preset cache. Among them, when receiving data for the first time after the record file is created, disable the kernel cache of the operating system using the unbuffered flag; The start address of the preset cache is a multiple of the data block in the storage device, and the maximum capacity of the preset cache is a multiple of the data page in the storage device.

10. A data reading device, characterized in that, Include: An acquisition module for obtaining a data read request; A filling module, which is used to open a record file storing serialized data in any record of a storage device according to the data reading request, read the file header of the record file, fill the file header content corresponding to the file header into the corresponding file header structure, and fill the interface content length into the interface content variable of the file header structure; A deserialization module, which is used to generate a data pool based on the filled file header structure, read the data header in the data pool, fill the data content corresponding to the data header into a string, obtain the data type of the data content, match a preset data template according to the data type name, and use the preset data template and the string to deserialize the serialized data to obtain target data.

11. A data acquisition device, characterized in that, The data acquisition device is used for: Identifying the actual capacity of the target data to be recorded; If the actual capacity is less than the remaining storage capacity of the preset cache, writing the target data into the preset cache and updating the write cache index of the preset cache; If the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, writing a part of the target data into the preset cache. When the preset cache is full, writing the data in the preset cache into the record file of the storage device through the operation interface of the operating system, copying the remaining data of the target data to the start address of the preset cache, and updating the write cache index of the preset cache. Among them, when receiving data for the first time after the record file is created, the kernel cache of the operating system is prohibited by using the bufferless flag; The start address of the preset cache is a multiple of the data block in the storage device, and the maximum capacity of the preset cache is a multiple of the data page in the storage device.

12. A data reading device, characterized in that, The data reading device is used for: Obtaining a data reading request; According to the data reading request, opening a record file storing serialized data in any record of the storage device, reading the file header of the record file, filling the file header content corresponding to the file header into the corresponding file header structure, and filling the interface content length into the interface content variable of the file header structure; Generating a data pool based on the filled file header structure, reading the data header in the data pool, filling the data content corresponding to the data header into a string, obtaining the data type of the data content, matching a preset data template according to the data type name, and using the preset data template and the string to deserialize the serialized data to obtain target data.

13. A vehicle, characterized in that, Including: A storage device, wherein one or more record files are created in the storage device; A data acquisition device for identifying the actual capacity of target data to be recorded; if the actual capacity is less than the remaining storage capacity of a preset cache, the target data is written into the preset cache, and the write cache index of the preset cache is updated; if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, a partial data of the target data is written into the preset cache. When the preset cache is full, the data in the preset cache is written into a record file of the storage device through an operation interface of the operating system, the remaining data of the target data is copied to the head address of the preset cache, and the write cache index of the preset cache is updated, wherein when receiving data for the first time after the record file is created, the kernel cache of the operating system is prohibited by using a buffer-free flag; The head address of the preset cache is a multiple of a data block in the storage device, and the maximum capacity of the preset cache is a multiple of a data page in the storage device.

14. The vehicle according to claim 13, wherein It further includes: A reading device for obtaining a data reading request; Opening a record file that stores serialized data in the storage device according to the data reading request, reading a file header of the record file, filling file header content corresponding to the file header into a corresponding file header structure, and filling an interface content length into an interface content variable of the file header structure; Generating a data pool based on the filled file header structure, reading a data header in the data pool, filling data content corresponding to the data header into a string, obtaining a data type of the data content, matching a preset data template according to the data type name, and deserializing the serialized data by using the preset data template and the string to obtain target data.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, This program is executed by a processor to be used for implementing the data recording method according to any one of claims 1-7, or the data reading method according to claim 8.

Images