A vehicle data processing method and device, a vehicle, and an electronic device

Abstract

The application relates to a vehicle data processing method and device, a vehicle and an electronic device. The method comprises the following steps: acquiring a data file of a vehicle; reading all binary files in the data file in parallel through multiple reading threads to obtain data of each binary file and write the data into a memory; distributing the data of each binary file to a parsing thread pool; parsing the corresponding data by the parsing thread pool to obtain multiple structured data; performing time alignment on the multiple structured data to obtain combined frame data; and serializing the combined frame data into a playable file. According to the application, the data is read through multiple threads, and the data is parsed in parallel through multiple parsing thread pools, so that different types of data are uniformly processed, the efficiency of obtaining the playable file is improved, and thus the problem of low efficiency of obtaining the playable file according to the vehicle data is solved.

Description

Technical Field

[0001] This invention relates to the field of automotive technology, and more specifically to a method, apparatus, vehicle, and electronic equipment for processing vehicle data. Background Technology

[0002] When a vehicle is being tested or in use, it can collect data through its sensors. This data is then uploaded to a server, which analyzes the vehicle's status to monitor or control it.

[0003] The data collected by sensors can be called vehicle data. When uploading vehicle data to the server, because the data is collected by different sensors and involves different aspects, it needs to be processed separately and then aligned to obtain a synchronized playable file. Therefore, the process of obtaining a playable file from vehicle data is complex, resulting in low efficiency. Summary of the Invention

[0004] This application provides a method, apparatus, vehicle, and electronic device for processing vehicle data to solve the problem of low efficiency in obtaining playable files from vehicle data.

[0005] In a first aspect, this application provides a method for processing vehicle data, comprising: acquiring a vehicle data file, wherein the data file is a file encapsulating vehicle data, and the vehicle data is data collected by sensors in the vehicle; reading all binary files in the data file in parallel through multiple reading threads to obtain data for each binary file, and writing the data into memory; determining the memory block containing data belonging to the same binary file in memory by reading the topic data of the binary files in the proto file of the data file, and allocating the memory block to a parsing thread pool; parsing the corresponding data by the parsing thread pool to obtain various structured data; performing time alignment on the various structured data to obtain combined frame data; and serializing the combined frame data into a playable file.

[0006] For example, before reading all binary files in a data file in parallel by multiple read threads, the above method further includes: reading the metadata of the file system, wherein the metadata of the file system is the metadata recorded by the file system during the process of encapsulating vehicle data into data files, and the metadata includes at least the file size of the binary files in the data file; and assigning each binary file in the data file to one of the multiple read threads according to the metadata.

[0007] This application allocates multiple binary files to multiple read threads based on the file size of the binary files in the file system metadata, thereby achieving a balanced distribution of binary file reading tasks and further improving the efficiency of multiple read threads reading multiple binary files in parallel.

[0008] For example, the parsing thread pool parses the corresponding data to obtain various structured data, including: if the data is video type, the parsing thread pool calls the video decoder to parse the data into a video format file, and the video frames in the video format file have timestamps; if the data is radar point cloud type, the parsing thread pool calls the radar decoder to parse the data into a drc file, and the data in the drc file has timestamps.

[0009] This application improves the parsing efficiency of binary byte streams by using different thread pools for parsing binary byte streams according to their data types.

[0010] For example, time alignment of multiple structured data to obtain combined frame data includes: determining multiple target timestamps based on the timestamps of video frames; at each target timestamp, searching for data in the drc file whose time interval between the timestamp and the target timestamp is within a preset interval; combining the found data with the video frames at the target timestamps as a combined frame; and obtaining the combined frames at each target timestamp to obtain combined frame data.

[0011] This application achieves time alignment between video frames and radar point cloud data in the .drc file by determining the target timestamp, thus realizing the effect of combining structured data into time-aligned combined frame data and improving the efficiency of processing vehicle data into playable files.

[0012] For example, after aligning multiple structured data by time to obtain combined frame data, the above method further includes: generating two-dimensional points for each field in each frame of data in the combined frame data according to the target timestamp; and generating field curves based on the two-dimensional points.

[0013] This application generates two-dimensional points for each field in each frame of combined frame data according to the target timestamp; and generates field curves based on the two-dimensional points, thereby generating visualization curves for each field in each frame of structured data, realizing the visualization of key vehicle data.

[0014] For example, after serializing the combined frame data into a playable file, the above method further includes: uploading the playable file, field curves, video format files, and DRC files to the server in parallel using different threads.

[0015] The solution proposed in this application uses different threads to upload playable files, field curves, video format files, and DRC files in parallel, thereby improving the efficiency of uploading files to the cloud or server.

[0016] Secondly, this application provides a vehicle data processing apparatus, comprising: an acquisition module for acquiring vehicle data files, wherein the data files are files encapsulating vehicle data, and the vehicle data are data collected by sensors in the vehicle; a reading module for reading all binary files in the data files in parallel through multiple reading threads, obtaining data from each binary file, and writing the data into memory; an allocation module for determining the memory block containing data belonging to the same binary file in memory by reading the topic data of the binary files in the proto file of the data files, and allocating the memory block to a parsing thread pool; a parsing module for parsing the corresponding data by the parsing thread pool to obtain various structured data; an alignment module for performing time alignment on the various structured data to obtain combined frame data; and a processing module for serializing the combined frame data into a playable file.

[0017] Thirdly, this application provides a vehicle, including: a sensor for collecting vehicle data; and a device for processing the aforementioned vehicle data.

[0018] Fourthly, this application provides a device comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to: acquire a vehicle data file, wherein the data file is a file encapsulating vehicle data, and the vehicle data is data collected by sensors in the vehicle; read all binary files in the data file in parallel using multiple read threads to obtain data for each binary file, and write the data into memory; allocate different parsing thread pools for data belonging to different binary files in memory according to the type of the binary files; parse the corresponding data by the parsing thread pools to obtain various structured data; time-align the various structured data to obtain combined frame data; and serialize the combined frame data into a playable file.

[0019] Fifthly, this application also provides a computer storage medium storing computer-executable instructions for executing the vehicle data processing method of any of the above claims of this application.

[0020] The beneficial effects of this application are:

[0021] This application addresses the issue of low efficiency in obtaining playable files from vehicle data by acquiring vehicle data files, where the data files are encapsulated vehicle data collected by sensors. Multiple read threads read all binary files within the data files in parallel, obtaining the data for each binary file and writing it into memory. The application then determines the memory block containing data belonging to the same binary file by reading the topic data from the proto file of the data files, and allocates this memory block to a parsing thread pool. The parsing thread pool parses the corresponding data to obtain various structured data. These structured data are then time-aligned to obtain combined frame data. Finally, the combined frame data is serialized into a playable file. This method, through multiple threads reading data and multiple parsing thread pools parsing the data in parallel, achieves unified processing of different data types, improving the efficiency of obtaining playable files and thus solving the problem of low efficiency in obtaining playable files from vehicle data. Attached Figure Description

[0022] Figure 1 This is a flowchart of a vehicle data processing method according to this application;

[0023] Figure 2 A flowchart illustrating another method for processing vehicle data according to this application;

[0024] Figure 3 This is a schematic diagram illustrating the overall information of the read data in this application, resulting in the proto interface file data.

[0025] Figure 4 This is a schematic diagram illustrating the parsing of binary data in memory to obtain structured data for this application.

[0026] Figure 5 This is a schematic diagram illustrating the time synchronization of structured data according to this application;

[0027] Figure 6 A diagram illustrating the uploaded documents for this application;

[0028] Figure 7 This is a schematic diagram of the vehicle data processing device of this application;

[0029] Figure 8 This is a schematic diagram of a vehicle data processing device according to this application. Detailed Implementation

[0030] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0031] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0032] For ease of description, spatial relative terms may be used in the text to describe the relative position or movement of one element or feature relative to another element or feature, as shown in the figure. These relative terms include, for example, "inside," "outside," "middle," "outer," "below," "below," "above," "front," "back," etc. Such spatial relative terms are intended to include different orientations of the device in use or operation, other than those depicted in the figure. For example, if the device in the figure undergoes a positional flip, orientation change, or change of motion, these directional indications will change accordingly. For instance, an element described as "below other elements or features" or "below other elements or features" will subsequently be oriented "above other elements or features" or "above other elements or features." Therefore, the example term "below" can include both upper and lower orientations. The device may be otherwise oriented (rotated 90 degrees or in other directions), and the spatial relative descriptors used in the text will be interpreted accordingly.

[0033] To address the low efficiency of obtaining playable files from vehicle data in existing technologies, this application provides a vehicle data processing method that improves the efficiency of obtaining playable files from vehicle data.

[0034] Figure 1 This is a flowchart of a vehicle data processing method provided in the embodiments of this application, including:

[0035] S1, acquire the vehicle's data file, where the data file is a file encapsulated with vehicle data, and the vehicle data is the data collected by the sensors in the vehicle;

[0036] S2 reads all binary files in the data file in parallel through multiple reading threads, obtains the data of each binary file, and writes the data into memory;

[0037] S3, by reading the topic data of the binary file in the proto file of the data file, determines the memory block where the data belonging to the same binary file is located in memory, and allocates the memory block to a parsing thread pool;

[0038] S4, the parsing thread pool parses the corresponding data to obtain various structured data;

[0039] S5 performs time alignment on various structured data to obtain combined frame data;

[0040] S6 serializes the combined frame data into a playable file.

[0041] This application can be applied to scenarios where vehicle data is acquired and processed into playable files for playback. For example, during vehicle testing or control, vehicle data can be acquired, processed by the methods described in this application, and a playable file can be obtained. This playable file can be stored locally or in the cloud and can be played in real time or at any time to view the vehicle's status or control the vehicle.

[0042] In the above scenarios, the vehicle can be a gasoline-powered vehicle, a pure electric vehicle (BEV), a hybrid electric vehicle (HEV), a plug-in hybrid electric vehicle (PHEV), an extended-range electric vehicle (EREV), or a fuel cell electric vehicle (FCEV), but is not limited to these.

[0043] The vehicle data mentioned above can be data collected by sensors deployed on the vehicle. These sensors can include various types, such as cameras for collecting video data, lidar for collecting point cloud data, lidar for ranging, millimeter-wave radar, and infrared cameras for thermal imaging. Additionally, they can include sensors for speed, acceleration, angular velocity, steering angle, gear position, and braking status. Each sensor is responsible for collecting one type of data. For example, cameras collect video data, lidar collects point cloud data, and so on. If there are multiple cameras, each camera collects one channel of video data.

[0044] The data collected by sensors can be called vehicle data, and it can be of different types depending on the sensor used. In this example, vehicle data collected by different sensors can be encapsulated to obtain vehicle data files. During encapsulation, the data collected by each sensor can be stored in a folder named "topic". For example, in a multi-camera setup, each camera collects one video stream, which is stored in a folder named "topic". Each video stream from one camera is stored as a .bin file. Thus, vehicle data collected by multiple sensors will be encapsulated into multiple folders named "topic", each folder containing one or more .bin files, and each .bin file corresponds to one sensor.

[0045] When encapsulating vehicle data into data files, if the vehicle data is to be stored as multiple .bin files in multiple folders named with a topic, then a proto interface file must first be encapsulated. The proto interface file contains the encapsulation rules for packaging the vehicle data collected by sensors into binary .bin files. When each sensor collects vehicle data, the rules in this proto interface file can be followed to store the collected vehicle data into multiple .bin files in multiple folders named with a topic.

[0046] A combination of a proto interface file and multiple .bin files in multiple folders named "topic" can be used as the data file in this application. The data file can be stored locally on the vehicle.

[0047] After obtaining the aforementioned data files, the first step is to read the binary data from these files into memory. This step can be achieved using multiple read threads to read all the binary files in parallel; the number of read threads is not limited. While using multiple read threads to read all the binary files in the data file, the binary data from each binary file is read into memory and stored there. This allows the binary data in the binary files to be parsed in memory, resulting in structured data.

[0048] When using multiple threads to read data from multiple binary files in parallel, several scenarios arise because the number of threads may differ from the number of binary files. If the number of threads exceeds the number of binary files, only a subset of the threads are used to read the binary files in parallel, rather than using all threads. Each binary file is assigned to a separate thread to complete the parallel reading of multiple binary files. However, if the number of threads is less than the number of binary files, it is not possible to read all binary files simultaneously in parallel. In this case, the binary files need to be allocated, determining which reading thread each binary file is assigned to. In this scenario, all binary files are assigned to different threads; binary files assigned to the same thread are read serially, while binary files from different threads are read in parallel.

[0049] After multiple threads read the binary data from all binary files into memory, the binary data is stored in a memory block. Additionally, the system records which binary file the data was read from, including the topic name, timestamp range, and file index.

[0050] Because the records contain information such as topic name, timestamp range, and file index, it's possible to distinguish which binary file the binary data in different memory blocks belongs to. Memory blocks containing binary data belonging to the same binary file are then handled by a single parsing thread pool. This allows for parallel parsing of binary data in memory using multiple parsing thread pools. Each parsing thread pool can contain one or more parsing threads. If multiple parsing threads are included, they parse the memory blocks containing binary data belonging to the same binary file in parallel.

[0051] After being parsed by multiple parsing thread pools, the binary data in memory is parsed into structured data. For example, the binary data of a binary file captured by a camera is parsed into video format, and the binary data of a binary file captured by radar point cloud is parsed into point cloud format, etc.

[0052] After obtaining multiple types of structured data, time alignment can be performed on them. This alignment aligns all structured data to the same timestamp, allowing them to be combined into a unified timeline frame. This combined frame data can then be serialized into a playable file.

[0053] This application addresses the issue of low efficiency in obtaining playable files from vehicle data by acquiring vehicle data files, where the data files are encapsulated vehicle data collected by sensors. Multiple reading threads read all binary files within the data files in parallel, obtaining data from each binary file and writing it into memory. Different parsing thread pools are allocated to data belonging to different binary files in memory based on their type. The parsing thread pools parse the corresponding data to obtain various structured data. These structured data are then time-aligned to obtain combined frame data. A method for serializing the combined frame data into a playable file is employed. This approach, combining multiple threads for data reading and multiple parsing thread pools for parallel data parsing, achieves unified processing of different data types, improving the efficiency of obtaining playable files and thus solving the problem of low efficiency in obtaining playable files from vehicle data.

[0054] As an optional example, before reading all binary files in the data file in parallel by multiple read threads, the above method further includes: reading the metadata of the file system, wherein the metadata of the file system is the metadata recorded by the file system during the process of encapsulating vehicle data into data files, and the metadata includes at least the file size of the binary files in the data file; and assigning each binary file in the data file to one of the multiple read threads according to the metadata.

[0055] In this application, before using multiple read threads to read all binary files in a data file in parallel, it is possible to first obtain the number of binary files in the data file and the basic information of each binary file, such as file size and start and end timestamps of the collected vehicle data. The size of each binary file is at least obtained by reading the file system's metadata. File system metadata is data recorded by the file system. During the process of encapsulating sensor-collected data into binary files, the file system can be called to record the size of the binary files, and this recorded size is saved as metadata. Before using the read thread to read the binary files, the size of each binary file is determined based on the metadata saved in the file system, and then the binary files can be allocated to different read threads according to their sizes.

[0056] When allocating binary files to different read threads, dynamic weighting can be performed across all read threads based on the total size of the binary files already allocated to each thread. For example, with 10 read threads and 200 binary files, 10 binary files can be randomly or sequentially allocated to the 10 read threads. Then, the total size of the binary files for each read thread is calculated. If the total sizes of the binary files for the 10 read threads are not significantly different (a threshold is used to determine if the difference is large enough), the 10 binary files can continue to be allocated to the 10 read threads. If the total sizes of the binary files for the 10 read threads differ significantly, the binary files are allocated one by one until the total sizes of the binary files for the 10 read threads are roughly equal, and then the 10 binary files are allocated to the 10 read threads.

[0057] This scheme allows 200 binary files to be distributed among 10 reading threads, with each thread handling a relatively small total size of binary files.

[0058] If a certain reading thread processes the data faster during the process, the binary file in the slower reading thread can be transferred to that thread for processing.

[0059] This application allocates multiple binary files to multiple read threads based on the file size of the binary files in the file system metadata, thereby achieving a balanced distribution of binary file reading tasks and further improving the efficiency of multiple read threads reading multiple binary files in parallel.

[0060] As an optional example, in this application, since the binary files are read into memory by multiple parallel reading threads, if the source of the binary data in memory is not recorded, the binary data from multiple binary files will be mixed together and indistinguishable. Therefore, in this application, the proto file of the data file can be read first. The proto file stores the syntax declaration, package name, message structure, field definitions (type / name / tag number), nested types, enumerations, and other information of the binary files. The field definitions can be used to distinguish the binary files. Based on the different binary files identified, after the binary files are read into memory, memory blocks are allocated, grouping the memory blocks containing the binary data belonging to the same binary file together.

[0061] After memory blocks are allocated, they are divided into multiple groups, and each group is handled by a single parsing thread pool. Each parsing thread pool parses a group of memory blocks, resulting in structured data corresponding to a binary file. Each parsing thread pool may contain one or more parsing threads. If there is only one parsing thread, it parses all memory blocks in a group. If there are multiple parsing threads, they can parse all memory blocks in a group in parallel.

[0062] This application determines the memory block containing data belonging to the same binary file by reading the proto file of the data file; it then allocates the memory block to a parsing thread pool, thereby enabling parallel parsing of the binary data of different binary files using multiple parsing thread pools, which improves the efficiency of parsing binary data and obtaining structured data.

[0063] As an optional example, the parsing thread pool parses the corresponding data to obtain various structured data, including: if the data is video type, the parsing thread pool calls the video decoder to parse the data into a video format file, and the video frames in the video format file have timestamps; if the data is radar point cloud type, the parsing thread pool calls the radar decoder to parse the data into a .drc file, and the data in the .drc file has timestamps.

[0064] In this application, when using a parsing thread pool to parse data in their respective memory blocks, if the memory block stores binary data of video type, the parsing thread pool is used to call the video decoder to parse that part of the data. The main steps include:

[0065] 1. Parse the idle threads in the thread pool to retrieve binary byte streams from the shared queue.

[0066] 2. Query the pre-registered decoder registry to identify the dedicated decoder used to parse the binary byte stream.

[0067] 3. The dedicated decoder is invoked to perform the parsing action. This includes decapsulation: identifying the container format and stripping the file header; frame decoding: calling the FFmpeg library to decompress frame by frame, restoring the binary byte stream data to the original pixel frames; generating a structured object from the original pixel frames to obtain the decoded frame sequence; and storing the list of decoded frames into the frame assembly buffer. The parsing thread then returns to the thread pool, waiting for new tasks. The decoded frames can be used to generate video format files.

[0068] If decoding fails, a log is logged and an empty frame is returned. The empty frame can be skipped when synchronizing the structured data in the future.

[0069] Additionally, if the memory block stores binary data of radar point cloud type, a parsing thread pool is used to call the point cloud decoder to parse this data. The main steps include:

[0070] 1. Parse the idle threads in the thread pool to retrieve binary byte streams from the shared task queue;

[0071] 2. Query the pre-registered decoder registry to identify the dedicated decoder used for parsing the binary byte stream.

[0072] 3. A dedicated decoder is invoked to read the first 8 bytes of the binary byte stream (magic number) and verify its format validity, thereby determining the type and protocol version of the binary byte stream; metadata is parsed to obtain the frame timestamp, the number of points in the point cloud, the radar model / coordinate system identifier, and the data version number; the point data stream is parsed multiple times in a loop, and then the information of each point is read point by point according to a predefined binary layout to construct a point cloud object; after generating structured data, the structured point cloud object is stored in a dedicated frame assembly buffer; the thread releases resources and returns to the thread pool to wait for new tasks. The decoded point cloud data can be stored as a .drc format file.

[0073] Invalid frames that are parsed out can be automatically skipped during subsequent time synchronization of structured data.

[0074] This application improves the parsing efficiency of binary byte streams by using different thread pools for parsing binary byte streams according to their data types.

[0075] As an optional example, time alignment of multiple structured data to obtain combined frame data includes: determining multiple target timestamps based on the timestamps of video frames; for each target timestamp, searching for data in the drc file whose time interval between the timestamp and the target timestamp is within a preset interval; combining the found data with the video frames of the target timestamps as a combined frame; and obtaining the combined frames for each target timestamp to obtain combined frame data.

[0076] In this application, there are multiple schemes for time alignment of various structured data. One of them is to use frame skipping to align the structured data. Taking the alignment of video frames and radar point cloud data as an example, multiple target timestamps are determined from the video frames based on the timestamps of the video frames, such as determining the target timestamps at fixed intervals or at random intervals. The determined target timestamps are distributed across different video frames. Then, for each target timestamp, data in the DRC file whose time interval with the target timestamp is within a preset interval is selected and combined with the video frame data of the target timestamp to form a combined frame for the target timestamp. For example, after determining the target timestamps at fixed intervals, for one of the target timestamps pointing to the 3rd frame of the video frame, radar point cloud data with a time interval within 0.05 seconds of the target timestamp is searched from the radar point cloud. If the 2nd frame of the radar point cloud is found, the 3rd frame of the video frame and the 2nd frame of the radar point cloud are combined as a frame under the target timestamp. Next, iterate through the combined frames of the next target timestamp until all target timestamps have been traversed to obtain the combined frame data.

[0077] Another approach to time alignment of multi-structured data is to use a holistic alignment method. For example, taking the alignment of video frames and radar point cloud data as an example, for video frame data, multiple target timestamps are determined, recorded using the time the video frame data was acquired. Similarly, for radar point cloud data, multiple target timestamps are determined, also recorded using the time the radar point cloud data was acquired. Then, the target timestamps of the video frame data and the radar point cloud data are time-aligned, so that the video frames in the video frame data correspond to the radar point cloud frames in the radar point cloud data, thus completing the time alignment.

[0078] This application achieves time alignment between video frames and radar point cloud data in the .drc file by determining the target timestamp, thus realizing the effect of combining structured data into time-aligned combined frame data and improving the efficiency of processing vehicle data into playable files.

[0079] As an optional example, after aligning multiple structured data by time to obtain combined frame data, the above method further includes: generating two-dimensional points for each field in each frame of data in the combined frame data according to the target timestamp; and generating field curves based on the two-dimensional points.

[0080] In this application, in addition to time alignment of structured data, field data can be extracted from the structured data to generate two-dimensional points, thus obtaining field curves. During processing, for each field of each frame of data in each type of structured data, a field curve is generated. Specifically, each field is converted into a two-dimensional point (X, Y) form. The X-axis (horizontal axis) typically represents the timestamp, i.e., the time when each frame was recorded or generated, allowing viewing the data's trend along the time dimension. The Y-axis (vertical axis) represents the value of a specific field, which can be any data collected by any sensor, such as the depth field in a video frame of structured video data. During data extraction, the required timestamps and field values ​​are extracted from the structured data. For each data frame, there is a corresponding timestamp and multiple fields, each field corresponding to a field value. When generating two-dimensional points, for each field, the frame's timestamp is used as the X-coordinate, and the field's value is used as the Y-coordinate to generate a two-dimensional point (X, Y). These generated two-dimensional points are cached or stored in memory for subsequent data processing or visualization operations. When it's necessary to display trends in data changes, these two-dimensional points can be read from memory, and plotting tools or libraries can be used to draw the corresponding field curves. Users can then visually observe how the data changes over time. For example, a two-dimensional point (X, Y) represents the depth of field in structured video data at a specific timestamp. By generating curve data from all two-dimensional points, a field curve representing the change in depth of field in the structured video data with respect to timestamps is obtained.

[0081] In addition, this application can also encapsulate each frame of data in each binary file into JSON format, thereby obtaining a file format that can be directly read and searched, and reserved for subsequent analysis or reuse.

[0082] This application generates two-dimensional points for each field in each frame of combined frame data according to the target timestamp; and generates field curves based on the two-dimensional points, thereby generating visualization curves for each field in each frame of structured data, realizing the visualization of key vehicle data.

[0083] As an optional example, after serializing the combined frame data into a playable file, the above method also includes uploading the playable file, field curves, video format files, and .drc files to the server in parallel using different threads.

[0084] In this application, the combined frame data obtained after time alignment of structured data is serialized into a playable file, which can then be uploaded to a server or the cloud. In addition, each type of structured data parsed in this application, such as video format files, DRC files, and JSON format files converted from the original binary files, is uploaded to the cloud in parallel using separate threads. For example, video data is uploaded in chunks using independent threads, point cloud data is uploaded concurrently using independent threads, and large amounts of small archived data, such as field curves and JSON data, can be packaged and uploaded using independent threads.

[0085] The solution proposed in this application uses different threads to upload playable files, field curves, video format files, and DRC files in parallel, thereby improving the efficiency of uploading files to the cloud or server.

[0086] Figure 2 This is a flowchart of the process from sensor-collected vehicle data to the conversion of the vehicle data into a playable file, and then uploading the playable file.

[0087] Data input can be data files obtained in this application, organized as folders. Each data unit is a main folder containing two parts: 1. A subdirectory titled "Overall Data Information": containing proto interface files (describing recording time, topic information, version information, etc.); 2. A subdirectory titled "Signal File Collection": each topic corresponds to multiple .bin files (binary files), representing the data content of each frame for the entire time period of that topic. During data reading, the overall data information is first read to obtain overall metadata. Then, multiple threads are started to read the .bin files of multiple topics in parallel, caching the read binary data in memory (runtime cache), significantly improving data reading speed.

[0088] For example, in the receiving system, a directory structure is provided for each recorded data: root directory / 2025-08-xxx_data, which includes: the proto interface file msg_def.proto and the metadata file meta.json in the info / directory, and the topics / directory, where each topic (such as camera_front.bin, lidar.bin, imu.bin) stores the frame binary files for all time periods of that topic. The system first reads info / meta.json to parse out basic information such as the recording start and end times, the list of topics, and the version number; then the system spawns multiple threads, each thread is responsible for reading the .bin files of one or more topics into the memory cache.

[0089] Then, data parsing is performed to obtain various types of structured data. This step also reads the overall data information to obtain proto files, determines the type of .bin files, and writes the proto files by reading them and dynamically loading all interface definitions. For custom structure files, decoders (such as video decoders, radar decoders, etc.) are pre-registered. Structured data is obtained through parallel parsing using a multi-parse thread pool. The structured data undergoes rendering data transformation and encapsulation, performs protobuf deserialization, generates MP4 files from video data, decodes and compresses radar point clouds into drc files, and after data encapsulation and coordinate system transformation, generates frame data usable for visual rendering, which is then compressed and uploaded.

[0090] Based on the msg_def.proto file, the system dynamically loads the proto interface definition and parses the data structure of each topic. If custom structured data exists (such as video or radar), the system loads the resources required for that topic type through a pre-registered decoder module. Subsequently, each topic uses a dedicated thread pool for decoding. This includes: protobuf decoding: video data is deserialized to generate mp4 file segments; radar data is decoded and compressed to generate drc files. After processing, the data is uniformly encapsulated and transformed to the required coordinate system to generate rendering frame data; field curve generation: for each frame, the system iterates through the numerical fields in the proto, generating (timestamp, value) pairs and caching them in memory for subsequent visualization; single-frame JSON output: the entire frame of proto data is encapsulated into a JSON string and written to the local output / json / directory.

[0091] Furthermore, after time alignment of the structured data, combined frame data is obtained. The system establishes a time base based on the configured rendering frame rate (e.g., 10fps) and triggers a frame assembly task every 100ms starting from the first frame (determining a target timestamp every 100ms). The frame assembly task checks whether the corresponding frames for each topic exist in the cache and whether the duration exceeds the allowed range (e.g., ±50ms). If it exceeds the range, the data for that topic is discarded; otherwise, the frame data for each topic is assembled into a combined structured proto protocol body and serialized and written to the output / combined / directory.

[0092] The serialized playable files are uploaded. During upload, the system scans various data subdirectories under `output / ` and processes MP4, drc, single-frame JSON, curve JSON, and combined proto files respectively. For small JSON files (such as curves and frame JSON), the system first packages and compresses them into zip or tar files, and then multiple threads concurrently upload all files to the remote data bucket. MP4 / drc / combined proto files are uploaded using separate thread pools to ensure maximum overall upload speed, while logging is recorded during the upload process to track the status.

[0093] Figure 3 This diagram illustrates how the application reads overall data information to obtain proto interface file data. By reading the overall information of the business data, the file start and end times, version information, etc., are parsed out to obtain data such as the type and size of the binary file. The corresponding reading thread then reads the binary file, obtains the binary data, and writes it into memory.

[0094] Figure 4 This diagram illustrates the process of parsing binary data in memory to obtain structured data. For each binary file, a corresponding thread pool is used for parsing. For example, proto-A data is serialized, deserialized, assembled, rendered, and transformed to obtain structured data. Furthermore, individual field data for each frame is extracted into field curves, and each frame is output as a JSON file. Video data is generated as an MP4 file, and laser point cloud data is generated as a DRC file.

[0095] Figure 5 This is a schematic diagram illustrating the time synchronization of structured data according to this application. Different rendering data are synchronized in time using frame skipping to obtain combined frame data.

[0096] Figure 6 This is a schematic diagram illustrating the file upload process of this application. Video format files, .drc files, and JSON format files converted from the original binary files are all uploaded to the cloud in parallel using separate threads.

[0097] Figure 7 This is a schematic diagram of a vehicle data processing device provided in an embodiment of this application, including:

[0098] The acquisition module 701 is used to acquire the vehicle's data file, wherein the data file is a file obtained by encapsulating vehicle data, and the vehicle data is the data collected by the sensors in the vehicle;

[0099] The reading module 702 is used to read all binary files in the data file in parallel through multiple reading threads, obtain the data of each binary file, and write the data into memory;

[0100] The allocation module 703 is used to determine the memory block containing data belonging to the same binary file in memory by reading the topic data of the binary file in the proto file of the data file, and allocate the memory block to a parsing thread pool.

[0101] Parsing module 704 is used to parse the corresponding data by the parsing thread pool to obtain various types of structured data;

[0102] Alignment module 705 is used to perform time alignment on various structured data to obtain combined frame data;

[0103] Processing module 706 is used to serialize the combined frame data into a playable file.

[0104] This application can be applied to scenarios where vehicle data is acquired and processed into playable files for playback. For example, during vehicle testing or control, vehicle data can be acquired, processed by the methods described in this application, and a playable file can be obtained. This playable file can be stored locally or in the cloud and can be played in real time or at any time to view the vehicle's status or control the vehicle.

[0105] The vehicle data mentioned above can be data collected by sensors deployed on the vehicle. These sensors can include various types, such as cameras for collecting video data, lidar for collecting point cloud data, lidar for ranging, millimeter-wave radar, and infrared cameras for thermal imaging. Additionally, they can include sensors for speed, acceleration, angular velocity, steering angle, gear position, and braking status. Each sensor is responsible for collecting one type of data. For example, cameras collect video data, lidar collects point cloud data, and so on. If there are multiple cameras, each camera collects one channel of video data.

[0106] The data collected by sensors can be called vehicle data, and it can be of different types depending on the sensor used. In this example, vehicle data collected by different sensors can be encapsulated to obtain vehicle data files. During encapsulation, the data collected by each sensor can be stored in a folder named "topic". For example, in a multi-camera setup, each camera collects one video stream, which is stored in a folder named "topic". Each video stream from one camera is stored as a .bin file. Thus, vehicle data collected by multiple sensors will be encapsulated into multiple folders named "topic", each folder containing one or more .bin files, and each .bin file corresponds to one sensor.

[0107] When encapsulating vehicle data into data files, if the vehicle data is to be stored as multiple .bin files in multiple folders named with a topic, then a proto interface file must first be encapsulated. The proto interface file contains the encapsulation rules for packaging the vehicle data collected by sensors into binary .bin files. When each sensor collects vehicle data, the rules in this proto interface file can be followed to store the collected vehicle data into multiple .bin files in multiple folders named with a topic.

[0108] A combination of a proto interface file and multiple .bin files in multiple folders named "topic" can be used as the data file in this application. The data file can be stored locally on the vehicle.

[0109] After obtaining the aforementioned data files, the first step is to read the binary data from these files into memory. This step can be achieved using multiple read threads to read all the binary files in parallel; the number of read threads is not limited. While using multiple read threads to read all the binary files in the data file, the binary data from each binary file is read into memory and stored there. This allows the binary data in the binary files to be parsed in memory, resulting in structured data.

[0110] When attempting to read data from multiple binary files in parallel using multiple threads, several scenarios arise because the number of threads may differ from the number of binary files. If the number of threads exceeds the number of binary files, only a subset of the threads are used for parallel reading; not all threads are required. Each binary file is assigned a separate thread to handle the parallel reading of multiple binary files. However, if the number of threads is less than the number of binary files, it's impossible to read all binary files simultaneously in parallel. In this case, the binary files need to be allocated, determining which thread each binary file is assigned to. In this scenario, all binary files are assigned to different threads; binary files assigned to the same thread are read serially, while binary files from different threads are read in parallel.

[0111] After multiple threads read the binary data from all binary files into memory, the binary data is stored in a memory block. Additionally, the system records which binary file the data was read from, including the topic name, timestamp range, and file index.

[0112] Because the records contain information such as topic name, timestamp range, and file index, it's possible to distinguish which binary file the binary data in different memory blocks belongs to. Memory blocks containing binary data belonging to the same binary file are then handled by a single parsing thread pool. This allows for parallel parsing of binary data in memory using multiple parsing thread pools. Each parsing thread pool can contain one or more parsing threads. If multiple parsing threads are included, they parse the memory blocks containing binary data belonging to the same binary file in parallel.

[0113] After being parsed by multiple parsing thread pools, the binary data in memory is parsed into structured data. For example, the binary data of a binary file captured by a camera is parsed into video format, and the binary data of a binary file captured by radar point cloud is parsed into point cloud format, etc.

[0114] After obtaining multiple types of structured data, time alignment can be performed on them. This alignment aligns all structured data to the same timestamp, allowing them to be combined into a unified timeline frame. This combined frame data can then be serialized into a playable file.

[0115] This application addresses the issue of low efficiency in obtaining playable files from vehicle data by acquiring vehicle data files, where the data files are encapsulated vehicle data collected by sensors. Multiple reading threads read all binary files within the data files in parallel, obtaining data from each binary file and writing it into memory. Different parsing thread pools are allocated to data belonging to different binary files in memory based on their type. The parsing thread pools parse the corresponding data to obtain various structured data. These structured data are then time-aligned to obtain combined frame data. A method for serializing the combined frame data into a playable file is employed. This approach, combining multiple threads for data reading and multiple parsing thread pools for parallel data parsing, achieves unified processing of different data types, improving the efficiency of obtaining playable files and thus solving the problem of low efficiency in obtaining playable files from vehicle data.

[0116] Other examples of this application can be found in the examples above, and will not be repeated here.

[0117] This application also provides a vehicle, including: a sensor for collecting vehicle data; and a device for processing the aforementioned vehicle data. Other examples of this application are provided above and will not be repeated here.

[0118] like Figure 8As shown in the figure, this application embodiment provides a vehicle data processing device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0119] Memory 113 is used to store computer programs;

[0120] In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the vehicle data processing method provided in any of the foregoing method embodiments, including: acquiring a vehicle data file, wherein the data file is a file encapsulating vehicle data, and the vehicle data is data collected by sensors in the vehicle; reading all binary files in the data file in parallel through multiple reading threads to obtain data for each binary file, and writing the data into memory; allocating different parsing thread pools for data belonging to different binary files in memory according to the type of the binary files; parsing the corresponding data by the parsing thread pools to obtain various structured data; performing time alignment on the various structured data to obtain combined frame data; and serializing the combined frame data into a playable file.

[0121] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the vehicle data processing method provided in any of the foregoing method embodiments.

[0122] The device embodiments described above are illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment, depending on actual needs.

[0123] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments, or some parts of the embodiments.

[0124] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0125] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for processing vehicle data, characterized in that, include: Obtain the vehicle's data file, wherein the data file is a file encapsulating vehicle data, and the vehicle data is data collected by sensors in the vehicle; All binary files in the data file are read in parallel by multiple read threads to obtain the data of each binary file, and the data is written into memory; By reading the topic data of the binary file in the proto file of the data file, the memory block containing the data belonging to the same binary file in the memory is determined, and the memory block is allocated to a parsing thread pool; The parsing thread pool parses the corresponding data to obtain various types of structured data; The various structured data are time-aligned to obtain combined frame data; The combined frame data is serialized into a playable file; The parsing thread pool parses the corresponding data to obtain various structured data, including: if the data is video data, the parsing thread pool calls the video decoder to parse the data into a video format file, and the video frames in the video format file have timestamps; if the data is radar point cloud data, the parsing thread pool calls the radar decoder to parse the data into a drc file, and the data in the drc file has timestamps. The process of time-aligning the various structured data to obtain combined frame data includes: determining multiple target timestamps based on the timestamps of the video frames; for each target timestamp, searching for data in the drc file whose time interval between the timestamp and the target timestamp is within a preset interval; combining the found data with the video frames of the target timestamps as a combined frame; and obtaining the combined frames for each target timestamp to obtain the combined frame data.

2. The method according to claim 1, characterized in that, Before reading all binary files in the data file in parallel using multiple read threads, the method further includes: Read the metadata of the file system, wherein the metadata of the file system is the metadata recorded by calling the file system during the process of encapsulating the vehicle data into a data file, and the metadata includes at least the file size of the binary file in the data file; Based on the metadata, each binary file in the data file is assigned to one of the plurality of read threads.

3. The method according to claim 2, characterized in that, After performing time alignment on the various structured data to obtain combined frame data, the method further includes: For each field in each frame of the combined frame data, a two-dimensional point is generated according to the target timestamp; Generate field curves based on the two-dimensional points.

4. The method according to claim 3, characterized in that, After serializing the combined frame data into a playable file, the method further includes: The playable file, the field curve, the video format file, and the .drc file are uploaded to the server in parallel using different threads.

5. A vehicle data processing device, characterized in that, include: The acquisition module is used to acquire vehicle data files, wherein the data file is a file encapsulating vehicle data, and the vehicle data is data collected by sensors in the vehicle; The reading module is used to read all binary files in the data file in parallel through multiple reading threads, obtain the data of each binary file, and write the data into memory; The allocation module is used to allocate different parsing thread pools to data belonging to different binary files in memory according to the type of the binary file; The parsing module is used to parse the corresponding data by the parsing thread pool to obtain various types of structured data; An alignment module is used to perform time alignment on the various structured data to obtain combined frame data; The processing module is used to serialize the combined frame data into a playable file; The parsing module includes a parsing unit, configured to, if the data is video data, use a parsing thread pool to call a video decoder to parse the data into a video format file, wherein the video frames in the video format file have timestamps; and if the data is radar point cloud data, use a parsing thread pool to call a radar decoder to parse the data into a drc file, wherein the data in the drc file has timestamps. The process of time-aligning the various structured data to obtain combined frame data includes: determining multiple target timestamps based on the timestamps of the video frames; for each target timestamp, searching for data in the drc file whose time interval between the timestamp and the target timestamp is within a preset interval; combining the found data with the video frames of the target timestamps as a combined frame; and obtaining the combined frames for each target timestamp to obtain the combined frame data.

6. A vehicle, characterized in that, include: Sensors are used to collect vehicle data; It also includes the vehicle data processing device as described in claim 5.

7. A vehicle data processing device, characterized in that, The system includes at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus. The processor is configured to: acquire a vehicle data file, wherein the data file is a file encapsulating vehicle data, and the vehicle data is data collected by sensors in the vehicle; read all binary files in the data file in parallel using multiple read threads, obtain data for each binary file, and write the data into memory; allocate different parsing thread pools to data belonging to different binary files in memory according to the type of the binary files; parse the corresponding data using the parsing thread pools to obtain various structured data; perform time alignment on the various structured data to obtain combined frame data; and serialize the combined frame data into a playable file. The thread pool parses the corresponding data to obtain various structured data, including: if the data is video data, the parsing thread pool calls the video decoder to parse the data into a video format file, where the video frames in the video format file have timestamps; if the data is radar point cloud data, the parsing thread pool calls the radar decoder to parse the data into a DRC file, where the data in the DRC file has timestamps; wherein, time alignment of the various structured data to obtain combined frame data includes: determining multiple target timestamps based on the timestamps of the video frames; at each target timestamp, searching the data in the DRC file where the time interval between the timestamp and the target timestamp is within a preset interval; combining the found data with the video frames of the target timestamps as a combined frame; obtaining the combined frames for each target timestamp to obtain the combined frame data.

Images