Automatic driving multi-source data acquisition method and system

Abstract

The invention provides an automatic driving multi-source data collection method which comprises the steps that a plurality of independent threads are created, MPU data, ADECU data and / or F9K data are collected and written into a first Buffer, and each thread is bound with a unique port number; collecting CAN and / or CAN FD data and writing the CAN and / or CAN FD data into the first Buffer; pulling the RTSP and / or RTMP stream of the foresight and / or surround view camera and storing the RTSP and / or RTMP stream as an independent video file; and when the data volume of the first Buffer exceeds a preset threshold value, immediately switching to a second Buffer to continue to collect and write new data and monitor the data volume, writing the existing data of the first Buffer into a storage medium, emptying the existing data for later use, and waiting for next Buffer switching. According to the method, a mixed acquisition framework of MPU, ADECU, F9K, CAN and CAN FD data and / or independent video files and a double-Buffer hot switching mechanism are originally created for the high-load data of the emergency scene of the commercial vehicle, the problem of frame loss of the high-load data is solved, the reliability of the system is improved, and the integrity and traceability of the data of the automatic driving emergency scene are ensured.

Description

Technical Field

[0001] This application relates to the field of data acquisition technology, and in particular to a method and system for acquiring multi-source data for autonomous driving. Background Technology

[0002] In the field of autonomous driving technology, especially in the research and development of commercial vehicles (such as freight trucks and buses), the complete acquisition and synchronous processing of multi-source heterogeneous data under real-world road conditions is crucial for verifying the reliability of autonomous driving systems. Current mainstream data acquisition solutions suffer from the following technical bottlenecks:

[0003] 1. High-load data frame loss problem.

[0004] Existing systems typically employ single-threaded UDP transmission or fixed buffer mechanisms when acquiring data from control units such as MPU, ADECU, and F9K. When a vehicle encounters complex and sudden scenarios (such as emergency obstacle avoidance or multi-target intersections), the surge in data traffic can lead to buffer overflows. For example, the CAN / CAN FD bus can generate over 2000 frames of data per second; a single-buffer architecture is highly susceptible to the loss of critical control commands (such as braking signals), impacting subsequent accident root cause analysis.

[0005] 2. Lack of synchronization of multimodal data.

[0006] Traditional solutions store sensor data (such as MPU and CAN) and video data (forward / surround view cameras) separately. Due to the lack of a unified time reference, it is difficult to accurately align control commands with the surrounding visuals during later analysis. A typical example is that the ADECU issues a steering command at t = 10.5 s, but the corresponding image from the surround view video can only be located up to t = 10.8 s ± 0.3 s due to storage delay, making it impossible to verify the accuracy of the decision.

[0007] 3. Inefficient storage architecture.

[0008] Camera data is typically written directly to disk as an RTMP / RTSP stream, while control data needs to be relayed to the SOC via MCU-SPI for processing. This independent transmission of the two types of data leads to I / O resource contention, especially on embedded storage media such as eMMC. When video writes occupy the full bandwidth, the storage latency of control data can reach hundreds of milliseconds, exacerbating data asynchrony. Summary of the Invention

[0009] This application aims to solve at least one of the technical problems existing in the prior art mentioned above, and proposes a method and system for multi-source data acquisition in autonomous driving. It aims to overcome the industry pain point of data integrity loss in sudden scenarios of commercial vehicles and provide a highly reliable data foundation for the iteration of autonomous driving algorithms.

[0010] In a first aspect, embodiments of this application provide a method for acquiring multi-source data for autonomous driving, including:

[0011] Create several independent threads to collect MPU, ADECU and / or F9K data respectively and write them to the first buffer. Each thread is bound to a unique port number.

[0012] Acquire CAN and / or CAN FD data and write it to the first buffer;

[0013] Pull the RTSP and / or RTMP streams from the forward-looking and / or surround-view cameras and store them as separate video files;

[0014] When the amount of data in the first buffer exceeds a preset threshold, the system immediately switches to the second buffer to continue collecting and writing new data and monitoring the amount of data. At the same time, the existing data in the first buffer is written to the storage medium and cleared for later use, waiting for the next buffer switch.

[0015] Furthermore, when the data volume in the first buffer exceeds a preset threshold, the system immediately switches to the second buffer to continue collecting and writing new data and monitoring the data volume. Simultaneously, the existing data in the first buffer is written to the storage medium and cleared for later use, awaiting the next buffer switch. This includes:

[0016] The system monitors the capacity fill rate of the first buffer. When the fill rate exceeds a first preset threshold, it pauses the acquisition and writing of data and immediately switches to the second buffer to continue acquiring and writing new data. At the same time, it writes the existing data in the first buffer to the storage medium and clears it for later use.

[0017] The system monitors the capacity fill rate of the second buffer. When the fill rate exceeds the second preset threshold, it pauses the acquisition and writing of data and immediately switches to the first buffer to continue acquiring and writing new data. At the same time, it writes the existing data in the second buffer to the storage medium and clears it for later use. This logic is repeated in a loop.

[0018] Furthermore, the step of retrieving the RTSP and / or RTMP streams from the forward-looking and / or surround-view cameras and storing them as independent video files includes:

[0019] Assign a first storage path to the forward-view camera and a second storage path to the surround-view camera. The file names should include timestamps and vehicle IDs.

[0020] Furthermore, it also includes:

[0021] The data parsing steps include: aligning MPU, ADECU, F9K, CAN and / or CAN FD data according to the acquisition timestamp, and reconstructing the timing logic of vehicle control commands and sensor responses.

[0022] Furthermore, it also includes:

[0023] The UDP thread management steps include: allocating an independent memory pool for each thread and dynamically adjusting the UDP socket buffer size based on real-time network throughput.

[0024] Secondly, embodiments of this application provide an autonomous driving multi-source data acquisition system, including:

[0025] The multi-protocol acquisition module creates several independent threads to acquire MPU, ADECU, and / or F9K data respectively and write them to the first buffer. Each thread is bound to a unique port number; it also acquires CAN and / or CAN FD data and writes them to the first buffer.

[0026] The video processing module pulls RTSP and / or RTMP streams from the forward-looking and / or surround-view cameras and stores them as independent video files;

[0027] A dual-buffer storage control module controls the first buffer and the second buffer to alternately perform data acquisition and writing operations;

[0028] Storage medium for storing MPU, ADECU, F9K, CAN, CAN FD data and / or standalone video files.

[0029] Furthermore, the dual-buffer storage control module includes:

[0030] The status monitoring submodule monitors the capacity fill rate of the first buffer and the second buffer in real time.

[0031] The switching logic submodule generates a buffer switching instruction when the capacity fill rate exceeds a preset fill rate threshold.

[0032] The pointer management submodule resets the read and write pointers of the Buffer and updates the storage address mapping.

[0033] Furthermore, the video processing module includes:

[0034] The video splitting unit allocates storage paths according to camera type and embeds timestamps and vehicle IDs into file metadata.

[0035] Furthermore, it also includes:

[0036] The data parsing engine aligns MPU, ADECU, F9K, CAN, and / or CAN FD data according to the acquisition timestamp, reconstructing the timing logic of vehicle control commands and sensor responses.

[0037] Furthermore, it also includes:

[0038] The UDP thread management unit allocates an independent memory pool for each thread and dynamically adjusts the UDP socket buffer size based on real-time network throughput.

[0039] This application provides a multi-source data acquisition method for autonomous driving. For high-load data in sudden scenarios of commercial vehicles, it has created a unique hybrid acquisition architecture for MPU, ADECU, F9K, CAN, CAN FD data and / or independent video files, as well as a dual-buffer hot-switching mechanism. This solves the problem of frame loss in high-load data, improves system reliability, and ensures the integrity and traceability of autonomous driving emergency scenario data. Attached Figure Description

[0040] Figure 1 This application provides a core flowchart of a multi-source data acquisition method for autonomous driving.

[0041] Figure 2 This is a data structure diagram of the autonomous driving data provided in the embodiments of this application. Detailed Implementation

[0042] To enable those skilled in the art to better understand the technical solutions of this application, exemplary embodiments of this application are described below with reference to the accompanying drawings, including various details of the embodiments of this application to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. Unless otherwise specified, the various embodiments of this application and the features within those embodiments can be combined with each other.

[0043] As used herein, the term “and / or” includes any and all combinations of one or more of the associated enumerated entries. The terminology used herein is for describing particular embodiments only and is not intended to limit the application. As used herein, the singular forms “a” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated features, integrals, steps, operations, elements, and / or components is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0044] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It should also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0045] Definition of key technical terms:

[0046] (1) MPU: Microprocessor Unit, usually refers to a microprocessor unit used to process various computing tasks.

[0047] (2) ADECU: Advanced Driver Assistance System Electronic Control Unit, which refers to the electronic control unit of the advanced driver assistance system, which focuses on the control and management of the autonomous driving system.

[0048] (3) F9K: This is an Advanced Driver Assistance System (ADAS) control unit specifically designed for autonomous driving and advanced driver assistance functions. The F9K typically integrates multiple functions, including sensor data processing, environmental perception, execution of control algorithms, and communication capabilities. It can process data from multiple sensors (such as radar, LiDAR, and cameras) and implement autonomous driving functions (such as lane keeping, adaptive cruise control, etc.).

[0049] (4) CAN: Controller Area Network, is a high-efficiency and reliable automotive communication protocol designed for distributed real-time control systems. Its differential signal transmission and priority arbitration mechanism enable stable data transmission even in complex environments.

[0050] (5) CAN FD: CAN with Flexible Data Rate, which is an upgraded version of traditional CAN. It retains the physical layer but optimizes the CAN protocol. Traditional CAN is suitable for low-speed, small-data-volume scenarios, while CAN FD supports higher bandwidth requirements (such as in-vehicle entertainment systems, ADAS, etc.).

[0051] (6) Buffer: Buffer memory.

[0052] (7) FFmpeg: It is an open-source cross-platform multimedia processing framework that provides audio and video encoding, decoding, transcoding, format conversion, streaming media processing and other functions, and supports almost all mainstream audio and video formats and encoding and decoding standards.

[0053] As described in the background section of this specification, current mainstream data acquisition solutions suffer from technical bottlenecks such as frame loss under high load, lack of multimodal data synchronization, and inefficient storage architecture. In view of this, this application proposes a multi-source data acquisition method and system for autonomous driving, aiming to overcome the industry pain point of data integrity loss in sudden scenarios in commercial vehicles and provide a highly reliable data foundation for the iteration of autonomous driving algorithms.

[0054] refer to Figure 1 One embodiment of this application provides a method for acquiring multi-source data for autonomous driving, which may specifically include the following steps.

[0055] Step 1: Create several independent threads to collect MPU, ADECU and / or F9K data respectively and write them to the first buffer. Each thread is bound to a unique port number.

[0056] Data in autonomous driving is mainly divided into MPU, ADECU, F9K, forward and surround view cameras, CAN and CAN FD, etc., and these data are categorized according to... Figure 2 The data is stored in a specific manner. The forward-looking and surround-view cameras are stored as separate files for users to review and replay.

[0057] In autonomous driving systems, the MPU, F9K, and ADECU can work together to achieve comprehensive driver assistance and autonomous driving functions. The MPU can be used to provide broader computing power, while the F9K and ADECU focus on the specific implementation and control of autonomous driving and advanced driver assistance.

[0058] The data from MPU, ADECU, and F9K is collected via UDP protocol. Three independent threads are created for data collection, and different port numbers are used to distinguish the data. The collected data is then written into the first buffer.

[0059] Preferably, the multi-source data acquisition method for autonomous driving further includes a UDP thread management step. The UDP thread management step includes: allocating an independent memory pool for each thread and adaptively and dynamically adjusting the UDP socket buffer size based on real-time network throughput. In some embodiments, the MPU acquisition thread memory pool is fixed at 128 MB, and when network latency increases, the UDP buffer can be adaptively expanded from 32 KB to 64 KB. This configuration optimizes high-concurrency data processing capabilities and reduces the risk of data loss due to network jitter.

[0060] Step 2: Acquire CAN and / or CAN FD data and write it to the first buffer.

[0061] Data such as CAN and CAN FD is acquired by the MCU, and the acquired data is transmitted to the SOC via SPI and also written into the aforementioned first buffer.

[0062] Step 3: Extract the RTSP and / or RTMP streams from the front-view and / or surround-view cameras and store them as independent video files. Specifically, use FFmpeg to extract the RTSP / RTMP streams from the front-view / surround-view cameras and store them directly as independent video files.

[0063] More specifically, a first storage path is allocated to the front-view camera, and a second storage path is allocated to the surround-view camera. The file names include timestamps and vehicle IDs. In some embodiments, the front-view camera video data is stored as / data / front / 20250731_0830_VIN123.mp4, and the surround-view camera video data is stored as / data / surround / same-prefix.mp4. This configuration supports rapid retrieval and playback based on spatiotemporal dimensions, enabling quick verification of the accuracy of autonomous driving decisions.

[0064] Step 4: When the amount of data in the first buffer exceeds the preset threshold, immediately switch to the second buffer to continue collecting and writing new data and monitoring the amount of data. At the same time, write the existing data in the first buffer to the storage medium and clear it for later use, waiting for the next buffer switch.

[0065] When acquiring large amounts of data from MPU, ADECU, F9K, CAN, and / or CAN FD, data frame loss may occur. To address this issue, this application employs a dual-buffer approach for data processing: one buffer handles data acquisition, while the other handles data writing. After data acquisition is complete, the buffer can be switched between for both acquisition and writing.

[0066] Step 4 may specifically include:

[0067] The capacity fill rate of the first buffer is monitored. When it exceeds the first preset fill rate threshold, the acquisition and writing of data is paused and the system immediately switches to the second buffer to continue acquiring and writing new data. At the same time, the existing data in the first buffer is written to the storage medium and cleared for later use.

[0068] The system monitors the capacity fill rate of the second buffer. When the fill rate exceeds the second preset threshold, it pauses the acquisition and writing of data and immediately switches to the first buffer to continue acquiring and writing new data. At the same time, it writes the existing data in the second buffer to the storage medium and clears it for later use. This logic is repeated in a loop.

[0069] In some embodiments, for example, a buffer switch is triggered when the first buffer is filled to 85%. During the transfer of existing data in the first buffer to the storage medium, newly acquired data (newly acquired MPU, ADECU, F9K, CAN, and / or CANFD data) is written to the second buffer. After the existing data in the first buffer is transferred to the storage medium, its capacity is cleared for future use. A buffer switch is triggered when the second buffer is filled to 80%. During the transfer of existing data in the second buffer to the storage medium, newly acquired data is immediately written to the first buffer. After the existing data in the second buffer is transferred to the storage medium, its capacity is cleared for future use. This logic repeats continuously. This configuration enables uninterrupted acquisition and writing, avoiding data overwriting and loss.

[0070] After the user collects the data, it is analyzed according to the time of collection to reconstruct the entire process during driving. Camera data also allows observation of the surrounding environment, thus verifying the correctness of the autonomous driving's processing. Specifically, the data from the MPU, ADECU, F9K, CAN, and / or CAN FD are aligned according to the collection timestamp to reconstruct the timing logic of vehicle control commands and sensor responses. In some embodiments, the braking command time of the ADECU (t=12.5 s) is overlaid and analyzed with the frame of the forward-looking camera (t=12.5 s±0.1 s). This setup allows for accurate reproduction of the autonomous driving decision-making process and pinpointing defects in handling unexpected scenarios.

[0071] This application constructs a hybrid acquisition channel based on "UDP+SPI+FFmpeg," achieving hardware resource decoupling through port isolation and thread allocation. The designed dual-buffer hot-switching mechanism dynamically avoids frame loss under high load. An embedded spatiotemporal alignment engine enables microsecond-level synchronous parsing of control commands and video frames. This application forms a closed loop through multi-threaded port isolation acquisition → dynamic dual-buffer switching to prevent frame loss → spatiotemporal alignment parsing, ensuring data integrity and traceability. This overcomes the industry pain point of data integrity loss in sudden commercial vehicle scenarios, providing a highly reliable data foundation for autonomous driving algorithm iteration.

[0072] Another embodiment of this application proposes an autonomous driving multi-source data acquisition system, which may include multiple functional modules such as a multi-protocol acquisition module, a video processing module, a dual-buffer storage control module, and a storage medium.

[0073] The multi-protocol acquisition module creates several independent threads to acquire MPU, ADECU, and / or F9K data and write them to the first buffer. Each thread is bound to a unique port number; it also acquires CAN and / or CAN FD data and writes them to the first buffer.

[0074] The video processing module retrieves RTSP and / or RTMP streams from forward-looking and / or surround-view cameras and stores them as independent video files. The video processing module includes:

[0075] The video splitting unit allocates storage paths according to camera type and embeds timestamps and vehicle IDs into file metadata.

[0076] The dual-buffer storage control module controls the first buffer and the second buffer to alternately perform data acquisition and writing operations.

[0077] The dual-buffer storage control module may also include a status monitoring submodule, a switching logic submodule, and a pointer management submodule.

[0078] The status monitoring submodule monitors the capacity fill rate of the first buffer and the second buffer in real time.

[0079] The switching logic submodule generates a buffer switching instruction when the capacity fill rate exceeds a preset fill rate threshold.

[0080] The pointer management submodule resets the read and write pointers of the Buffer and updates the storage address mapping.

[0081] Storage medium for storing MPU, ADECU, F9K, CAN, CAN FD data and / or standalone video files.

[0082] Preferably, the autonomous driving multi-source data acquisition system may further include:

[0083] The data parsing engine aligns MPU, ADECU, F9K, CAN, and / or CAN FD data according to the acquisition timestamp, reconstructing the timing logic of vehicle control commands and sensor responses.

[0084] The UDP thread management unit allocates an independent memory pool for each thread and dynamically adjusts the UDP socket buffer size based on real-time network throughput.

[0085] For the specific technical details of each functional module in the autonomous driving multi-source data acquisition system proposed in this embodiment, please refer to the aforementioned specific embodiment of an autonomous driving multi-source data acquisition method, which will not be repeated here.

[0086] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0087] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for general illustrative purposes only and should not be construed as limiting. In some embodiments, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this application as set forth by the appended claims.

Claims

1. A method for acquiring multi-source data for autonomous driving, characterized in that, include: Create several independent threads to collect MPU, ADECU and / or F9K data respectively and write them to the first buffer. Each thread is bound to a unique port number. Acquire CAN and / or CAN FD data and write it to the first buffer; Pull the RTSP and / or RTMP streams from the forward-looking and / or surround-view cameras and store them as separate video files; When the amount of data in the first buffer exceeds a preset threshold, the system immediately switches to the second buffer to continue collecting and writing new data and monitoring the amount of data. At the same time, the existing data in the first buffer is written to the storage medium and cleared for later use, waiting for the next buffer switch.

2. The method for acquiring multi-source data for autonomous driving according to claim 1, characterized in that, When the data volume in the first buffer exceeds a preset threshold, the system immediately switches to the second buffer to continue collecting and writing new data while monitoring the data volume. Simultaneously, the existing data in the first buffer is written to the storage medium and cleared for later use, awaiting the next buffer switch. This includes: The system monitors the capacity fill rate of the first buffer. When the fill rate exceeds a first preset threshold, it pauses the acquisition and writing of data and immediately switches to the second buffer to continue acquiring and writing new data. At the same time, it writes the existing data in the first buffer to the storage medium and clears it for later use. The system monitors the capacity fill rate of the second buffer. When the fill rate exceeds the second preset threshold, it pauses the acquisition and writing of data and immediately switches to the first buffer to continue acquiring and writing new data. At the same time, it writes the existing data in the second buffer to the storage medium and clears it for later use. This logic is repeated in a loop.

3. The method for acquiring multi-source data for autonomous driving according to claim 1, characterized in that, The process of retrieving RTSP and / or RTMP streams from forward-looking and / or surround-view cameras and storing them as independent video files includes: Assign a first storage path to the forward-view camera and a second storage path to the surround-view camera. The file names should include timestamps and vehicle IDs.

4. The method for acquiring multi-source data for autonomous driving according to claim 1, characterized in that, Also includes: The data parsing steps include: aligning MPU, ADECU, F9K, CAN and / or CAN FD data according to the acquisition timestamp, and reconstructing the timing logic of vehicle control commands and sensor responses.

5. The method for acquiring multi-source data for autonomous driving according to claim 1, characterized in that, Also includes: The UDP thread management steps include: allocating an independent memory pool for each thread and dynamically adjusting the UDP socket buffer size based on real-time network throughput.

6. An autonomous driving multi-source data acquisition system, characterized in that, include: The multi-protocol acquisition module creates several independent threads to acquire MPU, ADECU, and / or F9K data respectively and write them to the first buffer. Each thread is bound to a unique port number; it also acquires CAN and / or CAN FD data and writes them to the first buffer. The video processing module pulls RTSP and / or RTMP streams from the forward-looking and / or surround-view cameras and stores them as independent video files; A dual-buffer storage control module controls the first buffer and the second buffer to alternately perform data acquisition and writing operations; Storage medium for storing MPU, ADECU, F9K, CAN, CAN FD data and / or standalone video files.

7. The autonomous driving multi-source data acquisition system according to claim 6, characterized in that, The dual-buffer storage control module includes: The status monitoring submodule monitors the capacity fill rate of the first buffer and the second buffer in real time. The switching logic submodule generates a buffer switching instruction when the capacity fill rate exceeds a preset fill rate threshold. The pointer management submodule resets the read and write pointers of the Buffer and updates the storage address mapping.

8. The autonomous driving multi-source data acquisition system according to claim 6, characterized in that, The video processing module includes: The video splitting unit allocates storage paths according to camera type and embeds timestamps and vehicle IDs into file metadata.

9. The autonomous driving multi-source data acquisition system according to claim 6, characterized in that, Also includes: The data parsing engine aligns MPU, ADECU, F9K, CAN, and / or CAN FD data according to the acquisition timestamp, reconstructing the timing logic of vehicle control commands and sensor responses.

10. The autonomous driving multi-source data acquisition system according to claim 6, characterized in that, Also includes: The UDP thread management unit allocates an independent memory pool for each thread and dynamically adjusts the UDP socket buffer size based on real-time network throughput.

Images