A three-dimensional target detection method based on vehicle-mounted chip

Abstract

The present invention specifically discloses a three-dimensional target detection method based on an on-board chip, including: performing point cloud processing on the real vehicle radar data collected by the 4D millimeter wave radar to form a point cloud data set; constructing a three-dimensional target detection neural network model, and training it through the point cloud data set; compressing the trained three-dimensional target detection neural network model, and converting it into a binary model file supported by the on-board chip, and storing it in the on-board chip; putting the constructed three-dimensional target model reasoning executable file into the on-board chip for operation, and reading the binary model file when performing three-dimensional target detection to perform target three-dimensional reasoning. The present invention can improve the stability and real-time performance of the on-board chip operation and increase the accuracy of three-dimensional target detection.

Description

Technical Field

[0001] The present invention relates to the technical field of in-vehicle entertainment equipment, and more specifically, to a three-dimensional target detection method for an in-vehicle chip. Background Art

[0002] With the rapid development of autonomous driving technology, advanced vehicle perception systems have become the key to improving driving safety and driving experience. Among them, three-dimensional target detection plays a vital role in obstacle identification and obstacle avoidance strategy formulation in complex road environments. 4D millimeter-wave radar, as an emerging sensor, can not only provide the three-dimensional coordinates of the target object, but also measure the relative speed of the target through the Doppler effect, providing the vehicle with a more comprehensive and accurate environmental perception capability. However, three-dimensional target detection relies on high-performance GPUs for complex image processing and data analysis. Limited by the computing resources and power consumption requirements of on-board chips, it often leads to increased processing delays, affecting the real-time and stability of the system. Therefore, how to achieve efficient and accurate three-dimensional target detection with limited resources has become a hot topic and challenge in current research. Summary of the invention

[0003] The present invention provides a three-dimensional target detection method based on an on-board chip, which solves the problem that the three-dimensional target detection of obstacles by the existing on-board advanced perception system is limited by the computing resources and power consumption requirements of the on-board chip, which easily affects the real-time performance and stability of the system. The method can improve the stability and real-time performance of the on-board chip operation and increase the accuracy of three-dimensional target detection.

[0004] To achieve the above objectives, the present invention provides the following technical solutions:

[0005] A three-dimensional target detection method based on a vehicle-mounted chip, comprising:

[0006] Perform point cloud processing on the real vehicle radar data collected by the 4D millimeter wave radar to form a point cloud data set;

[0007] Constructing a 3D object detection neural network model and training it using the point cloud dataset;

[0008] Compressing the trained three-dimensional object detection neural network model and converting it into a binary model file supported by the vehicle chip, and storing it in the vehicle chip;

[0009] The constructed executable file for three-dimensional target model reasoning is placed in the vehicle-mounted chip for running, and the binary model file is read when performing three-dimensional target detection to perform three-dimensional target reasoning.

[0010] Preferably, compressing the trained three-dimensional object detection neural network model and converting it into a binary model file supported by the vehicle-mounted chip and storing it in the vehicle-mounted chip includes:

[0011] Use pruning algorithms and pattern quantization to compress and optimize the 3D object detection neural network model to reduce the model's computational workload and storage requirements;

[0012] Adapt and optimize the model according to the hardware characteristics and performance requirements of the vehicle chip;

[0013] Export the trained model to the format set by the vehicle chip, and then compile it into a specific binary model file using the AI ​​tool chain or optimization framework specific to the vehicle chip;

[0014] Save the binary model file into the appropriate directory of the onboard chip.

[0015] Preferably, the constructed three-dimensional target model reasoning executable file includes:

[0016] Set up dependent libraries on the vehicle chip;

[0017] Create an inference engine corresponding to the vehicle chip;

[0018] Load weights and create execution context;

[0019] According to the input and output binding shapes of the engine, calculate the required buffer size, use the corresponding API of the on-board chip to allocate device memory for the input and output buffers, and create a list of input and output bindings for subsequent inference operations.

[0020] Preferably, the performing three-dimensional reasoning of the target includes:

[0021] Copy input data from host memory to input buffer in device memory;

[0022] Perform reasoning operations using execution context and binding lists;

[0023] Copies the output result from device memory back to the output buffer in host memory;

[0024] Perform necessary post-processing operations on the output results to calculate the semantic information with the maximum probability of each grid in the three-dimensional space;

[0025] The three-dimensional space grid information is displayed in a graphical interface, and each category is given a different color.

[0026] Preferably, the inference engine is a TensorRT engine.

[0027] Preferably, the dependent libraries include: opencv and PCL.

[0028] Preferably, the construction of a three-dimensional target detection neural network model includes:

[0029] An occupancy grid network is used, and an attention mechanism is set to extract features. The voxelized point cloud features are compressed into 2D features through convolution, and up and down sampling of different dimensions are performed. Then, the features are placed in the attention module to effectively extract the features of key parts.

[0030] Preferably, the training using the point cloud data set includes:

[0031] Divide the training set, validation set, and test set by random sampling;

[0032] Generate more training samples through data augmentation.

[0033] Preferably, the data enhancement includes:

[0034] Simulation data augmentation by simulating different weather conditions, lighting conditions, and road conditions;

[0035] Use existing real data to enhance the real data through rotation, scaling, and translation.

[0036] The present invention provides a three-dimensional target detection method based on an on-board chip. The method compresses a trained three-dimensional target detection neural grid model and converts it into a binary model file supported by the on-board chip, and constructs an executable file for three-dimensional target model reasoning to perform target three-dimensional reasoning. The method solves the problem that the three-dimensional target detection of obstacles by the existing on-board advanced perception system is limited by the computing resources and power consumption requirements of the on-board chip, which easily affects the real-time and stability of the system. The method can improve the stability and real-time performance of the on-board chip operation and increase the accuracy of three-dimensional target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the specific embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below.

[0038] Figure 1 It is a schematic diagram of a three-dimensional target detection method based on a vehicle-mounted chip provided by the present invention. DETAILED DESCRIPTION

[0039] In order to enable persons skilled in the art to better understand the solutions of the embodiments of the present invention, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and implementation modes.

[0040] In view of the problem that the three-dimensional target detection of obstacles by the current on-board advanced perception system is limited by the computing resources and power consumption requirements of the on-board chip, which easily affects the real-time and stability of the system, the present invention provides a three-dimensional target detection method based on the on-board chip, which solves the problem that the three-dimensional target detection of obstacles by the existing on-board advanced perception system is limited by the computing resources and power consumption requirements of the on-board chip, which easily affects the real-time and stability of the system. The method can improve the stability and real-time performance of the on-board chip operation and increase the accuracy of three-dimensional target detection.

[0041] like Figure 1 As shown, a three-dimensional target detection method based on a vehicle-mounted chip includes:

[0042] S1: Perform point cloud processing on the real vehicle radar data collected by the 4D millimeter wave radar to form a point cloud data set.

[0043] S2: Build a 3D target detection neural network model and train it using the point cloud dataset.

[0044] S3: compress the trained three-dimensional object detection neural network model, convert it into a binary model file supported by the vehicle chip, and store it in the vehicle chip.

[0045] S4: Put the constructed executable file of three-dimensional target model reasoning into the vehicle-mounted chip for running, and read the binary model file when performing three-dimensional target detection to perform three-dimensional target reasoning.

[0046] Specifically, the vehicle-mounted chip equipped with the target detection system and 4D millimeter wave sensor is integrated into the vehicle system to ensure that the chip can communicate and work together with other parts of the system. The integrated system is tested and verified on the road to ensure that the performance of the target detection system meets the requirements of the vehicle system. This method can improve the detection ability of small targets and occluded targets on the road, provide accurate information input for subsequent autonomous driving road planning, improve the stability and real-time performance of the vehicle-mounted chip operation, and increase the accuracy of three-dimensional target detection.

[0047] Further, compressing the trained three-dimensional target detection neural network model and converting it into a binary model file supported by the vehicle chip and storing it in the vehicle chip includes:

[0048] Use pruning algorithms and pattern quantization to compress and optimize the 3D object detection neural network model to reduce the model's computational workload and storage requirements;

[0049] Adapt and optimize the model according to the hardware characteristics and performance requirements of the vehicle chip;

[0050] Export the trained model to the format set by the vehicle chip, and then compile it into a specific binary model file using the AI ​​tool chain or optimization framework specific to the vehicle chip;

[0051] Save the binary model file into the appropriate directory of the onboard chip.

[0052] Specifically, after the model training is completed, the trained model needs to be compressed and converted into a binary file supported by the vehicle chip, and pushed to the chip. The inference code can be written in C++ language to implement inference. The specific details are as follows:

[0053] (1) Model compression and optimization: Use the L1 norm pruning algorithm and int8 mode quantization to reduce the computational workload and storage requirements of the model.

[0054] (2) Model format conversion: export the trained deep learning model (pth) to onnx, and then use the AI ​​tool chain or optimization framework specific to the vehicle chip to compile it into a specific binary file (such as using the TensorRT tool to convert the model to engine format).

[0055] (3) Model adaptation: Adapt and optimize the model according to the hardware characteristics and performance requirements of the vehicle-mounted chip. This may include adjusting the input and output formats of the model, optimizing the model's memory usage and computing efficiency, etc. Common vehicle-mounted chips include CPU, GPU, FPGA, ASIC, and SoC with AI computing capabilities. Ensure that the adapted model can efficiently call the chip's AI computing unit.

[0056] (4) The model file is saved in the appropriate directory of the chip to ensure that it can be read by the next step of inference code.

[0057] Furthermore, the constructed three-dimensional target model reasoning executable file includes:

[0058] Set up dependent libraries on the vehicle chip;

[0059] Create an inference engine corresponding to the vehicle chip;

[0060] Load weights and create execution context;

[0061] According to the input and output binding shapes of the engine, calculate the required buffer size, use the corresponding API of the on-board chip to allocate device memory for the input and output buffers, and create a list of input and output bindings for subsequent inference operations.

[0062] Further, the target three-dimensional reasoning includes:

[0063] Copy input data from host memory to input buffer in device memory;

[0064] Perform reasoning operations using execution context and binding lists;

[0065] Copies the output result from device memory back to the output buffer in host memory;

[0066] Perform necessary post-processing operations on the output results to calculate the semantic information with the maximum probability of each grid in the three-dimensional space;

[0067] The three-dimensional space grid information is displayed in a graphical interface, and each category is given a different color.

[0068] Furthermore, the inference engine is a TensorRT engine.

[0069] Furthermore, the dependent libraries include: opencv and PCL.

[0070] In actual applications, the required reasoning code can be developed in C++ language, including point cloud data reading, point cloud voxelization, densification, model initialization, cache creation, model reasoning, result post-processing, visualization and other modules. Use a cross-compiler to compile the code into an executable file under the ARM architecture and put it into the vehicle chip for running.

[0071] The details are as follows:

[0072] (1) Preparation of dependent libraries:

[0073] Make sure that the necessary dependent libraries for model reasoning, such as opencv, PCL, etc., have been installed on the vehicle chip.

[0074] (2) Building a chip inference engine:

[0075] Create an inference engine corresponding to the chip, such as Nvidia's TensorRT engine.

[0076] (3) Load weights and create execution context:

[0077] Load weights from the model binary file into the engine. Use the loaded engine to create an execution context for subsequent inference operations.

[0078] (4) Allocate input and output buffers:

[0079] Calculate the required buffer size based on the engine's input and output binding shapes. Use the chip's corresponding API to allocate device memory for the input and output buffers. Create a list of input and output bindings for subsequent inference operations.

[0080] (5) Reasoning operation:

[0081] Prepare input data: Copy input data from host memory to input buffer in device memory.

[0082] Perform inference: Perform inference operations using the execution context and binding list.

[0083] Get output results: Copy the output results from device memory back to the output buffer in host memory.

[0084] (6) Post-processing:

[0085] Perform necessary post-processing operations on the output results to calculate the semantic information with the maximum probability of each grid in the three-dimensional space, such as cars, buses, pedestrians, non-motor vehicles, etc.

[0086] (7) Visualization:

[0087] The three-dimensional space grid information is displayed in a graphical interface, and each category is given a different color.

[0088] Further, the construction of a three-dimensional target detection neural network model includes:

[0089] An occupancy grid network is used, and an attention mechanism is set to extract features. The voxelized point cloud features are compressed into 2D features through convolution, and up and down sampling of different dimensions are performed. Then, the features are placed in the attention module to effectively extract the features of key parts.

[0090] In practical applications, the occupancy grid network used has shown significant advantages in 3D object detection. These advantages are mainly reflected in the following aspects: The occupancy network achieves dense perception of 3D space by dividing the world into grid cells and defining which cells are occupied and which are free. The occupancy network can predict the shape and position of occluded objects based on the surrounding semantic information. This ability makes the occupancy network more superior in 3D object detection in complex scenes.

[0091] Further, the training using the point cloud data set includes:

[0092] Divide the training set, validation set, and test set by random sampling;

[0093] Generate more training samples through data augmentation.

[0094] Further, the data enhancement includes:

[0095] Simulation data augmentation by simulating different weather conditions, lighting conditions, and road conditions;

[0096] Use existing real data to enhance the real data through rotation, scaling, and translation.

[0097] In practical applications, data collection should include a variety of traffic scenarios, such as highways, urban roads, rural roads, etc., and should cover different traffic conditions, weather conditions, road types, etc. During the collection of millimeter-wave radar data, pay attention to the operating status of the monitoring equipment and the quality of data collection. Save the collected millimeter-wave radar data to the storage device. Perform data processing on the collected radar data. Data processing includes: Noise removal: The data collected by the millimeter-wave radar may contain various noises, such as environmental noise, equipment noise, etc., which need to be removed by filtering and other methods. Filling missing values: For the missing parts in the data, interpolation or other methods need to be used to fill them to ensure the integrity of the data.

[0098] After preparing the data set for model training, you can start training the model. Before training the model, you need to design the model's loss function, optimizer, learning rate update strategy, data batch size, number of iterations, etc.

[0099] The specific details of occupancy model training are described as follows:

[0100] The loss function is a weighted combination of Cross-Entropy Loss and Lovasz loss:

[0101] Cross-Entropy Loss,In the model, the task is to classify each location in the scene,as either an occupied class (road surface, vehicle, pedestrian, non-motor vehicle, building, etc.) or,unoccupied.

[0102] Lovasz loss is a loss function designed for semantic segmentation tasks. It combines Lovasz expansion and Softmax loss to better handle multi-category segmentation tasks.

[0103] The optimizer selects Adam, the learning rate update strategy selects the cosine annealing strategy, and the initial learning rate is set to 0.001.

[0104] The training equipment can be NVIDIA A800 server, the training card is set to 8 cards, the data batch size is set to 12G, and the number of iterations is set to 80 rounds. Start training on the point cloud data collected in 3 and output the weight model pth file. In the subsequent training process, the performance of the model can be continuously optimized by continuously adjusting the model structure and hyperparameters.

[0105] It can be seen that the present invention provides a three-dimensional target detection method based on an on-board chip, which compresses the trained three-dimensional target detection neural grid model and converts it into a binary model file supported by the on-board chip, and constructs an executable file for three-dimensional target model reasoning to perform target three-dimensional reasoning. This solves the problem that the three-dimensional target detection of obstacles by the existing on-board advanced perception system is limited by the computing resources and power consumption requirements of the on-board chip, which easily affects the real-time and stability of the system. It can improve the stability and real-time performance of the on-board chip operation and increase the accuracy of three-dimensional target detection.

[0106] The above describes in detail the structure, features and effects of the present invention based on the embodiments shown in the drawings. The above is only a preferred embodiment of the present invention, but the present invention is not limited to the scope of implementation shown in the drawings. Any changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments with equivalent changes, which still do not exceed the spirit covered by the description and drawings, should be within the protection scope of the present invention.

Claims

1. A three-dimensional target detection method based on an on-board chip, characterized in that: include: Perform point cloud processing on the real vehicle radar data collected by the 4D millimeter wave radar to form a point cloud data set; Constructing a 3D object detection neural network model and training it using the point cloud dataset; Compressing the trained three-dimensional object detection neural network model and converting it into a binary model file supported by the vehicle chip, and storing it in the vehicle chip; The constructed executable file for three-dimensional target model reasoning is placed in the vehicle-mounted chip for running, and the binary model file is read when performing three-dimensional target detection to perform three-dimensional target reasoning.

2. The three-dimensional target detection method based on the vehicle-mounted chip according to claim 1 is characterized in that: The step of compressing the trained three-dimensional target detection neural network model and converting it into a binary model file supported by the vehicle-mounted chip and storing it in the vehicle-mounted chip includes: Use pruning algorithms and pattern quantization to compress and optimize the 3D object detection neural network model to reduce the model's computational workload and storage requirements; Adapt and optimize the model according to the hardware characteristics and performance requirements of the vehicle chip; Export the trained model to the format set by the vehicle chip, and then compile it into a specific binary model file using the AI ​​tool chain or optimization framework specific to the vehicle chip; Save the binary model file into the appropriate directory of the onboard chip.

3. The three-dimensional target detection method based on the vehicle-mounted chip according to claim 2 is characterized in that: The constructed three-dimensional target model reasoning executable file includes: Set up dependent libraries on the vehicle chip; Create an inference engine corresponding to the vehicle chip; Load weights and create execution context; According to the input and output binding shapes of the engine, calculate the required buffer size, use the corresponding API of the on-board chip to allocate device memory for the input and output buffers, and create a list of input and output bindings for subsequent inference operations.

4. The three-dimensional target detection method based on the vehicle-mounted chip according to claim 3 is characterized in that: The performing of target three-dimensional reasoning includes: Copy input data from host memory to input buffer in device memory; Perform inference operations using execution context and binding list; Copies the output result from device memory back to the output buffer in host memory; Perform necessary post-processing operations on the output results to calculate the semantic information with the maximum probability of each grid in the three-dimensional space; The three-dimensional space grid information is displayed in a graphical interface, and each category is given a different color.

5. The three-dimensional target detection method based on vehicle-mounted chip according to claim 4 is characterized in that: The inference engine is the TensorRT engine.

6. The three-dimensional target detection method based on the vehicle-mounted chip according to claim 5 is characterized in that: The dependent libraries include: opencv and PCL.

7. The three-dimensional target detection method based on vehicle-mounted chips according to claim 6 is characterized in that: The constructing of a three-dimensional target detection neural network model comprises: An occupancy grid network is used, and an attention mechanism is set to extract features. The voxelized point cloud features are compressed into 2D features through convolution, and up and down sampling of different dimensions are performed. Then, the features are placed in the attention module to effectively extract the features of key parts.

8. The three-dimensional target detection method based on vehicle-mounted chips according to claim 7 is characterized in that: The training using the point cloud data set includes: Divide the training set, validation set, and test set by random sampling; Generate more training samples through data augmentation.

9. The three-dimensional target detection method based on vehicle-mounted chips according to claim 8 is characterized in that: The data enhancement includes: Simulation data augmentation by simulating different weather conditions, lighting conditions, and road conditions; Use existing real data to enhance the real data through rotation, scaling, and translation.

Images