A traffic target detection system based on vehicle-mounted sensor fusion
Through multi-sensor fusion and convolutional neural network processing, the problems of low detection rate of single sensors and easy failure of multiple sensors are solved, high-precision and stable traffic target detection is achieved, and the environmental perception capability of the intelligent driving system is improved.
Patent Information
- Application Number
- CN202210000172.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-03
- Publication Date
- 2025-09-16
- Estimated Expiration
- 2042-01-03
AI Technical Summary
In existing technologies, 3D target detection based on a single sensor in traffic scenarios suffers from low detection rates, reliance on assumptions or insufficient prior information, multi-sensor fusion methods are prone to failure under environmental disturbances, and the sensor data fusion effect is poor, failing to fully utilize their respective advantages.
Using sensors such as RGB cameras, night vision cameras, millimeter-wave radars and lidars, data fusion is performed through the intelligent driving controller. The sensor data is uniformly projected onto a bird's-eye view using a fusion algorithm, and feature extraction and fusion are performed. 3D target detection is performed in combination with a convolutional neural network.
It improves the accuracy and robustness of 3D target detection, enhances the stability and robustness of intelligent driving systems in extreme environments, and achieves more efficient traffic target detection.
Smart Images
Figure CN114283394B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent connected vehicles, and in particular to a traffic target detection system integrating vehicle-mounted sensors. Background Art
[0002] Detecting objects in the surrounding environment of intelligent vehicles is a key research topic. Currently, object detection technology is gradually transitioning from 2D object detection to 3D object detection. This is primarily due to its ability to output information such as the category of objects within the sensing range, as well as their position, length, width, height, and azimuth in three-dimensional space. Furthermore, 3D object detection is more suitable for subsequent tasks such as tracking, prediction, and path planning.
[0003] For the research on 3D target detection in the environment, there are already a large number of sensor technologies available for reference. Image-based 3D target detection mainly uses two-dimensional frame information and target orientation information, combined with prior assumptions, to obtain the three-dimensional frame of the target in the image. 3D target detection based on laser point cloud first segments the target point cloud through clustering, and then regresses the segmented point cloud parts to complete 3D detection. However, due to sensor defects, 3D detection based on a single sensor is difficult to further improve the detection rate through sensor algorithms alone. Target detection methods based on vehicle-mounted sensors have been continuously studied in recent years, but traffic target detection systems in traffic scenarios still have some problems, which are mainly reflected as follows:
[0004] First, 3D object detection based on a single sensor cannot directly and completely obtain the characteristics of traffic targets and usually relies on certain assumptions or prior information. For example, although a single-camera image contains a large amount of semantic information, the detection effect is poor in extreme environments and there is no depth information. LiDAR can form a point cloud in three-dimensional space through laser ranging, but the point cloud is relatively sparse and prone to false detection or missed detection due to the small number of target reflection points. Millimeter wave is even sparser, and target-level point cloud data is usually unable to accurately determine the category and 3D frame of the detected target.
[0005] Second, multi-sensor back-end fusion (target-level fusion) relies on the target detection algorithms of individual sensors, failing to fully leverage the strengths of each sensor and resulting in significant semantic loss. Back-end data fusion methods must consider the fusion of 3D target feature data from each sensor, including data association, state updates, and shape fusion. Traditional point or elliptical target models are not suitable for 3D box target model data fusion, and current research on 3D box fusion is limited.
[0006] Third, currently, the most effective methods for multi-sensor front-end fusion (pixel-level and feature-level fusion) are all based on neural network methods. Most of these methods rely on tight coupling between sensor data. If a sensor fails or the data is significantly different, it can directly lead to detection system failure and poor resistance to environmental disturbances.
[0007] The environmental perception system is a key component of intelligent vehicles, providing a wealth of environmental information and surrounding information to enable the smooth completion of autonomous driving tasks. Research on environmental perception systems based on multi-sensor fusion has important practical application value. Summary of the Invention
[0008] The purpose of the present invention is to provide a traffic target detection system with vehicle-mounted sensor fusion in order to overcome the defects of the above-mentioned prior art.
[0009] The technical solution adopted by the present invention is: a traffic target detection system with vehicle-mounted sensor fusion, including sensors and an intelligent driving controller. The sensors include an RGB camera, a night vision camera, a millimeter-wave radar, and a lidar. The intelligent driving controller includes three modules: a fusion algorithm for outputting 3D target information, motion planning, and motion control. The sensors input raw data to the intelligent driving controller via a data line. After the intelligent driving controller's fusion algorithm passes through, the control-by-wire chassis is controlled via CAN signals to complete the autonomous driving task.
[0010] The fusion algorithm includes an image processing module, a point cloud processing module, a feature fusion module and a detection head module. The RGB image collected by the RGB camera and the night vision image collected by the night vision camera are respectively input into the image processing modules corresponding to the two sensor modalities, and the target-level point cloud of the millimeter-wave radar and the laser point cloud of the lidar are respectively input into the corresponding point cloud processing modules; the image processing module and the point cloud processing module input the image and point cloud information into the feature fusion module, which then transmits the information to the detection head module, and finally the detection head module outputs the 3D target.
[0011] Furthermore, the features of each point in the target-level point cloud of the millimeter-wave radar include at least a transverse coordinate, a longitudinal coordinate, a transverse velocity, and a longitudinal velocity in the millimeter-wave radar coordinate system.
[0012] Furthermore, the features of each point in the laser point cloud of the laser radar include at least the X-axis coordinate, Y-axis coordinate, Z-axis coordinate and reflectivity in the millimeter wave radar coordinate system.
[0013] Furthermore, the intelligent driving controller integrates the Linux system internally, and the communication architecture adopts ROS or LCM to complete information interaction between software modules.
[0014] Furthermore, the fusion algorithm pre-processes the raw sensor data and inputs it into the neural network of the fusion algorithm; the neural network outputs 3D target information within the perception area, mainly including center coordinates (x, y, z), external dimensions (width w, length l, height h) and orientation (θ).
[0015] Furthermore, the neural network consists of three parts: data encoding module, feature fusion module and 3D object detection module.
[0016] Furthermore, the feature fusion module stacks the weighted sensor data features together along the ground normal direction, thereby converting them into a pseudo image feature map with a fixed length, width and a fixed number of channels; the feature fusion module inputs the pseudo image feature map into a pre-trained convolutional neural network model framework, and uses upsampling convolution on the output features of the last three modules of the convolutional neural network model to obtain features of the same size; the features of the same size are stacked along the channel direction, i.e., the first dimension, to form a fused feature.
[0017] Furthermore, the 3D target detection module includes a sharing layer, a classification layer and a regression layer; after the fusion features pass through the sharing layer, they pass through the classification layer and the regression layer respectively to classify the category of the corresponding preset anchor frame and regress the 3D frame parameters, wherein the 3D frame parameters include the position deviation value, shape deviation value and heading angle difference value with the corresponding preset anchor frame.
[0018] The image processing module encodes the input three-channel image through a pre-trained backbone network. The generated feature map is passed through two upsampling modules to obtain a feature map and a depth prediction map, respectively. The feature map represents the feature value at each pixel position in the feature map; the depth prediction map represents the depth value at each pixel position in the feature map. The feature value and depth value at each pixel position in the feature map can be mapped to a three-dimensional space to form a spatial feature map in the shape of a viewing cone. The spatial feature map is sampled to generate a voxel feature map of a fixed size. Each column of the voxel feature map is encoded separately, and the maximum value is taken in the vertical direction to finally generate the bird's-eye view features of the image. On the other hand, the feature map of the backbone network is passed through a weighted regression network to obtain the image data weight W.
[0019] The point cloud processing module first distributes each point in the point cloud to the columns of the horizontal road grid. A multi-layer perceptron extracts features from all points in each column, maximizing the numerical channel. This generates a fixed-length feature vector for each column. These feature vectors are then remapped to the horizontal road grid to generate laser bird's-eye view features. These features are then passed through the encoding and decoding module to generate point cloud bird's-eye view features. Another branch of the encoding and decoding module is passed through a weighted regression module to obtain point cloud data weights.
[0020] Compared with the prior art, the present invention has the following advantages:
[0021] The present invention solves some of the current problems of heterogeneous multi-sensor fusion by uniformly projecting the data of each vehicle-mounted sensor onto a bird's-eye view of the horizontal ground through data processing, thereby converting the data of different sensors into a unified expression and into a data format that can be processed by convolutional neural networks.
[0022] The present invention introduces the weight of sensor data into the fusion algorithm, which can highlight the importance of each sensor data under different extreme weather conditions, thereby improving the robustness and stability of the environmental perception module of the intelligent driving system.
[0023] Based on the laser radar point cloud data, the present invention integrates the millimeter wave radar point cloud data, and the 3D target detection accuracy is significantly improved; after further integration of image data, the 3D target detection achieves the best performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 This is an architecture diagram of a traffic target detection system based on vehicle-mounted sensor fusion.
[0025] Figure 2 This is the architecture diagram of the perception algorithm based on multi-sensor fusion.
[0026] Figure 3 This is the network structure diagram of the image processing module.
[0027] Figure 4 This is the network structure diagram of the point cloud processing module.
[0028] Figure 5 This is the network structure diagram of the feature fusion and target detection module.
[0029] Figure 6 The graph shows the accuracy change of the training dataset, validation dataset, and test dataset with the number of iterations. DETAILED DESCRIPTION
[0030] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the embodiments described are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.
[0031] like Figure 1As shown in the figure, the present invention provides a traffic target detection system using on-board sensor fusion. First, the modules of the intelligent driving system are arranged in the intelligent driving controller. Sensors such as RGB cameras, night vision cameras, millimeter-wave radars, and lidars are connected to the corresponding ports of the intelligent driving controller via data cables. The CAN port of the intelligent driving controller is interconnected with the CAN line of the wire-controlled chassis. Each module within the intelligent driving controller adopts the ROS, ZCM, or LCM communication architecture, so that the input and output signals of each algorithm can be transmitted to each other.
[0032] The fusion algorithm belongs to the intelligent driving environment perception system, which is used to process the input multi-sensor raw data and obtain the 3D information of the surrounding environment targets of the intelligent driving car, such as Figure 2 As shown in the figure, the image processing module and point cloud processing module in the fusion algorithm each process the raw signals from their corresponding sensors, extracting features from the heterogeneous sensor data and converting them into a uniformly sized bird's-eye view feature map. All bird's-eye view feature maps are stacked channel-wise and then pass through the feature fusion module to extract higher-dimensional features. These features are then input into the detection head module for target classification and 3D information regression.
[0033] The image processing module in the fusion algorithm mainly processes the three-channel image features, converts the features into three-dimensional space, and then generates a bird's-eye view feature map, such as Figure 3 As shown in the figure, a branch is also used to obtain the weight of the current sensor data, which reflects its importance in the later fusion process. The backbone network in the image processing module is a pre-trained network, typically using the first five blocks of VGG16 or networks such as Resnet101. Because pre-trained networks can effectively extract semantic information from images, they can accelerate network convergence in the later stages of the fusion algorithm, thereby obtaining more accurate 3D object detection results. Since infrared images are single-channel grayscale images, they need to be copied along the channel direction to form three-channel images before being imported into the fusion algorithm.
[0034] The point cloud processing module in the fusion algorithm also needs to convert the original point cloud information into a bird's-eye view feature map and generate the weight of the current point cloud data, such as Figure 4 To further extract high-dimensional semantic information from the global point cloud, the laser bird's-eye view features are fed into the encoding / decoding network module. This outputs the point cloud bird's-eye view features, so that the features of each pixel in the feature map contain information about the global point cloud. The encoding / decoding network module also uses the weighted regression module to further derive the weights of the current point cloud.
[0035] like Figure 5As shown in the figure, the feature fusion module further fuses the bird's-eye view feature maps from each sensor and extracts semantic information at different scales. This extracted semantic information is then stacked and used in the 3D object detection module. The detection module processes this stacked semantic information through a convolutional layer, followed by two convolutional layers to classify the object category corresponding to each anchor box and regress the deviation value of each anchor box's 3D information.
[0036] Through back propagation of the loss function of categories and deviation values, end-to-end training is formed to further correct the network weights, thereby completing target detection and 3D information estimation of traffic targets during the application process.
[0037] like Figure 6 As shown in the figure, during training, the accuracy of the training set gradually improved, but the accuracy of the validation and test sets first increased and then decreased. This indicates that the network initially learned gradually and was able to perform well in the 3D object detection task, but then overfitting occurred. Following conventional practice, the accuracy of each training stage was calculated. Experimental results show that compared to 3D object detection based on single-modal lidar, the proposed method significantly improves overall detection accuracy after integrating millimeter-wave radar information. Further integration of image information achieves the best accuracy, reaching 33%.
[0038] The method of the present invention addresses some of the current issues with heterogeneous multi-sensor fusion. By uniformly projecting the data from each onboard sensor onto a bird's-eye view of the horizontal ground through data processing, the data from different sensors is converted into a unified representation and a data format that can be processed by convolutional neural networks. Furthermore, by incorporating sensor data weights into the fusion algorithm, the method can highlight the importance of each sensor's data in different extreme weather conditions, thereby improving the robustness and stability of the intelligent driving system's environmental perception module. This method offers advantages such as strong practicality and reliability.
[0039] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.
Claims
1. A traffic target detection system based on vehicle-mounted sensor fusion, comprising a sensor and an intelligent driving controller, characterized in that: The sensors include an RGB camera, a night vision camera, a millimeter-wave radar, and a lidar. The intelligent driving controller includes three modules: a fusion algorithm for outputting 3D target information, motion planning, and motion control. The sensors input raw data into the intelligent driving controller via a data cable. After the intelligent driving controller's fusion algorithm passes through it, it controls the controlled-by-wire chassis via CAN signals and completes the autonomous driving task. The fusion algorithm includes an image processing module, a point cloud processing module, a feature fusion module and a detection head module; the RGB image collected by the RGB camera and the infrared image collected by the night vision camera are respectively input into the image processing modules corresponding to the two sensor modalities. Since the infrared image is a single-channel grayscale image, the single-channel grayscale image needs to be copied along the channel direction to become a three-channel image before importing the fusion algorithm; the target-level point cloud of the millimeter-wave radar and the laser point cloud of the lidar are respectively input into the corresponding point cloud processing modules; the image processing module and the point cloud processing module input the image and point cloud information into the feature fusion module, and fuse them according to the fusion weight, and then transmit them to the detection head module, and finally the detection head module outputs the 3D target; The image processing module encodes the input three-channel image through the pre-trained backbone network, and the generated feature map is passed through two upsampling modules to obtain the feature map and depth prediction map respectively; The feature map represents the feature value at each pixel position of the feature map; the depth prediction map represents the depth value at each pixel position of the feature map; the feature map can be mapped to a three-dimensional space to form a spatial feature map in the shape of a cone through the feature value and its depth value at each pixel position of the feature map; the spatial feature map is sampled to generate a voxel feature map of a fixed size; each column of the voxel feature map is encoded separately, and the maximum value is taken in the vertical direction, and finally the image bird's-eye view feature is generated; on the other hand, the feature map of the backbone network is passed through a weighted regression network to obtain the image data weight W; The point cloud processing module first distributes each point in the point cloud to the body columns of the horizontal road grid; all points in each body column are subjected to feature extraction through a multi-layer perceptron, and the maximum value is obtained in the direction of the number channel, so that each body column obtains a feature vector of fixed length; The feature vector is remapped to the horizontal road surface grid to obtain a bird's-eye view feature; the bird's-eye view feature is passed through an encoding and decoding module to generate a point cloud bird's-eye view feature; On the other hand, the other branch of the encoding and decoding module passes through the weight regression module to obtain the point cloud data weight.
2. The vehicle-mounted sensor fusion traffic target detection system according to claim 1, characterized in that: The features of each point in the target-level point cloud of the millimeter-wave radar include at least a transverse coordinate, a longitudinal coordinate, a transverse velocity, and a longitudinal velocity in the millimeter-wave radar coordinate system.
3. The vehicle-mounted sensor fusion traffic target detection system according to claim 1, characterized in that: The features of each point in the laser point cloud of the laser radar include at least the X-axis coordinate, Y-axis coordinate, Z-axis coordinate and reflectivity in the millimeter wave radar coordinate system.
4. The vehicle-mounted sensor fusion traffic target detection system according to claim 1, characterized in that: The intelligent driving controller integrates the Linux system internally, and the communication architecture adopts ROS or LCM to complete information interaction between software modules.
5. The vehicle-mounted sensor fusion traffic target detection system according to claim 1, characterized in that: The fusion algorithm pre-processes the raw sensor data and inputs it into the neural network of the fusion algorithm; the neural network outputs 3D target information within the perception area, mainly including center coordinates (x, y, z), external dimensions (width w, length l, height h) and orientation (θ).
6. The vehicle-mounted sensor fusion traffic target detection system according to claim 5, characterized in that: The neural network consists of three parts: data encoding module, feature fusion module and 3D target detection module.
7. The vehicle-mounted sensor fusion traffic target detection system according to claim 6, characterized in that: The feature fusion module stacks the weighted sensor data features together along the ground normal direction, thereby converting them into a pseudo image feature map with fixed length, width and fixed number of channels; the feature fusion module inputs the pseudo image feature map into the pre-trained convolutional neural network model framework, and uses upsampling convolution on the output features of the last three modules of the convolutional neural network model to obtain features of the same size; the features of the same size are stacked along the first dimension in the channel direction to form fused features.
8. The vehicle-mounted sensor fusion traffic target detection system according to claim 7, characterized in that: The 3D object detection module includes a sharing layer, a classification layer and a regression layer; After the fusion features pass through the shared layer, they pass through the classification layer and regression layer respectively to classify the category of the corresponding preset anchor frame and regress the 3D frame parameters, where the 3D frame parameters include the position deviation value, shape deviation value and heading angle difference value with the corresponding preset anchor frame.