A multi-sensor fusion target detection system and detection method

Abstract

The present invention discloses a multi-sensor fusion target detection system and detection method. The system solves the real-time fusion processing requirements of intelligent driving for local perception of massive multi-source heterogeneous data and external collaborative data. Even if the existing hardware system can meet the real-time requirements, its cost will be more expensive. This system adopts a distributed computing platform, which is more economical while meeting performance requirements. By fusing millimeter-wave radar and visual information, this method solves the limitations of single-camera perception caused by external light source changes and foggy and rainy weather, so that objects can be detected even in extreme weather. At the same time, by fusing lidar information, accurate distance information of objects can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a target detection system and a detection method that are fused with multiple sensors. Background Art

[0002] The autonomous driving vehicle perceives the environment through sensors. The current mainstream sensors include lidar, millimeter wave radar, ultrasonic radar, and cameras. However, each sensor has its own advantages and disadvantages, and the sensors have different adaptability to the environment. The detection of the autonomous driving perception system not only needs to identify the category of the current object, but also measure the relative distance between the obstacle and the autonomous driving vehicle.

[0003] Due to the defects of the sensors themselves, in harsh environments such as fog, rain, and night, the performance of a single type of sensor is limited and it is unable to accurately detect obstacles. Since both LiDAR and millimeter-wave radar have good ranging effects, the information of vehicle detection is fused according to the coordinate transformation obtained by sensor calibration. For vehicle detection problems where the weather affects the environment, multi-sensor fusion can be used to complement the environment perception and improve the environmental adaptability of detection.

[0004] However, most existing multi-sensor fusion systems take up a lot of computing power, and ordinary computing platforms lack computing power and cannot meet real-time requirements. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art and to improve a multi-sensor fusion target detection system and detection method.

[0006] The technical solution for achieving the purpose of the present invention is:

[0007] A multi-sensor fusion target detection system includes a first AGX module, a second AGX module, a third AGX module, an LTE-5G module, a WiFi module, a switch module, a CAN transceiver module, a display module, an image acquisition module, a laser radar, a millimeter-wave radar and a power management module, wherein the first AGX module serves as a host, and the second AGX module and the third AGX module serve as slaves, wherein the laser radar is connected to the second AGX module through the switch module, the millimeter-wave radar and the image acquisition module are connected to the third AGX module through the CAN transceiver module and the switch module respectively, the second AGX module and the third AGX module are also connected to the first AGX module through the switch module, the LTE-5G module and the WiFi module are respectively connected to the first AGX module, the first AGX module is also connected to the display module, and the power management module supplies power to the first AGX module, the second AGX module, the third AGX module, the LTE-5G module, the WiFi module, the switch module, the CAN transceiver module, the display module, the image acquisition module, the laser radar and the millimeter-wave radar.

[0008] The first AGX module, the second AGX module and the third AGX module are data processing modules based on NVIDIA jetson_AGX_Xavier.

[0009] The image acquisition module is a video camera or a camera.

[0010] A multi-sensor fusion target detection method comprises the following steps:

[0011] 1) Fusion of millimeter wave and visual data. The specific steps are as follows:

[0012] 1-1) Calibrate the millimeter wave radar and image acquisition module, calculate the external parameters, so that the target point in the radar coordinate system can be mapped to the pixel coordinate system, and the image of the image acquisition module can show the position of the object detected by the millimeter wave;

[0013] 1-2) The image acquisition module and millimeter wave radar input the collected data into the third AGX module, synchronize the millimeter wave data and the picture data, select the timestamp of each picture as the sampling point, and select the millimeter wave data closest to the timestamp to match it;

[0014] 1-3) After normalizing the lateral velocity, longitudinal velocity, and RCS in the millimeter wave data, convert them into a range of 0 to 255 to generate a millimeter wave image;

[0015] 1-4) Using the method in step 1-3) to simultaneously process the data set containing millimeter wave data and image data to generate a training set;

[0016] 1-5) Establish an improved millimeter wave and image fusion detection model based on yolov4-tiny, including a millimeter wave feature extraction network and an image and millimeter wave fusion detection network, as follows:

[0017] The millimeter wave feature extraction network comprises a first convolution layer, a second convolution layer, a first maximum pooling layer, a third convolution layer, a second maximum pooling layer, a fourth convolution layer, a third maximum pooling layer, a fifth convolution layer, a fourth maximum pooling layer, a sixth convolution layer, a fifth maximum pooling layer and a seventh convolution layer which are sequentially connected, the millimeter wave image generated in step 1-3) is input into the millimeter wave feature extraction network from the first convolution layer, and finally the sixth convolution layer outputs the first feature image, and the seventh convolution layer outputs the second feature image;

[0018] The detection network for fusion of image and millimeter wave comprises Yolov4-tiny backbone network, first spatial attention module, second spatial attention module, eighth convolution layer, ninth convolution layer, tenth convolution layer, eleventh convolution layer, first upsampling layer, twelfth convolution layer, thirteenth convolution layer, fourteenth convolution layer, fifteenth convolution layer, small target detection layer and large target detection layer; the image collected by the image acquisition module is passed through the Yolov4-tiny backbone network to obtain the third feature image and the fourth feature image, one branch of the second feature map output by the millimeter wave feature extraction network is first concat-operated with the fourth feature image to perform feature map splicing, and another branch of the second feature map is passed through the first spatial attention module to enhance the feature weight of the specific area, and then multiplied with the feature map after the concat operation and enters the eighth convolution layer. , after passing through the eighth convolution layer, the ninth convolution layer, and the tenth convolution layer in sequence, a branch of the tenth convolution layer generates the fifth feature image, and another branch of the tenth convolution layer passes through the eleventh convolution layer and the first upsampling in sequence, and then performs a concat operation with the third feature image, and then multiplies it with the first feature map after passing through the second spatial attention module, and then passes through the twelfth convolution layer, the thirteenth convolution layer, and the fourteenth convolution layer in sequence; the output of a branch of the fourteenth convolution layer is concat-operated with the fifth feature image and input into the fifteenth convolution layer, and then passes through the large target detection layer; the output of a branch of the fourteenth convolution layer is sent to the small target detection layer, and the small target detection layer and the large target detection layer respectively output prediction values, and the prediction values ​​are filtered out by non-maximum suppression NMS to generate the final prediction result, which includes prediction box border information, confidence, and category information;

[0019] 1-6) After the training set obtained in step 1-4) is subjected to data enhancement operations such as random brightness change, color change, and size change, it is input into the millimeter wave and image fusion detection model improved based on yolov4-tiny to train the model, and then the real-time collected image data and millimeter wave data are input into the trained model to obtain the prediction box border information, confidence, and category information, and the detection results are transmitted to the first AGX module;

[0020] 2) LiDAR obstacle detection, the specific steps are as follows:

[0021] 2-1) The laser radar transmits the collected information to the second AGX module. The second AGX module receives the information transmitted by the laser radar and performs Euclidean clustering on the information to obtain the coordinates of the target's three-dimensional detection frame in the point cloud coordinate system and its corresponding distance;

[0022] 2-2) The coordinates of the 3D detection frame in the point cloud coordinate system are converted to the pixel coordinate system of the camera through an external calibration method to obtain the detection frame information and distance of the target, and then the detection frame information and distance of the target are transmitted to the first AGX module;

[0023] 3) The first AGX module performs IOU calculation based on the results obtained in step 1-6) and the results of step 2-2). If the IOU of the lidar detection frame and the detection frame output by the camera millimeter wave fusion is greater than a certain threshold, it means that the same object is detected, and the distance, category and orientation of the detected target are output;

[0024] 4) The output result obtained in step 3 is displayed in the display module or transmitted to the host computer through the LTE-5G module or WiFi module.

[0025] In step 1-3), the millimeter wave image is first generated as an image with the same size as the corresponding camera image and all pixel values ​​are zero, with each millimeter wave point as the center of the circle. O i , set a point R i The distance is 1m from the center of the circle in the millimeter wave coordinate system. The calibration method in step 1-1) is used to R i Convert to the pixel coordinate system, calculate the pixel distance between the point in the converted pixel coordinate system and the center of the circle, and draw a circle with this distance as the radius. Fill the pixel values ​​in this range with the lateral velocity, longitudinal velocity, and RCS converted above, and finally obtain the millimeter wave image.

[0026] The present invention provides a multi-sensor fusion target detection system and detection method. Compared with the existing system, the system solves the real-time fusion processing requirements of intelligent driving for local perception of massive multi-source heterogeneous data and external collaborative data. Even if the existing hardware system can meet the real-time requirements, its cost will be more expensive. This system adopts a distributed computing platform, which is more economical while meeting performance requirements. By fusing millimeter-wave radar and visual information, this method solves the limitations of single-camera perception caused by external light source changes and foggy and rainy weather, so that objects can be detected even in extreme weather. At the same time, by fusing lidar information, accurate distance information of objects can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a structural block diagram of a multi-sensor fusion target detection system;

[0028] Figure 2 Schematic diagram of the millimeter wave feature extraction network;

[0029] Figure 3 This is a schematic diagram of the millimeter wave vision fusion detection model based on yolov4-tiny;

[0030] Figure 4 Schematic diagram of millimeter wave image. DETAILED DESCRIPTION

[0031] The present invention is further described below with reference to the accompanying drawings and embodiments, but the present invention is not limited thereto.

[0032] Example:

[0033] like Figure 1 As shown, a multi-sensor fusion target detection system includes a first AGX module, a second AGX module, a third AGX module, an LTE-5G module, a WiFi module, a switch module, a CAN transceiver module, a display module, an image acquisition module, a laser radar, a millimeter-wave radar and a power management module. The first AGX module serves as a host, and the second AGX module and the third AGX module serve as slaves. The image acquisition module and the millimeter-wave radar are connected to the third AGX module through the switch module and the CAN transceiver module respectively, the laser radar is connected to the second AGX module through the switch, and the second AGX module and the third AGX module are also connected to the first AGX module through the switch module, the LTE-5G module and the WiFi module are respectively connected to the first AGX module, and the first AGX module is also connected to the display module, and the power management module supplies power to the first AGX module, the second AGX module, the third AGX module, the LTE-5G module, the WiFi module, the switch module, the CAN transceiver module, the display module, the image acquisition module, the laser radar and the millimeter-wave radar.

[0034] The first AGX module, the second AGX module and the third AGX module are data processing modules based on NVIDIA jetson_AGX_Xavier. They are computing boxes that integrate fast computing, fast startup and fast optimization. They have super computing power and are suitable for hardware devices in the fields of autonomous driving, artificial intelligence and automation.

[0035] The image acquisition module is a video camera or a camera.

[0036] The switch module is an 8-way Gigabit switch module.

[0037] The CAN transceiver module is a 3-way CAN transceiver module.

[0038] When using this system, the devices used for perception are laser radar, millimeter-wave radar, and image acquisition module. The laser radar is connected to the 8-way Gigabit switch module via an Ethernet cable to communicate with the computing box. Its data will be sent to the second AGX module for processing and output the distance information of the object; the millimeter-wave radar is connected via a 3-way CAN transceiver module, and its data is received by the third AGX module. At the same time, the image data obtained by the camera of the image acquisition module is transmitted via Ethernet and is also received by the third AGX module; the millimeter-wave data and image data are fused in the third AGX module to output the object type, position in the image, and distance information of the object.

[0039] A multi-sensor fusion target detection method comprises the following steps:

[0040] 1) Fusion of millimeter wave and visual data. The specific steps are as follows:

[0041] 1-1) Calibrate the millimeter wave radar and image acquisition module, calculate the external parameters, so that the target point in the radar coordinate system can be mapped to the pixel coordinate system, and the image of the image acquisition module can show the position of the object detected by the millimeter wave;

[0042] 1-2) The image acquisition module and millimeter wave radar input the collected data into the third AGX module, synchronize the millimeter wave data and the picture data, select the timestamp of each picture as the sampling point, and select the millimeter wave data closest to the timestamp to match it;

[0043] 1-3) After normalizing the lateral velocity, longitudinal velocity, and RCS in the millimeter wave data, convert them to the range of 0 to 255 to generate a millimeter wave image, such as Figure 4 As shown, the calculation formula for normalizing the lateral velocity, longitudinal velocity, and RCS is as follows:

[0044]

[0045]

[0046]

[0047] The millimeter wave image is first generated as an image with the same size as the corresponding camera image, with all pixel values ​​of 0, with each millimeter wave point as the center of the circle. O i , set a point R i The distance is 1m from the center of the circle in the millimeter wave coordinate system. The calibration method in step 1-1) is used to R i Convert to the pixel coordinate system, calculate the pixel distance between the point in the converted pixel coordinate system and the center of the circle, and draw a circle with this distance as the radius. Fill the pixel values ​​in this range with the lateral velocity, longitudinal velocity, and RCS converted above, and finally obtain the millimeter wave image.

[0048] 1-4) Using the method in step 1-3) to simultaneously process the data set containing millimeter wave data and image data to generate a training set;

[0049] 1-5) Establish an improved millimeter wave and image fusion detection model based on yolov4-tiny, including a millimeter wave feature extraction network and an image and millimeter wave fusion detection network, as follows:

[0050] The millimeter wave feature extraction network, such as Figure 2 As shown, it includes a first convolution layer, a second convolution layer, a first maximum pooling layer, a third convolution layer, a second maximum pooling layer, a fourth convolution layer, a third maximum pooling layer, a fifth convolution layer, a fourth maximum pooling layer, a sixth convolution layer, a fifth maximum pooling layer and a seventh convolution layer connected in sequence, the millimeter wave image generated in step 1-3) is input into the millimeter wave feature extraction network from the first convolution layer, and finally the sixth convolution layer outputs the first feature image, and the seventh convolution layer outputs the second feature image;

[0051] Figure 2 In the figure, the convolution layer (m, n) represents a convolution layer with m convolution kernels and n×n size. The maximum pooling layer is used to adjust the size of the feature map. The processed millimeter wave passes through the first convolution layer, the second convolution layer, the first maximum pooling layer, the third convolution layer, the second maximum pooling layer, the fourth convolution layer, the third maximum pooling layer, the fifth convolution layer, the fourth maximum pooling layer, the sixth convolution layer, the fifth maximum pooling layer, and the seventh convolution layer. The size of the first feature map output is 26×26 and the dimension is 128; the size of the second feature map is 13×13 and the dimension is 128.

[0052] The image and millimeter wave fusion detection network, such as Figure 3 As shown, it includes a Yolov4-tiny backbone network, a first spatial attention module, a second spatial attention module, an eighth convolutional layer, a ninth convolutional layer, a tenth convolutional layer, an eleventh convolutional layer, a first upsampling layer, a twelfth convolutional layer, a thirteenth convolutional layer, a fourteenth convolutional layer, a fifteenth convolutional layer, a small target detection layer and a large target detection layer; the image collected by the image acquisition module passes through the Yolov4-tiny backbone network to obtain a third feature image and a fourth feature image, a branch of the second feature map output by the millimeter wave feature extraction network is first concat-operated with the fourth feature image to perform feature map splicing, and another branch of the second feature map is enhanced by the first spatial attention module for a specific area feature weight, and then multiplied with the feature map after the concat operation, and then enters the eighth convolutional layer, and passes through the eighth convolutional layer in sequence. After the convolution layer, the ninth convolution layer, and the tenth convolution layer, a branch of the tenth convolution layer generates the fifth feature image. Another branch of the tenth convolution layer then passes through the eleventh convolution layer and the first upsampling, and then performs a concat operation with the third feature image, and then multiplies it with the first feature map after the second spatial attention module, and then passes through the twelfth convolution layer, the thirteenth convolution layer, and the fourteenth convolution layer in sequence; the output of a branch of the fourteenth convolution layer is concat-operated with the fifth feature image and input into the fifteenth convolution layer, and then passes through the large target detection layer; the output of a branch of the fourteenth convolution layer is sent to the small target detection layer, and the small target detection layer and the large target detection layer respectively output prediction values. The prediction values ​​are filtered out by non-maximum suppression NMS to generate the final prediction result, which includes the prediction box border information, confidence, and category information;

[0053] The prediction value output by the detection layer is S*S*3*(4+1+B), where S means that the original image is divided into S*S grids, 3 is the number of anchorboxes in each layer, 4 is the border size (x, y, w, h), 1 is the confidence value, and B is the number of types.

[0054] 1-6) After the training set obtained in step 1-4) is subjected to data enhancement operations such as random brightness change, color change, and size change, it is input into the millimeter wave and image fusion detection model improved based on yolov4-tiny to train the model, and then the real-time collected image data and millimeter wave data are input into the trained model to obtain the prediction box border information, confidence, and category information, and the detection results are transmitted to the first AGX module;

[0055] 2) LiDAR obstacle detection, the specific steps are as follows:

[0056] 2-1) The laser radar transmits the collected information to the second AGX module. The second AGX module receives the information transmitted by the laser radar and performs Euclidean clustering on the information to obtain the coordinates of the target's three-dimensional detection frame in the point cloud coordinate system and its corresponding distance;

[0057] 2-2) The coordinates of the 3D detection frame in the point cloud coordinate system are converted to the pixel coordinate system of the camera through an external calibration method to obtain the detection frame information and distance of the target, and then the detection frame information and distance of the target are transmitted to the first AGX module;

[0058] 3) The first AGX module performs IOU calculation based on the results obtained in step 1-6) and the results of step 2-2). If the IOU of the LiDAR detection frame and the detection frame output by the camera millimeter wave fusion is greater than a certain threshold, it means that the same object is detected. Based on the precise distance information detected by the LiDAR, the distance, category and orientation of the detected target are output;

[0059] 4) The output result obtained in step 3 is displayed in the display module or transmitted to the host computer through the LTE-5G module or WiFi module.

Claims

1. A multi-sensor fusion target detection method, characterized in that: A target detection system with multi-sensor fusion is used for target detection. The target detection system includes a first AGX module, a second AGX module, a third AGX module, an LTE-5G module, a WiFi module, a switch module, a CAN transceiver module, a display module, an image acquisition module, a laser radar, a millimeter-wave radar and a power management module. The first AGX module serves as a host, and the second AGX module and the third AGX module serve as slaves. The laser radar is connected to the second AGX module through a switch module, and the millimeter-wave radar and the image acquisition module are connected to the third AGX module through a CAN transceiver module and a switch module, respectively. The second AGX module and the third AGX module are also connected to the first AGX module through a switch module. The LTE-5G module and the WiFi module are respectively connected to the first AGX module, and the first AGX module is also connected to the display module. The power management module supplies power to the first AGX module, the second AGX module, the third AGX module, the LTE-5G module, the WiFi module, the switch module, the CAN transceiver module, the display module, the image acquisition module, the laser radar and the millimeter-wave radar. The method comprises the following steps: 1) Fusion of millimeter wave and visual data. The specific steps are as follows: 1-1) Calibrate the millimeter wave radar and image acquisition module, calculate the external parameters, so that the target point in the radar coordinate system can be mapped to the pixel coordinate system, and the image of the image acquisition module can show the position of the object detected by the millimeter wave; 1-2) The image acquisition module and the millimeter wave radar input the collected data into the third AGX module, synchronize the millimeter wave data and the picture data, select the timestamp of each picture as the sampling point, and select the millimeter wave data closest to the timestamp to match it; 1-3) After normalizing the lateral velocity, longitudinal velocity, and RCS in the millimeter wave data, convert them into a range of 0 to 255 to generate a millimeter wave image; 1-4) using the method of step 1-3) to simultaneously process the data set containing millimeter wave data and image data to generate a training set; 1-5) Establish a millimeter wave and image fusion detection model based on yolov4-tiny, including a millimeter wave feature extraction network and an image and millimeter wave fusion detection network, as follows: The millimeter wave feature extraction network comprises a first convolutional layer, a second convolutional layer, a first maximum pooling layer, a third convolutional layer, a second maximum pooling layer, a fourth convolutional layer, a third maximum pooling layer, a fifth convolutional layer, a fourth maximum pooling layer, a sixth convolutional layer, a fifth maximum pooling layer and a seventh convolutional layer connected in sequence, the millimeter wave image generated in step 1-3) is input into the millimeter wave feature extraction network from the first convolutional layer, and finally the sixth convolutional layer outputs the first feature image, and the seventh convolutional layer outputs the second feature image; The detection network for fusion of image and millimeter wave comprises Yolov4-tiny backbone network, first spatial attention module, second spatial attention module, eighth convolution layer, ninth convolution layer, tenth convolution layer, eleventh convolution layer, first upsampling layer, twelfth convolution layer, thirteenth convolution layer, fourteenth convolution layer, fifteenth convolution layer, small target detection layer and large target detection layer; the image collected by the image acquisition module is passed through the Yolov4-tiny backbone network to obtain the third feature image and the fourth feature image, one branch of the second feature map output by the millimeter wave feature extraction network is first concat-operated with the fourth feature image to perform feature map splicing, and another branch of the second feature map is passed through the first spatial attention module to enhance the feature weight of the specific area, and then multiplied with the feature map after the concat operation and enters the eighth convolution layer. , after passing through the eighth convolution layer, the ninth convolution layer, and the tenth convolution layer in sequence, a branch of the tenth convolution layer generates the fifth feature image, and another branch of the tenth convolution layer passes through the eleventh convolution layer and the first upsampling in sequence, and then performs a concat operation with the third feature image, and then multiplies it with the first feature map after passing through the second spatial attention module, and then passes through the twelfth convolution layer, the thirteenth convolution layer, and the fourteenth convolution layer in sequence; the output of a branch of the fourteenth convolution layer is concat-operated with the fifth feature image and input into the fifteenth convolution layer, and then passes through the large target detection layer; the output of a branch of the fourteenth convolution layer is sent to the small target detection layer, and the small target detection layer and the large target detection layer respectively output prediction values, and the prediction values ​​are filtered out by non-maximum suppression NMS to generate the final prediction result, which includes prediction box border information, confidence, and category information; 1-6) After the training set obtained in step 1-4) is subjected to data enhancement operations such as random brightness change, color change, and size change, it is input into the millimeter wave and image fusion detection model improved based on yolov4-tiny to train the model, and then the real-time collected image data and millimeter wave data are input into the trained model to obtain the prediction box border information, confidence, and category information, and the detection result is transmitted to the first AGX module; 2) LiDAR obstacle detection, the specific steps are as follows: 2-1) The laser radar transmits the collected information to the second AGX module. The second AGX module receives the information transmitted by the laser radar and performs Euclidean clustering on the information to obtain the coordinates of the target's three-dimensional detection frame in the point cloud coordinate system and its corresponding distance; 2-2) The coordinates of the three-dimensional detection frame in the point cloud coordinate system are converted to the pixel coordinate system of the camera through an external calibration method to obtain the detection frame information and distance of the target, and then the detection frame information and distance of the target are transmitted to the first AGX module; 3) The first AGX module performs IOU calculation based on the result obtained in step 1-6) and the result of step 2-2). If the IOU of the detection frame output by the lidar and the camera millimeter wave fusion is greater than a certain threshold, it means that the same object is detected, and the distance, category and orientation of the detected target are output; 4) The output result obtained in step 3) is displayed in the display module or transmitted to the host computer through the LTE-5G module or WiFi module.

2. The target detection method of multi-sensor fusion according to claim 1, characterized in that: In step 1-3), the millimeter wave image is first generated as an image with the same size as the corresponding camera image and all pixel values ​​are 0, with each millimeter wave point as the center of the circle O i , set a point R i The distance is 1m from the center of the circle in the millimeter wave coordinate system. The calibration method of step 1-1) is used to calibrate R i Convert to the pixel coordinate system, calculate the pixel distance between the point in the converted pixel coordinate system and the center of the circle, and draw a circle with this distance as the radius. Fill the pixel values ​​in this range with the lateral velocity, longitudinal velocity, and RCS converted above, and finally obtain the millimeter wave image.

3. The target detection method of multi-sensor fusion according to claim 1, characterized in that: The first AGX module, the second AGX module and the third AGX module are data processing modules based on NVIDIA jetson_AGX_Xavier.

4. The target detection method of multi-sensor fusion according to claim 1, characterized in that: The image acquisition module is a video camera or a camera.

Images