Automatic driving target detection system and optimization method based on multi-sensor fusion in Internet of Vehicles environment

Abstract

The invention discloses an automatic driving target detection system based on multi-sensor fusion in an Internet of Vehicles environment and an optimization method, and belongs to the technical field of automatic driving. Comprising a visual camera target detection module, a laser radar target detection module, a target tracking module, a laser radar and visual camera joint calibration module, a laser point cloud projection-to-image module and a laser radar information and visual information fusion module. According to the automatic driving target detection system based on multi-sensor fusion in the Internet of Vehicles environment and the optimization method, the extraction capability of key features is enhanced, the convergence speed and the positioning precision of visual detection on a target vehicle are improved, the lightweight requirement of a model is ensured, and the loss of a detection result and false detection are avoided.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an autonomous driving target detection system and optimization method based on multi-sensor fusion in a connected vehicle environment. Background Art

[0002] With the continuous deepening of the research on target detection algorithms, traditional autonomous driving target detection algorithms have been able to meet the target detection problems in general traffic scenarios. In complex traffic scenarios, when autonomous driving vehicles are identifying surrounding targets, it is very easy for traffic targets to be blocked, resulting in the inability to fully present the appearance, shape and color features of the blocked traffic targets, which increases the difficulty of the target detection algorithm to identify traffic targets; in addition, in the process of identifying small targets at a long distance, due to the small number of pixels in the image, the features that can be extracted during target detection lack sufficient context information, which makes it difficult for the target detection model to use other information in the scene to assist detection, and the detection accuracy is low. At present, the perception systems of autonomous driving vehicles mostly use two sensors, lidar and visual cameras, both of which have their own advantages and disadvantages. Visual camera sensors can obtain rich visual information of the surrounding environment and traffic targets, but due to the lack of depth information, they cannot obtain accurate three-dimensional spatial information of objects and are easily disturbed by the external environment; while lidar has good robustness and can accurately obtain three-dimensional spatial information of objects, but it cannot obtain information such as color, texture and shape of traffic targets, and its recognition ability is poor. Therefore, it is difficult for a single sensor to meet the requirements of accuracy and robustness of autonomous driving target detection. The present invention proposes to fuse the perception information of laser radar and visual camera, which can take advantage of the advantages of both and make up for their shortcomings.

[0003] The target detection method of the fusion of lidar information and visual information for autonomous driving can usually be divided into pixel-level fusion, feature-level fusion and decision-level fusion according to the different stages of fusion. The networks of most existing pixel-level fusion and feature-level fusion methods are relatively complex, making it difficult to achieve real-time performance, and the application effect is not good. The decision-level fusion method performs target detection on different data separately and then fuses them. The fusion method is relatively flexible. However, there are few studies on decision-level fusion methods, and their applications are relatively limited. There are still problems such as low target detection rate and poor real-time performance. Therefore, an autonomous driving target detection system and optimization method based on multi-sensor fusion in a connected vehicle environment are proposed to achieve an efficient, safe and high-quality driving process for autonomous driving vehicles. Summary of the invention

[0004] The purpose of the present invention is to provide an automatic driving target detection system and optimization method based on multi-sensor fusion in a vehicle networking environment, which enhances the ability to extract key features, improves the convergence speed and positioning accuracy of visual detection of target vehicles, ensures the lightweight requirements of the model, and avoids the loss of detection results and false detection.

[0005] To achieve the above-mentioned objectives, the present invention provides an autonomous driving target detection system based on multi-sensor fusion in a vehicle networking environment, including a visual camera target detection module, a laser radar target detection module, a target tracking module, a laser radar visual camera joint calibration module, a laser point cloud projection to image module and a laser radar information and visual information fusion module. The visual camera target detection module and the laser radar target detection module are both connected to the target tracking module, the target tracking module is connected to the laser radar visual camera joint calibration module, the laser radar visual camera joint calibration module is connected to the laser point cloud projection to image module, and the laser point cloud projection to image module is connected to the laser radar information and visual camera information fusion module.

[0006] Preferably, the visual camera target detection module adopts a multi-scale prediction method, adds a small target detection layer on the original three-layer detection layer, introduces the CBAM attention mechanism module, optimizes the loss function of network training, and uses the Wise-IOU loss function instead of the default CIOU loss function in YOLOV7-Tiny to perform target detection on social vehicles and autonomous driving vehicles to obtain image detection frames;

[0007] The LiDAR target detection module uses the ConvNeXt module to improve the two-dimensional convolution downsampling module in the backbone network, and embeds the SimAM attention module into the backbone network to perform three-dimensional target detection on social vehicles and other autonomous driving vehicles to obtain point cloud 3D detection frames;

[0008] The target tracking module performs Hungarian matching between the tracked target and the newly detected target between the adjacent frames of the detection results of the laser radar target detection module and the visual camera target detection module, and obtains the association matrix of the detected target. The association matrix is ​​used to calculate the association distance between each object and the newly created object, and the Hungarian algorithm is used to perform the optimal matching between each object and the newly created object;

[0009] The LiDAR visual camera joint calibration module performs joint calibration on the visual camera and LiDAR based on the joint calibration algorithm of the visual camera and LiDAR, transforms the measurement values ​​of the visual camera and LiDAR into a coordinate system, and obtains the joint calibration result of the visual camera and LiDAR, that is, the external parameter matrix. When performing joint calibration, it is usually necessary to consider the internal and external parameters of the sensor;

[0010] The laser point cloud is projected to the image module. The obtained external parameter matrix is ​​used to project the three-dimensional target detection result obtained by the laser radar target detection module to the pixel coordinate system of the camera image through the external parameter matrix obtained by the joint calibration module to obtain the initial point cloud 3D detection frame of each radar detection target.

[0011] The lidar information and visual information fusion module uses the Hungarian matching algorithm to optimally match the projected lidar point cloud 3D detection frame and the image detection frame of the visual camera and their tracking sequences, and performs secondary screening on the unmatched image detection frames and point cloud 3D detection frames to exclude image detection frames and point cloud 3D detection frames that are not in the tracking sequence and cannot match each other, and finally selects the image detection frames and point cloud 3D detection frames with higher reliability as the final fusion results for output.

[0012] The optimization method of automatic driving target detection based on multi-sensor fusion in the vehicle networking environment includes the following steps:

[0013] S1, detecting and identifying the target in the collected image through the visual camera target detection module;

[0014] S2, performing three-dimensional target detection on the collected point cloud data through the laser radar target detection module;

[0015] S3, through the target tracking module, establish the tracking sequence of the laser radar and the visual camera, and use the Hungarian matching algorithm to perform the optimal matching of the previous and next frame targets;

[0016] S4. The visual camera and the laser radar are jointly calibrated by the laser radar visual camera joint calibration module to obtain the visual camera and laser radar joint calibration result, i.e., the external parameter matrix;

[0017] S5. The obtained extrinsic parameter matrix is ​​passed to the laser point cloud projection to image module. The three-dimensional target detection result obtained by the laser radar target detection module is projected into the pixel coordinate system of the camera image using the extrinsic parameter matrix to obtain each initial point cloud 3D detection frame of the radar detection target;

[0018] S6. Finally, the lidar information and visual information fusion module uses the Hungarian matching algorithm to optimally match the tracking sequence of the lidar's point cloud 3D detection frame and the visual camera's image detection frame, fuses the point cloud 3D detection frame and the image detection frame, and outputs the fusion result after classifying all image detection frames and point cloud 3D detection frames according to whether the fused image detection frame and point cloud 3D detection frame are matched. The image detection frames and point cloud 3D detection frames with higher reliability are selected as the final fusion results for output and secondary screening.

[0019] Preferably, the visual camera target detection module in S1 improves the YOLOv7-Tiny model, mainly by introducing the FPN network idea. The improved YOLOv7-Tiny model performs convolution and upsampling operations on the P3 layer in the Neck layer to increase the size of the feature map, and splices it with the F2 layer with rich detail information in the backbone layer, and then fully mixes the image features through the ELAN-Tiny module to output the additionally constructed feature map P2. The CBAM attention mechanism module introduced therein enhances the model's ability to focus on key features by introducing a channel attention module and a spatial attention module in the convolutional neural network.

[0020] Preferably, the laser radar target detection module in S2 improves the Point Pillars algorithm, and mainly uses ConvNeXt to improve the two-dimensional convolution downsampling module in the backbone network. The ConvNeXt module is mainly used for feature extraction. It aggregates feature information at multiple scales through top-down downsampling and corresponding upsampling, enhances the feature extraction ability of the algorithm, and embeds the SimAM attention module into the backbone network. The SimAM attention module works by inferring the three-dimensional attention weights in the feature map, does not add additional network parameters, and has the characteristics of plug-and-play.

[0021] Preferably, the target tracking in S3 includes the following steps:

[0022] S3.1, calculating the relative distance between the target detection frame of the previous frame and the target detection frame of the current frame;

[0023]

[0024] in, is the relative distance; a i is the target detection frame of the previous frame, a i ∈A i (i=1,2,3...m), the coordinates of its center point are (x i ,y i );b j is the target detection box of the current frame, b j ∈B j (j=1,2,3...n), the coordinates of its center point are (x j ,y j );

[0025] S3.2, calculating the difference between the number of targets detected in the previous frame and the number of targets detected in the current frame;

[0026] F=M i -N j ;

[0027] Among them, F is the difference; M i is the number of targets detected in the previous frame; N j is the number of targets detected in the current frame;

[0028] S3.3, using the obtained relative distance values ​​to establish a correlation matrix;

[0029]

[0030] Among them, Q m×n is the correlation matrix;

[0031] S3.4, construct a matching mathematical model based on the Hungarian algorithm;

[0032]

[0033] Among them, W is the data association algorithm, X m×n Is a binary matrix marking whether the target detection box is completely matched; when X m×n When X is 1, it means that the target detection frame of the previous frame completely matches the target detection frame of the current frame. m×n When it is 0, it means that the target detection frame of the previous frame does not match the target detection frame of the current frame.

[0034] Preferably, the joint calibration in S4 includes the following steps:

[0035] S4.1, transform from the laser radar coordinate system to the camera coordinate system;

[0036]

[0037] Among them, R is the rotation matrix; T is the translation vector; any point P in the laser radar coordinate system L The coordinates of (x L ,y L ,z L );P C (x C ,y C ,z C ) is the point coordinate under the visual camera coordinate;

[0038] S4.2, converting the data of the camera coordinate system to the image coordinate system;

[0039]

[0040] Among them, P O (x O ,y O ) is the coordinate point P of the visual camera following the pinhole model principle C (x C ,y C ,zC ) is the projection point in the image coordinate system; f is the focal length of the visual camera;

[0041] S4.3, converting the data of the image coordinate system to the pixel coordinate system;

[0042]

[0043] Among them, u and v are the midpoint P of the laser radar coordinate system L (x L ,y L ,z L ) in the pixel coordinate system; u O 、v O is the coordinate of the optical center O in the pixel coordinate system; dx and dy are the quantized values ​​from the image coordinate system to the pixel coordinate system;

[0044] S4.4. By combining the above calculation process, the relationship between the laser radar coordinate system and the pixel coordinate system can be obtained when the data of the visual camera coordinate system is converted to the pixel coordinate system;

[0045] The data of the visual camera coordinate system is converted to the pixel coordinate system as follows:

[0046]

[0047] The relationship between the laser radar coordinate system and the pixel coordinate system:

[0048]

[0049] Among them, K is the camera internal parameter.

[0050] Preferably, the laser radar information and visual information fusion module in S6 uses the Hungarian matching algorithm and the obtained relative distance to construct an association matrix to calculate the optimal match between the image detection frame and the point cloud 3D detection frame, and performs secondary screening on the unmatched image detection frame and the point cloud 3D detection frame, and the secondary screening includes the following steps:

[0051] S6.1. Extract the unmatched laser radar point cloud 3D detection frame. If the point cloud 3D detection frame is marked with an ID in the target tracking module and is tracked by the target, it is considered that the point cloud 3D detection frame is not matched due to image occlusion, and the laser radar point cloud 3D detection frame is output;

[0052] S6.2, extract the image detection frame of the unmatched visual camera. If the image detection frame is marked with an ID in the target tracking module and there is a point cloud projection in the image detection frame, it is considered that the target is too far away and is not matched, and the image detection frame of the visual camera is output;

[0053] S6.3. If the above two conditions are not met, the match is deemed to have failed and no output operation is performed.

[0054] Therefore, the present invention adopts the above-mentioned automatic driving target detection system and optimization method based on multi-sensor fusion in the Internet of Vehicles environment, which has the following beneficial effects:

[0055] (1) This patent proposes an improved visual target detection module to address the problems of traffic target occlusion and difficulty in detecting small targets at a distance in complex traffic environments. Based on the YOLOV7-Tiny framework, a small target detection layer and a CBAM attention mechanism module are added, the training loss function is optimized, and the WIOU loss function is used instead of the default CIOU loss function in YOLOV7-Tiny to improve the accuracy and speed of target recognition and ensure the lightweight requirements of the model.

[0056] (2) This patent proposes an improved LiDAR target detection module to address the problems of vehicle occlusion and low small target detection accuracy in complex traffic environments. Based on the Point Pillars algorithm, the ConvNeXt module and the self-attention mechanism SimAM module are added to the backbone network to improve detection accuracy and efficiency, and ensure the lightweight requirements of the autonomous driving target detection model.

[0057] (3) The present invention aims to solve the problems of lost detection results, false detection and poor detection accuracy in traditional autonomous driving target detection models. It adds a Hungarian matching algorithm to realize target detection and tracking of visual cameras and lidars, matches the detection results of previous and next frames, avoids lost detection results and false detection, and further improves detection accuracy.

[0058] (4) The present invention aims to solve the problem that a single sensor cannot meet the target detection needs of autonomous driving vehicles in complex traffic scenes with multiple targets blocked and many small targets. It combines the advantages of radar and camera, realizes multi-sensor decision-level fusion by joint calibration of laser radar and visual camera, projection of laser point cloud to image, and fusion module of laser radar information and visual information. Ultimately, it improves the accuracy of target detection of autonomous driving, ensures the safety of autonomous driving, and has good practical value.

[0059] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flow chart of the target detection optimization method of the present invention;

[0061] Figure 2 It is a data flow diagram of the target detection optimization method of the present invention;

[0062] Figure 3 is an improved diagram of the visual camera target detection module in the present invention;

[0063] Figure 4 This is an improved diagram of the laser radar information and visual information fusion module in the present invention. DETAILED DESCRIPTION

[0064] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0065] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0066] Embodiment 1

[0067] The present invention provides an automatic driving target detection system based on multi-sensor fusion in a vehicle networking environment, including a visual camera target detection module, a laser radar target detection module, a target tracking module, a laser radar visual camera joint calibration module, a laser point cloud projection to image module and a laser radar information and visual information fusion module. The visual camera target detection module and the laser radar target detection module are both connected to the target tracking module, the target tracking module is connected to the laser radar visual camera joint calibration module, the laser radar visual camera joint calibration module is connected to the laser point cloud projection to image module, and the laser point cloud projection to image module is connected to the laser radar information and visual camera information fusion module.

[0068] The visual camera target detection module proposes an improved YOLOV7-Tiny algorithm to address the problems of vehicle occlusion in complex traffic environments and difficulty in detecting small targets at long distances when autonomous vehicles use visual cameras to detect targets of social vehicles and autonomous vehicles. A multi-scale prediction method is used to add a small target detection layer on the original three-layer detection layer, introduce the CBAM attention mechanism module, optimize the loss function of network training, and use the Wise-IOU loss function instead of the default CIOU loss function in YOLOV7-Tiny to detect targets of social vehicles and autonomous vehicles and obtain image detection frames.

[0069] The LiDAR target detection module aims to solve the problem of inaccurate recognition results when autonomous vehicles use LiDAR to detect three-dimensional targets in complex traffic environments with multiple targets occluded and many small targets. A LiDAR three-dimensional target detection method based on the improved PointPillars algorithm is proposed. The ConvNeXt module is used to improve the two-dimensional convolution downsampling module in the backbone network, and the SimAM attention module is embedded in the backbone network to perform three-dimensional target detection on social vehicles and other autonomous vehicles to obtain point cloud 3D detection frames.

[0070] The target tracking module aims to solve the problem of target detection loss caused by a large number of traffic participants and random position changes in complex traffic scenes. A method for tracking targets in the detection frames of laser radar and visual cameras is proposed. Hungarian matching is performed between the tracked targets and the newly detected targets in the adjacent frames of the detection results of the laser radar target detection module and the visual camera target detection module to obtain the association matrix of the detected targets. The association matrix is ​​used to calculate the association distance between each object and the newly created object, and the Hungarian algorithm is used to optimally match each object with the newly created object.

[0071] The LiDAR visual camera joint calibration module performs joint calibration on the visual camera and LiDAR based on the joint calibration algorithm of the visual camera and LiDAR, transforms the measurement values ​​of the visual camera and LiDAR into a coordinate system, and obtains the joint calibration result of the visual camera and LiDAR, that is, the external parameter matrix. When performing joint calibration, it is usually necessary to consider the intrinsic and extrinsic parameters of the sensor.

[0072] The laser point cloud projection to image module aims to solve the problem of inconsistent coordinate systems of collected point cloud and image data, and proposes a method to project point cloud from the original laser radar coordinate system to the pixel coordinate system where the image is located. Using the obtained external parameter matrix, the three-dimensional target detection result obtained by the laser radar target detection module is projected into the pixel coordinate system of the camera image through the external parameter matrix obtained by the joint calibration module, and the initial point cloud 3D detection frame of each radar detection target is obtained.

[0073] To address the low accuracy of the fusion of lidar information and visual information, the lidar information and visual information fusion module uses the Hungarian matching algorithm to optimally match the projected lidar point cloud 3D detection frame, the image detection frame of the visual camera and their tracking sequences, and performs secondary screening on the unmatched image detection frames and point cloud 3D detection frames to exclude image detection frames and point cloud 3D detection frames that are not in the tracking sequence and cannot be matched with each other. Finally, image detection frames and point cloud 3D detection frames with higher reliability are selected and output as the final fusion results.

[0074] like Figure 1 and Figure 2 As shown in the figure, the optimization method of automatic driving target detection based on multi-sensor fusion in the vehicle networking environment includes the following steps:

[0075] S1, detecting and identifying the target in the collected image through the visual camera target detection module;

[0076] like Figure 3As shown in the figure, the YOLOv7-Tiny model of the visual camera target detection module is mainly improved by introducing the FPN network idea. The improved YOLOv7-Tiny model performs convolution and upsampling operations on the feature map of the P3 layer in the Neck layer to increase the size of the feature map, and splices it with the feature map of the F2 layer with rich detail information in the backbone layer. Then, the image features are fully mixed through the ELAN-Tiny module, and the additional feature map P2 is output, so that the model can more effectively reduce the probability of missed detection when detecting vehicles, especially for densely distributed vehicles and targets of different sizes. The CBAM attention mechanism module introduced in it enhances the model's ability to focus on key features by introducing channel attention modules and spatial attention modules in the convolutional neural network. At the same time, the bounding box loss function Wise-IOU v3 (WIOU) based on the dynamic non-monotonic focusing mechanism is introduced, and the "outlier degree" is used instead of IOU to evaluate the quality of the anchor box, and a better gradient gain allocation strategy is provided.

[0077] S2, performing three-dimensional target detection on the collected point cloud data through the laser radar target detection module;

[0078] The Point Pillars algorithm of the LiDAR target detection module mainly uses ConvNeXt to improve the two-dimensional convolution downsampling module in the backbone network. The ConvNeXt module is mainly used for feature extraction. It aggregates feature information at multiple scales through top-down downsampling and corresponding upsampling to enhance the algorithm's feature extraction capabilities. The SimAM attention module is embedded in the backbone network. The SimAM attention module works by inferring the three-dimensional attention weights in the feature map, does not add additional network parameters, and has plug-and-play characteristics. The introduction of the ConvNeXt module and the SimAM attention module can improve the feature extraction capabilities of the backbone network, obtain richer feature information, and thus improve the detection accuracy of the model.

[0079] S3, through the target tracking module, establish the tracking sequence of the laser radar and the visual camera, and use the Hungarian matching algorithm to perform the optimal matching of the previous and next frame targets;

[0080] Target tracking includes the following steps:

[0081] S3.1, calculating the relative distance between the target detection frame of the previous frame and the target detection frame of the current frame;

[0082]

[0083] in, is the relative distance; a i is the target detection frame of the previous frame, a i ∈Ai (i=1,2,3...m), the coordinates of its center point are (x i ,y i );b j is the target detection box of the current frame, b j ∈B j (j=1,2,3...n), the coordinates of its center point are (x j ,y j );

[0084] S3.2, calculating the difference between the number of targets detected in the previous frame and the number of targets detected in the current frame;

[0085] F=M i -N j ;

[0086] Among them, F is the difference; M i is the number of targets detected in the previous frame; N j is the number of targets detected in the current frame;

[0087] The difference can help us understand the changes in the detected targets between consecutive frames. If the difference is positive, it means that new targets appear in the current frame; if the difference is negative, it means that some targets disappear in the current frame.

[0088] S3.3, using the obtained relative distance values ​​to establish a correlation matrix;

[0089]

[0090] Among them, Q m×n is the correlation matrix;

[0091] S3.4, construct a matching mathematical model based on the Hungarian algorithm;

[0092]

[0093] Among them, W is the data association algorithm, X m×n Is a binary matrix marking whether the target detection box is completely matched; when X m×n When X is 1, it means that the target detection frame of the previous frame completely matches the target detection frame of the current frame. m×n When it is 0, it means that the target detection frame of the previous frame does not match the target detection frame of the current frame.

[0094] S4. The visual camera and the laser radar are jointly calibrated by the laser radar visual camera joint calibration module to obtain the visual camera and laser radar joint calibration result, i.e., the external parameter matrix;

[0095] The joint calibration includes the following steps:

[0096] S4.1, transform from the laser radar coordinate system to the camera coordinate system;

[0097]

[0098] Among them, R is the rotation matrix; T is the translation vector; any point P in the laser radar coordinate system L The coordinates of (x L ,y L ,z L );P C (x C ,y C ,z C ) is the point coordinate under the visual camera coordinate;

[0099] S4.2, converting the data of the camera coordinate system to the image coordinate system;

[0100]

[0101] Among them, P O (x O ,y O ) is the coordinate point P of the visual camera following the pinhole model principle C (x C ,y C ,z C ) is the projection point in the image coordinate system; f is the focal length of the visual camera;

[0102] S4.3, converting the data of the image coordinate system to the pixel coordinate system;

[0103]

[0104] Among them, u and v are the midpoint P of the laser radar coordinate system L (x L ,y L ,z L ) in the pixel coordinate system; u O 、v O is the coordinate of the optical center O in the pixel coordinate system; dx and dy are the quantized values ​​from the image coordinate system to the pixel coordinate system;

[0105] S4.4. By combining the above calculation process, the relationship between the laser radar coordinate system and the pixel coordinate system can be obtained when the data of the visual camera coordinate system is converted to the pixel coordinate system;

[0106] The data of the visual camera coordinate system is converted to the pixel coordinate system as follows:

[0107]

[0108] The relationship between the laser radar coordinate system and the pixel coordinate system:

[0109]

[0110] Among them, K is the camera internal parameter.

[0111] S5, pass the obtained external parameter matrix to the laser point cloud projection to the image module, and use the external parameter matrix to convert the three-dimensional target detection result P obtained by the laser radar target detection module L (x L ,y L ,z L ), projected into the pixel coordinate system (u, v) of the camera image, so that the target in the image detection frame has the spatial coordinate information measured by the lidar, and the initial point cloud 3D detection frames of the radar detection targets are obtained.

[0112] S6. Finally, the lidar information and visual information fusion module uses the Hungarian matching algorithm to optimally match the tracking sequence of the lidar's point cloud 3D detection frame and the visual camera's image detection frame, fuses the point cloud 3D detection frame and the image detection frame, and outputs the fusion result after classifying all image detection frames and point cloud 3D detection frames according to whether the fused image detection frame and point cloud 3D detection frame are matched. The image detection frames and point cloud 3D detection frames with higher reliability are selected as the final fusion results for output and secondary screening.

[0113] like Figure 4 As shown, let the image detection box in the current frame be c i ∈C i (i=1,2,3...m), the point cloud 3D detection box in the current frame is d j ∈D j (j=1,2,3...n), the coordinates of the center point of the image detection frame are (x ci ,y ci ), the coordinates of the center point of the point cloud 3D detection box are (x dj ,y dj ), calculate the relative distance between the image detection box and the point cloud 3D detection box

[0114] The LiDAR information and visual information fusion module uses the Hungarian matching algorithm and the relative distance to construct the correlation matrix to calculate the optimal match between the image detection frame and the point cloud 3D detection frame, and performs secondary screening on the unmatched image detection frames and point cloud 3D detection frames. The secondary screening includes the following steps:

[0115] S6.1. Extract the unmatched laser radar point cloud 3D detection frame. If the point cloud 3D detection frame is marked with an ID in the target tracking module and is tracked by the target, it is considered that the point cloud 3D detection frame is not matched due to image occlusion, and the laser radar point cloud 3D detection frame is output;

[0116] S6.2, extract the image detection frame of the unmatched visual camera. If the image detection frame is marked with an ID in the target tracking module and there is a point cloud projection in the image detection frame, it is considered that the target is too far away and is not matched, and the image detection frame of the visual camera is output;

[0117] S6.3. If the above two conditions are not met, the match is deemed to have failed and no output operation is performed.

[0118] Therefore, the present invention adopts the autonomous driving target detection system and optimization method based on multi-sensor fusion in the above-mentioned vehicle networking environment, which enhances the ability to extract key features, improves the convergence speed and positioning accuracy of visual detection of target vehicles, ensures the lightweight requirements of the model, and avoids the loss of detection results and false detection.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. An autonomous driving target detection system based on multi-sensor fusion in a connected vehicle environment, characterized by: It includes a visual camera target detection module, a lidar target detection module, a target tracking module, a lidar visual camera joint calibration module, a laser point cloud projection to image module and a lidar information and visual information fusion module. The visual camera target detection module and the lidar target detection module are both connected to the target tracking module, the target tracking module is connected to the lidar visual camera joint calibration module, the lidar visual camera joint calibration module is connected to the laser point cloud projection to image module, and the laser point cloud projection to image module is connected to the lidar information and visual camera information fusion module.

2. The automatic driving target detection system based on multi-sensor fusion in the vehicle networking environment according to claim 1 is characterized by: The visual camera target detection module adopts a multi-scale prediction method. On top of the original three-layer detection layer, a small target detection layer is added. The CBAM attention mechanism module is introduced to optimize the loss function of network training. At the same time, the Wise-IOU loss function is used instead of the default CIOU loss function in YOLOV7-Tiny to detect social vehicles and autonomous driving vehicles and obtain image detection frames. The LiDAR target detection module uses the ConvNeXt module to improve the two-dimensional convolution downsampling module in the backbone network, and embeds the SimAM attention module into the backbone network to perform three-dimensional target detection on social vehicles and other autonomous driving vehicles to obtain point cloud 3D detection frames; The target tracking module performs Hungarian matching between the tracked target and the newly detected target between the adjacent frames of the detection results of the laser radar target detection module and the visual camera target detection module, and obtains the association matrix of the detected target. The association matrix is ​​used to calculate the association distance between each object and the newly created object, and the Hungarian algorithm is used to perform the optimal matching between each object and the newly created object; The LiDAR visual camera joint calibration module performs joint calibration on the visual camera and LiDAR based on the joint calibration algorithm of the visual camera and LiDAR, transforms the measurement values ​​of the visual camera and LiDAR into a coordinate system, and obtains the joint calibration result of the visual camera and LiDAR, that is, the external parameter matrix. When performing joint calibration, it is usually necessary to consider the internal and external parameters of the sensor; The laser point cloud is projected to the image module. The obtained external parameter matrix is ​​used to project the three-dimensional target detection result obtained by the laser radar target detection module to the pixel coordinate system of the camera image through the external parameter matrix obtained by the joint calibration module to obtain the initial point cloud 3D detection frame of each radar detection target. The lidar information and visual information fusion module uses the Hungarian matching algorithm to optimally match the projected lidar point cloud 3D detection frame and the image detection frame of the visual camera and their tracking sequences, and performs secondary screening on the unmatched image detection frames and point cloud 3D detection frames to exclude image detection frames and point cloud 3D detection frames that are not in the tracking sequence and cannot match each other, and finally selects the image detection frames and point cloud 3D detection frames with higher reliability as the final fusion results for output.

3. The optimization method for automatic driving target detection based on multi-sensor fusion in a connected vehicle environment according to any one of claim 1, characterized in that: The following steps are involved: S1, detecting and identifying the target in the collected image through the visual camera target detection module; S2, performing three-dimensional target detection on the collected point cloud data through the laser radar target detection module; S3, through the target tracking module, establish the tracking sequence of the laser radar and the visual camera, and use the Hungarian matching algorithm to perform the optimal matching of the previous and next frame targets; S4. The visual camera and the laser radar are jointly calibrated by the laser radar visual camera joint calibration module to obtain the visual camera and laser radar joint calibration result, i.e., the external parameter matrix; S5. The obtained extrinsic parameter matrix is ​​passed to the laser point cloud projection to image module. The three-dimensional target detection result obtained by the laser radar target detection module is projected into the pixel coordinate system of the camera image using the extrinsic parameter matrix to obtain each initial point cloud 3D detection frame of the radar detection target; S6. Finally, the lidar information and visual information fusion module uses the Hungarian matching algorithm to optimally match the tracking sequence of the lidar's point cloud 3D detection frame and the visual camera's image detection frame, fuses the point cloud 3D detection frame and the image detection frame, and outputs the fusion result after classifying all image detection frames and point cloud 3D detection frames according to whether the fused image detection frame and point cloud 3D detection frame are matched. The image detection frames and point cloud 3D detection frames with higher reliability are selected as the final fusion results for output and secondary screening.

4. The optimization method for automatic driving target detection based on multi-sensor fusion in a connected vehicle environment according to claim 3, characterized in that: The visual camera target detection module in S1 improves the YOLOv7-Tiny model, mainly by introducing the FPN network idea. The improved YOLOv7-Tiny model performs convolution and upsampling operations on the P3 layer in the Neck layer to increase the size of the feature map, and splices it with the F2 layer with rich detail information in the backbone layer. The image features are fully mixed through the ELAN-Tiny module to output the additionally constructed feature map P2. The CBAM attention mechanism module introduced in it enhances the model's ability to focus on key features by introducing channel attention modules and spatial attention modules in the convolutional neural network.

5. The optimization method for automatic driving target detection based on multi-sensor fusion in a connected vehicle environment according to claim 3, characterized in that: The lidar target detection module in S2 improves the Point Pillars algorithm, mainly using ConvNeXt to improve the two-dimensional convolution downsampling module in the backbone network. The ConvNeXt module is mainly used for feature extraction. It aggregates feature information at multiple scales through top-down downsampling and corresponding upsampling, enhances the feature extraction capability of the algorithm, and embeds the SimAM attention module into the backbone network. The SimAM attention module works by inferring the three-dimensional attention weights in the feature map, does not add additional network parameters, and has the characteristics of plug-and-play.

6. The optimization method for automatic driving target detection based on multi-sensor fusion in a connected vehicle environment according to claim 3, characterized in that: Target tracking in S3 includes the following steps: S3.1, calculating the relative distance between the target detection frame of the previous frame and the target detection frame of the current frame; in, is the relative distance; a i is the target detection frame of the previous frame, a i ∈A i (i=1,2,3...m), the coordinates of its center point are (x i ,y i );b j is the target detection box of the current frame, b j ∈B j (j=1,2,3...n), the coordinates of its center point are (x j ,y j ); S3.2, calculating the difference between the number of targets detected in the previous frame and the number of targets detected in the current frame; F=M i -N j ; Among them, F is the difference; M i is the number of targets detected in the previous frame; N j is the number of targets detected in the current frame; S3.3, using the obtained relative distance values ​​to establish a correlation matrix; Among them, Q m×n is the correlation matrix; S3.4, construct a matching mathematical model based on the Hungarian algorithm; Among them, W is the data association algorithm, X m×n Is a binary matrix marking whether the target detection box is completely matched; when X m×n When X is 1, it means that the target detection frame of the previous frame completely matches the target detection frame of the current frame. m×n When it is 0, it means that the target detection frame of the previous frame does not match the target detection frame of the current frame.

7. The optimization method for automatic driving target detection based on multi-sensor fusion in a connected vehicle environment according to claim 3, characterized in that: The joint calibration in S4 includes the following steps: S4.1, transform from the laser radar coordinate system to the camera coordinate system; Among them, R is the rotation matrix; T is the translation vector; any point P in the laser radar coordinate system L The coordinates of (x L ,y L ,z L );P C (x C ,y C ,z C ) is the point coordinate under the visual camera coordinate; S4.2, converting the data of the camera coordinate system to the image coordinate system; Among them, P O (x O ,y O ) is the coordinate point P of the visual camera following the pinhole model principle C (x C ,y C ,z C ) is the projection point in the image coordinate system; f is the focal length of the visual camera; S4.3, converting the data of the image coordinate system to the pixel coordinate system; Among them, u and v are the midpoint P of the laser radar coordinate system L (x L ,y L ,z L ) in the pixel coordinate system; u O 、v O is the coordinate of the optical center O in the pixel coordinate system; dx and dy are the quantized values ​​from the image coordinate system to the pixel coordinate system; S4.

4. By combining the above calculation process, the relationship between the laser radar coordinate system and the pixel coordinate system can be obtained when the data of the visual camera coordinate system is converted to the pixel coordinate system; The data of the visual camera coordinate system is converted to the pixel coordinate system as follows: The relationship between the laser radar coordinate system and the pixel coordinate system: Among them, K is the camera internal parameter.

8. The optimization method for automatic driving target detection based on multi-sensor fusion in a connected vehicle environment according to claim 3, characterized in that: The laser radar information and visual information fusion module in S6 uses the Hungarian matching algorithm and the relative distance to construct the correlation matrix to calculate the optimal match between the image detection frame and the point cloud 3D detection frame, and performs secondary screening on the unmatched image detection frames and point cloud 3D detection frames. The secondary screening includes the following steps: S6.

1. Extract the unmatched laser radar point cloud 3D detection frame. If the point cloud 3D detection frame is marked with an ID in the target tracking module and is tracked by the target, it is considered that the point cloud 3D detection frame is not matched due to image occlusion, and the laser radar point cloud 3D detection frame is output; S6.2, extract the image detection frame of the unmatched visual camera. If the image detection frame is marked with an ID in the target tracking module and there is a point cloud projection in the image detection frame, it is considered that the target is too far away and is not matched, and the image detection frame of the visual camera is output; S6.

3. If the above two conditions are not met, the match is deemed to have failed and no output operation is performed.

Images