Automatic driving target detection system and optimization method based on multi-sensor fusion in vehicle networking environment

Abstract

The application discloses an automatic driving target detection system and optimization method based on multi-sensor fusion in a vehicle networking environment, belongs to the technical field of automatic driving, and comprises a visual camera target detection module, a laser radar target detection module, a target tracking module, a laser radar visual camera joint calibration module, a laser point cloud projection to an image module and a laser radar information and visual information fusion module.The automatic driving target detection system and optimization method based on multi-sensor fusion in the vehicle networking environment have the advantages that the extraction capability of key features is enhanced, the convergence speed and positioning accuracy of visual detection on a target vehicle are improved, the model lightweight demand is ensured, and the loss and false detection of detection results are avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to an automatic driving target detection system and optimization method based on multi-sensor fusion in a vehicle networking environment. BACKGROUND

[0002] With the continuous deepening of target detection algorithm research, traditional automatic driving target detection algorithms have been able to meet the target detection problem in general traffic scenarios. In complex traffic scenarios, in the process of identifying surrounding targets, the automatic driving vehicle is prone to occlusion of traffic targets, which makes the appearance, shape and color features of the occluded traffic targets not fully presented, increasing the difficulty of the target detection algorithm in identifying traffic targets. In addition, in the process of identifying small targets at a distance, the lack of sufficient context information of the features that can be extracted during target detection due to the small number of pixels in the image makes it difficult for the target detection model to use other information in the scene to assist in detection, resulting in low detection accuracy. At present, the perception system of the automatic driving vehicle mostly uses two kinds of sensors, laser radar and visual camera. Both of these two sensors have their own advantages and disadvantages. The visual camera sensor can obtain rich visual information of the surrounding environment and traffic targets, but due to the lack of depth information, it cannot obtain accurate three-dimensional spatial information of the object, and is easily disturbed by the external environment. Laser radar has good robustness and can accurately obtain three-dimensional spatial information of the object, but cannot obtain color, texture and shape information of the traffic target, and has poor recognition ability. Therefore, a single sensor cannot meet the requirements of accuracy and robustness of automatic driving target detection. The present application proposes to fuse the perception information of laser radar and visual camera, which can take advantage of both and make up for the shortcomings of both.

[0003] The target detection method of automatic driving laser radar information and visual information fusion can be divided into pixel-level fusion, feature-level fusion and decision-level fusion according to different stages of fusion. Most of the existing pixel-level fusion and feature-level fusion methods have complex networks, which are difficult to realize real-time and have poor application effect. The decision-level fusion method detects targets separately for different data and then fuses them, and the fusion method is relatively flexible. However, there are few studies on decision-level fusion methods, and the application is relatively limited, and there are still problems of low target detection rate and poor real-time performance. Therefore, the automatic driving target detection system and optimization method based on multi-sensor fusion in a vehicle networking environment are proposed to realize the efficient, safe and high-quality driving process of automatic driving vehicles. SUMMARY

[0004] The application aims to provide an automatic driving target detection system based on multi-sensor fusion in a vehicle networking environment and an optimization method, which enhances the extraction capability of key features, improves the convergence speed and positioning accuracy of visual detection on target vehicles, guarantees the lightweight demand of the model, and avoids detection result loss and false detection.

[0005] To achieve the above-mentioned purpose, the application provides an automatic driving target detection system based on multi-sensor fusion in a vehicle networking environment, which comprises a visual camera target detection module, a laser radar target detection module, a target tracking module, a laser radar visual camera joint calibration module, a laser point cloud projection to image module, and a laser radar information and visual information fusion module, the visual camera target detection module and the laser radar target detection module are connected with the target tracking module, the target tracking module is connected with the laser radar visual camera joint calibration module, the laser radar visual camera joint calibration module is connected with the laser point cloud projection to image module, and the laser point cloud projection to image module is connected with the laser radar information and visual camera information fusion module.

[0006] Preferably, the visual camera target detection module adopts a multi-scale prediction method, adds a small target detection layer on the original three detection layers, introduces a CBAM attention mechanism module, optimizes the loss function of network training, and simultaneously uses a Wise-IOU loss function to replace the default CIOU loss function in YOLOV7-Tiny, so as to perform target detection on social vehicles and automatic driving vehicles to obtain an image detection frame.

[0007] The laser radar target detection module improves the two-dimensional convolution downsampling module in the backbone network by using a ConvNeXt module, and embeds a SimAM attention module in the backbone network, so as to perform three-dimensional target detection on social vehicles and other automatic driving vehicles to obtain a point cloud 3D detection frame.

[0008] The target tracking module performs Hungarian matching between tracked targets and newly detected targets in adjacent frames of the detection results of the laser radar target detection module and the visual camera target detection module, obtains an association matrix of the detection targets, the association matrix is used to calculate the association distance between each object and a newly built object, and the Hungarian algorithm is used to perform optimal matching for each object and the newly built object.

[0009] The laser radar visual camera joint calibration module performs joint calibration of the visual camera and the laser radar based on a joint calibration algorithm of the visual camera and the laser radar, transforms the measurement values of the visual camera and the laser radar to a coordinate system, obtains a joint calibration result of the visual camera and the laser radar, i.e., an extrinsic matrix, and when performing joint calibration, the intrinsic parameters and the extrinsic parameters of the sensors need to be considered.

[0010] The laser point cloud is projected onto the image module. The obtained extrinsic parameter matrix is ​​used to project the three-dimensional target detection results obtained by the laser radar target detection module onto the pixel coordinate system of the camera image through the extrinsic parameter matrix obtained by the joint calibration module to obtain the initial point cloud 3D detection frames of the radar detection targets.

[0011] The lidar information and visual information fusion module uses the Hungarian matching algorithm to optimally match the projected lidar point cloud 3D detection frame, the visual camera's image detection frame, and their tracking sequences. It then performs a secondary screening of the unmatched image detection frames and point cloud 3D detection frames to exclude image detection frames and point cloud 3D detection frames that are not in the tracking sequence and cannot be matched with each other. Finally, the image detection frames and point cloud 3D detection frames with higher reliability are selected and output as the final fusion results.

[0012] The optimization method for autonomous driving target detection based on multi-sensor fusion in the Internet of Vehicles environment includes the following steps:

[0013] S1, detect and identify the target in the collected image through the visual camera target detection module;

[0014] S2. Perform three-dimensional target detection on the collected point cloud data through the laser radar target detection module;

[0015] S3. Through the target tracking module, establish the tracking sequence of the lidar and visual camera, and use the Hungarian matching algorithm to perform the optimal matching of the previous and next frame targets;

[0016] S4. Perform joint calibration of the visual camera and the lidar through the lidar-vision camera joint calibration module to obtain the joint calibration result of the visual camera and the lidar, i.e., the extrinsic parameter matrix;

[0017] S5. The obtained extrinsic parameter matrix is ​​passed to the laser point cloud projection to image module. The extrinsic parameter matrix is ​​used to project the three-dimensional target detection results obtained by the laser radar target detection module into the pixel coordinate system of the camera image to obtain the initial point cloud 3D detection frames of the radar detection targets.

[0018] S6. Finally, the lidar information and visual information fusion module uses the Hungarian matching algorithm to optimally match the tracking sequence of the lidar's point cloud 3D detection frame and the visual camera's image detection frame, fuses the point cloud 3D detection frame and the image detection frame, and classifies all image detection frames and point cloud 3D detection frames according to whether the fused image detection frame and point cloud 3D detection frame are matched, and outputs the fusion result. The image detection frame and point cloud 3D detection frame with higher reliability are selected as the final fusion result for output and secondary screening.

[0019] Preferably, the visual camera target detection module in S1 improves the YOLOv7-Tiny model, mainly by introducing the FPN network idea for improvement. The improved YOLOv7-Tiny model performs convolution and upsampling operations on the P3 layer in the Neck layer, increases the size of the feature map, and is spliced with the F2 layer in the backbone layer which is rich in detailed information. Then, the image features are fully mixed through the ELAN-Tiny module to output the additionally constructed feature map P2. The CBAM attention mechanism module is introduced, which enhances the model's attention to key features by introducing a channel attention module and a spatial attention module in the convolutional neural network.

[0020] Preferably, the laser radar target detection module in S2 improves the Point Pillars algorithm, mainly using ConvNeXt to improve the two-dimensional convolution downsampling module in the backbone network. The ConvNeXt module is mainly used for feature extraction. It aggregates feature information at multiple scales through top-down downsampling and corresponding upsampling, enhances the feature extraction capability of the algorithm, and embeds the SimAM attention module in the backbone network. The SimAM attention module works by inferring three-dimensional attention weights in the feature map, without increasing additional network parameters, and has the characteristics of plug and play.

[0021] Preferably, the target tracking in S3 includes the following steps:

[0022] S3.1, calculate the relative distance between the target detection box of the previous frame and the target detection box of the current frame;

[0023]

[0024] wherein, is the relative distance; a i is the target detection box of the previous frame, a i ∈A i (i = 1, 2, 3...m), with the center point coordinates (x i , y i ); b j is the target detection box of the current frame, b j ∈B j (j = 1, 2, 3...n), with the center point coordinates (x j , y j );

[0025] S3.2, calculate the difference between the number of targets detected in the previous frame and the number of targets detected in the current frame;

[0026] F = M i -N j ;

[0027] Wherein, F is the difference value; M i is the number of targets detected in the previous frame; N j is the number of targets detected in the current frame;

[0028] S3.3, establishing a correlation matrix by using the relative distance value obtained;

[0029]

[0030] Wherein, Q m×n is the correlation matrix;

[0031] S3.4, constructing a matching mathematical model based on the Hungarian algorithm;

[0032]

[0033] Wherein, W is a data association algorithm, X m×n is a binary matrix indicating whether the target detection frame is completely matched; when X m×n is 1, it means that the target detection frame of the previous frame is completely matched with the target detection frame of the current frame, when X m×n is 0, it means that the target detection frame of the previous frame is not matched with the target detection frame of the current frame.

[0034] Preferably, the joint calibration in S4 includes the following steps:

[0035] S4.1, converting from the laser radar coordinate system to the camera coordinate system;

[0036]

[0037] Wherein, R is a rotation matrix; T is a translation vector; the coordinates of any point P L in the laser radar coordinate system are (x L , y L , z L ); P C (x C , y C , z C ) is the point coordinate in the visual camera coordinate;

[0038] S4.2, converting the data of the camera coordinate system to the image coordinate system;

[0039]

[0040] Wherein, P O (x O , y O ) is the visual camera coordinate point P C (x C , y C , zC ) the projection point in the image coordinate system; f is the focal length of the vision camera;

[0041] S4.3, convert the data of the image coordinate system to the pixel coordinate system;

[0042]

[0043] wherein u, v are the coordinates of the point P L (x L ,y L ,z L ) in the pixel coordinate system; u O , v O are the coordinates of the optical center O in the pixel coordinate system; dx, dy are the quantization values from the image coordinate system to the pixel coordinate system;

[0044] S4.4, by integrating the above calculation process, the relationship between the laser radar coordinate system and the pixel coordinate system can be obtained when the data of the vision camera coordinate system is converted to the pixel coordinate system;

[0045] The data of the vision camera coordinate system converted to the pixel coordinate system is:

[0046]

[0047] The relationship between the laser radar coordinate system and the pixel coordinate system is:

[0048]

[0049] wherein K is the camera intrinsic parameter.

[0050] Preferably, the laser radar information and vision information fusion module in S6 uses the Hungarian matching algorithm and the relative distance obtained to construct a correlation matrix to calculate the optimal matching of the image detection frame and the point cloud 3D detection frame, and performs secondary screening on the unmatched image detection frame and the point cloud 3D detection frame, the secondary screening including the following steps:

[0051] S6.1, extract the unmatched laser radar point cloud 3D detection frame, if the point cloud 3D detection frame is marked with an ID in the target tracking module, it is considered that the image occlusion causes the point cloud 3D detection frame to be unmatched, and the point cloud 3D detection frame of the laser radar is output;

[0052] S6.2, extract the unmatched image detection frame of the vision camera, if the image detection frame is marked with an ID in the target tracking module, and the image detection frame has a point cloud projection, it is considered that the target distance is too far to cause the unmatched, and the image detection frame of the vision camera is output;

[0053] S6.3, if the above two conditions are not met, it is considered that the matching fails, and no output operation is performed.

[0054] Therefore, the application adopts the automatic driving target detection system and optimization method based on multi-sensor fusion in the above Internet of Vehicles environment, which has the following beneficial effects:

[0055] (1) The present application improves the visual target detection module to solve the problem of traffic target occlusion and detection of small targets at a long distance in complex traffic environment. Based on the YOLOV7-Tiny framework, small target detection layer and CBAM attention mechanism module are added, and the training loss function is optimized. The WIOU loss function is used instead of the default CIOU loss function in YOLOV7-Tiny, which improves the accuracy and speed of target recognition and ensures the lightweight demand of the model.

[0056] (2) The present application improves the laser radar target detection module to solve the problems of vehicle occlusion and low small target detection accuracy in complex traffic environment. Based on the Point Pillars algorithm, ConvNeXt module and self-attention mechanism SimAM module are added in the backbone network to improve the detection accuracy and efficiency and ensure the lightweight demand of the automatic driving target detection model.

[0057] (3) The present application increases the Hungarian matching algorithm to realize the target detection and tracking of visual camera and laser radar to solve the problems of detection result loss and false detection, low detection accuracy of traditional automatic driving target detection model. The detection results of the front and rear frames are matched to avoid detection result loss and false detection, and further improve the detection accuracy.

[0058] (4) The present application combines the advantages of radar and camera to realize multi-sensor decision-level fusion by laser radar and visual camera joint calibration, laser point cloud projection to image, laser radar information and visual information fusion module to solve the problem that single sensor cannot meet the target detection demand of automatic driving vehicle in complex traffic scene with multiple target occlusion and small targets. Finally, the accuracy of automatic driving target detection is improved, the safety of automatic driving is ensured, and it has good practical value.

[0059] The technical solutions of the present application will be further described in detail below with the help of the drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is the flow chart of the target detection optimization method of the present application;

[0061] Figure 2 is the data flow chart of the target detection optimization method of the present application;

[0062] Figure 3 is the improvement diagram of the visual camera target detection module in the present application;

[0063] Figure 4 Figure is an improved diagram of the laser radar information and visual information fusion module in the application. DETAILED DESCRIPTION

[0064] The technical solutions of the application are further described below through the drawings and examples.

[0065] Unless otherwise defined, the technical terms or scientific terms used in the application shall have the usual meanings understood by those skilled in the art to which the application belongs.

[0066] Example one

[0067] The application provides an automatic driving target detection system based on multi-sensor fusion in a vehicle networking environment, which comprises a visual camera target detection module, a laser radar target detection module, a target tracking module, a laser radar visual camera joint calibration module, a laser point cloud projection to image module, and a laser radar information and visual information fusion module. The visual camera target detection module and the laser radar target detection module are connected with the target tracking module, the target tracking module is connected with the laser radar visual camera joint calibration module, the laser radar visual camera joint calibration module is connected with the laser point cloud projection to image module, and the laser point cloud projection to image module is connected with the laser radar information and visual camera information fusion module.

[0068] The visual camera target detection module proposes an improved YOLOV7-Tiny algorithm to solve the problems of vehicle occlusion in complex traffic environment and difficulty in detecting small targets at a long distance when the automatic driving vehicle detects social vehicles and automatic driving vehicles through a visual camera. A multi-scale prediction method is used to add a small target detection layer on the original three detection layers, introduce a CBAM attention mechanism module, optimize the loss function of network training, and use a Wise-IOU loss function instead of the default CIOU loss function in YOLOV7-Tiny to detect the target detection frame of social vehicles and automatic driving vehicles.

[0069] The laser radar target detection module proposes a laser radar three-dimensional target detection method based on an improved PointPillars algorithm to solve the problem of inaccurate recognition results when the automatic driving vehicle detects three-dimensional targets through a laser radar in a complex traffic environment with multi-target occlusion and many small targets. A ConvNeXt module is used to improve the two-dimensional convolution downsampling module in the backbone network, and a SimAM attention module is embedded in the backbone network to detect the point cloud 3D detection frame of social vehicles and other automatic driving vehicles.

[0070] The target tracking module proposes a method of tracking the detection boxes of the laser radar and the vision camera to solve the problem of target detection loss caused by the large number of traffic participants and the random change of their positions in a complex traffic scene. The tracked targets and newly detected targets between adjacent frames of the detection results of the laser radar target detection module and the vision camera target detection module are subjected to Hungarian matching, and a correlation matrix of the detection targets is obtained. The correlation matrix is used to calculate the correlation distance between each object and a newly built object, and the Hungarian algorithm is used to perform optimal matching for each object and the newly built object.

[0071] The laser radar and vision camera joint calibration module performs joint calibration of the vision camera and the laser radar based on a joint calibration algorithm of the vision camera and the laser radar, transforms the measurement values of the vision camera and the laser radar to a coordinate system, and obtains a joint calibration result of the vision camera and the laser radar, i.e., an extrinsic parameter matrix. When performing joint calibration, the intrinsic and extrinsic parameters of the sensors need to be considered.

[0072] The laser point cloud projection to image module proposes a method of projecting the point cloud from the original laser radar coordinate system to the pixel coordinate system of the image to solve the problem of non-uniform coordinate systems of the collected point cloud and image data. The three-dimensional target detection result obtained by the laser radar target detection module is projected into the pixel coordinate system of the camera image by using the extrinsic parameter matrix obtained by the joint calibration module, and the initial point cloud 3D detection box of the radar detection target is obtained.

[0073] The laser radar information and vision information fusion module adopts the Hungarian matching algorithm to perform optimal matching on the projected laser radar point cloud 3D detection box, the image detection box of the vision camera, and the tracking sequences of the two to solve the problem of low fusion accuracy of laser radar information and vision information. The image detection boxes and point cloud 3D detection boxes that are not matched are subjected to secondary screening to exclude the image detection boxes and point cloud 3D detection boxes that are not in the tracking sequence and cannot be matched with each other, and the image detection boxes and point cloud 3D detection boxes with high reliability are finally screened as the final fusion result for output.

[0074] As shown in Figure 1 and Figure 2 , the optimization method of automatic driving target detection based on multi-sensor fusion in a vehicle networking environment includes the following steps:

[0075] S1, detecting and identifying the targets in the collected images by the vision camera target detection module;

[0076] As shown in Figure 3As shown, the YOLOv7-Tiny model of the visual camera target detection module is mainly improved by introducing the FPN network idea. The improved YOLOv7-Tiny model performs convolution and upsampling operations on the feature map of the P3 layer in the Neck layer, increases the size of the feature map, and splices the feature map of the F2 layer rich in detailed information in the backbone layer, and then fully mixes the image features through the ELAN-Tiny module to output the additionally constructed feature map P2, so that the model can more effectively reduce the probability of missed detection when detecting vehicles, especially for densely distributed vehicles and targets of different sizes. The CBAM attention mechanism module introduced therein enhances the model's attention to key features by introducing a channel attention module and a spatial attention module in the convolutional neural network. At the same time, the boundary box loss function Wise-IOU v3 (WIOU) based on the dynamic non-monotonic focusing mechanism is introduced, which uses "outlier degree" to replace IOU to evaluate the quality of anchor boxes and provides a more excellent gradient gain allocation strategy.

[0077] S2, performing three-dimensional target detection on the collected point cloud data through the laser radar target detection module;

[0078] The Point Pillars algorithm of the laser radar target detection module mainly improves the two-dimensional convolution downsampling module in the backbone network by using the ConvNeXt module. The ConvNeXt module is mainly used for feature extraction, which aggregates feature information at multiple scales through top-down downsampling and corresponding upsampling to enhance the feature extraction capability of the algorithm. The SimAM attention module is embedded in the backbone network, which works by inferring three-dimensional attention weights in the feature map without adding additional network parameters and has the feature of plug and play. The introduction of the ConvNeXt module and the SimAM attention module can improve the feature extraction capability of the backbone network, obtain more rich feature information, and thus improve the detection accuracy of the model.

[0079] S3, establishing a tracking sequence of the laser radar and the visual camera through the target tracking module, and performing optimal matching of the front and rear frames of the target by using the Hungarian matching algorithm;

[0080] The target tracking includes the following steps:

[0081] S3.1, calculating the relative distance between the target detection frame of the previous frame and the target detection frame of the current frame;

[0082]

[0083] wherein, is the relative distance; a i is the target detection frame of the previous frame, a i ∈Ai (i = 1, 2, 3...m), where the center point coordinates are (x i , y i ); b j is the target detection frame of the current frame, b j ∈B j (j = 1, 2, 3...n), where the center point coordinates are (x j , y j );

[0084] S3.2, calculate the difference between the number of detected targets in the previous frame and the number of detected targets in the current frame;

[0085] F = M i -N j ;

[0086] Where F is the difference; M i is the number of detected targets in the previous frame; N j is the number of detected targets in the current frame;

[0087] The difference can help us understand the change of the detected targets between consecutive frames. If the difference is positive, it means that new targets appear in the current frame; if the difference is negative, it means that some targets disappear in the current frame.

[0088] S3.3, use the relative distance value obtained to establish a correlation matrix;

[0089]

[0090] Where Q m×n is the correlation matrix;

[0091] S3.4, construct a matching mathematical model based on the Hungarian algorithm;

[0092]

[0093] Where W is the data association algorithm, X m×n is a binary matrix that marks whether the target detection frame is completely matched; when X m×n is 1, it means that the target detection frame of the previous frame is completely matched with the target detection frame of the current frame, when X m×n is 0, it means that the target detection frame of the previous frame is not matched with the target detection frame of the current frame.

[0094] S4, through the laser radar vision camera joint calibration module, the vision camera and the laser radar are jointly calibrated, and the vision camera, laser radar joint calibration result is obtained, that is, the extrinsic matrix;

[0095] The joint calibration includes the following steps:

[0096] S4.1. Convert from the lidar coordinate system to the camera coordinate system;

[0097]

[0098] Among them, R is the rotation matrix; T is the translation vector; any point P in the laser radar coordinate system L The coordinates of (x L ,y L ,z L );P C (x C ,y C ,z C ) is the point coordinate in the visual camera coordinate;

[0099] S4.2, converting the data of the camera coordinate system to the image coordinate system;

[0100]

[0101] Among them, P O (x O ,y O ) is the coordinate point P of the visual camera following the pinhole model principle C (x C ,y C ,z C ) is the projection point in the image coordinate system; f is the focal length of the visual camera;

[0102] S4.3. Convert the data in the image coordinate system to the pixel coordinate system;

[0103]

[0104] Among them, u and v are the midpoint P of the laser radar coordinate system L (x L ,y L ,z L ) coordinates in the pixel coordinate system; u O 、v O is the coordinate of the optical center O in the pixel coordinate system; dx and dy are the quantized values ​​from the image coordinate system to the pixel coordinate system;

[0105] S4.4. By combining the above calculation process, the relationship between the lidar coordinate system and the pixel coordinate system can be obtained when the data of the visual camera coordinate system is converted to the pixel coordinate system;

[0106] The data of the visual camera coordinate system is converted to the pixel coordinate system as follows:

[0107]

[0108] The relationship between the lidar coordinate system and the pixel coordinate system:

[0109]

[0110] Among them, K is the camera intrinsic parameter.

[0111] S5, pass the obtained external parameter matrix to the laser point cloud projection to the image module, and use the external parameter matrix to convert the three-dimensional target detection result P obtained by the laser radar target detection module into L (x L ,y L ,z L ), projected into the pixel coordinate system (u, v) of the camera image, so that the target within the image detection frame has the spatial coordinate information measured by the lidar, and the initial point cloud 3D detection frames of the radar detection targets are obtained.

[0112] S6. Finally, the lidar information and visual information fusion module uses the Hungarian matching algorithm to optimally match the tracking sequence of the lidar's point cloud 3D detection frame and the visual camera's image detection frame, fuses the point cloud 3D detection frame and the image detection frame, and classifies all image detection frames and point cloud 3D detection frames according to whether the fused image detection frame and point cloud 3D detection frame are matched, and outputs the fusion result. The image detection frame and point cloud 3D detection frame with higher reliability are selected as the final fusion result for output and secondary screening.

[0113] like Figure 4 As shown, let the image detection frame in the current frame be c i ∈C i (i=1,2,3...m), the point cloud 3D detection box in the current frame is d j ∈D j (j=1,2,3...n), the coordinates of the center point of the image detection frame are (x ci ,y ci ), the coordinates of the center point of the point cloud 3D detection frame are (x dj ,y dj ), calculate the relative distance between the image detection frame and the point cloud 3D detection frame

[0114] The LiDAR and visual information fusion module uses the Hungarian matching algorithm and the relative distance to construct an association matrix to calculate the optimal match between the image detection frame and the point cloud 3D detection frame. It then performs a secondary screening of the unmatched image detection frames and point cloud 3D detection frames. The secondary screening includes the following steps:

[0115] S6.1, extract the unmatched lidar point cloud 3D detection box, if the point cloud 3D detection box is marked with ID in the target tracking module, it is considered that the image occlusion causes the point cloud 3D detection box to be unmatched, and the point cloud 3D detection box of the lidar is output;

[0116] S6.2, extract the unmatched image detection box of the visual camera, if the image detection box is marked with ID in the target tracking module, and there is point cloud projection in the image detection box, it is considered that the target distance is too far to cause the mismatch, and the image detection box of the visual camera is output.

[0117] S6.3, if the above two conditions are not met, it is considered that the matching fails, and no output operation is performed.

[0118] Therefore, the automatic driving target detection system and the optimization method based on multi-sensor fusion in the vehicle networking environment enhance the extraction ability of key features, improve the convergence speed and positioning accuracy of visual detection on target vehicles, guarantee the lightweight demand of the model, and avoid the loss and false detection of detection results.

[0119] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application and not to limit them, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that: it can still modify or equivalently replace the technical solutions of the present application, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. An automatic driving target detection system based on multi-sensor fusion in a vehicle networking environment, characterized in that: The application relates to a laser radar and visual camera joint calibration module, a laser radar target detection module, a visual camera target detection module, a target tracking module, a laser radar and visual information fusion module and a laser radar point cloud projection to an image module. The visual camera target detection module improves a YOLOv7-Tiny model, the improved YOLOv7-Tiny model performs convolution and up-sampling operations on a P3 layer in a Neck layer, increases the size of a feature map, and is spliced with a F2 layer in a backbone layer which is rich in detailed information, then fully mixes image features through an ELAN-Tiny module, and outputs an additionally constructed feature map P2, wherein a CBAM attention mechanism module is introduced, channel attention modules and spatial attention modules are introduced into a convolutional neural network, and the attention ability of the model to features is enhanced. The laser radar target detection module improves a two-dimensional convolution down-sampling module in a backbone network through a ConvNeXt module, and embeds a SimAM attention module in the backbone network, so that three-dimensional target detection is performed on social vehicles and other automatic driving vehicles to obtain a point cloud 3D detection frame. The target tracking module performs Hungarian matching between tracked targets and newly detected targets between adjacent frames of detection results of the laser radar target detection module and the visual camera target detection module, calculates the relative distance between the target detection frame of the last frame and the target detection frame of the current frame, calculates the difference between the number of detected targets in the last frame and the number of detected targets in the current frame, and establishes a correlation matrix by using the obtained relative distance value. The laser radar information and visual information fusion module adopts a Hungarian matching algorithm to optimally match the projected laser radar point cloud 3D detection frame, the image detection frame of the visual camera and the tracking sequences of the two, performs secondary screening on the unmatched image detection frame and the point cloud 3D detection frame, excludes the image detection frame and the point cloud 3D detection frame which are not in the tracking sequence and cannot be matched with each other, and finally screens the image detection frame and the point cloud 3D detection frame with high reliability as the final fusion result for output. The secondary screening comprises: extracting the unmatched laser radar point cloud 3D detection frame, if the point cloud 3D detection frame is marked with an ID in the target tracking module, it is considered that the image is blocked, the point cloud 3D detection frame is not matched, and the point cloud 3D detection frame of the laser radar is output; extracting the unmatched image detection frame of the visual camera, if the image detection frame is marked with an ID in the target tracking module and the image detection frame has a point cloud projection, it is considered that the target is too far away and is not matched, and the image detection frame of the visual camera is output; if the above two conditions are not met, it is considered that the matching fails, and no output operation is performed. 2.The automatic driving target detection system based on multi-sensor fusion in a vehicle Internet environment according to claim 1, characterized in that: The laser radar visual camera joint calibration module performs joint calibration on the visual camera and the laser radar based on a joint calibration algorithm of the visual camera and the laser radar, transforms the measurement values of the visual camera and the laser radar to a coordinate system, obtains a joint calibration result of the visual camera and the laser radar, that is, an extrinsic matrix, and usually needs to consider the intrinsic parameters and extrinsic parameters of the sensors during joint calibration. The laser point cloud projection to image module projects the three-dimensional target detection result obtained by the laser radar target detection module into the pixel coordinate system of the camera image by using the extrinsic parameter matrix obtained by the joint calibration module, to obtain the initial point cloud 3D detection frame of each radar detection target.

3. The optimization method of automatic driving target detection based on multi-sensor fusion in a vehicle networking environment, using the automatic driving target detection system based on multi-sensor fusion in a vehicle networking environment according to any one of claims 1-2, characterized in that, The method comprises the following steps: S1, detecting and identifying the target in the collected image by the visual camera target detection module; S2, performing three-dimensional target detection on the collected point cloud data by the laser radar target detection module; S3, establishing a tracking sequence of the laser radar and the visual camera by the target tracking module, and performing optimal matching of the front and rear frame targets by using the Hungarian matching algorithm; S4, performing joint calibration on the visual camera and the laser radar by the laser radar and visual camera joint calibration module, to obtain the joint calibration result of the visual camera and the laser radar, i.e., the extrinsic parameter matrix; S5, transmitting the obtained extrinsic parameter matrix to the laser point cloud projection to image module, and projecting the three-dimensional target detection result obtained by the laser radar target detection module into the pixel coordinate system of the camera image by using the extrinsic parameter matrix, to obtain the initial point cloud 3D detection frame of each radar detection target; S6, finally, the laser radar information and visual information fusion module performs optimal matching on the tracking sequence of the point cloud 3D detection frame of the laser radar and the image detection frame of the visual camera by using the Hungarian matching algorithm, fuses the point cloud 3D detection frame and the image detection frame, classifies all the image detection frames and the point cloud 3D detection frames according to whether the fused image detection frame and the point cloud 3D detection frame are matched, and outputs the fusion result, and selects the image detection frame and the point cloud 3D detection frame with high reliability as the final fusion result for output and secondary screening.

4. The optimization method for automatic driving target detection based on multi-sensor fusion in a vehicle Internet environment according to claim 3, characterized in that: In S2, the laser radar target detection module improves the Point Pillars algorithm, adopts a ConvNeXt to improve the two-dimensional convolution downsampling module in the backbone network, the ConvNeXt module is used for feature extraction, it aggregates feature information at multiple scales through top-down downsampling and corresponding upsampling, enhances the feature extraction capability of the algorithm, and embeds a SimAM attention module in the backbone network, the SimAM attention module works by inferring three-dimensional attention weights in the feature map, does not increase additional network parameters, and has the characteristic of plug and play.

5. The optimization method for automatic driving target detection based on multi-sensor fusion in a vehicle Internet environment according to claim 3, characterized in that: The target tracking in S3 comprises the following steps: S3.1, calculating the relative distance of the target detection frame of the previous frame and the target detection frame of the current frame; ； wherein, is a relative distance; is a target detection frame of a previous frame, wherein the center point coordinates are ; is a target detection frame of a current frame, wherein the center point coordinates are ; S3.2, calculating the difference between the number of detected targets in the previous frame and the number of detected targets in the current frame; ； wherein, is the difference; is the number of targets detected in the previous frame; is the number of targets detected in the current frame; S3.3, establishing an association matrix by using the obtained relative distance value; ； wherein is the incidence matrix; S3.4, constructing a matching mathematical model based on the Hungarian algorithm; ； wherein, is a data association algorithm, is a binary matrix marking whether a target detection box is a perfect match; when is 1, it means that the target detection box of the previous frame is a perfect match with the target detection box of the current frame, when is 0, it means that the target detection box of the previous frame is not a match with the target detection box of the current frame.

6. The optimization method for automatic driving target detection based on multi-sensor fusion in a vehicle Internet environment according to claim 3, characterized in that: The joint calibration in S4 comprises the following steps: S4.1, converting from the laser radar coordinate system to the camera coordinate system; ； wherein, is a rotation matrix; is a translation vector; coordinates of any point in the LiDAR coordinate system are ; are the point coordinates in the vision camera coordinates; S4.2, converting the data of the camera coordinate system to the image coordinate system; ； wherein, is the focal length of the vision camera; is the projection point in the image coordinate system; is the focal length of the vision camera; S4.3, converting the data of the image coordinate system to the pixel coordinate system; ； wherein , is the point in the laser radar coordinate system is the coordinate in the pixel coordinate system; , is the coordinate of the optical center O in the pixel coordinate system; , is the quantization value from the image coordinate system to the pixel coordinate system; S4.4, through the comprehensive calculation process can be obtained by the above-mentioned visual camera coordinate system data conversion to the pixel coordinate system, the relationship between the laser radar coordinate system and the pixel coordinate system; The data conversion of the visual camera coordinate system to the pixel coordinate system is: ； The relationship between the laser radar coordinate system and the pixel coordinate system is: ； wherein, is the camera intrinsic parameter.

Images