A multimodal fusion detection method for intelligent connected vehicles based on vehicle-road collaboration

Abstract

The present invention relates to a vehicle-road collaborative detection method based on V2X communication and multi-sensor fusion, which belongs to the field of intelligent vehicle target detection. The method comprises the following steps: obtaining a vehicle-perspective road image, point cloud data, and a BEV road image at a traffic intersection, and obtaining a target detection result in the vehicle-perspective image; obtaining vehicle and pedestrian target detection results in a laser radar point cloud, and projecting the result onto a vehicle acquisition image plane; obtaining vehicle and pedestrian target detection results in a BEV image at a traffic intersection by using V2X communication technology; performing a perspective conversion on the BEV image at the traffic intersection by using a weight conversion method based on perspective transformation, and obtaining converted target information; correcting vehicle and pedestrian targets at the vehicle-perspective according to the converted target; and cross-modally fusing the corrected vehicle-perspective target with the projected point cloud target by constructing a convolutional neural network, and obtaining an accurate road target detection result.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent vehicle detection and relates to a multimodal fusion detection method for intelligent connected vehicles based on vehicle-road collaboration. Background Art

[0002] Intelligent connected vehicles rely on computer vision, deep learning, wireless network communication and other technologies, and are equipped with a variety of sensors, so that intelligent connected vehicles have basic functions such as environmental perception, path planning, decision control, etc., so as to achieve autonomous driving. Among them, accurate detection of vehicles, pedestrians and other targets in complex scenes is an important part of intelligent connected vehicles to achieve environmental perception, which can provide accurate scene information for autonomous navigation map construction, path planning, etc.

[0003] In order to realize target detection of intelligent connected vehicles, accurate road information can be obtained through on-board cameras and laser radars. However, in actual applications, the detection results of sensors at the same viewing angle are often relatively simple and not robust. In addition, when intelligent connected vehicles are driving at intersections in real complex traffic environments with heavy traffic and people, detection errors may occur, which in turn leads to wrong decisions of intelligent connected vehicles and creates safety hazards. At present, a large number of studies have been conducted on the detection technology of intelligent connected vehicles at home and abroad, but most detection algorithms only consider the single-vehicle single-sensor or single-vehicle multi-sensor situations on highways or simple roads, and there is little research on vehicle-road collaborative detection under complex road conditions at traffic intersections. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a vehicle-road cooperative detection method based on V2X communication and multiple sensors to realize the application of precise target detection of intelligent networked vehicles at complex traffic intersections. The method uses V2X communication technology and vehicle-mounted cameras, vehicle-mounted laser radars and road traffic cameras to provide a solution for target detection of intelligent networked vehicles, enabling them to achieve accurate target detection in complex environments, thereby improving their safety.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A multimodal fusion detection method for intelligent connected vehicles based on vehicle-road collaboration, when an intelligent connected vehicle drives to a certain distance range of a complex traffic intersection with a traffic monitoring camera, the intelligent connected vehicle detects objects within its own range in combination with its own driving needs, specifically including the following steps:

[0007] S1: Obtain road images and point cloud data, and use deep learning target detection methods to obtain vehicle, non-motor vehicle, and pedestrian detection results in the image; pre-process the point cloud data, and extract features of the target in the point cloud according to the features of the target to be detected;

[0008] S2: Based on the detection results of vehicle images and the target features extracted from the point cloud, the L2 attention voxel network is used to obtain accurate point cloud data of road targets, realize three-dimensional detection of targets at traffic intersections, classify vehicles and pedestrians in the point cloud, and obtain accurate point cloud object detection results;

[0009] S3: Use V2X communication technology to obtain real-time video data from traffic monitoring cameras at traffic intersections, extract frames from the video data to construct image data; use deep learning target detection methods to obtain vehicle, non-motorized vehicle, and pedestrian detection results in the image;

[0010] S4: In step S4, the intelligent connected vehicle is positioned using the position of the intelligent connected vehicle and the distance between the intelligent connected vehicle and the camera at the intersection, and the target detection result of the on-board camera of the intelligent connected vehicle is corrected by using V2X communication technology and a weight transfer method based on perspective transformation;

[0011] S5: Build a fusion network to perform cross-modal fusion on the precise radar point cloud obtained in step S3 and the corrected detection result in step S4 to obtain the precise traffic road conditions from the perspective of the intelligent connected vehicle.

[0012] Further, the step S1 specifically includes the following steps:

[0013] S11: Use cameras and lidar to obtain road images and point cloud data;

[0014] S12: Preprocess the image data, use the nearest neighbor interpolation algorithm to calculate and map the input pixels to the non-integer coordinates of the input image, and use the median filter to eliminate image noise; use the road condition dataset KITTI to train the YOLOv5 neural network, and use the trained model to obtain the image-based road target detection results; at the same time, convert the target data in the image to the image coordinate system through the following formula to obtain the target position:

[0015]

[0016] in, and Represent the homogeneous coordinates of the midpoint in the world coordinate system and the image coordinate system respectively, s is the scaling factor, G is the intrinsic parameter of the camera, and [R|t] is the extrinsic parameter of the camera, including the rotation parameter R and the translation parameter t;

[0017] S13: Based on the prior knowledge of the traffic target, the point cloud data in the rough area is first clustered, and then the targets other than the traffic target are filtered out in the rough area to obtain a more accurate point cloud area containing the traffic target;

[0018] S14: In the point cloud area obtained in step S13, based on the shape characteristics and point cloud reflection intensity that distinguish traffic targets from other targets, feature extraction is performed on targets in the point cloud that resemble motor vehicles, non-motor vehicles, pedestrians, etc., and a rough point cloud traffic target detection result is obtained.

[0019] Further, the step S2 specifically includes the following steps:

[0020] S21: Based on the image detection results, the position of the intelligent networked vehicle is taken as the dot to capture the point cloud data and image in the front half of the intelligent networked vehicle. Then, based on the correspondence between the image and the point cloud, the points in the point cloud of the area are projected onto the image using the following transformation formula:

[0021]

[0022] Among them, Ax+By+Cz+D=0 represents the three-dimensional coordinate space, n=[x,y,z] and n0=[x0,y0,z0] represent the point cloud coordinates and the coordinates of any point in the three-dimensional coordinate system respectively, and N=[A,B,C] is the plane normal vector in the image coordinate system; the points that can be projected on the image are potential points where traffic targets exist, and these points are aggregated into a rough area containing traffic targets;

[0023] S22: Add L2 attention to the voxel network, add an attention layer before each output layer, calculate the L2 norm normalization for each target point cloud, and use it as the L2 attention weight; multiply the normalized point cloud target vector by the weight matrix to obtain a weighted vector, align the weighted vector with the target annotation point cloud and evaluate it to obtain a score, then apply the score to the softmax function to obtain a probability distribution, multiply it with the original point cloud vector and sum it to obtain the final point cloud target;

[0024] S23: Based on the rough traffic target detection result point cloud area obtained in step S21, it is evenly divided into multiple 3D grids, i.e., voxels, which are input into the L2 attention voxel network for feature processing, convolution, upsampling, downsampling, and removing false detection results, and obtaining accurate point cloud-based traffic target detection results under the L2 normalized attention constraint.

[0025] Further, the step S3 specifically includes the following steps:

[0026] S31: using V2X communication technology to obtain video data collected by a traffic monitoring camera at a traffic intersection, and performing frame extraction processing on the video data to obtain BEV image data at the traffic intersection;

[0027] S32: Using the obtained traffic intersection BEV image data to train the YOLOv5 neural network, and using the trained model to obtain road target detection results based on the traffic intersection image.

[0028] Further, the step S4 specifically includes the following steps:

[0029] S41: Estimating the position of the intelligent connected vehicle:

[0030] 1) Using laser radar, obtain the distance between the intelligent connected car and the traffic camera at the traffic intersection in the current frame;

[0031] 2) Estimating and verifying the current position of the intelligent connected vehicle through the position of the previous frame and the distance traveled by the intelligent connected vehicle between frames;

[0032] S42: The real-time data of the traffic camera at the traffic intersection is transmitted to the intelligent networked vehicle through V2X communication technology. The target detection results of the vehicle image data and the traffic camera image data in steps S12 and S32 are used to obtain the four corresponding pixel coordinate points of the specific target, and the corresponding transformation matrix of the current time frame and the next time frame is obtained through the variable distance perspective conversion formula; at the same time, the image transformation matrix of the next time frame is derived through each sub-matrix and the vehicle driving data information, and the conversion result of each target point in the next frame is obtained, so as to realize the perspective conversion of the traffic camera data and the vehicle camera data, and convert the perspective of the traffic camera data into the vehicle camera data; the variable distance perspective conversion formula is:

[0033]

[0034]

[0035]

[0036] in, is the average confidence of the detected target in time frame t, K is the transformation matrix, R is the linear transformation submatrix, S is the translation transformation submatrix, P is the perspective transformation submatrix, Q is the full scale transformation submatrix, i = [xy 1] and j = [uv w] represent the coordinate points of the vehicle image data and the traffic camera image data, Δ1 and Δ2 are the weights of these two factors, d t is the distance from the smart connected car to the traffic camera at time t, and h is the height of the traffic camera from the ground;

[0037] S43: Using a spatial feature fusion method, the targets detected in the converted traffic camera data frame and the common pedestrian targets and vehicle targets in the vehicle camera data frame are fused and corrected to obtain accurate target data and trust scores from the perspective of the intelligent connected car;

[0038] Further, the step S5 specifically includes the following steps:

[0039] S51: Convert the precise radar point cloud obtained in step S3 and the corrected detection result in step S4 into a set of consistent joint detection sparse vectors, where the corrected sparse vector of the detected target is I img ={x1,y1,x2,y2,x3,y3,x4,y4,score}, where x1,y1,x2,y2,x3,y3,x4,y4 are the coordinates of the four vertices of the detected target, and score is the confidence score; the sparse tensor of the precise lidar target point cloud is I lidar ={x,y,z,h,w,l,α,score}, where x,y,z are the coordinates of the center point of the target point cloud, h,w,l are the length, width and height of the point cloud range, α is the rotation angle of the point cloud target, and score is the confidence score; the constructed consistent detection joint sparse vector is I total ={score img ,score lidar ,cfr,l2}, cfr is the cross fusion ratio of the targets between two different modalities, which is specifically defined as:

[0040]

[0041] Where img is the image target, point is the point cloud target; a, b are the weight ratios of the targets in the image and point cloud respectively, and a+b=1; l2 is the L2 normalized distance between the two;

[0042] S52: Eliminate the consistent detection joint sparse vectors with empty IoU;

[0043] S53: Construct a fusion network to confirm the joint detection sparse vector and obtain the final detection target: use five convolutional layers to perform convolution operations on the joint detection sparse vector, and finally obtain a 1×n×1 tensor, where n is the number of non-empty targets of all detection joint sparse vectors; at the same time, for these non-empty targets, save their indexes to mark the targets; map the obtained tensor to the desired learning target, i.e., the probability score map, through the maximum pooling operation;

[0044] S54: According to the obtained probability score, the corresponding precise point cloud target is characterized as a projected point cloud sequence, the projected point cloud sequence is directly merged into the vehicle camera data frame together with the feature indicating its timestamp, and the projected point cloud is aligned with the image frame. The position and mode of the features are fused through the projection fusion mechanism and connectivity search, and finally cross-modal fusion is achieved to obtain the final high-precision target detection result.

[0045] The beneficial effects of the present invention are:

[0046] 1) The present invention uses existing and fixed traffic cameras at traffic intersections in the city as reference objects and data input, which can reduce the implementation costs such as installing specific markers, and at the same time pre-process the laser radar point cloud, avoiding the processing of huge data when using laser radar;

[0047] 2) Compared with other existing collaborative target detection methods, the present invention uses traffic camera data at traffic intersections on the road as an auxiliary to detect target correction. Traffic camera at traffic intersection is stable, in a fixed position for a long time, and there is no occlusion, which can well assist intelligent networked vehicles to accurately detect targets at traffic intersections with frequent situations;

[0048] 3) The present invention utilizes V2X communication technology and multi-sensor fusion, and for specific complex scenes at traffic intersections, according to the change in the distance between the intelligent connected vehicle and the traffic camera at the traffic intersection between the previous and next frames, a convolutional neural network is constructed to fuse the image data and the lidar point cloud data, which can effectively improve the accuracy and robustness of the target detection of the intelligent connected vehicle. Other advantages, objectives and features of the present invention will be explained to some extent in the subsequent description, and to some extent, will be obvious to those skilled in the art based on the investigation and research below, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be achieved and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0050] Figure 1 This is a main flow chart of the multimodal fusion detection method described in an embodiment of the present invention;

[0051] Figure 2 A diagram of the installation location of the equipment described in the embodiment of the invention;

[0052] Figure 3 It is a schematic diagram of the detection method described in the embodiment of the invention. DETAILED DESCRIPTION

[0053] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0054] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0055] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0056] See also Figure 1 to Figure 3 The present invention provides a multimodal fusion detection method for intelligent connected vehicles based on vehicle-road collaboration. First, it is assumed that the intelligent connected vehicle is at an intersection with a traffic monitoring camera. The intelligent connected vehicle needs to detect targets such as vehicles and pedestrians on the road in accordance with its own driving needs, including the following steps:

[0057] Step 1: Use the on-board camera and LiDAR to obtain road images and point cloud data. Use the deep learning traffic condition target detection method to obtain the detection results of the target of interest in the image. Then preprocess the point cloud data and extract the features of the target of interest in the point cloud; the specific steps include:

[0058] 101: Use cameras and laser radars to obtain road images and point cloud data. First, install a device bracket on the top of the smart connected car. The camera is a fixed video acquisition and is installed in front of the top of the smart connected car. The installation position is 1.79 meters from the ground and collects images in front of the vehicle at a rate of 30 frames per second. The laser radar is installed in the center of the roof of the smart connected car, 1.99 meters from the ground, and collects road point cloud data at a rate of 10 frames per second;

[0059] 102: Preprocess the image data, use the nearest neighbor interpolation algorithm to calculate and map the input pixels to the non-integer coordinates of the input image, and use the median filter to eliminate image noise; use the road condition dataset KITTI to train the YOLOv5 neural network, and use the trained model to obtain the image-based road target detection results; at the same time, convert the target data in the image to the image coordinate system through the following formula to obtain the target position:

[0060]

[0061] in, and Represent the homogeneous coordinates of the midpoint in the world coordinate system and the image coordinate system respectively, s is the scaling factor, G is the intrinsic parameter of the camera, and [R|t] is the extrinsic parameter of the camera, including the rotation parameter R and the translation parameter t;

[0062] 103: Based on the prior knowledge of traffic targets, the point cloud data in the rough area is first clustered, and then the targets other than traffic targets are filtered out in the rough area by point set counting filtering, height filtering and shape filtering. A more accurate point cloud area containing traffic targets is obtained;

[0063] 104: In the point cloud region obtained in step 103, based on the shape characteristics of the traffic target that are different from other targets and the point cloud reflection intensity, feature extraction is performed on the targets in the point cloud that are similar to motor vehicles, non-motor vehicles, pedestrians, etc., and a rough point cloud traffic target detection result is obtained.

[0064] In step 2, based on the image detection results of the vehicle-mounted camera and the target features extracted from the point cloud, an L2 attention voxel network is constructed to add attention distribution constraints to the point cloud target detection and reduce the gap in attention weights, making the detection model more stable and reliable, thereby detecting and classifying traffic targets in the point cloud and obtaining accurate point cloud traffic target detection results.

[0065] The specific steps include:

[0066] 201: According to the image detection results, with a radius of 30m and around the position of the intelligent connected vehicle itself as the origin, the point cloud data and image in the front half area of ​​the intelligent connected vehicle are captured, and then according to the correspondence between the image and the point cloud, the points in the point cloud of the area are projected onto the image through the following transformation formula.

[0067]

[0068] Among them, Ax+By+Cz+D=0 represents the three-dimensional coordinate space, n=[x,y,z] and n0=[x0,y0,z0] represent the point cloud coordinates and the coordinates of any point in the three-dimensional coordinate system respectively, and N=[A,B,C] is the plane normal vector in the image coordinate system. These points that can be projected on the image are considered to be potential points where traffic targets exist, and these points are aggregated into a rough area containing traffic targets.

[0069] 202: Construct and train an L2 attention voxel network. Add L2 attention to the voxel network, add an attention layer before each output layer, calculate the L2 norm normalization for each target point cloud, and use it as the L2 attention weight; multiply the normalized point cloud target vector with the weight matrix to obtain a weighted vector, align the weighted vector with the target annotated point cloud and evaluate it to obtain a score, then apply the score to the softmax function to obtain a probability distribution, multiply it with the original point cloud vector and sum it to obtain the final point cloud target.

[0070] 203: Based on the rough traffic target detection result point cloud area obtained in step 201, it is evenly divided into 3D grids, i.e., voxels, which are input into the L2 attention voxel network for further feature processing, convolution, upsampling, downsampling and other operations, and the false detection results are removed, and the accurate point cloud-based traffic target detection results are obtained under the L2 normalized attention constraint.

[0071] Step 3: Use V2X communication technology to obtain real-time video data from traffic monitoring cameras at traffic intersections, extract frames from the video data and construct image data. Use deep learning target detection methods to obtain detection results for vehicles, non-motorized vehicles, pedestrians, etc. in the image;

[0072] 301: using V2X communication technology to obtain video data collected by a traffic monitoring camera at a traffic intersection, and performing frame extraction processing on the video data to obtain BEV image data of the traffic intersection;

[0073] 302: Using the obtained traffic intersection BEV image data to train the YOLOv5 neural network, and using the trained model to obtain a road target detection result based on the traffic intersection image.

[0074] Step 4: Use the position of the intelligent connected vehicle and the distance between it and the intersection camera to locate the intelligent connected vehicle, and use V2X communication technology and a weight transfer method based on perspective transformation to correct the target detection results of the intelligent connected vehicle's onboard camera. The specific steps include:

[0075] 401: Estimate the position of the intelligent connected vehicle. 1) Since the traffic camera at the traffic intersection is fixed, the laser radar is used to obtain the distance between the intelligent connected vehicle and the traffic camera at the traffic intersection in the current frame; 2) Then, the current position of the intelligent connected vehicle is estimated and verified based on the position of the previous frame and the distance traveled by the intelligent connected vehicle in the time between frames, eliminating the interference of vehicles on the radar data, and waiting for more accurate position information.

[0076] 402: Transmitting the real-time data of the traffic camera at the traffic intersection to the intelligent connected vehicle through V2X communication technology, taking the four corresponding pixel coordinate points of the specific target by using the target detection results of the vehicle image data and the target detection results of the traffic camera image data in step 102 and step 302, and obtaining the corresponding transformation matrix of the current time frame and the next time frame by the variable distance perspective conversion formula; at the same time, deriving the image transformation matrix of the next time frame through each sub-matrix and the vehicle driving data information, obtaining the conversion result of each target point in the next frame, realizing the perspective conversion of the traffic camera data and the vehicle camera data, and converting the perspective of the traffic camera data into the vehicle camera data, wherein the variable distance perspective conversion formula includes:

[0077]

[0078]

[0079]

[0080] in, is the average confidence of the detected target in time frame t, K is the transformation matrix, R is the linear transformation submatrix, S is the translation transformation submatrix, P is the perspective transformation submatrix, Q is the full scale transformation submatrix, i = [xy 1] and j = [uv w] represent the coordinate points of the vehicle image data and the traffic camera image data, Δ1 and Δ2 are the weights of these two factors, d t is the distance from the smart connected car to the traffic camera at time t, and h is the height of the traffic camera from the ground.

[0081] 403: The targets detected in the converted traffic camera data frame are fused and corrected by the spatial feature fusion method, and the common pedestrian targets, vehicle targets, etc. in the vehicle-mounted camera data frame are fused and corrected to obtain accurate target data and trust scores under the traffic camera data.

[0082] Step 5: Build a multi-constraint fusion network to perform cross-modal fusion on the precise radar point cloud obtained in step 3 and the corrected detection results in step 4 to obtain the precise traffic road conditions from the perspective of the intelligent connected vehicle.

[0083] 501: Convert the precise radar point cloud obtained in step 302 and the corrected detection result in step 403 into a set of consistent joint detection sparse vectors. The corrected sparse vector of the detected target is I img ={x1,y1,x2,y2,x3,y3,x4,y4,score}, where x1,y1,x2,y2,x3,y3,x4,y4 are the coordinates of the four vertices of the detected target, and score is the confidence score. The sparse tensor of the precise LiDAR target point cloud is I lidar ={x,y,z,h,w,l,α,score}, where x,y,z are the coordinates of the center point of the target point cloud, h,w,l are the length, width and height of the point cloud range, α is the rotation angle of the point cloud target, and score is the confidence score. The constructed consistent detection joint sparse vector is I total ={score img ,score lidar ,cfr,l2}, cfr is the cross fusion ratio of the targets between two different modalities, specifically defined as Where img is the image target, point is the point cloud target; a, b are the weight ratios of the targets in the image and point cloud respectively, and a+b=1; l2 is the L2 normalized distance between the two;

[0084] 502: Since there is no one-to-one correspondence between the precise radar point cloud target and the corrected detection result when constructing a consistent joint detection sparse vector, the consistent detection joint sparse vector with an empty cfr is removed, that is, the joint sparse vectors that do not belong to the same semantics are deleted;

[0085] 503: Construct a fusion network to confirm the joint detection sparse vector and obtain the final detection target. Use five convolutional layers to perform convolution operations on the joint detection sparse vector, and finally obtain a 1×n×1 tensor, where n is the number of non-empty targets in all detection joint sparse vectors. At the same time, for these non-empty targets, save their indexes to mark the targets. Map the obtained tensor to the desired learning target, that is, the probability score map, through the maximum pooling operation;

[0086] 504: According to the obtained probability score, the corresponding precise point cloud target is characterized as a projected point cloud sequence, the projected point cloud sequence is directly merged into the vehicle camera data frame together with the feature indicating its timestamp, and the projected point cloud is aligned with the image frame. The position and mode of the features are fused through the projection fusion mechanism and connectivity search, and finally cross-modal fusion is achieved to obtain the final high-precision target detection result.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A multi-modal fusion detection method for intelligent connected vehicles based on vehicle-road collaboration, characterized in that: When an intelligent connected vehicle drives within a certain distance of a complex traffic intersection equipped with a traffic monitoring camera, the intelligent connected vehicle detects objects within its own range in accordance with its own driving needs, specifically including the following steps: S1: Obtain road images and point cloud data, and use deep learning target detection methods to obtain vehicle, non-motor vehicle, and pedestrian detection results in the image; pre-process the point cloud data, and extract features of the target in the point cloud according to the features of the target to be detected; S2: Based on the detection results of the vehicle collected images and the target features extracted from the point cloud, accurate point cloud data of the road target is obtained through the L2 attention voxel network, three-dimensional detection of targets at traffic intersections is achieved, vehicles and pedestrians in the point cloud are classified, and accurate point cloud object detection results are obtained; the step S2 specifically includes the following steps: S21: Based on the image detection results, the position of the intelligent networked vehicle is taken as the dot to capture the point cloud data and image in the front half of the intelligent networked vehicle. Then, based on the correspondence between the image and the point cloud, the points in the point cloud of the area are projected onto the image using the following transformation formula: Among them, Ax+By+Cz+D=0 represents the three-dimensional coordinate space, n=[x,y,z] and n0=[x0,y0,z0] represent the point cloud coordinates and the coordinates of any point in the three-dimensional coordinate system respectively, and N=[A,B,C] is the plane normal vector in the image coordinate system; the points that can be projected on the image are potential points where traffic targets exist, and these points are aggregated into a rough area containing traffic targets; S22: Add L2 attention to the voxel network, add an attention layer before each output layer, calculate the L2 norm normalization for each target point cloud, and use it as the L2 attention weight; multiply the normalized point cloud target vector by the weight matrix to obtain a weighted vector, align the weighted vector with the target annotated point cloud and evaluate it to obtain a score, then apply the score to the softmax function to obtain a probability distribution, multiply it with the original point cloud vector and sum it to obtain the final point cloud target; S23: Based on the rough traffic target detection result point cloud area obtained in step S21, it is evenly divided into multiple 3D grids, i.e., voxels, which are input into the L2 attention voxel network for feature processing, convolution, upsampling, downsampling, and removing false detection results, and obtaining accurate point cloud-based traffic target detection results under the L2 normalized attention constraint; S3: Use V2X communication technology to obtain real-time video data from traffic monitoring cameras at traffic intersections, extract frames from the video data and construct image data; use the YOLOv5 target detection method to obtain vehicle, non-motorized vehicle, and pedestrian detection results in the image; S4: In step S4, the intelligent connected vehicle is positioned using the position of the intelligent connected vehicle and the distance between the intelligent connected vehicle and the camera at the intersection, and the target detection result of the on-board camera of the intelligent connected vehicle is corrected by using V2X communication technology and a weight conversion method based on perspective transformation; S5: Build a fusion network to perform cross-modal fusion on the precise radar point cloud obtained in step S3 and the corrected detection result in step S4 to obtain the precise traffic road conditions from the perspective of the intelligent connected vehicle.

2. The multimodal fusion detection method for intelligent connected vehicles based on vehicle-road collaboration according to claim 1 is characterized in that: The step S1 specifically includes the following steps: S11: Use cameras and lidar to obtain road images and point cloud data; S12: Preprocess the image data, use the nearest neighbor interpolation algorithm to calculate and map the input pixels to the non-integer coordinates of the input image, and use the median filter to eliminate image noise; use the road condition dataset KITTI to train the YOLOv5 neural network, and use the trained model to obtain the image-based road target detection results; at the same time, convert the target data in the image to the image coordinate system through the following formula to obtain the target position: in, and Represent the homogeneous coordinates of the midpoint in the world coordinate system and the image coordinate system respectively, s is the scaling factor, G is the intrinsic parameter of the camera, and [R|t] is the extrinsic parameter of the camera, including the rotation parameter R and the translation parameter t; S13: Based on the prior knowledge of the traffic target, the point cloud data in the rough area is first clustered, and then the targets other than the traffic target are filtered out in the rough area to obtain a more accurate point cloud area containing the traffic target; S14: In the point cloud area obtained in step S13, based on the shape characteristics and point cloud reflection intensity that distinguish traffic targets from other targets, feature extraction is performed on targets in the point cloud that resemble motor vehicles, non-motor vehicles, pedestrians, etc., and a rough point cloud traffic target detection result is obtained.

3. The multimodal fusion detection method for intelligent connected vehicles based on vehicle-road collaboration according to claim 1 is characterized in that: The step S3 specifically comprises the following steps: S31: using V2X communication technology to obtain video data collected by a traffic monitoring camera at a traffic intersection, and performing frame extraction processing on the video data to obtain BEV image data at the traffic intersection; S32: Using the obtained traffic intersection BEV image data to train the YOLOv5 neural network, and using the trained model to obtain road target detection results based on the traffic intersection image.

4. The multimodal fusion detection method for intelligent connected vehicles based on vehicle-road collaboration according to claim 1 is characterized in that: The step S4 specifically comprises the following steps: S41: Estimating the position of the intelligent connected vehicle: 1) Using laser radar, obtain the distance between the intelligent connected car and the traffic camera at the traffic intersection in the current frame; 2) Estimating and verifying the current position of the intelligent connected vehicle through the position of the previous frame and the distance traveled by the intelligent connected vehicle between frames; S42: The real-time data of the traffic camera at the traffic intersection is transmitted to the intelligent networked vehicle through V2X communication technology. The target detection results of the vehicle image data and the traffic camera image data in steps S12 and S32 are used to obtain the four corresponding pixel coordinate points of the specific target, and the corresponding transformation matrix of the current time frame and the next time frame is obtained through the variable distance perspective conversion formula; at the same time, the image transformation matrix of the next time frame is derived through each sub-matrix and the vehicle driving data information, and the conversion result of each target point in the next frame is obtained, so as to realize the perspective conversion of the traffic camera data and the vehicle camera data, and convert the perspective of the traffic camera data into the vehicle camera data; the variable distance perspective conversion formula is: in, is the average confidence of the detected target in time frame t, K is the transformation matrix, R is the linear transformation submatrix, S is the translation transformation submatrix, P is the perspective transformation submatrix, Q is the full scale transformation submatrix, i = [xy 1] and j = [uvw] represent the coordinate points of the vehicle image data and the traffic camera image data, Δ1 and Δ2 are the weights of these two factors, d t is the distance from the smart connected car to the traffic camera at time t, and h is the height of the traffic camera from the ground; S43: The targets detected in the converted traffic camera data frame are fused and corrected by the spatial feature fusion method, and the common pedestrian targets and vehicle targets in the vehicle camera data frame are fused and corrected to obtain accurate target data and trust scores from the perspective of the intelligent connected vehicle.

5. The multimodal fusion detection method for intelligent connected vehicles based on vehicle-road collaboration according to claim 1 is characterized in that: The step S5 specifically comprises the following steps: S51: Convert the precise radar point cloud obtained in step S3 and the corrected detection result in step S4 into a set of consistent joint detection sparse vectors, where the corrected sparse vector of the detected target is I img ={x1,y1,x2,y2,x3,y3,x4,y4,score}, where x1,y1,x2,y2,x3,y3,x4,y4 are the coordinates of the four vertices of the detected target, and score is the confidence score; the sparse tensor of the precise lidar target point cloud is I lidar ={x,y,z,h,w,l,α,score}, where x,y,z are the coordinates of the center point of the target point cloud, h,w,l are the length, width and height of the point cloud range, α is the rotation angle of the point cloud target, and score is the confidence score; the constructed consistent detection joint sparse vector is I total ={score img ,score lidar ,cfr,l2}, cfr is the cross fusion ratio of the targets between two different modalities, which is specifically defined as: Where img is the image target, point is the point cloud target; a, b are the weight ratios of the targets in the image and point cloud respectively, and a+b=1; l2 is the L2 normalized distance between the two; S52: Eliminate the consistent detection joint sparse vector with empty IoU; S53: Construct a fusion network to confirm the joint detection sparse vector and obtain the final detection target: use five convolutional layers to perform convolution operations on the joint detection sparse vector, and finally obtain a 1×n×1 tensor, where n is the number of non-empty targets of all detection joint sparse vectors; at the same time, for these non-empty targets, save their indexes to mark the targets; map the obtained tensor to the desired learning target, i.e., the probability score map, through the maximum pooling operation; S54: According to the obtained probability score, the corresponding precise point cloud target is characterized as a projected point cloud sequence, the projected point cloud sequence is directly merged into the vehicle camera data frame together with the feature indicating its timestamp, and the projected point cloud is aligned with the image frame. The position and mode of the features are fused through the projection fusion mechanism and connectivity search, and finally cross-modal fusion is achieved to obtain the final high-precision target detection result.

Images