Multi-vehicle cooperative detection method based on image and v2v message fusion

Abstract

The present application relates to the technical field of intelligent transportation, and discloses a multi-vehicle cooperative detection method based on image and V2V message fusion, comprising: constructing a multi-vehicle cooperative detection system model, which comprises a plurality of intelligent networked vehicles; based on the multi-vehicle cooperative detection system model, obtaining a vehicle surrounding environment graph, and performing discrete depth layer division on the vehicle surrounding environment graph to extract first BEV features; based on the multi-vehicle cooperative detection system model, obtaining V2V messages of the intelligent networked vehicles, extracting position and direction angle information of vehicles around the intelligent networked vehicles, and performing coordinate conversion to extract second BEV features; according to the second BEV features, defining a mask to perform feature fusion on the first BEV features and the second BEV features to generate fusion features; the method improves the detection reliability and accuracy of surrounding vehicles.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, in particular to a multi-vehicle cooperative detection method based on image and V2V message fusion. BACKGROUND

[0002] Internet of Vehicles (IoV) refers to the interconnection of vehicles, infrastructure and the Internet, enabling information exchange to provide a wide range of services for drivers and passengers. Intelligent and Connected Vehicles (ICVs) as a component of IoV combine the communication capabilities of Vehicle-to-Vehicle (V2V), Vehicle-to-Pedestrian and Vehicle-to-Infrastructure (V2I), where the infrastructure refers to RoadSide Units (RSUs) on both sides of the road, which use cameras, sensors and other data sources to identify and locate various objects in the vehicle or entities in the vehicle environment. The application of intelligent and connected vehicles aims to improve the overall safety, efficiency and autonomy of vehicles. By accurately detecting and identifying objects in real time, intelligent and connected vehicles can make informed decisions and take appropriate actions to prevent collisions, bypass obstacles and improve overall driving experience. In the past decade, intelligent and connected vehicles have developed rapidly under the impetus of deep learning technology, and many scholars have applied deep learning to the development of intelligent and connected vehicles. Environmental perception technology, as one of the key technologies essential for the intelligentization and networking of vehicles, has also been fully developed. Environmental perception aims to comprehensively obtain the driving conditions around the vehicle to help drivers and intelligent driving systems make decisions, reduce or eliminate human-caused traffic accidents and improve travel efficiency.

[0003] Modern intelligent and connected vehicles are usually equipped with cameras, radars and LiDARs to perceive the surrounding environment. However, these sensors have obvious limitations. Although cameras can provide high-resolution vehicle surrounding environment maps that are closer to human vision, vehicle surrounding environment maps lack sufficient three-dimensional spatial information, and objects are usually presented in two-dimensional form, making it difficult to accurately obtain depth information. In contrast, LiDAR and millimeter wave radar can directly obtain three-dimensional information of objects, but due to their low resolution and high equipment cost, practical deployment faces economic challenges.

[0004] To solve the above problems, researchers have devoted to multi-modal fusion detection. First, by integrating data from different sensors, the shortcomings of different sensors can be made up, complement each other, and the accuracy and robustness of environmental perception can be improved. However, these works focus on single vehicle perspective, and in non-line-of-sight conditions, the sensors equipped by a single vehicle cannot fully obtain the surrounding scene information; second, intelligent connected vehicles communicate through on-board units (OBU), and use OBU to directly exchange information between vehicles, such as speed, position and driving direction, which can help vehicles perceive the surrounding driving conditions. However, current vehicle target detection has not considered the information obtained from the Internet of Vehicles, and V2V messages have not been effectively utilized, and the degree of networking of vehicles is low, and only using vehicle surrounding environment map and point cloud data may lead to missed detection of vehicles or pedestrians in non-line-of-sight, which indirectly leads to the decline of target detection integrity. SUMMARY

[0005] In view of the above problems in the prior art, the present application provides a multi-vehicle cooperative detection method based on image and V2V message fusion, which is used to solve the defects that the traditional visual vehicle detection precision in the prior art multi-modal fusion detection method is low due to the obstruction of obstacles and the V2V messages are not fully utilized.

[0006] In order to achieve the above application purpose, the technical scheme adopted by the present application is:

[0007] The multi-vehicle cooperative detection method based on image and V2V message fusion comprises the following steps:

[0008] S1, a multi-vehicle cooperative detection system model is constructed, which comprises a plurality of intelligent connected vehicles;

[0009] S2, based on the multi-vehicle cooperative detection system model, the vehicle surrounding environment map is obtained, and the vehicle surrounding environment map is divided into discrete depth layers to extract the first BEV feature;

[0010] S3, based on the multi-vehicle cooperative detection system model, the V2V message of the intelligent connected vehicle is obtained, the position and direction angle information of the vehicle around the intelligent connected vehicle is extracted and coordinate conversion is performed to extract the second BEV feature;

[0011] S4, according to the second BEV feature, the first BEV feature and the second BEV feature are fused by defining a mask to generate a fusion feature.

[0012] The present application has the following advantages:

[0013] The multi-vehicle cooperative detection method based on image and V2V message fusion provided in the present application, by constructing a multi-vehicle cooperative detection system model, using an intelligent connected vehicle to shoot a vehicle surrounding environment map, and combining V2V messages transmitted by a vehicle-mounted unit, through the fusion of structured message data and the vehicle surrounding environment map for vehicle-road cooperative detection, the surrounding vehicle detection reliability is improved, the problem that the traditional visual vehicle detection precision is greatly affected by weather conditions and image quality is solved, and the influence of obstacles is reduced, most importantly, combined with V2V messages, wireless communication can be carried out, visual obstruction is avoided, thereby helping the intelligent connected vehicle to obtain the state of the surrounding vehicles, not only the advantages of V2V messages can be fully utilized, but also the surrounding vehicle detection accuracy is improved, and the influence of obstacles is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 A flowchart of the multi-vehicle cooperative detection method based on image and V2V message fusion provided in the present application is shown.

[0015] Figure 2 A schematic diagram of the application scene of the multi-vehicle cooperative detection system model in the embodiment is shown.

[0016] Figure 3 A schematic diagram of the fusion process of the first BEV feature and the second BEV feature in the embodiment is shown.

[0017] Figure 4 A schematic diagram of the view frustum point cloud generated by the vehicle surrounding environment map is shown.

[0018] Figure 5 A schematic diagram of the BSM message format in the embodiment is shown.

[0019] Figure 6 A schematic diagram of the comparison results of the intelligent connected vehicle at different scales in the embodiment is shown.

[0020] Figure 7 A schematic diagram of the intersection over union results of the intelligent connected vehicle at different scales and different resolutions in the embodiment is shown.

[0021] Figure 8 A schematic diagram of the frame number change of the intelligent connected vehicle at different scales and different resolutions in the embodiment is shown.

[0022] Figure 9 A schematic diagram of the camera distribution of the intelligent connected vehicle in the embodiment is shown.

[0023] Figure 10 A schematic diagram of the coordinate conversion process of the V2V message in the embodiment is shown. DETAILED DESCRIPTION

[0024] The specific embodiments of the present application are described below to facilitate the understanding of the present application for those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all the inventions utilizing the concept of the present application are within the scope of protection.

[0025] As shown in Figure 1 The multi-vehicle cooperative detection method based on image and V2V message fusion includes the following steps S1-S4:

[0026] S1, a multi-vehicle cooperative detection system model is constructed, which includes a plurality of intelligent connected vehicles.

[0027] Specifically, in step S1, the intelligent connected vehicles are each equipped with a plurality of cameras and a vehicle-mounted unit.

[0028] In this embodiment, Figure 2 The application scenario of the multi-vehicle cooperative detection system model is shown, and the scenario is considered to be an urban road environment, which envelops a plurality of intelligent connected vehicles, and each intelligent connected vehicle is equipped with a plurality of cameras (vehicle-mounted cameras) for shooting vehicle surrounding environment maps. Specifically, as shown in Figure 9 Figure 9 The distribution of the cameras installed on each intelligent connected vehicle is shown, i.e. each intelligent connected vehicle is equipped with six cameras (surrounding view cameras), which are installed on the front, right front, left front, rear, right rear, and left rear of the vehicle, respectively, so as to collect six vehicle surrounding environment maps.

[0029] Meanwhile, each intelligent connected vehicle is also equipped with a vehicle-mounted unit for V2V message exchange with surrounding intelligent connected vehicles, i.e. intelligent connected vehicles driving on the road periodically broadcast V2V messages, and vehicles exchange information through such V2V messages, which are encoded into binary messages according to the ASN.1 standard. After receiving the message, other vehicles first decode and restore the original message through the communication protocol stack, and then parse the vehicle ID, position, direction angle, etc. fields in the message to extract valid information for subsequent steps and fusion detection with the vehicle surrounding environment maps obtained by the vehicle-mounted cameras.

[0030] S2, based on the multi-vehicle cooperative detection system model, the vehicle surrounding environment maps are obtained, and the vehicle surrounding environment maps are divided into discrete depth layers to extract the first BEV feature.

[0031] ​In this embodiment, for the six 2D vehicle surrounding environment maps of the intelligent connected vehicle surrounding environment collected by the input ring view camera, the 3D view cone point cloud features are generated by predicting the discrete depth distribution of the vehicle surrounding environment map feature points, so as to project the view cone point cloud features to the BEV grid by using the camera parameters, aggregate multi-view data, and finally form the first BEV feature, thereby realizing direct image generation of spatially aligned BEV features and improving the geometric accuracy and multi-view fusion consistency of image perception. The benefits of adding discrete depth are: discrete depth can avoid continuous estimation error amplification and improve projection accuracy.

[0032] Specifically, step S2 specifically includes S21-S26:

[0033] S21, using several cameras of the intelligent connected vehicle, collecting several vehicle surrounding environment maps of the intelligent connected vehicle.

[0034] In this embodiment, Figure 3 The vehicle surrounding view is displayed, that is, each intelligent connected vehicle uses six ring view cameras around the vehicle body to take pictures, and six 2D vehicle surrounding environment maps can be taken, which are the vehicle surrounding view of the intelligent connected vehicle. Since the picture taken by a monocular camera is a process of converting 3D to 2D, the vehicle information obtained through the picture inevitably loses depth information, and the depth information is the distance information between an object and an observation point (such as a camera), which is an important part of the vehicle position in a 3D space. In order to better perceive the vehicle position information, it is necessary to obtain the depth value of each pixel in the vehicle surrounding environment map, and the present application selects to predict the probability distribution of each pixel on multiple discrete depth layers, and calculates the depth by weighting the probability distribution.

[0035] S22, inputting each vehicle surrounding environment map into an image encoder, setting a depth starting value d0, a depth interval Δ, and a discrete depth layer K, and then performing feature extraction and depth discretization on all pixel points of each vehicle surrounding environment map to generate semantic features and depth probabilities of each pixel point.

[0036] Specifically, the formula for depth discretization in step S22 is:

[0037] D={d0,d0+Δ,…,d0+(K-1)Δ}

[0038] Where D represents the discrete depth of each pixel point.

[0039] In this embodiment, as shown in Figure 3 The vehicle surrounding environment map is input into the image encoder, and 2D image feature extraction is performed first. For each pixel point in the vehicle surrounding environment map, a semantic feature and a depth probability Where C=64 is the number of semantic feature output channels, and K=41 represents the number of discrete depth layers.

[0040] S23. Weight the semantic features of each pixel with the depth probability to generate weighted semantic features for each pixel, i.e.:

[0041] c p,d =α d ·c

[0042] Among them, c p,d Let α represent the weighted semantic features of a pixel at depth d, where c represents the semantic features of the pixel. d This represents the depth probability of a pixel with depth d.

[0043] In this embodiment, the depth probability α of a pixel with depth d is... d That is, the probability distribution of each pixel in the image along the discrete depth d direction, where d is the depth value and satisfies ∑ d α d =1, at each depth d, define feature c p,d =α d If c, then each pixel p at each depth d has a corresponding feature c. p,d , represents the weighted semantic feature of pixel p at depth d.

[0044] S24. Generate a view frustum cloud based on the weighted semantic features of each pixel.

[0045] In this embodiment, each image of the vehicle's surrounding environment is discretized into K layers to obtain a view frustum cloud, meaning that a view frustum cloud has K layers.

[0046] Therefore, steps S22-S23 above constitute the generation process of the view frustum point cloud, the principle of which is: to collect six images with a width of W... img H is the height img The vehicle's surrounding environment map is used as input. Figure 3 The image encoder shown in this invention is a ResNet network. Then, in the depth direction, starting from the initial depth distance d0, K discrete depth values ​​are generated at depth intervals of Δ, so that a final K×H image can be generated. img ×W img The view frustum cloud (3D view frustum cloud) is obtained by discretizing the depth of each image of the vehicle's surrounding environment into K layers. A view frustum cloud consists of K layers, and the coordinates of each pixel in each layer are (h, w, d), where h is the x-axis coordinate, w is the y-axis coordinate, and d is the depth value corresponding to that pixel. Specifically:

[0047] likeFigure 4 as shown, Figure 4 The generated view frustum point cloud process is shown, wherein since the RGB picture is composed of three color channels, each picture of a vehicle surrounding environment map with a width of W img and a height of H img can be represented as is a real set, in order to construct a 3D view frustum point cloud, for each pixel point p(h, w) in each vehicle surrounding environment map, by setting K = 41 discrete depth layers D = {d0, d0+Δ,…,d0+(K-1)Δ}, Δ is the depth interval, that is, K discrete intervals are divided in the depth direction, each interval interval Δ, for example, d0=4n, interval Δ=1m, then the depth range is 4~45m; finally, each pixel point p corresponds to K 3D points {h, w, d} in the point cloud space, and H img ×W img pixel points on the 2D vehicle surrounding environment map finally form a 3D view frustum point cloud with a size of K×H img ×W img .

[0048] S25, each pixel point in the view frustum point cloud is converted into a pixel point in the 3D world coordinate system through the intrinsic matrix and the extrinsic matrix of the camera.

[0049] In this embodiment, in order to project the view frustum point cloud obtained in the previous step onto the BEV plane, thereby generating a grid BEV feature, that is, a first BEV feature, it is necessary to first convert the pixel point p(h, w, d) into a point (x, y, z) in the 3D world coordinate system, and this process depends on the intrinsic matrix I and the extrinsic matrix E of the camera, wherein the camera intrinsic matrix I projects the point (x c , y c , z c ) in the camera coordinate system into the image plane (h, w), and its form is: f x and f y are the focal lengths of the camera in the x and y directions, (c x , c y ) are the x and y axis coordinates of the origin of the camera coordinate system in the vehicle environment map plane; the camera extrinsic matrix E converts the point (x, y, z) in the world coordinate system into the camera coordinate system (x c , y c , z c ), and its form is: R 3×3 is a rotation matrix, that is, the rotation of the world coordinate system to the camera coordinate system, t 3×1is a translation vector, i.e. the translation from the world coordinate system to the camera coordinate system; therefore, the specific process of converting each pixel point in the frustum point cloud into a pixel point in the 3D world coordinate system through the intrinsic matrix I and the extrinsic matrix E of the camera is as follows:

[0050] Specifically, step S25 specifically includes S251-S253:

[0051] S251, based on the intrinsic matrix of the camera, convert each pixel point to the normalized plane in the camera coordinate system, i.e.

[0052]

[0053] wherein u and v respectively represent the x and y axis coordinates of each pixel point in the normalized plane in the camera coordinate system, h and w represent the x and y axis coordinates of the pixel point in the frustum point cloud (also refer to the x and y axis coordinates of the pixel point in the vehicle environment map, i.e. after depth discretization, the x and y axis coordinates of the pixel point in the vehicle environment map do not change, only the depth value is added), c x and c y respectively represent the x and y axis coordinates of the origin of the camera coordinate system in the vehicle surrounding environment map plane, f x and f y respectively represent the focal length of the camera in the x and y axis directions.

[0054] S252, based on the depth, obtain the pixel point with depth information in the camera coordinate system, i.e.

[0055]

[0056] wherein x c , y c , and z c respectively represent the x, y, and z axis coordinates of the pixel point with depth information in the camera coordinate system.

[0057] S253, based on the extrinsic matrix of the camera, convert the pixel point with depth information in the camera coordinate system to the 3D world coordinate system, i.e.

[0058]

[0059] wherein E represents the extrinsic matrix of the camera, x, y, and z respectively represent the x, y, and z axis coordinates of each pixel point in the 3D world coordinate system, R represents the rotation matrix, T represents the transpose, and t represents the translation vector.

[0060] S26, divide the BEV grid, project the pixel point in the 3D world coordinate system into each BEV grid, and sum all the pixel points in each BEV grid for pool processing, to finally generate the first BEV feature.

[0061] Specifically, the formula for sum-pooling all the pixel points of each BEV grid in step S26 is:

[0062]

[0063] wherein BEV x,y represents the feature point after sum-pooling, represents the center coordinate of a certain BEV grid, represents the range of the BEV grid centered at represents the coordinate of the pixel point actually falling into a certain BEV grid.

[0064] In this embodiment, since the pixel point coordinate after the coordinate conversion in the above step is (x, y, z), and the height z can be generally ignored in the BEV, the finally generated BEV feature has a dimension of 1 in the z-axis direction, i.e., a first BEV feature map with a size of N x N is generated. The specific generation process is as follows: the BEV is divided into N x N grids or Pillars, each coordinate-converted pixel point is divided into the nearest grid, in order to aggregate the features of all the pixel points in the grid into a unified representation, the features of all the pixel points in each grid are sum-pooled while the feature information of all the pixel points is retained and the influence of the salient features is enhanced, the final result of the sum-pooling is that only one feature value is left in each BEV grid, which is the final BEV feature value, and then the BEV feature values of all the BEV grids constitute the first BEV feature.

[0065] S3, based on the multi-vehicle cooperative detection system model, obtaining the V2V message of the intelligent connected vehicle, extracting the position and direction angle information of the surrounding vehicles of the intelligent connected vehicle and performing coordinate conversion, and extracting the second BEV feature.

[0066] In this embodiment, based on the V2V message, the relative position map of the surrounding intelligent connected vehicles is obtained through the latitude and longitude coordinates and the direction angle of the surrounding intelligent connected vehicles, i.e., the process of obtaining the second BEV feature, so as to further enrich the scene perception dimension and make up for the deficiency of the single modal of the image. The V2V message is the Basic Safety Message (BSM), which contains the dynamic state information of the vehicle, such as the current position coordinate (latitude and longitude), direction angle, etc. The BSM message format is as follows: Figure 5 ​As shown, according to the definition in the “Cooperative Intelligent Transport System Dedicated Short-Range Communication Part 3: Network Layer and Application Layer Specification” (GB / T 31024.3-2019) published by the State Administration for Market Regulation and the State Administration for Standardization in 2019, the parameter lat is the latitude value, the northern latitude is positive, the southern latitude is negative, the value range is -900000000~900000001, the resolution is 1E-7°, and the value 900000001 represents unknown or invalid; the parameter long is the longitude value, the east longitude is positive, the west longitude is negative, the resolution is 1E-7°, and the value range is -1799999999~1800000001, and the value 1800000001 is unknown or invalid; the parameter heading is the direction angle (also known as the heading angle) of the vehicle or the traffic participant, which is the clockwise angle between the motion direction and the positive north direction. The unit is 0.0125°. Therefore, according to the above V2V message, the driving conditions around the vehicle can be obtained to perform the second BEV feature extraction, specifically as follows:

[0067] Specifically, step S3 specifically includes S31-S36:

[0068] S31, using the on-board unit of the intelligent connected vehicle, collecting V2V messages of each intelligent connected vehicle, which includes the latitude and longitude and the direction angle of the intelligent connected vehicle.

[0069] S32, setting a center intelligent connected vehicle, and taking the center intelligent connected vehicle as the coordinate origin, performing unit conversion on the latitude and longitude of the surrounding intelligent connected vehicles, that is:

[0070]

[0071] Wherein, ΔY represents the number of meters in the latitude direction, ΔLat represents the latitude difference obtained by subtracting the latitude of the origin from the latitude of the surrounding intelligent connected vehicle, L φ represents the distance corresponding to each degree of latitude on the earth, ΔX represents the number of meters in the longitude direction, ΔLon represents the longitude difference obtained by subtracting the longitude of the origin from the longitude of the surrounding intelligent connected vehicle, cos represents the cosine function, and lat current represents the current latitude.

[0072] In this embodiment, the center intelligent connected vehicle is taken as the coordinate origin, and the coordinates are (x self =0, y self =0), and the direction angle is θ self =0. However, since the latitude and longitude of the surrounding intelligent connected vehicles are in degrees (°), but the final detection result needs to be displayed in a coordinate system in meters (m) level, the above formula is used to convert the latitude and longitude of the surrounding intelligent connected vehicles into meters, wherein L φ =111320 (m / °), and cos(lat current) is used to correct the influence of the earth latitude. Since the earth is an ellipsoid, the distance corresponding to 1 degree of longitude is different at different latitudes. Therefore, in order to correct the change of the distance of each degree of longitude due to the difference in latitude, it is necessary to multiply cos (current latitude) to adjust.

[0073] S33, the direction angle of the surrounding intelligent connected vehicle is rotated and transformed, that is:

[0074]

[0075] wherein (x rot ,y rot ) represents the coordinate point of the surrounding intelligent connected vehicle after rotation transformation, R0 represents the rotation matrix, Δθ represents the angle difference, θ self represents the angle of the center intelligent connected vehicle rotating clockwise relative to the positive north direction, θ other represents the angle of the surrounding intelligent connected vehicle rotating clockwise relative to the positive north direction, (x org ,y org ) represents the original coordinate point of the surrounding intelligent connected vehicle before rotation transformation.

[0076] In this embodiment, Figure 10 the process of coordinate conversion is shown. The orange vehicle framed by the red frame is defined as the center intelligent connected vehicle, which can also be called the ego vehicle. In the perspective of the center intelligent connected vehicle, the detected vehicles around the center intelligent connected vehicle are defined as the surrounding intelligent connected vehicles. Since the original rotation angle is based on the positive north, the purpose of rotation transformation is to take the orientation of the center intelligent connected vehicle as the reference, take the center intelligent connected vehicle as the coordinate origin, and uniformly convert the position coordinates of the surrounding intelligent connected vehicles to the relative coordinate system with the orientation of the ego vehicle as the reference, so that the spatial relationship between multiple vehicles is expressed under the same direction reference, which is convenient for subsequent steps of BEV fusion. However, since the surrounding intelligent connected vehicles have a volume, they cannot be regarded as a point and only the center point is calculated, but the four corners (vertices) need to be calculated, as shown in step S34.

[0077] S34, the center point, the length and the width of the surrounding intelligent connected vehicle are obtained to obtain the coordinates of the four corners of the vehicle body rectangle of the surrounding intelligent connected vehicle relative to the center point before rotation, and the coordinates of the four corners of the vehicle body rectangle of the surrounding intelligent connected vehicle after rotation are calculated, that is:

[0078]

[0079]

[0080] wherein Rotated CornersCorners represents the coordinates of the four corners of the vehicle body rectangle of the surrounding intelligent connected vehicle relative to the center point before rotation, L veh , W veh represents the length and width of the surrounding intelligent connected vehicle, (x0, y0) represents the center point coordinates of the surrounding intelligent connected vehicle, (x1, y1), (x2, y2), (x3, y3), (x4, y4) respectively represent the coordinates of the four corners of the vehicle body rectangle after rotation.

[0081] In this embodiment, the purpose of drawing the vehicle body rectangle of the surrounding intelligent connected vehicle is to visually display the vehicle space occupation in the BEV space and provide geometric priori and visual basis for subsequent vehicle detection.

[0082] S35, divide the BEV grid, and set each BEV grid value to 0;

[0083] In this embodiment, since the above step obtains the N*N BEV feature according to the vehicle surrounding environment map, in order to enable the subsequent step to fuse the vehicle surrounding environment map and the V2V message, it is necessary to convert the V2V message into an N*N BEV feature of the same size, therefore, in this embodiment, N*N BEV grids are also set when the BEV grid is divided, and all grid values are set to 0, so at this time the BEV grid is only a zero matrix, and is not the final second BEV feature, and the process of generating the second BEV feature is: covering the divided BEV grid to the vehicle body rectangle after rotation in the above step, checking which grid points fall within the vehicle body rectangle after rotation obtained in the above step, falling within the vehicle body rectangle indicates that the grid is occupied by the vehicle, indicating that there is a vehicle at this position, then the BEV grid value at this position can be set to 1, checking whether all grid points fall within the vehicle rectangle, and finally obtaining the second BEV feature, the specific operation is as follows:

[0084] S36, covering the divided BEV grid to the vehicle body rectangle of the surrounding intelligent connected vehicle after rotation, setting the value of the BEV grid falling within the vehicle body rectangle to 1, and finally generating the second BEV feature, which is specifically:

[0085] S361, determining the minimum and maximum coordinates of the rectangle surrounding the surrounding intelligent connected vehicle parallel to the BEV grid, that is:

[0086] x min = min(x1, x2, x3, x4), x max = max(x1, x2, x3, x4)

[0087] y min = min(y1, y2, y3, y4), y max= max(y1, y2, y3, y4)

[0088] wherein (x min , y min ) represents the minimum coordinates of the rectangle surrounding the surrounding intelligent connected vehicle parallel to the BEV grid, and (x max , y max ) represents the maximum coordinates of the rectangle surrounding the surrounding intelligent connected vehicle parallel to the BEV grid.

[0089] In this embodiment, by determining the minimum and maximum coordinates of the rectangle surrounding the surrounding intelligent connected vehicle parallel to the BEV grid, the approximate coverage area of the vehicle in the BEV grid can be quickly located, which facilitates subsequent efficient filling of the BEV feature map and improves the accuracy and computational efficiency of grid projection.

[0090] S362, the center coordinates of each BEV grid are calculated, i.e.:

[0091]

[0092] wherein i and j represent the column index and row index of the BEV grid respectively, l represents the length or width of each BEV grid, and N represents the total length or total width of all BEV grids, represents the center coordinates of each BEV grid.

[0093] S363, each BEV grid is sequentially numbered from small to large, and then the center coordinates of a BEV grid are selected in turn in the order from small to large.

[0094] S364, it is judged whether the center coordinates of the BEV grid satisfy the relationship formula If yes, step S365 is executed, otherwise, step S363 is executed.

[0095] S365, a ray is drawn from the center coordinates of the BEV grid to the right horizontal direction, it is judged whether the ray intersects with each edge of the surrounding intelligent connected vehicle, if yes, step S366 is executed, otherwise, step S363 is executed.

[0096] In this embodiment, since the angle of the intelligent connected vehicle may not be parallel to the grid line, the grid center coordinates cannot be used to judge whether it is in the vehicle body rectangle, but a ray can be drawn from the center coordinates of each BEV grid to the right horizontal direction to judge whether the ray intersects with each edge of the surrounding intelligent connected vehicle, so as to judge whether the grid is in the vehicle body rectangle, and the specific operation is as follows:

[0097] Specifically, the specific process of judging in step S365 whether the ray intersects with each edge of the center intelligent connected vehicle is as follows:

[0098] Step1, each edge of the surrounding intelligent connected vehicle is represented by the coordinates of any adjacent two corners of the rotated body rectangle of the surrounding intelligent connected vehicle, that is:

[0099]

[0100] wherein, V ′ represents the edge composed of the corners V i1 and V i1+1 , V i1 (x i1 , y i1 ) represents the x-axis coordinate and y-axis coordinate of the i1th corner, V i1+1 (x i1+1 , y i1+1 ) represents the x-axis coordinate and y-axis coordinate of the i1+1th corner.

[0101] Step2, determine whether the y-axis coordinates of the adjacent two corners of each edge are not equal, that is, y i1 ≠ y i1+1 , if yes, execute Step3, otherwise, the ray has no intersection with the edge.

[0102] Step3, calculate the x-axis coordinate of the intersection point of the ray and the edge, that is:

[0103]

[0104] wherein, x intersect represents the x-axis coordinate of the intersection point, represents the y-axis coordinate of the center coordinate point of the BEV grid.

[0105] Step4, determine whether the y-axis coordinate of the center coordinate point of the BEV grid is within the y-axis coordinate range of the edge, that is: if yes, execute Step5, otherwise, the ray has no intersection with the edge.

[0106] Step5, determine whether the intersection point of the ray and the edge is on the right side of the ray, that is if yes, the ray has one intersection point with the edge, otherwise, the ray has no intersection with the edge; wherein, represents the x-axis coordinate of the center coordinate point of the BEV grid.

[0107] S366, the number of intersection points of the ray and each edge of the surrounding intelligent connected vehicle is obtained, if the number of intersection points is odd, the center coordinate point of the BEV grid is in the vehicle body rectangle of the surrounding intelligent connected vehicle, and the value of the BEV grid is set to 1, if the number of intersection points is even, the center coordinate point of each BEV grid is outside the vehicle body rectangle of the surrounding intelligent connected vehicle, the value of the BEV grid is unchanged, and finally the second BEV feature is generated.

[0108] S4, according to the second BEV feature, the first BEV feature and the second BEV feature are fused by defining a mask to generate a fusion feature.

[0109] In this embodiment, for the target detection task, the vehicle sending the V2V message is equivalent to the vehicle position known, because the V2V message contains the position coordinates, direction angle and other driving information of the vehicle, and since the real road scene will contain non-connected vehicles, the position information of all vehicles cannot be obtained. In summary, the second BEV feature obtained from the V2V message is the vehicle position known area, and the first BEV feature obtained from the image is the predicted area, that is, the known area needs to be used to assist in detecting vehicles. The reason why the present application does not directly add the two is that direct addition will cause information confusion, thereby destroying the integrity of the data and the training effect of the model. The value of the second BEV feature known area obtained from the V2V message is a true value, and the value of the unknown area is zero. If direct addition is performed, the true value of the known area will be covered by the prediction result obtained from the image, resulting in information loss. On the other hand, the first BEV feature obtained from the vehicle surrounding environment map is a prediction value, which is usually a probability value between 0 and 1. Direct addition will cause the value of the known area to exceed 1, resulting in a mismatch in the value. Therefore, in order to clearly distinguish between the known area and the unknown area and avoid the problems caused by direct addition, the present application selects to use a mask to distinguish between the known area and the unknown area, so that the data in different areas can be processed in a targeted manner during the fusion process. The specific operation is as follows:

[0110] Specifically, step S4 specifically includes S41-S42:

[0111] S41, based on the second BEV feature, a mask is defined, that is:

[0112]

[0113] wherein G V represents the second BEV feature, and mask represents the mask area.

[0114] In this embodiment, it is assumed that the second BEV feature obtained from the V2V message is G V , and the first BEV feature obtained from the vehicle surrounding environment map is G c , and the known area of the above step is GV In this embodiment, the value of the known region is non-zero, and the value of the unknown region is zero, so a mask is defined, the known region is defined as 1, and the unknown region is defined as 0, so as to mark which region is known and which region is unknown.

[0115] S42, based on the mask, the first BEV feature is fused with the second BEV feature to generate a fusion feature, that is:

[0116] G fused = mask G V + (1-mask) G c

[0117] Wherein, G fused represents the fusion feature, A c represents the first BEV feature

[0118] In this embodiment, in order to verify the effectiveness of the multi-vehicle cooperative detection method based on image and V2V message fusion proposed in the present application, the following experiments are performed:

[0119] First, the nuScenes dataset is used, which is a large dataset composed of vehicle surrounding environment maps and point cloud data of one thousand scenes, each scene lasting 20 seconds, wherein the vehicle surrounding environment map comes from six cameras respectively deployed in front, right front, left front, rear, right rear and left rear of the vehicle. Since the nuScenes does not contain V2V messages, the position, direction angle and other dynamic information of the vehicle cannot be obtained, therefore the vehicle annotation information provided by the dataset is used to obtain the vehicle surrounding top view, for each frame of vehicle annotation information, the 3D bounding box can be converted to the BEV perspective, and the vehicle region is mapped to the BEV plane through coordinate transformation, and finally the driving situation around the vehicle is obtained, which can be used instead of the second BEV feature obtained from the V2V message. Considering that the real road scene is not all vehicles are intelligent networked cars, the proportion of intelligent networked cars is set to 0, 0.2, 0.4, 0.6, 0.8 and 1 respectively in the experiment. The second BEV feature obtained from the V2V message can be obtained by setting part of the region to zero, first considering two extreme cases, one is that the number of intelligent networked cars on the road is zero, at this time there is no additional information to assist, only relying on the pictures of the six cameras to detect vehicles; the other case is that all the vehicles on the road are intelligent networked cars, at this time the vehicle obtains the V2V message of the surrounding intelligent networked cars and extracts the position, direction angle and other useful information, which can completely draw the BEV graph without pictures.

[0120] At the same time, for the evaluation of the method proposed in the present application, an indicator commonly used in the target detection field, Intersection over Union (IoU), is selected, which is usually used to measure the degree of overlap between two regions, and the calculation formula is:

[0121]

[0122] Wherein, A is the target region predicted by the method proposed in this invention, B is the target region labeled in the ground truth, TP is the number of correctly classified pixels (True Positives), FP is the number of misclassified pixels (False Positives), FN is the number of pixels not correctly classified into the category (False Negatives), A∩B represents the overlapping part between the predicted and ground truth regions (i.e., TP), and A∪B represents the merged region between the predicted and ground truth regions (i.e., TP+FP+FN).

[0123] Then, in order to simulate real road scenarios, the proportions of intelligent connected vehicles were adjusted. Set to different values, when At this time, it means that all vehicles on the road are non-intelligent connected vehicles, and there is no information exchange between them; when This means that all vehicles on the road are connected and intelligent vehicles, allowing for the acquisition of their location information. To simulate deployment in a vehicle-to-everything (V2X) network, this experiment was not trained on multiple GPUs, but rather ran on a single NVIDIA RTX 3090 (24GB) graphics card. The CPU used was a 14vCPU Intel(R) Xeon(R) Platinum 8362 CPU@2.80GHz, built on Ubuntu 18.04, using ResNet as the backbone network. All experiments were trained for 150k steps, with validation performed every 10k steps. The batch size was set to 4, the number of worker threads used for data loading was set to 10, the Adam optimizer was used, and the learning rate was 1×10⁻⁶. -3 The weight decays to 1×10 -7 For object segmentation, a binary cross-entropy loss is used with a positive sample weight of 1.0. For lane segmentation, the positive sample weight is 5.0. Ground truth targets are generated by projecting 3D bounding boxes onto the BEV plane. The transformation layer in the nuScenes map is converted to the vehicle coordinate system and rasterized to identify the areas where vehicles can drive and lane boundaries. Specific simulation parameters are shown in Table 1.

[0124] Table 1 Simulation Parameter Settings

[0125]

[0126]

[0127] Based on the above experimental setup, the experimental results are as follows: Figure 6 As shown, Figure 6 Showcasing different proportions of intelligent connected vehicles The comparison results below, where is located Figure 6 The green vehicle in the center is the data collection vehicle, and the surrounding vehicles are the vehicles being tested. Figure 6 The right side shows the test results, from top to bottom: the proportion of intelligent connected vehicles. The values ​​are set to 0, 0.5, and 0.9 respectively. The green rectangle represents the data collection vehicle, the blue rectangle represents the detected vehicle, and the orange area represents the road. Figure 6 The top and bottom sides show the actual shooting process, clearly demonstrating the situation around the vehicle. The area circled in red in the right-hand inspection results shows the impact of obstructions on the inspection results. At this time, the method proposed in this invention only relies on images to detect surrounding vehicles. When there is an obstruction in front, the obstructed vehicle cannot be detected. As can be seen from the above real-world footage, there is more than one vehicle directly in front, but due to obstruction, the second obstructed vehicle cannot be clearly identified in the image. In this case, if the obstructed vehicle is a connected vehicle, it will send a V2V message to the vehicles behind, thus revealing its location. and As the test results show, obscured vehicles can be detected very well; on the other hand, the areas circled in orange demonstrate the impact of distance on the detection results. Sometimes, due to the distance between the vehicle and the detector, the detection results are poor, and in some cases, the vehicle cannot be detected at all. If the vehicle at a distance is a connected car, such as... and As the detection results show, the vehicle's position can be detected very well.

[0128] Furthermore, by changing the resolution of the input image, the impact of the addition of intelligent connected vehicles on the detection results can be observed, and the results are as follows: Figure 7 As shown, it can be observed that the detection performance at low resolution (64*176) significantly impacts the performance of intelligent connected vehicles. Extremely sensitive, from arrive The IoU (Interval of Units) improved from 0.181 to 0.919, a nearly five-fold increase. This is because low-resolution images lose details, such as small objects in the distance, while intelligent connected vehicles compensate for the blind spots of a single vehicle by sharing multi-view data through V2V (vehicle-to-vehicle) communication. At high resolution (256*704), the improvement is even more significant. The IoU remains at 0.342 because high-resolution images themselves contain rich features, providing sufficient information and reducing reliance on intelligent connected vehicle assistance. Furthermore, across all resolutions, as the proportion of intelligent connected vehicles... The improvement in both the IoU and the addition of intelligent connected vehicles significantly enhances the detection performance of the proposed method.

[0129] Simultaneously, the detection frame rate of the method proposed in this invention was calculated, i.e., the number of frames processed per second (FPS); since the V2V messages are fused regardless of... The FPS didn't change much regardless of the setting, so we divided it into two scenarios: one with intelligent connected vehicles. and unconnected intelligent vehicles The scene. From Figure 8 As can be seen, the inclusion of V2V messages from connected vehicles introduces additional computational overhead for data processing. Therefore, while the detection accuracy is higher with the participation of connected vehicles, the overall FPS of the proposed method decreases. Thus, in practical applications, a trade-off between detection accuracy and efficiency is necessary. In summary, connected vehicles are key to improving accuracy at low resolution, but at the cost of reduced FPS, while high resolution can reduce reliance on connected vehicles.

[0130] In summary, the multi-vehicle cooperative detection method based on image and V2V message fusion proposed in this invention constructs a multi-vehicle cooperative detection system model, utilizes images of the vehicle's surrounding environment captured by intelligent connected vehicles, and combines this with V2V messages transmitted by the onboard unit. By fusing structured message data with the vehicle's surrounding environment image, vehicle-road cooperative detection is performed, improving the reliability of surrounding vehicle detection. This solves the problem that the accuracy of traditional visual vehicle detection is greatly affected by weather conditions and image quality, while also reducing the impact of obstacle occlusion. Most importantly, by combining V2V messages, wireless communication is possible, avoiding visual obstruction, thereby helping intelligent connected vehicles obtain the status of surrounding vehicles. This not only fully utilizes the advantages of V2V messages but also improves the accuracy of surrounding vehicle detection and reduces the impact of obstacles.

[0131] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0132] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A multi-vehicle cooperative detection method based on image and V2V message fusion, characterized in that, The method comprises the following steps: S1, constructing a multi-vehicle cooperative detection system model, which comprises a plurality of intelligent connected vehicles; S2, based on the multi-vehicle cooperative detection system model, obtaining a vehicle surrounding environment map, and discretely dividing the depth layer of the vehicle surrounding environment map to extract first BEV features; S3, based on the multi-vehicle cooperative detection system model, obtaining V2V messages of the intelligent connected vehicle, extracting position and direction angle information of vehicles around the intelligent connected vehicle, and performing coordinate conversion to extract second BEV features; S4, according to the second BEV features, defining a mask, and performing feature fusion on the first BEV features and the second BEV features to generate fusion features, specifically: S41, based on the second BEV features, defining a mask, that is: wherein, represents a second BEV feature, represents a mask region; S42, based on the mask, performing feature fusion on the first BEV features and the second BEV features to generate fusion features, that is: wherein, represents a fusion feature, represents a first BEV feature.

2. The method of claim 1, wherein, In step S1, the intelligent connected vehicles are each provided with a plurality of cameras and a vehicle-mounted unit.

3. The method of claim 2, wherein, Step S2 specifically comprises: S21, using the plurality of cameras of the intelligent connected vehicle to collect a plurality of vehicle surrounding environment maps of the intelligent connected vehicle; S22, input the surrounding environment map of each vehicle into an image encoder, and perform feature extraction and depth discretization on all pixel points of the surrounding environment map of each vehicle by setting a depth start value , a depth interval , and a discrete depth layer , to generate semantic features and depth probabilities of each pixel point. S23, weighting the semantic features and depth probabilities of each pixel point to generate weighted semantic features of each pixel point, that is: wherein, represents a weighted semantic feature of a pixel point with a depth of represents a semantic feature of a pixel point, represents a depth probability of a pixel point with a depth of .​ S24, based on the weighted semantic features of each pixel point, generating a view frustum point cloud; S25, converting each pixel point in the view frustum point cloud into a pixel point in a 3D world coordinate system through an intrinsic matrix and an extrinsic matrix of the camera; S26, dividing BEV grids and projecting the pixel points in the 3D world coordinate system into each BEV grid, and performing sum pooling on all pixel points in each BEV grid to finally generate first BEV features.

4. The method of claim 3, wherein, The formula for depth discretization in step S22 is: wherein, represents the discrete depth of each pixel.

5. The method of claim 4, wherein, Step S25 specifically comprises: S251, based on the intrinsic matrix of the camera, converting each pixel point to a normalized plane in the camera coordinate system, that is: wherein, , respectively represent the normalized plane coordinates of each pixel point in the camera coordinate system, , axis coordinates, , represent the normalized plane coordinates of each pixel point in the frustum point cloud, , axis coordinates, , respectively represent the normalized plane coordinates of the camera coordinate system origin in the vehicle surrounding map plane, , axis coordinates, , respectively represent the focal length of the camera in the , axis direction; S252, based on the depth, obtaining a pixel point with depth information in the camera coordinate system, that is: wherein, , , respectively represent the coordinates of a pixel point with depth information in the camera coordinate system , , axis coordinates; S253, based on the extrinsic matrix of the camera, converting the pixel point with depth information in the camera coordinate system to a 3D world coordinate system, that is: wherein, represents an extrinsic matrix of the camera, , , respectively represent the 3D world coordinates of each pixel point of the 3D world coordinate system, , , axis coordinates, represents a rotation matrix, represents a transpose, represents a translation vector.

6. The method of claim 5, wherein, The formula for performing sum pooling on all pixel points in each BEV grid in step S26 is: in, Represents the feature points after summation and pooling. This represents the center coordinates of a certain BEV grid. Indicated by The BEV grid range centered on the center, This represents the coordinates of a pixel that actually falls within a certain BEV grid.

7. The method of claim 6, wherein, Step S3 specifically comprises: S31, using the vehicle-mounted unit of the intelligent connected vehicle to collect V2V messages of each intelligent connected vehicle, which includes the latitude, longitude and direction angle of the intelligent connected vehicle; S32, setting a center intelligent connected vehicle and taking the center intelligent connected vehicle as the coordinate origin to perform unit conversion on the latitude and longitude of the surrounding intelligent connected vehicles, that is: wherein, represents the number of meters in the latitude direction, represents a latitude difference obtained by subtracting the original point latitude from the latitude of the surrounding intelligent connected vehicle, represents a distance corresponding to each degree of latitude on the earth, represents the number of meters in the longitude direction, represents a longitude difference obtained by subtracting the original point longitude from the longitude of the surrounding intelligent connected vehicle, represents a cosine function, represents the current latitude; S33, performing rotation transformation on the direction angles of the surrounding intelligent connected vehicles, that is: wherein, represents a coordinate point of the surrounding intelligent connected vehicle after rotation transformation, represents a rotation matrix, represents an angle difference, represents an angle of clockwise rotation of the center intelligent connected vehicle relative to the positive north direction, represents an angle of clockwise rotation of the surrounding intelligent connected vehicle relative to the positive north direction, represents an original coordinate point of the surrounding intelligent connected vehicle before rotation transformation; S34, obtaining the center point, vehicle length and vehicle width of the surrounding intelligent connected vehicles to obtain the coordinates of the four corners of the vehicle body rectangle of the surrounding intelligent connected vehicles relative to the center point, and calculating the coordinates of the four corners of the vehicle body rectangle of the surrounding intelligent connected vehicles after rotation, that is: wherein, represents the coordinates of the four corners of the body rectangle of the surrounding intelligent connected vehicle relative to the center point after rotation, represents the coordinates of the four corners of the body rectangle of the surrounding intelligent connected vehicle relative to the center point before rotation, represents the length and width of the surrounding intelligent connected vehicle, represents the coordinates of the center point of the surrounding intelligent connected vehicle, respectively represent the coordinates of the four corners of the body rectangle of the surrounding intelligent connected vehicle after rotation.​​​​ S35, dividing BEV grids and setting each BEV grid value to 0; S36, cover the divided BEV grid to the rotated surrounding intelligent connected vehicle body rectangle, set the value of the BEV grid falling on the vehicle body rectangle to 1, and finally generate the second BEV feature.

8. The method of claim 7, wherein, Step S36 specifically includes: S361, determine the minimum and maximum coordinates of the rectangle surrounding the surrounding intelligent connected vehicle parallel to the BEV grid, that is: wherein, represents the minimum coordinates of a rectangle surrounding the surrounding intelligent connected vehicles parallel to the BEV grid, represents the maximum coordinates of a rectangle surrounding the surrounding intelligent connected vehicles parallel to the BEV grid; S362, calculate the center coordinates of each BEV grid, that is: wherein, , represents the column index, the row index of the BEV grid, respectively, represents the length or width of each BEV grid, represents the total length or total width of all BEV grids, represents the center coordinate of each BEV grid; S363, sequentially number each BEV grid from small to large, and then select the center coordinates of a BEV grid in order from small to large; S364, judge whether the center coordinate of the BEV grid satisfies the relationship formula , if yes, execute step S365, otherwise, execute step S363; S365, draw a ray from the center coordinate point of the BEV grid to the right horizontal direction, and judge whether the ray intersects with each edge of the surrounding intelligent connected vehicle, if yes, execute step S366, otherwise, execute step S363; S366, get the intersection point number of the ray with each edge of the surrounding intelligent connected vehicle, if the intersection point number is odd, the center coordinate point of the BEV grid is in the vehicle body rectangle of the surrounding intelligent connected vehicle, and the value of the BEV grid is set to 1, if the intersection point number is even, the center coordinate point of each BEV grid is outside the vehicle body rectangle of the surrounding intelligent connected vehicle, the value of the BEV grid is unchanged, and finally the second BEV feature is generated.

9. The method of claim 8, wherein, The specific process of judging whether the ray intersects with each edge of the surrounding intelligent connected vehicle in step S365 is: Step 1, express each edge of the surrounding intelligent connected vehicle with the coordinates of any two adjacent corners of the rotated vehicle body rectangle of the surrounding intelligent connected vehicle, that is: wherein, represents a side composed of angles to , represents an axis coordinate of the first angle, axis coordinate, axis coordinate, represents an axis coordinate of the second angle, axis coordinate, axis coordinate; Step 2, judging whether the two adjacent angles of each edge are not equal, i.e. , if yes, executing Step 3, otherwise, the ray has no intersection with the edge. , if yes, executing Step 3, otherwise, the ray has no intersection with the edge. Step 3, calculating the intersection point of the ray and the edge axis coordinates, i.e. wherein, the intersection point of the axis and the plane axis coordinates, the center coordinate point of the BEV grid axis coordinates; Step4、judge whether the axis coordinate of the center coordinate point of the BEV grid is in the axis coordinate range of the edge, i.e. , if yes, execute Step5, otherwise, the ray has no intersection with the edge. Step5、judge whether the axis coordinate of the center coordinate point of the BEV grid is in the axis coordinate range of the edge, i.e. , if yes, execute Step6, otherwise, the ray has no intersection with the edge. Step6、judge whether the axis coordinate of the center coordinate point of the BEV grid is in the axis coordinate range of the edge, i Step 5, judging whether the intersection point of the ray and the edge is on the right side of the ray, i.e. , if yes, the ray intersects the edge once, otherwise, the ray does not intersect the edge; wherein, represents the axis coordinate of the center coordinate point of the BEV grid.

Images