An automatic driving cooperative perception method based on image three-dimensional reconstruction

Abstract

The application discloses an automatic driving cooperative perception method based on image three-dimensional reconstruction and belongs to the technical field of intelligent traffic cooperative perception. The steps are as follows: collecting original images of vehicle-mounted cameras with labels to construct a data set, and dividing the data set into a training set and a test set; constructing a target detection network, designing a loss function, and training the target detection network by using the training set; inputting camera images of an automatic driving vehicle to be detected into the target detection network which has been pre-trained, and reconstructing pixel positions of a two-dimensional image output by the target detection network into three-dimensional positions in the real world; and fusing three-dimensional position information obtained from camera images of multiple automatic driving vehicles to obtain a final target detection result. The application can solve the problems of limited single vehicle perception distance and missing two-dimensional image depth information in existing vehicle detection and positioning technologies, and realizes sharing and fusion of detection results between vehicles through a core information communication strategy.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation cooperative perception technology, specifically relating to an autonomous driving cooperative perception method based on image 3D reconstruction. Background Technology

[0002] In recent years, autonomous driving technology, as a core direction in the field of intelligent transportation, has made significant progress, attracting widespread attention from academia and industry. One of the core challenges of autonomous driving systems is how to achieve reliable and safe driving in complex and dynamic environments. To meet this challenge, the system must possess a high level of environmental adaptability, among which image perception, as a key technology, has become an indispensable component of autonomous driving systems.

[0003] Traditional single-vehicle perception systems primarily rely on sensors installed on the vehicle, such as cameras, radar, and lidar, to perceive the surrounding environment. Among these, image sensors play a crucial role in environmental perception due to their ability to provide high resolution and rich visual information. However, these perception systems have inherent limitations: their field of view is restricted by occlusion and distance, and they are susceptible to adverse weather conditions. Furthermore, the performance of single-vehicle perception systems in complex and dynamic traffic scenarios often falls short of the requirements for efficient perception and decision-making. These limitations not only constrain the overall performance of autonomous driving systems but also affect their safety and reliability in practical deployments.

[0004] To address this issue, the concept of collaborative perception has been proposed in recent years. Collaborative perception improves the robustness and accuracy of a perception system by fusing data from multiple sensors. Through cooperation among multiple agents and the data fusion capabilities of multiple sensors, collaborative perception significantly enhances the robustness and accuracy of the perception system. Particularly in autonomous driving scenarios, collaborative perception can expand the perception range and improve the detection capability of occluded objects by sharing visual data among multiple agents, such as... Figure 1 As shown, this method can not only effectively reduce the perception blind spots of a single agent, but also improve the decision-making efficiency of the system in dynamic environments.

[0005] Image data fusion is a key step in collaborative perception. In autonomous driving scenarios, image data often originates from multiple cameras with different perspectives, resolutions, and shooting conditions. How to efficiently fuse this image data to form a comprehensive understanding of the environment is a current research hotspot. This process involves not only image stitching and correction but also the extraction of useful features from multi-source data and comprehensive analysis combined with data from other sensors. However, existing technologies still have limitations in terms of detection accuracy for small targets at long distances, affecting the performance of collaborative perception. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, this invention proposes a cooperative perception method for autonomous driving based on image 3D reconstruction. The method is rationally designed, overcomes the shortcomings of existing technologies, and has good results.

[0007] An autonomous driving cooperative perception method based on image 3D reconstruction includes the following steps:

[0008] Step 1: Collect labeled raw images from vehicle-mounted cameras to construct a dataset, and divide it into training and test sets;

[0009] Step 2: Construct an object detection network, design a loss function, and train and test the object detection network using training and testing sets;

[0010] Step 3: Input the camera image of the autonomous vehicle to be detected into the pre-trained target detection network, and reconstruct the pixel positions of the two-dimensional image output by the target detection network to the three-dimensional positions in the real world.

[0011] Step 4: Fuse the 3D position information obtained from camera images of multiple autonomous vehicles to obtain the final target detection result.

[0012] Furthermore, in step 1, images of multiple autonomous vehicles on multiple road sections are collected using vehicle-mounted cameras, or simulation tools are used to perform simulations. All the original images collected from the vehicle-mounted cameras are used to construct a dataset, and the dataset is divided into a training set and a test set according to the proportions.

[0013] Furthermore, in step 2, the target detection network includes a feature focusing module and a detection head, and the specific working process and loss function are as follows:

[0014] Step 2.1: After scaling and normalizing the images in the dataset, input them into the encoder. Multi-layer convolutional networks and feature pyramid networks are used to extract features, capturing object information at different scales. The encoder's output feature map is... Where Φ(·) is the feature extraction operation, It is the l-th image captured by the autonomous vehicle, l∈[1,n], n is the number of images, C is the depth, H is the height, and W is the width;

[0015] Step 2.2: Transfer the feature map The input is an adaptive average pooling layer, which aggregates the spatial information of the feature map into a single depth vector, representing the global semantics of each depth. The computation process is as follows: in, It is the global feature vector corresponding to the c-th layer depth. It is the feature vector in the i-th row and j-th column corresponding to the c-th layer depth;

[0016] Step 2.3, The depth weights are calculated by inputting two fully connected layers. The calculation process is as follows: in, Let c be the weight vector at depth c. ε is the ReLU activation function, and ε is the Sigmoid function. It is a depth-based global feature vector; it is weighted at the depth of each layer of the original feature map, and the calculation method is as follows: in, These are the features updated after the depth of layer c;

[0017] Step 2.4, let the feature map For feature maps Perform average pooling and max pooling respectively to obtain the feature map after average pooling. and the feature map after max pooling Will and The features are concatenated and then weighted using convolution operations to generate a feature map. The expression is: Where ι represents the convolution operation; forward propagation performs inference on the input feature map, utilizing the forward propagation function. The final output feature map is obtained.

[0018] Step 2.5: Weight the feature map by attention. The input detection head consists of two parts: non-maximum suppression (NMS) and scaled bounding boxes. NMS works as follows: first, detection boxes exceeding a confidence threshold are filtered out; the remaining boxes are grouped according to their predicted categories; within each category, boxes are sorted from highest to lowest confidence; the highest-confidence boxes are retained; and the cross-union ratio (CUI) of the remaining boxes is calculated. Based on the CUI, boxes exceeding a threshold are removed. Scaled bounding boxes are calculated first in the image processed by the detection head. and the original input image The scaling ratio is determined, then the coordinates are inversely scaled, and the bounding box on the original image is generated based on the center coordinates and size of the inversely scaled image.

[0019] The detection head ultimately outputs an information vector I = {I1, I2, ..., I...} of n target detection boxes. n}, where the information vector I of the i-th target detection box i =(δ,θ,t) u ,t v ,w,h), where δ is the target category, θ is the detection box confidence, and t u t vThe coordinates of the center of the detection box are represented by w, where w is the width of the detection box and h is the height of the detection box.

[0020] Step 2.6: Taking into account the object detection classification loss, object detection bounding box regression loss, and object confidence loss, a loss function is constructed. Where L1 is the object detection classification loss, L2 is the bounding box regression loss, L3 is the object confidence loss, and λ is the target confidence loss. i It is the weighting coefficient of the loss.

[0021] Furthermore, step 3 includes the following sub-steps:

[0022] Step 3.1: Let O1-X1Y1Z1 represent the world coordinate system and O2-X2Y2Z2 represent the camera coordinate system. When the camera rotates around the X1, Y1, and Z1 axes of the world coordinate system, it forms the roll angle α, pitch angle β, and yaw angle γ, respectively, thus obtaining three rotation matrix components R. α R β R γ Then we get the rotation matrix R = R α R β R γ S = [S1, S2, S3] T Let S1, S2, and S3 be the translation vectors, representing the translation components of a point on the X1, Y1, and Z1 axes, respectively. After rotation and translation, the relationship is obtained. Where U1 = (x1, y1, z1, 1) and U2 = (x2, y2, z2, 1), x1, y1, z1 are the coordinate values ​​in the world coordinate system, and x2, y2, z2 are the coordinate values ​​in the camera coordinate system;

[0023] Step 3.2: Represent the image coordinate system as O3-X3Y3. The relationship between the camera coordinate system and the image coordinate system is a perspective projection, expressed as: Where f is the camera focal length, and (x,y) are the coordinates of the imaging point in the image;

[0024] Step 3.3: Represent the pixel coordinate system using O4-UV. The relationship between image coordinates and pixel coordinates is expressed as follows: (u,v) are the coordinates in the pixel coordinate system, (u0,v0) are the coordinates of the origin of the image coordinate system in the pixel coordinate system, and d x d y It is the length and width of each pixel;

[0025] Step 3.4: Combining the analysis from steps 3.1 to 3.3, the transformation relationship from the world coordinate system to the pixel coordinate system is obtained as follows: Where B1 is the camera intrinsic parameter matrix, B2 is called the camera extrinsic parameter matrix, and g x and g y It is the camera's scale factor in the u-axis and v-axis directions;

[0026] Step 3.5: When the camera and the horizontal plane are at an angle, the extension of the optical center and the point will intersect the ground at a point P. The image point of P in the pixel coordinate system is P′. Let Q be the projection of P in the Z2 direction in the camera coordinate system; assuming ω is the angle between the optical axis and O2Q, then we have Where K is the vertical pixel value of the image;

[0027] Step 3.6: According to the principle of alternate angles within parallel lines, the angle σ between O2Q and the ground can be considered as the angle between O2Q and the horizontal line, where β is the camera's pitch angle. Therefore, the angle σ = β + ω. Where z1 represents the height of the camera above the ground; for each target detection box in the pixel coordinate system, the bottom center of the target detection box is set as the target point P′, then the pixel coordinates of P′ are... However, when the target is not directly in front of the camera, the target point deviates from the center of the vehicle, affecting the detection results. Therefore, it is necessary to update the u... p The value is represented as Using the transformation relationships between different coordinate systems, the transformation function for the target point from the pixel coordinate system to the world coordinate system is obtained as follows:

[0028] Furthermore, step 4 includes the following sub-steps:

[0029] Step 4.1: Consider N autonomous vehicles in a hybrid scenario where driverless and human-driven vehicles coexist. Each autonomous vehicle has the ability to perceive, detect, and communicate. Let be the image, camera parameters, and pose acquired by the r-th autonomous vehicle, respectively. Given a certain communication budget E, the expression for maximizing the 3D detection performance of each autonomous vehicle is:

[0030]

[0031] in, It is the message transmitted from the q-th autonomous vehicle to the r-th autonomous vehicle, where q and r ∈ [1, N] and q ≠ r. It is an object detection network. It is the transformation function for the target point from the pixel coordinate system to the world coordinate system. It is a function for fusing perception results from multiple autonomous vehicles. These are real-world measurements. It is a metric for evaluating detection performance;

[0032] Step 4.2: Each autonomous vehicle will perceive vehicles in the surrounding environment through its own camera sensors and generate the location information of these vehicles. The perceived location data is transmitted in the form of a message (τ, a, M), where τ represents a timestamp, a is the number of the data source vehicle, and M is a list of the locations of the target objects perceived by the vehicle, i.e., the real-world coordinates of the target points obtained in Step 3.6. To avoid timing errors, the data must be time-aligned to ensure that the data is based on the same time segment.

[0033] Step 4.3: After receiving the location information of surrounding objects from nearby autonomous vehicles, determine whether to fuse the data into a single target object; use Euclidean distance to measure the similarity between any two detection points. and The formula for calculating the Euclidean distance between two detection points is:

[0034] Step 4.4: Assuming the distance D between the m detection points is less than the threshold η, then these m detection points are considered to represent the same target object; update the coordinates P′ of the merged detection points using the weighted average, as shown in the formula. Where, p t It is the position of the t-th detection point. is the weight of the detection point, and p′ represents the overall position of the target object after collaborative perception by multiple vehicles.

[0035] The beneficial technical effects of this invention are as follows:

[0036] This invention proposes a collaborative perception method for autonomous driving based on image 3D reconstruction, which can solve the problems of limited perception distance of a single vehicle and lack of depth information in two-dimensional images in existing vehicle detection and localization technologies. It effectively improves the accuracy of long-distance vehicle detection and the accuracy of depth information recovery, can accurately estimate the position of vehicles in the real world, and realizes the sharing and fusion of detection results among vehicles through core information communication strategies. Attached Figure Description

[0037] Figure 1 This is a flowchart of the collaborative perception method based on image 3D reconstruction in this invention.

[0038] Figure 2 This is a flowchart illustrating the workflow of the target detection network in this invention.

[0039] Figure 3 This is a flowchart illustrating the process of restoring image depth information using the pinhole camera imaging principle in this invention. Detailed Implementation

[0040] The specific embodiments of the present invention will be further described below with reference to specific examples:

[0041] An autonomous driving cooperative perception method based on image 3D reconstruction, such as Figure 1 As shown, it includes the following steps:

[0042] Step 1: Collect labeled raw images from vehicle-mounted cameras to construct a dataset, and divide it into training and test sets;

[0043] Step 2: Construct an object detection network, design a loss function, and train and test the object detection network using training and testing sets;

[0044] Step 3: Input the camera image of the autonomous vehicle to be detected into the pre-trained target detection network, and reconstruct the pixel positions of the two-dimensional image output by the target detection network to the three-dimensional positions in the real world.

[0045] Step 4: Fuse the 3D position information obtained from camera images of multiple autonomous vehicles to obtain the final target detection result.

[0046] Specifically, in step 1, smart transportation practitioners use on-board cameras to collect images of multiple autonomous vehicles on multiple road sections, or use simulation tools to perform simulations, construct a dataset from all the original images collected from the on-board cameras, and divide the dataset into training and test sets according to the proportions.

[0047] Specifically, in step 2, the target detection network includes a feature focusing module and a detection head, such as... Figure 2 As shown, the specific working process and loss function are as follows:

[0048] Step 2.1: After scaling and normalizing the images in the dataset, input them into the encoder. Multi-layer convolutional networks and feature pyramid networks are used to extract features, capturing object information at different scales. The encoder's output feature map is... Where Φ(·) is the feature extraction operation, It is the l-th image captured by the autonomous vehicle, l∈[1,n], n is the number of images, C is the depth, H is the height, and W is the width;

[0049] Step 2.2: Transfer the feature map The input is an adaptive average pooling layer, which aggregates the spatial information of the feature map into a single depth vector, representing the global semantics of each depth. The computation process is as follows: in It is the global feature vector corresponding to the c-th layer depth. It is the feature vector in the i-th row and j-th column corresponding to the c-th layer depth;

[0050] Step 2.3: In order to extract and learn input features and help the network become more robust to small perturbations in the input data, the following steps are taken: The depth weights are calculated by inputting two fully connected layers. The calculation process is as follows: in, Let c be the weight vector at depth c. ε is the ReLU activation function, and ε is the Sigmoid function. It is a depth-based global feature vector; it is weighted at the depth of each layer of the original feature map, and the calculation method is as follows: in, These are the features updated after the depth of layer I;

[0051] Step 2.4, let the feature map For feature maps Perform average pooling and max pooling separately to reduce the depth dimension, and obtain the feature map after average pooling. and the feature map after max pooling Will and The features are concatenated and then weighted using convolution operations to generate a feature map. The expression is: Where ι represents the forward propagation of the convolution operation to infer the input feature map, utilizing the forward propagation function. The final output feature map is obtained. By using depth focusing and position focusing, the detection head can obtain richer geometric and semantic information;

[0052] Step 2.5: Detect the head using attention-weighted feature maps. As input, output classes, and regression objects, the detection head consists of two parts: non-maximum suppression (NMS) and scale boxes, used to infer the final prediction result. The NMS part filters out overlapping detection boxes based on a confidence threshold while predicting different categories. Specifically: first, detection boxes with confidence scores above the threshold are selected; the remaining detection boxes are grouped according to their predicted categories; within each category, the detection boxes are sorted from highest to lowest confidence; the detection box with the highest confidence score is retained; and the intersection-union ratio (IU) of the remaining boxes is calculated. Based on the IU, boxes exceeding the threshold are removed. The scale box part maps the model-predicted bounding boxes from the feature map space output by the network back to the space of the original input image. Specifically: first, the image after processing by the detection head is calculated... and the original input image The scaling ratio is determined, then the coordinates are inversely scaled, and the bounding box on the original image is generated based on the center coordinates and size of the inversely scaled image.

[0053] The detection head ultimately outputs an information vector I = {I1, I2, ..., I...} of n target detection boxes. n}, where the information vector I of the i-th target detection box i =(δ,θ,t) u ,t v ,w,h), where δ is the target category, θ is the detection box confidence, and t u t v The coordinates of the center of the detection box are represented by w, where w is the width of the detection box and h is the height of the detection box.

[0054] Step 2.6: Since object detection is a classification and localization problem, we comprehensively consider the object detection classification loss, the object detection bounding box regression loss, and the object confidence loss to construct its loss function. Where L1 is the object detection classification loss, L2 is the bounding box regression loss, L3 is the object confidence loss, and λ is the target confidence loss. s It is the weighting coefficient of the loss.

[0055] Specifically, in step 3, such as Figure 3 As shown, it includes the following sub-steps:

[0056] Step 3.1: Let O1-X1Y1Z1 represent the world coordinate system, and O2-X2Y2Z2 represent the camera coordinate system. When the camera rotates around the X1, Y1, and Z1 axes of the world coordinate system, respectively, it forms a roll angle α, a pitch angle β, and a yaw angle γ. Taking a rotation angle α around X1 as an example, the transformation relationship between world coordinates and camera coordinates is as follows: Similarly, a similar transformation relationship exists when rotating around the Y1 and Z1 axes by angles β and γ, respectively. This yields three rotation matrix components R. α R β R γ Then we get the rotation matrix R = R α R β R γ S = [S1, S2, c3] T Let S1, S2, and S3 be the translation vectors, representing the translation components of a point on the X1, Y1, and Z1 axes, respectively. After rotation and translation, the relationship is obtained. Where U1 = (x1, y1, z1, 1) and U2 = (x2, y2, z2, 1), x1, y1, z1 are the coordinate values ​​in the world coordinate system, and x2, y2, z2 are the coordinate values ​​in the camera coordinate system.

[0057] Step 3.2: Represent the image coordinate system as O3-X3Y3. In the image coordinate system, the position of each point is represented by rows and columns. The relationship between the camera coordinate system and the image coordinate system is perspective projection, expressed as: Where f is the camera focal length, and (x,y) are the coordinates of the imaging point in the image;

[0058] Step 3.3: Represent the pixel coordinate system using O4-UV. The transformation from image coordinate system to pixel coordinate system is to convert the actual position of the object into the pixel position. Image coordinate system is usually continuous, while pixel coordinate system is discrete. On display devices, images need to be converted to pixel coordinate system to adapt to the screen resolution, with each pixel corresponding to a point on the screen. The relationship between image coordinates and pixel coordinates is expressed as: (u,v) are the coordinates in the pixel coordinate system, (u0,v0) are the coordinates of the origin of the image coordinate system in the pixel coordinate system, and d x d y It is the length and width of each pixel;

[0059] Step 3.4: Combining the analysis from steps 3.1 to 3.3, the transformation relationship from the world coordinate system to the pixel coordinate system is obtained as follows: Where B1 is the camera intrinsic parameter matrix, B2 is called the camera extrinsic parameter matrix, and g x and g y These are the camera's scale factors along the u-axis and v-axis, where B1, B2, and g are the scale factors. x and g y Collectively referred to as vehicle camera coefficients;

[0060] Step 3.5: When the camera and the horizontal plane are at an angle, the extension of the optical center and the point will intersect the ground at a point P. The image point of P in the pixel coordinate system is P′. Let Q be the projection of P in the Z2 direction in the camera coordinate system; assuming ω is the angle between the optical axis and O2Q, then we have Where K is the vertical pixel value of the image;

[0061] Step 3.6: According to the principle of alternate angles within parallel lines, the angle σ between O2Q and the ground can be considered as the angle between O2Q and the horizontal line, where β is the camera's pitch angle. Therefore, the angle σ = β + ω. Where z1 represents the height of the camera above the ground; for each target detection box in the pixel coordinate system, the bottom center of the target detection box is set as the target point P′, then the pixel coordinates of P′ are... However, when the target is not directly in front of the camera, the target point deviates from the center of the vehicle, affecting the detection results. Therefore, it is necessary to update the u... p The value is represented as Using the transformation relationships between different coordinate systems, the transformation function for the target point from the pixel coordinate system to the world coordinate system is obtained as follows:

[0062] Specifically, step 4 includes the following sub-steps:

[0063] Step 4.1: Consider N autonomous vehicles in a hybrid scenario where driverless and human-driven vehicles coexist. Each autonomous vehicle has the ability to perceive, detect, and communicate. Let be the image, camera parameters, and pose acquired by the r-th autonomous vehicle, respectively. Given a certain communication budget E, the expression for maximizing the 3D detection performance of each autonomous vehicle is:

[0064]

[0065] in, It is the message transmitted from the q-th agent to the r-th agent, where q and r ∈ [1, N] and q ≠ r. It is an object detection network. It is the transformation function for the target point from the pixel coordinate system to the world coordinate system. It is a function for fusing perception results from multiple autonomous vehicles. These are real-world measurements. It is a metric for evaluating detection performance;

[0066] Step 4.2: Autonomous vehicles share perception data via wireless communication (such as V2V, V2X), but the bandwidth of wireless networks is limited. Each autonomous vehicle uses its own camera sensors to perceive vehicles in the surrounding environment and generate the position information of these objects, usually represented as three-dimensional coordinates. The perceived position data is transmitted in the form of a message (τ, a, M), where τ represents a timestamp, a is the data source vehicle number, and M is a list of the positions of the target objects perceived by the vehicle. To avoid timing errors, the data must be time-aligned to ensure that the data is based on the same time segment.

[0067] Step 4.3: After receiving the location information of surrounding objects from nearby autonomous vehicles, determine whether to fuse the data into a single target object; use Euclidean distance to measure the similarity between any two detection points. and The formula for calculating the Euclidean distance between two detection points is:

[0068] Step 4.4: Assuming the distance D between the m detection points is less than the threshold η, then these m detection points are considered to represent the same target object; update the coordinates p′ of the merged detection points using the weighted average, as shown in the formula. Where, p t It is the position of the t-th detection point. is the weight of the detection point, and p′ represents the overall position of the target object after collaborative perception by multiple vehicles.

[0069] To demonstrate the feasibility and superiority of the method of the present invention, the following comparative experiments are presented.

[0070] The method of this invention was compared with five existing methods—When2com, V2X-ViT, Where2comm, CoCa3D, and CollaborativeBEV—on three datasets: OPV2V, DAIR-V2X, and V2XSet. Specifically, the When2com method improves the efficiency and accuracy of collaborative perception by introducing a time synchronization mechanism to optimize communication and information fusion among multiple agents; the V2X-ViT method utilizes a Transformer-based architecture combined with vehicle-to-everything (V2X) communication technology to achieve efficient feature extraction and information fusion, significantly improving the performance of multi-agent collaborative perception; Where2comm proposes a communication-efficient collaborative perception framework that uses a pragmatic compression mechanism guided by a spatial confidence graph to help agents decide who to communicate with and whose information to aggregate, thereby achieving efficient multi-agent collaboration; and the CoCa3D method significantly improves the performance of 3D object detection using only cameras through multi-agent collaboration. This method demonstrates that, with sufficient collaboration, cameras can outperform LiDAR in certain real-world scenarios. CollaborativeBEV is a collaborative perception framework based on bird's-eye view (BEV) that generates pedestrian occupancy maps through information sharing among multiple agents, effectively solving the occlusion problem in complex scenes and making it suitable for target detection in crowded environments. Comparative experimental results are shown in Table 1.

[0071] Table 1 Performance comparison of each method

[0072]

[0073] Wherein, mAP@0.3, mAP@0.5, and mAP@0.7 are the average accuracies when the Intersection over Union (IoU) thresholds are 0.3, 0.5, and 0.7, respectively. Higher values ​​indicate higher accuracy in collaborative sensing. Experimental results show that the method of this invention achieves the best prediction performance.

[0074] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A cooperative perception method for autonomous driving based on image 3D reconstruction, characterized in that, Includes the following steps: Step 1: Collect labeled raw images from vehicle-mounted cameras to construct a dataset, and divide it into training and test sets; Step 2: Construct an object detection network, design a loss function, and train and test the object detection network using training and testing sets; Step 3: Input the camera image of the autonomous vehicle to be detected into the pre-trained target detection network, and reconstruct the pixel positions of the two-dimensional image output by the target detection network to the three-dimensional positions in the real world. Step 4: Fuse the 3D position information obtained from camera images of multiple autonomous vehicles to obtain the final target detection result; Step 4 includes the following sub-steps: Step 4.1: Consider the hybrid scenario where autonomous and human driving coexist. Each of the four autonomous vehicles possesses the ability to perceive, detect, and communicate; [The text abruptly ends here, likely due to an incomplete sentence or missing information.] , , They are the first Given a communication budget E, the expression for maximizing the 3D detection performance of each autonomous vehicle, based on images, camera parameters, and pose data collected by the autonomous vehicle, is as follows: ； in, It is the first The first autonomous vehicle to the Messages transmitted by an autonomous vehicle and , It is an object detection network. It is the transformation function for the target point from the pixel coordinate system to the world coordinate system. It is a function for fusing perception results from multiple autonomous vehicles. These are real-world measurements. Measurement of detection performance; Step 4.2: Each autonomous vehicle uses its own camera sensors to perceive vehicles in the surrounding environment and generate location information for these vehicles. The perceived location data is then transmitted as a message. Transmitted in the form of, where, Represents a timestamp. It is the vehicle number of the data source. This is a list of the locations of the target objects sensed by the vehicle, i.e., the real-world coordinates of the target points obtained in step 3.6; to avoid timing errors, the data must be time-aligned to ensure that the data are based on the same time segment; Step 4.3: After receiving the location information of surrounding objects from nearby autonomous vehicles, determine whether to fuse the data into a single target object; use Euclidean distance to measure the similarity between any two detection points. and The formula for calculating the Euclidean distance between two detection points is: ; Step 4.4, Assumption The distance between each detection point Less than the threshold Then it is considered that Each detection point represents the same target object; the coordinates of the merged detection points are updated using a weighted average. The formula is ,in, It is the first The location of each testing point It is the weight of that detection point. This indicates the overall position of the target object after being sensed collaboratively by multiple vehicles.

2. The autonomous driving cooperative perception method based on image 3D reconstruction according to claim 1, characterized in that, In step 1, images of multiple autonomous vehicles on multiple road sections are collected using vehicle-mounted cameras, or simulation tools are used to perform simulations. All the original images collected from the vehicle-mounted cameras are used to construct a dataset, and the dataset is divided into a training set and a test set according to the proportion.

3. The autonomous driving cooperative perception method based on image 3D reconstruction according to claim 2, characterized in that, In step 2, the target detection network includes a feature focusing module and a detection head. The specific working process and loss function are as follows: Step 2.1: After scaling and normalizing the images in the dataset, input them into the encoder. Multi-layer convolutional networks and feature pyramid networks are used to extract features, capturing object information at different scales. The encoder's output feature map is... ,in, It is a feature extraction operation. This is the first data collection point for autonomous vehicles. One image, , For the number of images, It's about depth. It's about height. It is the width; Step 2.2: Transfer the feature map The input is an adaptive average pooling layer, which aggregates the spatial information of the feature map into a single depth vector, representing the global semantics of each depth. The computation process is as follows: ,in, It is the first The global feature vector corresponding to the layer depth. It is the first The layer depth corresponding to the first Line number eigenvectors of the column; Step 2.3, The depth weights are calculated by inputting two fully connected layers. The calculation process is as follows: ,in, For the first Layer depth weight vector It is the ReLU activation function. It is the Sigmoid function. It is a depth-based global feature vector; it is weighted at the depth of each layer of the original feature map, and the calculation method is as follows: ,in, It is the first Features updated after layer depth; Step 2.4, let the feature map For feature maps Perform average pooling and max pooling respectively to obtain the feature map after average pooling. and the feature map after max pooling ,Will and The features are concatenated and then weighted using convolution operations to generate a feature map. The expression is: ,in It's a convolution operation; forward propagation performs inference on the input feature map, using the forward propagation function. The final output feature map is obtained. , ; Step 2.5: Weight the feature map by attention. The input detection head consists of two parts: non-maximum suppression (NMS) and scaled bounding boxes. NMS works as follows: first, detection boxes exceeding a confidence threshold are filtered out; the remaining boxes are grouped according to their predicted categories; within each category, boxes are sorted from highest to lowest confidence; the highest-confidence boxes are retained; and the cross-union ratio (CUI) of the remaining boxes is calculated. Based on the CUI, boxes exceeding a threshold are removed. Scaled bounding boxes are calculated first in the image processed by the detection head. and the original input image The scaling ratio is determined, then the coordinates are inversely scaled, and the bounding box on the original image is generated based on the center coordinates and size of the inversely scaled image. Final output of the detection head Information vector of each target detection box , of which Information vector of each target detection box , It is the target category. It is the confidence score of the detection box. Indicates the coordinates of the center of the detection box. It is the width of the detection frame. It is the height of the detection frame; Step 2.6: Taking into account the object detection classification loss, object detection bounding box regression loss, and object confidence loss, a loss function is constructed. ,in, It is the object detection classification loss. It is the bounding box regression loss. It is the target confidence loss. It is the weighting coefficient of the loss.

4. The autonomous driving cooperative perception method based on image 3D reconstruction according to claim 3, characterized in that, Step 3 includes the following sub-steps: Step 3.1, use To represent the world coordinate system, use Represents the camera coordinate system, as the camera revolves around the world coordinate system. axis, shaft and As the shaft rotates, it forms roll angles respectively. Pitch angle and deflection angle Thus, three rotation matrix components are obtained. , , Then the rotation matrix is ​​obtained. , It is a translation vector. They are points at , , The translation components on the axis, after rotation and translation, yield the relationship. ,in, , , These are the coordinates in the world coordinate system. These are the coordinate values ​​in the camera coordinate system. Step 3.2, use The image coordinate system represents the perspective projection relationship between the camera coordinate system and the image coordinate system, expressed as: ,in It's the camera's focal length. These are the coordinates of the imaging point in the image; Step 3.3, use The pixel coordinate system is represented as follows: , These are the coordinates in the pixel coordinate system. These are the coordinates of the origin of the image coordinate system in the pixel coordinate system. , It is the length and width of each pixel; Step 3.4: Combining the analysis from steps 3.1 to 3.3, the transformation relationship from the world coordinate system to the pixel coordinate system is obtained as follows: ,in, For the camera's intrinsic parameter matrix, This is called the camera extrinsic parameter matrix. and It's the camera. axis Scale factor along the axis; Step 3.5: When the camera is at an angle to the horizontal plane, the extension of the optical center and the point will inevitably intersect the ground. , The imaging point in the pixel coordinate system is ,set up yes In camera coordinate system Projection of direction; assumption It is the optical axis and The included angle is then... ,in, These are the vertical pixel values ​​of the image; Step 3.6: According to the principle of alternate angles within parallel lines, Angle with the ground Can be regarded as The angle between the horizontal line and the horizontal line If it is the camera's pitch angle, then the included angle is... Then there is ,in This represents the camera's height above the ground; for each target detection box in the pixel coordinate system, the bottom center of the target detection box is set as the target point. ,but pixel coordinates However, when the target is not directly in front of the camera, the target point deviates from the center of the vehicle, affecting the detection results. Therefore, an update is needed. The value is represented as Using the transformation relationships between different coordinate systems, the transformation function for transforming the target point from the pixel coordinate system to the world coordinate system is obtained as follows: .

Images