Automatic driving collaborative perception method based on image three-dimensional reconstruction

Abstract

The invention discloses an automatic driving collaborative perception method based on image three-dimensional reconstruction, and belongs to the technical field of intelligent traffic collaborative perception, and the method comprises the steps: collecting original images of a vehicle-mounted camera with labels to construct a data set, and dividing the data set into a training set and a test set; constructing a target detection network, designing a loss function, and training the target detection network by using the training set; inputting an automatic driving vehicle camera image to be detected into a pre-trained target detection network, and reconstructing a pixel position of a two-dimensional image output by the target detection network to a three-dimensional position of a real world; and fusing the three-dimensional position information obtained by the camera images of the plurality of autonomous vehicles to obtain a final target detection result. According to the invention, the problems of limited sensing distance of a single vehicle and lack of depth information of a two-dimensional image in the existing vehicle detection and positioning technology can be solved, and the sharing and fusion of detection results among vehicles are realized through a core information communication strategy.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent traffic collaborative perception technology, and specifically relates to an autonomous driving collaborative perception method based on three-dimensional image reconstruction. Background Art

[0002] In recent years, autonomous driving technology, a core area of intelligent transportation, has made significant progress, attracting widespread attention from both academia and industry. One of the core challenges of autonomous driving systems is achieving reliable and safe driving in complex and dynamic environments. To meet this challenge, systems must possess a high level of environmental scalability. Image perception, as a key technology, has become an indispensable component of autonomous driving systems.

[0003] Traditional single-vehicle perception systems primarily rely on sensors mounted on the vehicle, such as cameras, radar, and lidar, to perceive the surrounding environment. Image sensors, owing to their ability to provide high resolution and rich visual information, play a crucial role in environmental perception. However, these perception systems have inherent limitations: their field of view is limited by occlusion and distance, and they are susceptible to adverse weather conditions. Furthermore, single-vehicle perception systems often struggle to meet the requirements for efficient perception and decision-making in complex and dynamic traffic scenarios. These limitations not only restrict the overall performance of autonomous driving systems but also impact their safety and reliability in practical deployments.

[0004] To solve this problem, the concept of collaborative perception has been proposed in recent years. Collaborative perception improves the robustness and accuracy of the perception system by fusing data from multiple sensors. Collaborative perception significantly improves the robustness and accuracy of the perception system through the collaboration between multiple agents and the data fusion capabilities of multiple sensors. Especially in unmanned driving scenarios, collaborative perception can expand the perception range and improve the detection ability of occluded objects by sharing visual data between multiple agents. Figure 1 This method can not only effectively reduce the perception blind spots of a single agent, but also improve the decision-making efficiency of the system in a dynamic environment.

[0005] Image data fusion is a key component of collaborative perception. In autonomous driving scenarios, image data often comes from multiple cameras with varying perspectives, resolutions, and shooting conditions. Efficiently fusing this image data to form a comprehensive understanding of the environment is a hot topic in current research. This process involves not only image stitching and correction but also the extraction of useful features from multi-source data and comprehensive analysis combined with data from other sensors. However, existing technologies still have limitations in accurately detecting small targets at long distances, hindering the performance of collaborative perception. Summary of the Invention

[0006] In response to the above-mentioned problems existing in the prior art, the present invention proposes an autonomous driving collaborative perception method based on three-dimensional image reconstruction. The method has a reasonable design, solves the shortcomings of the prior art, and has good results.

[0007] A collaborative perception method for autonomous driving based on three-dimensional image reconstruction includes the following steps:

[0008] Step 1: Collect labeled original images from the vehicle camera to construct a dataset and divide it into a training set and a test set;

[0009] Step 2: Build a target detection network, design a loss function, and use the training set and test set to train and test the target detection network;

[0010] Step 3: Input the camera image of the autonomous driving vehicle to be detected into the pre-trained object detection network, and reconstruct the pixel positions of the two-dimensional image output by the object detection network into real-world three-dimensional positions;

[0011] Step 4: Fuse the 3D position information obtained from the camera images of multiple autonomous vehicles to obtain the final target detection result.

[0012] Furthermore, in step 1, a vehicle-mounted camera is used to capture images of multiple autonomous driving vehicles on multiple road sections, or a simulation tool is used for simulation, and all the collected original images of the vehicle-mounted cameras are constructed into a data set, and the data set is divided into a training set and a test set in proportion.

[0013] Furthermore, in step 2, the target detection network includes a feature focusing module and a detection head. The specific working process and loss function are as follows:

[0014] Step 2.1: After resizing and normalizing the image in the dataset, the image is input into the encoder, and features are extracted using a multi-layer convolutional network and a feature pyramid network to capture object information at different scales. The output feature map of the encoder is Among them, Φ(·) is the feature extraction operation, is the lth image captured by the autonomous vehicle, l∈[1,n], n is the number of images, C is the depth, H is the height, and W is the width;

[0015] Step 2.2: Feature map Input the adaptive average pooling layer to aggregate the spatial information of the feature map into a single depth vector, representing the global semantics of each depth. The calculation process is in, is the global eigenvector corresponding to the depth of layer c, is the feature vector of row i and column j corresponding to the depth of layer c;

[0016] Step 2.3, Input two fully connected layers to calculate the depth weight, the calculation process is in, is the weight vector of the c-th layer depth, is the ReLU activation function, ε is the Sigmoid function, is the depth global feature vector; the depth of each layer of the original feature map is weighted and the calculation method is in, is the feature after the depth update of layer c;

[0017] Step 2.4, let the feature map Feature Map Perform average pooling and maximum pooling respectively to obtain the feature map after average pooling And the feature map after maximum pooling Will and Splicing and generating weighted feature maps through convolution operations The expression is: Where ι is the convolution operation; forward propagation performs inference on the input feature map, using the forward propagation function Get the final output feature map

[0018] Step 2.5: The feature map weighted by attention Input detection head, the detection head consists of two parts: non-maximum suppression and scale box. Non-maximum suppression is specifically as follows: first, according to the confidence threshold, the detection boxes above the threshold are filtered out, the remaining detection boxes are grouped according to the predicted categories, and the detection boxes in each category are sorted from high to low according to the confidence, the detection box with the highest confidence is retained, and the intersection of union and the ratio of the remaining boxes is calculated. Based on the intersection of union and the ratio of the remaining boxes, the boxes exceeding the threshold are removed; the scale box is specifically as follows: first, the image processed by the detection head is calculated. and the original input image The scaling ratio is then reversed, and the bounding box on the original image is generated based on the reverse-scaled center coordinates and size.

[0019] The detection head finally outputs the information vector of n target detection boxes I={I1,I2,...,I n}, where the information vector I of the i-th target detection box i =(δ,θ,t u ,t v ,w,h), δ is the target category, θ is the detection box confidence, t u , t vIndicates the coordinates of the center of the detection frame, w is the width of the detection frame, and h is the height of the detection frame;

[0020] Step 2.6: Comprehensively consider the target detection classification loss, target detection bounding box regression loss, and target confidence loss, and then construct its loss function Among them, L1 is the target detection classification loss, L2 is the bounding box regression loss, L3 is the target confidence loss, and λ i is the weighting coefficient of the loss.

[0021] Furthermore, the step 3 includes the following sub-steps:

[0022] Step 3.1, use O1-X1Y1Z1 to represent the world coordinate system, and use O2-X2Y2Z2 to represent the camera coordinate system. When the camera rotates around the X1 axis, Y1 axis, and Z1 axis of the world coordinate system, the roll angle α, pitch angle β, and yaw angle γ are formed respectively, thus obtaining the three rotation matrix components R α , R β , R γ , then we get the rotation matrix R = R α R β R γ , S=[S1,S2,S3] T is the translation vector, S1, S2, S3 are the translation components of the point on the X1, Y1, and Z1 axes respectively. After rotation and translation, we get the relationship Among them, U1 = (x1, y1, z1, 1), U2 = (x2, y2, z2, 1), x1, y1, z1 are the coordinate values in the world coordinate system, and x2, y2, z2 are the coordinate values in the camera coordinate system;

[0023] Step 3.2: Use O3-X3Y3 to represent the image coordinate system. The relationship from the camera coordinate system to the image coordinate system is a perspective projection. The expression is: Where f is the focal length of the camera, (x, y) is the coordinate of the imaging point in the image;

[0024] Step 3.3: Use O4-UV to represent the pixel coordinate system. The relationship between image coordinates and pixel coordinates is expressed as: (u,v) is the coordinate value in the pixel coordinate system, (u0,v0) is the coordinate of the origin of the image coordinate system in the pixel coordinate system, d x d y is the length and width of each pixel;

[0025] Step 3.4: Based on the analysis of steps 3.1-3.3, the conversion relationship from the world coordinate system to the pixel coordinate system is: Among them, B1 is the camera internal parameter matrix, B2 is called the camera external parameter matrix, g x and g y is the scale factor of the camera in the u-axis and v-axis directions;

[0026] Step 3.5: When the camera is at an angle to the horizontal plane, the extension line of the optical center and the point will intersect the ground at a point P. The imaging point of P in the pixel coordinate system is P′. Let Q be the projection of P in the Z2 direction in the camera coordinate system. Assuming ω is the angle between the optical axis and O2Q, then Where K is the vertical pixel value of the image;

[0027] Step 3.6: According to the principle of phase difference within parallel lines, the angle σ between O2Q and the ground can be regarded as the angle between O2Q and the horizontal line. β is the pitch angle of the camera, so the angle σ=β+ω, then we have Where z1 represents the height of the camera from the ground; for each target detection frame in the pixel coordinate system, the bottom center of the target detection frame is set as the target point P′, then the pixel coordinate of P′ is However, when the target is not in front of the camera, the target point will deviate from the center of the vehicle, which will affect the detection results. Therefore, it is necessary to update u p The value of Using the conversion relationship between various coordinate systems, the conversion function of the target point from the pixel coordinate system to the world coordinate system is obtained as follows:

[0028] Furthermore, the step 4 includes the following sub-steps:

[0029] Step 4.1: Consider N autonomous vehicles in a mixed scenario where both unmanned and manned vehicles coexist. Each autonomous vehicle has the ability to perceive, detect, and communicate. are the images, camera parameters, and postures collected by the r-th autonomous vehicle. Given a certain communication budget E, the 3D detection performance of each autonomous vehicle is maximized, and the expression is:

[0030]

[0031] in, is the message transmitted from the qth autonomous vehicle to the rth autonomous vehicle, q, r∈[1,N] and q≠r, is the target detection network, is the conversion function of the target point from the pixel coordinate system to the world coordinate system, is the information fusion function of the perception results of multiple autonomous vehicles, is a real-world measurement, It is a metric for evaluating detection performance;

[0032] In step 4.2, each autonomous vehicle uses its camera sensor to sense surrounding vehicles and generates their location information. This location data is transmitted as a message (τ, a, M), where τ represents a timestamp, a is the vehicle ID, and M is the location list of the target objects sensed by the vehicle, i.e., the real-world coordinates of the target points obtained in step 3.6. To avoid timing errors, the data must be time-aligned to ensure that the data is based on the same time segment.

[0033] Step 4.3: After receiving the position information of surrounding objects sent by the surrounding autonomous driving vehicles, determine whether to fuse the data into the same target object; use the Euclidean distance to measure the similarity between any two detection points. and Two detection points, the Euclidean distance formula between the two points is

[0034] Step 4.4: Assume that the distance D between the m detection points is less than the threshold η, then these m detection points are considered to represent the same target object; use the weighted average to update the coordinates P′ of the merged detection points, the formula is Among them, p t is the position of the t-th detection point, is the weight of the detection point, and p′ represents the comprehensive position of the target object after collaborative perception by multiple vehicles.

[0035] Beneficial technical effects brought about by the present invention:

[0036] The present invention proposes a collaborative perception method for autonomous driving based on three-dimensional image reconstruction, which can solve the problems of limited perception distance of a single vehicle and missing depth information of two-dimensional images in existing vehicle detection and positioning technologies. It effectively improves the accuracy of long-distance vehicle detection and depth information recovery, can accurately estimate the position of the vehicle in the real world, and realize the sharing and integration of detection results between vehicles through core information communication strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of the collaborative perception method based on image three-dimensional reconstruction in the present invention.

[0038] Figure 2 This is a workflow diagram of the target detection network in the present invention.

[0039] Figure 3 This is a flowchart of restoring image depth information using the pinhole camera imaging principle in the present invention. DETAILED DESCRIPTION

[0040] The specific implementation of the present invention will be further described below with reference to specific embodiments:

[0041] A collaborative perception method for autonomous driving based on image 3D reconstruction, such as Figure 1 As shown, the following steps are included:

[0042] Step 1: Collect labeled original images from the vehicle camera to construct a dataset and divide it into a training set and a test set;

[0043] Step 2: Build a target detection network, design a loss function, and use the training set and test set to train and test the target detection network;

[0044] Step 3: Input the camera image of the autonomous driving vehicle to be detected into the pre-trained object detection network, and reconstruct the pixel positions of the two-dimensional image output by the object detection network into real-world three-dimensional positions;

[0045] Step 4: Fuse the 3D position information obtained from the camera images of multiple autonomous vehicles to obtain the final target detection result.

[0046] Specifically, in step 1, smart transportation practitioners use on-board cameras to collect images of multiple autonomous vehicles on multiple road sections, or use simulation tools for simulation, construct all the collected original images from on-board cameras into a dataset, and divide the dataset into training sets and test sets in proportion.

[0047] Specifically, in step 2, the target detection network includes a feature focusing module and a detection head, such as Figure 2 As shown, the specific working process and loss function are as follows:

[0048] Step 2.1: After resizing and normalizing the image in the dataset, the image is input into the encoder, and features are extracted using a multi-layer convolutional network and a feature pyramid network to capture object information at different scales. The output feature map of the encoder is Among them, Φ(·) is the feature extraction operation, is the lth image captured by the autonomous vehicle, l∈[1,n], n is the number of images, C is the depth, H is the height, and W is the width;

[0049] Step 2.2: Feature map Input the adaptive average pooling layer to aggregate the spatial information of the feature map into a single depth vector, representing the global semantics of each depth. The calculation process is in is the global feature vector corresponding to the depth of layer c, is the feature vector of row i and column j corresponding to the depth of layer c;

[0050] Step 2.3: In order to extract and learn input features and help the network be more robust to small perturbations of the input data, Input two fully connected layers to calculate the depth weight, the calculation process is in, is the weight vector of the c-th layer depth, is the ReLU activation function, ε is the Sigmoid function, is the depth global feature vector; the depth of each layer of the original feature map is weighted and the calculation method is in, is the feature after the depth update of layer I;

[0051] Step 2.4, let the feature map Feature Map Perform average pooling and maximum pooling respectively to reduce the depth dimension and obtain the feature map after average pooling And the feature map after maximum pooling Will and Splicing and generating weighted feature maps through convolution operations The expression is: Where ι is the forward propagation of the convolution operation to reason about the input feature map, using the forward propagation function Get the final output feature map Through depth focusing and position focusing, the detection head can obtain richer geometric and semantic information;

[0052] Step 2.5: Detection head with attention-weighted feature map As input, output class and regression objects, the detection head includes two parts: non-maximum suppression and scale box to infer the final prediction results; the non-maximum suppression part filters out some overlapping detection boxes through the confidence threshold, and predicts different categories at the same time, specifically: first, according to the confidence threshold, filter out the detection boxes above the threshold, group the remaining detection boxes according to the predicted categories, sort the detection boxes in each category from high to low according to the confidence, retain the detection box with the highest confidence, and calculate the intersection of the remaining boxes. Based on the intersection of the two ratios, remove the boxes that exceed the threshold; the scale box part maps the bounding box predicted by the model from the feature map space output by the network back to the space of the original input image, specifically: first calculate the image processed by the detection head and the original input image The scaling ratio is then reversed, and the bounding box on the original image is generated based on the reverse-scaled center coordinates and size.

[0053] The detection head finally outputs the information vector of n target detection boxes I={I1,I2,...,I n}, where the information vector I of the i-th target detection box i =(δ,θ,t u ,t v ,w,h), δ is the target category, θ is the detection box confidence, t u , t v Indicates the coordinates of the center of the detection frame, w is the width of the detection frame, and h is the height of the detection frame;

[0054] Step 2.6: Because target detection is a classification and positioning problem, the target detection classification loss, target detection bounding box regression loss, and target confidence loss are comprehensively considered to construct its loss function. Among them, L1 is the target detection classification loss, L2 is the bounding box regression loss, L3 is the target confidence loss, and λ s is the weighting coefficient of the loss.

[0055] Specifically, in step 3, Figure 3 As shown, it includes the following sub-steps:

[0056] Step 3.1. Use O1-X1Y1Z1 to represent the world coordinate system, and O2-X2Y2Z2 to represent the camera coordinate system. When the camera rotates around the X1 axis, Y1 axis, and Z1 axis of the world coordinate system, the roll angle α, pitch angle β, and yaw angle γ are formed respectively. Taking the rotation angle α around X1 as an example, the conversion relationship between world coordinates and camera coordinates is as follows: Similarly, when rotating around the Y1 axis and the Z1 axis by β and γ angles respectively, there is a similar transformation relationship. Thus, the three rotation matrix components R are obtained. α , R β , R γ , then we get the rotation matrix R = R α R β R γ , S=[S1,S2,c3] T is the translation vector, S1, S2, S3 are the translation components of the point on the X1, Y1, and Z1 axes respectively. After rotation and translation, we get the relationship Among them, U1 = (x1, y1, z1, 1), U2 = (x2, y2, z2, 1), x1, y1, z1 are the coordinate values in the world coordinate system, and x2, y2, z2 are the coordinate values in the camera coordinate system;

[0057] Step 3.2, use O3-X3Y3 to represent the image coordinate system. In the image coordinate system, the position of each point is represented by rows and columns. The relationship from the camera coordinate system to the image coordinate system is a perspective projection, and the expression is: Where f is the focal length of the camera, (x, y) is the coordinate of the imaging point in the image;

[0058] Step 3.3: Use O4-UV to represent the pixel coordinate system. The conversion from the image coordinate system to the pixel coordinate system is to convert the actual position of the object into the pixel position. The image coordinate system is usually continuous, while the pixel coordinate system is discrete. On the display device, the image needs to be converted to the pixel coordinate system to adapt to the screen resolution. Each pixel corresponds to a point on the screen. The relationship between image coordinates and pixel coordinates is expressed as: (u,v) is the coordinate value in the pixel coordinate system, (u0,v0) is the coordinate of the origin of the image coordinate system in the pixel coordinate system, d x d y is the length and width of each pixel;

[0059] Step 3.4: Based on the analysis of steps 3.1-3.3, the conversion relationship from the world coordinate system to the pixel coordinate system is: Among them, B1 is the camera internal parameter matrix, B2 is called the camera external parameter matrix, g x and g y is the scale factor of the camera in the u-axis v-axis direction, where B1, B2, g x and g y Collectively referred to as the vehicle camera coefficient;

[0060] Step 3.5: When the camera is at an angle to the horizontal plane, the extension line of the optical center and the point will intersect the ground at a point P. The imaging point of P in the pixel coordinate system is P′. Let Q be the projection of P in the Z2 direction in the camera coordinate system. Assuming ω is the angle between the optical axis and O2Q, then Where K is the vertical pixel value of the image;

[0061] Step 3.6: According to the principle of phase difference within parallel lines, the angle σ between O2Q and the ground can be regarded as the angle between O2Q and the horizontal line. β is the pitch angle of the camera, so the angle σ=β+ω, then we have Where z1 represents the height of the camera from the ground; for each target detection frame in the pixel coordinate system, the bottom center of the target detection frame is set as the target point P′, then the pixel coordinate of P′ is However, when the target is not in front of the camera, the target point will deviate from the center of the vehicle, which will affect the detection results. Therefore, it is necessary to update u p The value of Using the conversion relationship between various coordinate systems, the conversion function of the target point from the pixel coordinate system to the world coordinate system is obtained as follows:

[0062] Specifically, step 4 includes the following sub-steps:

[0063] Step 4.1: Consider N autonomous vehicles in a mixed scenario where both unmanned and manned vehicles coexist. Each autonomous vehicle has the ability to perceive, detect, and communicate. are the images, camera parameters, and postures collected by the r-th autonomous vehicle. Given a certain communication budget E, the 3D detection performance of each autonomous vehicle is maximized, and the expression is:

[0064]

[0065] in, is the message transmitted by the qth agent to the rth agent, q, r∈[1,N] and q≠r, is the target detection network, is the conversion function of the target point from the pixel coordinate system to the world coordinate system, is the information fusion function of the perception results of multiple autonomous vehicles, is a real-world measurement, It is a metric for evaluating detection performance;

[0066] Step 4.2: Autonomous vehicles share perception data through wireless communications (such as V2V and V2X), but wireless networks have limited bandwidth. Each autonomous vehicle uses its own camera sensor to perceive vehicles in the surrounding environment and generates location information for these objects, typically expressed as three-dimensional coordinates. The perceived location data is transmitted in the form of a message (τ, a, M), where τ represents a timestamp, a is the vehicle ID of the data source, and M is a list of the locations of the target objects perceived by the vehicle. To avoid timing errors, the data must be time-aligned to ensure that the data is based on the same time segment.

[0067] Step 4.3: After receiving the position information of surrounding objects sent by the surrounding autonomous driving vehicles, determine whether to fuse the data into the same target object; use the Euclidean distance to measure the similarity between any two detection points. and Two detection points, the Euclidean distance formula between the two points is

[0068] Step 4.4: Assume that the distance D between the m detection points is less than the threshold η, then these m detection points are considered to represent the same target object; use the weighted average to update the coordinates p′ of the merged detection points, the formula is Among them, p t is the position of the t-th detection point, is the weight of the detection point, and p′ represents the comprehensive position of the target object after collaborative perception by multiple vehicles.

[0069] In order to demonstrate the feasibility and superiority of the method of the present invention, the following comparative experiments are given.

[0070] The method of the present invention was compared with five existing methods: When2com, V2X-ViT, Where2comm, CoCa3D, and CollaborativeBEV, on three datasets: OPV2V, DAIR-V2X, and V2XSet. The When2com method optimizes communication and information fusion between multiple agents by introducing a time synchronization mechanism, thereby improving the efficiency and accuracy of collaborative perception. The V2X-ViT method utilizes a Transformer-based architecture combined with vehicle-to-everything (V2X) communication technology to achieve efficient feature extraction and information fusion, significantly improving the performance of multi-agent collaborative perception. Where2comm proposes a communication-efficient collaborative perception framework that uses a pragmatic compression mechanism guided by spatial confidence maps to help agents decide with whom to communicate and whose information to aggregate, thereby achieving efficient multi-agent collaboration. The CoCa3D method significantly improves the 3D object detection performance of cameras alone through multi-agent collaboration. This method demonstrates that, with sufficient collaboration, cameras can surpass LiDAR detection performance in certain real-world scenarios. CollaborativeBEV is a collaborative perception framework based on a bird's-eye view (BEV). By sharing information between multiple agents, it generates pedestrian occupancy maps, effectively solving occlusion problems in complex scenes and making it suitable for target detection in crowded environments. The comparative experimental results are shown in Table 1.

[0071] Table 1 Performance comparison of various methods

[0072]

[0073] Among them, mAP@0.3, mAP@0.5, and mAP@0.7 are the average precisions when the intersection over union (IoU) thresholds are 0.3, 0.5, and 0.7, respectively. The larger the value, the higher the accuracy of collaborative perception. The experimental results show that the proposed method achieves the best prediction results.

[0074] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the scope of protection of the present invention.

Claims

1. A collaborative perception method for autonomous driving based on three-dimensional image reconstruction, characterized in that: The following steps are involved: Step 1: Collect labeled original images from the vehicle camera to construct a dataset and divide it into a training set and a test set; Step 2: Build a target detection network, design a loss function, and use the training set and test set to train and test the target detection network; Step 3: Input the camera image of the autonomous driving vehicle to be detected into the pre-trained object detection network, and reconstruct the pixel positions of the two-dimensional image output by the object detection network into real-world three-dimensional positions; Step 4: Fuse the 3D position information obtained from the camera images of multiple autonomous vehicles to obtain the final target detection result.

2. The autonomous driving collaborative perception method based on three-dimensional image reconstruction according to claim 1, characterized in that: In step 1, a vehicle-mounted camera is used to capture images of multiple autonomous driving vehicles on multiple road sections, or a simulation tool is used to perform simulation, and all the collected original images from the vehicle-mounted cameras are constructed into a dataset, and the dataset is divided into a training set and a test set in proportion.

3. The autonomous driving collaborative perception method based on three-dimensional image reconstruction according to claim 2, characterized in that: In step 2, the target detection network includes a feature focusing module and a detection head. The specific working process and loss function are as follows: Step 2.1: After resizing and normalizing the image in the dataset, the image is input into the encoder, and features are extracted using a multi-layer convolutional network and a feature pyramid network to capture object information at different scales. The output feature map of the encoder is Among them, Φ(·) is the feature extraction operation, is the lth image captured by the autonomous vehicle, l∈[1,n], n is the number of images, C is the depth, H is the height, and W is the width; Step 2.2: Feature map Input the adaptive average pooling layer to aggregate the spatial information of the feature map into a single depth vector, representing the global semantics of each depth. The calculation process is in, is the global feature vector corresponding to the depth of layer c, is the feature vector of row i and column j corresponding to the depth of layer c; Step 2.3, Input two fully connected layers to calculate the depth weight, the calculation process is in, is the weight vector of the c-th layer depth, is the ReLU activation function, ε is the Sigmoid function, is the depth global feature vector; the depth of each layer of the original feature map is weighted and the calculation method is in, is the feature after the depth update of layer c; Step 2.4, let the feature map Feature Map Perform average pooling and maximum pooling respectively to obtain the feature map after average pooling And the feature map after maximum pooling Will and Splicing and generating weighted feature maps through convolution operations The expression is: Where ι is the convolution operation; forward propagation performs inference on the input feature map, using the forward propagation function Get the final output feature map Step 2.5: The feature map weighted by attention Input detection head, the detection head consists of two parts: non-maximum suppression and scale box. Non-maximum suppression is specifically as follows: first, according to the confidence threshold, the detection boxes above the threshold are filtered out, the remaining detection boxes are grouped according to the predicted categories, and the detection boxes in each category are sorted from high to low according to the confidence, the detection box with the highest confidence is retained, and the intersection of union and the ratio of the remaining boxes is calculated. Based on the intersection of union and the ratio of the remaining boxes, the boxes exceeding the threshold are removed; the scale box is specifically as follows: first, the image processed by the detection head is calculated. and the original input image The scaling ratio is then reversed, and the bounding box on the original image is generated based on the reverse-scaled center coordinates and size. The detection head finally outputs the information vector of n target detection boxes I={I1,I2,...,I n }, where the information vector I of the i-th target detection box i =(δ,I,t u ,t v ,w,h), δ is the target category, θ is the detection box confidence, t u , t v Indicates the coordinates of the center of the detection frame, w is the width of the detection frame, and h is the height of the detection frame; Step 2.6: Comprehensively consider the target detection classification loss, target detection bounding box regression loss, and target confidence loss, and then construct its loss function Among them, L1 is the target detection classification loss, L2 is the bounding box regression loss, L3 is the target confidence loss, and λ s is the weighting coefficient of the loss.

4. The autonomous driving collaborative perception method based on three-dimensional image reconstruction according to claim 3, characterized in that: The step 3 includes the following sub-steps: Step 3.1, use O1-X1Y1Z1 to represent the world coordinate system, and use O2-X2Y2Z2 to represent the camera coordinate system. When the camera rotates around the X1 axis, Y1 axis, and Z1 axis of the world coordinate system, the roll angle α, pitch angle β, and yaw angle γ are formed respectively, thus obtaining the three rotation matrix components R α , R β , R γ , then we get the rotation matrix R = R α R β R γ , S=[S1,S2,S3] T is the translation vector, S1, S2, S3 are the translation components of the point on the X1, Y1, and Z1 axes respectively. After rotation and translation, we get the relationship Among them, U1 = (x1, y1, z1, 1), U2 = (x2, y2, z2, 1), x1, y1, z1 are the coordinate values in the world coordinate system, and x2, y2, z2 are the coordinate values in the camera coordinate system; Step 3.2: Use O3-X3Y3 to represent the image coordinate system. The relationship from the camera coordinate system to the image coordinate system is a perspective projection. The expression is: Where f is the focal length of the camera, (x, y) is the coordinate of the imaging point in the image; Step 3.3: Use O4-UV to represent the pixel coordinate system. The relationship between image coordinates and pixel coordinates is expressed as: (u,v) is the coordinate value in the pixel coordinate system, (u0,v0) is the coordinate of the origin of the image coordinate system in the pixel coordinate system, d x d y is the length and width of each pixel; Step 3.4: Based on the analysis of steps 3.1-3.3, the conversion relationship from the world coordinate system to the pixel coordinate system is: Among them, B1 is the camera internal parameter matrix, B2 is called the camera external parameter matrix, g x and g y is the scale factor of the camera in the u-axis and v-axis directions; Step 3.5: When the camera is at an angle to the horizontal plane, the extension line of the optical center and the point will intersect the ground at a point P. The imaging point of P in the pixel coordinate system is P′. Let Q be the projection of P in the Z2 direction in the camera coordinate system. Assuming ω is the angle between the optical axis and O2Q, then Where K is the vertical pixel value of the image; Step 3.6: According to the principle of phase difference between parallel lines, the angle σ between O2θ and the ground can be regarded as the angle between O2Q and the horizontal line. β is the pitch angle of the camera, so the angle σ=β+ω, then we have Where z1 represents the height of the camera from the ground; for each target detection frame in the pixel coordinate system, the bottom center of the target detection frame is set as the target point P′, then the pixel coordinate of P′ is However, when the target is not in front of the camera, the target point will deviate from the center of the vehicle, which will affect the detection results. Therefore, it is necessary to update u p The value of Using the conversion relationship between various coordinate systems, the conversion function of the target point from the pixel coordinate system to the world coordinate system is obtained as follows:

5. The autonomous driving collaborative perception method based on three-dimensional image reconstruction according to claim 4, characterized in that: The step 4 includes the following sub-steps: Step 4.1: Consider N autonomous vehicles in a mixed scenario where both unmanned and manned vehicles coexist. Each autonomous vehicle has the ability to perceive, detect, and communicate. are the images, camera parameters, and postures collected by the r-th autonomous vehicle. Given a certain communication budget E, the 3D detection performance of each autonomous vehicle is maximized, and the expression is: in, is the message transmitted from the qth autonomous vehicle to the rth autonomous vehicle, q, r∈[1,N] and q≠r, is the target detection network, is the conversion function of the target point from the pixel coordinate system to the world coordinate system, is the information fusion function of the perception results of multiple autonomous vehicles, is a real-world measurement, It is a metric for evaluating detection performance; In step 4.2, each autonomous vehicle uses its camera sensor to sense surrounding vehicles and generates their location information. This location data is transmitted as a message (τ, a, M), where τ represents a timestamp, a is the vehicle ID, and M is the location list of the target objects sensed by the vehicle, i.e., the real-world coordinates of the target points obtained in step 3.

6. To avoid timing errors, the data must be time-aligned to ensure that the data is based on the same time segment. Step 4.3: After receiving the position information of surrounding objects sent by the surrounding autonomous driving vehicles, determine whether to fuse the data into the same target object; use the Euclidean distance to measure the similarity between any two detection points. and Two detection points, the Euclidean distance formula between the two points is Step 4.4: Assume that the distance D between the m detection points is less than the threshold η, then these m detection points are considered to represent the same target object; use the weighted average to update the coordinates p′ of the merged detection points, the formula is Among them, p t is the position of the t-th detection point, is the weight of the detection point, and p′ represents the comprehensive position of the target object after collaborative perception by multiple vehicles.

Images