Multi-view fusion autonomous vehicle swarm self-calibration cooperative perception method, device and training method

Abstract

A multi-view fusion autonomous vehicle swarm self-calibration collaborative perception method, device, and training method are proposed. This method acquires synchronized images of overlapping multi-view fields of view, extracts two-dimensional features from each view image, predicts depth distribution, and generates global multi-view BEV features containing target appearance and spatial consistency information. A neural network with large convolutional kernels is used as a decoder for spatial aggregation, achieving complementary multi-view features. End-to-end training is used to jointly optimize model parameters and extrinsic parameter matrices, enabling simultaneous output of target pose BEV results and autonomous vehicle pose BEV results. The target pose BEV results provide the target's pose distribution information in the region, and the autonomous vehicle pose BEV results yield the extrinsic parameter matrices between autonomous vehicles, achieving self-calibration of the extrinsic parameter matrices between autonomous vehicle swarms. This effectively improves the flexibility, robustness, and deployment efficiency of swarm collaborative perception.

Description

Technical Field

[0001] This invention relates to the field of unmanned vehicle swarm perception technology, and in particular to a multi-view fusion unmanned vehicle swarm self-calibration collaborative perception method, device and training method. Background Technology

[0002] Unmanned vehicle swarm perception is of great significance in national defense, engineering, and special rescue fields. Unmanned vehicles equipped with cameras can observe targets from multiple perspectives from different locations, which can solve the problem of target occlusion in special environments: for the area that needs to be perceived, the unmanned vehicle swarm is deployed in a distributed manner around the area. By collecting multi-view overlapping synchronous images of the area, collaborative perception of targets in the area can be achieved, and the bird's eye view (BEV) perception results of the target area can be output. The BEV view can clearly show the location of the target, overcome the influence of single camera occlusion, and help improve situational awareness capabilities.

[0003] However, current research on collaborative perception in autonomous vehicle swarms largely focuses on multi-view perception with cameras in fixed poses. Camera poses are generally equated with vehicle poses. Before deployment, multi-view collaborative perception algorithms for autonomous vehicles require calibrating the extrinsic parameters between cameras using a checkerboard pattern; that is, the relative poses between cameras or autonomous vehicles are needed to output accurate BEV perception results. However, calibrating the extrinsic parameters means that the relative poses of the autonomous vehicles in the swarm cannot change during observation, otherwise the calibration results will be corrupted, limiting the flexibility of collaborative perception in autonomous vehicle swarms. The need for extrinsic parameter calibration also limits the deployment efficiency of collaborative perception in autonomous vehicle swarms. Furthermore, most current multi-view collaborative perception algorithms use a fixed number of cameras, meaning the number of input images is fixed, which cannot adapt to changes in the number of platforms in the swarm during collaborative perception, limiting the deployment efficiency and system scalability. The calibration results for extrinsic parameters are prone to errors as the poses of the autonomous vehicle swarm change, thus affecting the perception results. Since extrinsic parameters and perception results are obtained step-by-step, they are prone to introducing accumulated distributed errors. Summary of the Invention

[0004] To address the aforementioned shortcomings of existing technologies, this disclosure provides a multi-view fusion autonomous vehicle swarm self-calibration collaborative perception method and a corresponding model training method for perceiving targets such as pedestrians and vehicles, in order to improve the collaborative perception flexibility, robustness, and deployment efficiency of autonomous vehicle swarm systems.

[0005] The multi-view fusion autonomous vehicle cluster self-calibration and cooperative perception method disclosed herein mainly includes the following steps:

[0006] S1: Initialization of self-calibration and collaborative perception processing of multi-view fusion unmanned vehicle cluster, including: BEV grid initialization, reference coordinate system initialization, extrinsic parameter matrix initialization, camera intrinsic parameter initialization, learning rate initialization, depth interval initialization, and voxel mesh partitioning initialization.

[0007] S2: Distribute the cluster of unmanned vehicles around the target area, facing the target area, and collect images from different perspectives at the same time. There is overlap in the field of view between the images from different perspectives.

[0008] S3: Extract, decode and optimize the consistency features of the multi-view overlapping field-of-view synchronized images acquired by the unmanned vehicle cluster to obtain the target pose BEV result and the unmanned vehicle pose BEV result;

[0009] S4: Result the target pose BEV Results of autonomous vehicle posture BEV By using nonmaximum suppression, the pose parameters (x, y, r) of the target and the autonomous vehicle are extracted, reprojected onto the two-dimensional BEV plane, and input and displayed on the interactive interface.

[0010] S5: Obtain the target pose distribution information in the region from the target pose BEV result; obtain the extrinsic parameter matrix between autonomous vehicles based on the autonomous vehicle pose BEV result, and realize the self-calibration of the extrinsic parameter matrix between autonomous vehicle clusters.

[0011] Furthermore, step S1 specifically includes:

[0012] S11, BEV grid initialization: Based on the target perception area size, divide the horizontal and vertical dimensions of the final output BEV result grid, and set the number of orientation channels;

[0013] BEV grid size is calculated using the following formula:

[0014]

[0015] Among them, H B W B H represents the vertical and horizontal pixel counts of the final output BEV result, respectively. W W W These are the actual vertical and horizontal dimensions of the target perception region, respectively, and Δ is the actual size of a single BEV pixel.

[0016] If the number of orientation channels is set to K, then the angle of each orientation channel is calculated by the following formula:

[0017]

[0018] Where k = 0, 1, ..., K-1 is the orientation channel index, r kLet be the orientation angle of the k-th orientation channel;

[0019] S12, Reference coordinate system initialization: Select the camera coordinate system of an autonomous vehicle as the reference coordinate system V0, that is, set the origin of the autonomous vehicle in the final output target pose BEV and autonomous vehicle pose BEV results as: x = 0, y = 0, r = 0; where x, y, r represent the two-dimensional coordinate position and orientation angle in the BEV, respectively.

[0020] S13, Extrinsic parameter matrix initialization: Randomly initialize the extrinsic parameter matrices of other unmanned vehicle cameras relative to the reference coordinate system and set them as learnable parameters, then optimize them through backpropagation;

[0021] S13, Camera Intrinsic Parameter Initialization: Read the camera's factory intrinsic parameter calibration results through the camera interface to keep the camera's focal length, principal point, and distortion coefficient unchanged;

[0022] S15, Learning Rate Initialization: Set the learning rate for the extrinsic parameter matrix and model parameters, and set the parameters for the optimizer and learning rate scheduler;

[0023] S16, Depth Interval Initialization: Set the minimum depth d for the depth distribution prediction network. min and maximum depth d max Divide the logarithmic interval depth range:

[0024]

[0025] Where D is the number of depth intervals, d = 0, 1, ..., D-1 are the depth interval indices, and z d This represents the actual distance of the d-th depth interval;

[0026] S17, Voxel Mesh Generation Initialization: Let the voxel mesh size for the voxel feature generation process be V. X ×V Y ×V Z The voxel has a side length of s and a spatial range of [x]. min ,x max ]×[y min ,y max ]×[z min ,z max ].

[0027] Furthermore, step S3 specifically includes:

[0028] S31, For each input image, a two-dimensional feature extraction network extracts two-dimensional features:

[0029] A deep convolutional neural network is used as a shared backbone to extract multi-scale two-dimensional features from images at various viewpoints.

[0030] The extracted two-dimensional features are represented by the following formula:

[0031] F i =F 2D (I i )

[0032] Among them, F i ∈R C×H×W Let F be the two-dimensional feature extracted from the i-th image, where C is the number of feature channels, H and W are the feature dimensions, which are also the pixel dimensions of the input image, and F is the feature size. 2D (·) represents a two-dimensional feature extraction network, I i ∈R 3×H×W Let i be the input image of the i-th type, where i = 1, 2, ..., N is the index of the input image, and N is the number of autonomous vehicles in the cluster.

[0033] S32: For each input image, the depth probability distribution is predicted by the depth distribution prediction network:

[0034] A deep convolutional neural network is used as a shared backbone to extract the depth probability distribution of each image;

[0035] The extracted depth probability distribution is represented by the following formula:

[0036] D i =F depth (I i )

[0037] Among them, D i ∈R D×H×W Let F be the depth probability distribution of the i-th image. depth (·) represents a deep distribution prediction network;

[0038] S33: Transform the position (h, w) and depth zd of each pixel in each image to the reference coordinate system;

[0039] Coordinate transformation is represented as:

[0040]

[0041] Among them, R i Let t be a rotation matrix. i E is the translation vector. i =[R i |t i ] represents the extrinsic parameter matrix from camera i to the reference coordinate system, K i For camera internal parameters;

[0042] Calculate voxel indexes:

[0043]

[0044] If v is satisfiedx ∈[0,V X -1],v y ∈[0,V Y -1],v z ∈[0,V Z If -1 is found, the point is retained; otherwise, it is discarded.

[0045] S34: For each image, for each transformed effective voxel position (v x ,v y ,v z By accumulating voxel features and voxel weights, the features of images from different viewpoints are transformed to the same coordinate system.

[0046] The voxel features of each image are represented as follows:

[0047] V i [c,v x ,v y ,v z ]+=F i (c,h,w)·D i (d,h,w)

[0048] The voxel weights for each image are represented as follows:

[0049] W i [v x ,v y ,v z ]+=D i (d,h,w)

[0050] Normalizing the voxel features of each image yields:

[0051]

[0052] in, ε is a local minimum value to prevent division by zero;

[0053] S35: Voctet features of each image i global The summation pooling along the height z-axis is compressed into BEV features, represented as follows:

[0054] V i BEV =sum z {V i global [c,v x ,v y ,v z ]}

[0055] in,

[0056] BEV features V of all images i BEV Element-wise summation generates global multi-view BEV features;

[0057] Global multi-view BEV features can be represented as:

[0058] V BEV =sum i {V i BEV}

[0059] in,

[0060] S36, global multi-view BEV feature V BEV Input the target feature decoding network and the autonomous vehicle feature decoding network respectively: The target feature decoding network and the autonomous vehicle feature decoding network are constructed using the same feature decoding network architecture. The neural network with large convolutional kernels is used as the decoder to perform spatial aggregation, extract target appearance and spatial consistency information, and realize multi-view feature complementarity.

[0061] The target feature decoding network and the autonomous vehicle feature decoding network output the target pose BEV result, respectively. Results of autonomous vehicle posture BEV Represented as:

[0062]

[0063] in, and All are three-dimensional heatmap matrices, satisfying In H B W B The dimension displays the two-dimensional relative positioning of the target or autonomous vehicle. The value of each point represents the probability that the target or autonomous vehicle appears at that point. The K channel displays the orientation information of the target or autonomous vehicle. The value of each point represents the probability that the target or autonomous vehicle is facing that angle.

[0064] Furthermore, in step S5, the method for obtaining the extrinsic parameter matrix between the unmanned vehicles is as follows:

[0065] The BEV coordinate system is set to be the same as the world coordinate system, meaning the x and y axes of the BEV are aligned with the world coordinate system and their origins coincide. For driverless vehicle i and driverless vehicle j, their BEV coordinates are respectively (x... i ,y i ) and (x j ,y j ), with orientation angles θ i and θ j The ratio of the actual size of the BEV to the world coordinate system is sm / pixel;

[0066] The actual coordinates of driverless vehicles i and j in the world coordinate system are represented as follows:

[0067] (u i ,v i )=(s·x i ,s·y i )

[0068] (u j ,v j )=(s·x j ,s·y j )

[0069] The translation vector from driverless car j to driverless car i is represented as:

[0070]

[0071] The relative rotation angle is:

[0072] θ r =θ j -θ i

[0073] The rotation matrix is ​​then:

[0074]

[0075] This leads to the extrinsic parameter matrix:

[0076]

[0077] The multi-view fusion autonomous vehicle swarm self-calibration cooperative perception device applying the above method mainly includes:

[0078] A multi-view image acquisition module is used to acquire synchronized images of the target area from different viewpoints;

[0079] The two-dimensional feature extraction network module is used to extract two-dimensional features from each input image;

[0080] The depth distribution prediction network module is used to predict the depth probability distribution of each input image; the coordinate transformation module transforms the position and depth of each pixel in each image to the reference coordinate system.

[0081] The voxel feature generation module is used to accumulate voxel features and voxel weights for each image and for each converted effective voxel position, transforming the features of images from different viewpoints to the same coordinate system.

[0082] The BEV feature generation module is used to sum the voxel features of each image along the height axis and then pool and compress them into BEV features.

[0083] The feature decoding network module includes a target feature decoding network and an autonomous vehicle feature decoding network with the same architecture, which output the target pose BEV result and the autonomous vehicle pose BEV result, respectively.

[0084] The self-calibration module is used to obtain the external parameter matrix between autonomous vehicles based on the autonomous vehicle pose BEV results.

[0085] The training method applied to the above-mentioned collaborative sensing device mainly includes the following steps:

[0086] Step 1: Construct a dataset. Each dataset includes synchronized images from multiple perspectives and labels. The synchronized images are captured simultaneously by a cluster of autonomous vehicles around the target area, and the labels are the ground truth poses of the corresponding target BEVs. The positioning and attitude of BEVs with autonomous vehicles is truly valuable. Each dataset contains a different number of viewpoints and labels;

[0087] Step 2: The multi-view fusion unmanned vehicle cluster self-calibration and collaborative perception device is initialized;

[0088] Step 3: Load the training dataset;

[0089] Step 4: From the input multi-view images, randomly select one view and discard it. Input the remaining images into the multi-view fusion autonomous vehicle cluster self-calibration and cooperative perception device, and output the target pose BEV result. Results of autonomous vehicle posture BEV

[0090] Step 5: Calculate the total loss function, which is a weighted sum of the target pose loss function and the autonomous vehicle pose loss function; use the target and autonomous vehicle pose ground truth monitoring device model to enable it to simultaneously output the target pose BEV result and the autonomous vehicle pose BEV result.

[0091] Step 6: Optimize the device model parameters and extrinsic parameter matrix jointly through backpropagation using gradient descent;

[0092] Step 7: Repeat steps 3 through 6 until all training set data has been loaded.

[0093] Furthermore, the initialization in step 2 specifically includes: BEV grid initialization, reference coordinate system initialization, extrinsic parameter matrix initialization, camera intrinsic parameter initialization, learning rate initialization, and depth interval initialization.

[0094] Furthermore, step 5 specifically includes:

[0095] Step 51, calculate the target pose loss function L target :

[0096] For the target BEV pose view label B target H is processed using a two-dimensional Gaussian kernel. B W B The pose soft ground truth is obtained by processing the K channels using a one-dimensional Gaussian kernel.

[0097] Using both mean squared error loss and Focal loss to supervise the heatmap, the target pose loss function is expressed as:

[0098]

[0099] Where α1 and α2 are weights, MSELoss(·) represents the mean squared error loss function, which is used to force the model to regress to the true value, and FocalLoss(·) is used to reduce the impact of sample imbalance.

[0100] Step 52, calculate the pose loss function L of the unmanned vehicle. UGV :

[0101] For the pose view label B of the autonomous vehicle BEV UGV H is processed using a two-dimensional Gaussian kernel. B W B The pose soft ground truth is obtained by processing the K channels using a one-dimensional Gaussian kernel.

[0102] Using a Euclidean loss-supervised heatmap, the autonomous vehicle pose loss function is expressed as:

[0103]

[0104] Step 53: Calculate the total loss function L, expressed as:

[0105] L=λ1L target +λ2L UGV

[0106] Where λ1 and λ2 are weights.

[0107] Furthermore, the method also includes the following steps:

[0108] Step 8: Load the test dataset, input the multi-view images into the multi-view fusion unmanned vehicle cluster self-calibration cooperative perception device, and output the target pose BEV result and the unmanned vehicle pose BEV result.

[0109] Compared with the prior art, the beneficial effects of this disclosure are:

[0110] ① Construct target pose loss function and autonomous vehicle pose loss function. Input multi-view overlapping field-of-view synchronized image. By extracting two-dimensional features from each view image, predicting depth distribution, and generating global multi-view BEV features containing target appearance and spatial consistency information, use a neural network with large convolutional kernels as a decoder for spatial aggregation to achieve multi-view feature complementarity. Through end-to-end training, jointly optimize model parameters and extrinsic matrix, and simultaneously output target pose BEV results and autonomous vehicle pose BEV results. From the target pose BEV results, the pose distribution information of the target in the region can be obtained. Based on the autonomous vehicle pose BEV results, the extrinsic matrix between autonomous vehicles can be obtained, realizing the self-calibration of extrinsic matrix between autonomous vehicle clusters, thereby improving the flexibility, robustness, and deployment efficiency of cluster collaborative perception.

[0111] ② Input multi-view overlapping vision synchronization images, the generated global multi-view BEV feature dimension is fixed, which can support a variable number of input images. The dataset contains different numbers of view data and labels. During training, the multi-view images in the training dataset are randomly discarded, and the remaining images are input into the model to enhance the robustness of the model to changes in the number of input images. The output target pose BEV results and autonomous vehicle pose BEV results can maintain cooperative perception when the number of autonomous vehicles in the cluster changes, enhance the scalability of the system, and thus improve the flexibility, robustness and deployment efficiency of cluster cooperative perception.

[0112] ③ Construct an autonomous vehicle pose loss function, input multi-view overlapping vision synchronization images, generate global multi-view BEV features containing target appearance and spatial consistency information, train and optimize the extrinsic matrix end-to-end, and output autonomous vehicle pose BEV results. Based on the autonomous vehicle pose BEV results, the extrinsic matrix between autonomous vehicles can be obtained, realizing the self-calibration of the extrinsic matrix between autonomous vehicle clusters, thereby improving the deployment efficiency of cluster collaborative perception.

[0113] ④ Input multi-view overlapping field-of-view synchronized images, generate global multi-view BEV features containing target appearance and spatial consistency information, use a neural network with large convolutional kernels as a decoder to perform spatial aggregation, realize multi-view feature complementarity, and thus realize collaborative perception of the target when the single view is occluded. It can output the target pose BEV result, and obtain the target pose distribution information in the region from the target pose BEV result, thereby improving the robustness of cluster collaborative perception.

[0114] ⑤ Input multi-view overlapping field-of-view synchronized images, generate global multi-view BEV features containing target appearance and spatial consistency information, use a neural network with large convolutional kernels as a decoder to perform spatial aggregation, realize multi-view feature complementarity, maintain cooperative perception effect when the observation position of the unmanned vehicle cluster changes, output target pose BEV results, thereby improving the robustness of cluster cooperative perception.

[0115] ⑥ The target pose loss function and the autonomous vehicle pose loss function are weighted to form the total loss function. The global optimization efficiency is improved by end-to-end training and joint optimization of model parameters and extrinsic matrix. Attached Figure Description

[0116] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments of this disclosure taken in conjunction with the accompanying drawings, in which the same reference numerals generally represent the same components.

[0117] Figure 1 This is a schematic diagram of an exemplary unmanned vehicle cluster system structure;

[0118] Figure 2 The flowchart below illustrates the self-calibration and collaborative perception processing of a multi-view fusion unmanned vehicle cluster.

[0119] Figure 3 This is an example diagram illustrating the deployment of a cluster of unmanned vehicles around a target area.

[0120] Figure 4 This is a schematic diagram illustrating the deployment of a multi-view fusion autonomous vehicle cluster self-calibration and collaborative perception model according to this disclosure;

[0121] Figure 5 This is a schematic diagram of a two-dimensional feature extraction network structure;

[0122] Figure 6 This is a schematic diagram of the deep distribution prediction network structure;

[0123] Figure 7 This is a schematic diagram of the feature decoding network structure;

[0124] Figure 8 The result is the target pose after nonmaximum suppression and reprojection onto the BEV;

[0125] Figure 9 The pose of the unmanned vehicle is obtained by nonmaximum suppression and reprojection onto the BEV.

[0126] Figure 10 This is a flowchart of the training and testing process for a multi-view fusion autonomous vehicle cluster self-calibration collaborative perception model based on this disclosure. Detailed Implementation

[0127] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0128] This disclosure provides a multi-view fusion autonomous vehicle swarm self-calibration collaborative perception method, device, and training method.

[0129] In one exemplary implementation, the autonomous vehicle swarm system mainly consists of an autonomous vehicle swarm and a task controller, as shown in the attached figure. Figure 1-4 As shown.

[0130] The autonomous vehicle cluster acquires synchronized images and sends them to the task controller. The task controller then runs a multi-view fusion autonomous vehicle cluster self-calibrated cooperative perception model to perform cooperative perception processing and output the target and autonomous vehicle pose BEV perception results. Among these:

[0131] (1) The unmanned vehicle cluster consists of multiple unmanned vehicles, each of which includes an onboard computer, power supply, sensor module and networking communication module;

[0132] The onboard computer is connected to the sensor module and the network communication module. As the core module for the control of the unmanned vehicle, it controls the image acquisition and network communication functions of the unmanned vehicle.

[0133] The power supply provides power to the unmanned vehicle, enabling the sensor modules, onboard computer, and network communication module to function properly.

[0134] The sensor module is connected to the onboard computer. It uses a camera to acquire images and sends the acquired images to the onboard computer. The images are then sent to the task controller through the network communication module as input to the multi-view fusion autonomous vehicle cluster self-calibration and cooperative perception model. The camera is installed at the geometric center of the autonomous vehicle and faces the front of the vehicle. The camera pose is the same as the autonomous vehicle pose.

[0135] The networking communication module is connected to the onboard computer, enabling the unmanned vehicle to join the cluster system communication network for sending and receiving data.

[0136] (2) The task controller consists of a computing server, a network communication module, and an interactive interface, wherein:

[0137] The computing server is connected to the network communication module and the interactive interface, providing computing power, video memory, memory, and storage space computing resources. It runs a multi-view fusion autonomous vehicle cluster self-calibration cooperative perception model. It receives synchronous images collected from the autonomous vehicle cluster through the network communication module, performs cooperative perception processing, and outputs the obtained target and autonomous vehicle pose BEV perception results to the interactive interface for display.

[0138] The networking communication module is connected to the computing server, enabling the task controller to join the cluster system communication network for sending and receiving data;

[0139] The interactive interface is connected to the computing server, receiving and displaying the target and autonomous vehicle (BEV) position and pose perception results from the computing server.

[0140] In this implementation, as shown in the attached document Figure 2 As shown, the multi-view fusion autonomous vehicle cluster self-calibration and collaborative perception processing process mainly includes the following steps:

[0141] Step 1: Distribute the autonomous vehicle swarm at different locations around the target area, facing the target area, and collect images from different perspectives at the same time, with overlapping views between the images. Each autonomous vehicle has a unique digital identity. By perceiving the target from multiple perspectives, collaborative perception of the target is achieved even when a single perspective is obstructed, thereby improving the robustness of the swarm's collaborative perception.

[0142] Step 2: The unmanned vehicle cluster sends multi-view images to the task controller through the network communication module. The task controller receives the multi-view images through the network communication module and inputs them into the computing server. The computing server runs the multi-view fusion unmanned vehicle cluster self-calibration cooperative perception model.

[0143] Step 3: Initialize the multi-view fusion autonomous vehicle cluster self-calibration collaborative perception model, including BEV grid initialization, reference coordinate system initialization, extrinsic parameter matrix initialization, camera intrinsic parameter initialization, learning rate initialization, depth interval initialization, and voxel mesh partitioning initialization.

[0144] Step 3-1: BEV Grid Initialization. Based on the target sensing area size, divide the final output BEV result grid into horizontal and vertical dimensions, and set the number of orientation channels. The BEV grid size is calculated using the following formula:

[0145]

[0146] Among them, H B W B H represents the vertical and horizontal pixel counts of the final output BEV result, respectively. W W W Δ represents the actual vertical and horizontal dimensions of the target perception region, respectively, and Δ represents the actual size of a single BEV pixel.

[0147] If the number of orientation channels is set to K, then the angle of each orientation channel can be calculated by the following formula:

[0148]

[0149] Where k = 0, 1, ..., K-1 is the orientation channel index, r k Let be the orientation angle of the k-th orientation channel.

[0150] Step 3-2: Initialize the reference coordinate system. Select the camera coordinate system of an autonomous vehicle as the reference coordinate system V0. That is, set the origin x = 0, y = 0, r = 0 in the final output target pose BEV and autonomous vehicle pose BEV results. Here, x, y, and r represent the two-dimensional coordinate position and orientation angle in the BEV, respectively.

[0151] Step 3-3: Initialize the extrinsic parameter matrix. Randomly initialize the extrinsic parameter matrix of other unmanned vehicle cameras relative to the reference coordinate system and set it as a learnable parameter. Optimize it through backpropagation.

[0152] Steps 3-4: Initialize camera intrinsic parameters. Read the camera's factory intrinsic parameter calibration results through the camera interface, keeping the camera's focal length, principal point, and distortion coefficient unchanged.

[0153] Steps 3-5: Initialize the learning rate, set the learning rate of the extrinsic parameter matrix and model parameters, and set the parameters of the optimizer and learning rate scheduler.

[0154] Steps 3-6: Depth interval initialization, setting the minimum depth d for the depth distribution prediction network. min and maximum depth d max Divide the logarithmic interval depth range:

[0155]

[0156] Where D is the number of depth intervals, d = 0, 1, ..., D-1 are the depth interval indices, and z d This represents the actual distance of the d-th depth interval.

[0157] Steps 3-7: Voxel mesh generation initialization, setting the voxel mesh size for the voxel feature generation process to V. X ×V Y ×V Z The voxel has a side length of s and a spatial range of [x]. min ,x max ]×[y min ,y max ]×[z min ,z max ].

[0158] Step 4: Input the multi-view images into the multi-view fusion autonomous vehicle cluster self-calibration cooperative perception model, and output the target pose BEV result and the autonomous vehicle pose BEV result. The input multi-view overlapping field-of-view synchronized images contain the appearance and spatial information of the target from different viewpoints. Through consistency feature extraction, decoding, and optimization by the multi-view fusion autonomous vehicle cluster self-calibration cooperative perception model, the target pose BEV result and the autonomous vehicle pose BEV result can be directly output. From the target pose BEV result, the pose distribution information of the target in the region can be obtained. Based on the autonomous vehicle pose BEV result, the extrinsic parameter matrix between autonomous vehicles can be obtained, realizing the self-calibration of the extrinsic parameter matrix between autonomous vehicle clusters, thereby improving the flexibility, robustness, and deployment efficiency of cluster cooperative perception.

[0159] Step 4-1: For each input image, extract two-dimensional features using a two-dimensional feature extraction network. (See attached image.) Figure 5 As shown, a deep convolutional neural network is used as a shared backbone to extract multi-scale two-dimensional features from images at various viewpoints. These two-dimensional features contain appearance consistency information of the target and are used to generate subsequent global multi-view BEV features.

[0160] The extracted two-dimensional features can be represented by the following formula:

[0161] F i =F 2D (I i )

[0162] Among them, F i ∈R C×H×W Let F be the two-dimensional feature extracted from the i-th image, where C is the number of feature channels, H and W are the feature dimensions (also the pixel dimensions of the input image), and F is the feature size. 2D (·) represents a two-dimensional feature extraction network, I i ∈R 3×H×W Let i be the input image of the i-th type, where i = 1, 2, ..., N is the index of the input image, and N is the number of autonomous vehicles.

[0163] Step 4-2: For each input image, predict the depth probability distribution using a depth distribution prediction network. (See attached image.) Figure 6 As shown, a deep convolutional neural network is used as a shared backbone to extract the depth probability distribution of each image. The depth probability distribution contains spatial consistency information of the target, which is used to generate subsequent global multi-view BEV features.

[0164] The extracted depth probability distribution can be represented by the following formula:

[0165] D i =F depth (I i )

[0166] Among them, D i∈R D×H×W Let F be the depth probability distribution of the i-th image. depth (·) represents a deep distribution prediction network.

[0167] Step 4-3: Assign the position (h, w) and depth z of each pixel in each image. d Transform to the reference coordinate system.

[0168] Coordinate transformation can be expressed as:

[0169]

[0170] Among them, R i Let t be a rotation matrix. i E is the translation vector. i =[R i |t i ] represents the extrinsic parameter matrix from camera i to the reference coordinate system, K i This refers to the camera's internal parameters.

[0171] Calculate voxel indexes:

[0172]

[0173] If v is satisfied x ∈[0,V X -1],v y ∈[0,V Y -1],v z ∈[0,V Z If -1 is selected, the point is retained; otherwise, it is discarded.

[0174] Step 4-4: For each image, for each transformed effective voxel position (v x ,v y ,v z The voxel features and voxel weights are accumulated to transform the features of images from different viewpoints to the same coordinate system.

[0175] The voxel features of each image can be represented as:

[0176] V i [c,v x ,v y ,v z ]+=F i (c,h,w)·D i (d,h,w)

[0177] The voxel weights of each image can be represented as:

[0178] W i [v x ,v y ,v z]+=D i (d,h,w)

[0179] Normalizing the voxel features of each image yields:

[0180]

[0181] in, ε is a minimum value to prevent division by zero.

[0182] Steps 4-5: Extract the voxel features V of each image i global Sum-pooling along the height z-axis to compress into BEV features can be expressed as:

[0183] V i BEV =sum z {V i global [c,v x ,v y ,v z ]}

[0184] in,

[0185] BEV features V of all images i BEV Element-wise summation generates global multi-view BEV features. These global multi-view BEV features contain target appearance and spatial consistency information. With fixed dimensions, they can support a variable number of input images, thus maintaining collaborative perception even when the number of autonomous vehicles in the swarm changes. This enhances the system's scalability and improves the flexibility of swarm collaborative perception.

[0186] Global multi-view BEV features can be represented as:

[0187] V BEV =sum i {V i BEV}

[0188] in,

[0189] Steps 4-6: Transfer global multi-view BEV features V BEV Input the target feature decoding network and the autonomous vehicle feature decoding network respectively. Construct the target feature decoding network and the autonomous vehicle feature decoding network using the same feature decoding network architecture, as shown in the appendix. Figure 7As shown, a neural network with a large convolutional kernel is used as a decoder to perform spatial aggregation, extract target appearance and spatial consistency information, and achieve multi-view feature complementarity. This enables the autonomous vehicle cluster to maintain cooperative perception when the observation position changes, and achieves cooperative perception of the target when the single view is occluded, thereby improving the robustness of cluster cooperative perception.

[0190] The target feature decoding network and the autonomous vehicle feature decoding network output the target pose BEV result, respectively. Results of autonomous vehicle posture BEV It can be represented as:

[0191]

[0192] in, and All are three-dimensional heatmap matrices, satisfying In H B W B The dimension displays the two-dimensional relative positioning of the target or autonomous vehicle. The value of each point represents the probability that the target or autonomous vehicle appears at that point. The K channel displays the orientation information of the target or autonomous vehicle. The value of each point represents the probability that the target or autonomous vehicle is facing that angle.

[0193] Step 5: Obtain the target pose BEV result Results of autonomous vehicle posture BEV The pose parameters (x, y, r) of the target and the autonomous vehicle are extracted using nonmaximum suppression, reprojected onto the 2D BEV plane, input and displayed on the interactive interface, as shown in the attached figure. Figure 8 and 9 As shown.

[0194] The target pose BEV results can be used to obtain the pose distribution information of the target in the region; the relative pose between autonomous vehicles can be obtained based on the autonomous vehicle pose BEV results, and the translation matrix + rotation matrix between autonomous vehicles can be obtained, which is the extrinsic parameter matrix between autonomous vehicles, thus realizing the self-calibration of the extrinsic parameter matrix between autonomous vehicle clusters.

[0195] As can be seen, the multi-view fusion autonomous vehicle cluster self-calibration and cooperative perception model described in this embodiment mainly includes:

[0196] A two-dimensional feature extraction network is used to extract two-dimensional features from each input image;

[0197] A depth distribution prediction network is used to predict the depth probability distribution for each input image.

[0198] The feature decoding network includes a target feature decoding network and an autonomous vehicle feature decoding network with the same architecture, which output the target pose BEV result and the autonomous vehicle pose BEV result, respectively.

[0199] An exemplary training method for the above network model parameters is shown in the appendix. Figure 10 As shown, the main steps include the following:

[0200] Step 1: Construct a dataset. Each dataset consists of synchronized images from multiple perspectives and labels. The synchronized images are captured simultaneously by a cluster of autonomous vehicles around the perception area, and the labels are the ground truth poses of the corresponding target BEVs. The positioning and attitude of BEVs with autonomous vehicles is truly valuable. The dataset contains different numbers of viewpoints and labels, which can enhance the model's robustness to changes in the number of input images, enabling the model to support a variable number of input images. This allows the model to maintain cooperative perception when the number of autonomous vehicles in the swarm changes, improving the flexibility and robustness of swarm cooperative perception and enhancing the system's scalability.

[0201] Step 2: Initialize the multi-view fusion autonomous vehicle cluster self-calibration collaborative perception model, including BEV grid initialization, reference coordinate system initialization, extrinsic parameter matrix initialization, camera intrinsic parameter initialization, learning rate initialization, and depth interval initialization.

[0202] Step 3: Load the training dataset.

[0203] Step 4: Randomly discard a single viewpoint image from each group of multi-view images, and input the remaining images into the multi-view fusion autonomous vehicle cluster self-calibration cooperative perception model. The output is the target pose BEV result and the autonomous vehicle pose BEV result. Randomly discarding multi-view images in the training dataset during training enhances the model's robustness to changes in the number of input images, enabling the model to support a variable number of input images. This maintains cooperative perception even when the number of autonomous vehicles in the cluster changes, improving the flexibility and robustness of cluster cooperative perception and enhancing system scalability.

[0204] Step 5: Calculate the total loss function, which is a weighted sum of the target pose loss function and the autonomous vehicle pose loss function. A ground truth supervised model for both target and autonomous vehicle poses enables the model to simultaneously output both the target pose BEV result and the autonomous vehicle pose BEV result, thereby improving the cluster's perception flexibility, robustness, and deployment efficiency.

[0205] Step 5-1: Calculate the target pose loss function L target For the target BEV pose view label B target H is processed using a two-dimensional Gaussian kernel. B W B The pose soft ground truth is obtained by processing the K channels using a one-dimensional Gaussian kernel. The target pose loss function can optimize the model parameters so that the model outputs the target pose BEV result.

[0206] Using both mean squared error loss and Focal loss to simultaneously supervise the heatmap, the target pose loss function can be expressed as:

[0207]

[0208] Where α1 and α2 are weights, MSELoss(·) represents the mean squared error loss function, which is used to force the model to regress to the true value, and FocalLoss(·) is used to reduce the impact of sample imbalance.

[0209] Step 5-2: Calculate the pose loss function L of the unmanned vehicle. UGV For the pose view label B of the autonomous vehicle (BEV) UGV H is processed using a two-dimensional Gaussian kernel. B W B The pose soft ground truth is obtained by processing the K channels using a one-dimensional Gaussian kernel. The autonomous vehicle pose loss function can optimize the extrinsic parameter matrix, enabling the model to output the autonomous vehicle pose BEV result. Based on the autonomous vehicle pose BEV result, the extrinsic parameter matrix between autonomous vehicles can be obtained, realizing the self-calibration of the extrinsic parameter matrix between autonomous vehicle clusters, thereby improving the deployment efficiency of cluster collaborative perception.

[0210] Using a Euclidean loss-supervised heatmap, the pose loss function of the autonomous vehicle can be expressed as:

[0211]

[0212] Step 5-3: Calculate the total loss function L, which can be expressed as:

[0213] L=λ1L target +λ2L UGV

[0214] Where λ1 and λ2 are weights.

[0215] Step 6: Optimize model parameters and extrinsic parameters jointly via backpropagation using gradient descent. The target pose loss function and the autonomous vehicle pose loss function are weighted to form the total loss function. End-to-end training is used to jointly optimize model parameters and extrinsic parameters, enabling the model to simultaneously output both target pose BEV results and autonomous vehicle pose BEV results. The target pose BEV results provide the target's pose distribution information within the region, and the autonomous vehicle pose BEV results yield the extrinsic parameters between autonomous vehicles. This allows for self-calibration of extrinsic parameters among the autonomous vehicle cluster, thereby improving the flexibility, robustness, and deployment efficiency of cluster collaborative perception. The synchronous optimization of model parameters and extrinsic parameters enhances global optimization efficiency.

[0216] Step 7: Repeat steps 3 through 6 until all training set data has been loaded.

[0217] Step 8: Load the test dataset, input the multi-view images into the multi-view fusion autonomous vehicle cluster self-calibration cooperative perception model, and output the target pose BEV result and the autonomous vehicle pose BEV result.

[0218] The above technical solutions are merely exemplary embodiments of the present invention. For those skilled in the art, based on the application methods and principles disclosed in the present invention, it is easy to make various types of improvements or modifications, and not limited to the methods described in the specific embodiments of the present invention. Therefore, the methods described above are merely preferred and not restrictive.

Claims

1. A multi-view fusion autonomous vehicle swarm self-calibration and cooperative perception method, comprising the following steps: S1: Initialization of self-calibration and collaborative perception processing of multi-view fusion unmanned vehicle cluster, including: BEV grid initialization, reference coordinate system initialization, extrinsic parameter matrix initialization, camera intrinsic parameter initialization, learning rate initialization, depth interval initialization, and voxel mesh partitioning initialization. S2: Distribute the cluster of unmanned vehicles around the target area, facing the target area, and collect images from different perspectives at the same time. There is overlap in the field of view between the images from different perspectives. S3: Extract, decode and optimize the consistency features of the multi-view overlapping field-of-view synchronized images acquired by the unmanned vehicle cluster to obtain the target pose BEV result and the unmanned vehicle pose BEV result; S4: Result the target pose BEV Results of autonomous vehicle posture BEV The pose parameters of the target and the autonomous vehicle are extracted through nonmaximum suppression. x , y , r The image is reprojected onto a two-dimensional BEV plane, input, and displayed on the interactive interface. S5: Obtain the target pose distribution information in the region from the target pose BEV results; obtain the extrinsic parameter matrix between autonomous vehicles based on the autonomous vehicle pose BEV results, and realize the self-calibration of the extrinsic parameter matrix between autonomous vehicle clusters; Step S3 specifically includes: S31, For each input image, a two-dimensional feature extraction network extracts two-dimensional features: A deep convolutional neural network is used as a shared backbone to extract multi-scale two-dimensional features from images at various viewpoints. The extracted two-dimensional features are represented by the following formula: in, For the first i Two-dimensional features extracted from the image C The number of feature channels, H , W This refers to the feature size, which is also the pixel size of the input image. This represents a two-dimensional feature extraction network. For the input of the first i Image i =1,2, …, N For the input image index, N The number of driverless vehicles in the cluster; S32: For each input image, the depth probability distribution is predicted by the depth distribution prediction network: A deep convolutional neural network is used as a shared backbone to extract the depth probability distribution of each image; The extracted depth probability distribution is represented by the following formula: in, For the first i Depth probability distribution of the image This represents a deep distribution prediction network; D The number of depth intervals; S33: For each pixel position (h, w) and depth z in each image... d Transform to the reference coordinate system. z d For the first d The actual distance of each depth interval d = 0, 1, …, D -1 represents the depth range index; Coordinate transformation is represented as: in, R i For rotation matrix, t i It is a translation vector. Indicates camera i The extrinsic matrix of the reference coordinate system K i For camera internal parameters; Calculate the voxel index: If satisfied , , If the point is not valid, keep it; otherwise, discard it. S34: For each image, for each transformed effective voxel position ( v x , v y , v z By accumulating voxel features and voxel weights, the features of images from different viewpoints are transformed to the same coordinate system. The voxel features of each image are represented as follows: The voxel weights for each image are represented as follows: Normalizing the voxel features of each image yields: in, , It is set to a very small value to prevent division by zero; S35: Voxel features of each image Along height z Axial summation pooling compression is used to extract BEV features, represented as follows: in, ; BEV features of all images Element-wise summation generates global multi-view BEV features; Global multi-view BEV features can be represented as: in, ; S36 will integrate global multi-view BEV features. Input the target feature decoding network and the autonomous vehicle feature decoding network respectively: The target feature decoding network and the autonomous vehicle feature decoding network are constructed using the same feature decoding network architecture. The neural network with large convolutional kernels is used as the decoder to perform spatial aggregation, extract target appearance and spatial consistency information, and realize multi-view feature complementarity. The target feature decoding network and the autonomous vehicle feature decoding network output the target pose BEV result, respectively. Results of autonomous vehicle posture BEV , is represented as: in, and All are three-dimensional heatmap matrices, satisfying ,exist H B , W B The dimension displays the two-dimensional relative positioning of the target or autonomous vehicle. The value of each point represents the probability that the target or autonomous vehicle appears at that point. K The channel displays the orientation information of the target or the autonomous vehicle, and the value of each point represents the probability that the target or the autonomous vehicle is facing that angle; H B , W B These represent the vertical and horizontal pixel counts of the final BEV output, respectively.

2. The method according to claim 1, characterized in that, Step S1 specifically includes: S11, BEV grid initialization: Based on the target perception area size, divide the horizontal and vertical dimensions of the final output BEV result grid, and set the number of orientation channels; BEV grid size is calculated using the following formula: in, H B , W B These represent the vertical and horizontal pixel counts of the final output BEV result, respectively. H W , W W These are the actual vertical and horizontal dimensions of the target perception region, respectively, and Δ is the actual size of a single BEV pixel. The number of facing channels is set to K The angle for each direction towards the channel is calculated using the following formula: in, k = 0, 1, …, K -1 represents the orientation channel index. r k For the first k The orientation angle of the passageway; S12, Reference coordinate system initialization: Select the camera coordinate system of an autonomous vehicle as the reference coordinate system. V 0, meaning the origin of the coordinate system is set as the target pose BEV and the autonomous vehicle pose BEV result in the final output. x = 0, y = 0, r =0; where, x , y , r These represent the two-dimensional coordinate position and orientation angle in the BEV, respectively. S13, Extrinsic parameter matrix initialization: Randomly initialize the extrinsic parameter matrices of other unmanned vehicle cameras relative to the reference coordinate system and set them as learnable parameters, then optimize them through backpropagation; S13, Camera Intrinsic Parameter Initialization: Read the camera's factory intrinsic parameter calibration results through the camera interface to keep the camera's focal length, principal point, and distortion coefficient unchanged; S15, Learning Rate Initialization: Set the learning rate for the extrinsic parameter matrix and model parameters, and set the parameters for the optimizer and learning rate scheduler; S16, Depth Interval Initialization: Set the minimum depth for the depth distribution prediction network. d min and maximum depth d max Divide the logarithmic interval depth range: in, D The number of depth intervals, d = 0, 1, …, D -1 represents the depth range index. z d For the first d The actual distance of each depth interval; S17, Voxel Mesh Generation Initialization: Let the voxel mesh size for the voxel feature generation process be... V X × V Y × V Z The side length of the voxel is s The spatial range is [ x min , x max ]×[ y min , y max ]×[ z min , z max ].

3. The method according to claim 1 or 2, characterized in that, In step S5, the method for obtaining the extrinsic parameter matrix between the unmanned vehicles is as follows: Set the BEV coordinate system to be the same as the world coordinate system, that is, the BEV's coordinate system... x , y The axis direction is consistent with the world coordinate system and coincides with the origin; for autonomous vehicles i and driverless cars j Their BEV coordinates are ( x i , y i )and( x j , y j ), the orientation angles are respectively θ i and θ j The ratio of the BEV's actual size to its size in the world coordinate system is s m / pixel; driverless car i and driverless cars j The actual coordinates in the world coordinate system are represented as follows: driverless car j To driverless cars i The translation vector is represented as: The relative rotation angle is: The rotation matrix is ​​then: This leads to the extrinsic parameter matrix: 。 4. A multi-view fusion autonomous vehicle swarm self-calibration and cooperative perception device applying the method described in claim 3, characterized in that, include: A multi-view image acquisition module is used to acquire synchronized images of the target area from different viewpoints; The two-dimensional feature extraction network module is used to extract two-dimensional features from each input image; The depth distribution prediction network module is used to predict the depth probability distribution for each input image; The coordinate transformation module transforms the position and depth of each pixel in each image to the reference coordinate system; The voxel feature generation module is used to accumulate voxel features and voxel weights for each image and for each converted effective voxel position, transforming the features of images from different viewpoints to the same coordinate system. The BEV feature generation module is used to sum the voxel features of each image along the height axis and then pool and compress them into BEV features. The feature decoding network module includes a target feature decoding network and an autonomous vehicle feature decoding network with the same architecture, which output the target pose BEV result and the autonomous vehicle pose BEV result, respectively. The self-calibration module is used to obtain the external parameter matrix between autonomous vehicles based on the autonomous vehicle pose BEV results.

5. A training method for the multi-view fusion autonomous vehicle swarm self-calibration cooperative perception device as described in claim 4, comprising the following steps: Step 1: Construct a dataset. Each dataset includes synchronized images from multiple perspectives and labels. The synchronized images are captured simultaneously by a cluster of autonomous vehicles around the target area, and the labels are the ground truth poses of the corresponding target BEVs. The positioning and attitude of BEVs with autonomous vehicles is truly valuable. Each set of data contains a different number of viewpoints and labels; Step 2: The multi-view fusion unmanned vehicle cluster self-calibration and collaborative perception device is initialized; Step 3: Load the training dataset; Step 4: From the input multi-view images, randomly select one view and discard it. Input the remaining images into the multi-view fusion autonomous vehicle cluster self-calibration and cooperative perception device, and output the target pose BEV result. Results of autonomous vehicle posture BEV ; Step 5: Calculate the total loss function, which is a weighted sum of the target pose loss function and the autonomous vehicle pose loss function; use the ground truth supervision model for target and autonomous vehicle poses to enable it to simultaneously output the target pose BEV result and the autonomous vehicle pose BEV result. Step 6: Optimize the device model parameters and extrinsic parameter matrix jointly through backpropagation using gradient descent; Step 7: Repeat steps 3 through 6 until all training set data has been loaded.

6. The method according to claim 5, characterized in that, The initialization in step 2 specifically includes: BEV grid initialization, reference coordinate system initialization, extrinsic parameter matrix initialization, camera intrinsic parameter initialization, learning rate initialization, and depth interval initialization.

7. The method according to claim 5 or 6, characterized in that, Step 5 specifically includes: Step 51, calculate the target pose loss function. L target : For the target BEV pose view label Two-dimensional Gaussian kernel processing is used. H B , W B Dimensions, processed using a one-dimensional Gaussian kernel K The channel obtains the pose soft truth value ; Using both mean squared error loss and Focal loss to supervise the heatmap, the target pose loss function is expressed as: in, α 1 and α 2 represents the weight. This represents the mean squared error loss function, used to force the model to regress to the true value. Used to reduce the impact of sample imbalance; Step 52, calculate the pose loss function of the unmanned vehicle. L UGV : For autonomous vehicle (BEV) pose view labels Two-dimensional Gaussian kernel processing is used. H B , W B Dimensions, processed using a one-dimensional Gaussian kernel K The channel obtains the pose soft truth value ; Using a Euclidean loss-supervised heatmap, the autonomous vehicle pose loss function is expressed as: ； Step 53: Calculate the total loss function L , is represented as: Where λ1 and λ2 are weights.

8. The method according to claim 5 or 6, characterized in that, It also includes the following steps: Step 8: Load the test dataset, input the multi-view images into the multi-view fusion unmanned vehicle cluster self-calibration cooperative perception device, and output the target pose BEV result and the unmanned vehicle pose BEV result.

Images