Multi-view fusion unmanned vehicle cluster self-calibration cooperative sensing method and device and training method

Abstract

The invention discloses a multi-view fusion unmanned vehicle cluster self-calibration cooperative sensing method and device and a training method, and the method comprises the steps: obtaining a multi-view overlapped view synchronous image, extracting two-dimensional features of each view image, predicting depth distribution, and generating a global multi-view BEV feature containing target appearance and space consistency information; a neural network with a large convolution kernel is used as a decoder to carry out space aggregation, multi-view feature complementation is realized, a target pose BEV result and an unmanned vehicle pose BEV result can be output at the same time through end-to-end training and joint optimization of model parameters and an external parameter matrix, pose distribution information of a target in a region can be obtained from the target pose BEV result, and the target pose distribution information of the unmanned vehicle can be obtained. According to an unmanned vehicle attitude BEV result, an external parameter matrix between unmanned vehicles can be obtained, and self-calibration of the external parameter matrix between unmanned vehicle clusters is realized, so that the flexibility, robustness and deployment efficiency of cluster collaborative awareness are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of unmanned vehicle cluster perception technology, and in particular to a multi-perspective fusion unmanned vehicle cluster self-calibration collaborative perception method, device and training method. Background Art

[0002] Unmanned vehicle cluster perception is of great significance in the fields of national defense, engineering, and special rescue. Unmanned vehicles are equipped with cameras and can observe targets from multiple perspectives from different positions, which can solve the problem of targets being easily obscured in special environments: for areas that need to be perceived, unmanned vehicle clusters are dispersed around the area. By collecting synchronized images of multiple perspectives and overlapping fields of view in the area, collaborative perception of targets in the area can be achieved, and a bird's eye view (BEV) perception result of the target area can be output. The BEV view can clearly show the location of the target, overcoming the influence of single camera being easily obscured, which is conducive to improving situational awareness capabilities.

[0003] However, current research on collaborative perception for swarms of autonomous vehicles (AVs) has largely focused on multi-view perception using fixed-pose cameras. Camera pose is generally equated with AV pose. Before deployment, multi-view collaborative perception algorithms for AV cameras require inter-camera extrinsic calibration using a checkerboard grid. This requires the relative poses of the cameras or AVs to be known in order to produce accurate BEV perception results. However, calibrating inter-camera extrinsic parameters means the relative poses of the AVs in the swarm cannot change during observation, otherwise the calibration results will be corrupted. This limits the flexibility of swarm collaborative perception. The need for external parameter calibration also restricts the deployment efficiency of swarm collaborative perception. Furthermore, most current multi-view collaborative perception algorithms use a fixed number of cameras, meaning a fixed number of input images. This makes them unable to adapt to changes in the number of platforms in the swarm during collaborative perception, limiting the deployment efficiency and system scalability. Calibration of external parameters is prone to errors as the swarm pose changes, which in turn affects perception results. External parameters and perception results are acquired step by step, which can easily introduce distributional error accumulation. Summary of the Invention

[0004] In response to the above-mentioned shortcomings of the existing technology, the present disclosure provides a multi-perspective fusion unmanned vehicle cluster self-calibration collaborative perception method and a corresponding model training method for perception targets such as pedestrians and vehicles, so as to improve the collaborative perception flexibility, robustness and deployment efficiency of the unmanned vehicle cluster system.

[0005] The multi-view fusion unmanned vehicle cluster self-calibration collaborative perception method provided by the present disclosure mainly includes the following steps:

[0006] S1: Initialization of collaborative perception processing for multi-view fusion unmanned vehicle cluster self-calibration, including: BEV grid initialization, reference coordinate system initialization, extrinsic matrix initialization, camera intrinsic parameter initialization, learning rate initialization, depth interval initialization, and voxel grid division initialization;

[0007] S2: Deploy the unmanned vehicle cluster dispersedly around the target area, facing the target area, and collect images from different perspectives at the same time. There is overlap between the fields of view of the images from different perspectives.

[0008] S3: Consistent feature extraction, decoding and optimization are performed on the multi-view overlapping field of view synchronized images obtained by the unmanned vehicle cluster to obtain the BEV results of the target pose and the unmanned vehicle pose;

[0009] S4: BEV result of target pose and BEV results for autonomous vehicle posture Through non-maximum suppression, the pose parameters (x, y, r) of the target and the unmanned vehicle are extracted, reprojected onto the two-dimensional BEV plane, input and displayed on the interactive interface;

[0010] S5: Obtain the target's posture distribution information in the area from the target posture BEV result; obtain the external parameter matrix between the unmanned vehicles based on the unmanned vehicle posture BEV result, and realize the self-calibration of the external parameter matrix between the unmanned vehicle clusters.

[0011] Furthermore, the step S1 specifically includes:

[0012] S11, BEV grid initialization: divide the horizontal and vertical dimensions of the final output BEV result grid according to the size of the target perception area, and set the number of orientation channels;

[0013] The BEV grid size is calculated as follows:

[0014]

[0015] Among them, H B ,W B are the number of vertical and horizontal pixels of the final output BEV result, H W ,W W are the actual vertical and horizontal sizes of the target perception area, respectively, and Δ is the actual size corresponding to a single pixel of BEV;

[0016] The number of orientation channels is set to K, and the angle of each orientation channel is calculated as follows:

[0017]

[0018] Among them, k = 0, 1, ..., K-1 is the index of the direction channel, r kis the orientation angle of the k-th orientation channel;

[0019] S12, reference coordinate system initialization: Select the camera coordinate system of an unmanned vehicle as the reference coordinate system V0, that is, set the coordinate origin x = 0, y = 0, r = 0 in the final output target pose BEV and the unmanned vehicle pose BEV result of the unmanned vehicle to be: x = 0, y = 0, r = 0; where x, y, and r represent the two-dimensional coordinate position and orientation angle in BEV respectively;

[0020] S13, extrinsic parameter matrix initialization: randomly initialize the extrinsic parameter matrices of other unmanned vehicle cameras relative to the reference coordinate system, set them as learnable parameters, and optimize them through backpropagation;

[0021] S13, camera internal parameter initialization: read the camera factory internal parameter calibration results through the camera interface, keeping the camera focal length, principal point, and distortion coefficient unchanged;

[0022] S15, learning rate initialization: set the learning rate of the external parameter matrix and model parameters, and set the parameters of the optimizer and learning rate scheduler;

[0023] S16, depth interval initialization: set the minimum depth d of the depth distribution prediction network min and the maximum depth d max , divided into logarithmically spaced depth intervals:

[0024]

[0025] Where D is the number of depth intervals, d = 0, 1, ..., D-1 is the depth interval index, z d is the actual distance of the dth depth interval;

[0026] S17, voxel grid initialization: Assume that the voxel grid size of the voxel feature generation process is V X ×V Y ×V Z , the voxel side length is s, and the spatial range is [x min ,x max ]×[y min ,y max ]×[z min ,z max ].

[0027] Furthermore, the step S3 specifically includes:

[0028] S31, for each input image, extract two-dimensional features through the two-dimensional feature extraction network:

[0029] A deep convolutional neural network is used as a shared backbone to extract multi-scale two-dimensional features from images of different viewpoints;

[0030] The extracted two-dimensional features are expressed as follows:

[0031] F i =F 2D (I i )

[0032] Among them, F i ∈R C×H×W is the two-dimensional feature extracted from the i-th image, C is the number of feature channels, H and W are the feature sizes, which are also the pixel sizes of the input image, and F 2D (·) represents the two-dimensional feature extraction network, I i ∈R 3×H×W is the i-th input image, i = 1, 2, ..., N is the input image index, and N is the number of unmanned vehicles in the cluster;

[0033] S32: For each input image, the depth distribution prediction network predicts the depth probability distribution:

[0034] A deep convolutional neural network is used as a shared backbone to extract the depth probability distribution of each image;

[0035] The extracted depth probability distribution is expressed as follows:

[0036] D i =F depth (I i )

[0037] Among them, D i ∈R D×H×W is the depth probability distribution of the i-th image, F depth (·) represents the deep distribution prediction network;

[0038] S33: Convert each pixel position (h, w) and depth zd of each image to a reference coordinate system;

[0039] The coordinate transformation is expressed as:

[0040]

[0041] Among them, R i is the rotation matrix, t i is the translation vector, E i =[R i |t i ] represents the external parameter matrix of camera i to the reference coordinate system, K i is the camera internal parameter;

[0042] Calculate the voxel index:

[0043]

[0044] If vx ∈[0,V X -1],v y ∈[0,V Y -1],v z ∈[0,V Z -1], then the point is retained, otherwise it is discarded;

[0045] S34: For each image, for each transformed valid voxel position (v x ,v y ,v z ) Accumulate voxel features and voxel weights to transform the features of each view image into the same coordinate system:

[0046] The voxel features of each image are represented as:

[0047] V i [c,v x ,v y ,v z ]+=F i (c,h,w)·D i (d,h,w)

[0048] The voxel weight of each image is expressed as:

[0049] W i [v x ,v y ,v z ]+=D i (d,h,w)

[0050] Then normalizing the voxel features of each image can be obtained:

[0051]

[0052] in, ε is a minimum value to prevent division by zero;

[0053] S35: The voxel feature V of each image i global The summation and pooling along the height z axis are compressed into BEV features, which are expressed as:

[0054] V i BEV =sum z {V i global [c,v x ,v y ,v z ]}

[0055] in,

[0056] The BEV features V of all images i BEV Element-by-element summation to generate global multi-view BEV features;

[0057] The global multi-view BEV feature can be expressed as:

[0058] V BEV =sum i {V i BEV}

[0059] in,

[0060] S36, the global multi-view BEV feature V BEV Input the target feature decoding network and the unmanned vehicle feature decoding network separately: The feature decoding network with the same architecture is used to construct the target feature decoding network and the unmanned vehicle feature decoding network respectively. A neural network with a large convolution kernel is used as the decoder to perform spatial aggregation, extract the target appearance and spatial consistency information, and realize multi-view feature complementarity;

[0061] The target feature decoding network and the unmanned vehicle feature decoding network output the target pose BEV results respectively and BEV results for autonomous vehicle posture Expressed as:

[0062]

[0063] in, and They are all three-dimensional heat map matrices, satisfying In H B 、W B The dimension displays the two-dimensional relative positioning of the target or unmanned vehicle. The value of each point represents the probability of the target or unmanned vehicle appearing at that point. The K channel displays the orientation information of the target or unmanned vehicle. The value of each point represents the probability of the target or unmanned vehicle facing that angle.

[0064] Furthermore, in step S5, the method for obtaining the extrinsic parameter matrix between the unmanned vehicles is:

[0065] The BEV coordinate system is set to be the same as the world coordinate system, that is, the x-axis and y-axis directions of the BEV are consistent with the world coordinate system and the origin coincides with the world coordinate system; for unmanned vehicle i and unmanned vehicle j, their BEV coordinates are (x i ,y i ) and (x j ,y j ), the orientation angles are θ i and θ j , the ratio of BEV to the actual size of the world coordinate system is sm / pixel;

[0066] The actual coordinates of unmanned vehicle i and unmanned vehicle j in the world coordinate system are expressed as:

[0067] (u i ,v i )=(s·x i ,s·y i )

[0068] (u j ,v j )=(s·x j ,s·y j )

[0069] The translation vector from unmanned vehicle j to unmanned vehicle i is expressed as:

[0070]

[0071] The relative rotation angle is:

[0072] θ r =θ j -θ i

[0073] Then the rotation matrix is:

[0074]

[0075] Then we get the external parameter matrix:

[0076]

[0077] The multi-view fusion unmanned vehicle cluster self-calibration collaborative perception device using the above method mainly includes:

[0078] Multi-view image acquisition module, used to acquire synchronized images of the target area from different viewpoints;

[0079] A two-dimensional feature extraction network module is used to extract the two-dimensional features of each input image;

[0080] The depth distribution prediction network module is used to predict the depth probability distribution of each input image; the coordinate transformation module converts the position and depth of each pixel of each image into the reference coordinate system;

[0081] The voxel feature generation module is used to accumulate voxel features and voxel weights for each valid voxel position after transformation in each image, and transform the features of each view image into the same coordinate system;

[0082] The BEV feature generation module is used to sum the voxel features of each image along the height axis and compress them into BEV features through pooling;

[0083] The feature decoding network module includes a target feature decoding network and an unmanned vehicle feature decoding network with the same architecture, which output the target pose BEV results and the unmanned vehicle pose BEV results respectively;

[0084] The self-calibration module is used to obtain the external parameter matrix between the unmanned vehicles based on the BEV results of the unmanned vehicle posture.

[0085] The training method applied to the above-mentioned collaborative sensing device mainly includes the following steps:

[0086] Step 1: Construct a dataset. Each set of data includes synchronized images and labels from multiple perspectives. The synchronized images are taken by a cluster of unmanned vehicles around the target area at the same time, and the labels are the corresponding target BEV pose truth values. and the true value of the BEV pose of the unmanned vehicle Each set of data contains different amounts of perspective data and labels;

[0087] Step 2: Initialize the multi-view fusion unmanned vehicle cluster self-calibration collaborative perception device;

[0088] Step 3: Load the training dataset;

[0089] Step 4: Randomly select one perspective image from the input multi-view images and discard it, input the remaining images into the multi-view fusion unmanned vehicle cluster self-calibration collaborative perception device, and output the target pose BEV result and BEV results for autonomous vehicle posture

[0090] Step 5: Calculate the total loss function, which is composed of the weighted loss function of the target pose and the unmanned vehicle pose; supervise the device model with the target and unmanned vehicle pose truth values, so that it can simultaneously output the target pose BEV result and the unmanned vehicle pose BEV result;

[0091] Step 6: Jointly optimize the device model parameters and the external parameter matrix through back propagation of the gradient descent method;

[0092] Step 7: Repeat steps 3 to 6 until all training set data is loaded.

[0093] Furthermore, the initialization in step 2 specifically includes: BEV grid initialization, reference coordinate system initialization, extrinsic parameter matrix initialization, camera intrinsic parameter initialization, learning rate initialization, and depth interval initialization.

[0094] Furthermore, the step 5 specifically includes:

[0095] Step 51, calculate the target pose loss function L target :

[0096] For the target BEV pose view label B target , using a two-dimensional Gaussian kernel to process H B ,W B Dimension, use one-dimensional Gaussian kernel to process K channel to get the soft truth value of pose

[0097] Using mean square error loss and focal loss to supervise the heat map at the same time, the target pose loss function is expressed as:

[0098]

[0099] Among them, α1 and α2 are weights, MSELoss(·) represents the mean square error loss function, which is used to force the model to regress to the true value, and FocalLoss(·) is used to reduce the impact of sample imbalance;

[0100] Step 52: Calculate the posture loss function L of the unmanned vehicle UGV :

[0101] For the BEV pose view label B UGV , using a two-dimensional Gaussian kernel to process H B ,W B Dimension, use one-dimensional Gaussian kernel to process K channel to get the soft truth value of pose

[0102] Using the Euclidean loss to supervise the heat map, the posture loss function of the unmanned vehicle is expressed as:

[0103]

[0104] Step 53: Calculate the total loss function L, expressed as:

[0105] L=λ1L target +λ2L UGV

[0106] Among them, λ1 and λ2 are weights.

[0107] Furthermore, the method further comprises the following steps:

[0108] Step 8: Load the test data set, input the multi-view images into the multi-view fusion unmanned vehicle cluster self-calibration collaborative perception device, and output the target pose BEV results and the unmanned vehicle pose BEV results.

[0109] Compared with the prior art, the present invention has the following advantages:

[0110] ① Construct the target pose loss function and the unmanned vehicle pose loss function, input multi-view overlapping field of view synchronized images, extract two-dimensional features from each view image, predict depth distribution and generate global multi-view BEV features containing target appearance and spatial consistency information, use a neural network with a large convolution kernel as a decoder for spatial aggregation, and achieve multi-view feature complementarity. Through end-to-end training, jointly optimize the model parameters and external parameter matrix, and can simultaneously output the target pose BEV results and the unmanned vehicle pose BEV results. The target pose BEV results can be used to obtain the pose distribution information of the target in the area. According to the unmanned vehicle pose BEV results, the external parameter matrix between unmanned vehicles can be obtained, and the self-calibration of the external parameter matrix between unmanned vehicle clusters can be achieved, thereby improving the flexibility, robustness and deployment efficiency of cluster collaborative perception;

[0111] ② Input multi-view overlapping field of view synchronized images. The generated global multi-view BEV feature has a fixed dimension and can support a variable number of input images. The dataset contains different numbers of view data and labels. During training, the multi-view images in the training dataset are randomly discarded, and the remaining images are input into the model to enhance the model's robustness to changes in the number of input images. The BEV results of the target pose and the unmanned vehicle pose are output. This can maintain collaborative perception when the number of unmanned vehicle clusters changes, enhance the scalability of the system, and thus improve the flexibility, robustness and deployment efficiency of cluster collaborative perception;

[0112] ③ Construct an unmanned vehicle pose loss function, input multi-view overlapping field of view synchronized images, generate global multi-view BEV features containing target appearance and spatial consistency information, and optimize the external parameter matrix through end-to-end training. The BEV result of the unmanned vehicle pose can be output. Based on the BEV result of the unmanned vehicle pose, the external parameter matrix between unmanned vehicles can be obtained, and the external parameter matrix between unmanned vehicle clusters can be self-calibrated, thereby improving the deployment efficiency of cluster collaborative perception;

[0113] ④ Input multi-view overlapping field of view synchronized images to generate global multi-view BEV features containing target appearance and spatial consistency information. Use a neural network with a large convolution kernel as a decoder for spatial aggregation to achieve multi-view feature complementarity, thereby realizing collaborative perception of the target when a single view is occluded. It can output the target pose BEV result, from which the pose distribution information of the target in the area can be obtained, thereby improving the robustness of cluster collaborative perception.

[0114] ⑤ Input multi-view overlapping field of view synchronized images to generate global multi-view BEV features containing target appearance and spatial consistency information. Use a neural network with a large convolution kernel as a decoder for spatial aggregation to achieve multi-view feature complementarity. This can maintain collaborative perception when the observation position of the unmanned vehicle cluster changes, and output the target pose BEV results, thereby improving the robustness of cluster collaborative perception.

[0115] ⑥ The target pose loss function and the unmanned vehicle pose loss function are weighted to form a total loss function, and the global optimization efficiency is improved through end-to-end training and joint optimization of model parameters and external parameter matrices. BRIEF DESCRIPTION OF THE DRAWINGS

[0116] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0117] Figure 1 Schematic diagram of an exemplary unmanned vehicle swarm system structure;

[0118] Figure 2 This is a flowchart of the self-calibration collaborative perception processing of a multi-view fusion unmanned vehicle cluster according to the present disclosure;

[0119] Figure 3 A schematic diagram of an exemplary unmanned vehicle swarm deployed around the target area;

[0120] Figure 4 A schematic diagram of the deployment of a multi-view fusion unmanned vehicle cluster self-calibration collaborative perception model according to the present disclosure;

[0121] Figure 5 Schematic diagram of the two-dimensional feature extraction network structure;

[0122] Figure 6 Schematic diagram of the depth distribution prediction network structure;

[0123] Figure 7 Schematic diagram of the feature decoding network structure;

[0124] Figure 8 The target pose result after non-maximum suppression and reprojection to BEV;

[0125] Figure 9 This is the pose result of the unmanned vehicle after non-maximum suppression and reprojection to BEV;

[0126] Figure 10 This is a flowchart for training and testing a multi-view fusion unmanned vehicle cluster self-calibration collaborative perception model according to the present disclosure. DETAILED DESCRIPTION

[0127] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0128] The present invention provides a multi-perspective fusion unmanned vehicle cluster self-calibration collaborative perception method, device and training method.

[0129] In an exemplary embodiment, the unmanned vehicle cluster system mainly consists of an unmanned vehicle cluster and a mission controller, as shown in the attached Figure 1-4 shown.

[0130] The unmanned vehicle cluster collects synchronized images and sends them to the mission controller, which runs a multi-view fusion unmanned vehicle cluster self-calibration collaborative perception model to perform collaborative perception processing and output the target and unmanned vehicle posture BEV perception results.

[0131] (1) An unmanned vehicle cluster consists of multiple unmanned vehicles, each of which contains an onboard computer, power supply, sensor module, and networking communication module;

[0132] The onboard computer is connected to the sensor module and the networking communication module, serving as the core module for controlling the unmanned vehicle, controlling the image acquisition and networking communication functions of the unmanned vehicle;

[0133] The power supply provides power to the unmanned vehicle, enabling the sensor module, on-board computer and networking communication module to operate normally;

[0134] The sensor module is connected to the vehicle's computer and uses a camera to capture images. The captured images are sent to the vehicle's computer and then to the mission controller via the networking communication module. These images serve as input to the multi-view fusion unmanned vehicle swarm self-calibration collaborative perception model. The camera is mounted at the geometric center of the unmanned vehicle and faces the vehicle's front. The camera's pose is the same as the vehicle's.

[0135] The networking communication module is connected to the on-board computer, enabling the unmanned vehicle to join the cluster system communication network for sending and receiving data.

[0136] (2) The task controller consists of a computing server, a networking communication module, and an interactive interface, where:

[0137] The computing server is connected to the networking communication module and the interactive interface, providing computing resources such as computing power, video memory, memory, and storage space. It runs the multi-view fusion unmanned vehicle cluster self-calibration collaborative perception model, receives synchronized images collected from the unmanned vehicle cluster through the networking communication module, performs collaborative perception processing, and outputs the obtained target and unmanned vehicle posture BEV perception results to the interactive interface for display.

[0138] The networking communication module is connected to the computing server, allowing the task controller to join the cluster system communication network for sending and receiving data;

[0139] The interactive interface is connected to the computing server, receives and displays the target and BEV perception results of the unmanned vehicle posture from the computing server.

[0140] In this embodiment, as shown in the attached Figure 2 As shown in the figure, the multi-view fusion unmanned vehicle cluster self-calibration collaborative perception processing process mainly includes the following steps:

[0141] Step 1: Deploy a swarm of autonomous vehicles at different locations around the target area. Each vehicle faces the target area and collects simultaneous images from different perspectives, with overlap between the images. Each autonomous vehicle has a unique digital identity. This multi-perspective perception of the target enables collaborative perception even when a single perspective is obstructed, thereby improving the robustness of the swarm's collaborative perception.

[0142] Step 2: The unmanned vehicle cluster sends the multi-view images to the mission controller through the networking communication module. The mission controller receives the multi-view images through the networking communication module and inputs them into the computing server. The computing server runs the multi-view fusion unmanned vehicle cluster self-calibration collaborative perception model.

[0143] Step 3: Initialize the multi-view fusion unmanned vehicle cluster self-calibration collaborative perception model, including BEV grid initialization, reference coordinate system initialization, external parameter matrix initialization, camera intrinsic parameter initialization, learning rate initialization, depth interval initialization, and voxel grid division initialization.

[0144] Step 3-1: Initialize the BEV grid. Divide the horizontal and vertical dimensions of the final output BEV result grid according to the target perception area size, and set the number of orientation channels. The BEV grid size is calculated by the following formula:

[0145]

[0146] Among them, H B ,W B are the number of vertical and horizontal pixels of the final output BEV result, H W ,W W are the actual vertical and horizontal sizes of the target perception area, respectively, and Δ is the actual size corresponding to a single pixel of BEV.

[0147] The number of orientation channels is set to K, and the angle of each orientation channel can be calculated as follows:

[0148]

[0149] Among them, k = 0, 1, ..., K-1 is the index of the direction channel, r k is the orientation angle of the k-th orientation channel.

[0150] Step 3-2: Initialize the reference coordinate system. Select the camera coordinate system of an unmanned vehicle as the reference coordinate system V0. This means that the coordinate origin of the unmanned vehicle in the final output target pose BEV and the unmanned vehicle pose BEV result is set to x = 0, y = 0, and r = 0. Here, x, y, and r represent the two-dimensional coordinate position and orientation angle in BEV, respectively.

[0151] Step 3-3: Initialize the extrinsic parameter matrix. Randomly initialize the extrinsic parameter matrices of other unmanned vehicle cameras relative to the reference coordinate system, set them as learnable parameters, and optimize them through backpropagation.

[0152] Step 3-4: Initialize the camera's internal parameters. Read the camera's factory internal calibration results through the camera interface to keep the camera's focal length, principal point, and distortion coefficient unchanged.

[0153] Steps 3-5: Initialize the learning rate, set the learning rate of the external parameter matrix and model parameters, and set the parameters of the optimizer and learning rate scheduler.

[0154] Step 3-6: Initialize the depth interval and set the minimum depth d of the depth distribution prediction network min and the maximum depth d max , divided into logarithmically spaced depth intervals:

[0155]

[0156] Where D is the number of depth intervals, d = 0, 1, ..., D-1 is the depth interval index, z d is the actual distance of the dth depth interval.

[0157] Step 3-7: Initialize voxel grid division and set the voxel grid size of the voxel feature generation process to V X ×V Y ×V Z , the voxel side length is s, and the spatial range is [x min ,x max ]×[y min ,y max ]×[z min ,z max ].

[0158] Step 4: Input the multi-view images into the multi-view fusion unmanned vehicle cluster self-calibration collaborative perception model, and output the target pose BEV results and the unmanned vehicle pose BEV results. The input multi-view overlapping field of view synchronized images contain the target's appearance and spatial information from different perspectives. The multi-view fusion unmanned vehicle cluster self-calibration collaborative perception model performs consistent feature extraction, decoding, and optimization, and directly outputs the target pose BEV results and the unmanned vehicle pose BEV results. The target pose BEV results can be used to obtain the pose distribution information of the target in the area. Based on the unmanned vehicle pose BEV results, the extrinsic parameter matrix between the unmanned vehicles can be obtained, enabling self-calibration of the extrinsic parameter matrix between the unmanned vehicle cluster, thereby improving the flexibility, robustness, and deployment efficiency of cluster collaborative perception.

[0159] Step 4-1: For each input image, extract the two-dimensional features through the two-dimensional feature extraction network. Figure 5 As shown in the figure, a deep convolutional neural network is used as a shared backbone to extract multi-scale two-dimensional features from each view. The two-dimensional features contain information about the appearance consistency of the target and are used to generate subsequent global multi-view BEV features.

[0160] The extracted two-dimensional features can be expressed as follows:

[0161] F i =F 2D (I i )

[0162] Among them, F i ∈R C×H×W is the two-dimensional feature extracted from the i-th image, C is the number of feature channels, H, W is the feature size, which is also the pixel size of the input image, and F 2D (·) represents the two-dimensional feature extraction network, I i ∈R 3×H×W is the i-th input image, i=1,2,…,N is the input image index, and N is the number of unmanned vehicles.

[0163] Step 4-2: For each input image, the depth distribution prediction network predicts the depth probability distribution. Figure 6 As shown in the figure, a deep convolutional neural network is used as a shared backbone to extract the depth probability distribution of each image. The depth probability distribution contains the spatial consistency information of the target and is used to generate the subsequent global multi-view BEV features.

[0164] The extracted depth probability distribution can be expressed as follows:

[0165] D i =F depth (I i )

[0166] Among them, D i∈R D×H×W is the depth probability distribution of the i-th image, F depth (·) denotes a deep distribution prediction network.

[0167] Step 4-3: Each pixel position (h, w) and depth z of each image d Transform to the reference coordinate system.

[0168] The coordinate transformation can be expressed as:

[0169]

[0170] Among them, R i is the rotation matrix, t i is the translation vector, E i =[R i |t i ] represents the external parameter matrix of camera i to the reference coordinate system, K i is the camera internal parameter.

[0171] Calculate the voxel index:

[0172]

[0173] If v x ∈[0,V X -1],v y ∈[0,V Y -1],v z ∈[0,V Z -1], then the point is retained; otherwise, it is discarded.

[0174] Step 4-4: For each image, for each transformed valid voxel position (v x ,v y ,v z ) accumulates voxel features and voxel weights, and transforms the features of images from each perspective into the same coordinate system.

[0175] The voxel features of each image can be expressed as:

[0176] V i [c,v x ,v y ,v z ]+=F i (c,h,w)·D i (d,h,w)

[0177] The voxel weight of each image can be expressed as:

[0178] W i [v x ,v y ,v z]+=D i (d,h,w)

[0179] Then normalizing the voxel features of each image can be obtained:

[0180]

[0181] in, ε is a very small value to prevent division by zero.

[0182] Step 4-5: The voxel feature V of each image i global The summation and pooling along the height z axis are compressed into BEV features, which can be expressed as:

[0183] V i BEV =sum z {V i global [c,v x ,v y ,v z ]}

[0184] in,

[0185] The BEV features V of all images i BEV The element-by-element summation generates a global multi-view BEV feature. This feature contains information about target appearance and spatial consistency. Its dimensionality is fixed, allowing it to support a variable number of input images. This allows it to maintain collaborative perception even when the number of autonomous vehicle clusters changes, enhancing system scalability and the flexibility of cluster collaborative perception.

[0186] The global multi-view BEV feature can be expressed as:

[0187] V BEV =sum i {V i BEV}

[0188] in,

[0189] Step 4-6: Transform the global multi-view BEV feature V BEV Input the target feature decoding network and the unmanned vehicle feature decoding network respectively. Use the feature decoding network with the same architecture to construct the target feature decoding network and the unmanned vehicle feature decoding network respectively, as shown in the attached figure. Figure 7As shown in the figure, a neural network with a large convolution kernel is used as a decoder to perform spatial aggregation, extract target appearance and spatial consistency information, and achieve multi-view feature complementarity. It can maintain collaborative perception when the observation position of the unmanned vehicle cluster changes, and realize collaborative perception of the target when a single view is blocked, thereby improving the robustness of cluster collaborative perception.

[0190] The target feature decoding network and the unmanned vehicle feature decoding network output the target pose BEV results respectively and BEV results for autonomous vehicle posture It can be expressed as:

[0191]

[0192] in, and They are all three-dimensional heat map matrices, satisfying In H B ,W B The dimension displays the two-dimensional relative positioning of the target or unmanned vehicle. The value of each point represents the probability of the target or unmanned vehicle appearing at that point. The K channel displays the orientation information of the target or unmanned vehicle. The value of each point represents the probability of the target or unmanned vehicle facing that angle.

[0193] Step 5: BEV result of target pose and BEV results for autonomous vehicle posture The pose parameters (x, y, r) of the target and the unmanned vehicle are extracted by non-maximum suppression, reprojected to the two-dimensional BEV plane, input and displayed on the interactive interface, as shown in the attached figure. Figure 8 and 9 shown.

[0194] The pose distribution information of the target in the area can be obtained from the BEV results of the target pose; the relative pose between the unmanned vehicles can be obtained according to the BEV results of the unmanned vehicle pose, from which the translation matrix + rotation matrix between the unmanned vehicles can be obtained, that is, the extrinsic parameter matrix between the unmanned vehicles, realizing the self-calibration of the extrinsic parameter matrix between the unmanned vehicle clusters.

[0195] It can be seen that the multi-view fusion unmanned vehicle cluster self-calibration collaborative perception model described in this embodiment mainly includes:

[0196] A two-dimensional feature extraction network is used to extract the two-dimensional features of each input image;

[0197] Depth distribution prediction network, used to predict the depth probability distribution of each input image;

[0198] The feature decoding network includes a target feature decoding network and an unmanned vehicle feature decoding network with the same architecture, which output the target pose BEV results and the unmanned vehicle pose BEV results respectively.

[0199] The exemplary training method for the above network model parameters is shown in the attached Figure 10 As shown, it mainly includes the following steps:

[0200] Step 1: Construct a dataset. Each set of data consists of synchronized images and labels from multiple perspectives. The synchronized images are taken simultaneously by a cluster of unmanned vehicles around the perception area, and the labels are the corresponding target BEV pose truth values. and the true value of the BEV pose of the unmanned vehicle The dataset contains different amounts of perspective data and labels, which can enhance the model's robustness to changes in the number of input images, enabling the model to support a variable number of input images, thereby maintaining collaborative perception when the number of unmanned vehicle clusters changes, improving the flexibility and robustness of cluster collaborative perception, and enhancing the scalability of the system.

[0201] Step 2: Initialize the multi-view fusion unmanned vehicle cluster self-calibration collaborative perception model, including BEV grid initialization, reference coordinate system initialization, external parameter matrix initialization, camera internal parameter initialization, learning rate initialization, and depth interval initialization.

[0202] Step 3: Load the training dataset.

[0203] Step 4: Randomly discard a single view image from each set of multi-view images and input the remaining images into the multi-view fusion unmanned vehicle cluster self-calibration collaborative perception model, outputting the BEV results for the target pose and the unmanned vehicle pose. Randomly discarding multi-view images from the training dataset during training enhances the model's robustness to changes in the number of input images, enabling the model to support a variable number of input images, thereby maintaining collaborative perception even when the number of unmanned vehicle clusters changes. This improves the flexibility and robustness of cluster collaborative perception and enhances system scalability.

[0204] Step 5: Calculate the total loss function, which is composed of the weighted loss function of the target pose and the self-driving car pose. The model is supervised by the ground truth of the target and self-driving car poses, enabling the model to simultaneously output the target pose BEV results and the self-driving car pose BEV results, thereby improving the flexibility, robustness and deployment efficiency of cluster perception.

[0205] Step 5-1: Calculate the target pose loss function L target For the target BEV pose view label B target , using a two-dimensional Gaussian kernel to process H B ,W B Dimension, use one-dimensional Gaussian kernel to process K channel to get the soft truth value of pose The target pose loss function can optimize the model parameters so that the model outputs the target pose BEV result.

[0206] Using mean square error loss and focal loss to supervise the heat map at the same time, the target pose loss function can be expressed as:

[0207]

[0208] Among them, α1 and α2 are weights, MSELoss(·) represents the mean square error loss function, which is used to force the model to regress to the true value, and FocalLoss(·) is used to reduce the impact of sample imbalance.

[0209] Step 5-2: Calculate the pose loss function L of the unmanned vehicle UGV For the BEV pose view label B UGV , using a two-dimensional Gaussian kernel to process H B ,W B Dimension, use one-dimensional Gaussian kernel to process K channel to get the soft truth value of pose The unmanned vehicle posture loss function can optimize the extrinsic parameter matrix so that the model outputs the unmanned vehicle posture BEV result. According to the unmanned vehicle posture BEV result, the extrinsic parameter matrix between unmanned vehicles can be obtained, and the self-calibration of the extrinsic parameter matrix between unmanned vehicle clusters can be realized, thereby improving the deployment efficiency of cluster collaborative perception.

[0210] Using the Euclidean loss to supervise the heat map, the pose loss function of the unmanned vehicle can be expressed as:

[0211]

[0212] Step 5-3: Calculate the total loss function L, which can be expressed as

[0213] L=λ1L target +λ2L UGV

[0214] Among them, λ1 and λ2 are weights.

[0215] Step 6: Jointly optimize the model parameters and extrinsic parameter matrix through gradient descent backpropagation. The target pose loss function and the unmanned vehicle pose loss function are weighted to form a total loss function. End-to-end training is then used to jointly optimize the model parameters and extrinsic parameter matrix, enabling the model to simultaneously output the target pose BEV results and the unmanned vehicle pose BEV results. The target pose BEV results provide information about the pose distribution of the target in the area. The unmanned vehicle pose BEV results provide information about the extrinsic parameter matrix between unmanned vehicles, enabling self-calibration of the extrinsic parameter matrix between unmanned vehicle clusters. This improves the flexibility, robustness, and deployment efficiency of cluster collaborative perception. The model parameters and extrinsic parameter matrix are optimized simultaneously, enhancing global optimization efficiency.

[0216] Step 7: Repeat steps 3 to 6 until all training set data is loaded.

[0217] Step 8: Load the test dataset, input the multi-view images into the multi-view fusion unmanned vehicle cluster self-calibration collaborative perception model, and output the target pose BEV results and the unmanned vehicle pose BEV results.

[0218] The above technical solutions are only exemplary embodiments of the present invention. For those skilled in the art, it is easy to make various types of improvements or modifications based on the application methods and principles disclosed in the present invention, and are not limited to the methods described in the above specific embodiments of the present invention. Therefore, the methods described above are only preferred and do not have a restrictive meaning.

Claims

1. A multi-view fusion unmanned vehicle cluster self-calibration collaborative perception method, comprising the following steps: S1: Initialization of collaborative perception processing for multi-view fusion unmanned vehicle cluster self-calibration, including: BEV grid initialization, reference coordinate system initialization, extrinsic matrix initialization, camera intrinsic parameter initialization, learning rate initialization, depth interval initialization, and voxel grid division initialization; S2: Deploy the unmanned vehicle cluster around the target area, facing the target area, and collect images from different perspectives at the same time. There is overlap between the fields of view of the images from different perspectives. S3: Consistent feature extraction, decoding and optimization are performed on the multi-view overlapping field of view synchronized images obtained by the unmanned vehicle cluster to obtain the BEV results of the target pose and the unmanned vehicle pose; S4: BEV result of target pose and BEV results for autonomous vehicle posture Through non-maximum suppression, the pose parameters (x, y, r) of the target and the unmanned vehicle are extracted, reprojected onto the two-dimensional BEV plane, input and displayed on the interactive interface; S5: Obtain the target's posture distribution information in the area from the target posture BEV result; obtain the external parameter matrix between the unmanned vehicles based on the unmanned vehicle posture BEV result, and realize the self-calibration of the external parameter matrix between the unmanned vehicle clusters.

2. The method according to claim 1, characterized in that The step S1 specifically includes: S11, BEV grid initialization: divide the horizontal and vertical dimensions of the final output BEV result grid according to the size of the target perception area, and set the number of orientation channels; The BEV grid size is calculated as follows: Among them, H B ,W B are the number of vertical and horizontal pixels of the final BEV output result, H W ,W W are the actual vertical and horizontal sizes of the target perception area, respectively, and Δ is the actual size corresponding to a single pixel of BEV; The number of orientation channels is set to K, and the angle of each orientation channel is calculated as follows: Among them, k = 0, 1, ..., K-1 is the index of the direction channel, r k is the orientation angle of the k-th orientation channel; S12, reference coordinate system initialization: Select the camera coordinate system of an unmanned vehicle as the reference coordinate system V0, that is, set the coordinate origin x = 0, y = 0, r = 0 in the final output target pose BEV and the unmanned vehicle pose BEV result of the unmanned vehicle to be: x = 0, y = 0, r = 0; where x, y, and r represent the two-dimensional coordinate position and orientation angle in BEV respectively; S13, extrinsic parameter matrix initialization: randomly initialize the extrinsic parameter matrices of other unmanned vehicle cameras relative to the reference coordinate system, set them as learnable parameters, and optimize them through backpropagation; S13, camera internal parameter initialization: read the camera factory internal parameter calibration results through the camera interface, keeping the camera focal length, principal point, and distortion coefficient unchanged; S15, learning rate initialization: set the learning rate of the external parameter matrix and model parameters, and set the parameters of the optimizer and learning rate scheduler; S16, depth interval initialization: set the minimum depth d of the depth distribution prediction network min and the maximum depth d max , divided into logarithmically spaced depth intervals: Where D is the number of depth intervals, d = 0, 1, ..., D-1 is the depth interval index, z d is the actual distance of the dth depth interval; S17, voxel grid initialization: Assume that the voxel grid size of the voxel feature generation process is V X ×V Y ×V Z , the voxel side length is s, and the spatial range is [x min ,x max ]×[y min ,y max ]×[z min ,z max ].

3. The method according to claim 2, characterized in that The step S3 specifically includes: S31, for each input image, extract two-dimensional features through the two-dimensional feature extraction network: A deep convolutional neural network is used as a shared backbone to extract multi-scale two-dimensional features from images of different viewpoints; The extracted two-dimensional features are expressed as follows: F i =F 2D (I i ) Among them, F i ∈R C×H×W is the two-dimensional feature extracted from the i-th image, C is the number of feature channels, H and W are the feature sizes, which are also the pixel sizes of the input image, and F 2D (·) represents the two-dimensional feature extraction network, I i ∈R 3×H×W is the i-th input image, i = 1, 2, ..., N is the input image index, and N is the number of unmanned vehicles in the cluster; S32: For each input image, the depth distribution prediction network predicts the depth probability distribution: A deep convolutional neural network is used as a shared backbone to extract the depth probability distribution of each image; The extracted depth probability distribution is expressed as follows: D i =F depth (I i ) Among them, D i ∈R D×H×W is the depth probability distribution of the i-th image, F depth (·) represents the deep distribution prediction network; S33: Convert each pixel position (h, w) and depth zd of each image to a reference coordinate system; The coordinate transformation is expressed as: Among them, R i is the rotation matrix, t i is the translation vector, E i =[R i |t i ] represents the external parameter matrix of camera i to the reference coordinate system, K i is the camera internal reference; Calculate the voxel index: If v x ∈[0,V X -1],v y ∈[0,V Y -1],v z ∈[0,V Z -1], then the point is retained, otherwise it is discarded; S34: For each image, for each transformed valid voxel position (v x ,v y ,v z ) Accumulate voxel features and voxel weights to transform the features of each view image into the same coordinate system: The voxel features of each image are represented as: V i [c,v x ,v y ,v z ]+=F i (c,h,w)·D i (d,h,w) The voxel weight of each image is expressed as: W i [v x ,v y ,v z ]+=D i (d,h,w) Then normalizing the voxel features of each image can be obtained: in, ε is a minimum value to prevent division by zero; S35: The voxel feature V of each image i global The summation and pooling along the height z axis are compressed into BEV features, which are expressed as: V i BEV =sum z {V i global [c,v x ,v y ,v z ]} in, The BEV features V of all images i BEV Element-by-element summation to generate global multi-view BEV features; The global multi-view BEV feature can be expressed as: V BEV =sum i {V i BEV } in, S36, the global multi-view BEV feature V BEV Input the target feature decoding network and the unmanned vehicle feature decoding network separately: The feature decoding network with the same architecture is used to construct the target feature decoding network and the unmanned vehicle feature decoding network respectively. A neural network with a large convolution kernel is used as the decoder to perform spatial aggregation, extract the target appearance and spatial consistency information, and realize multi-view feature complementarity; The target feature decoding network and the unmanned vehicle feature decoding network output the target pose BEV results respectively and BEV results for autonomous vehicle posture Expressed as: in, and They are all three-dimensional heat map matrices, satisfying In H B 、W B The dimension displays the two-dimensional relative positioning of the target or unmanned vehicle. The value of each point represents the probability of the target or unmanned vehicle appearing at that point. The K channel displays the orientation information of the target or unmanned vehicle. The value of each point represents the probability of the target or unmanned vehicle facing that angle.

4. The method according to any one of claims 1 to 3, characterized in that In step S5, the method for obtaining the extrinsic parameter matrix between the unmanned vehicles is: The BEV coordinate system is set to be the same as the world coordinate system, that is, the x-axis and y-axis directions of the BEV are consistent with the world coordinate system and the origin coincides with the world coordinate system; for unmanned vehicle i and unmanned vehicle j, their BEV coordinates are (x i ,y i ) and (x j ,y j ), the orientation angles are θ i and θ j , the ratio of BEV to the actual size of the world coordinate system is sm / pixel; The actual coordinates of unmanned vehicle i and unmanned vehicle j in the world coordinate system are expressed as: (u i ,v i )=(s·x i ,s·y i ) (u j ,v j )=(s·x j ,s·y j ) The translation vector from unmanned vehicle j to unmanned vehicle i is expressed as: The relative rotation angle is: i r =θ j -θ i Then the rotation matrix is: Then we get the external parameter matrix:

5. A multi-view fusion unmanned vehicle cluster self-calibration collaborative perception device using the method of claim 4, characterized in that: include: Multi-view image acquisition module, used to acquire synchronized images of the target area from different viewpoints; A two-dimensional feature extraction network module is used to extract the two-dimensional features of each input image; Depth distribution prediction network module, used to predict the depth probability distribution of each input image; The coordinate transformation module converts the position and depth of each pixel of each image into a reference coordinate system; The voxel feature generation module is used to accumulate voxel features and voxel weights for each valid voxel position after transformation in each image, and transform the features of each view image into the same coordinate system; The BEV feature generation module is used to sum the voxel features of each image along the height axis and compress them into BEV features through pooling; The feature decoding network module includes a target feature decoding network and an unmanned vehicle feature decoding network with the same architecture, which output the target pose BEV results and the unmanned vehicle pose BEV results respectively; The self-calibration module is used to obtain the external parameter matrix between the unmanned vehicles based on the BEV results of the unmanned vehicle posture.

6. A training method for the multi-view fusion unmanned vehicle swarm self-calibration collaborative perception device of claim 5, comprising the following steps: Step 1: Construct a dataset. Each set of data includes synchronized images and labels from multiple perspectives. The synchronized images are taken by a cluster of unmanned vehicles around the target area at the same time, and the labels are the corresponding target BEV pose truth values. and the true value of the BEV pose of the unmanned vehicle Each set of data contains different amounts of perspective data and labels; Step 2: Initialize the multi-view fusion unmanned vehicle cluster self-calibration collaborative perception device; Step 3: Load the training dataset; Step 4: Randomly select one perspective image from the input multi-view images and discard it, input the remaining images into the multi-view fusion unmanned vehicle cluster self-calibration collaborative perception device, and output the target pose BEV result and BEV results for autonomous vehicle posture Step 5: Calculate the total loss function, which is composed of the weighted loss function of the target pose and the unmanned vehicle pose; supervise the device model with the target and unmanned vehicle pose truth values, so that it can simultaneously output the target pose BEV result and the unmanned vehicle pose BEV result; Step 6: Jointly optimize the device model parameters and the external parameter matrix through back propagation of the gradient descent method; Step 7: Repeat steps 3 to 6 until all training set data is loaded.

7. The method according to claim 6, characterized in that The initialization in step 2 specifically includes: BEV grid initialization, reference coordinate system initialization, extrinsic parameter matrix initialization, camera intrinsic parameter initialization, learning rate initialization, and depth interval initialization.

8. The method according to claim 6 or 7, characterized in that The step 5 specifically includes: Step 51, calculate the target pose loss function L target : For the target BEV pose view label B target , using a two-dimensional Gaussian kernel to process H B ,W B Dimension, use one-dimensional Gaussian kernel to process K channel to get the soft truth value of pose Using mean square error loss and focal loss to supervise the heat map at the same time, the target pose loss function is expressed as: Among them, α1 and α2 are weights, MSELoss(·) represents the mean square error loss function, which is used to force the model to regress to the true value, and FocalLoss(·) is used to reduce the impact of sample imbalance; Step 52: Calculate the posture loss function L of the unmanned vehicle UGV : For the BEV pose view label B UGV , using a two-dimensional Gaussian kernel to process H B ,W B Dimension, use one-dimensional Gaussian kernel to process K channel to get the soft truth value of pose Using the Euclidean loss to supervise the heat map, the pose loss function of the unmanned vehicle is expressed as: Step 53: Calculate the total loss function L, expressed as: L=λ1L target +λ2L UGV Among them, λ1 and λ2 are weights.

9. The method according to claim 6 or 7, characterized in that The following steps are also included: Step 8: Load the test data set, input the multi-view images into the multi-view fusion unmanned vehicle cluster self-calibration collaborative perception device, and output the target pose BEV results and the unmanned vehicle pose BEV results.

Images