Multi-unmanned aerial vehicle cooperative 3D sensing method based on channel adaptation

Abstract

A multi-unmanned aerial vehicle cooperative 3D sensing method based on channel adaptation relates to the field of communication, and comprises establishing a multi-unmanned aerial vehicle cooperative 3D sensing framework based on channel adaptation, a BEV semantic coding module and a BEV semantic decoding module. And the unmanned aerial vehicle encodes and fuses the acquired image into semantic features and shares the semantic features to other unmanned aerial vehicles through a complex channel environment, and receives the decoded semantic features of the unmanned aerial vehicle to enhance the 3D perception ability of the unmanned aerial vehicle. The BEV semantic coding module is used for adjusting and adapting a process of fusing self multi-view features into BEV features according to a channel state and compressing and coding the BEV features into semantic features by the unmanned aerial vehicle at the transmitting end; and the BEV semantic decoding module is used for the receiving end unmanned aerial vehicle to decode the semantic features according to the channel state and fuse a plurality of unmanned aerial vehicle features into a perceptible feature vector. According to the method, the interference of channel noise on cooperative 3D sensing is reduced, and the cooperative 3D sensing performance of multiple unmanned aerial vehicles in a complex channel environment is improved.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a multi-UAV cooperative 3D perception method based on channel adaptation. Background Technology

[0002] Traditionally, single UAVs based on deep neural network methods have made significant progress in target detection tasks. As the deployment scale of UAVs increases, multiple UAVs can share perception information and cooperate to perform target detection, which can greatly expand the perception range and improve the perception accuracy in complex environments.

[0003] Literature (Who2com: Collaborative Perception via Learnable Handshake Communication (Y.-C. Liu, J. Tian, ​​C.-Y. Ma, N. Glaser, C.-W. Kuo and Z. Kira, "Who2com: Collaborative Perception via Learnable Handshake Communication," 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France, 2020, pp. 6876-6883, doi:10.1109 / ICRA40945.2020.9197364. proposes a multi-agent cooperative sensing method. This scheme considers cooperative sensing in ideal channel environments but with limited bandwidth, proposing a three-stage communication mechanism. This mechanism includes the following steps: 1) Request phase: Performance-constrained agents broadcast compressed requests based on their visual observations; 2) Matching phase: Other agents calculate the matching score between their visual observations and the received requests using a learned matching algorithm; 3) Connection phase: Constrained agents select communication objects to improve their prediction accuracy in downstream sensing tasks (such as semantic segmentation). The entire mechanism adopts an end-to-end training approach, requiring only supervision signals from downstream tasks.

[0004] This scheme considers a multi-agent cooperative perception task with N agents in the environment, each capable of acquiring observation information. However, due to environmental occlusion and other factors, the target agent's observation information is incomplete. The target agent aims to complete its own observation information by integrating information from other agents. The proposed communication mechanism comprises three core steps: request, matching, and connection. Specifically, the target agent first broadcasts its request message to neighboring normal agents. Normal agents then calculate the matching score between their own keys and the request message. After the normal agents return the matching scores, the degraded agents select n agents based on these scores to establish connections (i.e., receive information). In each step, agents can achieve information compression through a key generator, message generator, image encoder, and task decoder. However, this scheme only considers bandwidth-constrained transmission scenarios in cooperative perception tasks, assuming ideal channels. In reality, however, the channel environment is a noisy wireless channel. Agents share features and cooperate through independent wireless links, and each communication link is a noisy and time-varying fading channel, causing varying degrees of interference to data transmission and thus reducing the target agent's perception accuracy.

[0005] In summary, there is a need to explore a new method for collaborative sensing among multiple UAVs in complex wireless channel environments. Summary of the Invention

[0006] The purpose of this invention is to propose a channel-adaptive multi-UAV cooperative 3D perception method to improve the performance of multi-UAV cooperative 3D perception in complex wireless channel environments.

[0007] The technical solution adopted by this invention to solve the technical problem is as follows:

[0008] This invention provides a channel-adaptive multi-UAV cooperative 3D perception method, which mainly includes the following steps:

[0009] (1) Establish a multi-UAV collaborative 3D perception framework based on channel adaptation. Consider multiple UAVs to perform collaborative 3D perception in a complex wireless channel environment. The transmitting UAV needs to fuse and encode the image data and channel information acquired by its own sensors into a semantic feature vector, and share the semantic feature vector with other UAVs through the complex wireless channel environment. The receiving UAV enhances its own 3D perception capability by decoding the semantic feature vector.

[0010] (2) Design a BEV semantic coding module and a BEV semantic decoding module; the BEV semantic coding module includes a multi-view BEV transformation module and a channel coding module, which are used to adjust and adapt the process of fusing its own multi-view features into BEV features according to the channel state of the transmitting UAV, and compress and encode it into a semantic feature vector; the BEV semantic decoding module includes a channel decoding module and a multi-UAV feature fusion module, which are used to adaptively decode the received semantic features according to the channel state of the receiving UAV, and fuse the decoded multiple UAV features into a perceptible feature vector.

[0011] Furthermore, the transmitting UAV uses a WinTransformer network and a GeneralizedLSSFPN network as image encoders to perform multi-scale feature extraction and fusion, obtaining feature data and feature fusion data from K perspectives; the mathematical expression for the feature data from the K perspectives is:

[0012] F n =f GLSSFPN (f SwinT (X n ))

[0013] in, This represents the image data acquired by the sending drone n, where H0 is the image height, W0 is the image width, and f is the image height. SwinT (·) indicates that the SwinTransformer network is used for multi-scale feature extraction, f GLSSFPN (·) indicates that a GeneralizedLSSFPN network is used for multi-scale feature fusion. This represents the extracted feature data from K perspectives, where C1 is the number of feature data channels, H1 is the feature data height, and W1 is the feature data width.

[0014] The calculation process for the feature fusion data is represented as follows:

[0015] S3←f Interpolate (S3)

[0016] S2←f Conv3×3Norm (f Conv1×1Norm (S2|S3))

[0017] S2←f Interpolate (S2)

[0018] S1←f Conv3×3Norm (f Conv1×1Norm (S1|S2))

[0019] F n =S1

[0020] Where {S1,S2,S3} are the three-stage features given by the SwinTransformer network, ·|· represents the Concat operation in the channel layer, f Interpolate (·) denotes the interpolation upsampling function, f Conv1×1Norm (·) represents a 1×1 convolution operation and a BatchNorm2d operation, f Conv3×3Norm (·) indicates the operation of 3×3 convolution and BatchNorm2d.

[0021] Furthermore, the semantic features R received by the receiving drone n' nn' Represented as:

[0022] R nn' =h*T nn' +ω

[0023] Where h is the channel fading coefficient; ω is the noise coefficient, which follows a circularly symmetric Gaussian distribution CN(0,σ). 2 ), σ 2 T represents the noise power of the channel. nn' The semantic features generated by sending drone n to drone n'.

[0024] Furthermore, the BEV semantic decoding module decodes and fuses the received semantic features into perceptible BEV features P based on the channel state of each link. n' The receiving UAV uses a 3D target detection module to detect BEV features. n' Target detection is performed; the 3D target detection module includes a DownSample network, a ResNet network, an LSSFPN network, and a Head network. The target detection process of the receiving UAV n' is represented as follows:

[0025] O n' =f Head (f LSSFPN (f ResNet (f DS (P n' ))))

[0026] Among them, f DS (·) indicates that downsampling is performed using the DownSample network, f Resnet (·) indicates that a ResNet network is used for multi-scale feature transformation, f LSSFPN (·) indicates the use of the LSSFPN network for multi-scale feature fusion, f Head (·) indicates that the head network is used to make final adjustments to the channel output based on the 3D object detection task. n' This indicates the output of the task.

[0027] Furthermore, the DownSample network consists of a three-layer convolutional network, each layer including convolution operations, BatchNorm2d operations, and ReLU operations; the Head network is a single-layer convolutional network used to adjust the final channel output to adapt to the object detection task; the LSSFPN network performs multi-scale fusion of the two-stage features {S1, S3} given by the ResNet network, and its specific calculation process is expressed as follows:

[0028] S3←f Interpolate (S3)

[0029] S1←f Conv3×3NormRelu (f Conv1×1NormRelu (S1|S3))

[0030] S1←f Conv3×3NormRelu (f UpSample (S1))

[0031] O n' =S1

[0032] Where ·|· represents the Concat operation at the channel layer, f Interpolate (·) denotes the interpolation upsampling function, f Conv1×1NormRelu (·) represents a 1×1 convolution operation, a BatchNorm2d operation, and a ReLU operation, f Conv3×3NormRelu (·) represents 3×3 convolution operation, BatchNorm2d operation, and ReLU operation.

[0033] Furthermore, the multi-view BEV transformation module transforms and fuses the feature data from the K-viewpoint into a unified BEV viewpoint to obtain BEV viewpoint features. The specific implementation process is as follows:

[0034] Construct a 3D mesh M in pixel coordinate system pic And it is mapped onto a 3D mesh M in the world coordinate system using the camera's intrinsic and extrinsic parameter matrices. world For multi-view image features F n First, calculate the probability distribution D at different depths. p and semantic features D f Next, the semantic feature distribution D at different depths is calculated; finally, the semantic feature distributions within each grid are summed, and the final BEV view feature V is obtained through a downsampling network. n :

[0035] V n =f DS (f BEVT (F n ,I,E))

[0036] Among them, fBEVT (·,·,·) represents the multi-view BEV transformation module; This is the intrinsic parameter matrix of the camera; f is the extrinsic parameter matrix of the camera; DS (·) indicates a downsampling network used to filter compressed features.

[0037] Furthermore, the channel coding module uses the channel signal-to-noise ratio (SNR) between the transmitting UAV n and the receiving UAV n'. nn' BEV perspective feature V n Adjustments and adaptations are made, and the data is compressed and encoded into semantic features T. nn' :

[0038] T nn' =f CAE (V n SNR nn' )

[0039] Among them, f CAE (·,·) represents the channel coding module, SNR nn' Let n be the signal-to-noise ratio of the transmitting drone n and the receiving drone n'.

[0040] The channel coding module includes eight fully connected layers (FC), seven SNR feature generation modules (SM), and a transmission compression layer (TC). The transmission compression layer (TC) implements channel compression using one fully connected layer. The signal-to-noise ratio (SNR) is generated by the SNR feature generation modules (SM). nn' Mapped to vector sm, which is the vector sm generated by the j-th SNR feature generation module SM. j Represented as:

[0041] sm j ←(f Relu (f Linear (SNR nn' )))

[0042] sm j ←(f Relu (f Linear (sm j )))

[0043] sm j ←(f Sigmoid (f Linear (sm j )))

[0044] Among them, f Linear (·) represents a linear operation, f Relu (·) represents the ReLU activation function operation, f Sigmoid (·) represents the Sigmoid activation function operation.

[0045] Furthermore, the channel decoding module for the received semantic feature R nn' The decoding process is represented as follows:

[0046] V' n =f CAD (R nn' SNR nn' )

[0047] Among them, f CAD (·,·) represents the channel decoding module, SNR nn' Let n be the signal-to-noise ratio of the transmitting drone n and the receiving drone n'.

[0048] The channel decoding module includes a transmission decompression layer (TDC), eight fully connected layers (FC), and seven SNR feature generation modules (SM). The transmission decompression layer (TDC) implements channel compression through one fully connected layer.

[0049] Furthermore, the multi-UAV feature fusion module will integrate the BEV features V generated by the receiving UAV n'. n' and the decoding feature set {V'1,...,V' n ,...,V' N ,n≠n'} are fused and upsampled into perceptible features P n' :

[0050] P n' =f US (f Fusion ({V'1,...,V' n ,...,V' N ,n≠n'},V n' ))

[0051] Among them, f Fusion (·) indicates a multi-UAV feature fusion module; f US (·) indicates an upsampling network;

[0052] The multi-UAV feature fusion module uses a ConvGRU network for L rounds of fusion, first averaging m values ​​of all received semantic features. n' The characteristics of the l-th round of fusion are: The specific calculation process is as follows:

[0053]

[0054] in, The initial features of the receiving drone n' in round 0 are represented by ·|·; ·|· represents the Concat operation at the channel layer; f ConvGRU(·) indicates that the same ConvGRU network is used for cyclic computation in each round; U represents the fusion feature of the receiving UAV n' in the (L+1)th round; n' This represents the final output feature of the multi-UAV feature fusion module for the receiving UAV n'.

[0055] Furthermore, the upsampling network is constructed using a three-layer network, each layer including an interpolation upsampling module and a Conv3×3 NormReLU network with a stride of 1, as shown below:

[0056] P n' ←f Conv3×3NormRelu (f Interpolate (U n' ))

[0057] P n' ←f Conv3×3NormRelu (f Interpolate (P n' ))

[0058] P n' ←f Conv3×3NormRelu (f Interpolate (P n' ))

[0059] Among them, f Interpolate (·) denotes the interpolation upsampling function, f Conv3×3NormRelu (·) represents 3×3 convolution operation, BatchNorm2d operation, and ReLU operation.

[0060] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0061] This invention provides a channel-adaptive multi-UAV cooperative 3D perception method, and studies a multi-UAV cooperative 3D perception system suitable for complex wireless channel environments. From the perspective of cooperative transmission, this invention considers multi-UAV cooperative 3D perception in complex wireless channel environments, firstly establishing a channel-adaptive multi-UAV cooperative 3D perception framework. Each UAV needs to encode and fuse the images acquired by its own sensors into semantic feature vectors, and share these semantic feature vectors with other UAVs through the complex wireless channel environment. The receiving UAV can decode the semantic feature vectors to enhance its own 3D perception capabilities. From the perspective of channel transmission, the original feature vectors are easily distorted by channel noise during transmission. To reduce the interference of channel noise on cooperative 3D perception, this invention first designs a BEV semantic encoding module and a BEV semantic decoding module based on a channel-adaptive mechanism. By dynamically tracking the channel link conditions, the transmitting UAV adjusts and adapts the BEV feature fusion process through the BEV semantic encoding module, and further compresses and encodes it into semantic features; the receiving UAV receives the semantic features from multiple other UAVs through the BEV semantic decoding module, and decodes and fuses them into perceptible feature vectors. This invention improves the performance of multi-UAV cooperative 3D perception in complex wireless channel environments and has broad application prospects. Attached Figure Description

[0062] Figure 1 This is a multi-UAV collaborative 3D perception framework based on channel adaptation.

[0063] Figure 2 For the image encoder network at the transmitting end.

[0064] Figure 3 This is a 3D target detection module for the receiving end.

[0065] Figure 4 This is the BEV semantic encoding module at the sending end.

[0066] Figure 5 This is the channel coding module for the transmitting end.

[0067] Figure 6 This is the SNR feature generation module.

[0068] Figure 7 This is the BEV semantic decoding module for the receiving end.

[0069] Figure 8 This is the channel decoding module for the receiving end. Detailed Implementation

[0070] The present invention will be further described in detail below with reference to the accompanying drawings.

[0071] This invention provides a channel-adaptive multi-UAV cooperative 3D perception method. First, a channel-adaptive multi-UAV cooperative 3D perception framework is established. Then, based on this framework, a BEV semantic encoding module and a BEV semantic decoding module are designed. The transmitting UAV needs to fuse and encode the image data acquired by its own sensors and channel information into a semantic feature vector, and share this semantic feature vector with other UAVs through a complex wireless channel environment. The receiving UAV can decode the semantic feature vector to enhance its own 3D perception capabilities.

[0072] like Figure 1 As shown, the present invention provides a multi-UAV cooperative 3D perception method based on channel adaptation, the specific implementation process of which is as follows:

[0073] S1: Acquisition of multi-view images;

[0074] This invention considers a multi-drone collaborative 3D perception scenario. It is assumed that there are N drones in the scenario, N≥2, and each drone is equipped with K cameras to acquire image data from K perspectives, K≥1.

[0075] S2: Multi-scale feature extraction and fusion to generate multi-view features;

[0076] The transmitting UAV first uses a WinTransformer network and a Generalized LSSFPN network as image encoders for multi-scale feature extraction and fusion. The image encoder network is as follows: Figure 2 As shown, after the SwinTransformer network extracts the features of stages 1, 2, and 3, the Generalized LSSFPN network fuses these features at multiple scales.

[0077] Specifically, assume that the image data acquired by the transmitting drone n is Given that H0 is the image height and W0 is the image width, the process of multi-scale feature extraction and fusion using an image encoder can be represented as follows:

[0078] F n =f GLSSFPN (f SwinT (X n ))

[0079] Among them, f SwinT (·) indicates that the SwinTransformer network is used for multi-scale feature extraction, f GLSSFPN (·) indicates that a GeneralizedLSSFPN network is used for multi-scale feature fusion. This represents the extracted feature data from K perspectives, where C1 is the number of feature data channels, H1 is the feature data height, and W1 is the feature data width.

[0080] Specifically, for the three-stage features {S1, S2, S3} given by the SwinTransformer network, the GeneralizedLSSFPN network is used for feature fusion. The specific implementation process is as follows:

[0081] S3←f Interpolate (S3)

[0082] S2←f Conv3×3Norm (f Conv1×1Norm (S2|S3))

[0083] S2←f Interpolate (S2)

[0084] S1←f Conv3×3Norm (f Conv1×1Norm (S1|S2))

[0085] F n =S1

[0086] Where ·|· represents the Concat operation at the channel layer, f Interpolate (·) denotes the interpolation upsampling function, f Conv1×1Norm (·) represents a 1×1 convolution operation and a BatchNorm2d operation, f Conv3×3Norm (·) indicates the operation of 3×3 convolution and BatchNorm2d.

[0087] S3: Semantic feature vector generation;

[0088] Subsequently, the transmitting UAV uses the BEV semantic coding module to compress and encode the feature data from K perspectives into transmittable semantic feature vectors based on the channel state, and then transmits them through a complex wireless channel environment. The transmitting UAV n needs to share the semantic feature vectors with all other N-1 UAVs; therefore, the total number of semantic feature vectors generated based on different channel signal-to-noise ratios is {T}. n1 ,...,T nn' ,...,T nN ,n≠n'}.

[0089] S4: Semantic feature vector transmission;

[0090] The semantic features generated by the transmitting UAV need to be transmitted through a noisy wireless channel. Considering a scenario where UAVs are conducting cooperative sensing at high altitudes, the wireless channel adopts an additive white Gaussian noise (AWGN) channel model. For the transmission process from the transmitting UAV n to the receiving UAV n', assuming the channel fading coefficient is h and the noise figure is ω, the semantic features R received by the receiving UAV n' are... nn' It can be represented as:

[0091] R nn' =h*T nn' +ω

[0092] In the AWGN channel model, the fading coefficient h is always 1, and the noise figure ω follows a circularly symmetric Gaussian distribution CN(0,σ). 2 ), σ 2 This represents the noise power of the channel.

[0093] S5: Semantic Feature Vector Decoding and Fusion with 3D Object Detection;

[0094] The receiving drone receives semantic feature vectors shared by multiple drones and uses the BEV semantic decoding module to adaptively decode the received semantic feature vectors, fusing them into a perceptible feature vector. Specifically, the receiving drone n' receives semantic features shared by other drones. These features are affected by channel noise interference and can be represented as {R}. 1n' ,...,R nn' ,...,R Nn' The BEV semantic decoding module decodes and fuses the received semantic features into perceptible BEV features P based on the channel state of each link. n' .

[0095] Subsequently, the receiving UAV n' uses the 3D target detection module to analyze the BEV features P n' Perform target detection, such as Figure 3 As shown, the 3D target detection module mainly includes a DownSample network, a ResNet network, an LSSFPN network, and a Head network. Therefore, the decoding and perception process of the receiving UAV n' can be represented as:

[0096] O n' =f Head (f LSSFPN (f ResNet (f DS (P n' ))))

[0097] Among them, f DS (·) indicates that downsampling is performed using the DownSample network, f Resnet (·) indicates that a ResNet network is used for multi-scale feature transformation, f LSSFPN (·) indicates the use of the LSSFPN network for multi-scale feature fusion, f Head (·) indicates that the head network is used to make final adjustments to the channel output based on the 3D object detection task. n' This indicates the output of the task.

[0098] like Figure 3As shown, the DownSample network consists of three convolutional layers, each including convolution operations, BatchNorm2d operations, and ReLU operations. The Head network is a single-layer convolutional network used to adjust the final channel output to adapt to the object detection task. The LSSFPN network fuses the two-stage features {S1, S3} given by the ResNet network at multiple scales. Its specific implementation process is as follows:

[0099] S3←f Interpolate (S3)

[0100] S1←f Conv3×3NormRelu (f Conv1×1NormRelu (S1|S3))

[0101] S1←f Conv3×3NormRelu (f UpSample (S1))

[0102] O n' =S1

[0103] Where ·|· represents the Concat operation at the channel layer, f Interpolate (·) denotes the interpolation upsampling function, f Conv1×1NormRelu (·) represents a 1×1 convolution operation, a BatchNorm2d operation, and a ReLU operation, f Conv3×3NormRelu (·) represents 3×3 convolution operation, BatchNorm2d operation, and ReLU operation.

[0104] S6: Training phase;

[0105] This invention provides a channel-adaptive multi-UAV cooperative 3D perception method, employing an end-to-end training approach. The training objective is to enable the model to detect 3D targets as accurately as possible in complex wireless channel environments. Therefore, the CenterNet loss function is used during training, and its mathematical expression is as follows:

[0106]

[0107] Among them, O n Y is the predicted value; n The actual value; For the heatmap loss of CenterNet, Gaussian Focal Loss is used; λ is the bounding box loss of CenterNet, using L1Loss; λ is the weight hyperparameter of the bounding box loss.

[0108] In order to reduce the interference of channel noise on collaborative 3D perception, this invention designs a BEV semantic coding module and a BEV semantic decoding module based on a channel adaptive mechanism.

[0109] like Figure 4 As shown, the BEV semantic coding module mainly includes a multi-view BEV transformation module and a channel coding module. These modules are used by the transmitting UAV to adjust and adapt its multi-view features to BEV features based on the channel conditions, and then compress and encode them into semantic features. For the transmitting UAV n, the input to the BEV semantic coding module is feature data from K views. The multi-view BEV transformation module converts the feature data F from K views. n Transformation fusion into BEV view features C2 represents the number of BEV view feature channels, H2 represents the BEV view feature height, and W2 represents the BEV view feature width, which can be expressed as:

[0110] V n =f DS (f BEVT (F n ,I,E))

[0111] Among them, f BEVT (·,·,·) represents the multi-view BEV transformation module; This is the intrinsic parameter matrix of the camera; f is the extrinsic parameter matrix of the camera; DS (·) indicates a downsampling network used to filter compressed features.

[0112] The multi-view BEV transformation module transforms and fuses the feature data from the K-viewpoint into a unified BEV viewpoint to obtain BEV viewpoint features. The specific implementation process is as follows:

[0113] The multi-view BEV transformation module maps pixel features to BEV features by calculating the probability distribution of each pixel at different depths. First, a 3D mesh M in the pixel coordinate system is constructed. pic The camera's intrinsic parameter matrix is ​​used to map it onto a 3D mesh M in the camera coordinate system. pic Then, it is mapped to a 3D mesh M in the world coordinate system using an extrinsic parameter matrix. world For multi-view image features F n First, calculate the probability distribution D at different depths. p and semantic features D f Next, the semantic feature distribution D at different depths is calculated; finally, the semantic feature distributions within each grid are summed, and the final BEV view feature V is obtained through a downsampling network. n The specific calculation process is as follows:

[0114]

[0115] M world =R cam2world M cam +T cam2world

[0116] D p =f Softmax (f Conv1×1 (F n ))

[0117] D f =f Conv1×1 (F n )

[0118] D = D p ⊙D f

[0119]

[0120] V n =f DS (B n )

[0121] Among them, E -1 The inverse of the extrinsic parameter matrix can be decomposed into a rotation matrix R. cam2world and displacement matrix T cam2world M pic Representing a 3D mesh in pixel coordinates, it can be decomposed into vectors with three dimensions: x, y, and d; I -1 M represents the inverse matrix of the intrinsic parameter matrix; cam Represents a 3D mesh in the camera coordinate system; f Softmax (·) indicates performing a Softmax operation on the depth dimension; f Conv1×1 (·) represents a 1×1 convolution operation; M world (b,h,w) represents the grid located at (h,w) in batch b; B n (b,h,w) represents the BEV feature located in the (h,w) grid under batch b; f k f represents the k-th vector in the semantic feature distribution D; DS (·) represents a downsampling network, consisting of three Conv3×3 NormReLU networks with a stride of 2, which can be represented as:

[0122] V n =f Conv3×3NormRelu (f Conv3×3NormRelu (f Conv3×3NormRelu (B n )))

[0123] Among them, fConv3×3NormRelu (·) represents 3×3 convolution operation, BatchNorm2d operation, and ReLU operation.

[0124] For the transmission process between the transmitting drone n and the receiving drone n', the channel coding module determines the signal-to-noise ratio (SNR) of the channel between the transmitting drone n and the receiving drone n'. nn' BEV perspective feature V n Adjustments and adaptations are made, and the data is compressed and encoded into semantic features T. nn' , can be represented as:

[0125] T nn' =f CAE (V n SNR nn' )

[0126] Among them, f CAE (·,·) represents the channel coding module, which generates semantic features based on the channel signal-to-noise ratio (SNR). nn' Let n be the signal-to-noise ratio of the transmitting drone n and the receiving drone n'.

[0127] like Figure 5 As shown, the channel coding module mainly consists of 8 fully connected layers (FC), 7 SNR feature generation modules (SM), and a transmission compression layer (TC). The transmission compression layer (TC) implements channel compression using one fully connected layer. The SNR feature generation modules (SM) are as follows: Figure 6 As shown, the signal-to-noise ratio (SNR) is generated through the SNR feature generation module SM. nn' The mapping is to the signal-to-noise ratio modulation vector sm, which is the vector sm generated by the j-th SNR feature generation module SM. j It can be represented as:

[0128] sm j ←(f Relu (f Linear (SNR nn' )))

[0129] sm j ←(f Relu (f Linear (sm j )))

[0130] sm j ←(f Sigmoid (f Linear (sm j )))

[0131] Among them, f Linear (·) represents a linear operation, f Relu (·) represents the ReLU activation function operation, f Sigmoid(·) represents the Sigmoid activation function operation.

[0132] The transmitting drone n needs to share semantic features with all other N-1 drones. Therefore, the total semantic features generated based on different channel signal-to-noise ratios are {T}. n1 ,...,T nn' ,...,T nN ,n≠n'}.

[0133] like Figure 7 As shown, the BEV semantic decoding module mainly includes a channel decoding module and a multi-UAV feature fusion module. The receiving UAV adaptively decodes the received semantic features based on the channel state and fuses the decoded features from multiple UAVs into a perceptible feature vector. The receiving UAV n' receives semantic features shared by other UAVs. These features are affected by channel noise interference and can be represented as {R}. 1n' ,...,R nn' ,...,R Nn' For the receiving UAV n', the input to the BEV semantic decoding module is the received semantic features {R}. 1n' ,...,R nn' ,...,R Nn' The channel decoding module decodes the received semantic features according to the signal-to-noise ratio of each channel, and all received semantic features can be decoded into a feature set {V'1,...,V'}. n ,...,V' N For the received semantic features R, ..., n≠n'}, nn' The decoding process can be represented as:

[0134] V' n =f CAD (R nn' SNR nn' )

[0135] Among them, f CAD (·,·) represents the channel decoding module, used to decode the received semantic features based on the channel signal-to-noise ratio (SNR). nn' Let n be the signal-to-noise ratio of the transmitting drone n and the receiving drone n'.

[0136] like Figure 8 As shown, the channel decoding module mainly consists of a transmission decompression layer (TDC) and eight fully connected layers.

[0137] It consists of FC and 7 SNR feature generation modules SM, where the transmission decompression layer TDC is implemented by a fully connected layer to achieve channel compression.

[0138] Subsequently, the multi-UAV feature fusion module will combine the BEV features V generated by the receiving UAV n' itself. n' and the decoding feature set {V'1,...,V' n ,...,V' N ,n≠n'} are fused and upsampled into perceptible features P n' , can be represented as:

[0139] P n' =f US (f Fusion ({V'1,...,V' n ,...,V' N ,n≠n'},V n' ))

[0140] Among them, f Fusion (·) indicates a multi-UAV feature fusion module, used by the receiving UAV to fuse features from multiple UAVs; f US (·) indicates an upsampling network.

[0141] The multi-UAV feature fusion module uses a ConvGRU network for L rounds of fusion, first averaging m values ​​of all received semantic features. n' The characteristics of the l-th round of fusion are: The specific calculation process is as follows:

[0142]

[0143] in, The initial features of the receiving drone n' in round 0 are represented by ·|·; ·|· represents the Concat operation at the channel layer; f ConvGRU (·) indicates that the same ConvGRU network is used for cyclic computation in each round;

[0144] U represents the fusion feature of the receiving UAV n' in the (L+1)th round; n' This represents the final output feature of the multi-UAV feature fusion module for the receiving UAV n'.

[0145] upsampling network f US (·) Using a three-layer network, each layer includes an interpolation upsampling module and a Conv3×3 NormReLU network with a stride of 1, it can be represented as:

[0146] P n' ←f Conv3×3NormRelu (f Interpolate (U n' ))

[0147] Pn' ←f Conv3×3NormRelu (f Interpolate (P n' ))

[0148] P n' ←f Conv3×3NormRelu (f Interpolate (P n' ))

[0149] Among them, f Interpolate (·) denotes the interpolation upsampling function, f Conv3×3NormRelu (·) represents 3×3 convolution operation, BatchNorm2d operation, and ReLU operation.

[0150] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-UAV cooperative 3D perception method based on channel adaptation, characterized in that, Includes the following steps: (1) Establish a multi-UAV collaborative 3D perception framework based on channel adaptation. Consider multiple UAVs to perform collaborative 3D perception in a complex wireless channel environment. The transmitting UAV needs to fuse and encode the image data and channel information acquired by its own sensors into a semantic feature vector, and share the semantic feature vector with other UAVs through the complex wireless channel environment. The receiving UAV enhances its own 3D perception capability by decoding the semantic feature vector. (2) Design a BEV semantic coding module and a BEV semantic decoding module; the BEV semantic coding module includes a multi-view BEV transformation module and a channel coding module, which are used to adjust and adapt the process of fusing its own multi-view features into BEV features according to the channel state of the transmitting UAV, and compress and encode it into a semantic feature vector; the BEV semantic decoding module includes a channel decoding module and a multi-UAV feature fusion module, which are used to adaptively decode the received semantic features according to the channel state of the receiving UAV, and fuse the decoded multiple UAV features into a perceptible feature vector.

2. The multi-UAV cooperative 3D perception method based on channel adaptation according to claim 1, characterized in that, The transmitting UAV uses a WinTransformer network and a GeneralizedLSSFPN network as image encoders to perform multi-scale feature extraction and fusion, obtaining feature data and feature fusion data from K perspectives; the mathematical expression for the feature data from the K perspectives is: F n =f GLSSFPN (f SwinT (X n )) in, This represents the image data acquired by the sending drone n, where H0 is the image height, W0 is the image width, and f is the image height. SwinT (·) indicates that the SwinTransformer network is used for multi-scale feature extraction, f GLSSFPN (·) indicates that a GeneralizedLSSFPN network is used for multi-scale feature fusion. This represents the extracted feature data from K perspectives, where C1 is the number of feature data channels, H1 is the feature data height, and W1 is the feature data width. The calculation process for the feature fusion data is represented as follows: S3←f Interpolate (S3) S2←f Conv3×3Norm (f Conv1×1Norm (S2|S3)) S2←f Interpolate (S2) S1←f Conv3×3Norm (f Conv1×1Norm (S1|S2)) F n =S1 Where {S1,S2,S3} are the three-stage features given by the SwinTransformer network, ·|· represents the Concat operation in the channel layer, f Interpolate (·) denotes the interpolation upsampling function, f Conv1×1Norm (·) represents a 1×1 convolution operation and a BatchNorm2d operation, f Conv3×3Norm (·) indicates the operation of 3×3 convolution and BatchNorm2d.

3. The multi-UAV cooperative 3D perception method based on channel adaptation according to claim 1, characterized in that, The semantic features R received by the receiving drone n' nn' Represented as: R nn' =h*T nn' +ω Where h is the channel fading coefficient; ω is the noise coefficient, which follows a circularly symmetric Gaussian distribution CN(0,σ). 2 ), σ 2 T represents the noise power of the channel. nn' The semantic features generated by sending drone n to drone n'.

4. The multi-UAV cooperative 3D perception method based on channel adaptation according to claim 1, characterized in that, The BEV semantic decoding module decodes and fuses the received semantic features into perceptible BEV features P based on the channel state of each link. n' The receiving UAV uses a 3D target detection module to detect BEV features. n' Target detection is performed; the 3D target detection module includes a DownSample network, a ResNet network, an LSSFPN network, and a Head network. The decoding and perception process of the receiving UAV n' is represented as follows: O n' =f Head (f LSSFPN (f ResNet (f DS (P n' )))) Among them, f DS (·) indicates that downsampling is performed using the DownSample network, f Resnet (·) indicates that a ResNet network is used for multi-scale feature transformation, f LSSFPN (·) indicates the use of the LSSFPN network for multi-scale feature fusion, f Head (·) indicates that the head network is used to make final adjustments to the channel output based on the 3D object detection task. n' This indicates the output of the task.

5. A multi-UAV cooperative 3D perception method based on channel adaptation according to claim 4, characterized in that, The DownSample network consists of three convolutional layers, each layer including convolution, BatchNorm2d, and ReLU operations. The Head network is a single-layer convolutional network used to adjust the final channel output to adapt to the object detection task. The LSSFPN network performs multi-scale fusion of the two-stage features {S1, S3} given by the ResNet network, and its specific calculation process is as follows: S3←f Interpolate (S3) S1←f Conv3×3NormRelu (f Conv1×1NormRelu (S1|S3)) S1←f Conv3×3NormRelu (f Interpolate (S1)) THE n' =S1 Where ·|· represents the Concat operation at the channel layer, f Interpolate (·) denotes the interpolation upsampling function, f Conv1×1NormRelu (·) represents a 1×1 convolution operation, a BatchNorm2d operation, and a ReLU operation, f Conv3×3NormRelu (·) represents 3×3 convolution operation, BatchNorm2d operation, and ReLU operation.

6. The multi-UAV cooperative 3D perception method based on channel adaptation according to claim 1, characterized in that, The multi-view BEV transformation module transforms and fuses the feature data from the K-viewpoint into a unified BEV viewpoint to obtain BEV viewpoint features. The specific implementation process is as follows: Construct a 3D mesh M in pixel coordinate system pic And it is mapped onto a 3D mesh M in the world coordinate system using the camera's intrinsic and extrinsic parameter matrices. world For multi-view image features F n First, calculate the probability distribution D at different depths. p and semantic features D f Next, the semantic feature distribution D at different depths is calculated; finally, the semantic feature distributions within each grid are summed, and the final BEV view feature V is obtained through a downsampling network. n : V n =f DS (f BEVT (F n ,I,E)) Among them, f BEVT (·,·,·) represents the multi-view BEV transformation module; This is the intrinsic parameter matrix of the camera; f is the extrinsic parameter matrix of the camera; DS (·) indicates a downsampling network used to filter compressed features.

7. A multi-UAV cooperative 3D perception method based on channel adaptation according to claim 6, characterized in that, The channel coding module uses the channel signal-to-noise ratio (SNR) between the transmitting UAV n and the receiving UAV n'. nn' BEV perspective feature V n Adjustments and adaptations are made, and the data is compressed and encoded into semantic features T. nn' : T nn' =f CAE (V n ,SNR nn' ) Among them, f CAE (·,·) represents the channel coding module, SNR nn' Let n be the signal-to-noise ratio of the transmitting drone n and the receiving drone n'. The channel coding module includes eight fully connected layers (FC), seven SNR feature generation modules (SM), and a transmission compression layer (TC). The transmission compression layer (TC) implements channel compression using one fully connected layer. The signal-to-noise ratio (SNR) is generated by the SNR feature generation modules (SM). nn' Mapped to vector sm, which is the vector sm generated by the j-th SNR feature generation module SM. j Represented as: sm j ←(f Relu (f Linear (SNR nn' ))) sm j ←(f Relu (f Linear (sm j ))) sm j ←(f Sigmoid (f Linear (sm j ))) Among them, f Linear (·) represents a linear operation, f Relu (·) represents the ReLU activation function operation, f Sigmoid (·) represents the Sigmoid activation function operation.

8. A multi-UAV cooperative 3D perception method based on channel adaptation according to claim 1, characterized in that, The channel decoding module receives semantic features R. nn' The decoding process is represented as follows: In' n =f CAD (R nn' ,SNR nn' ) Among them, f CAD (·,·) represents the channel decoding module, SNR nn' Let n be the signal-to-noise ratio of the transmitting drone n and the receiving drone n'. The channel decoding module includes a transmission decompression layer (TDC), eight fully connected layers (FC), and seven SNR feature generation modules (SM). The transmission decompression layer (TDC) implements channel compression through one fully connected layer.

9. A multi-UAV cooperative 3D perception method based on channel adaptation according to claim 1, characterized in that, The multi-UAV feature fusion module will integrate the BEV features V generated by the receiving UAV n'. n' and the decoding feature set {V'1,...,V' n ,...,V' N ,n≠n'} are fused and upsampled into perceptible features P n' : P n' =f US (f Fusion ({V'1,...,V' n ,...,V' N ,n≠n'},V n' )) Among them, f Fusion (·) indicates a multi-UAV feature fusion module; f US (·) indicates an upsampling network; The multi-UAV feature fusion module uses a ConvGRU network for L rounds of fusion, first averaging m values ​​of all received semantic features. n' The characteristics of the l-th round of fusion are: The specific calculation process is as follows: in, The initial features of the receiving drone n' in round 0 are represented by ·|·; ·|· represents the Concat operation at the channel layer; f ConvGRU (·) indicates that the same ConvGRU network is used for cyclic computation in each round; U represents the fusion feature of the receiving UAV n' in the (L+1)th round; n' This represents the final output feature of the multi-UAV feature fusion module for the receiving UAV n'.

10. A multi-UAV cooperative 3D perception method based on channel adaptation according to claim 9, characterized in that, The upsampling network is constructed using a three-layer network. Each layer includes an interpolation upsampling module and a Conv3×3 NormReLU network with a stride of 1, as shown below: P n' ←f Conv3×3NormRelu (f Interpolate (U n' )) P n' ←f Conv3×3NormRelu (f Interpolate (P n' )) P n' ←f Conv3×3NormRelu (f Interpolate (P n' )) Among them, f Interpolate (·) denotes the interpolation upsampling function, f Conv3×3NormRelu (·) represents 3×3 convolution operation, BatchNorm2d operation, and ReLU operation.

Images