An air-ground cooperative target detection method based on a double-branch network architecture and an electronic device

Abstract

The application provides a kind of air-ground cooperative target detection method and electronic equipment based on double-branch network architecture, method specifically includes the following steps: using the two encoders of double-branch network respectively to carry out feature extraction to air perspective and ground perspective, and using the cross attention module in the encoder to interact the features of different perspectives;Cross-perspective feature fusion module in the encoder is used to fuse the features of two perspectives;The fused features are mapped to the keys of the multi-head self-attention layer in the two decoders of double-branch network, and finally the target class and the bounding box are output through the feedforward neural network of the decoder;Design a joint loss function to train the double-branch network.The technical scheme of the application overcomes the problem that the detection method in the prior art cannot significantly improve the robustness in complex environment and recognize multi-perspective targets.

Description

Technical Field

[0001] This invention relates to the field of target detection, and specifically to an air-ground cooperative target detection method and electronic device based on a dual-branch network architecture. Background Technology

[0002] In recent years, with the rapid development of smart integrated utility tunnel construction, the inspection tasks of underground utility tunnels and other hazardous locations have faced numerous challenges, such as insufficient lighting, poor signal strength, and difficulties in ensuring worker safety. Therefore, using unmanned platform equipment for surveying has become an important direction for future development. However, single-view unmanned equipment has limitations in daily inspection processes, such as limited field of view and poor robustness, making it difficult to meet the target detection needs in complex environments. Therefore, air-ground dual-view joint inspection technology has emerged.

[0003] Significant progress has been made in the field of multi-drone cooperative detection. For example, existing multi-UAV cooperative detection methods based on genetic algorithms can adapt to dynamically changing detection environments and effectively improve the target recognition rate and detection accuracy of multi-drone systems. Existing task allocation methods for multi-drone cooperative operations successfully avoid getting trapped in local optima while ensuring convergence speed. These methods achieve multi-drone cooperation from the control domain, but routine inspections in scenarios such as underground utility tunnels mainly rely on target recognition technology. Therefore, simply studying cooperation in the control domain is insufficient to solve practical problems.

[0004] Therefore, there is a need for an air-ground cooperative target detection method and electronic device based on a dual-branch network architecture that can significantly improve robustness and multi-view target recognition capabilities in complex environments. Summary of the Invention

[0005] The main objective of this invention is to provide an air-ground cooperative target detection method and electronic device based on a dual-branch network architecture, in order to solve the problem that existing detection methods cannot significantly improve robustness and perform multi-view target recognition in complex environments.

[0006] To achieve the above objectives, this invention provides an air-ground cooperative target detection method based on a dual-branch network architecture, specifically including the following steps: S1 utilizes two encoders in a dual-branch network to extract features from the aerial and ground perspectives respectively, and uses the cross-attention module in the encoder to interact with the features from different perspectives. S2, the cross-view feature fusion module in the encoder is used to fuse the features of the two views; S3 maps the fused features to the keys of the multi-head self-attention layers in the two decoders of the dual-branch network, and finally outputs the target category and bounding box through the feedforward neural network of the decoder. S4. Design a joint loss function to train the dual-branch network.

[0007] Furthermore, the two encoders in step S1 have the same structure. One encoder includes: a first multi-head self-attention layer, a first residual connection and layer normalization Add&Norm, a cross-attention module, a second residual connection and layer normalization, a first feedforward neural network FFN, and a third residual connection and layer normalization, connected in sequence.

[0008] Furthermore, the two encoders include: a first encoder and a second encoder; the input to the first encoder is aerial view features and position information, and the input to the second encoder is ground view features and position information; the first residual connection in the first encoder is connected to the layer normalization and the cross-attention module in the second encoder, and the first residual connection in the second encoder is connected to the layer normalization and the cross-attention module in the first encoder. The cross-attention formula of the first encoder is: ; in, Cross-attention features from an aerial perspective For normalized exponential functions, This is the query vector from an aerial perspective. The key vector is the ground-view vector. for transpose, This is the value vector from the ground perspective. The dimension of the key.

[0009] Furthermore, step S2 specifically includes the following steps: S2.1, the output features of the first encoder and the second encoder are respectively ,in, For sequence length, For batch size, For feature dimensions, on the feature dimensions spliced ​​together ; S2.2, the concatenated features are used to calculate the two-dimensional weight vector at each location using a multilayer perceptron (MLP). And through the softmax function Normalization to dynamic weights , ; ; Among them, subscript Indicates the sequence position. For sequence position The output characteristics of the first encoder, For sequence position The output characteristics of the second encoder, For sequence position Two-dimensional weight vector, ; S2.3, perform weighted fusion of the two features to obtain the fused features. and : ; ; in, Features resulting from the fusion of aerial perspectives. This is the fused feature from the ground perspective.

[0010] Furthermore, the two decoders have the same structure. The inputs of the first decoder and the second decoder are learnable target query parameters. The first decoder includes: a second multi-head self-attention layer, a fourth residual connection and layer normalization layer, a third multi-head self-attention layer, a fifth residual connection and layer normalization layer, a second feedforward neural network and a sixth residual connection and layer normalization layer, which are connected in sequence. The sixth residual connection and layer normalization layer connects the third feedforward neural network and the fourth feedforward neural network. The third feedforward neural network outputs the target category, and the fourth feedforward neural network outputs the bounding box.

[0011] Furthermore, This is mapped to the value vector in the third multi-head self-attention layer of the first decoder. This is mapped to the value vector in the third multi-head self-attention layer of the second decoder. The features output by the fourth residual connection and layer normalization of the first decoder are concatenated with the learnable target query parameters and mapped to the key vector in the third multi-head self-attention layer of the first decoder. The features output by the fourth residual connection and layer normalization of the second decoder are concatenated with the learnable target query parameters and mapped to the key vector in the third multi-head self-attention layer of the second decoder.

[0012] Furthermore, step S4 specifically involves: spatial loss. and ground loss The data is sent back to its respective decoder, and the loss after fusion is recorded. The data is fed back to the two encoders for joint optimization, specifically as follows: ; ; in, It is an open space branch. For space loss or ground loss , These are the weighting coefficients for the corresponding tasks. To lose weight, For various scalar losses, For weights.

[0013] Furthermore, the aerial viewpoint features are mapped to the value vector of the first multi-head self-attention layer in the first encoder, and the aerial viewpoint features and position information are concatenated and mapped to the key vector and query vector of the first multi-head self-attention layer in the first encoder; the ground viewpoint features are mapped to the value vector of the first multi-head self-attention layer in the second encoder, and the ground viewpoint features and position information are concatenated and mapped to the key vector and query vector of the first multi-head self-attention layer in the second encoder.

[0014] Furthermore, the learnable target query parameters are mapped to the value vector, key vector, and query vector of the second multi-head self-attention layer in the first decoder and the second decoder.

[0015] The present invention also provides an electronic device, which includes a processor and a memory communicatively connected to the processor; the memory stores instructions that are executed by the processor, and the instructions are executed by the processor to cause the processor to perform, for example, an air-ground cooperative target detection method based on a dual-branch network architecture.

[0016] The present invention has the following beneficial effects: This invention proposes a novel air-ground cooperative detection algorithm with a dual-branch network architecture. The key feature is the use of a Transformer architecture to construct a dual-branch network. Through a cross-attention mechanism and dynamic fusion of deep interactive multi-view feature information, the algorithm significantly improves the model's robustness in complex environments and its ability to identify targets from multiple perspectives. Furthermore, this invention designs a joint loss backpropagation mechanism, ensuring that the fused loss can be jointly optimized. This air-ground cooperative detection algorithm, designed for dangerous scenarios such as complex underground spaces, effectively compensates for the limited field of view of a single target and reduces the false negative rate. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A diagram of the dual-branch network architecture of the present invention is shown.

[0018] Figure 2 The figure shows the result of target detection using an air-ground cooperative target detection method based on a dual-branch network architecture provided by the present invention. Detailed Implementation

[0019] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example 1 A method for air-ground cooperative target detection based on a dual-branch network architecture includes the following steps: S1 utilizes two encoders in a dual-branch network to extract features from the aerial and ground perspectives respectively, and uses the cross-attention module in the encoder to interact with the features from different perspectives. S2, the cross-view feature fusion module in the encoder is used to fuse the features of the two views; S3 maps the fused features to the keys of the multi-head self-attention layers in the two decoders of the dual-branch network, and finally outputs the target category and bounding box through the feedforward neural network of the decoder. S4. Design a joint loss function to train the dual-branch network.

[0021] To effectively perform collaborative target detection from both air and ground perspectives, this invention proposes a novel dual-branch network architecture. This architecture consists of two independent Encoder-Decoder structures, processing input data from the air and ground perspectives respectively. Each Encoder-Decoder structure is responsible for acquiring target features from different perspectives and generating corresponding target detection results. Through this structure, the network can learn features from both perspectives separately, thereby improving detection accuracy. Specifically... Figure 1 As shown.

[0022] Input images from both aerial and ground perspectives are first processed for feature extraction using two independent encoders. In each encoder, a self-attention mechanism is employed to globally model the data from each perspective. The processed features are then passed through two separate decoders for target prediction. Each decoder independently generates a prediction, including the target category and bounding box. This structure enables target recognition along independent paths for both aerial and ground perspectives.

[0023] Specifically, the two encoders in step S1 have the same structure. One encoder includes: a first multi-head self-attention layer, a first residual connection and layer normalization Add&Norm, a cross-attention module, a second residual connection and layer normalization, a first feedforward neural network FFN, and a third residual connection and layer normalization, connected in sequence.

[0024] In the encoder part of the dual-branch network, to effectively perform the interaction and fusion of aerial and ground-based dual-view features, we optimized the traditional encoder structure. Specifically, by introducing a cross-attention mechanism, we enhanced information transfer and feature interaction between the aerial and ground-based perspectives. Figure 2 As shown, each encoder contains two main multi-head self-attention modules and a cross-attention module. First, the input features from the aerial and ground perspectives are processed by self-attention layers to learn features, capturing the relationships and information within each perspective. However, a single self-attention mechanism cannot capture the potential correlations between the two perspectives, so a cross-attention module is introduced to allow the features of each perspective to interact with the features of the other perspective.

[0025] Specifically, the two encoders include a first encoder and a second encoder; the inputs of the first encoder are aerial view features and position information, and the inputs of the second encoder are ground view features and position information; the first residual connection in the first encoder is connected to the layer normalization and the cross-attention module in the second encoder, and the first residual connection in the second encoder is connected to the layer normalization and the cross-attention module in the first encoder. The cross-attention formula of the first encoder (taking the aerial view branch as an example) is: ; in, Cross-attention features from an aerial perspective For normalized exponential functions, This is the query vector from an aerial perspective. The key vector is the ground-view vector. for transpose, This is the value vector from the ground perspective. The dimension of the key.

[0026] The cross-attention module uses aerial features as queries and ground-view features as keys and values, and performs cross-fusion through a self-attention mechanism. Similarly, ground-view features are also cross-attentioned with aerial features, meaning that ground-view features are used as queries and aerial features as keys and values, and the cross-fusion is performed through a self-attention mechanism.

[0027] This cross-attention mechanism allows features from two perspectives to be effectively integrated in the same network structure, significantly improving the model's robustness in complex environments and its ability to identify targets from multiple perspectives.

[0028] Specifically, to more flexibly fuse the features output from both the air and ground encoders, this invention designs a cross-view feature fusion module. Step S2 specifically includes the following steps: S2.1, the output features of the first encoder and the second encoder are respectively ,in, For sequence length, For batch size, For feature dimensions, on the feature dimensions spliced ​​together ; S2.2, the concatenated features are used to calculate the two-dimensional weight vector at each location using a multilayer perceptron (MLP). And through the softmax function Normalization to dynamic weights To accommodate the different importance of aerial and ground perspectives in various scenarios: ; ; Among them, subscript Indicates the sequence position. For sequence position The output characteristics of the first encoder, For sequence position The output characteristics of the second encoder, For sequence position Two-dimensional weight vector, ; S2.3, perform weighted fusion of the two features to obtain the fused features. and : ; ; in, Features resulting from the fusion of aerial perspectives. This is the fused feature from the ground perspective.

[0029] The fused features are immediately fed into their respective decoders, thereby preserving complementary information from both perspectives while allowing the network to adaptively adjust the fusion ratio of the two features, thus improving the accuracy and robustness of object detection in complex scenes.

[0030] Specifically, the two decoders have the same structure. The inputs of the first decoder and the second decoder are learnable target query parameters. The first decoder includes: a second multi-head self-attention layer, a fourth residual connection and layer normalization layer, a third multi-head self-attention layer, a fifth residual connection and layer normalization layer, a second feedforward neural network and a sixth residual connection and layer normalization layer, which are connected in sequence. The sixth residual connection and layer normalization layer connects the third feedforward neural network and the fourth feedforward neural network. The third feedforward neural network outputs the target category, and the fourth feedforward neural network outputs the bounding box.

[0031] Specifically, This is mapped to the value vector in the third multi-head self-attention layer of the first decoder. This is mapped to the value vector in the third multi-head self-attention layer of the second decoder. The features output by the fourth residual connection and layer normalization of the first decoder are concatenated with the learnable target query parameters (Objectqueries) and mapped to the query vector of the third multi-head self-attention layer in the first decoder. The features, after being concatenated with the learnable target query parameters (Objectqueries), are mapped to the key vector in the third multi-head self-attention layer of the second decoder. The features output by the fourth residual connection and layer normalization of the second decoder are concatenated with the learnable target query parameters and mapped to the query vector in the third multi-head self-attention layer of the second decoder.

[0032] Specifically, while a single ground viewpoint can provide local information, it cannot fully understand the context of the entire scene, thus requiring the fusion of cross-viewpoint features. To enhance information sharing between these two viewpoints, we propose a joint loss backpropagation mechanism. This mechanism ensures that the losses of the two decoders can be backpropagated to their respective decoders, while the fused loss... The data will be fed back to the Encoder for joint optimization. Step S4 specifically involves: space loss. and ground loss The data is sent back to its respective decoder, and the loss after fusion is recorded. The data is fed back to the two encoders for joint optimization, specifically as follows: ; ; in, It is an open space branch. For space loss or ground loss , These are the weighting coefficients for the corresponding tasks. To lose weight, For various scalar losses, For weights.

[0033] By weighted fusion of the losses from the two perspectives, the network can adaptively adjust the weights between the perspectives during training, thereby improving detection accuracy.

[0034] Specifically, the aerial view features are mapped to the value vector of the first multi-head self-attention layer in the first encoder, and the aerial view features and position information are concatenated and mapped to the key vector and query vector of the first multi-head self-attention layer in the first encoder; the ground view features are mapped to the value vector of the first multi-head self-attention layer in the second encoder, and the ground view features and position information are concatenated and mapped to the key vector and query vector of the first multi-head self-attention layer in the second encoder.

[0035] Specifically, the learnable object queries are mapped to the value vector, key vector, and query vector of the second multi-head self-attention layer in the first and second decoders.

[0036] Example 2 An electronic device includes a processor and a memory communicatively connected to the processor; the memory stores instructions that are executed by the processor to cause the processor to perform an air-ground cooperative target detection method based on a dual-branch network architecture according to Embodiment 1.

[0037] To comprehensively test the effectiveness of the proposed dual-branch network architecture-based air-ground cooperative detection algorithm, a self-made underground utility tunnel dataset was created. This dataset covers various types of targets commonly found in hazardous scenarios, such as boxes, rats, pressure gauges, shovels, traffic cones, construction barriers, warning posts, safety indicator lights, and cameras. The dataset employs an instance-level multi-task joint annotation framework, providing the target detection bounding boxes and their category information. The dataset consists of 1019 image pairs. While ensuring image quality, it also introduces challenging factors such as blurring due to device vibration and low lighting, increasing the complexity of real-world application scenarios.

[0038] This invention includes model comparison, ablation experiments, and inference detection experiments to demonstrate the effectiveness and advantages of the algorithm.

[0039] Model Comparison Experiment: To verify the advantages of the Transformer-based dual-branch structure of this invention over conventional models, a typical convolutional object detection network—Faster RCNN—was modified into an air-ground dual-branch version, and comparative experiments were conducted on the same complex underground scene dataset. Table 1 shows the mAP results of the three models under the condition of IoU=0.5.

[0040] Table 1 Model Comparison Experiment As can be seen, the Transformer-based dual-branch architecture of this invention achieves a 19.8% improvement over dual-branch convolutional networks, indicating that the proposed architecture can more effectively fuse aerial and ground viewpoints and significantly improve detection accuracy in complex underground scenarios.

[0041] Ablation experiments: On the same dataset, this embodiment compares the performance of two fusion strategies: static average fusion and dynamic weight fusion. The results are shown in Table 2.

[0042] Table 2 Ablation Experiment The results show that the introduction of the dynamic weight fusion module improved mAP by approximately 0.3 percentage points. This indicates that adaptively assigning fusion weights to the aerial and ground feature branches can more effectively utilize complementary information between viewpoints, thereby significantly improving detection performance.

[0043] Inference Detection Experiment: To verify the advantages of the open-ground dual-branch structure in terms of viewpoint coverage and object category recognition, this embodiment recorded the number of different object categories recognized by each method throughout the entire test set (a total of 7 categories: boxes, shovels, traffic cones, construction guardrails, warning posts, safety indicator lights, and surveillance cameras). The results are shown in Table 3.

[0044] Table 3 Reasoning Detection Experiment As shown in Table 3, although each single-view method can identify most categories within its advantageous perspective, blind spots still exist. In contrast, the proposed air-ground dual-branch cooperative detection method successfully integrates complementary information from two perspectives, achieving complete identification of all seven target categories. The advantages of the dual-branch architecture have been demonstrated.

[0045] Therefore, by visualizing the detection results of complex underground space scenarios, the core advantages of the air-ground collaborative detection network in terms of complementary perspectives are intuitively demonstrated. For example... Figure 2 As shown, the system marks all identified targets with red boxes and displays the target category and corresponding confidence score in real time in the upper left corner of the bounding box. The final detection visualization results not only reflect the detection accuracy but also highlight the unique advantages of the air-ground cooperative mechanism in handling viewpoint boundaries and incomplete targets. Figure 2 For example, when a ground-based view captures a half-section of a road cone at the edge of an image but an aerial view cannot effectively distinguish it, the cross-view compensation of information from the ground-aerial collaborative algorithm ensures the integrity of the detection results. This complementary enhancement of multi-dimensional information essentially improves the system's scene perception depth and survivability robustness in confined underground spaces.

[0046] In summary, compared to single-view detection, this invention effectively overcomes the problem of limited field of view and reduces the false negative rate. This advantage stems from the dual-branch network architecture proposed in this invention.

[0047] Compared to convolutional networks, the Transformer architecture is more effective at fusing multi-view information and has better accuracy. This advantage stems from improvements to the Encoder and the design of the cross-view feature fusion module.

[0048] This method is more effective than existing detection networks. This advantage stems from verification experiments on the air-ground cooperative detection method with a dual-branch architecture, effectively demonstrating the beneficial effects of this invention.

[0049] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A method for air-ground cooperative target detection based on a dual-branch network architecture, characterized in that, Specifically, the steps include the following: S1 utilizes two encoders in a dual-branch network to extract features from the aerial and ground perspectives respectively, and uses the cross-attention module in the encoder to interact with the features from different perspectives. S2, the cross-view feature fusion module in the encoder is used to fuse the features of the two views; S3 maps the fused features to the keys of the multi-head self-attention layers in the two decoders of the dual-branch network, and finally outputs the target category and bounding box through the feedforward neural network of the decoder. S4. Design a joint loss function to train the dual-branch network.

2. The air-ground cooperative target detection method based on a dual-branch network architecture according to claim 1, characterized in that, The two encoders in step S1 have the same structure. One encoder includes: a first multi-head self-attention layer, a first residual connection and layer normalization Add&Norm, a cross-attention module, a second residual connection and layer normalization, a first feedforward neural network FFN, and a third residual connection and layer normalization, connected in sequence.

3. The air-ground cooperative target detection method based on a dual-branch network architecture according to claim 2, characterized in that, The system comprises two encoders: a first encoder and a second encoder. The input to the first encoder is aerial view features and position information, while the input to the second encoder is ground view features and position information. A first residual connection in the first encoder is connected to the layer normalization module and the cross-attention module in the second encoder. Similarly, a first residual connection in the second encoder is connected to both the layer normalization module and the cross-attention module in the first encoder. The cross-attention formula for the first encoder is: ； in, Cross-attention features from an aerial perspective For normalized exponential functions, This is the query vector from an aerial perspective. The key vector is the ground-view vector. for transpose, This is the value vector from the ground perspective. The dimension of the key.

4. The air-ground cooperative target detection method based on a dual-branch network architecture according to claim 1, characterized in that, Step S2 specifically includes the following steps: S2.1, the output features of the first encoder and the second encoder are respectively ,in, For sequence length, For batch size, For feature dimensions, on the feature dimensions, spliced ​​together ; S2.2, the concatenated features are used to calculate the two-dimensional weight vector at each location using a multilayer perceptron (MLP). And through the softmax function Normalization to dynamic weights , ； ； Among them, subscript Indicates the sequence position. For sequence position The output characteristics of the first encoder, For sequence position The output characteristics of the second encoder, For sequence position Two-dimensional weight vector, ; S2.3, perform weighted fusion of the two features to obtain the fused features. and : ； ； in, Features resulting from the fusion of aerial perspectives. This is the fused feature from the ground perspective.

5. The air-ground cooperative target detection method based on a dual-branch network architecture according to claim 4, characterized in that, The two decoders have the same structure. The input to the first decoder and the second decoder is the learnable target query parameters. The first decoder includes: a second multi-head self-attention layer, a fourth residual connection and layer normalization layer, a third multi-head self-attention layer, a fifth residual connection and layer normalization layer, a second feedforward neural network and a sixth residual connection and layer normalization layer, which are connected in sequence. The sixth residual connection and layer normalization layer connects the third feedforward neural network and the fourth feedforward neural network. The third feedforward neural network outputs the target category, and the fourth feedforward neural network outputs the bounding box.

6. The air-ground cooperative target detection method based on a dual-branch network architecture according to claim 5, characterized in that, This is mapped to the value vector in the third multi-head self-attention layer of the first decoder. This is mapped to the value vector in the third multi-head self-attention layer of the second decoder. The features output by the fourth residual connection and layer normalization of the first decoder are concatenated with the learnable target query parameters and mapped to the key vector in the third multi-head self-attention layer of the first decoder. The features output by the fourth residual connection and layer normalization of the second decoder are concatenated with the learnable target query parameters and mapped to the key vector in the third multi-head self-attention layer of the second decoder.

7. The air-ground cooperative target detection method based on a dual-branch network architecture according to claim 1, characterized in that, Step S4 specifically involves: space loss and ground loss The data is sent back to its respective decoder, and the loss after fusion is recorded. The data is fed back to the two encoders for joint optimization, specifically as follows: ； ； in, It is an open space branch. For space loss or ground loss , These are the weighting coefficients for the corresponding tasks. To lose weight, For various scalar losses, For weights.

8. The air-ground cooperative target detection method based on a dual-branch network architecture according to claim 3, characterized in that, The aerial view features are mapped to the value vector of the first multi-head self-attention layer in the first encoder. The aerial view features and position information are concatenated and mapped to the key vector and query vector of the first multi-head self-attention layer in the first encoder. The ground view features are mapped to the value vector of the first multi-head self-attention layer in the second encoder. The ground view features and position information are concatenated and mapped to the key vector and query vector of the first multi-head self-attention layer in the second encoder.

9. The air-ground cooperative target detection method based on a dual-branch network architecture according to claim 5, characterized in that, The learnable target query parameters are mapped to the value vector, key vector, and query vector of the second multi-head self-attention layer in the first decoder and the second decoder.

10. An electronic device, characterized in that, The electronic device includes a processor and a memory communicatively connected to the processor; the memory stores instructions that are executed by the processor to cause the processor to perform the air-ground cooperative target detection method based on a dual-branch network architecture as described in any one of claims 1-9.

Images