Unmanned aerial vehicle multi-modal remote sensing image target detection method and device based on hybrid Mamb-CNN network

Abstract

The invention provides an unmanned aerial vehicle multi-mode remote sensing image target detection method and device based on a hybrid Mamb-CNN network. The method comprises the steps of obtaining a visible light image and a thermal infrared image to be detected; inputting the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result; the target detection network is constructed based on a twinborn feature extraction network, the twinborn feature extraction network comprises a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head, the visible light branch network extracts visible light local features in a visible light image, the infrared branch network extracts infrared local features in a thermal infrared image, and the cross-modal fusion module carries out cross-modal fusion; the cross-modal fusion module fuses the visible light local features and the infrared local features to obtain multi-scale features, the neck layer is based on the multi-scale features to obtain global fusion features, and the detection head is based on the global fusion features to obtain remote sensing detection results, so that the precision and robustness of multi-modal target detection are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a method and device for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on a hybrid Mamba-CNN network. Background Art

[0002] UAV remote sensing has become an important supplement to traditional satellite remote sensing due to its high flexibility and low operating cost. UAV remote sensing is widely used in environmental monitoring, disaster response and urban planning. It is equipped with various modal sensors, such as infrared, visible light and LiDAR, to achieve all-weather ground target monitoring. Due to the differences in spectral range, resolution and imaging conditions of sensor data of different modalities, efficient fusion methods are urgently needed to achieve accurate, effective and fast ground target detection. At present, the existing multimodal fusion methods are mainly based on the research of Convolutional Neural Network (CNN) and Transformer framework. However, CNN-based methods are difficult to effectively learn global context information due to the limitation of their receptive field. Transformers networks perform well in capturing long-range dependencies and global information, but their high computational burden limits their application in remote sensing target detection. Therefore, researchers combine CNN and Transformer methods to solve the above problems. For example, C2Former uses CNN as the backbone network to obtain image features and realizes feature alignment and complementarity between modalities based on the cross-attention module of Transformer, but it cannot avoid the computational complexity of Transformer network. To reduce the computational burden, researchers map features to lower dimensions when calculating global attention, which inevitably leads to information loss and affects the robustness of the fusion.

[0003] In recent years, the Mamba network has been able to efficiently capture the global contextual information of the target with linear computational complexity. Various enhanced versions based on the Mamba architecture have also achieved explosive success in the field of computer vision, including Vision Mamba, VMamba, MIM-ISTD, and Mamba-YOLO. Recently, Mamba-related research has also been widely used in the multimodal field. Fusion-Mamba maps cross-modal features to the hidden state space for interaction, reduces the differences between cross-modal features, and enhances the consistency of fused feature representations. Sigma is a Mamba semantic segmentation framework based on a twin network, which uses Mamba for feature extraction and integrates the Mamba fusion mechanism to select key information in different modalities.

[0004] However, the above methods mainly combine some CNNs with the original visual Mamba blocks through the query-key-value (QKV) mechanism, which on the one hand fails to fully utilize the advantages of the Mamba architecture, and on the other hand introduces redundant parameters, limiting the representation ability of feature fusion. Summary of the invention

[0005] The present invention provides a method and device for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on a hybrid Mamba-CNN network, so as to solve the problem that in the prior art, some CNNs are mainly combined with the Mamba blocks of the original vision through a query-key-value mechanism, which fails to fully utilize the advantages of the Mamba architecture and introduces redundant parameters, thereby limiting the representation ability of feature fusion.

[0006] The present invention provides a method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on a hybrid Mamba-CNN network, comprising the following steps: Acquire visible light images and thermal infrared images to be detected; Inputting the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result output by the target detection network; The target detection network is constructed based on a twin feature extraction network, which includes a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head. The visible light branch network is used to extract visible light local features in the visible light image, and the infrared branch network is used to extract infrared local features in the thermal infrared image. The cross-modal fusion module is used to fuse the visible light local features and the infrared local features to obtain multi-scale features. The neck layer is used to obtain global fusion features based on the multi-scale features. The detection head is used to obtain the remote sensing detection results based on the global fusion features.

[0007] According to a method for detecting targets in multimodal remote sensing images of unmanned aerial vehicles based on a hybrid Mamba-CNN network provided by the present invention, the cross-modal fusion module includes a first cross-modal fusion layer, a second cross-modal fusion layer and a third cross-modal fusion layer; The first cross-modal fusion layer is used to fuse the first visible light local feature and the first infrared local feature to obtain a first multi-scale feature; the first visible light local feature is output by the second feature extraction layer of the visible light branch network; the first infrared local feature is output by the second feature extraction layer of the infrared branch network; The second cross-modal fusion layer is used to fuse the second visible light local feature and the second infrared local feature to obtain a second multi-scale feature; the second visible light local feature is output by the third feature extraction layer of the visible light branch network; the second infrared local feature is output by the third feature extraction layer of the infrared branch network; The third cross-modal fusion layer is used to fuse the third visible light local feature and the third infrared local feature to obtain a third multi-scale feature; the third visible light local feature is output by the fifth feature extraction layer of the visible light branch network; the third infrared local feature is output by the fifth feature extraction layer of the infrared branch network; Correspondingly, the neck layer is used to obtain the global fusion feature based on the first multi-scale feature, the second multi-scale feature and the third multi-scale feature.

[0008] According to a method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on a hybrid Mamba-CNN network provided by the present invention, the cross-modal fusion module includes a normalization layer, a linear layer, a depth-separable convolution layer, an activation layer, a scan expansion and fusion layer, a selective state space model, a scan merging layer, a first residual layer and a second residual layer; The scanning expansion and fusion layer is used to flatten the visible light features and infrared features output by the depthwise separable convolution layer in four directions respectively to obtain one-dimensional visible light features and one-dimensional infrared features, and perform regional-level feature fusion on the one-dimensional visible light features and the one-dimensional infrared features to obtain multiple one-dimensional fusion sequences; The selective state space model is used to extract features from the multiple one-dimensional fusion sequences respectively to obtain multiple one-dimensional fusion features; The scanning merging layer is used to perform feature combination on the multiple one-dimensional fusion features, and project the combined features to the size of the original input feature space to obtain a merged feature; The first residual layer is used to obtain a first complementary feature based on the combined feature and the visible light local feature, and to obtain a second complementary feature based on the combined feature and the infrared local feature; The second residual layer is used to obtain a multi-scale first feature based on the first complementary feature, and to obtain a multi-scale second feature based on the second complementary feature.

[0009] According to a method for detecting a target in a multimodal remote sensing image of an unmanned aerial vehicle based on a hybrid Mamba-CNN network provided by the present invention, the training step of the target detection network includes: Obtaining a sample visible light image and a sample thermal infrared image, as well as remote sensing dataset labels corresponding to the sample visible light image and the sample thermal infrared image, and determining an initial target detection network; Inputting the sample visible light image and the sample thermal infrared image into the initial target detection network to obtain a predicted remote sensing detection result output by the initial target detection network; Based on the difference between the remote sensing data set label and the predicted remote sensing detection result, the initial target detection network is iterated on parameters to obtain the target detection network.

[0010] According to a method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on a hybrid Mamba-CNN network provided by the present invention, the method performs parameter iteration on the initial target detection network based on the difference between the remote sensing dataset label and the predicted remote sensing detection result, including: Determining a classification loss and a confidence loss based on differences between categories in the remote sensing dataset labels and categories in the predicted remote sensing detection results; Based on the classification loss and the confidence loss, a target loss is determined, and based on the target loss, parameter iteration is performed on the initial target detection network.

[0011] According to a method for detecting a target in a multimodal remote sensing image of an unmanned aerial vehicle based on a hybrid Mamba-CNN network provided by the present invention, the target loss is determined based on the classification loss and the confidence loss, including: Determining a bounding box regression loss based on a difference between the true box coordinates in the remote sensing dataset label and the predicted box coordinates in the predicted remote sensing detection result; The object loss is determined based on the classification loss, the confidence loss, and the bounding box regression loss.

[0012] The present invention also provides a UAV multimodal remote sensing image target detection device based on a hybrid Mamba-CNN network, comprising the following units: An acquisition unit, used for acquiring a visible light image and a thermal infrared image to be detected; An input unit, used to input the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result output by the target detection network; The target detection network is constructed based on a twin feature extraction network, which includes a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head. The visible light branch network is used to extract visible light local features in the visible light image, and the infrared branch network is used to extract infrared local features in the thermal infrared image. The cross-modal fusion module is used to fuse the visible light local features and the infrared local features to obtain multi-scale features. The neck layer is used to obtain global fusion features based on the multi-scale features. The detection head is used to obtain the remote sensing detection results based on the global fusion features.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on a hybrid Mamba-CNN network as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on a hybrid Mamba-CNN network as described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for detecting a target in a multi-modal remote sensing image of an unmanned aerial vehicle based on a hybrid Mamba-CNN network as described above is implemented.

[0016] The present invention provides a method and device for detecting a target of a multimodal remote sensing image of an unmanned aerial vehicle based on a hybrid Mamba-CNN network, obtains a visible light image and a thermal infrared image to be detected, inputs the visible light image and the thermal infrared image into a target detection network, obtains a remote sensing detection result output by the target detection network, and the target detection network is constructed based on a twin feature extraction network, and the twin feature extraction network includes a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer, and a detection head. The cross-modal fusion module is used to fuse the visible light local features and the infrared local features to obtain multi-scale features, and the neck layer is used to obtain global fusion features based on multi-scale features, and the detection head is used to obtain remote sensing detection results based on global fusion features. The learning of single-modal local features and the fusion of cross-modal global features in this process effectively enhance the discriminability of small targets, and improves the ability to distinguish between different categories by using local information, further enhancing the accuracy and robustness of multi-modal target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is one of the flow charts of the method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on the hybrid Mamba-CNN network provided by the present invention.

[0019] Figure 2This is the second flow chart of the method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on the hybrid Mamba-CNN network provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of the cross-modal fusion module provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of a UAV multimodal remote sensing image target detection device based on a hybrid Mamba-CNN network provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or precedence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type.

[0025] This paper aims at the influence of the shooting distance and imaging mechanism of UAV on ground targets. The targets in the image have small size, dense distribution, low distinction between categories and other problems. A target detection network based on four-directional selective scanning fusion strategy is proposed, named RemoteDet-Mamba.

[0026] Figure 1 This is one of the flow charts of the method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on a hybrid Mamba-CNN network provided by the present invention. Figure 2 FIG2 is a flow chart of the method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on a hybrid Mamba-CNN network provided by the present invention. Figure 1 , Figure 2 As shown, the method includes step 110 and step 120.

[0027] Step 110: Acquire a visible light image and a thermal infrared image to be detected.

[0028] Specifically, a visible light image and a thermal infrared image to be detected can be obtained, and the visible light image and the thermal infrared image can be obtained by a visible light imaging device and a thermal infrared imaging device carried by a drone.

[0029] Step 120, inputting the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result output by the target detection network; The target detection network is constructed based on a twin feature extraction network, which includes a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head. The visible light branch network is used to extract visible light local features in the visible light image, and the infrared branch network is used to extract infrared local features in the thermal infrared image. The cross-modal fusion module is used to fuse the visible light local features and the infrared local features to obtain multi-scale features. The neck layer is used to obtain global fusion features based on the multi-scale features. The detection head is used to obtain the remote sensing detection results based on the global fusion features.

[0030] Specifically, after obtaining the visible light image and the thermal infrared image, the visible light image and the thermal infrared image can be input into a target detection network to obtain a remote sensing detection result output by the target detection network.

[0031] Among them, the target detection network is built based on the twin feature extraction network, which includes a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer, and a detection head. The visible light branch network is used to extract visible light local features in visible light images, and the infrared branch network is used to extract infrared local features in thermal infrared images.

[0032] After obtaining the local features of the two modal targets using the twin feature extraction network with a convolutional neural network structure, the subsequent multimodal fusion process is carried out based on the Mamba framework, that is, the cross-modal fusion Mamba module (Cross FusionMamba, CFM) is used to fuse the visible light local features and the infrared local features to obtain multi-scale features.

[0033] Here, there are multiple cross-modal fusion modules in the target detection network, and each cross-modal fusion module may include a normalization layer, a linear layer, a depth-separable convolutional layer, an activation layer, a scan expansion and fusion layer, a selective state space model, a scan merging layer, a first residual layer, a feedforward neural network and a second residual layer.

[0034] The neck layer (Neck) is used to obtain global fusion features based on multi-scale features, and the detection head is used to obtain remote sensing detection results based on global fusion features.

[0035] In addition, the target detection network also includes a spatial pyramid pooling module (Spatial Pyramid Pooling Fast Feature Map, SPFF) and a residual module C3.

[0036] It can be understood that the twin convolutional neural network can extract multi-scale features from two multimodal images, such as visible light images and thermal infrared images. The cross-modal fusion module is located between the two modal feature extraction networks, and can realize the global feature fusion of deep local feature blocks between the two modalities. The final global fusion feature is composed of the integrated output of the cross-modal fusion module, the visible light branch network and the infrared branch network, thereby enhancing the accuracy and robustness of multimodal target detection.

[0037] It should be noted that the learning of unimodal local features and the fusion of cross-modal global features effectively enhance the discriminability of small targets and improve the ability to distinguish between different categories by using local information. In addition, Mamba's serial processing mechanism significantly improves the detection speed.

[0038] The method provided by the embodiment of the present invention obtains visible light images and thermal infrared images to be detected, inputs the visible light images and thermal infrared images into the target detection network, and obtains remote sensing detection results output by the target detection network. The target detection network is constructed based on a twin feature extraction network. The twin feature extraction network includes a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer, and a detection head. The cross-modal fusion module is used to fuse visible light local features and infrared local features to obtain multi-scale features. The neck layer is used to obtain global fusion features based on multi-scale features. The detection head is used to obtain remote sensing detection results based on global fusion features. In this process, the learning of single-modal local features and the fusion of cross-modal global features effectively enhance the discriminability of small targets, and improves the ability to distinguish between different categories by using local information, further enhancing the accuracy and robustness of multi-modal target detection.

[0039] The embodiment of the present invention aims to solve the problem of multimodal drone target detection in remote sensing images with high-density targets and complex scenes, and provides a new detection framework RemoteDet-Mamba. The framework integrates the complementary advantages of CNN and Mamba architectures, and has both the multi-scale local feature representation capability of CNN and the linear complexity and global receptive field provided by Mamba architecture.

[0040] Based on the above embodiment, the cross-modal fusion module includes a first cross-modal fusion layer, a second cross-modal fusion layer and a third cross-modal fusion layer; The first cross-modal fusion layer is used to fuse the first visible light local feature and the first infrared local feature to obtain a first multi-scale feature; the first visible light local feature is output by the second feature extraction layer of the visible light branch network; the first infrared local feature is output by the second feature extraction layer of the infrared branch network; The second cross-modal fusion layer is used to fuse the second visible light local feature and the second infrared local feature to obtain a second multi-scale feature; the second visible light local feature is output by the third feature extraction layer of the visible light branch network; the second infrared local feature is output by the third feature extraction layer of the infrared branch network; The third cross-modal fusion layer is used to fuse the third visible light local feature and the third infrared local feature to obtain a third multi-scale feature; the third visible light local feature is output by the fifth feature extraction layer of the visible light branch network; the third infrared local feature is output by the fifth feature extraction layer of the infrared branch network; Correspondingly, the neck layer is used to obtain the global fusion feature based on the first multi-scale feature, the second multi-scale feature and the third multi-scale feature.

[0041] Specifically, the cross-modal fusion module may include four cross-modal fusion layers, and the embodiment of the present invention requires the use of a first cross-modal fusion layer, a second cross-modal fusion layer, and a third cross-modal fusion layer.

[0042] Taking two modal inputs as an example, the input image is defined as , where S=1, 2 represent two different modalities. These images are first fed into the convolutional neural network block in the object detection network to extract their multi-scale features:

[0043] in, represents the features extracted from the i-th layer, and Represent the local features of the two modalities respectively. In order to perform multi-scale feature extraction, the input of the Neck part consists of three parts, which are the fusion outputs of the 2nd, 3rd, and 5th layers of the twin feature extraction network, which can be expressed as .

[0044]

[0045] In the Neck part, multi-scale feature fusion is performed to generate It will be sent to the detection head, and then generate the final remote sensing detection results.

[0046] That is, the first visible light local feature is output by the second feature extraction layer of the visible light branch network, the first infrared local feature is output by the second feature extraction layer of the infrared branch network, the second visible light local feature is output by the third feature extraction layer of the visible light branch network, the second infrared local feature is output by the third feature extraction layer of the infrared branch network, the third visible light local feature is output by the fifth feature extraction layer of the visible light branch network, and the third infrared local feature is output by the fifth feature extraction layer of the infrared branch network.

[0047] Correspondingly, the neck layer is used to perform feature fusion on the first multi-scale features, the second multi-scale features and the third multi-scale features to obtain a global fusion feature.

[0048] Here, the feature fusion of the first multi-scale feature, the second multi-scale feature and the third multi-scale feature may be performed by concatenating the first multi-scale feature, the second multi-scale feature and the third multi-scale feature, or by weighting the first multi-scale feature, the second multi-scale feature and the third multi-scale feature using an attention mechanism and then concatenating them, which is not specifically limited in the embodiment of the present invention.

[0049] Based on the above embodiments, Figure 3 is a schematic diagram of the structure of the cross-modal fusion module provided by the present invention, such as Figure 3 As shown, the cross-modal fusion module includes a normalization layer, a linear layer, a depth-separable convolution layer, an activation layer, a scan expansion and fusion layer, a selective state space model, a scan merging layer, a first residual layer, and a second residual layer; The scanning expansion and fusion layer is used to flatten the visible light features and infrared features output by the depthwise separable convolution layer in four directions respectively to obtain one-dimensional visible light features and one-dimensional infrared features, and perform regional-level feature fusion on the one-dimensional visible light features and the one-dimensional infrared features to obtain multiple one-dimensional fusion sequences; The selective state space model is used to extract features from the multiple one-dimensional fusion sequences respectively to obtain multiple one-dimensional fusion features; The scanning merging layer is used to perform feature combination on the multiple one-dimensional fusion features, and project the combined features to the size of the original input feature space to obtain a merged feature; The first residual layer is used to obtain a first complementary feature based on the combined feature and the visible light local feature, and to obtain a second complementary feature based on the combined feature and the infrared local feature; The second residual layer is used to obtain a multi-scale first feature based on the first complementary feature, and to obtain a multi-scale second feature based on the second complementary feature.

[0050] Specifically, after using the twin feature extraction network with a convolutional neural network structure to obtain the local features of the two modal targets, the cross-modal fusion module based on the Mamba framework performs the subsequent multimodal fusion process. Figure 3 is a schematic diagram of the structure of the cross-modal fusion module provided by the present invention, such as Figure 3 Assume , They represent the local features of visible light and thermal infrared modalities respectively, and the LayerNorm layer is used to normalize the input features to accelerate model convergence.

[0051]

[0052] in, Represents the LayerNorm operation (normalization layer), Denotes a projection operation (linear layer) using a linear transformation. Depthwise separable convolution is applied to the two modality features to facilitate communication between channels:

[0053] in, represents a depth-wise separable convolution operation, Represents the SiLU activation function. The Mamba architecture uses a unique selective scanning mechanism to adjust parameters according to the input data to achieve selective scanning fusion processing of two modal features. Like VMamba, and Flatten in four directions (up, down, left and right) to generate 1D sequences, each of size HW×C, and perform region-level feature fusion by element addition:

[0054] in, represents the scanning operation along the i-th direction.

[0055] The selective state-space model processes these one-dimensional fused sequences individually for feature extraction, producing four outputs, denoted as y1, y2, y3, and y4.

[0056]

[0057] in, represents the i-th selective state-space model (S6 block). The scan-and-merge layer is used to unfold and recombine the output of the S6 block into , and project it back to the original input feature space size, that is:

[0058] in, represents the merged features, Represents the flattening operation along the i-th direction. The original input features of the two modalities are passed through the residual connection to obtain the first complementary features and the second complementary feature :

[0059] Finally, the complementary features are processed by layer normalization and feedforward neural network respectively, and then residually connected with the complementary features to obtain the final output.

[0060]

[0061] Among them, GELU is the activation function, W1, W2, b1 and b2 are the weights and biases of the linear projection.

[0062] In terms of multimodal fusion, the embodiment of the present invention constructs a CFM module based on the Mamba architecture, which performs comprehensive global scanning from four directions through a selective scanning mechanism, effectively isolates dense detection objects, and promotes fine and targeted feature fusion.

[0063] Furthermore, the framework of the target detection network of the present invention is mainly composed of a twin convolutional neural network and a cross-modal fusion Mamba module. First, multi-scale local information is extracted through the twin network encoder, and the CFM module uses a design based on the Mamba selective scanning two-dimensional mechanism (2D-Selective-Scan for Vision Data, SS2D) to scan the extracted multi-scale features in four directions to achieve decoupling of dense targets and selective feature fusion. The linear scanning strategy of the embodiment of the present invention has linear time complexity, can capture long-range dependencies at the level of local feature blocks, and significantly improve the accuracy and efficiency of target detection.

[0064] Based on the above embodiment, the training step of the target detection network includes: Step 210, obtaining a sample visible light image and a sample thermal infrared image, as well as remote sensing data set labels corresponding to the sample visible light image and the sample thermal infrared image, and determining an initial target detection network; Step 220, inputting the sample visible light image and the sample thermal infrared image into the initial target detection network to obtain a predicted remote sensing detection result output by the initial target detection network; Step 230, based on the difference between the remote sensing data set label and the predicted remote sensing detection result, the initial target detection network is iterated to obtain the target detection network.

[0065] Specifically, in order to better obtain the target detection network and improve the accuracy and reliability of target detection, training can be performed based on the following steps: First, obtain sample visible light images and sample thermal infrared images, as well as remote sensing dataset labels corresponding to the sample visible light images and sample thermal infrared images, and determine an initial target detection network.

[0066] The parameters of the initial target detection network here may be randomly generated or pre-set, and the embodiment of the present invention does not specifically limit this.

[0067] Then, the sample visible light image and the sample thermal infrared image are input into the initial target detection network to obtain the predicted remote sensing detection result output by the initial target detection network.

[0068] After obtaining the predicted remote sensing detection result based on the initial target detection network, the parameters of the initial target detection network can be iterated based on the difference between the remote sensing dataset label and the predicted remote sensing detection result, and the initial target detection network after the parameter iteration is used as the target detection network.

[0069] It can be understood that the greater the difference between the remote sensing dataset label and the predicted remote sensing detection result, the greater the target loss; the smaller the difference between the remote sensing dataset label and the predicted remote sensing detection result, the smaller the target loss.

[0070] Based on the above embodiment, the step 230 of iterating parameters of the initial target detection network based on the difference between the remote sensing data set label and the predicted remote sensing detection result includes: Step 231, determining classification loss and confidence loss based on the difference between the categories in the remote sensing dataset label and the categories in the predicted remote sensing detection result; Step 232: determine the target loss based on the classification loss and the confidence loss, and perform parameter iteration on the initial target detection network based on the target loss.

[0071] Specifically, the classification loss is determined based on the difference between the categories in the remote sensing dataset labels and the categories in the predicted remote sensing detection results. and confidence loss .

[0072] in, , Both are based on smoothed binary cross entropy (Smooth BCE), To enhance numerical stability, the latter When calculating, a CIoU (Complete-IOU) loss is added between horizontal boundaries to speed up the training process.

[0073] After obtaining the classification loss and the confidence loss, the target loss can be determined based on the sum of the classification loss and the confidence loss, or the weighted sum, and the parameters of the initial target detection network can be iterated based on the target loss.

[0074] Based on the above embodiment, determining the target loss based on the classification loss and the confidence loss in step 232 includes: Step 2321, determining a bounding box regression loss based on a difference between the real box coordinates in the remote sensing dataset label and the predicted box coordinates in the predicted remote sensing detection result; Step 2322, determining the target loss based on the classification loss, the confidence loss, and the bounding box regression loss.

[0075] Specifically, the bounding box regression loss can be determined based on the difference between the true box coordinates in the remote sensing dataset label and the predicted box coordinates in the predicted remote sensing detection results. .

[0076] It can be understood that the greater the difference between the true box coordinates in the remote sensing dataset label and the predicted box coordinates in the predicted remote sensing detection results, the greater the bounding box regression loss; the smaller the difference between the true box coordinates in the remote sensing dataset label and the predicted box coordinates in the predicted remote sensing detection results, the smaller the bounding box regression loss.

[0077] Finally, the target loss can be determined based on the classification loss, confidence loss, and bounding box regression loss. The formula is as follows:

[0078] in, represents the target loss, represents the bounding box regression loss, represents the confidence loss, represents the classification loss, Using Complete-IOU (CIoU).

[0079] The following is a description of the UAV multimodal remote sensing image target detection device based on the hybrid Mamba-CNN network provided by the present invention. The UAV multimodal remote sensing image target detection device based on the hybrid Mamba-CNN network described below and the UAV multimodal remote sensing image target detection method based on the hybrid Mamba-CNN network described above can be referred to each other.

[0080] Based on any of the above embodiments, the present invention provides a multi-modal remote sensing image target detection device for unmanned aerial vehicles based on a hybrid Mamba-CNN network, Figure 4 : is a structural schematic diagram of a multi-modal remote sensing image target detection device for unmanned aerial vehicles based on a hybrid Mamba-CNN network provided by the present invention, such as Figure 4 As shown, the device comprises: An acquisition unit 410 is used to acquire a visible light image and a thermal infrared image to be detected; An input unit 420, used to input the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result output by the target detection network; The target detection network is constructed based on a twin feature extraction network, which includes a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head. The visible light branch network is used to extract visible light local features in the visible light image, and the infrared branch network is used to extract infrared local features in the thermal infrared image. The cross-modal fusion module is used to fuse the visible light local features and the infrared local features to obtain multi-scale features. The neck layer is used to obtain global fusion features based on the multi-scale features. The detection head is used to obtain the remote sensing detection results based on the global fusion features.

[0081] The device provided by the embodiment of the present invention obtains the visible light image and thermal infrared image to be detected, inputs the visible light image and thermal infrared image into the target detection network, and obtains the remote sensing detection result output by the target detection network. The target detection network is constructed based on the twin feature extraction network. The twin feature extraction network includes a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head. The cross-modal fusion module is used to fuse the visible light local features and the infrared local features to obtain multi-scale features. The neck layer is used to obtain global fusion features based on multi-scale features. The detection head is used to obtain remote sensing detection results based on global fusion features. The learning of single-modal local features and the fusion of cross-modal global features in this process effectively enhance the discriminability of small targets, and improve the ability to distinguish between different categories by using local information, further enhancing the accuracy and robustness of multi-modal target detection.

[0082] Based on any of the above embodiments, the cross-modal fusion module includes a first cross-modal fusion layer, a second cross-modal fusion layer and a third cross-modal fusion layer; The first cross-modal fusion layer is used to fuse the first visible light local feature and the first infrared local feature to obtain a first multi-scale feature; the first visible light local feature is output by the second feature extraction layer of the visible light branch network; the first infrared local feature is output by the second feature extraction layer of the infrared branch network; The second cross-modal fusion layer is used to fuse the second visible light local feature and the second infrared local feature to obtain a second multi-scale feature; the second visible light local feature is output by the third feature extraction layer of the visible light branch network; the second infrared local feature is output by the third feature extraction layer of the infrared branch network; The third cross-modal fusion layer is used to fuse the third visible light local feature and the third infrared local feature to obtain a third multi-scale feature; the third visible light local feature is output by the fifth feature extraction layer of the visible light branch network; the third infrared local feature is output by the fifth feature extraction layer of the infrared branch network; Correspondingly, the neck layer is used to obtain the global fusion feature based on the first multi-scale feature, the second multi-scale feature and the third multi-scale feature.

[0083] Based on any of the above embodiments, the cross-modal fusion module includes a normalization layer, a linear layer, a depth-separable convolution layer, an activation layer, a scan expansion and fusion layer, a selective state space model, a scan merging layer, a first residual layer, and a second residual layer; The scanning expansion and fusion layer is used to flatten the visible light features and infrared features output by the depthwise separable convolution layer in four directions respectively to obtain one-dimensional visible light features and one-dimensional infrared features, and perform regional-level feature fusion on the one-dimensional visible light features and the one-dimensional infrared features to obtain multiple one-dimensional fusion sequences; The selective state space model is used to extract features from the multiple one-dimensional fusion sequences respectively to obtain multiple one-dimensional fusion features; The scanning merging layer is used to perform feature combination on the multiple one-dimensional fusion features, and project the combined features to the size of the original input feature space to obtain a merged feature; The first residual layer is used to obtain a first complementary feature based on the combined feature and the visible light local feature, and to obtain a second complementary feature based on the combined feature and the infrared local feature; The second residual layer is used to obtain a multi-scale first feature based on the first complementary feature, and to obtain a multi-scale second feature based on the second complementary feature.

[0084] Based on any of the above embodiments, it further includes a training unit, wherein the training unit is specifically used for: A sample acquisition unit is used to acquire a sample visible light image and a sample thermal infrared image, as well as remote sensing data set labels corresponding to the sample visible light image and the sample thermal infrared image, and determine an initial target detection network; An input unit, used to input the sample visible light image and the sample thermal infrared image into the initial target detection network to obtain a predicted remote sensing detection result output by the initial target detection network; A parameter iteration unit is used to perform parameter iteration on the initial target detection network based on the difference between the remote sensing data set label and the predicted remote sensing detection result to obtain the target detection network.

[0085] Based on any of the above embodiments, the parameter iteration unit is specifically used to: A loss determination unit, configured to determine a classification loss and a confidence loss based on a difference between a category in the remote sensing dataset label and a category in the predicted remote sensing detection result; A parameter iteration subunit is used to determine a target loss based on the classification loss and the confidence loss, and to perform parameter iteration on the initial target detection network based on the target loss.

[0086] Based on any of the above embodiments, the parameter iteration subunit is specifically used for: Determining a bounding box regression loss based on a difference between the true box coordinates in the remote sensing dataset label and the predicted box coordinates in the predicted remote sensing detection result; The object loss is determined based on the classification loss, the confidence loss, and the bounding box regression loss.

[0087] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute the UAV multimodal remote sensing image target detection method based on the hybrid Mamba-CNN network, the method comprising: obtaining a visible light image and a thermal infrared image to be detected; inputting the visible light image and the thermal infrared image into the target detection network to obtain a remote sensing detection result output by the target detection network; the target detection network is constructed based on a twin feature extraction network, the twin feature extraction network comprises a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head, the visible light branch network is used to extract visible light local features in the visible light image, the infrared branch network is used to extract infrared local features in the thermal infrared image, the cross-modal fusion module is used to fuse the visible light local features and the infrared local features to obtain multi-scale features, the neck layer is used to obtain global fusion features based on the multi-scale features, and the detection head is used to obtain the remote sensing detection results based on the global fusion features.

[0088] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0089] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the UAV multimodal remote sensing image target detection method based on a hybrid Mamba-CNN network provided by the above methods, the method including: obtaining a visible light image and a thermal infrared image to be detected; inputting the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result output by the target detection network; the target detection network is constructed based on a twin feature extraction network, and the twin feature extraction network includes a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head, the visible light branch network is used to extract visible light local features in the visible light image, the infrared branch network is used to extract infrared local features in the thermal infrared image, the cross-modal fusion module is used to fuse the visible light local features and the infrared local features to obtain multi-scale features, the neck layer is used to obtain global fusion features based on the multi-scale features, and the detection head is used to obtain the remote sensing detection results based on the global fusion features.

[0090] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the UAV multimodal remote sensing image target detection method based on a hybrid Mamba-CNN network provided by the above-mentioned methods, the method comprising: obtaining a visible light image and a thermal infrared image to be detected; inputting the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result output by the target detection network; the target detection network is constructed based on a twin feature extraction network, the twin feature extraction network comprising a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head, the visible light branch network is used to extract visible light local features in the visible light image, the infrared branch network is used to extract infrared local features in the thermal infrared image, the cross-modal fusion module is used to fuse the visible light local features and the infrared local features to obtain multi-scale features, the neck layer is used to obtain global fusion features based on the multi-scale features, and the detection head is used to obtain the remote sensing detection results based on the global fusion features.

[0091] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on hybrid Mamba-CNN network, characterized in that: include: Acquire visible light images and thermal infrared images to be detected; Inputting the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result output by the target detection network; The target detection network is constructed based on a twin feature extraction network, which includes a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head. The visible light branch network is used to extract visible light local features in the visible light image, and the infrared branch network is used to extract infrared local features in the thermal infrared image. The cross-modal fusion module is used to fuse the visible light local features and the infrared local features to obtain multi-scale features. The neck layer is used to obtain global fusion features based on the multi-scale features. The detection head is used to obtain the remote sensing detection results based on the global fusion features.

2. The method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on hybrid Mamba-CNN network according to claim 1, characterized in that: The cross-modal fusion module includes a first cross-modal fusion layer, a second cross-modal fusion layer and a third cross-modal fusion layer; The first cross-modal fusion layer is used to fuse the first visible light local feature and the first infrared local feature to obtain a first multi-scale feature; the first visible light local feature is output by the second feature extraction layer of the visible light branch network; the first infrared local feature is output by the second feature extraction layer of the infrared branch network; The second cross-modal fusion layer is used to fuse the second visible light local feature and the second infrared local feature to obtain a second multi-scale feature; the second visible light local feature is output by the third feature extraction layer of the visible light branch network; the second infrared local feature is output by the third feature extraction layer of the infrared branch network; The third cross-modal fusion layer is used to fuse the third visible light local feature and the third infrared local feature to obtain a third multi-scale feature; the third visible light local feature is output by the fifth feature extraction layer of the visible light branch network; the third infrared local feature is output by the fifth feature extraction layer of the infrared branch network; Correspondingly, the neck layer is used to obtain the global fusion feature based on the first multi-scale feature, the second multi-scale feature and the third multi-scale feature.

3. The method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on hybrid Mamba-CNN network according to claim 1, characterized in that: The cross-modal fusion module includes a normalization layer, a linear layer, a depth-separable convolution layer, an activation layer, a scan expansion and fusion layer, a selective state space model, a scan merging layer, a first residual layer, and a second residual layer; The scanning expansion and fusion layer is used to flatten the visible light features and infrared features output by the depthwise separable convolution layer in four directions respectively to obtain one-dimensional visible light features and one-dimensional infrared features, and perform regional-level feature fusion on the one-dimensional visible light features and the one-dimensional infrared features to obtain multiple one-dimensional fusion sequences; The selective state space model is used to extract features from the multiple one-dimensional fusion sequences respectively to obtain multiple one-dimensional fusion features; The scanning merging layer is used to perform feature combination on the multiple one-dimensional fusion features, and project the combined features to the size of the original input feature space to obtain a merged feature; The first residual layer is used to obtain a first complementary feature based on the combined feature and the visible light local feature, and to obtain a second complementary feature based on the combined feature and the infrared local feature; The second residual layer is used to obtain a multi-scale first feature based on the first complementary feature, and to obtain a multi-scale second feature based on the second complementary feature.

4. The method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on a hybrid Mamba-CNN network according to any one of claims 1 to 3, characterized in that: The training steps of the target detection network include: Obtaining a sample visible light image and a sample thermal infrared image, as well as remote sensing dataset labels corresponding to the sample visible light image and the sample thermal infrared image, and determining an initial target detection network; Inputting the sample visible light image and the sample thermal infrared image into the initial target detection network to obtain a predicted remote sensing detection result output by the initial target detection network; Based on the difference between the remote sensing data set label and the predicted remote sensing detection result, the initial target detection network is iterated on parameters to obtain the target detection network.

5. The method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on hybrid Mamba-CNN network according to claim 4 is characterized in that: The performing parameter iteration on the initial target detection network based on the difference between the remote sensing data set label and the predicted remote sensing detection result comprises: Determining a classification loss and a confidence loss based on differences between categories in the remote sensing dataset labels and categories in the predicted remote sensing detection results; Based on the classification loss and the confidence loss, a target loss is determined, and based on the target loss, parameter iteration is performed on the initial target detection network.

6. The method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on hybrid Mamba-CNN network according to claim 5, characterized in that: The determining the target loss based on the classification loss and the confidence loss includes: Determining a bounding box regression loss based on a difference between the true box coordinates in the remote sensing dataset label and the predicted box coordinates in the predicted remote sensing detection result; The object loss is determined based on the classification loss, the confidence loss, and the bounding box regression loss.

7. A UAV multimodal remote sensing image target detection device based on a hybrid Mamba-CNN network, characterized in that: include: An acquisition unit, used for acquiring a visible light image and a thermal infrared image to be detected; An input unit, used to input the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result output by the target detection network; The target detection network is constructed based on a twin feature extraction network, which includes a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head. The visible light branch network is used to extract visible light local features in the visible light image, and the infrared branch network is used to extract infrared local features in the thermal infrared image. The cross-modal fusion module is used to fuse the visible light local features and the infrared local features to obtain multi-scale features. The neck layer is used to obtain global fusion features based on the multi-scale features. The detection head is used to obtain the remote sensing detection results based on the global fusion features.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on a hybrid Mamba-CNN network is implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on a hybrid Mamba-CNN network is implemented as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on a hybrid Mamba-CNN network is implemented as described in any one of claims 1 to 6.

Images