Unmanned aerial vehicle multi-modal remote sensing image target detection method and device based on hybrid Mamba-CNN network

Abstract

The application provides a kind of unmanned aerial vehicle multimodal remote sensing image target detection method and device based on mixed Mamba-CNN network, wherein the method comprises: obtaining visible light image and thermal infrared image to be detected;Visible light image and thermal infrared image are input into target detection network, and remote sensing detection result is obtained;Target detection network is constructed based on twin feature extraction network, and twin feature extraction network includes visible light branch network, infrared branch network, cross-modal fusion module, neck layer and detection head, visible light branch network extracts visible light local feature in visible light image, infrared branch network extracts infrared local feature in thermal infrared image, cross-modal fusion module fuses visible light local feature and infrared local feature, obtains multi-scale feature, neck layer is based on multi-scale feature, obtains global fusion feature, and detection head is based on global fusion feature, obtains remote sensing detection result, and the precision and robustness of multimodal target detection are enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and particularly relates to a multi-modal remote sensing image target detection method and device based on a mixed Mamba-CNN network. BACKGROUND

[0002] Unmanned aerial vehicle (UAV) remote sensing has become an important supplement to traditional satellite remote sensing due to its high flexibility and low operating cost. UAV remote sensing is widely used in environmental monitoring, disaster response and urban planning, and various modal sensors such as infrared, visible light and LiDAR are equipped to achieve all-weather ground target monitoring. Due to the differences in spectral range, resolution and imaging conditions of different modal sensor data, there is an urgent need for efficient fusion methods to achieve accurate, effective and rapid ground target detection. At present, existing multi-modal fusion methods are mainly based on research in the Convolutional Neural Network (CNN) and Transformer framework. However, due to the limitations of the receptive field, CNN-based methods are difficult to effectively learn global context information. Transformers network performs well in capturing long-range dependencies and global information, but its high computational burden limits its application in remote sensing target detection. Therefore, researchers combine CNN and Transformer methods to solve the above problems, for example, C2Former uses CNN as the backbone network to obtain the features of the image, and realizes the feature alignment and complementarity between modalities based on the cross-attention module of the Transformer, but cannot avoid the computational complexity of the Transformer network. In order to reduce the computational burden, researchers map the features to a lower dimension when computing the global attention, which inevitably leads to the loss of information, affecting the robustness of fusion.

[0003] In recent years, Mamba network can efficiently capture the global context information of the target with linear computational complexity. Various enhanced versions based on Mamba architecture have also achieved explosive success in the field of computer vision, including Vision Mamba, VMamba, MIM-ISTD and Mamba-YOLO. Recently, Mamba-related research has also been widely applied in the multi-modal field. Fusion-Mamba maps cross-modal features to a hidden state space for interaction, reduces the differences between cross-modal features, and enhances the consistency of the fused feature representation. Sigma is a Mamba semantic segmentation framework based on twin networks, which uses Mamba for feature extraction and integrates a Mamba fusion mechanism to select key information in different modalities.

[0004] However, the above method mainly combines some CNNs with the original visual Mamba block through a query-key-value (QKV) mechanism, which on the one hand fails to fully utilize the advantages of the Mamba architecture, and on the other hand introduces redundant parameters, limiting the representation ability of feature fusion. SUMMARY

[0005] The application provides a UAV multi-modal remote sensing image target detection method and device based on a hybrid Mamba-CNN network, to solve the defects in the prior art that some CNNs are combined with the original visual Mamba block through a query-key-value mechanism, which on the one hand fails to fully utilize the advantages of the Mamba architecture, and on the other hand introduces redundant parameters, limiting the representation ability of feature fusion.

[0006] The application provides a UAV multi-modal remote sensing image target detection method based on a hybrid Mamba-CNN network, comprising the following steps:

[0007] Obtaining a visible light image and a thermal infrared image to be detected;

[0008] Inputting the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result output by the target detection network;

[0009] The target detection network is constructed based on a twin feature extraction network, the twin feature extraction network comprises a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head, the visible light branch network is used for extracting visible light local features in the visible light image, the infrared branch network is used for extracting infrared local features in the thermal infrared image, the cross-modal fusion module is used for fusing the visible light local features and the infrared local features to obtain multi-scale features, the neck layer is used for obtaining global fusion features based on the multi-scale features, and the detection head is used for obtaining the remote sensing detection result based on the global fusion features.

[0010] According to the UAV multi-modal remote sensing image target detection method based on the hybrid Mamba-CNN network provided by the application, the cross-modal fusion module comprises a first cross-modal fusion layer, a second cross-modal fusion layer and a third cross-modal fusion layer;

[0011] The first cross-modal fusion layer is used for fusing first visible light local features and first infrared local features to obtain first multi-scale features; the first visible light local features are output by a second layer feature extraction layer of the visible light branch network; and the first infrared local features are output by a second layer feature extraction layer of the infrared branch network;

[0012] The second cross-modal fusion layer is configured to fuse the second visible light local feature and the second infrared local feature to obtain a second multi-scale feature; the second visible light local feature is output by a third layer feature extraction layer of the visible light branch network; and the second infrared local feature is output by a third layer feature extraction layer of the infrared branch network.

[0013] The third cross-modal fusion layer is configured to fuse the third visible light local feature and the third infrared local feature to obtain a third multi-scale feature; the third visible light local feature is output by a fifth layer feature extraction layer of the visible light branch network; and the third infrared local feature is output by a fifth layer feature extraction layer of the infrared branch network.

[0014] Correspondingly, the neck layer is configured to obtain the global fusion feature based on the first multi-scale feature, the second multi-scale feature and the third multi-scale feature.

[0015] According to the unmanned aerial vehicle multi-modal remote sensing image target detection method based on the mixed Mamba-CNN network, the cross-modal fusion module comprises a normalization layer, a linear layer, a depth separable convolution layer, an activation layer, a scanning expansion and fusion layer, a selective state space model, a scanning merging layer, a first residual layer and a second residual layer.

[0016] The scanning expansion and fusion layer is configured to flatten the visible light feature and the infrared feature output by the depth separable convolution layer in four directions respectively to obtain one-dimensional visible light features and one-dimensional infrared features, and perform region-level feature fusion on the one-dimensional visible light features and the one-dimensional infrared features to obtain a plurality of one-dimensional fusion sequences.

[0017] The selective state space model is configured to perform feature extraction on the plurality of one-dimensional fusion sequences respectively to obtain a plurality of one-dimensional fusion features.

[0018] The scanning merging layer is configured to combine the plurality of one-dimensional fusion features, and project the combined features to the size of the original input feature space to obtain a merged feature.

[0019] The first residual layer is configured to obtain a first complementary feature based on the merged feature and the visible light local feature, and obtain a second complementary feature based on the merged feature and the infrared local feature.

[0020] The second residual layer is configured to obtain a multi-scale first feature based on the first complementary feature, and obtain a multi-scale second feature based on the second complementary feature.

[0021] According to the unmanned aerial vehicle multi-modal remote sensing image target detection method based on the hybrid Mamba-CNN network provided by the application, the training step of the target detection network comprises:

[0022] Obtaining sample visible light images and sample thermal infrared images, and remote sensing data set labels corresponding to the sample visible light images and the sample thermal infrared images, and determining an initial target detection network;

[0023] Inputting the sample visible light images and the sample thermal infrared images into the initial target detection network to obtain a predicted remote sensing detection result output by the initial target detection network;

[0024] Based on the difference between the remote sensing data set labels and the predicted remote sensing detection result, the initial target detection network is iterated in parameters to obtain the target detection network.

[0025] According to the unmanned aerial vehicle multi-modal remote sensing image target detection method based on the hybrid Mamba-CNN network provided by the application, the training step of the target detection network comprises:

[0026] Based on the difference between the categories in the remote sensing data set labels and the categories in the predicted remote sensing detection result, a classification loss and a confidence loss are determined;

[0027] Based on the classification loss and the confidence loss, a target loss is determined, and the initial target detection network is iterated in parameters based on the target loss.

[0028] According to the unmanned aerial vehicle multi-modal remote sensing image target detection method based on the hybrid Mamba-CNN network provided by the application, the training step of the target detection network comprises:

[0029] Based on the difference between the real box coordinates in the remote sensing data set labels and the predicted box coordinates in the predicted remote sensing detection result, a boundary box regression loss is determined;

[0030] Based on the classification loss, the confidence loss, and the boundary box regression loss, the target loss is determined.

[0031] The application also provides an unmanned aerial vehicle multi-modal remote sensing image target detection device based on a hybrid Mamba-CNN network, comprising the following units:

[0032] The acquisition unit is used for acquiring visible light images and thermal infrared images to be detected;

[0033] An input unit is configured to input the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result output by the target detection network.

[0034] The target detection network is constructed based on a twin feature extraction network, the twin feature extraction network comprises a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head, the visible light branch network is configured to extract visible light local features in the visible light image, the infrared branch network is configured to extract infrared local features in the thermal infrared image, the cross-modal fusion module is configured to fuse the visible light local features and the infrared local features to obtain multi-scale features, the neck layer is configured to obtain global fusion features based on the multi-scale features, and the detection head is configured to obtain the remote sensing detection result based on the global fusion features.

[0035] The application further provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for unmanned aerial vehicle multi-modal remote sensing image target detection based on the hybrid Mamba-CNN network according to any one of the above.

[0036] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the method for unmanned aerial vehicle multi-modal remote sensing image target detection based on the hybrid Mamba-CNN network according to any one of the above.

[0037] The application further provides a computer program product comprising a computer program, wherein the computer program is executable by a processor to implement the method for unmanned aerial vehicle multi-modal remote sensing image target detection based on the hybrid Mamba-CNN network according to any one of the above.

[0038] The application provides a method and device for unmanned aerial vehicle multi-modal remote sensing image target detection based on a hybrid Mamba-CNN network, which acquires a visible light image and a thermal infrared image to be detected, inputs the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result output by the target detection network, and constructs the target detection network based on a twin feature extraction network, wherein the twin feature extraction network comprises a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head, the cross-modal fusion module is configured to fuse visible light local features and infrared local features to obtain multi-scale features, the neck layer is configured to obtain global fusion features based on the multi-scale features, and the detection head is configured to obtain the remote sensing detection result based on the global fusion features. This process effectively enhances the discriminability of small targets through learning of single-modal local features and fusion of cross-modal global features, improves the discrimination ability between different categories by using local information, and further enhances the precision and robustness of multi-modal target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 This is one of the flow charts of the method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on the hybrid Mamba-CNN network provided by the present invention.

[0041] Figure 2 This is the second flow chart of the method for detecting target in multimodal remote sensing images of unmanned aerial vehicles based on the hybrid Mamba-CNN network provided by the present invention.

[0042] Figure 3 It is a structural diagram of the cross-modal fusion module provided by the present invention.

[0043] Figure 4 It is a structural schematic diagram of the UAV multimodal remote sensing image target detection device based on the hybrid Mamba-CNN network provided by the present invention.

[0044] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0045] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0046] The terms "first," "second," and the like in the present invention are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type.

[0047] The present application aims at the influence of unmanned aerial vehicle on the shooting distance and imaging mechanism of ground targets, and the problems of small size, dense distribution and low distinction between categories of targets in images. A target detection network based on a four-direction selective scanning fusion strategy, named RemoteDet-Mamba, is proposed.

[0048] Figure 1 is one of the flowcharts of the unmanned aerial vehicle multi-modal remote sensing image target detection method based on the hybrid Mamba-CNN network provided by the present application, Figure 2 is another flowchart of the unmanned aerial vehicle multi-modal remote sensing image target detection method based on the hybrid Mamba-CNN network provided by the present application, as shown in Figure 1 、 Figure 2 The method comprises steps 110 and 120.

[0049] Step 110: Obtain the visible light image and the thermal infrared image to be detected.

[0050] Specifically, the visible light image and the thermal infrared image to be detected can be obtained by a visible light imaging device and a thermal infrared imaging device carried by an unmanned aerial vehicle.

[0051] Step 120: Input the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result output by the target detection network.

[0052] The target detection network is constructed based on a twin feature extraction network, which comprises a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head. The visible light branch network is used to extract visible light local features in the visible light image, the infrared branch network is used to extract infrared local features in the thermal infrared image, the cross-modal fusion module is used to fuse the visible light local features and the infrared local features to obtain multi-scale features, the neck layer is used to obtain global fusion features based on the multi-scale features, and the detection head is used to obtain the remote sensing detection result based on the global fusion features.

[0053] Specifically, after obtaining the visible light image and the thermal infrared image, the visible light image and the thermal infrared image can be input into the target detection network to obtain a remote sensing detection result output by the target detection network.

[0054] The target detection network is constructed based on a twin feature extraction network, which comprises a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head. The visible light branch network is used to extract visible light local features in the visible light image, the infrared branch network is used to extract infrared local features in the thermal infrared image.

[0055] After obtaining the local features of the two modal targets using the twin feature extraction network with a convolutional neural network structure, the subsequent multimodal fusion process is carried out based on the Mamba framework. The Cross Fusion Mamba module (CFM) is used to fuse the visible light local features and the infrared local features to obtain multi-scale features.

[0056] Here, there are multiple cross-modal fusion modules in the target detection network, each of which can include a normalization layer, a linear layer, a depth-separable convolutional layer, an activation layer, a scan expansion and fusion layer, a selective state space model, a scan merging layer, a first residual layer, a feedforward neural network and a second residual layer.

[0057] The neck layer (Neck) is used to obtain global fusion features based on multi-scale features, and the detection head is used to obtain remote sensing detection results based on global fusion features.

[0058] In addition, the target detection network also includes a spatial pyramid pooling module (Spatial Pyramid Pooling Fast Feature Map, SPFF) and a residual module C3.

[0059] It can be understood that the twin convolutional neural network can extract multi-scale features from two multimodal images, such as visible light images and thermal infrared images. The cross-modal fusion module is located between the two modal feature extraction networks, and can realize the global feature fusion of deep local feature blocks between the two modalities. The final global fusion feature is composed of the integrated output of the cross-modal fusion module, the visible light branch network and the infrared branch network, thereby enhancing the accuracy and robustness of multimodal target detection.

[0060] It should be noted that the fusion of learning unimodal local features and cross-modal global features effectively enhances the discriminability of small objects and improves the ability to distinguish between different categories by leveraging local information. In addition, Mamba's serial processing mechanism significantly improves detection speed.

[0061] The method provided by the embodiment of the application acquires a visible light image and a thermal infrared image to be detected, inputs the visible light image and the thermal infrared image into a target detection network, and obtains a remote sensing detection result output by the target detection network. The target detection network is constructed based on a twin feature extraction network. The twin feature extraction network includes a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer, and a detection head. The cross-modal fusion module is used for fusing visible light local features and infrared local features to obtain multi-scale features. The neck layer is used for obtaining global fusion features based on the multi-scale features. The detection head is used for obtaining the remote sensing detection result based on the global fusion features. This process effectively enhances the discriminability of small targets through learning of single-modal local features and fusion of cross-modal global features, improves the distinguishing ability between different categories through local information, and further enhances the precision and robustness of multi-modal target detection.

[0062] The embodiment of the application aims to solve the problem of multi-modal unmanned aerial vehicle target detection of remote sensing images with high-density targets and complex scenes, and provides a new detection framework RemoteDet-Mamba. The framework integrates the complementary advantages of CNN and Mamba architecture, has multi-scale local feature representation capability of CNN, and has linear complexity and global receptive field provided by the Mamba architecture.

[0063] Based on the above embodiment, the cross-modal fusion module includes a first cross-modal fusion layer, a second cross-modal fusion layer, and a third cross-modal fusion layer.

[0064] The first cross-modal fusion layer is used for fusing first visible light local features and first infrared local features to obtain first multi-scale features. The first visible light local features are output by a second layer feature extraction layer of the visible light branch network. The first infrared local features are output by a second layer feature extraction layer of the infrared branch network.

[0065] The second cross-modal fusion layer is used for fusing second visible light local features and second infrared local features to obtain second multi-scale features. The second visible light local features are output by a third layer feature extraction layer of the visible light branch network. The second infrared local features are output by a third layer feature extraction layer of the infrared branch network.

[0066] The third cross-modal fusion layer is used for fusing third visible light local features and third infrared local features to obtain third multi-scale features. The third visible light local features are output by a fifth layer feature extraction layer of the visible light branch network. The third infrared local features are output by a fifth layer feature extraction layer of the infrared branch network.

[0067] Correspondingly, the neck layer is configured to obtain the global fusion feature based on the first multi-scale feature, the second multi-scale feature and the third multi-scale feature.

[0068] Specifically, the cross-modal fusion module can include four cross-modal fusion layers, and the first cross-modal fusion layer, the second cross-modal fusion layer and the third cross-modal fusion layer are needed in the embodiment of the application.

[0069] Taking two modal inputs as an example, the input image is defined as where S = 1, 2 respectively represent two different modalities. These images are first sent to the convolutional neural network block in the target detection network to extract their multi-scale features:

[0070]

[0071] wherein, represents the feature extracted by the i-th layer, and respectively represent the local features of the two modalities. In order to perform multi-scale feature extraction, the input of the neck part contains three parts, which are the fusion outputs of the second, third and fifth layers of the twin feature extraction network, and can be represented as .

[0072]

[0073] In the neck part, multi-scale feature fusion is performed to generate which will be sent to the detection head to generate the final remote sensing detection result.

[0074] That is, the first visible light local feature is the output of the second layer feature extraction layer of the visible light branch network, the first infrared local feature is the output of the second layer feature extraction layer of the infrared branch network, the second visible light local feature is the output of the third layer feature extraction layer of the visible light branch network, the second infrared local feature is the output of the third layer feature extraction layer of the infrared branch network, the third visible light local feature is the output of the fifth layer feature extraction layer of the visible light branch network, and the third infrared local feature is the output of the fifth layer feature extraction layer of the infrared branch network.

[0075] Correspondingly, the neck layer is configured to obtain the global fusion feature based on the first multi-scale feature, the second multi-scale feature and the third multi-scale feature.

[0076] The feature fusion of the first multi-scale feature, the second multi-scale feature and the third multi-scale feature can be splicing the first multi-scale feature, the second multi-scale feature and the third multi-scale feature, or can be splicing the first multi-scale feature, the second multi-scale feature and the third multi-scale feature after weighting by using an attention mechanism, and the embodiment of the application does not make a specific limitation.

[0077] Based on the above embodiment, Figure 3 is a structural schematic diagram of a cross-modal fusion module provided by the application, as Figure 3 shown, the cross-modal fusion module comprises a normalization layer, a linear layer, a depth separable convolution layer, an activation layer, a scanning expansion and fusion layer, a selective state space model, a scanning merging layer, a first residual layer and a second residual layer.

[0078] The scanning expansion and fusion layer is used for respectively flattening the visible light feature and the infrared feature output by the depth separable convolution layer in four directions to obtain one-dimensional visible light features and one-dimensional infrared features, and performing region-level feature fusion on the one-dimensional visible light features and the one-dimensional infrared features to obtain a plurality of one-dimensional fusion sequences.

[0079] The selective state space model is used for respectively performing feature extraction on the plurality of one-dimensional fusion sequences to obtain a plurality of one-dimensional fusion features.

[0080] The scanning merging layer is used for combining the plurality of one-dimensional fusion features, and projecting the combined features to the size of the original input feature space to obtain a merged feature.

[0081] The first residual layer is used for obtaining a first complementary feature based on the merged feature and the visible light local feature, and obtaining a second complementary feature based on the merged feature and the infrared local feature.

[0082] The second residual layer is used for obtaining a multi-scale first feature based on the first complementary feature, and obtaining a multi-scale second feature based on the second complementary feature.

[0083] Specifically, after obtaining the local features of the two modal targets by using a convolutional neural network structure of a twin feature extraction network, a cross-modal fusion module based on a Mamba framework is used for subsequent multi-modal fusion process. Figure 3 is a structural schematic diagram of a cross-modal fusion module provided by the application, as Figure 3 shown. It is assumed that , respectively represent the local features of the visible light and the thermal infrared modal, and a LayerNorm layer is used for normalizing the input features to accelerate model convergence.

[0084]

[0085] in, Represents the LayerNorm operation (normalization layer), Denotes a projection operation using linear transformation (linear layer). Depthwise separable convolution is applied to the two modal features to facilitate communication between channels:

[0086]

[0087] in, represents a depth-wise separable convolution operation, Represents the SiLU activation function. The Mamba architecture uses a unique selective scanning mechanism to adjust parameters according to the input data, thereby achieving selective scanning fusion processing of two modal features. Like VMamba, and Flatten in four directions (up, down, left and right) to generate 1D sequences, each of size HW×C, and perform region-level feature fusion in an element-wise addition manner:

[0088]

[0089] in, represents the scanning operation along the i-th direction.

[0090] The selective state-space model processes these one-dimensional fusion sequences individually for feature extraction, producing four outputs, denoted as y1, y2, y3, and y4.

[0091]

[0092] in, represents the i-th selective state space model (S6 block). The scan-merge layer is used to unfold the output of the S6 block and recombine it into , and project it back to the original input feature space size, that is:

[0093]

[0094]

[0095] in, represents the merged features, Indicates the flattening operation along the i-th direction. The original input features of the two modalities are passed through the residual connection to obtain the first complementary features and the second complementary feature :

[0096]

[0097] Finally, the complementary features are respectively processed by layer normalization operation and feedforward neural network, and then the final output is obtained after residual connection with the complementary features.

[0098]

[0099]

[0100] wherein, GELU is an activation function, W1, W2, b1 and b2 are weights and biases of linear projection.

[0101] In terms of multi-modal fusion, the CFM module based on the Mamba architecture is constructed, a comprehensive global scan is performed from four directions through a selective scanning mechanism, dense detection objects are effectively isolated, and fine and targeted feature fusion is promoted.

[0102] Further, the framework of the target detection network of the application mainly consists of a twin convolutional neural network and a cross-modal fusion Mamba module. First, multi-scale local information is extracted by a twin network encoder, and the CFM module uses the design of a 2D-Selective-Scan for Vision Data (SS2D) based on Mamba selective scanning two-dimensional mechanism to scan the extracted multi-scale features in four directions, realizing decoupling of dense targets and selective feature fusion. The linear scanning strategy of the embodiment has a linear time complexity, can capture long-range dependencies at the local feature block level, and significantly improves the accuracy and efficiency of target detection.

[0103] Based on the above embodiment, the training steps of the target detection network include:

[0104] Step 210, acquiring a sample visible light image and a sample thermal infrared image, and a remote sensing data set label corresponding to the sample visible light image and the sample thermal infrared image, and determining an initial target detection network;

[0105] Step 220, inputting the sample visible light image and the sample thermal infrared image into the initial target detection network to obtain a predicted remote sensing detection result output by the initial target detection network;

[0106] Step 230, based on the difference between the remote sensing data set label and the predicted remote sensing detection result, iteratively adjusting the parameters of the initial target detection network to obtain the target detection network.

[0107] Specifically, in order to better obtain the target detection network and improve the accuracy and reliability of target detection, the training can be based on the following steps:

[0108] First, a sample visible light image and a sample thermal infrared image are acquired, as well as a sample visible light image and a sample thermal infrared image corresponding remote sensing dataset label, and an initial target detection network is determined.

[0109] The parameters of the initial target detection network here can be randomly generated or pre-set, and embodiments of the present application do not make specific limitations thereto.

[0110] Then, the sample visible light image and the sample thermal infrared image are input into the initial target detection network to obtain a predicted remote sensing detection result output by the initial target detection network.

[0111] After obtaining the predicted remote sensing detection result based on the initial target detection network, the initial target detection network can be iterated in parameters based on the difference between the remote sensing dataset label and the predicted remote sensing detection result, and the initial target detection network after the parameter iteration is taken as the target detection network.

[0112] It can be understood that the greater the difference between the remote sensing dataset label and the predicted remote sensing detection result, the greater the target loss; the smaller the difference between the remote sensing dataset label and the predicted remote sensing detection result, the smaller the target loss.

[0113] Based on the above embodiments, the step 230 of iterating the initial target detection network in parameters based on the difference between the remote sensing dataset label and the predicted remote sensing detection result comprises:

[0114] Step 231, based on the difference between the class in the remote sensing dataset label and the class in the predicted remote sensing detection result, determine the classification loss and the confidence loss;

[0115] Step 232, based on the classification loss and the confidence loss, determine the target loss, and based on the target loss, iterate the initial target detection network in parameters.

[0116] Specifically, based on the difference between the class in the remote sensing dataset label and the class in the predicted remote sensing detection result, the classification loss and the confidence loss are determined.

[0117] Wherein, , are both based on Smooth BCE, to enhance numerical stability, and the latter adds CIoU (Complete-IOU) loss between horizontal boundaries to accelerate the training process.

[0118] After obtaining the classification loss and the confidence loss, a target loss can be determined based on a sum or a weighted sum of the classification loss and the confidence loss, and the initial target detection network can be iterated based on the target loss.

[0119] Based on the above embodiments, determining the target loss based on the classification loss and the confidence loss in step 232 includes:

[0120] In step 2321, a bounding box regression loss is determined based on a difference between the real box coordinates in the remote sensing dataset label and the predicted box coordinates in the predicted remote sensing detection result.

[0121] In step 2322, the target loss is determined based on the classification loss, the confidence loss, and the bounding box regression loss.

[0122] Specifically, the bounding box regression loss can be determined based on a difference between the real box coordinates in the remote sensing dataset label and the predicted box coordinates in the predicted remote sensing detection result. .

[0123] It can be understood that the greater the difference between the real box coordinates in the remote sensing dataset label and the predicted box coordinates in the predicted remote sensing detection result, the greater the bounding box regression loss; the smaller the difference between the real box coordinates in the remote sensing dataset label and the predicted box coordinates in the predicted remote sensing detection result, the smaller the bounding box regression loss.

[0124] Finally, the target loss can be determined based on the classification loss, the confidence loss, and the bounding box regression loss, and the formula is as follows:

[0125]

[0126] wherein, represents the target loss, represents the bounding box regression loss, represents the confidence loss, represents the classification loss, Complete-IOU (CIoU) is adopted.

[0127] The unmanned aerial vehicle multi-modal remote sensing image target detection device based on the hybrid Mamba-CNN network provided by the present application will be described below. The unmanned aerial vehicle multi-modal remote sensing image target detection device described below can be correspondingly referred to the unmanned aerial vehicle multi-modal remote sensing image target detection method described above.

[0128] Based on any of the above embodiments, the present application provides an unmanned aerial vehicle multi-modal remote sensing image target detection device based on a hybrid Mamba-CNN network,Figure 4 : is a structural diagram of a multimodal remote sensing image target detection device for UAV based on a hybrid Mamba-CNN network provided by the present invention, such as Figure 4 As shown, the device includes:

[0129] An acquisition unit 410 is configured to acquire a visible light image and a thermal infrared image to be detected;

[0130] An input unit 420 is configured to input the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result output by the target detection network;

[0131] The target detection network is constructed based on a twin feature extraction network, which includes a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head. The visible light branch network is used to extract visible light local features in the visible light image, and the infrared branch network is used to extract infrared local features in the thermal infrared image. The cross-modal fusion module is used to fuse the visible light local features and the infrared local features to obtain multi-scale features. The neck layer is used to obtain global fusion features based on the multi-scale features. The detection head is used to obtain the remote sensing detection results based on the global fusion features.

[0132] The device provided by the embodiment of the present invention obtains the visible light image and thermal infrared image to be detected, inputs the visible light image and thermal infrared image into the target detection network, and obtains the remote sensing detection result output by the target detection network. The target detection network is constructed based on the twin feature extraction network. The twin feature extraction network includes a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head. The cross-modal fusion module is used to fuse the visible light local features and the infrared local features to obtain multi-scale features. The neck layer is used to obtain global fusion features based on the multi-scale features. The detection head is used to obtain remote sensing detection results based on the global fusion features. The learning of single-modal local features and the fusion of cross-modal global features in this process effectively enhance the discriminability of small targets, and improves the ability to distinguish between different categories by using local information, further enhancing the accuracy and robustness of multi-modal target detection.

[0133] Based on any of the above embodiments, the cross-modal fusion module includes a first cross-modal fusion layer, a second cross-modal fusion layer, and a third cross-modal fusion layer;

[0134] The first cross-modal fusion layer is used to fuse the first visible light local feature and the first infrared local feature to obtain a first multi-scale feature; the first visible light local feature is output by the second feature extraction layer of the visible light branch network; the first infrared local feature is output by the second feature extraction layer of the infrared branch network;

[0135] The second cross-modal fusion layer is configured to fuse the second visible light local feature and the second infrared local feature to obtain a second multi-scale feature; the second visible light local feature is output by a third layer feature extraction layer of the visible light branch network; and the second infrared local feature is output by a third layer feature extraction layer of the infrared branch network.

[0136] The third cross-modal fusion layer is configured to fuse the third visible light local feature and the third infrared local feature to obtain a third multi-scale feature; the third visible light local feature is output by a fifth layer feature extraction layer of the visible light branch network; and the third infrared local feature is output by a fifth layer feature extraction layer of the infrared branch network.

[0137] Correspondingly, the neck layer is configured to obtain the global fusion feature based on the first multi-scale feature, the second multi-scale feature and the third multi-scale feature.

[0138] According to any one of the above embodiments, the cross-modal fusion module comprises a normalization layer, a linear layer, a depth separable convolution layer, an activation layer, a scan expansion and fusion layer, a selective state space model, a scan merging layer, a first residual layer and a second residual layer.

[0139] The scan expansion and fusion layer is configured to flatten the visible light feature and the infrared feature output by the depth separable convolution layer in four directions respectively to obtain one-dimensional visible light features and one-dimensional infrared features, and perform region-level feature fusion on the one-dimensional visible light features and the one-dimensional infrared features to obtain a plurality of one-dimensional fusion sequences.

[0140] The selective state space model is configured to perform feature extraction on the plurality of one-dimensional fusion sequences respectively to obtain a plurality of one-dimensional fusion features.

[0141] The scan merging layer is configured to combine the plurality of one-dimensional fusion features, and project the combined features to the size of the original input feature space to obtain a merged feature.

[0142] The first residual layer is configured to obtain a first complementary feature based on the merged feature and the visible light local feature, and obtain a second complementary feature based on the merged feature and the infrared local feature.

[0143] The second residual layer is configured to obtain a multi-scale first feature based on the first complementary feature, and obtain a multi-scale second feature based on the second complementary feature.

[0144] According to any one of the above embodiments, the method further comprises a training unit, and the training unit is specifically configured to:

[0145] An acquisition unit is configured to acquire a sample visible light image and a sample thermal infrared image, and a remote sensing dataset label corresponding to the sample visible light image and the sample thermal infrared image, and determine an initial target detection network;

[0146] An input unit is configured to input the sample visible light image and the sample thermal infrared image into the initial target detection network to obtain a predicted remote sensing detection result output by the initial target detection network.

[0147] A parameter iteration unit is configured to perform parameter iteration on the initial target detection network based on a difference between the remote sensing dataset label and the predicted remote sensing detection result, to obtain the target detection network.

[0148] According to any one of the above embodiments, the parameter iteration unit is specifically configured to:

[0149] A determination unit is configured to determine a classification loss and a confidence loss based on a difference between a category in the remote sensing dataset label and a category in the predicted remote sensing detection result.

[0150] A parameter iteration sub-unit is configured to determine a target loss based on the classification loss and the confidence loss, and perform parameter iteration on the initial target detection network based on the target loss.

[0151] According to any one of the above embodiments, the parameter iteration sub-unit is specifically configured to:

[0152] A bounding box regression loss is determined based on a difference between a real box coordinate in the remote sensing dataset label and a predicted box coordinate in the predicted remote sensing detection result.

[0153] The target loss is determined based on the classification loss, the confidence loss, and the bounding box regression loss.

[0154] Figure 5 is a structural schematic diagram of an electronic device provided by the present application, as Figure 5As shown, the electronic device can include a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 complete mutual communication through the communications bus 540. The processor 510 can invoke a logic instruction in the memory 530 to execute a multi-modal remote sensing image target detection method for a drone based on a hybrid Mamba-CNN network, the method comprising: acquiring a visible light image and a thermal infrared image to be detected; inputting the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result output by the target detection network; the target detection network is constructed based on a twin feature extraction network, the twin feature extraction network comprising a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer, and a detection head, the visible light branch network is used to extract visible light local features in the visible light image, the infrared branch network is used to extract infrared local features in the thermal infrared image, the cross-modal fusion module is used to fuse the visible light local features and the infrared local features to obtain multi-scale features, the neck layer is used to obtain global fusion features based on the multi-scale features, and the detection head is used to obtain the remote sensing detection result based on the global fusion features.

[0155] In addition, the logic instruction in the memory 530 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0156] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program being stored on a non-transitory computer-readable storage medium, and the computer program being executable by a processor to enable a computer to perform the method of target detection based on a hybrid Mamba-CNN network for multi-modal remote sensing images of a UAV, the method comprising: obtaining a visible light image and a thermal infrared image to be detected; inputting the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result output by the target detection network; the target detection network being constructed based on a twin feature extraction network, the twin feature extraction network comprising a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer, and a detection head, the visible light branch network being configured to extract visible light local features in the visible light image, the infrared branch network being configured to extract infrared local features in the thermal infrared image, the cross-modal fusion module being configured to fuse the visible light local features and the infrared local features to obtain multi-scale features, the neck layer being configured to obtain global fusion features based on the multi-scale features, and the detection head being configured to obtain the remote sensing detection result based on the global fusion features.

[0157] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement a method of target detection based on a hybrid Mamba-CNN network for multi-modal remote sensing images of a UAV, the method comprising: obtaining a visible light image and a thermal infrared image to be detected; inputting the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result output by the target detection network; the target detection network being constructed based on a twin feature extraction network, the twin feature extraction network comprising a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer, and a detection head, the visible light branch network being configured to extract visible light local features in the visible light image, the infrared branch network being configured to extract infrared local features in the thermal infrared image, the cross-modal fusion module being configured to fuse the visible light local features and the infrared local features to obtain multi-scale features, the neck layer being configured to obtain global fusion features based on the multi-scale features, and the detection head being configured to obtain the remote sensing detection result based on the global fusion features.

[0158] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0159] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0160] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for unmanned aerial vehicle multi-modal remote sensing image target detection based on a hybrid Mamba-CNN network, characterized in that, The method comprises: acquiring a visible light image and a thermal infrared image to be detected; inputting the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result output by the target detection network; the target detection network is constructed based on a twin feature extraction network, and the twin feature extraction network comprises a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer and a detection head; the visible light branch network is used to extract visible light local features in the visible light image; the infrared branch network is used to extract infrared local features in the thermal infrared image; the cross-modal fusion module is used to fuse the visible light local features and the infrared local features to obtain multi-scale features; the neck layer is used to obtain global fusion features based on the multi-scale features; and the detection head is used to obtain the remote sensing detection result based on the global fusion features; the cross-modal fusion module comprises a first cross-modal fusion layer, a second cross-modal fusion layer and a third cross-modal fusion layer; the first cross-modal fusion layer is used to fuse first visible light local features and first infrared local features to obtain first multi-scale features; the first visible light local features are output by a second layer feature extraction layer of the visible light branch network; and the first infrared local features are output by a second layer feature extraction layer of the infrared branch network; the second cross-modal fusion layer is used to fuse second visible light local features and second infrared local features to obtain second multi-scale features; the second visible light local features are output by a third layer feature extraction layer of the visible light branch network; and the second infrared local features are output by a third layer feature extraction layer of the infrared branch network; the third cross-modal fusion layer is used to fuse third visible light local features and third infrared local features to obtain third multi-scale features; the third visible light local features are output by a fifth layer feature extraction layer of the visible light branch network; and the third infrared local features are output by a fifth layer feature extraction layer of the infrared branch network; correspondingly, the neck layer is used to obtain the global fusion features based on the first multi-scale features, the second multi-scale features and the third multi-scale features; the cross-modal fusion module comprises a normalization layer, a linear layer, a depth separable convolution layer, an activation layer, a scan expansion and fusion layer, a selective state space model, a scan merging layer, a first residual layer and a second residual layer; the scan expansion and fusion layer is used to flatten one-dimensional visible light features and one-dimensional infrared features in four directions respectively based on visible light features and infrared features output by the depth separable convolution layer, to obtain the one-dimensional visible light features and the one-dimensional infrared features, and to perform regional-level feature fusion on the one-dimensional visible light features and the one-dimensional infrared features to obtain a plurality of one-dimensional fusion sequences; the selective state space model is used to perform feature extraction on the plurality of one-dimensional fusion sequences respectively to obtain a plurality of one-dimensional fusion features; the scan merging layer is used to combine the plurality of one-dimensional fusion features, and project the combined features to the size of the original input feature space to obtain merged features; and The first residual layer is configured to obtain a first complementary feature based on the merged feature and the visible light local feature, and obtain a second complementary feature based on the merged feature and the infrared local feature; The second residual layer is configured to obtain a multi-scale first feature based on the first complementary feature, and obtain a multi-scale second feature based on the second complementary feature.

2. The unmanned aerial vehicle multi-modal remote sensing image target detection method based on the hybrid Mamba-CNN network according to claim 1, characterized in that, The training step of the target detection network comprises: obtaining a sample visible light image and a sample thermal infrared image, and a remote sensing dataset label corresponding to the sample visible light image and the sample thermal infrared image, and determining an initial target detection network; inputting the sample visible light image and the sample thermal infrared image into the initial target detection network to obtain a predicted remote sensing detection result output by the initial target detection network; performing parameter iteration on the initial target detection network based on a difference between the remote sensing dataset label and the predicted remote sensing detection result to obtain the target detection network.

3. The unmanned aerial vehicle multi-modal remote sensing image target detection method based on the hybrid Mamba-CNN network according to claim 2, characterized in that, The parameter iteration on the initial target detection network based on the difference between the remote sensing dataset label and the predicted remote sensing detection result comprises: determining a classification loss and a confidence loss based on a difference between a category in the remote sensing dataset label and a category in the predicted remote sensing detection result; determining a target loss based on the classification loss and the confidence loss, and performing parameter iteration on the initial target detection network based on the target loss.

4. The unmanned aerial vehicle multi-modal remote sensing image target detection method based on the hybrid Mamba-CNN network according to claim 3, characterized in that, The determination of the target loss based on the classification loss and the confidence loss comprises: determining a bounding box regression loss based on a difference between a real box coordinate in the remote sensing dataset label and a predicted box coordinate in the predicted remote sensing detection result; determining the target loss based on the classification loss, the confidence loss, and the bounding box regression loss.

5. An unmanned aerial vehicle multi-modal remote sensing image target detection device based on a hybrid Mamba-CNN network, characterized in that, comprises: an acquisition unit configured to acquire a visible light image and a thermal infrared image to be detected; an input unit configured to input the visible light image and the thermal infrared image into a target detection network to obtain a remote sensing detection result output by the target detection network; The target detection network is constructed based on a twin feature extraction network, the twin feature extraction network comprises a visible light branch network, an infrared branch network, a cross-modal fusion module, a neck layer, and a detection head, the visible light branch network is configured to extract a visible light local feature in the visible light image, the infrared branch network is configured to extract an infrared local feature in the thermal infrared image, the cross-modal fusion module is configured to fuse the visible light local feature and the infrared local feature to obtain a multi-scale feature, the neck layer is configured to obtain a global fusion feature based on the multi-scale feature, and the detection head is configured to obtain the remote sensing detection result based on the global fusion feature; The cross-modal fusion module comprises a first cross-modal fusion layer, a second cross-modal fusion layer, and a third cross-modal fusion layer; The first cross-modal fusion layer is configured to fuse the first visible light local feature and the first infrared local feature to obtain a first multi-scale feature, the first visible light local feature being output by a second layer feature extraction layer of the visible light branch network, and the first infrared local feature being output by a second layer feature extraction layer of the infrared branch network; The second cross-modal fusion layer is configured to fuse the second visible light local feature and the second infrared local feature to obtain a second multi-scale feature, the second visible light local feature being output by a third layer feature extraction layer of the visible light branch network, and the second infrared local feature being output by a third layer feature extraction layer of the infrared branch network; The third cross-modal fusion layer is configured to fuse the third visible light local feature and the third infrared local feature to obtain a third multi-scale feature, the third visible light local feature being output by a fifth layer feature extraction layer of the visible light branch network, and the third infrared local feature being output by a fifth layer feature extraction layer of the infrared branch network; The neck layer is configured to obtain the global fusion feature based on the first multi-scale feature, the second multi-scale feature, and the third multi-scale feature; The cross-modal fusion module comprises a normalization layer, a linear layer, a depth separable convolution layer, an activation layer, a scan expansion and fusion layer, a selective state space model, a scan merging layer, a first residual layer, and a second residual layer; The scan expansion and fusion layer is configured to flatten the visible light feature and the infrared feature output by the depth separable convolution layer in four directions respectively to obtain one-dimensional visible light features and one-dimensional infrared features, and perform region-level feature fusion on the one-dimensional visible light features and the one-dimensional infrared features to obtain a plurality of one-dimensional fusion sequences; The selective state space model is configured to perform feature extraction on the plurality of one-dimensional fusion sequences respectively to obtain a plurality of one-dimensional fusion features; The scan merging layer is configured to combine the plurality of one-dimensional fusion features, and project the combined features to the size of the original input feature space to obtain a merged feature; The first residual layer is configured to obtain a first complementary feature based on the merged feature and the visible light local feature, and obtain a second complementary feature based on the merged feature and the infrared local feature; The second residual layer is configured to obtain a multi-scale first feature based on the first complementary feature, and obtain a multi-scale second feature based on the second complementary feature.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method for detecting a target in a multi-modal remote sensing image of a UAV based on a hybrid Mamba-CNN network according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method for detecting a target in a multi-modal remote sensing image of a UAV based on a hybrid Mamba-CNN network according to any one of claims 1 to 4.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method for detecting a target in a multi-modal remote sensing image of a UAV based on a hybrid Mamba-CNN network according to any one of claims 1 to 4.

Images