A multispectral remote sensing image target detection method and system based on complementary mask

Abstract

The application discloses a multispectral remote sensing image target detection method and system based on complementary mask, and belongs to the technical field of remote sensing image target detection, which solves the problems of low target recognition accuracy of existing detection methods and insufficient exploration of inter-modal difference information, etc. Based on the complementary mask cross attention mechanism and the differential complementary enhancement module, the application realizes effective interaction between different modal information, highlights the difference features between different modalities, and through the modal fusion module, utilizes the consistency information between different modalities to perform feature-level fusion on the two different modal remote sensing images, thereby improving the detection performance of the new model. The application can solve the problem of double-modal input target detection and can be applied to practical scenes such as all-weather monitoring, multi-target tracking, defect detection and quality control. Compared with the baseline method, the mean average precision index of the application increases, has less training cost, and is convenient for deployment on hardware.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image target detection technology, and more specifically, relates to a multispectral remote sensing image target detection method and system based on complementary masks. Background Technology

[0002] Currently, remote sensing image acquisition capabilities are continuously improving with the development of information technology, resulting in a large amount of data with different spatiotemporal resolutions. This data has been widely used in various downstream tasks. Among numerous remote sensing image processing tasks, multimodal remote sensing image target detection technology occupies a key position and is an important link in processing and applying remote sensing data. With the development of remote sensing technology, the data exhibits diverse characteristics, covering multiple modalities such as visible light, infrared, and radar, providing multi-dimensional information support for target detection.

[0003] Different modalities of remote sensing images carry unique information. For example, visible light images can intuitively show the color and texture features of ground objects, while infrared images are sensitive to temperature differences and can detect concealed targets or have an advantage at night. Radar images can penetrate clouds and fog, providing structural information about terrain features. In the field of target detection using multimodal remote sensing images, accurately and timely detecting small targets at multiple scales is a major challenge.

[0004] However, due to the large scale variation of targets, complex backgrounds, and limited number of labeled samples in remote sensing images, traditional single-modality target detection technology is difficult to meet the requirements. Although multimodal data fusion technology has the advantage of integrating different modal data compared to single-modality technology, existing multimodal fusion networks have low target recognition accuracy in remote sensing image target detection tasks and cannot fully explore the differences between modalities. Summary of the Invention

[0005] This application addresses the technical problems existing in the prior art by providing a method and system for target detection in multispectral remote sensing images based on complementary masks.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] This invention first provides a multispectral remote sensing image target detection method based on complementary masks, comprising the following steps:

[0008] S100. Construct a multispectral remote sensing image dataset and perform preprocessing, dividing the dataset into a training set, a validation set, and a test set;

[0009] S200. Construct a dual-branch feature extraction network suitable for dual-modal input, including a dual-stream feature extraction backbone network and a feature fusion layer, to extract features at all levels of remote sensing images of different modalities;

[0010] S300: Obtain image features at all levels, and send the feature maps of corresponding levels of the two modal remote sensing images into the feature interaction mask enhancement module to extract consistency and complementarity information between different modal features, thereby realizing the fusion of semantic information between different modal remote sensing images;

[0011] S400: The final fused feature map is obtained and fed into the feature fusion layer and the detection head step by step to realize the detection and recognition of each target in the multispectral remote sensing image.

[0012] Preferably, step S100 includes:

[0013] The system matches remote sensing images of different modalities and aligns remote sensing images of different modalities with target labels in corresponding scenes to construct a multispectral remote sensing image dataset adapted to a dual-branch feature extraction network. The dataset is then randomly divided into training, validation, and test sets.

[0014] Preferably, step S200 includes:

[0015] Based on the YOLOv11 feature extraction network, a bimodal feature extraction network branch suitable for modality 1 and modality 2 is constructed. The two branches have the same structure, are independent of each other, and do not affect each other. The two different modalities of remote sensing images are convolved and sampled step by step to extract multi-level features. The extracted three-level feature maps are then sent to the subsequent feature enhancement module.

[0016] Preferably, in step S300:

[0017] The feature interaction mask enhancement module includes a differential complementary enhancement module, which uses two different modal feature maps extracted from the feature extraction backbone network to obtain differential information, supplement the semantic information lacking in a single modality, and realize semantic information interaction enhancement between remote sensing images of different modalities.

[0018] Preferably, the differential complementary enhancement module includes:

[0019] S311. Feature extraction downsampling: First, convolution operations are performed on feature maps of different modalities at the same level using convolution kernels of different scales to extract semantic information at different scales. Then, pixel-by-pixel summation is performed on features of different scales within a modality to achieve the fusion of semantic information at different scales.

[0020]

[0021]

[0022]

[0023]

[0024]

[0025] In the formula, and These are the feature maps for mode 1 and mode 2, respectively. and These are convolution operations with different kernel sizes. and These are the results of convolution operations on the modality 1 feature map using different convolution kernels. and These are the results of convolution operations on modality 2 feature maps using different convolution kernels. and These are the feature maps of modality 1 and modality 2 after convolutional sampling and semantic scale fusion, respectively.

[0026] S312. Differential weighted activation: Subtract the feature maps of mode 1 and mode 2 to obtain the differences between the features of the two remote sensing images. Then, perform pooling on the obtained differential features to compress them into channel vectors, with each channel corresponding to a scalar value.

[0027]

[0028]

[0029] In the formula, The result is the element-wise difference calculation of the features of mode 1 and mode 2. For average pooling operation, This is the result after the average pooling operation, representing the difference between the two modal features in each channel;

[0030] S313, Complementary information interaction, multiply the original feature map of mode 1 and feature map of mode 2 with the previously obtained weights at the channel level to enhance the channels that are sensitive to the differences between the two modes, thereby extracting feature information of different importance in the channel dimension based on the differences between modes;

[0031] Then, the weighted complementary feature information is added to the original features, and the difference information between different modalities is interacted to achieve complementary enhancement:

[0032]

[0033]

[0034] In the formula, and The feature information after assigning different weights to the features of mode 1 and mode 2 respectively, and * indicates pixel-wise multiplication between tensor matrices.

[0035] Preferably, in step S300, the feature interaction mask enhancement module includes a mask enhancement module. The mask enhancement module is based on the Transformer method and uses Cross-Attention. On this basis, it uses a smaller feature map and adds a complementary mask module, so that the feature's ability to express single-modal semantic information is enhanced.

[0036] Preferably, the mask enhancement module includes:

[0037] S321. Adaptive feature downsampling: Multi-scale convolutional downsampling operations are performed on the dual-modal features output by the differential complementary enhancement module. Small convolutional kernels are used to extract local information, and large convolutional kernels are used to supplement surrounding semantics. For each modality, the features obtained from convolution at different scales are added pixel-by-pixel with adaptive weights. The weights are gradually learned and updated during training.

[0038]

[0039]

[0040]

[0041]

[0042]

[0043]

[0044] In the formula, and These are the feature maps belonging to mode 1 and mode 2, respectively, output by the differential complementary module. and These are convolution operations with different kernel sizes. and These are the feature maps of modality 1 and modality 2 after convolutional sampling and semantic scale fusion, respectively. and These are the weights used for pixel-level fusion of features from modality 1 and modality 2, respectively.

[0045] S322, Cross-Attention: After obtaining the features after convolutional downsampling, the Cross-Attention module is applied to the features of each modality to map the feature map into a tensor query matrix Q, a key matrix K, and a value matrix V, resulting in an attention map A.

[0046]

[0047]

[0048]

[0049]

[0050]

[0051]

[0052] In the formula, Features are input for mode 1. , , These are the weight matrices for calculating the query matrix Q, the key matrix K, and the value matrix V, respectively, used to map the size of the feature tensor from C×H×W to HW×C, where C, H, and W are the number of channels, height, and width of the input feature map, respectively.

[0053]

[0054]

[0055]

[0056] In the formula, Features are input for mode 2. , , We calculate the weight matrices for the query matrix Q, key matrix K, and value matrix V, respectively. After obtaining the query matrix Q, key matrix K, and value matrix V for the two modalities, we perform cross-calculation to obtain the cross-attention map A.

[0057]

[0058]

[0059] S333, weight mask, to obtain attention maps of two modalities and Afterwards, and A zero-mask is randomly added to the top to eliminate some weight information, resulting in a new attention weight map:

[0060]

[0061]

[0062] In the formula, This indicates a masking operation, which multiplies the final mask attention weights with the original input feature map at the pixel level to obtain the final features output by the complementary masking enhancement module.

[0063] Preferably, step S400 includes:

[0064] The feature fusion module fuses the feature maps after cross-modal feature information interaction, thereby transforming two independent bimodal features into fused modal information to perform the final target detection task.

[0065] First, dimensionality reduction is performed using convolution to extract the spatial feature weights of a single modality. Then, spatial feature weights of modality 1 and modality 2 are obtained through convolution. Finally, feature fusion is performed using the two spatial feature weights.

[0066] Preferably, the feature fusion module includes:

[0067] S411. Obtaining the spatial weights of independent modalities: First, dimensionality reduction is performed on the channel dimension using convolutional kernels of different sizes to obtain semantic information at different scales. Then, the channels are concatenated, and dimensionality reduction is performed again using convolutional operations to obtain the spatial weights of the spatial unimodality.

[0068]

[0069]

[0070]

[0071] In the formula, This is the input of Mode 1 in the feature fusion module. and These are the outputs obtained from convolutions at two different scales. This is a splicing operation along the channel dimension. The weights are extracted using mode 1 alone, while mode 2 contains the weights. It is also obtained in the same way;

[0072] S412. Obtaining the spatial weights of the hybrid modality: Convolutional downsampling is performed on the feature maps of the two modalities respectively. Then, the results of the convolutional sampling of the two modalities are added pixel by pixel. After activation using the sigmoid activation function, the spatial weights of the hybrid modality are obtained. :

[0073]

[0074]

[0075]

[0076] In the formula, This indicates an activation operation. This represents the pixel-by-pixel addition operation between feature tensors;

[0077] S413. Final feature fusion yields the weights. , , Next, the fused modal weights are added to the independent weights, then cross-multiplied with the original feature maps of the different modalities. Finally, a residual link is established with the original feature maps. The weighted features of the two modalities are then concatenated along the channel dimension, and channel dimensionality reduction is achieved through convolution operations to obtain the final features used for detection.

[0078]

[0079]

[0080]

[0081]

[0082] .

[0083] Furthermore, the present invention also provides a multispectral remote sensing image target detection system based on complementary masks, wherein the detection system executes the steps in the above-described multispectral remote sensing image target detection method based on complementary masks during operation.

[0084] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program configured to implement the steps of the above-described multispectral remote sensing image target detection method based on complementary masks when invoked by a processor.

[0085] Compared with the prior art, the present invention has the following beneficial effects:

[0086] This invention provides a method and system for target detection in multispectral remote sensing images based on complementary masks. It proposes a novel deep learning model, constructing a dual-branch feature extraction network suitable for multimodal remote sensing image input, based on the traditional single-branch feature extraction backbone network. The network also boasts high scalability, facilitating the addition of necessary feature enhancement and fusion modules. Furthermore, by feeding feature maps from corresponding levels of the two modalities of remote sensing images into a feature interaction mask enhancement module, it extracts consistency and complementarity information between different modal features, achieving semantic information fusion between different modalities. By considering the spatial distribution of feature maps through intra-modal and inter-modal perspectives, and by calculating weights to highlight regions of significant interest, it enhances the model's ability to acquire information on detailed and global features, better enabling feature fusion between different modalities, thereby strengthening the model's ability to detect targets in remote sensing images. Attached Figure Description

[0087] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0088] Figure 1 This is a flowchart of the detection method provided in an embodiment of the present invention;

[0089] Figure 2 This is a schematic diagram of the complete network structure of the target detection network provided in an embodiment of the present invention;

[0090] Figure 3 This is a schematic diagram of the differential complementary enhancement module network structure of the detection method provided in the embodiments of the present invention;

[0091] Figure 4 A schematic diagram of the modal fusion module network structure of the detection method provided in this embodiment of the invention;

[0092] Figure 5 Example diagram of the detection results of the detection method provided in the embodiments of the present invention;

[0093] Figure 6 This is a comparison chart of the detection results of the detection method provided in this embodiment of the invention with those of other methods. Detailed Implementation

[0094] To make the technical problems, solutions, and beneficial effects of this application clearer, the following detailed description of a multispectral remote sensing image target detection method and system based on complementary masks, in conjunction with the accompanying drawings and embodiments, is provided. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0095] Example 1

[0096] like Figure 1 As shown, this embodiment of the invention provides a multispectral remote sensing image target detection method based on complementary masks, including the following steps:

[0097] S100. Construct a multispectral remote sensing image dataset and perform preprocessing, dividing the dataset into a training set, a validation set, and a test set;

[0098] S200. Construct a dual-branch feature extraction network suitable for dual-modal input, including a dual-stream feature extraction backbone network and a feature fusion layer, to extract features at all levels of remote sensing images of different modalities;

[0099] S300: Obtain image features at all levels, and send the feature maps of corresponding levels of the two modal remote sensing images into the feature interaction mask enhancement module to extract consistency and complementarity information between different modal features, thereby realizing the fusion of semantic information between different modal remote sensing images;

[0100] S400: The final fused feature map is obtained and fed into the feature fusion layer and the detection head step by step to realize the detection and recognition of each target in the multispectral remote sensing image.

[0101] This invention proposes a novel deep learning model. Based on the traditional single-branch feature extraction backbone network, a dual-branch feature extraction network suitable for multimodal remote sensing image input is constructed. The network also boasts high scalability, facilitating the addition of necessary feature enhancement and fusion modules. Furthermore, by feeding feature maps from corresponding layers of the two modalities of remote sensing images into a feature interaction mask enhancement module, consistency and complementarity information between different modalities are extracted, achieving semantic information fusion between different modalities. The spatial distribution of feature maps is considered through intra-modal and inter-modal perspectives, and regions of significant interest are highlighted by calculating weights. This enhances the model's ability to acquire information on both detailed and global features, enabling better feature fusion between different modalities and thus improving the model's ability to detect targets in remote sensing images.

[0102] Example 2

[0103] like Figure 1 As shown, this embodiment of the invention provides a multispectral remote sensing image target detection method based on complementary masks, including the following steps:

[0104] S100. Construct a multispectral remote sensing image dataset and preprocess it, then divide the dataset into training, validation, and test sets by random sampling.

[0105] Specifically, this step matches remote sensing images of different modalities and aligns remote sensing images of different modalities in the corresponding scene with the target label to construct a multispectral remote sensing image dataset adapted to the dual-branch feature extraction network. The dataset is then randomly divided into training, validation, and test sets.

[0106] S200. Construct a dual-branch feature extraction network suitable for dual-modal input, including a dual-stream feature extraction backbone network and a feature fusion layer, to extract features at various levels from remote sensing images of different modalities.

[0107] Specifically, this step includes: a feature extraction network based on YOLOv11, such as... Figure 2As shown, a dual-modal feature extraction network branch suitable for modality 1 and modality 2 is constructed. The two branches have the same structure, are independent of each other, and do not affect each other. The network performs convolution and sampling on the two different modalities of the input remote sensing images step by step to extract multi-level features. The extracted three-level feature maps are then sent to the subsequent feature enhancement module.

[0108] S300: Obtain image features at all levels, and send the feature maps of corresponding levels of the two modal remote sensing images into the feature interaction mask enhancement module to extract consistency and complementarity information between different modal features, thereby realizing the fusion of semantic information between different modal remote sensing images.

[0109] Specifically, this step includes:

[0110] S310, the feature interaction mask enhancement module consists of two parts. The first part includes a differential complementary enhancement module, such as... Figure 3 As shown, it utilizes two different modal feature maps extracted from the feature extraction backbone network to obtain differential information, supplementing the semantic information lacking in a single modality, and realizing the semantic information interaction enhancement between remote sensing images of different modalities.

[0111] Specifically, it includes:

[0112] S311. Feature extraction downsampling: First, convolution operations are performed on feature maps of different modalities at the same level using convolution kernels of different scales to extract semantic information at different scales. Then, pixel-by-pixel summation is performed on features of different scales within a modality to achieve the fusion of semantic information at different scales.

[0113]

[0114]

[0115]

[0116]

[0117]

[0118] In the formula, and These are the feature maps for mode 1 and mode 2, respectively. and These are convolution operations with different kernel sizes. and These are the results of convolution operations on the modality 1 feature map using different convolution kernels. and These are the results of convolution operations on modality 2 feature maps using different convolution kernels. and These are the feature maps of modality 1 and modality 2 after convolutional sampling and semantic scale fusion, respectively.

[0119] S312. Differential weighted activation: Subtract the feature maps of mode 1 and mode 2 to obtain the differences between the features of the two remote sensing images. Then, perform pooling on the obtained differential features to compress them into channel vectors, with each channel corresponding to a scalar value.

[0120]

[0121]

[0122] In the formula, The result is the element-wise difference calculation of the features of mode 1 and mode 2. For average pooling operation, This is the result after the average pooling operation, representing the difference between the two modal features in each channel.

[0123] Furthermore, the difference channel vectors are then fed into a fully connected layer to map the differences between the two modalities in the channel dimension into channel weights:

[0124]

[0125] In the formula, fc () represents a full join operation. weights These are the channel weights obtained through mapping via a fully connected layer.

[0126] S313, Complementary information interaction, multiply the original feature map of mode 1 and feature map of mode 2 with the previously obtained weights at the channel level to enhance the channels that are sensitive to the differences between the two modes, thereby extracting feature information of different importance in the channel dimension based on the differences between modes;

[0127] Then, the weighted complementary feature information is added to the original features, and the difference information between different modalities is interacted to achieve complementary enhancement:

[0128]

[0129]

[0130] In the formula, and The feature information after assigning different weights to the features of mode 1 and mode 2 respectively, and * indicates pixel-wise multiplication between tensor matrices.

[0131] S320, Feature Interaction Masking Enhancement Module includes a masking enhancement module. The masking enhancement module is based on the Transformer method and uses Cross-Attention. On this basis, it uses a smaller feature map and adds a complementary masking module, which enhances the ability of features to express semantic information of a single modality.

[0132] Specifically, it includes:

[0133] S321. Adaptive feature downsampling: Multi-scale convolutional downsampling operations are performed on the dual-modal features output by the differential complementary enhancement module. Small convolutional kernels are used to extract local information, and large convolutional kernels are used to supplement surrounding semantics. For each modality, the features obtained from convolution at different scales are added pixel-by-pixel with adaptive weights. The weights are gradually learned and updated during training.

[0134]

[0135]

[0136]

[0137]

[0138]

[0139]

[0140] In the formula, and These are the feature maps belonging to mode 1 and mode 2, respectively, output by the differential complementary module. and These are convolution operations with different kernel sizes. and These are the feature maps of modality 1 and modality 2 after convolutional sampling and semantic scale fusion, respectively. and These are the weights used for pixel-level fusion of features from modality 1 and modality 2, respectively.

[0141] S322, Cross-Attention: After obtaining the convolutionally downsampled features in step S321, the Cross-Attention module is applied to the features of each modality to map the feature map into a tensor query matrix Q, a key matrix K, and a value matrix V, resulting in an attention map A.

[0142]

[0143]

[0144]

[0145] In the formula, Features are input for mode 1. , , These are the weight matrices for calculating the query matrix Q, the key matrix K, and the value matrix V, respectively, used to map the size of the feature tensor from C×H×W to HW×C, where C, H, and W are the number of channels, height, and width of the input feature map, respectively.

[0146]

[0147]

[0148]

[0149] In the formula, Features are input for mode 2. , , We calculate the weight matrices for the query matrix Q, key matrix K, and value matrix V, respectively. After obtaining the query matrix Q, key matrix K, and value matrix V for the two modalities, we perform cross-calculation to obtain the cross-attention map A.

[0150]

[0151] .

[0152] S333, weight mask, to obtain attention maps of two modalities and Afterwards, and A zero-mask is randomly added to the top to eliminate some weight information, resulting in a new attention weight map:

[0153]

[0154]

[0155] In the formula, This indicates a masking operation, which multiplies the final mask attention weights with the original input feature map at the pixel level to obtain the final features output by the complementary masking enhancement module.

[0156] S400: The final fused feature map is obtained and fed into the feature fusion layer and the detection head step by step to realize the detection and recognition of each target in the multispectral remote sensing image.

[0157] Specifically, this step includes:

[0158] S410, Feature Fusion Module, such as Figure 4 As shown, the feature maps after cross-modal feature information interaction are fused, thereby transforming two independent bimodal features into fused modal information to perform the final target detection task.

[0159] First, dimensionality reduction is performed using convolution to extract the spatial feature weights of a single modality. Then, spatial feature weights of modality 1 and modality 2 are obtained through convolution. Finally, feature fusion is performed using the two spatial feature weights.

[0160] Specifically, it includes:

[0161] S411. Obtaining the spatial weights of independent modalities: First, dimensionality reduction is performed on the channel dimension using convolutional kernels of different sizes to obtain semantic information at different scales. Then, the channels are concatenated, and dimensionality reduction is performed again using convolutional operations to obtain the spatial weights of the spatial unimodality.

[0162]

[0163]

[0164]

[0165] In the formula, This is the input of Mode 1 in the feature fusion module. and These are the outputs obtained from convolutions at two different scales. This is a splicing operation along the channel dimension. The weights are extracted using mode 1 alone, while mode 2 contains the weights. It is obtained in the same way.

[0166] S412. Obtaining the spatial weights of the hybrid modality: Convolutional downsampling is performed on the feature maps of the two modalities respectively. Then, the results of the convolutional sampling of the two modalities are added pixel by pixel. After activation using the sigmoid activation function, the spatial weights of the hybrid modality are obtained. :

[0167]

[0168]

[0169]

[0170] In the formula, This indicates an activation operation. This represents the operation of adding feature tensors pixel by pixel.

[0171] S413. Final feature fusion yields the weights. , , Next, the fused modal weights are added to the independent weights, then cross-multiplied with the original feature maps of the different modalities. Finally, a residual link is established with the original feature maps. The weighted features of the two modalities are then concatenated along the channel dimension, and channel dimensionality reduction is achieved through convolution operations to obtain the final features used for detection.

[0172]

[0173]

[0174]

[0175]

[0176] .

[0177] The final detection result of this embodiment is as follows: Figure 5 As shown.

[0178] Example 3

[0179] Furthermore, this embodiment of the invention also provides a multispectral remote sensing image target detection system based on complementary masks. This system has program modules corresponding to the steps described above, and the detection system executes the steps in the multispectral remote sensing image target detection method based on complementary masks during operation. Other combinations and connections in this embodiment are the same as in specific embodiment one.

[0180] Example 4

[0181] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing a computer program configured to implement the steps of the above-described multispectral remote sensing image target detection method based on complementary masks when invoked by a processor.

[0182] Other combinations and connections in this embodiment are the same as in Specific Implementation Scheme 1.

[0183] In summary, this invention proposes several embodiments of a multispectral remote sensing image target detection method and system based on complementary masks for multimodal remote sensing image target detection tasks. The detection method employs the PyTorch deep learning framework to extract semantic features from different modalities of remote sensing images, and then performs feature fusion through a newly proposed enhancement fusion module to ultimately achieve the desired detection effect. Figure 6As shown, tests were conducted on the publicly available dataset DroneVehicle in this field. The mean average accuracy (mAP) of this invention increased by 1.87% compared to the baseline method and by 4.4%-8.6% compared to other multimodal detection methods. The number of parameters was 11.41M, significantly less than other detection methods. Furthermore, this invention improves the detection accuracy and scalability of computer vision models in multimodal remote sensing image target detection tasks while maintaining a relatively small number of parameters, facilitating the deployment of the algorithm model on hardware devices.

[0184] This invention addresses the issues of single-modality remote sensing image target detection performance being susceptible to environmental influences and information redundancy between different modalities. It proposes a novel deep learning model: building upon the traditional single-branch feature extraction backbone network, a dual-branch feature extraction network suitable for multimodal remote sensing image input is constructed. This network also boasts high scalability, facilitating the addition of necessary feature enhancement and fusion modules. Furthermore, this invention improves upon the traditional self-attention mechanism by proposing differential complementary enhancement and complementary mask enhancement modules to achieve effective interaction of feature information between different modalities. This highlights the differences between modalities, suppresses redundant information, leverages the role of differential information, fully exploits the semantic information contained in a single modality, and improves the ability to extract target features. Finally, through a modality fusion module, the consistency information between different modalities is utilized to fuse remote sensing images from two different modalities at the feature level, further enhancing the detection performance of the new model.

[0185] Furthermore, this invention feeds feature maps from corresponding layers of two modal remote sensing images into a feature interaction mask enhancement module to extract consistency and complementarity information between features of different modalities, achieving semantic information fusion between remote sensing images of different modalities. It utilizes a modality fusion module, considering the spatial distribution of feature maps from intra-modal and inter-modal perspectives, and emphasizes regions of significant interest by calculating weights, thus improving the model's ability to acquire information on detailed and global features, better performing feature fusion between different modalities, and thereby enhancing the model's ability to detect targets in remote sensing images. This invention can solve the target detection problem with dual-modal input and can be applied to practical scenarios such as all-weather monitoring, multi-target tracking, defect detection, and quality control.

[0186] In the description of this invention, it should be understood that terms such as “length”, “width”, “upper”, “lower”, “front”, “rear”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inner”, and “outer” indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0187] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0188] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A target detection method for multispectral remote sensing images based on complementary masks, characterized in that, Includes the following steps: S100. Construct a multispectral remote sensing image dataset and perform preprocessing, dividing the dataset into a training set, a validation set, and a test set; S200. Construct a dual-branch feature extraction network suitable for dual-modal input, including a dual-stream feature extraction backbone network and a feature fusion layer, to extract features at all levels of remote sensing images of different modalities; S300: Obtain image features at all levels, and send the feature maps of corresponding levels of the two modal remote sensing images into the feature interaction mask enhancement module to extract consistency and complementarity information between different modal features, thereby realizing the fusion of semantic information between different modal remote sensing images; The feature interaction mask enhancement module includes a differential complementary enhancement module, which uses two different modal feature maps extracted from the feature extraction backbone network to obtain differential information, supplement the semantic information lacking in a single modality, and realize semantic information interaction enhancement between remote sensing images of different modalities. The feature interaction mask enhancement module includes a mask enhancement module, which is based on the Transformer method and uses Cross-Attention. On this basis, it uses a smaller feature map and adds a complementary mask module, so that the feature's ability to express single-modal semantic information is enhanced. The mask enhancement module includes: S321. Adaptive feature downsampling: Multi-scale convolutional downsampling operations are performed on the dual-modal features output by the differential complementary enhancement module. Small convolutional kernels are used to extract local information, and large convolutional kernels are used to supplement surrounding semantics. For each modality, the features obtained from convolution at different scales are added pixel-by-pixel with adaptive weights. The weights are gradually learned and updated during training. In the formula, and These are the feature maps belonging to mode 1 and mode 2, respectively, output by the differential complementary module. and These are convolution operations with different kernel sizes. and These are the feature maps of modality 1 and modality 2 after convolutional sampling and semantic scale fusion, respectively. and These are the weights used for pixel-level fusion of features from modality 1 and modality 2, respectively. S322, Cross-Attention: After obtaining the features after convolutional downsampling, the Cross-Attention module is applied to the features of each modality to map the feature map into a tensor query matrix Q, a key matrix K, and a value matrix V, resulting in an attention map A. In the formula, Features are input for mode 1. , , These are the weight matrices for calculating the query matrix Q, the key matrix K, and the value matrix V, respectively, used to map the size of the feature tensor from C×H×W to HW×C, where C, H, and W are the number of channels, height, and width of the input feature map, respectively. In the formula, Features are input for mode 2. , , We calculate the weight matrices for the query matrix Q, key matrix K, and value matrix V, respectively. After obtaining the query matrix Q, key matrix K, and value matrix V for the two modalities, we perform cross-calculation to obtain the cross-attention map A. S333, weight mask, to obtain attention maps of two modalities and Afterwards, and A zero-mask is randomly added to the top to eliminate some weight information, resulting in a new attention weight map: In the formula, This indicates a masking operation, which multiplies the final mask attention weights with the original input feature map at the pixel level to obtain the final features output by the complementary masking enhancement module. S400: The final fused feature map is obtained and fed into the feature fusion layer and the detection head step by step to realize the detection and recognition of each target in the multispectral remote sensing image.

2. The multispectral remote sensing image target detection method based on complementary masks according to claim 1, characterized in that, Step S100 includes: The system matches remote sensing images of different modalities and aligns remote sensing images of different modalities with target labels in corresponding scenes to construct a multispectral remote sensing image dataset adapted to a dual-branch feature extraction network. The dataset is then randomly divided into training, validation, and test sets.

3. The multispectral remote sensing image target detection method based on complementary masks according to claim 1, characterized in that, Step S200 includes: Based on the YOLOv11 feature extraction network, a bimodal feature extraction network branch suitable for modality 1 and modality 2 is constructed. The two branches have the same structure, are independent of each other, and do not affect each other. The two different modalities of remote sensing images are convolved and sampled step by step to extract multi-level features. The extracted three-level feature maps are then sent to the subsequent feature enhancement module.

4. The multispectral remote sensing image target detection method based on complementary masks according to claim 1, characterized in that, The differential complementary enhancement module includes: S311. Feature extraction downsampling: First, convolution operations are performed on feature maps of different modalities at the same level using convolution kernels of different scales to extract semantic information at different scales. Then, pixel-by-pixel summation is performed on features of different scales within a modality to achieve the fusion of semantic information at different scales. In the formula, and These are the feature maps for mode 1 and mode 2, respectively. and These are convolution operations with different kernel sizes. and These are the results of convolution operations on the modality 1 feature map using different convolution kernels. and These are the results of convolution operations on modality 2 feature maps using different convolution kernels. and These are the feature maps of modality 1 and modality 2 after convolutional sampling and semantic scale fusion, respectively. S312. Differential weighted activation: Subtract the feature maps of mode 1 and mode 2 to obtain the differences between the features of the two remote sensing images. Then, perform pooling on the obtained differential features to compress them into channel vectors, with each channel corresponding to a scalar value. In the formula, The result is the element-wise difference calculation of the features of mode 1 and mode 2. For average pooling operation, This is the result after the average pooling operation, representing the difference between the two modal features in each channel; S313, Complementary information interaction, multiply the original feature map of mode 1 and feature map of mode 2 with the previously obtained weights at the channel level to enhance the channels that are sensitive to the differences between the two modes, thereby extracting feature information of different importance in the channel dimension based on the differences between modes; Then, the weighted complementary feature information is added to the original features, and the difference information between different modalities is interacted to achieve complementary enhancement: In the formula, and The feature information after assigning different weights to the features of mode 1 and mode 2 respectively, and * indicates pixel-wise multiplication between tensor matrices.

5. The multispectral remote sensing image target detection method based on complementary masks according to claim 1, characterized in that, Step S400 includes: The feature fusion module fuses the feature maps after cross-modal feature information interaction, thereby transforming two independent bimodal features into fused modal information to perform the final target detection task. First, dimensionality reduction is performed using convolution to extract the spatial feature weights of a single modality. Then, spatial feature weights of modality 1 and modality 2 are obtained through convolution. Finally, feature fusion is performed using the two spatial feature weights.

6. The multispectral remote sensing image target detection method based on complementary masks according to claim 5, characterized in that, The feature fusion module includes: S411. Obtaining the spatial weights of independent modalities: First, dimensionality reduction is performed on the channel dimension using convolutional kernels of different sizes to obtain semantic information at different scales. Then, the channels are concatenated, and dimensionality reduction is performed again using convolutional operations to obtain the spatial weights of the spatial unimodality. In the formula, This is the input of Mode 1 in the feature fusion module. and These are the outputs obtained from convolutions at two different scales. This is a splicing operation along the channel dimension. The weights are extracted using mode 1 alone, while mode 2 contains the weights. It is also obtained in the same way; S412. Obtaining the spatial weights of the hybrid modality: Convolutional downsampling is performed on the feature maps of the two modalities respectively. Then, the results of the convolutional sampling of the two modalities are added pixel by pixel. After activation using the sigmoid activation function, the spatial weights of the hybrid modality are obtained. : In the formula, This indicates an activation operation. This represents the pixel-by-pixel addition operation between feature tensors; S413. Final feature fusion yields the weights. , , Next, the fused modal weights are added to the independent weights, then cross-multiplied with the original feature maps of the different modalities. Finally, a residual link is established with the original feature maps. The weighted features of the two modalities are then concatenated along the channel dimension, and channel dimensionality reduction is achieved through convolution operations to obtain the final features used for detection. 。 7. A multispectral remote sensing image target detection system based on complementary masks, characterized in that, When the detection system is running, it performs the steps of the multispectral remote sensing image target detection method based on complementary masks as described in any one of claims 1-6.

Images