Multi-modal remote sensing small target identification method based on common-differential mode collaborative interactive fusion

Abstract

The invention discloses a multi-modal remote sensing small target recognition method based on common-differential mode collaborative interactive fusion, and relates to the technical field of image recognition. Performing feature extraction and interactive fusion based on a double-flow backbone feature extraction network in the multi-modal remote sensing small target recognition model to obtain an infrared enhanced fusion feature and a visible light enhanced fusion feature respectively; the infrared enhancement fusion features and the visible light enhancement fusion features output by the corresponding convolution down-sampling module and the output layer in the two backbone feature extraction channels are spliced and then are subjected to neck feature fusion output to obtain a prediction image with a target recognition result. According to the method, infrared and visible light modal features are fully utilized, and common features and difference features of the infrared and visible light modal features are subjected to collaborative modeling and interactive fusion, so that the characterization level of the small target is effectively improved, and recognition of the remote sensing small target is more robust.

Description

Technical Field

[0001] This invention relates to a multimodal remote sensing small target recognition method based on common-difference mode collaborative interaction fusion, belonging to the field of image recognition technology. Background Technology

[0002] In recent years, with continuous breakthroughs in computer vision technology, remote sensing small target recognition has shown broad application prospects in fields such as military reconnaissance, disaster monitoring, and urban management. Small targets in remote sensing images are typically characterized by their small size, blurred edges, and weakened texture, and are easily affected by complex backgrounds. Although deep learning techniques such as convolutional neural networks (CNNs) and Transformers have made some progress in the accuracy of small target recognition, existing methods still suffer from bottlenecks such as high false negative rates and high false recognition rates due to insufficient feature representation. Therefore, how to effectively improve the robustness and accuracy of remote sensing small target recognition remains a key technical challenge that urgently needs to be overcome.

[0003] Most current small target recognition methods rely on a single visual sensor for detection and recognition. By using techniques such as feature pyramids, attention mechanisms, and multiple detectors, the model's ability to extract features from small targets has been improved, resulting in continuous breakthroughs in recognition performance. However, the detection effect of relying solely on a single visual feature is difficult to overcome the bottleneck of insufficient accuracy caused by limited information on small targets in remote sensing and complex environments. Therefore, combining multimodal information to improve the stability and robustness of small target recognition has become one of the important directions in current target recognition research.

[0004] Among multimodal features, deep fusion of infrared and visible light modes has become the most common and effective technical approach. The thermal radiation information carried by infrared images is fused with the rich texture, edge, and color details in visible light images to create a model that can effectively improve the detection and recognition performance of targets in complex scenarios such as low light, occlusion, and background interference, thus making up for the inherent limitations of single-modality in terms of perception capability and robustness.

[0005] However, existing methods for fusing infrared and visible light modes still have many limitations in practical designs. Early cross-modal feature joint fusion methods typically used feature splicing, weighted summation, and other methods to fuse and encode modal features, generating a unified fusion representation that was then directly used for subsequent detection head recognition and classification. This achieved a certain degree of efficient utilization of cross-modal information, but it extracted features separately from each modality for recognition, resulting in an insufficient fusion representation.

[0006] In recent years, interactive feature fusion strategies based on bidirectional flow of modal features have achieved deeper structural complementarity and semantic alignment of modal features. One approach focuses on interactive fusion of shared modal features, unifying semantic consistency between modalities by calculating fusion weights. However, it neglects the differences in fine-grained complementary structures between modalities, resulting in a lack of discriminative power in the fused representation. Another approach focuses on interactive fusion of differential features between modalities, effectively utilizing the complementary advantages of modalities by subtracting modal features and applying differential weights. However, this type of approach lacks sufficient modeling of the commonalities between modalities. Recently, some studies have further focused on utilizing both consistent and differential features simultaneously. The results show that this improves feature fusion capability and recognition performance to some extent. However, this type of approach, due to the lack of coordinated modeling and constraints on the inherent relationship between shared and differential information flows, is prone to semantic overlap and information redundancy in the feature representations of the two modalities.

[0007] Therefore, how to simultaneously consider shared and differing information between modalities during the fusion process and achieve effective collaborative modeling of the two remains a major challenge in multimodal feature fusion research.

[0008] Shortcomings of existing technology:

[0009] 1. Traditional single-modal recognition methods are limited by data from a single sensor, resulting in limited characterization of the target and difficulty in dealing with the diversity of targets in complex environments, leading to insufficient recognition accuracy and robustness.

[0010] 2. The development of deep learning has brought significant progress to target recognition. However, most multimodal recognition methods simply utilize modal shared summation information or differential complementary information, lacking collaborative modeling of the two. The few methods that take both into account are prone to introducing feature redundancy, reducing the discriminative ability of cross-modal features, which may lead to poor performance in small target recognition in scenarios with low signal-to-noise ratio and loss of target details. Summary of the Invention

[0011] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multimodal remote sensing small target identification method based on common-difference mode collaborative interaction fusion. While making full use of infrared and visible light modal features, it collaboratively models and interactively fuses their common and difference features, effectively improving the representation level of small targets and making the identification of remote sensing small targets more robust.

[0012] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0013] This invention provides a multimodal remote sensing small target identification method based on common-difference mode collaborative interaction fusion, comprising:

[0014] Acquire infrared and visible light images;

[0015] Infrared and visible light images are used as inputs, and a multimodal remote sensing small target recognition model is used to output a predicted image with target recognition results.

[0016] The multimodal remote sensing small target recognition model includes a backbone and a neck connected in sequence. The backbone adopts a dual-stream backbone feature extraction network. The dual-stream backbone feature extraction network includes two backbone feature extraction channels with the same structure. Each backbone feature extraction channel includes multiple convolutional downsampling modules and an output layer connected in sequence. These modules are used to extract multi-level multimodal features from infrared images and visible light images to obtain multi-level infrared modal features and multi-level visible light modal features, respectively.

[0017] The outputs of the corresponding convolutional downsampling modules in the two backbone feature extraction channels are subjected to bidirectional interactive fusion of modal features by the common-difference modality collaborative interactive fusion module to obtain bidirectional interactive fused features of two modal features;

[0018] The two-modal feature bidirectional interactive fusion feature is added element-wise to the output of the convolutional downsampling module to obtain the infrared enhancement fusion feature and the visible light enhancement fusion feature, which are then used as the input to the next convolutional downsampling module or output layer in the corresponding backbone feature extraction channel.

[0019] The infrared enhanced fusion features and visible light enhanced fusion features output by the corresponding convolutional downsampling module and output layer in the two backbone feature extraction channels are spliced ​​together and then output by neck feature fusion to obtain a predicted image with target recognition results.

[0020] Furthermore, the convolutional downsampling module is divided into a low-level convolutional downsampling module, a mid-level convolutional downsampling module, and a high-level convolutional downsampling module;

[0021] The low-level convolutional downsampling module includes a convolution-batch normalization-SiLU activation module, a cross-stage partial connection module, and a convolution-batch normalization-SiLU activation module connected in sequence.

[0022] The mid-layer convolutional downsampling module includes a cross-stage partial connection module and a convolution-batch normalization-SiLU activation module connected in sequence.

[0023] The high-level convolutional downsampling module includes a cross-stage partial connection module, a convolution-batch normalization-SiLU activation module, and a spatial pyramid pooling module connected in sequence.

[0024] The output layer employs a cross-stage partial connection module.

[0025] Furthermore, the low-level convolutional downsampling module uses infrared or visible light images as input, and the mid-level convolutional downsampling module, high-level convolutional downsampling module, and output layer use infrared-enhanced fusion features. Visible light enhanced fusion features As input, after feature extraction, the data undergoes channel alignment and scale alignment to obtain multi-level infrared modal features. Multi-level visible light mode characteristics .

[0026] Furthermore, the common-difference mode collaborative interaction fusion module includes a difference feature modulation module and a shared feature integration module;

[0027] The multi-level infrared modal features Multi-level visible light mode characteristics Both are used as inputs to two differential feature modulation modules, which perform differential operations. One of the differential feature modulation modules is used to calculate the incremental value weight of the difference brought by the infrared mode in the visible light branch. Another difference feature modulation module is used to calculate the incremental value weight of the difference brought by the visible light mode in the infrared branch. ;

[0028] The multi-level infrared modal features With multi-level visible light mode characteristics After addition, respectively with , Multiplication yields the infrared branch joint sharing characteristics and the visible light branch joint sharing characteristics after differential modulation;

[0029] The infrared branch joint sharing features and the visible light branch joint sharing features after differential modulation are respectively mapped to multi-level infrared mode features. Multi-level visible light mode characteristics By summing the residuals, the infrared modal features modulated by the difference feature are obtained. Visible light mode features modulated with difference features ;

[0030] Infrared mode features modulated with difference features Visible light mode features modulated with difference features The input shared feature integration module is used for concatenation and fusion, and the output is a two-modal feature bidirectional interactive fusion feature. .

[0031] Furthermore, the data processing procedure of the shared feature integration module includes:

[0032] The infrared modal features modulated by the difference features Visible light mode features modulated with difference features The features are refined by performing corresponding depthwise separation convolutional layers, and the outputs of the depthwise separation convolutional layers are added element-wise to obtain preliminary fused features. ;

[0033] The preliminary fusion features The inputs from the channel average pooling module and the channel max pooling module are added together to obtain spatial characterization features from different angles. ;

[0034] The spatial characterization features Spatial attention weight map is obtained through 1×1 convolution and activation function. ;

[0035] The spatial attention weight map With initial fusion characteristics Element-wise multiplication yields bidirectional interactive fusion features of the two modalities. .

[0036] Furthermore, the multimodal remote sensing small target recognition model also includes a super-resolution image reconstruction branch structure during the training phase;

[0037] The super-resolution image reconstruction branch structure includes a multi-scale feature integration encoder and a high-resolution reconstruction decoder connected in sequence.

[0038] The infrared enhancement fusion feature and the visible light enhancement fusion feature output by the low-level convolutional downsampling module in the two backbone feature extraction channels are spliced ​​together to obtain the low-level fusion spliced ​​feature. The infrared enhancement fusion feature and the visible light enhancement fusion feature output by the high-level convolutional downsampling module in the two backbone feature extraction channels are spliced ​​together to obtain the high-level fusion spliced ​​feature.

[0039] The low-level fusion and stitching features and the high-level fusion and stitching features are used as inputs to the multi-scale feature integration encoder.

[0040] The low-level features are sequentially passed through a 3×3 convolutional layer and a 1×1 convolutional layer to obtain the low-level feature remodeling features. The low-level feature remodeling features are added element by element to the low-level fusion splicing features to obtain the low-level fusion features.

[0041] The low-level fusion features are input into a multi-scale convolution branch, which includes three branches: the first branch includes a cascaded 5×5 convolutional layer and a 3×3 convolutional layer; the second branch includes a 3×3 convolutional layer; and the third branch includes a 1×1 convolutional layer.

[0042] The outputs of the multi-scale convolutional branches are concatenated and then integrated through a 1×1 convolutional layer to achieve the integration of scaled information and obtain multi-scale integrated features.

[0043] Significantly enhanced low-level fusion features are obtained by using multi-scale integration features to enhance the significance of low-level fusion features.

[0044] The high-level fusion and splicing features are sequentially passed through an upsampling layer and a 1×1 convolutional layer to obtain high-level feature alignment features;

[0045] By leveraging high-level feature alignment features to enhance low-level fusion features with high semantic guidance, we obtain low-level fusion features with enhanced multi-receptive field fusion weights.

[0046] The low-level fusion features enhanced by multi-receptive field fusion weights and the low-level fusion features enhanced by saliency are added element-wise, and the encoder output is obtained by passing through a 3×3 convolutional layer and the ReLU activation function.

[0047] The decoder employs an enhanced deep super-resolution network structure, and the encoder output is processed by the enhanced deep super-resolution network structure to obtain high-quality features.

[0048] Furthermore, the multimodal remote sensing small target recognition model also includes a detection head during the training phase. The detection head is connected to the neck output end and is used to perform target prediction, localization regression, and classification prediction on the stitched infrared enhanced fusion features and visible light enhanced fusion features to obtain target confidence, target bounding box, and class probability.

[0049] Furthermore, it also includes pre-training the multimodal remote sensing small target recognition model, the pre-training method comprising:

[0050] Obtain a training dataset, which consists of multiple paired infrared and visible light images; The training dataset is preprocessed to obtain the preprocessed training dataset;

[0051] The pre-processed training dataset is used as input to train the multimodal remote sensing small target recognition model. During the training process, the loss function is calculated, and the model parameters are adjusted according to the loss function until the maximum number of training rounds is reached, thus obtaining the pre-trained multimodal remote sensing small target recognition model.

[0052] Furthermore, the preprocessing includes:

[0053] The infrared and visible light images are annotated to adapt them to the model input format, and the center point coordinates and width and height of the target bounding boxes in the infrared and visible light images are normalized.

[0054] Establish a mapping relationship between target categories, and convert the category names of target objects within the target bounding box into integer category indices;

[0055] The resolutions of the infrared and visible light images are adjusted to a uniform size to ensure consistent input dimensions, resulting in preprocessed infrared and visible light images.

[0056] Furthermore, the expression for the loss function is:

[0057] ;

[0058] ;

[0059] ;

[0060] in, Represents the loss function. , , , , All represent weighting coefficients. This indicates the detection loss of the main trunk. This represents the reconstruction loss of the branch structure in super-resolution image reconstruction. Indicates the target confidence loss. Indicates the target bounding box loss. Represents the category probability loss. Indicates the index of the detection head. These correspond to three different scale detection heads. This indicates a low-resolution detection head used for small-scale features. A medium-resolution detection head representing medium-scale features. This indicates a high-resolution detection head used for large-scale features. This represents the error weights used to adjust the target confidence level. This indicates the error weights for adjusting the target bounding box. This represents the error weights that adjust the class probabilities. This represents the output result of the super-resolution image reconstruction branch structure. Represents a visible light image. This represents the L1 norm.

[0061] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0062] This invention fully considers the complementarity between multimodal features and the expression requirements of fine-grained features of small targets. By extracting backbone features from two streams and fusion with bidirectional interaction of common difference modes, it effectively improves the effectiveness and representation ability of cross-modal feature fusion. Furthermore, it introduces super-resolution auxiliary supervision during the training phase to enhance local detail features, thereby improving the accuracy of small target recognition and the model convergence efficiency without increasing inference overhead. Attached Figure Description

[0063] Figure 1 This is a flowchart illustrating a multimodal remote sensing small target identification method based on common-difference mode collaborative interaction fusion in one embodiment of the present invention;

[0064] Figure 2 This is a schematic diagram of the structure of the multimodal remote sensing small target recognition model in a multimodal remote sensing small target recognition method based on common-difference mode collaborative interaction fusion in one embodiment of the present invention;

[0065] Figure 3 This is a schematic diagram of the structure of the common-differential mode collaborative interaction fusion module in a multimodal remote sensing small target recognition method based on common-differential mode collaborative interaction fusion in one embodiment of the present invention;

[0066] Figure 4 This is a schematic diagram of the differential feature control module in a multimodal remote sensing small target recognition method based on common-difference mode collaborative interaction fusion in one embodiment of the present invention;

[0067] Figure 5 This is a schematic diagram of the shared feature integration module in a multimodal remote sensing small target recognition method based on common-difference mode collaborative interaction fusion in one embodiment of the present invention;

[0068] Figure 6 This is a schematic diagram of the super-resolution image reconstruction branch structure in a multimodal remote sensing small target recognition method based on common-difference mode collaborative interaction fusion in one embodiment of the present invention. Detailed Implementation

[0069] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0070] Example 1:

[0071] like Figure 1 As shown, this embodiment of the invention provides a multimodal remote sensing small target identification method based on common-difference mode collaborative interaction fusion, including the following steps:

[0072] Acquire infrared and visible light images.

[0073] The resolution of the infrared and visible light images is uniformly scaled to 640×640 to obtain preprocessed infrared and visible light images.

[0074] Using preprocessed infrared and visible light images as input, a predicted image with target recognition results is output based on a multimodal remote sensing small target recognition model.

[0075] like Figure 2As shown, the overall structure of the multimodal remote sensing small target recognition model (CIFYOLO) is based on the YOLOv5 model framework to ensure an end-to-end detection process and high detection accuracy. The YOLOv5 Focus module is removed to avoid image resolution degradation, and the single-stream backbone network of YOLOv5 is extended into a structurally consistent and parallel-running dual-stream backbone network.

[0076] Therefore, the multimodal remote sensing small target recognition model includes a backbone and a neck connected in sequence. The backbone includes a dual-stream backbone feature extraction network, which includes two backbone feature extraction channels with the same structure. Both backbone feature extraction channels adopt the CSPNet network structure. The consistent structure helps to reduce computational overhead while ensuring performance, enabling efficient implementation on embedded or edge hardware.

[0077] The backbone feature extraction channels are used to perform multi-level, multimodal feature extraction on infrared and visible light images, respectively. Each channel includes multiple convolutional downsampling modules and an output layer connected in sequence. In this embodiment, the convolutional downsampling modules are sequentially divided into a low-level convolutional downsampling module, a middle-level convolutional downsampling module, and a high-level convolutional downsampling module (in this embodiment, the low-level convolutional downsampling module is the third layer output of the backbone feature extraction channels, the middle-level convolutional downsampling module is the fifth layer output, and the high-level convolutional downsampling module is the eighth layer output).

[0078] The low-level convolutional downsampling module comprises a convolution-batch normalized-SiLU activation module, a cross-stage partial connection module, and a convolution-batch normalized-SiLU activation module connected in sequence, used for preliminary feature extraction. The mid-level convolutional downsampling module comprises a cross-stage partial connection module and a convolution-batch normalized-SiLU activation module connected in sequence, used for further downsampling feature extraction. The high-level convolutional downsampling module comprises a cross-stage partial connection module, a convolution-batch normalized-SiLU activation module, and a spatial pyramid pooling module connected in sequence, utilizing multi-scale pooling kernels (1×1, 5×5, 9×9) to capture contextual information under different receptive fields, enhancing the network's ability to perceive multi-scale targets. The output layer employs a cross-stage partial connection module for deep residual feature integration.

[0079] The data processing procedure in the backbone is as follows:

[0080] The low-level convolutional downsampling module takes infrared or visible light images as input. After extracting infrared or visible light modes through multiple convolutional blocks and downsampling layers, it performs channel alignment and scale alignment to obtain multi-level infrared modal features. Multi-level visible light mode characteristics In this embodiment, the multi-level infrared modal features and multi-level visible light modal features output by the low-level convolutional downsampling module, the middle-level convolutional downsampling module, and the high-level convolutional downsampling module are... They are 3, 5, and 8 respectively.

[0081] Multi-level infrared modal characteristics Multi-level visible light mode characteristics The input is fed into the common-difference modality collaborative interaction fusion module for bidirectional interactive fusion of modal features. For example... Figure 3 As shown, the Shared-Differential Modality Collaborative Interaction Fusion Module (SDCIF) includes two difference feature modulation (DFM) modules and one shared feature integration (SFI) module, wherein the shared feature integration module includes two deep separable convolutional layers.

[0082] Combination Figure 4 and Figure 5 The data processing in the common-difference modulus collaborative interaction fusion module includes:

[0083] Multi-level infrared modal characteristics Multi-level visible light mode characteristics Both are used as inputs to two difference feature modulation modules, one of which is used to calculate the difference increment value weight brought by the infrared mode in the visible light branch. Another difference feature modulation module is used to calculate the incremental value weight of the difference brought by the visible light mode in the infrared branch. The processing expression for the differential feature modulation module is:

[0084]

[0085]

[0086]

[0087]

[0088] in, Represents multi-level visible light mode characteristics Compared to multi-level infrared modal features The incremental value of the difference Represents multi-level infrared modal characteristics Compared to multi-level visible light modal characteristics The incremental value of the difference This represents the Sigmoid activation function. , Indicates the number of feature channels.

[0089] Multi-level infrared modal characteristics With multi-level visible light mode characteristics After addition, respectively with , Multiplication is performed to achieve a weighted adjustment of the difference increment bias on the infrared branch and the difference increment bias on the visible light branch to obtain the joint shared features of the infrared branch and the joint shared features of the visible light branch after difference modulation.

[0090] The infrared branch joint sharing features and the visible light branch joint sharing features after differential modulation are respectively mapped to multi-level infrared mode features. Multi-level visible light mode characteristics By summing the residuals, the infrared modal features modulated by the difference feature are obtained. Visible light mode features modulated with difference features The specific expression is:

[0091]

[0092]

[0093] in, express , The summation of modalities shares semantic features.

[0094] By guiding the model with spatial attention, further focus is placed on key regions of modal common response, and the infrared modal features modulated with differential features. Visible light mode features modulated with difference features As input to the shared feature integration module, the features are refined through corresponding depth-separated convolutional layers, and then element-wise summed to obtain preliminary fused features. This results in better boundary preservation and detail perception in the spatial plane, expressed as:

[0095]

[0096] in, This indicates a depthwise separable convolution operation.

[0097] To model the saliency response of fused features from a spatial dimension, the shared feature integration module introduces a lightweight and efficient spatial attention mechanism. (Initial feature fusion) After separately inputting into the channel average pooling module and the channel max pooling module, the values ​​are summed and compressed to characterize the importance of spatial regions from different perspectives, comprehensively capturing the target region and fusing them to obtain spatial characterization features from different angles. Its expression is:

[0098]

[0099] in, This indicates a channel average pooling operation. This indicates the channel max pooling operation.

[0100] Spatial characterization features Spatial attention weight map is obtained through 1×1 convolution and activation function. Its expression is:

[0101]

[0102] in, This represents a 1×1 convolution operation. This represents the Sigmoid activation function.

[0103] Spatial attention weight map With initial fusion characteristics Element-wise multiplication yields bidirectional interactive fusion features of the two modalities. Its expression is:

[0104]

[0105] in, This indicates element-wise multiplication.

[0106] Multi-level infrared modal characteristics Multi-level visible light mode characteristics The modal features obtained by the co-differential modal collaborative interaction fusion module are bidirectional interactive fusion features. Re-enter into the corresponding backbone feature extraction channel and multi-level infrared modal features or multi-level visible light modal characteristics The infrared enhanced fusion feature or the visible light enhanced fusion feature is obtained by adding elements one by one, and is used as the input of the middle layer convolutional downsampling module. The above operation is repeated until the infrared enhanced fusion feature or the visible light enhanced fusion feature is output by the output layer.

[0107] The infrared enhancement fusion features and visible light enhancement fusion features output from the low-level convolutional downsampling module, the middle-level convolutional downsampling module, the high-level convolutional downsampling module, and the output layer in the two backbone feature extraction channels are concatenated and then input into the neck area. After neck feature fusion, a predicted image with target recognition results is obtained.

[0108] In this embodiment, the co-differential modal collaborative interaction fusion module is a fusion architecture that uses shared guidance for main fusion and difference perception for auxiliary regulation. One module is a difference feature modulation module for modeling modal complementarity differences, and the other is a shared feature integration module for extracting and filtering shared semantic information. Specifically, the fusion module first constructs a unified shared feature expression space as the backbone path for multimodal information fusion, and then introduces the difference feature modulation module to calculate channel attention based on intermodal difference information, guiding shared features to enhance modality-specific content. The shared feature integration module then models the saliency of shared features through a spatial attention mechanism to achieve more refined spatial fusion and region alignment. The two modules work together to improve the completeness and effectiveness of the fusion expression from both channel and spatial dimensions, avoiding excessive redundant feature fusion.

[0109] The final obtained bidirectional interactive fusion feature of two modal features It integrates channel enhancement guided by differential information with spatial focusing mechanism driven by shared semantic perception. Differential features regulate the enhancement direction of shared representation, and shared features further guide the focusing and integration of modal complementary information in space. It is an efficient cross-modal fusion representation that collaboratively models in channel and spatial dimensions, providing rich target information for subsequent feature refinement and detection head localization and classification.

[0110] The multimodal remote sensing small target recognition model also includes a super-resolution image reconstruction branch structure during the training phase. During the training phase, the super-resolution image reconstruction branch structure is used to generate reconstructed images, and the reconstruction error is calculated as an auxiliary supervision signal for the training phase.

[0111] like Figure 6 As shown, the super-resolution image reconstruction branch structure includes a multi-scale feature integration encoder and a high-resolution reconstruction decoder connected in sequence.

[0112] Multi-level infrared modal characteristics Multi-level visible light mode characteristics After stitching, low-level fusion stitching features and multi-level infrared modal features are obtained. Multi-level visible light mode characteristics After splicing, high-level fusion splicing features are obtained.

[0113] The low-level fused and stitched features are used as input to the multi-scale feature integration encoder. They are sequentially passed through a 3×3 convolutional layer and a 1×1 convolutional layer for channel compression and feature reshaping to obtain the low-level reshaped features. The low-level reshaped features are then added element-wise to the low-level fused and stitched features to obtain the low-level fused features, whose expression is:

[0114]

[0115] in, Indicates low-level fusion characteristics. This represents a 3×3 convolution operation. This represents a 1×1 convolution operation.

[0116] Bundle In the multi-scale convolutional branch, the first branch uses a concatenated structure of 5×5 convolution followed by 3×3 convolution. The former captures global texture and contour information from a larger receptive field, while the latter further refines the feature representation, enhancing the contextual modeling capability for medium-to-large scale targets. The second branch uses 3×3 convolution to achieve a balance between receptive field and detail preservation, taking into account both structural strength and local details. The third branch uses 1×1 convolution to preserve subtle textures and fine responses of extremely small targets in the original features, avoiding the loss of key information during convolution. The three feature paths are then connected and fused, and a 1×1 convolution is used to achieve unified integration of cross-scale information, expressed as:

[0117]

[0118]

[0119]

[0120]

[0121] in, , , These represent the first branch, the second branch, and the third branch, respectively. This represents the 5×5 convolution operation in the first branch. This represents the 3×3 convolution operation in the first branch. This represents the 3×3 convolution operation in the second branch. This represents the 1×1 convolution operation in the third branch. This represents the output of the first branch. This indicates the output of the second branch. This represents the output of the third branch. This represents a 1×1 convolution operation. This indicates multi-scale integration features.

[0122] Utilizing multi-scale integration features Low-level fusion features Perform saliency enhancement to integrate multi-scale features The attention weights are obtained after the sigmoid activation function and then... Multiplication, its expression is:

[0123]

[0124] in, This indicates a significant enhancement of low-level fusion features.

[0125] The high-level fused and stitched features are used as input to the encoder, and sequentially passed through an upsampling layer and a 1×1 convolutional layer for feature alignment and channel compression to obtain the high-level feature-aligned features, the expression of which is:

[0126]

[0127] in, This indicates high-level feature alignment features. This indicates an upsampling operation.

[0128] Will Attention maps for high-level semantic features are calculated using the sigmoid activation function, and... Multiplication achieves high semantic guidance enhancement of low-level information, and its expression is:

[0129]

[0130] in, This indicates a low-level fusion feature with enhanced fusion weights across multiple receptive fields.

[0131] Will and Element-wise summation yields comprehensive features that preserve fine-grained texture while possessing global semantic awareness, effectively highlighting small target regions. Finally, 3×3 convolution and the ReLU activation function are used to further refine and introduce non-linear expressive capabilities, providing high-quality detail representations for subsequent resolution restoration. The expression is as follows:

[0132]

[0133] in, Indicates encoder output, This represents the ReLU activation function.

[0134] The decoder employs an enhanced depth super-resolution network architecture (EDSR).

[0135] Will Resolution restoration is performed on the input enhanced deep super-resolution network structure, passing through multiple cascaded residual blocks. Each residual block contains two 3×3 convolutional layers with a stride of 1, and a ReLU activation function is introduced to enhance the network's nonlinear modeling capability. After multiple residual blocks are stacked sequentially, the stacked result is added to the input features through a global residual connection, and then connected to an upsampling module to restore the features to a level close to the original. Figure 1 This achieves high spatial resolution, ultimately outputting high-quality features that are rich in detail and semantically clear. .

[0136] The multimodal remote sensing small target recognition model also includes a detection head during the training phase. The detection head is connected to the neck output end and is used to perform target prediction, localization regression and classification prediction on the stitched infrared enhanced fusion features and visible light enhanced fusion features to obtain target confidence, target bounding box and category probability.

[0137] After the multimodal remote sensing small target recognition model is built, it also needs to be pre-trained. The pre-training includes:

[0138] Obtain the training dataset, which consists of multiple paired infrared and visible light images.

[0139] The infrared and visible light images are preprocessed. Preprocessing includes labeling the infrared and visible light images to adapt them to the current network's input format, and normalizing the center coordinates and dimensions of the target bounding boxes. In this embodiment, the label files corresponding to the infrared and visible light images are set to .txt format. The target bounding boxes are normalized to [0,1]. Each target is recorded in the form of class_id, x_center, y_center, w, and h, where class_id represents the target bounding box number, x_center represents the x-coordinate of the target bounding box's center point, y_center represents the y-coordinate of the target bounding box's center point, w represents the width of the target bounding box, and h represents the height of the target bounding box.

[0140] Establish a mapping relationship between target categories, converting the category names of target objects within the target bounding box into integer category indices. Scale the resolution of both infrared and visible light images to a uniform 640×640 to obtain preprocessed infrared and visible light images.

[0141] The preprocessed training dataset is used as input to train the multimodal remote sensing small target recognition model. During training, the loss function is calculated, and its expression is as follows:

[0142] ;

[0143] ;

[0144] ;

[0145] in, Represents the loss function. , , , , All represent weighting coefficients. This indicates the detection loss of the main trunk. This represents the reconstruction loss of the branch structure in super-resolution image reconstruction. This represents the target confidence loss, which is calculated by comparing the predicted score with the true confidence score. The constraint model determines the existence of the target, thereby achieving accurate target prediction; This represents the target bounding box loss. By calculating the spatial geometric deviation between the predicted box and the ground truth box, the position parameters are finely adjusted to achieve pixel-level localization regression. This represents the category probability loss, which guides the model to identify specific attributes of the target by minimizing the difference between the category probability and the true label distribution, thus achieving reliable classification prediction. Indicates the index of the detection head. These correspond to three different scale detection heads. This indicates a low-resolution detection head used for small-scale features. A medium-resolution detection head representing medium-scale features. This indicates a high-resolution detection head used for large-scale features. These represent the error weights used to adjust the target confidence level. This indicates the error weights for adjusting the target bounding box. This represents the error weights that adjust the class probabilities. This represents the output result of the super-resolution image reconstruction branch structure. Represents a visible light image. This represents the L1 norm.

[0146] The model parameters are adjusted according to the loss function using the backpropagation algorithm to improve the accuracy of the model in locating and classifying small targets until the maximum number of training rounds is reached, thus obtaining a pre-trained multimodal remote sensing small target recognition model.

[0147] The following is a detailed description of the specific test plan:

[0148] A remote sensing small target dataset was obtained. In this embodiment, the VEDAI dataset was designed for 10-fold cross-validation. In each split, 1089 images were used for training, and another 121 images were used for testing. The dataset image size was set to 640×640. Categories with fewer than 50 samples, such as airplanes, motorcycles, and buses, were removed.

[0149] During training, infrared and visible light images were enhanced using Hue / Saturation Value (HSV), multi-scale, translation, horizontal flipping, and mosaic effects; these enhancements were removed during testing. The training process used standard Stochastic Gradient Descent (SGD) with a momentum coefficient of 0.937, a Nesterov acceleration gradient with weight decay of 0.0005, and a batch size of 2. The initial learning rate was set to 0.01. The training process consisted of 300 epochs.

[0150] This embodiment conducted a complete experiment on 10 folds of the VEDAI dataset, and the average of the 10 results was used as the final performance metric for the comparative experiment. Simultaneously, a generalization experiment was conducted on the vehicle-mounted multi-scale dataset M3FD to demonstrate the generalization ability of this embodiment. To objectively evaluate the multimodal remote sensing small target recognition method based on common-difference modality collaborative interaction fusion proposed in this embodiment, three commonly used evaluation metrics—precision, recall, and mean average precision (mAP)—were introduced to assess the model's accuracy in the target recognition task.

[0151] During the training phase, detection loss and super-resolution reconstruction loss are jointly optimized to improve the model's accuracy in locating and classifying small targets. During the testing phase, the infrared enhanced fusion features and visible light enhanced fusion features output from the output layer of the corresponding convolutional downsampling module in the two backbone feature extraction channels are spliced ​​together and input into the detection head for classification and localization, ultimately achieving high-precision recognition of multimodal remote sensing small targets.

[0152] To verify the comparative superiority of the model of this invention, this embodiment conducted a systematic comparative experiment with the current mainstream infrared-visible light fusion methods on the VEDAI dataset. The results are shown in Table 1:

[0153] Table 1: Remote sensing target category recognition accuracy of the model in this embodiment compared with other models

[0154]

[0155] The comparison results shown in Table 1 demonstrate that the CIFYOLO model used in this invention achieves an average accuracy of 77.1%, significantly outperforming all other methods and exceeding the benchmark method SuperYOLO by 2.01%. It achieves the highest average accuracy of 87.4%, 77.7%, and 73.4% for multiple remote sensing target categories, including Camping, Boat, and Van, respectively. Its recognition performance in other target categories also ranks among the top, exhibiting stronger recognition accuracy.

[0156] To demonstrate the effectiveness of the Common-Differential Mode Collaborative Interaction Fusion Module (SDCIF) and the Super-Resolution Image Reconstruction Branch Structure (LPE-SR), an ablation study was conducted on the VEDAI dataset, and the results are shown in Table 2:

[0157] Table 2: Effectiveness Evaluation Results of Common-Differential Mode Collaborative Interaction Fusion Module and Super-Resolution Image Reconstruction Branch Structure

[0158]

[0159] The results in Table 2 show that the model using only the SDCIF module has an average accuracy 1.2% higher than the model using only the LPE-SR module, and the model using only the SDCIF module also slightly outperforms the benchmark model SuperYOLO. This indicates that the multimodal fusion strategy of this invention achieves sufficient and effective feature mining between modalities, which is the main reason for improving the accuracy of small target recognition. The model using the LPE-SR module also effectively improves recognition accuracy, exceeding that of most existing multimodal remote sensing target recognition methods. This confirms the effectiveness of the SDCIF and LPE-SR modules of this invention in improving the accuracy of remote sensing small target recognition.

[0160] Furthermore, to verify the generalization ability of the model of this invention in other scenarios, this embodiment conducted a generalization experiment on the vehicle-mounted multi-scale dataset M3FD. Compared with the VEDAI remote sensing image dataset, which contains mostly small targets, the M3FD dataset presents a greater challenge to the detection of small targets in a context of mixed multi-scale targets, thus better demonstrating the ability of the model of this invention to identify small targets in complex multi-scale scenes.

[0161] Table 3: Generalization experimental results of the model in this embodiment and other models on the vehicle-mounted multi-scale dataset M3FD.

[0162]

[0163] As shown in Table 3, the method proposed in this embodiment achieves optimal recognition accuracy on M3FD, with an average accuracy of 87.2%. It also achieves optimal accuracy on all target categories except the Motorcycle category.

[0164] The multimodal remote sensing small target recognition model in this embodiment is adapted to hardware circuits and intelligent chip environments for inference deployment, achieving low-latency, low-power real-time small target detection. This provides a feasible hardware implementation method for resource-constrained scenarios such as UAVs, satellite payloads, and embedded sensing platforms. The implementation method is as follows:

[0165] The small target recognition model is mapped to a hardware circuit adaptation framework, establishing a correspondence between deep learning operators and hardware acceleration units in FPGAs, ASICs, or edge AI chips, thereby achieving the matching of model structure and hardware instruction set.

[0166] Deploy models in an integrated circuit chip environment to achieve low-latency calls and low-power operation.

[0167] The chip-based small target recognition system can be applied to resource-constrained scenarios such as drones, satellite payloads, and embedded sensing terminals to achieve real-time small target detection and recognition.

[0168] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0169] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0170] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0171] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0172] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A multimodal remote sensing small target identification method based on common-difference mode collaborative interaction fusion, characterized in that, include: Acquire infrared and visible light images; Infrared and visible light images are used as inputs, and a multimodal remote sensing small target recognition model is used to output a predicted image with target recognition results. The multimodal remote sensing small target recognition model includes a backbone and a neck connected in sequence. The backbone adopts a dual-stream backbone feature extraction network. The dual-stream backbone feature extraction network includes two backbone feature extraction channels with the same structure. Each backbone feature extraction channel includes multiple convolutional downsampling modules and an output layer connected in sequence. These modules are used to extract multi-level multimodal features from infrared images and visible light images to obtain multi-level infrared modal features and multi-level visible light modal features, respectively. The outputs of the corresponding convolutional downsampling modules in the two backbone feature extraction channels are subjected to bidirectional interactive fusion of modal features by the common-difference modality collaborative interactive fusion module to obtain bidirectional interactive fused features of two modal features; The two-modal feature bidirectional interactive fusion feature is added element-wise to the output of the convolutional downsampling module to obtain the infrared enhancement fusion feature and the visible light enhancement fusion feature, which are then used as the input to the next convolutional downsampling module or output layer in the corresponding backbone feature extraction channel. The infrared enhanced fusion features and visible light enhanced fusion features output by the corresponding convolutional downsampling module and output layer in the two backbone feature extraction channels are spliced ​​together and then output by neck feature fusion to obtain a predicted image with target recognition results.

2. The multimodal remote sensing small target recognition method based on common-difference mode collaborative interaction fusion according to claim 1, characterized in that, The convolutional downsampling module is divided into a low-level convolutional downsampling module, a mid-level convolutional downsampling module, and a high-level convolutional downsampling module; The low-level convolutional downsampling module includes a convolution-batch normalization-SiLU activation module, a cross-stage partial connection module, and a convolution-batch normalization-SiLU activation module connected in sequence. The mid-layer convolutional downsampling module includes a cross-stage partial connection module and a convolution-batch normalization-SiLU activation module connected in sequence. The high-level convolutional downsampling module includes a cross-stage partial connection module, a convolution-batch normalization-SiLU activation module, and a spatial pyramid pooling module connected in sequence. The output layer employs a cross-stage partial connection module.

3. The multimodal remote sensing small target identification method based on common-difference mode collaborative interaction fusion according to claim 2, characterized in that, The low-level convolutional downsampling module takes infrared or visible light images as input, while the mid-level convolutional downsampling module, high-level convolutional downsampling module, and output layer use infrared enhancement to fuse features. Visible light enhanced fusion features As input, after feature extraction, the data undergoes channel alignment and scale alignment to obtain multi-level infrared modal features. Multi-level visible light mode characteristics .

4. The multimodal remote sensing small target identification method based on common-difference mode collaborative interaction fusion according to claim 1, characterized in that, The common-differential mode collaborative interaction fusion module includes a differential feature modulation module and a shared feature integration module; The multi-level infrared modal features Multi-level visible light mode characteristics Both are used as inputs to two differential feature modulation modules, which perform differential operations. One of the differential feature modulation modules is used to calculate the incremental value weight of the difference brought by the infrared mode in the visible light branch. Another difference feature modulation module is used to calculate the incremental value weight of the difference brought by the visible light mode in the infrared branch. ; The multi-level infrared modal features With multi-level visible light mode characteristics After adding element by element, they are respectively with , Element-wise multiplication yields the infrared branch joint sharing characteristics and the visible light branch joint sharing characteristics after differential modulation; The infrared branch joint sharing features and the visible light branch joint sharing features after differential modulation are respectively mapped to multi-level infrared mode features. Multi-level visible light mode characteristics By summing the residuals, the infrared modal features modulated by the difference feature are obtained. Visible light mode features modulated with difference features ; Infrared mode features modulated with difference features Visible light mode features modulated with difference features The input shared feature integration module is used for concatenation and fusion, and the output is a two-modal feature bidirectional interactive fusion feature. .

5. The multimodal remote sensing small target identification method based on common-difference mode collaborative interaction fusion according to claim 4, characterized in that, The data processing procedure of the shared feature integration module includes: The infrared modal features modulated by the difference features Visible light mode features modulated with difference features The features are refined by performing corresponding depthwise separation convolutional layers, and the outputs of the depthwise separation convolutional layers are added element-wise to obtain preliminary fused features. ; The preliminary fusion features The inputs from the channel average pooling module and the channel max pooling module are added together to obtain spatial characterization features from different angles. ; The spatial characterization features Spatial attention weight map is obtained through 1×1 convolutional layers and activation functions. ; The spatial attention weight map With initial fusion characteristics Element-wise multiplication yields bidirectional interactive fusion features of the two modalities. .

6. The multimodal remote sensing small target identification method based on common-difference mode collaborative interaction fusion according to claim 1, characterized in that, The multimodal remote sensing small target recognition model also includes a super-resolution image reconstruction branch structure during the training phase; The super-resolution image reconstruction branch structure includes a multi-scale feature integration encoder and a high-resolution reconstruction decoder connected in sequence. The infrared enhancement fusion feature and the visible light enhancement fusion feature output by the low-level convolutional downsampling module in the two backbone feature extraction channels are spliced ​​together to obtain the low-level fusion spliced ​​feature. The infrared enhancement fusion feature and the visible light enhancement fusion feature output by the high-level convolutional downsampling module in the two backbone feature extraction channels are spliced ​​together to obtain the high-level fusion spliced ​​feature. The low-level fusion stitching features and high-level fusion stitching features are used as inputs to the multi-scale feature integration encoder. The low-level features are sequentially passed through a 3×3 convolutional layer and a 1×1 convolutional layer to obtain the low-level feature remodeling features. The low-level feature remodeling features and the low-level fusion stitching features are added element by element to obtain the low-level fusion features. The low-level fusion features are input into a multi-scale convolution branch, which includes three branches: the first branch includes a cascaded 5×5 convolutional layer and a 3×3 convolutional layer; the second branch includes a 3×3 convolutional layer; and the third branch includes a 1×1 convolutional layer. The outputs of the multi-scale convolutional branches are concatenated and then integrated through a 1×1 convolutional layer to achieve the integration of scaled information and obtain multi-scale integrated features. Significantly enhanced low-level fusion features are obtained by using multi-scale integration features to enhance the significance of low-level fusion features. The high-level fusion and splicing features are sequentially passed through an upsampling layer and a 1×1 convolutional layer to obtain high-level feature alignment features; By leveraging high-level feature alignment features to enhance low-level fusion features with high semantic guidance, we obtain low-level fusion features with enhanced multi-receptive field fusion weights. The low-level fusion features enhanced by multi-receptive field fusion weights and the low-level fusion features enhanced by saliency are added element-wise, and the encoder output is obtained by passing through a 3×3 convolutional layer and the ReLU activation function. The decoder employs an enhanced deep super-resolution network structure, and the encoder output is processed by the enhanced deep super-resolution network structure to obtain high-quality features.

7. The multimodal remote sensing small target recognition method based on common-difference mode collaborative interaction fusion according to claim 6, characterized in that, The multimodal remote sensing small target recognition model also includes a detection head during the training phase. The detection head is connected to the neck output end and is used to perform target prediction, localization regression and classification prediction on the stitched infrared enhanced fusion features and visible light enhanced fusion features to obtain target confidence, target bounding box and category probability.

8. The multimodal remote sensing small target identification method based on common-difference mode collaborative interaction fusion according to claim 7, characterized in that, It also includes pre-training the multimodal remote sensing small target recognition model, the pre-training method including: Obtain a training dataset, which consists of multiple paired infrared and visible light images; The training dataset is preprocessed to obtain the preprocessed training dataset; The pre-processed training dataset is used as input to train the multimodal remote sensing small target recognition model. During the training process, the loss function is calculated, and the model parameters are adjusted according to the loss function until the maximum number of training rounds is reached, thus obtaining the pre-trained multimodal remote sensing small target recognition model.

9. The multimodal remote sensing small target identification method based on common-difference mode collaborative interaction fusion according to claim 8, characterized in that, The preprocessing includes: The infrared and visible light images are annotated to adapt them to the model input format, and the center point coordinates and width and height of the target bounding boxes in the infrared and visible light images are normalized. Establish a mapping relationship between target categories, and convert the category names of target objects within the target bounding box into integer category indices; The resolutions of the infrared and visible light images are adjusted to a uniform size to ensure consistent input dimensions, resulting in preprocessed infrared and visible light images.

10. The multimodal remote sensing small target identification method based on common-difference mode collaborative interaction fusion according to claim 8, characterized in that, The expression for the loss function is: ； ； ； in, Represents the loss function. , , , , All represent weighting coefficients. This indicates the detection loss of the main trunk. This represents the reconstruction loss of the branch structure in super-resolution image reconstruction. Indicates the target confidence loss. Indicates the target bounding box loss. Represents the class probability loss. Indicates the index of the detection head. These correspond to three different scale detection heads. This indicates a low-resolution detection head used for small-scale features. Medium-resolution detection head representing medium-scale features. This indicates a high-resolution detection head used for large-scale features. This represents the error weights used to adjust the target confidence level. This indicates the error weights for adjusting the target bounding box. This represents the error weights that adjust the class probabilities. This represents the output result of the super-resolution image reconstruction branch structure. Represents a visible light image. This represents the L1 norm.

Images