Multi-modal remote sensing image open set target detection method, medium, equipment and product

Abstract

The invention provides a multi-modal remote sensing image open set target detection method, a medium, equipment and a product, and relates to the technical field of remote sensing image target detection, and the method comprises the steps: obtaining a multi-modal remote sensing image, and carrying out the preprocessing of the multi-modal remote sensing image; an open set target detection model is constructed based on a YOLO-UniOW model, a backbone network adopts a CSPDarknet architecture to extract multi-scale features, a layered complementary attention mixer is introduced into the backbone network, a Neck structure adopts a path aggregation network to perform multi-scale feature fusion, and a multi-scale target detection model is constructed. Self-adaptive feature enhancement modules are inserted into a bottom-to-top path and a top-to-bottom path of the path aggregation network respectively; and training an open set target detection model by using the preprocessed multi-modal remote sensing image, and performing target detection on the to-be-detected multi-modal remote sensing image by using the trained open set target detection model. According to the invention, multi-modal information can be fully fused, and the open category identification capability is high.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image target detection technology, and in particular to methods, media, equipment, and products for target detection in open sets of multimodal remote sensing images. Background Technology

[0002] Object detection, defined in the field of computer vision, refers to the detection and identification of objects of interest in an image, and the marking of their locations. It is a method that combines image detection and recognition, locating the target while simultaneously identifying its category. Remote sensing images are images acquired using aerial technology in a bird's-eye view format, and object detection is a core task in remote sensing image interpretation.

[0003] In remote sensing images, due to the diversity of targets and the constantly changing environment, the known target categories in the training set often cannot cover all targets appearing in real-world scenarios. Open-set target detection differs from traditional closed-set target detection; it requires the model to not only identify known target categories but also effectively identify new target categories not present in the training set. In remote sensing images, traditional closed-set target detection methods often fail to adapt to the emergence of new targets. Therefore, research on open-set target detection is particularly important, as it helps models adapt to unknown target categories, improving their generalization ability and flexibility in practical applications.

[0004] Remote sensing image data exhibits a multimodal development trend of collaborative modeling of images and textual cues. Textual information can serve as semantic guidance for image content, complementing optical, multispectral, and infrared image modalities. YOLO-UniOW (see the paper "YOLO-UniOW: Efficient Universal Open-World Object Detection") is a novel object detection model designed to address the limitations of traditional models in open-world scenarios. Traditional object detection models are typically limited to the categories encountered during training, while YOLO-UniOW overcomes this limitation by combining an open vocabulary with an open-world detection task. Through adaptive decision learning (AdaDL) and wildcard learning strategies, the YOLO-UniOW model not only improves detection efficiency but also achieves dynamic detection of unknown objects and vocabulary expansion.

[0005] However, due to the modal gap and alignment difficulties between images and text at the semantic level, existing methods often struggle to effectively integrate image and text information, limiting the model's understanding ability and generalization performance in open scenarios. Summary of the Invention

[0006] The purpose of this invention is to address the problems of insufficient multimodal information fusion and weak open-category recognition in target detection of remote sensing images, and to propose a method for open-set target detection in multimodal remote sensing images, comprising the following steps: S1. Acquire multimodal remote sensing images and preprocess them to obtain preprocessed multimodal remote sensing images; S2. An open set target detection model is constructed based on the YOLO-UniOW model. The backbone network adopts the CSPDarknet architecture to extract multi-scale features and introduces a hierarchical complementary attention mixer (HRAMi) into the backbone network. The Neck structure adopts a path aggregation network (PAN) to fuse multi-scale features. Adaptive feature enhancement modules (AFE) are inserted on the bottom-to-top and top-to-bottom paths of the PAN respectively. S3. Use the preprocessed multimodal remote sensing images to train the open set target detection model, and use the trained open set target detection model to perform target detection on the multimodal remote sensing images to be detected.

[0007] Furthermore, preprocessing of multimodal remote sensing images includes: image cropping using a sliding window, geometric alignment and histogram matching of multimodal images.

[0008] Furthermore, CSPDarknet consists of one Stem layer and four Stage layers connected sequentially; The input to CSPDarknet passes through the Stem layer, the first stage layer, the second stage layer, the third stage layer, and the fourth stage layer in sequence to obtain the output of CSPDarknet. The first, second, and third stage layers consist of convolutional modules and CSP modules, while the fourth stage layer consists of convolutional modules, SPP modules, and CSP modules. Add a hierarchical complementary attention mixer between the Stem layer and the first Stage layer, between the first Stage layer and the second Stage layer, and between the third Stage layer and the fourth Stage layer.

[0009] Furthermore, HRAMi includes 1×1 convolution, standard residual convolution branch, multi-scale attention fusion branch, fusion and residual connection operations; HRAMi's input features are convolved into 1×1 and then fed into a standard residual convolution branch and a multi-scale attention fusion branch. The features obtained from the two branches are weighted and fused. The fused features are then added to the input features through residual connections to form the final output.

[0010] Furthermore, AFE includes layer normalization, feature refinement module, spatial context module, and feature weighted fusion operation; The input features of the AFE are normalized by layers and then passed through the feature refinement module and the spatial context module. The output features of the feature refinement module and the spatial context module are then weighted and fused to obtain the output features of the AFE.

[0011] Furthermore, the open-set object detection model uses an adaptive threshold focus loss function during training, expressed as follows:

[0012] in, This represents the adaptive threshold focus loss function. Represents the category weight balancing factor. This represents the model's predicted probability of the current true class. Indicates the dynamic focus adjustment factor;

[0013] in, This indicates the sensitivity of the dynamic response controlled by hyperparameters. This represents the mean predicted probability of positive samples in the current training batch. It represents the standard deviation of the predicted probability of positive samples.

[0014] The present invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for detecting open sets of multimodal remote sensing images.

[0015] The present invention also proposes an electronic device, including a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store a computer program, the computer program including computer-readable instructions, and the processor is configured to invoke the computer-readable instructions to execute the above-described multimodal remote sensing image open set target detection method.

[0016] The present invention also proposes a computer program product, including a computer program / instruction that, when executed by a processor, implements the steps of the above-described multimodal remote sensing image open set target detection method.

[0017] The beneficial effects of the technical solution provided by this invention are: This invention constructs a multimodal remote sensing image open set detection model based on YOLO-UniOW. The HRAMi module is introduced into the backbone network to enhance the expressive power of context and channel features, effectively extracting differential target information between multimodal images. An AFE module is introduced in the feature fusion stage of the Neck to further highlight the saliency of target regions, suppress background interference, and improve the discriminative power of the fused features. Simultaneously, an Adaptive Threshold Focus Loss (ATFL) is proposed in the classification loss design to dynamically adjust the weights of positive and negative samples, effectively alleviating the difficulty of learning unknown classes in open set scenarios. This invention demonstrates significant advantages in target detection accuracy, open class adaptability, and multimodal fusion performance in complex remote sensing environments, and can be more stably applied to open set remote sensing image detection tasks. Attached Figure Description

[0018] Figure 1 This is a flowchart of a multimodal remote sensing image open set target detection method according to an embodiment of the present invention; Figure 2 This is the backbone network of an embodiment of the present invention, wherein, Figure 2 (a) shows the architecture diagram of CSPDarknet. Figure 2 (b) shows the architecture diagram of CSPDarknet with HRAMi introduced; Figure 3 This is a structural diagram of the hierarchical complementary attention mixer HRAMi; Figure 4 This is a Neck structure diagram of the Insert Adaptive Feature Enhancement Module (AFE) in an embodiment of the present invention; Figure 5 This is a structural diagram of the Adaptive Feature Enhancement (AFE) module; Figure 6 This is a block diagram of an electronic device according to an exemplary embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0020] The flowchart of the multimodal remote sensing image open set target detection method according to an embodiment of the present invention is as follows: Figure 1 Specifically, it includes the following steps: S1. Acquire remote sensing images. Remote sensing images are usually large in size and have multimodal attributes (such as RGB images and infrared images). First, the remote sensing images are preprocessed. Image cropping is performed using a sliding window method to ensure that all targets, especially edge targets, are completely covered. For multimodal images, geometric alignment and histogram matching are performed to ensure that different modal spaces correspond. The cropped image name includes the original image number and coordinate position information to facilitate the reconstruction of subsequent inference results.

[0021] S2. An open-set object detection model is built based on the YOLO-UniOW model. The YOLO-UniOW model possesses good structural generality and open feature modeling capabilities, demonstrating excellent performance in open-set object detection tasks. YOLO-UniOW introduces Adaptive Decision Learning (AdaDL), replacing computationally expensive cross-modal fusion with lightweight alignment in the CLIP latent space, achieving efficient detection without sacrificing generalization ability. Furthermore, YOLO-UniOW designs a wildcard learning strategy, capable of detecting out-of-distribution objects as "unknown," while supporting dynamic vocabulary expansion. AdaDL, through the introduction of Low-Rank Adaptation (LoRA) technology, dynamically adjusts text features in the CLIP text encoder to better align them with image features, thereby constructing a more effective decision boundary without cross-modal fusion. Wildcard learning, by introducing wildcard embeddings for the "unknown" category, enables the model to detect objects not in the predefined vocabulary and label them as "unknown." This strategy is optimized through self-supervised learning, dynamically adapting to the emergence of new categories. In addition, YOLO-UniOW is based on the dual-head matching mechanism of YOLOv10 and uses region text comparison learning to ensure that the model remains efficient and accurate when detecting known and unknown objects.

[0022] In this embodiment of the invention, the backbone network uses the CSPDarknet architecture to extract multi-scale features, referencing... Figure 2 , Figure 2 This is the backbone network of an embodiment of the present invention, wherein, Figure 2 (a) shows the architecture diagram of CSPDarknet. Figure 2 Figure (b) shows the architecture of CSPDarknet with HRAMi. CSPDarknet consists of one Stem layer and four Stage layers connected sequentially. The input of CSPDarknet passes through the Stem layer, the first Stage layer, the second Stage layer, the third Stage layer, and the fourth Stage layer in sequence to obtain the output of CSPDarknet. The first, second, and third Stage layers consist of convolutional modules and CSP modules, while the fourth Stage layer consists of a convolutional module, an SPP module, and a CSP module. The convolutional modules are 3×3 two-dimensional convolutions.

[0023] The CSPDarknet architecture suffers from insufficient inter-layer contextual representation and weak inter-channel dependencies in its feature extraction capabilities for multimodal inputs. Therefore, in a preferred embodiment of the present invention, a hierarchical complementary attention mixer (HRAMi) is added between the Stem layer and the first Stage layer, between the first Stage layer and the second Stage layer, and between the third Stage layer and the fourth Stage layer of the backbone network. The structure diagram of the hierarchical complementary attention mixer is shown in the reference diagram. Figure 3 The process includes 1×1 convolutions, a standard residual convolution branch, a multi-scale attention fusion branch, and fusion and residual connection operations. The standard residual convolution branch consists of two consecutive 3×3 convolutions (stride=1, padding=1), with a batch normalization layer and ReLU activation function between them. The standard residual convolution branch maintains the same dimensionality in its output feature map, extracting basic feature information within the local receptive field and enhancing structural stability. The multi-scale attention fusion branch first uses a 1×1 convolution for channel compression; then, a channel attention module (SE, Squeeze-and-Excitation) is introduced to adaptively weight the importance of each channel. HRAMi's input features first enter a 1×1 convolutional compression channel dimension, and then are input into a standard residual convolutional branch and a multi-scale attention fusion branch respectively. The features obtained from the two branches are weighted and fused. The fused features are added to the input features through residual connections to form the final output. This ensures that low-level semantics and high-level attention information participate in modeling together, and the output dimension is consistent with the input, making it easy to seamlessly integrate into the original YOLO-UniOW structure.

[0024] The Neck structure employs a Path Aggregation Network (PAN) for multi-scale feature fusion. The PAN includes a bottom-up path and a top-down path. However, the PAN has limited ability to distinguish target regions in complex backgrounds. In a preferred embodiment of the invention, an Adaptive Feature Enhancement (AFE) module is inserted into both the bottom-up and top-down paths of the PAN. This aims to guide the model to focus more on discriminative target regions using spatial and channel attention mechanisms, thereby improving feature fusion quality and downstream detection accuracy. A reference image shows a Neck structure with the AFE module inserted in this embodiment of the invention. Figure 4The Neck network fuses the bottom-level features M2, middle-level features M3, high-level features M4, and top-level features M5 extracted from the backbone network. The first, second, third, and fourth stage layers output features M2, M3, M4, and M5, respectively. The Neck network's top-to-bottom path sequentially yields top-level feature P5, high-level feature P4, middle-level feature P3, and bottom-level feature P2. Features P5 and P4 are upsampled and then inserted into an AFE (Automatic Feature Builder), serving as inputs to the AFE. The Neck network's bottom-to-top path sequentially yields bottom-level feature N2, middle-level feature N3, high-level feature N4, and top-level feature N5. An AFE is inserted after feature N3, with feature N3 serving as its input. The top-to-bottom path of the Neck is as follows: Feature M5 passes through a convolutional layer (Conv) to obtain feature P5; after upsampling, feature P5 is concatenated with feature M4 through the output of the AFE and input into the CSP, then through a convolutional layer to output feature P4; after upsampling, feature P4 is concatenated with feature M3 through the output of the AFE and input into the CSP, then through a convolutional layer to output feature P3; after upsampling, feature P3 is concatenated with feature M2 to obtain feature P2. The bottom-to-top path of the Neck is as follows: Feature P2 passes through the CSP and a convolutional layer to output feature N2; feature N2 and feature P3 are concatenated and then through the CSP and a convolutional layer to output feature N3; feature N3 is concatenated with feature P4 through the output of the AFE and then through the CSP and a convolutional layer to output feature N4; feature N4 and feature P5 are concatenated and then input into the CSP to obtain feature N5.

[0025] Structure diagram reference of Adaptive Feature Enhancement Module (AFE) Figure 5 This includes layer normalization, 1×1 convolutional layers, a feature refinement module (FRM), a spatial context module (SCM), and feature weighted fusion operations. First, the input features of the AFE are normalized, as shown below:

[0026] in, The features after normalization of the representation layer, Here, b represents the input features of the AFE, c represents the batch size, h and w represent the spatial dimensions, specifically the height and width of the feature map, respectively. , These are the mean and standard deviation of all spatial locations along channel c. To represent a very small positive constant, usually set to 10. -5 Or even smaller, mainly used to avoid the denominator being zero and to improve the numerical stability of normalized calculations.

[0027] Then, a 1×1 convolutional layer is used for channel compression, compressing the channels to... , to obtain features Channel compression reduces the computational complexity of subsequent operations, promotes information fusion between features, and provides a unified scale and expression space for feature enhancement modules.

[0028] feature The image is processed through a Feature Refinement Module (FRM) and a Spatial Context Module (SCM). FRM primarily enhances the detailed features of key regions in the image, compensating for edge and texture information lost due to downsampling. Its design is based on image sharpening and contrast enhancement, combining edge detection and high-frequency texture extraction to extract local high-frequency components and regional low-frequency responses, which are then fused. The introduction of FRM enables the model to better distinguish target boundaries, eliminate interference from blurred regions, and improve target clarity and detection robustness against complex backgrounds.

[0029] The calculation process is as follows:

[0030] in, This indicates the output characteristics of FRM. Indicates the feature Perform edge enhancement. Indicates the feature Perform local contrast enhancement.

[0031] SCM aims to enhance the spatial modeling ability of a model by expanding the receptive field, especially for identifying targets with large scale differences in remote sensing images. The process is as follows: Features... By combining 7×7 convolution, batch normalization (BatchNorm, BN), and ReLU activation, feature maps with stronger spatial context semantics are generated. This structure enables the model to focus on structural target regions in a wide-area scene, improving its ability to distinguish long-distance targets such as building clusters and road networks in remote sensing images.

[0032] The output features of the feature refinement module and the spatial context module are weighted and fused to obtain the enhanced features of the AFE output. The fusion process can be represented as: the output features of SCM and FRM and The features are concatenated along the channel dimension and further fused using a lightweight MLP network and a 1×1 convolutional layer to obtain the final enhanced feature map. This is so that it can be used in the subsequent PAN module for multi-scale feature fusion and target detection. It is represented as:

[0033] in, Representation of features and The splicing is performed along the channel dimension; MLP stands for MLP network. This represents a 1×1 convolutional layer.

[0034] The Adaptive Feature Enhancement (AFE) module significantly improves the semantic integrity and edge clarity of target regions in multi-scale remote sensing images by combining spatial context modeling and local detail extraction, effectively enhancing the model's ability to perceive and detect targets in complex scenes.

[0035] S3. Use the preprocessed multimodal remote sensing images to train the open set target detection model, and use the trained open set target detection model to perform target detection on the multimodal remote sensing images to be detected.

[0036] The distribution of target categories in remote sensing images is extremely imbalanced, and there are a large number of unlabeled class samples in open scenes. Traditional Focal Loss suffers from the problem of negative samples dominating. In this embodiment of the invention, the open set target detection model uses an adaptive threshold focus loss function (ATFL) to replace the classification loss branch and classification loss term in YOLO-UniOW during training, thereby improving the robustness and generalization ability of the model in open set multimodal remote sensing scenes. The expression of the ATFL loss function is as follows:

[0037] in, This represents the adaptive threshold focus loss function; This represents the category weighting balancing factor, used to weight the importance of different categories. This represents the model's predicted probability of the current true class. Indicates the dynamic focus adjustment factor;

[0038] in, This indicates the hyperparameter controlling the dynamic response sensitivity; the default value is 2.0. This represents the mean predicted probability of positive samples in the current training batch. It represents the standard deviation of the predicted probability of positive samples.

[0039] Unlike traditional Focal Loss which uses a fixed exponential factor Unlike other methods, ATFL employs an adaptive design, enabling the model to dynamically adjust the loss focus during training based on the prediction confidence distribution of the current batch. When the model's prediction accuracy is high (i.e., μ is close to 1), reduce The value should be adjusted to avoid over-penalization; when the model's prediction uncertainty increases ( When it is relatively large, increase This strengthens the focus on difficult samples; for unknown categories in open sets, because their predicted probability distribution is discrete, ATFL can appropriately increase the penalty coefficient to suppress them from being misclassified as known categories.

[0040] In one exemplary embodiment, a computer-readable storage medium is included, which stores a computer program that, when executed by a processor, implements the above-described multimodal remote sensing image open set target detection method.

[0041] Please see Figure 6 In one exemplary embodiment, the device further includes an electronic device including at least one processor, at least one memory, and at least one communication bus.

[0042] The memory stores a computer program, which includes computer-readable instructions. The processor calls the computer-readable instructions stored in the memory through the communication bus to execute the above-mentioned multimodal remote sensing image open set target detection method.

[0043] In one exemplary embodiment, a computer program product is proposed, including a computer program / instructions that, when executed by a processor, implement the steps of the multimodal remote sensing image open set target detection method described above.

[0044] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for target detection in open sets of multimodal remote sensing images, characterized in that, Includes the following steps: S1. Acquire multimodal remote sensing images and preprocess them to obtain preprocessed multimodal remote sensing images; S2. An open set target detection model is constructed based on the YOLO-UniOW model. The backbone network uses the CSPDarknet architecture to extract multi-scale features and introduces a hierarchical complementary attention mixer into the backbone network. The Neck structure uses a path aggregation network to fuse multi-scale features. Adaptive feature enhancement modules are inserted into the bottom-to-top and top-to-bottom paths of the path aggregation network. S3. Use the preprocessed multimodal remote sensing images to train the open set target detection model, and use the trained open set target detection model to perform target detection on the multimodal remote sensing images to be detected.

2. The method for open set target detection in multimodal remote sensing images according to claim 1, characterized in that, Preprocessing of multimodal remote sensing images includes: image cropping using a sliding window, geometric alignment and histogram matching for multimodal images.

3. The method for open set target detection in multimodal remote sensing images according to claim 1, characterized in that, The CSPDarknet consists of one Stem layer and four Stage layers connected sequentially. The input to CSPDarknet passes through the Stem layer, the first stage layer, the second stage layer, the third stage layer, and the fourth stage layer in sequence to obtain the output of CSPDarknet. The first, second, and third stage layers consist of convolutional modules and CSP modules, while the fourth stage layer consists of convolutional modules, SPP modules, and CSP modules. Add a hierarchical complementary attention mixer between the Stem layer and the first Stage layer, between the first Stage layer and the second Stage layer, and between the third Stage layer and the fourth Stage layer.

4. The method for open set target detection in multimodal remote sensing images according to claim 1, characterized in that, The hierarchical complementary attention mixer includes a 1×1 convolution, a standard residual convolution branch, a multi-scale attention fusion branch, and fusion and residual connection operations; The input features of the hierarchical complementary attention mixer are convolved into 1×1 and then fed into the standard residual convolution branch and the multi-scale attention fusion branch respectively. The features obtained from the two branches are weighted and fused. The fused features are added to the input features through residual connections to form the final output.

5. The method for open set target detection in multimodal remote sensing images according to claim 1, characterized in that, The adaptive feature enhancement module includes layer normalization, 1×1 convolutional layer, feature refinement module, spatial context module, and feature weighted fusion operation; The input features of the adaptive feature enhancement module are normalized by layers and then processed by 1×1 convolutional layers. After passing through the feature refinement module and the spatial context module, the output features of the feature refinement module and the spatial context module are then subjected to a feature weighted fusion operation to obtain the output features of the adaptive feature enhancement module.

6. The method for open set target detection in multimodal remote sensing images according to claim 1, characterized in that, The open-set object detection model uses an adaptive threshold focus loss function during training, expressed as follows: in, This represents the adaptive threshold focus loss function. Represents the category weight balancing factor. This represents the model's predicted probability of the current true class. Indicates the dynamic focus adjustment factor; in, This indicates the sensitivity of the dynamic response controlled by hyperparameters. This represents the mean predicted probability of positive samples in the current training batch. It represents the standard deviation of the predicted probability of positive samples.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

8. An electronic device, characterized in that, The device includes a processor and a memory, the processor being interconnected with the memory, wherein the memory is used to store a computer program, the computer program including computer-readable instructions, and the processor is configured to invoke the computer-readable instructions to perform the method as described in any one of claims 1-6.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-6.

Images