A multimodal Mamba remote sensing target detection model and method based on difference-guided and quality-enhanced candidate boxes
By adopting the Mamba model of differential guidance and quality-enhanced candidate boxes in multimodal remote sensing object detection, the challenges brought about by inter-modal and intra-modal differences are solved, and the target detection effect with high accuracy and low computational cost is achieved.
Patent Information
- Application Number
- CN202411561621.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-04
- Publication Date
- 2025-04-04
AI Technical Summary
The existing multimodal remote sensing object detection methods are difficult to effectively solve the challenges brought by differences between modes and intramodal differences, especially under different lighting conditions, which leads to higher detection accuracy and calculation costs.
A multimodal Mamba remote sensing object detection model based on differential guidance and quality-enhanced candidate boxes is adopted, including a differential guidance cross-modal fusion Mamba module, a multi-scale target perception attention module and a quality-enhanced candidate box network module. High-quality candidate boxes are generated by adaptively combining global context features, suppressing noise and focusing on the target area.
The accuracy and efficiency of multimodal object detection are improved, and the high-precision recognition effect is achieved at a lower computing cost, which can more effectively explore the differences between RGB modes and combine them with the differences between modes.
Smart Images
Figure CN119762737A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of remote sensing target detection, and in particular relates to a multimodal Mamba remote sensing target detection model and method based on difference guidance and quality enhanced candidate frames in the field. Background Art
[0002] Object detection is a fundamental task in remote sensing, and has been widely used in urban planning, traffic monitoring, disaster relief, security, and other fields. However, traditional object detection methods often face unique challenges brought by remote sensing images, such as high variability in the scale, orientation, and density of objects. To address these challenges, object-oriented object detection has been developed, which includes the detection of object angles. This approach significantly improves the accuracy and precision of detecting rotated, irregular, and densely packed objects in complex remote sensing environments. As a result, object detection has attracted extensive research attention and applications. However, these studies are mainly designed for normal visible light images, which often have sparse information and significant noise interference due to low illumination and complex weather conditions.
[0003] With the advancement of optical sensor technology, infrared (IR) images have been widely used as an additional modality for these challenges because they can stably reflect the thermal information and contours of objects regardless of lighting and weather conditions. Although IR images are less affected by lighting and weather conditions, they provide less detailed information about objects, such as color and texture. Therefore, it is useful to combine RGB and IR images for target detection, a technique also known as multimodal target detection.
[0004] Based on previous studies, the challenges of multimodal target detection in remote sensing images can be divided into two categories. One challenge is the inter-modality differences caused by modal features, shooting angles, calibration errors, and post-processing; the other challenge is the large number of intra-modality differences between RGB images. The image quality of the target may vary greatly under different lighting conditions. Specifically, artificial light sources at night, such as street lights and headlights, can produce misleading information. In addition, severe noise and shadows caused by extremely low lighting and underexposure interfere with feature extraction, resulting in a large amount of irrelevant information in the RGB modality, which hinders optimal fusion in subsequent steps.
[0005] Recent studies have attempted to address the fusion-related issues by combining convolutional neural networks (CNNs) and transformers. TSFADet
[13] designed a CNN-based alignment framework to address the weak alignment problem in paired image modalities. However, due to the fixed receptive field size, CNN-based methods have difficulty learning global contextual information, which is crucial for detecting many small objects in remote sensing images. Although transformers have shown effectiveness in capturing long-range dependencies and global information, their expensive computational burden limits their application in high spatial resolution images. Therefore, hybrid methods combining CNN and transformers have been proposed. For example, C2Former
[14] uses CNN to extract image features and then uses a multimodal cross-attention module based on the Transformer architecture to obtain aligned and complementary features, solving the problem of inaccurate cross-modal fusion. Although the results of these methods are encouraging, they still cannot avoid the quadratic computational complexity of the Transformer. In addition, in order to reduce the computational load, these methods usually project features to lower dimensions when calculating global attention, which inevitably impairs the fusion efficiency. In addition, existing methods cannot fully explore the differences between RGB modalities and combine them with the differences between modalities, which is crucial for effective multimodal object detection, especially under different lighting conditions. Summary of the invention
[0006] The technical problem to be solved by the present invention is to provide a multimodal Mamba remote sensing target detection model and method based on difference guidance and quality enhanced candidate frames.
[0007] The present invention adopts the following technical solution:
[0008] A multimodal Mamba remote sensing target detection model based on difference-guided and quality-enhanced candidate boxes, wherein the improvements include: a difference-guided cross-modal fusion Mamba module, a multi-scale target perception attention module, and a quality-enhanced candidate box network module;
[0009] The difference-guided cross-modal fusion Mamba module utilizes the difference information between the modalities to adaptively merge the features of the global context of RGB and IR images;
[0010] The multi-scale object-aware attention module enhances feature representation by focusing on relevant object regions within the RGB modality;
[0011] The quality-enhanced candidate box network module adopts an anchor-free paradigm, generates offsets and angles based on prior point training, and applies an adaptive training sample selection method for dynamic label assignment.
[0012] Furthermore, the cross-modal fusion Mamba module includes 6 layers; the first layer is the LayerNorm layer, which is used to normalize the input features by layer; the second layer is the projection layer, which uses linear transformation operations to project the features into a unified hidden space; the third layer is the residual layer, which is used to promote information exchange between channels; the fourth layer is the SiLU activation function layer, which is used to perform nonlinear transformation on the features; the fifth layer is the difference selective scanning module; the sixth layer is the residual layer, which multiplies the output features by the original features processed by the channel attention module element by element and sums them.
[0013] Furthermore, the difference selective scanning module effectively enhances the feature expression capability by selectively rescanning and reverse scanning the feature maps in combination with the channel attention module, and can further improve the fusion capability of the model by cross-scanning feature maps of different modalities.
[0014] Furthermore, the multi-scale target perception attention module includes 5 layers; the first layer is a convolution layer, which is used to perform initial processing on the single-layer features of the input; the second layer is a BatchNorm layer, which is used to normalize the input features in batches; the third layer is a pooling layer, including channel average pooling and channel maximum pooling, which is used to aggregate channel information; the fourth layer is a multi-scale convolution layer, including 3*3 convolution, 5*5 convolution and 7*7 convolution operations; the fifth layer is a residual layer, which combines the output features of the fourth layer with the re-weighted original input features to enhance the features.
[0015] Furthermore, the quality-enhanced candidate box network module includes two layers; the first layer is a layer attention aggregation layer, and the second layer is a spatial attention layer.
[0016] A multimodal Mamba remote sensing target detection method based on difference guidance and quality enhanced candidate frame, the improvement of which is that it includes the following steps:
[0017] Step 1, using the VEDAI remote sensing target detection dataset;
[0018] Step 2, constructing the remote sensing target detection model as claimed in claim 1;
[0019] Step 3: Design a loss function and train the remote sensing target detection model obtained in step 2 according to the loss function:
[0020]
[0021] Loss All =Loss det_cls +Loss det_reg
[0022] In the above formula, N is the total number of samples in the batch; y iis the true label of the i-th sample, y i The value is 0, 1, 2, 3, ..., p i is the probability that the model predicts that the i-th sample is a positive class, t i is the true value of the i-th sample bounding box, v i is the predicted value of the i-th sample bounding box, In the function, x represents the residual, which is the difference between the predicted value and the true value of the sample bounding box;
[0023] Loss det_cls Represents the category loss function of the target detection head, which uses the cross entropy loss function; Loss det_reg Represents the regression loss function of the target detection head. The regression loss function uses smooth L1 Loss function;
[0024] Step 4: Use the trained remote sensing target detection model to perform target detection on the input remote sensing image.
[0025] The beneficial effects of the present invention are:
[0026] The remote sensing target detection model disclosed in the present invention is a framework for multimodal target detection in remote sensing. Among them, the cross-modal fusion Mamba module integrates global context information by using modal difference information as a guide to adaptively resolve inter-modal conflicts, thereby achieving efficient cross-modal feature fusion. The multi-scale target perception attention module is used to suppress noise and focus on the target area, minimizing the impact of intra-modal changes on subsequent fusion. In order to ensure the effectiveness of the multi-scale target perception attention module, a quality-enhanced candidate box network module is applied to generate high-quality candidate boxes.
[0027] The remote sensing target detection method disclosed in the present invention further integrates RGB images and infrared images, improves the target detection accuracy, and can achieve high-precision recognition effect at a lower computing cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 It is a block diagram of the remote sensing target detection model disclosed in the present invention;
[0029] Figure 2 This is the block diagram of the composition of the cross-modal fusion Mamba module;
[0030] Figure 3 It is a block diagram of the components of the difference selective scanning module;
[0031] Figure 4 This is a block diagram of the composition of the multi-scale target perception attention module;
[0032] Figure 5 It is a block diagram of the composition of the quality enhanced candidate box network module;
[0033] Figure 6 It is a detection result diagram of the remote sensing target detection method disclosed in the present invention. DETAILED DESCRIPTION
[0034] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0035] Embodiment 1, this embodiment discloses a multimodal Mamba remote sensing target detection model based on difference guidance and quality enhanced candidate frame, such as Figure 1 As shown, it includes the difference-guided cross-modal fusion Mamba module, the multi-scale target-aware attention module, and the quality-enhanced candidate box (QEPN detection head) network module;
[0036] The difference-guided cross-modal fusion Mamba module utilizes the difference information between the modalities and adaptively merges the features of the global context of RGB and IR images to solve the information redundancy between the modalities;
[0037] That is, the ability of the Mamba structure to effectively capture long-range dependencies can be used to effectively fuse cross-modal features guided by modal difference information with the multi-scale target perception attention module; the multi-scale target perception attention module calculates multi-scale spatial self-attention to adaptively focus on the target area within the RGB modality, thereby providing higher quality RGB features for subsequent fusion modules;
[0038] The multi-scale target perception attention module enhances feature representation by focusing on relevant target areas within the RGB modality, solves intra-modality changes, and suppresses noise. That is, it calculates multi-scale spatial self-attention to adaptively focus on the target area within the RGB modality, thereby providing higher quality RGB features for the subsequent fusion module;
[0039] The quality-enhanced candidate box network module adopts an anchor-free paradigm, generates offsets and angles based on prior point training, and applies an adaptive training sample selection method for dynamic label assignment, thereby achieving adaptive sample selection with fewer hyperparameters, and then generating high-quality oriented candidate boxes and reducing background redundancy to facilitate fine-grained recognition without the need for additional trainable parameters.
[0040] like Figure 2As shown, the cross-modal fusion Mamba module includes 6 layers; the first layer is the LayerNorm layer, which is used to normalize the input features by layer to accelerate the convergence of the model; the second layer is the projection layer, which uses the linear transformation operation to project the features to a unified hidden space; the third layer is the residual layer, which is used to promote information exchange between channels; the fourth layer is the SiLU activation function layer, which is used to perform nonlinear transformation on the features; the fifth layer is the difference selective scanning module; the sixth layer is the residual layer, which is the residual layer processed by the channel attention module after the processing result of the first layer. This layer multiplies the output features and the original features processed by the channel attention module element by element and sums them up, and finally obtains high-quality fusion features. The joint action of these layers realizes the effective fusion of cross-modal features.
[0041] like Figure 3 As shown, the difference selective scanning module effectively enhances the feature expression capability by selectively rescanning and reverse scanning the feature maps, combining the channel attention module, embedding and linear transformation, and can further improve the fusion capability of the model by cross-linking feature maps of different modalities.
[0042] like Figure 4 As shown, the multi-scale target perception attention module includes 5 layers; the first layer is a convolution layer, which is used to perform initial processing on the single-layer features of the input; the second layer is a BatchNorm layer, which is used to normalize the input features in batches; the third layer is a pooling layer, including channel average pooling and channel maximum pooling, which is used to aggregate channel information; the fourth layer is a multi-scale convolution layer, including 3*3 convolution, 5*5 convolution and 7*7 convolution operations; the fifth layer is a residual layer, which combines the output features of the fourth layer with the re-weighted original input features to enhance the features.
[0043] like Figure 5 As shown, the quality enhanced candidate box network module includes two layers; the first layer is a layer attention aggregation layer, and the second layer is a spatial attention layer.
[0044] This embodiment also discloses a multimodal Mamba remote sensing target detection method based on difference guidance and quality enhanced candidate frames, comprising the following steps:
[0045] Step 1, using the VEDAI remote sensing target detection dataset;
[0046] This dataset is designed for target detection on remote sensing images. It contains 9 types of remote sensing ground objects, namely pickup trucks, airplanes, tractors, engineering vehicles, trucks, campers, vans, cars, and ships. Each data contains two types of images: visible light and near-infrared, with 1,248 images each, and each image has a corresponding label. This dataset is suitable for the field of remote sensing image analysis, such as environmental monitoring, disaster assessment and other application scenarios. At the same time, due to its high-quality annotations and diverse object categories, it is also widely used in research and development in the field of computer vision.
[0047] Step 2, constructing the remote sensing target detection model as claimed in claim 1;
[0048] Step 3: Design a loss function and train the remote sensing target detection model obtained in step 2 according to the loss function:
[0049]
[0050] Loss All =Loss det_cls +Loss det_reg
[0051] In the above formula, N is the total number of samples in the batch; y i is the true label of the i-th sample, y i The value is 0, 1, 2, 3, ..., p i is the probability that the model predicts that the i-th sample is a positive class, t i is the true value of the i-th sample bounding box, v i is the predicted value of the i-th sample bounding box, in smooth L1 In the function, x represents the residual, which is the difference between the predicted value and the true value of the sample bounding box;
[0052] Loss det_cls Represents the category loss function of the target detection head, which uses the cross entropy loss function; Loss det_reg Represents the regression loss function of the target detection head. The regression loss function uses Loss function;
[0053] Step 4: Use the trained remote sensing target detection model to perform target detection on the input remote sensing image.
[0054] Test results such as Figure 6 shown.
Claims
1. A multimodal Mamba remote sensing target detection model based on difference-guided and quality-enhanced candidate boxes, characterized by: Including difference-guided cross-modal fusion Mamba module, multi-scale target-aware attention module and quality-enhanced candidate box network module; The difference-guided cross-modal fusion Mamba module utilizes the difference information between the modalities to adaptively merge the features of the global context of RGB and IR images; The multi-scale object-aware attention module enhances feature representation by focusing on relevant object regions within the RGB modality; The quality-enhanced candidate box network module adopts an anchor-free paradigm, generates offsets and angles based on prior point training, and applies an adaptive training sample selection method for dynamic label assignment.
2. The multimodal Mamba remote sensing target detection model based on difference-guided and quality-enhanced candidate frames according to claim 1, characterized in that: The cross-modal fusion Mamba module includes 6 layers; the first layer is the LayerNorm layer, which is used to normalize the input features by layer; the second layer is the projection layer, which uses linear transformation operations to project the features into a unified hidden space; the third layer is the residual layer, which is used to promote information exchange between channels; the fourth layer is the SiLU activation function layer, which is used to perform nonlinear transformation on the features; The fifth layer is a difference selective scanning module; the sixth layer is a residual layer, which multiplies the output features element-wise with the original features processed by the channel attention module and sums them.
3. The multimodal Mamba remote sensing target detection model based on difference-guided and quality-enhanced candidate frames according to claim 2, characterized in that: The difference selective scanning module effectively enhances the feature expression capability by selectively rescanning and reverse scanning the feature maps in combination with the channel attention module, and can further improve the fusion capability of the model by cross-checking feature maps of different modalities.
4. The multimodal Mamba remote sensing target detection model based on difference-guided and quality-enhanced candidate frames according to claim 1, characterized in that: The multi-scale object perception attention module includes 5 layers; the first layer is a convolutional layer, which is used to perform initial processing on the input single-layer features; The second layer is the BatchNorm layer, which is used to normalize the input features in batches; the third layer is the pooling layer, including channel average pooling and channel maximum pooling, which is used to aggregate channel information; the fourth layer is the multi-scale convolution layer, including 3*3 convolution, 5*5 convolution and 7*7 convolution operations; the fifth layer is the residual layer, which combines the output features of the fourth layer with the re-weighted original input features to enhance the features.
5. The multimodal Mamba remote sensing target detection model based on difference-guided and quality-enhanced candidate frames according to claim 1, characterized in that: The quality enhanced candidate box network module includes two layers; the first layer is a layer attention aggregation layer, and the second layer is a spatial attention layer.
6. A multimodal Mamba remote sensing target detection method based on difference-guided and quality-enhanced candidate frames, characterized in that: The steps include: Step 1, using the VEDAI remote sensing target detection dataset; Step 2, constructing the remote sensing target detection model as claimed in claim 1; Step 3: Design a loss function and train the remote sensing target detection model obtained in step 2 according to the loss function: Loss All =Loss det_cls +Loss det_reg In the above formula, N is the total number of samples in the batch; y i is the true label of the i-th sample, y i The value is 0, 1, 2, 3, ..., p i is the probability that the model predicts that the i-th sample is a positive class, t i is the true value of the i-th sample bounding box, v i is the predicted value of the i-th sample bounding box, In the function, x represents the residual, which is the difference between the predicted value and the true value of the sample bounding box; Loss det_cls Represents the category loss function of the target detection head, and the category loss function uses the cross entropy loss function; Loss det_reg Represents the regression loss function of the target detection head. The regression loss function uses Loss function; Step 4: Use the trained remote sensing target detection model to perform target detection on the input remote sensing image.