Remote sensing target detection model and method based on dynamic receptive field and multi-modal fusion
By using a remote sensing target detection model that integrates dynamic receptive field and multimodal fusion, the problems of target scale differences and complex backgrounds in remote sensing target detection are solved, achieving target detection with high accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SPACE STAR TECH CO LTD
- Filing Date
- 2025-12-18
- Publication Date
- 2026-04-17
AI Technical Summary
In remote sensing target detection, the target scale varies greatly and the background is complex. Existing technologies are difficult to effectively detect small targets and avoid background interference.
A remote sensing target detection model based on dynamic receptive field and multimodal fusion is adopted. By extracting features from optical remote sensing images and SAR remote sensing images and adjusting the receptive field adaptively, and combining deformable convolution to perform cross-modal feature alignment and fusion, the detection results are output.
It achieves the acquisition of contextual information for small targets and the capture of global structural features for large targets, improving the accuracy and robustness of detection and reducing computational complexity.
Smart Images
Figure CN121883933A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent interpretation and information fusion technology of multi-source remote sensing images, specifically relating to a remote sensing target detection method and system based on dynamic receptive field and multimodal fusion, which is suitable for automatic target detection using multi-source remote sensing data. Background Technology
[0002] Remote sensing target detection, as a technology in the field of Earth observation, has wide applications in military reconnaissance, maritime surveillance, environmental monitoring, and urban planning. With the rapid development of remote sensing technology, the spatial, temporal, and spectral resolutions of remote sensing images are constantly improving, providing a data foundation for refined target detection.
[0003] Unlike natural images, targets in remote sensing images (such as aircraft, ships, and vehicles) vary greatly in scale, ranging from large ships tens of meters in size to small vehicles a few meters in size. Furthermore, remote sensing images often feature complex and diverse backgrounds, with targets potentially appearing in various environments such as airports, ports, cities, and wilderness areas, resulting in significant background interference. Summary of the Invention
[0004] Therefore, this invention proposes a target detection model and method based on a dynamic receptive field and by fusing target features from optical remote sensing images and SAR remote sensing images.
[0005] This application provides a remote sensing target detection model based on dynamic receptive field and multimodal fusion, including:
[0006] The optical remote sensing image and SAR remote sensing image acquisition module is used to acquire optical remote sensing images and SAR remote sensing images, and to preprocess the optical remote sensing images and SAR remote sensing images.
[0007] A dual-stream feature extraction network is used to extract optical features and SAR features from optical remote sensing images and SAR remote sensing images, respectively.
[0008] The modality-adaptive dynamic receptive field module is embedded in the dual-stream feature extraction network. The dynamic receptive field module is used to adaptively adjust the size of the spatial receptive field according to the input optical remote sensing image, SAR remote sensing image and target characteristics.
[0009] A cross-modal feature alignment and fusion module is used to dynamically weight and fuse optical features and SAR features;
[0010] The detection result output module is used to output detection results based on the fused optical features and SAR features.
[0011] In at least one embodiment, the preprocessing of the optical remote sensing image and the SAR remote sensing image acquisition module includes normalizing the optical remote sensing image, and the preprocessing of the SAR remote sensing image includes first taking the logarithm of the SAR remote sensing image and then normalizing it.
[0012] In at least one embodiment, wherein,
[0013] The dual-stream feature extraction network includes an optical feature extraction backbone network and a SAR feature extraction backbone network;
[0014] The optical feature extraction backbone network includes convolutional layers, batch normalization layers, activation layers, and max pooling layers, which are used to extract optical features from optical remote sensing images.
[0015] The SAR feature extraction backbone network includes convolutional layers, batch normalization layers, activation layers, and max pooling layers, which are used to extract SAR features from SAR remote sensing images.
[0016] In at least one embodiment, wherein,
[0017] The optical feature extraction backbone network and the SAR feature extraction backbone network in the dual-stream feature extraction network each have a dynamic receptive field module;
[0018] The dynamic receptive field module in the optical feature extraction backbone network includes, in sequence: a convolutional layer constructed from two convolutional kernels of different sizes, an average pooling layer, a max pooling layer, a convolutional layer, a sigmoid activation layer, a spatial attention computation layer, and another convolutional layer, which are used to adaptively adjust the receptive field size according to the target characteristics of the optical remote sensing image.
[0019] The dynamic receptive field module in the SAR feature extraction backbone network includes: a convolutional layer constructed from two convolutional kernels of different sizes, an average pooling and a max pooling layer, a convolutional layer, a sigmoid activation layer, a spatial attention computation layer, and a convolutional layer, which are used to adaptively adjust the receptive field size according to the target characteristics of SAR remote sensing images.
[0020] In at least one embodiment, the cross-modal feature alignment and fusion module includes: a deformable convolutional layer with optical features aligned to the SAR feature space, a deformable convolutional layer with SAR features aligned to the optical feature space, a convolutional layer, a batch normalization layer, a ReLU activation layer, a convolutional layer, and a Softamax layer for outputting attention weights and scores.
[0021] This application also provides a remote sensing target detection method based on dynamic receptive field and multimodal fusion, which uses the above model to perform remote sensing target detection. The method includes:
[0022] S1: Acquire optical remote sensing images and SAR remote sensing images using optical remote sensing image and SAR remote sensing image acquisition modules respectively, and perform preprocessing.
[0023] S2: Using a dual-stream feature extraction network, optical features and SAR features are extracted from optical remote sensing images and SAR remote sensing images, respectively;
[0024] In at least one embodiment, step S1 includes:
[0025] S11: Normalize optical remote sensing images;
[0026] S12: Take the logarithm of the SAR remote sensing image and then normalize it.
[0027] In at least one embodiment, step S2 includes:
[0028] S21: Utilize the optical feature extraction backbone network in the dual-stream feature extraction network to extract optical features from optical remote sensing images;
[0029] S22: SAR features in SAR remote sensing images are extracted using the SAR feature extraction backbone network in the dual-stream feature extraction network.
[0030] In at least one embodiment, step S3 includes:
[0031] S31: The dynamic receptive field module in the optical feature extraction backbone network adaptively adjusts the receptive field size according to the target characteristics of the optical remote sensing image;
[0032] S32: The dynamic receptive field module in the SAR feature extraction backbone network adaptively adjusts the receptive field size according to the target characteristics of the SAR remote sensing image.
[0033] In at least one embodiment, step S4 includes:
[0034] S41: First, the extracted optical remote sensing image features are concatenated with the SAR remote sensing image features and then convolved. After that, the optical features and the convolution result are combined to perform deformable convolution, and the optical features are aligned to the SAR feature space.
[0035] S42: First, the extracted SAR remote sensing image features are concatenated with the optical remote sensing image features and then convolved. After that, the SAR features and the convolution result are combined to perform deformable convolution, and the SAR features are aligned to the optical feature space.
[0036] S43: Based on the spatial attention mechanism, the optical features after bidirectional alignment are first concatenated with the SAR features, the concatenated result is then convolved, batch normalized, and then activated by ReLU. The activated result is then convolved and the attention weights and scores are output through Softamax. Finally, the fused features are calculated based on the calculated weights and the bidirectional aligned features.
[0037] Compared with existing technologies, this invention has the following significant advantages: First, it overcomes the limitation of a fixed receptive field in target detection by introducing a dynamic receptive field mechanism, enabling sufficient contextual information for small targets in remote sensing target detection; capturing global structural features for large targets; and avoiding unnecessary background interference. Second, it simultaneously utilizes feature information from both optical and SAR remote sensing images during detection, effectively improving detection accuracy. Finally, based on a bidirectional alignment mechanism using deformable convolution and feature vectors, it avoids the model's preference for a single modality during unidirectional alignment, improving the robustness and adaptability of detection. Attached Figure Description
[0038] The following description, in conjunction with the accompanying drawings, will further illustrate the above-mentioned features, technical characteristics, advantages, and implementation methods of this application in a clear and understandable manner. The accompanying drawings are for illustrative and explanatory purposes only and do not limit the scope of this application. Wherein:
[0039] Figure 1 The overall flowchart of the target detection method provided by the present invention;
[0040] Figure 2 This is a schematic diagram of a target detection model in one embodiment of the present invention;
[0041] Figure 3 This is a schematic diagram of the structure of a dynamic receptive field network in one embodiment of the present invention. Detailed Implementation
[0042] To better understand the above technical solutions, the technical solutions of this application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of this application and the specific features in the embodiments are detailed descriptions of the technical solutions of this application, rather than limitations on the technical solutions of this application. In the absence of conflict, the embodiments of this application and the technical features in the embodiments can be combined with each other.
[0043] Considering the significant differences in target scale and the complex and diverse backgrounds in remote sensing images, by setting an appropriate receptive field in neural network-based detection methods, sufficient contextual information can be provided for small targets, global structural features can be captured for large targets, and unnecessary background interference can be avoided.
[0044] Furthermore, considering the distinct characteristics of optical remote sensing imagery and SAR remote sensing imagery—optical remote sensing imagery possesses rich texture and spectral information but is significantly affected by weather and lighting conditions; SAR remote sensing imagery offers all-weather, all-day operation but its representation of target geometry is inferior to that of optical remote sensing imagery—combining target features from optical SAR remote sensing imagery can effectively improve target detection rates.
[0045] Therefore, this invention proposes a model and method for detecting targets based on a dynamic receptive field and by fusing target features from optical remote sensing images and SAR remote sensing images.
[0046] like Figure 2 and Figure 3 As shown, the remote sensing target detection model based on dynamic receptive field and multimodal fusion includes:
[0047] The optical remote sensing image and SAR remote sensing image acquisition module is used to acquire optical remote sensing images and SAR remote sensing images, and to preprocess the optical remote sensing images and SAR remote sensing images.
[0048] A dual-stream feature extraction network is used to extract optical features and SAR features from optical remote sensing images and SAR remote sensing images, respectively.
[0049] A modality-adaptive dynamic receptive field module is embedded in a dual-stream feature extraction network. The dynamic receptive field module is used to adaptively adjust the size of the spatial receptive field based on the input optical remote sensing image, SAR remote sensing image, and target characteristics.
[0050] A cross-modal feature alignment and fusion module is used to dynamically weight and fuse optical features and SAR features;
[0051] The detection result output module is used to output detection results based on the fused optical features and SAR features.
[0052] Among them, the preprocessing of optical remote sensing images and SAR remote sensing images in the optical remote sensing image acquisition module includes normalizing the optical remote sensing images, and the preprocessing of SAR remote sensing images includes first taking the logarithm of the SAR remote sensing images and then normalizing them.
[0053] The dual-stream feature extraction network includes an optical feature extraction backbone network and a SAR feature extraction backbone network. The optical feature extraction backbone network includes convolutional layers, batch normalization layers, activation layers, and max pooling layers, which are used to extract optical features from optical remote sensing images. The SAR feature extraction backbone network includes convolutional layers, batch normalization layers, activation layers, and max pooling layers, which are used to extract SAR features from SAR remote sensing images.
[0054] The dual-stream feature extraction network includes two core networks: the optical feature extraction backbone and the SAR feature extraction backbone. The dynamic receptive field module in the optical feature extraction backbone consists of: a convolutional layer constructed from two kernels of different sizes, an average pooling layer, a max pooling layer, a convolutional layer, a sigmoid activation layer, a spatial attention computation layer, and another convolutional layer. This module adaptively adjusts the receptive field size based on the target characteristics of the optical remote sensing image. Similarly, the dynamic receptive field module in the SAR feature extraction backbone consists of: a convolutional layer constructed from two kernels of different sizes, an average pooling layer, a max pooling layer, a convolutional layer, a sigmoid activation layer, a spatial attention computation layer, and another convolutional layer. This module also adaptively adjusts the receptive field size based on the target characteristics of the SAR remote sensing image.
[0055] The cross-modal feature alignment and fusion module includes: a deformable convolutional layer aligning optical features to the SAR feature space, a deformable convolutional layer aligning SAR features to the optical feature space, a convolutional layer, a batch normalization layer, a ReLU activation layer, and a Softamax layer for outputting attention weights and scores. The dynamic weighted fusion between optical features and SAR features specifically includes:
[0056] Fusion feature = Optical alignment feature × Optical alignment feature weight + SAR alignment feature × SAR alignment feature weight.
[0057] The detection result output module includes three independent convolutional layers, which output the classification, bounding box, and centrality of the detected target, respectively.
[0058] An embodiment of the present invention also provides a remote sensing target detection method based on dynamic receptive field and multimodal fusion, comprising the following steps:
[0059] S1: Acquire optical remote sensing images and SAR remote sensing images using optical remote sensing image and SAR remote sensing image acquisition modules respectively, and perform preprocessing.
[0060] Step S1 specifically includes:
[0061] S11: Normalize the optical remote sensing image to obtain ;
[0062] S12: Take the logarithm of the SAR remote sensing image and then normalize it to obtain... .
[0063] S2: Using a dual-stream feature extraction network, optical features and SAR features are extracted from optical remote sensing images and SAR remote sensing images, respectively;
[0064] Step S2 specifically includes:
[0065] S21: Optical features in optical remote sensing images are extracted using the optical feature extraction backbone network in the dual-stream feature extraction network, resulting in... .
[0066] S22: SAR features in SAR remote sensing images are extracted using the SAR feature extraction backbone network in the dual-stream feature extraction network. .
[0067] S3: The dynamic receptive field module adaptively adjusts the size of the spatial receptive field based on the input optical remote sensing image, SAR remote sensing image, and target characteristics.
[0068] Step S3 specifically includes:
[0069] S31: The dynamic receptive field module in the optical feature extraction backbone network adaptively adjusts the receptive field size according to the target characteristics of the optical remote sensing image. Specifically, the input, after being processed by the network, enters the neural network of the dynamic receptive field module. First, the input is convolved by two convolutional kernels of different sizes. The convolution results are concatenated, and spatial relationships are extracted using average pooling (AvgPool) and max pooling (MaxPool). The pooling features are then transformed into spatial attention feature maps through convolution. After sigmoid activation, the spatial attention is calculated by weighting the input features. A second convolution is then performed to obtain the attention features. The network structure is as follows: Figure 3 As shown;
[0070] S32: The dynamic receptive field module in the SAR feature extraction backbone network adaptively adjusts the receptive field size according to the target characteristics of SAR remote sensing images. Specifically, the input, after processing, enters the neural network of the dynamic receptive field module. First, the input is convolved with two kernels of different sizes. The convolution results are concatenated, and spatial relationships are extracted using average pooling (AvgPool) and max pooling (MaxPool). The pooling features are then transformed into spatial attention feature maps through convolution. After sigmoid activation, these features are weighted with the input features for spatial attention calculation. A second convolution is performed to obtain attention features, thus extracting the spatial attention features. .
[0071] S4: Through the cross-modal feature alignment and fusion module, optical features and SAR features are dynamically weighted and fused;
[0072] Step S4 specifically includes:
[0073] S41: First, the extracted optical remote sensing image features are concatenated with the SAR remote sensing image features and then convolved. After that, the optical features and the convolution result are combined to perform deformable convolution, and the optical features are aligned to the SAR feature space.
[0074] S42: First, the extracted SAR remote sensing image features are concatenated with the optical remote sensing image features and then convolved. After that, the SAR features and the convolution result are combined to perform deformable convolution, and the SAR features are aligned to the optical feature space.
[0075] S43: Based on the spatial attention mechanism, the bidirectional aligned optical features are first concatenated with the SAR features. The concatenated result is then convolved, followed by batch normalization and ReLU activation. The activated result is then convolved again, and attention weights and scores are output through Softamax. Finally, the fused features are calculated based on the calculated weights and the bidirectional aligned features. The formula for calculating the fused features is as follows:
[0076]
[0077] S5: Based on the fused features, candidate anchor boxes are proposed using RPN, and the detection results are output.
[0078] Step S5 specifically includes:
[0079] Using the fused features as input, three independent convolutional layers are used to predict the target's classification, bounding box, and centrality, respectively.
[0080] This invention discloses a remote sensing target detection method based on dynamic receptive field and multimodal fusion. The method first acquires and preprocesses optical remote sensing images and synthetic aperture radar (SAR) remote sensing images; then, it constructs feature extraction networks for both optical and SAR images to obtain their feature outputs; modality-adaptive dynamic receptive field modules are embedded in both the optical and SAR branches to dynamically adjust the target receptive field size according to target characteristics; next, a cross-modal feature alignment and fusion module is used to solve geometric distortion problems through deformable convolution, bidirectionally aligning the features of the optical and SAR images, and employing a spatial attention mechanism to achieve intelligent feature-level fusion; finally, the detection result is output based on the fused feature vector.
[0081] This invention fully leverages the complementary advantages of optical and SAR remote sensing imagery, and adaptively adjusts the size of the spatial receptive field through a dynamic receptive field mechanism. While ensuring detection accuracy, it significantly reduces computational complexity, making it suitable for deployment on satellite equipment.
[0082] Note that the above is merely a description of the technical principles of the present invention. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.
Claims
1. A remote sensing target detection model based on dynamic receptive field and multimodal fusion, comprising: The optical remote sensing image and SAR remote sensing image acquisition module is used to acquire optical remote sensing images and SAR remote sensing images, and to preprocess the optical remote sensing images and SAR remote sensing images. A dual-stream feature extraction network is used to extract optical features and SAR features from optical remote sensing images and SAR remote sensing images, respectively. The modality-adaptive dynamic receptive field module is embedded in the dual-stream feature extraction network. The dynamic receptive field module is used to adaptively adjust the size of the spatial receptive field according to the input optical remote sensing image, SAR remote sensing image and target characteristics. A cross-modal feature alignment and fusion module is used to dynamically weight and fuse optical features and SAR features; The detection result output module is used to output detection results based on the fused optical features and SAR features.
2. The model according to claim 1, wherein, In the optical remote sensing image and SAR remote sensing image acquisition modules, the preprocessing of optical remote sensing images includes normalization, and the preprocessing of SAR remote sensing images includes first taking the logarithm of the SAR remote sensing image and then normalizing it.
3. The model according to claim 1, wherein, The dual-stream feature extraction network includes an optical feature extraction backbone network and a SAR feature extraction backbone network; The optical feature extraction backbone network includes convolutional layers, batch normalization layers, activation layers, and max pooling layers, which are used to extract optical features from optical remote sensing images. The SAR feature extraction backbone network includes convolutional layers, batch normalization layers, activation layers, and max pooling layers, which are used to extract SAR features from SAR remote sensing images.
4. The model according to claim 1, wherein, The optical feature extraction backbone network and the SAR feature extraction backbone network in the dual-stream feature extraction network each have a dynamic receptive field module; The dynamic receptive field module in the optical feature extraction backbone network includes, in sequence: a convolutional layer constructed from two convolutional kernels of different sizes, an average pooling layer, a max pooling layer, a convolutional layer, a sigmoid activation layer, a spatial attention computation layer, and another convolutional layer, which are used to adaptively adjust the receptive field size according to the target characteristics of the optical remote sensing image. The dynamic receptive field module in the SAR feature extraction backbone network includes: a convolutional layer constructed from two convolutional kernels of different sizes, an average pooling and a max pooling layer, a convolutional layer, a sigmoid activation layer, a spatial attention computation layer, and a convolutional layer, which are used to adaptively adjust the receptive field size according to the target characteristics of SAR remote sensing images.
5. The model according to claim 1, wherein, The cross-modal feature alignment and fusion module includes: a deformable convolutional layer that aligns optical features to the SAR feature space, a deformable convolutional layer that aligns SAR features to the optical feature space, a convolutional layer, a batch normalization layer, a ReLU activation layer, a convolutional layer, and a Softamax layer, which are used to output attention weights and scores.
6. A remote sensing target detection method based on dynamic receptive field and multimodal fusion, utilizing the model according to claim 1 for remote sensing target detection, the method comprising: S1: Acquire optical remote sensing images and SAR remote sensing images using optical remote sensing image and SAR remote sensing image acquisition modules respectively, and perform preprocessing. S2: Using a dual-stream feature extraction network, optical features and SAR features are extracted from optical remote sensing images and SAR remote sensing images, respectively.
7. The method according to claim 6, wherein, Step S1 includes: S11: Normalize optical remote sensing images; S12: Take the logarithm of the SAR remote sensing image and then normalize it.
8. The method according to claim 6, wherein, Step S2 includes: S21: Utilize the optical feature extraction backbone network in the dual-stream feature extraction network to extract optical features from optical remote sensing images; S22: SAR features in SAR remote sensing images are extracted using the SAR feature extraction backbone network in the dual-stream feature extraction network.
9. The method according to claim 6, wherein, Step S3 includes: S31: The dynamic receptive field module in the optical feature extraction backbone network adaptively adjusts the receptive field size according to the target characteristics of the optical remote sensing image; S32: The dynamic receptive field module in the SAR feature extraction backbone network adaptively adjusts the receptive field size according to the target characteristics of the SAR remote sensing image.
10. The method according to claim 6, wherein, Step S4 includes: S41: First, the extracted optical remote sensing image features are concatenated with the SAR remote sensing image features and then convolved. After that, the optical features and the convolution result are combined to perform deformable convolution, and the optical features are aligned to the SAR feature space. S42: First, the extracted SAR remote sensing image features are concatenated with the optical remote sensing image features and then convolved. Then, the SAR features and the convolution result are combined to perform deformable convolution, and the SAR features are aligned to the optical feature space. S43: Based on the spatial attention mechanism, the optical features after bidirectional alignment are first concatenated with the SAR features, the concatenated result is then convolved, batch normalized, and then activated by ReLU. The activated result is then convolved and the attention weights and scores are output through Softamax. Finally, the fused features are calculated based on the calculated weights and the bidirectional aligned features.