SAR and optical image registration method based on multi-scale characteristic decomposition and cross-modal fusion

Abstract

The invention discloses an SAR and optical image registration method based on multi-scale characteristic decomposition and cross-modal fusion. The method comprises the following steps: acquiring a synthetic aperture radar and an optical image; and inputting the synthetic aperture radar, the optical image and the to-be-registered area into the trained multi-modal registration model, and outputting a corresponding actual area of the to-be-registered area on another modal image. The multi-modal registration model comprises the following three components: a multi-modal feature decomposition module, which is used for separating a structure component (MSF) and a texture component (DTF) from an original image; the cross-modal twin encoder is used for realizing feature semantic alignment; and the dynamic convolution matching module is used for realizing efficient matching and coordinate regression.

Description

Technical Field

[0001] The present invention relates to a synthetic aperture radar and optical image registration method, belonging to the field of machine learning and digital image processing, and has broad application prospects in the fields of electronic information technology, geographic information systems, remote sensing, etc. Background Art

[0002] In the field of remote sensing, SAR and optical image registration refers to the process of matching pixels or feature points of Synthetic Aperture Radar (SAR) images and optical remote sensing images (such as visible light and multispectral images) of the same scene to the same coordinate system through geometric alignment and spatial mapping. Its core goal is to eliminate geometric and radiometric differences caused by factors such as imaging principles, sensor viewing angles, and terrain undulations, achieving high-precision spatial consistency between heterogeneous images. This lays the foundation for subsequent image fusion, change detection, or joint interpretation of multi-source information.

[0003] Synthetic aperture radar (SAR) and optical remote sensing imagery, as two core means of Earth observation, are remarkably complementary in their imaging mechanisms and information expression. SAR, by actively emitting microwave signals and receiving scattered echoes from ground objects, has the ability to observe through clouds, rain, fog, and haze around the clock, making it particularly suitable for disaster response and concealed target detection. Optical imagery, relying on the reflective properties of solar radiation, offers high spatial resolution and intuitive spectral information, making it irreplaceable in target recognition and ground object classification. By performing high-precision registration on these two heterogeneous imagery, the strong penetration of SAR and the detailed expression capabilities of optical imagery can be combined, providing multi-dimensional data support for applications such as disaster monitoring, military reconnaissance, and environmental monitoring, becoming a key precursor technology for multimodal remote sensing analysis.

[0004] Despite the significant application value of SAR and optical image registration, its implementation faces the following fundamental challenges: 1. SAR images are subject to side-view geometric distortion (overlapping, perspective contraction) and speckle noise, exhibiting non-intuitive scattering characteristics. Optical images are susceptible to lighting conditions, atmospheric interference, and cloud obscuration, and lack microwave penetration. 2. SAR scattering intensity has no direct physical correlation with optical reflection intensity, rendering similarity metrics based on grayscale (such as mutual information) or texture (such as structural similarity) ineffective. Traditional feature extraction methods (such as SIFT and ORB) significantly degrade feature matching consistency in SAR images due to speckle noise. 3. Topographic relief (such as hillshade) and sensor viewpoint differences lead to local nonlinear deformations between images. Existing methods, which mostly use rigid or affine transformation models, struggle to adapt to such deformations, resulting in significant registration errors in mountainous areas and urban built-up areas. Non-parametric models based on thin plate splines (TPS) or B-splines, while capable of representing complex deformations, are computationally complex and susceptible to noise. 4. Deep learning-based methods (such as convolutional neural networks and Transformer) rely on large-scale annotated data and lack the ability to generalize and adapt to multi-temporal, multi-resolution, and multi-sensor scenarios. Existing models have difficulty explicitly integrating SAR imaging geometric prior knowledge (such as the slant range projection model), resulting in reduced accuracy during cross-regional testing.

[0005] Existing technologies have not yet effectively resolved the following contradictions and still have significant limitations:

[0006] 1. Traditional artificial features or single deep learning models are difficult to adapt to the nonlinear radiation differences between SAR and optical images;

[0007] 2. The challenge of balancing high-precision nonparametric deformation field estimation with lightweight deployment requirements;

[0008] 3. How to embed SAR imaging geometric constraints into the deep learning framework to reduce dependence on labeled data.

[0009] The above bottlenecks seriously restrict the deep fusion application of multimodal remote sensing data. There is an urgent need for an adaptive registration method that takes into account high precision, strong generalization and low complexity to break through the technical barriers of collaborative analysis of heterogeneous remote sensing information. Summary of the Invention

[0010] To address the technical bottlenecks of existing SAR and optical image registration methods in adapting to feature heterogeneity, modeling complex geometric deformations, and multimodal generalization, this paper proposes a SAR-optical image registration system based on multimodal feature decomposition and cross-modal feature compensation. By integrating a feature decomposition mechanism guided by physical priors with cross-modal encoding technology driven by deep learning, this system achieves high-precision and high-efficiency registration of heterogeneous remote sensing images. It also features a visual interactive user interface (GUI) to support end-to-end engineering deployment.

[0011] An embodiment of the present invention provides a SAR-optical image registration method, comprising:

[0012] Step 1: Dataset Preparation

[0013] In step 1, the "dataset preparation" procedure is as follows: The QXS-SAROPT dataset is used. This dataset consists of 20,000 pairs of corresponding image patches collected using the GaoFen-3 SAR satellite and Google Earth's optical satellite. These image pairs are distributed over the three port cities of Santiago, Shanghai, and Qingdao. Each image has a size of 256 x 256. This dataset includes inland areas, oceans, mountains, buildings, ports, and various land types, covering common natural and man-made objects and land types.

[0014] Step 2: Data preprocessing

[0015] The principle of "data preprocessing" described in step 2 is as follows: modal differences are eliminated and feature expression is enhanced through a three-level processing flow of radiation normalization, geometric alignment, and dynamic enhancement.

[0016] Step 3: Build a multimodal registration network framework

[0017] Among them, the "multimodal registration network" described in step three has the following structure: the multimodal registration network includes three components: a multimodal feature decomposition module; a cross-modal twin encoder; and a dynamic convolution matching module.

[0018] Step 4: Implementation of multimodal feature decomposition module

[0019] The multimodal feature decomposition module described in step 4 is implemented as follows: This module is implemented by cascading a Wiener filter and a Gabor encoder. The Wiener filter dynamically calculates mixing weights based on local total variation (LTV) to separate the image into a macrostructure component (MSF) and a detail texture component (DTF). The Gabor encoder uses a 4-directional, 4 / 8 pixel scale log-Gabor filter bank to generate 256-dimensional direction-sensitive texture features through reflection filling. This design explicitly decouples the common features of SAR and optical images, reducing the need for large amounts of training data.

[0020] Step 5: Cross-modal twin encoder design

[0021] Among them, the "cross-modal twin encoder" described in step five has the following structure: a hybrid attention mechanism and an improved residual structure are used: the SAR branch integrates a channel attention module, and dynamically calibrates the scattering feature response through global average pooling and a fully connected layer; the optical branch adopts a spatial attention module, which fuses the maximum / average pooling features and generates a spatial mask through a 7×7 convolution. The residual adaptation module introduces a pre-activation structure (InstanceNorm-ReLU-Conv) and a Dropout layer, outputs a 512-dimensional fusion feature (structure + texture), and realizes cross-modal feature alignment through a modality compensation dictionary and a dynamic attention mechanism. The cross-modal twin encoder can effectively reduce the mismatch rate.

[0022] Step 6: Implementation of dynamic convolution matching mechanism

[0023] The "dynamic convolution matching" described in step 6 is done as follows: the query features are reconstructed into a set of dynamically generated convolution kernels. The 256-channel features of each query sample are independently regarded as a convolution kernel, preserving their spatial perception capabilities; the library features are reconstructed into a single batch multi-channel input feature map. This reconstruction ensures that each query kernel only interacts with the library features of the corresponding batch to avoid cross-sample interference; a padding-free grouped convolution is performed to generate a heat map, so that each query kernel acts independently on the corresponding library feature channel, and the computational complexity is reduced from O(N 2 ) is reduced to O(N). The coordinates are regressed through 5×5 Gaussian blur and sub-pixel interpolation to fully cover all possible matching positions.

[0024] Step 7: Hybrid loss function optimization

[0025] The hybrid loss function described in step 7 is implemented as follows: It combines GIoU, DIoU, and center coordinate MSE to address sensitivity issues in non-overlapping regions. A contrastive loss is used to construct query-positive sample-hard negative sample triplets, maximizing feature distance via cosine similarity. Training utilizes a hierarchical parameter optimization strategy, coupled with the AdamW optimizer and OneCycleLR for dynamic scheduling.

[0026] Step 8: Multi-GPU training and visual monitoring

[0027] Among them, the "multi-GPU training and visual monitoring" described in step eight is done as follows: the training system implements multi-GPU parallel training based on the PyTorch framework, and increases the batch size through mixed precision training. Global gradient clipping is used during training to prevent gradient explosion, and the mean / standard deviation of the gradients at each layer is recorded through a real-time monitoring module. The visualization interface integrates multi-dimensional analysis functions: the input image pair overlays the prediction box (red) and the true value box (green), annotates the IoU, RMSE and heat map confidence; the feature map visualization module displays the spatial distribution of 512-dimensional fused features; the training curve dynamically updates the loss value, learning rate and GPU memory usage. The system supports HDF5 data streaming loading and memory mapping optimization. The volume of the model can be greatly compressed after quantization, and the engineering deployment cost can be greatly reduced.

[0028] Through the above steps, the algorithm can obtain a high-precision and high-generalization cross-modal image registration model. This model can achieve high-precision alignment of SAR and optical images.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] The present invention proposes a synthetic aperture radar and optical image registration framework. By integrating physical priors with a hierarchical feature decoupling framework of deep learning, the present invention breaks through the limitations of traditional methods in expressing heterogeneous remote sensing image features: based on a joint modeling mechanism of multimodal feature decomposition, it explicitly separates cross-modal common structures and modality-specific textures, significantly improving the robustness of registration in complex terrain scenes; the dynamic convolution matching algorithm innovatively implements efficient feature similarity calculation, while ensuring sub-pixel accuracy, it greatly reduces computational complexity; the hybrid loss function effectively solves the difficult problem of balancing local deformation sensitivity and mismatch suppression through the collaborative optimization of geometric constraints and semantic discrimination goals. In terms of system design, a fully decoupled multi-task training strategy and a lightweight deployment architecture are adopted, taking into account the real-time requirements of the model on the embedded device side and the cross-sensor generalization capability, providing a solution that is both theoretically innovative and engineering practical for the efficient registration of multimodal remote sensing data, and has extremely low generalization cost and extremely strong generalization capability.

[0031] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0032] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0034] Figure 1 Schematic diagram of the framework designed for this algorithm.

[0035] Figure 2 Multimodal feature decomposition module designed for this algorithm.

[0036] Figure 3 The cross-modal twin encoder network designed for this algorithm. DETAILED DESCRIPTION

[0037] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0038] An embodiment of the present invention provides a synthetic aperture radar and optical image registration method, which specifically includes the following steps:

[0039] S1, acquire synthetic aperture radar and optical images;

[0040] S2. Input the synthetic aperture radar and optical images and the area to be registered into a trained multimodal registration model, and output the actual area corresponding to the area to be registered on another modality image.

[0041] In the above S2, the training method of the multimodal registration model is as follows:

[0042] Step 1: Acquire registered synthetic aperture radar and optical images as a training dataset; preprocess the training dataset to eliminate modal differences and enhance feature expression;

[0043] Step 2: Design a multimodal registration network, which specifically includes the following three parts: a multimodal feature decomposition module for separating the structure component (MSF) and texture component (DTF) from the original image; a cross-modal twin encoder for achieving feature semantic alignment; and a dynamic convolutional matching module for efficient matching and coordinate regression.

[0044] Step 3: Use the registered synthetic aperture radar and optical images to train a multimodal registration network;

[0045] The schematic diagram of the multimodal registration network framework can be found in Figure 1 shown.

[0046] Next, the above steps 1 to 3 are described in detail.

[0047] In step 1 above, for the synthetic aperture radar and optical image registration task, this embodiment of the present invention uses the QXS-SAROPT dataset. This dataset consists of 20,000 pairs of corresponding image patches collected using the GaoFen-3 SAR satellite and Google Earth's optical satellite. These image pairs are distributed over the three port cities of Santiago, Shanghai, and Qingdao. Each image has a size of 256 x 256. This dataset includes inland, ocean, mountainous, building, port, and various land types, covering common natural and man-made objects and land types. Furthermore, the images need to be automatically annotated, with the same areas forming annotated boxes.

[0048] In step 2 above, the multimodal registration network specifically includes the following three parts: a multimodal feature decomposition module for separating the structure component (MSF) and texture component (DTF) from the original image; a cross-modal twin encoder for achieving feature semantic alignment; and a dynamic convolution matching module for achieving efficient matching and coordinate regression. The specific definition of the above structure is as follows:

[0049] The multimodal feature decomposition module consists of a cascade of Wiener filters and Gabor encoders. Through a physical model-guided feature decoupling mechanism, it explicitly separates cross-modal shared structural features (Macro-Structural Features, MSF) and modality-specific texture features (Detail-Textural Features, DTF) from the original image. Its core design is as follows:

[0050] The Wiener filter fuses the filtering result with the original image through dynamic weights to suppress noise and retain the cross-modal common structure: First, the input image Perform 7×7 Gaussian filtering (standard deviation σ=1.6) to obtain the smoothed result I filtered =G σ *I, where the Gaussian kernel is defined as:

[0051]

[0052] Then the local total variation (LTV) is calculated to calculate the gradient amplitude of the original image and the filtered image respectively. The specific formula is as follows:

[0053]

[0054] in and is the horizontal and vertical gradient operator (Sobel kernel). Dynamic weight calculation is performed based on the LTV difference to generate a mixed weight ω∈[0,1] H×W , the formula is:

[0055]

[0056] Where a1 and a2 are preset thresholds, ∈ = 1×10-6 to avoid division by zero errors. Finally, feature decomposition is performed to output the structure component MSF and texture component DTF:

[0057] MSF=w⊙I filtered +(1-w)⊙I,DTF=I-MSF

[0058] The design preserves original details in smooth areas (λ<0.25), suppresses noise in areas of drastic changes (λ>0.5), and uses linear transitions in the intermediate states.

[0059] The Gabor encoder extracts direction-sensitive multi-scale texture features from the texture component DTF, which is implemented by a learnable log-Gabor filter bank. The filter is defined in the frequency domain as:

[0060]

[0061] Where f0 is the center frequency, corresponding to different scale parameters, σ f =0.75f0,θ k is the direction angle, σ θ =π / 8. In the spatial domain, the spatial domain filter kernel G is generated by inverse Fourier transform s (x,y), its envelope function and carrier function are:

[0062]

[0063] where x θ =xcosθ+ysinθ,y θ =-xsinθ+ycosθ.

[0064] In order to perform multi-scale feature extraction, the DTF is filtered in multiple directions and the response maps are spliced.

[0065] Cross-modal twin encoder module: The cross-modal twin encoder consists of a SAR branch and an optical branch. It uses a two-stream architecture to process the features of the SAR and optical modalities respectively, enhances the modality-specific expression through the attention mechanism, and then fuses the structure component (MSF) and texture component (DTF) to generate semantically aligned cross-modal features. Finally, cross-modal feature alignment is achieved through the modality compensation dictionary and dynamic attention mechanism. Its core structure is as follows:

[0066] The input image first passes through the shared underlying convolutional network to extract basic features, and then strengthens key features based on modal differences. The SAR branch focuses on the scattering intensity distribution and dynamically calibrates the importance of feature channels through channel attention; the optical branch emphasizes high-frequency details (such as building edges) and focuses on key areas through spatial attention. Both branches use pre-activated residual blocks to deepen feature expression and avoid gradient vanishing. Texture features extracted with Gabor encoder Splicing along the channel dimension to form a 512-dimensional fusion feature This feature encodes both cross-modal common structure and modality-specific texture, providing robust representation primitives for subsequent matching. Finally, cross-modal feature compensation is performed on the different modal features to achieve feature alignment. The two-branch structure can be modeled as follows:

[0067] Channel attention weight:

[0068] α c =σ(W2·ReLU(W1·GAP(F low )))

[0069] Spatial Attention Mask:

[0070] M s =σ(Conv 7×7 (Concat(MaxPool(F low ),AvgPool(F low ))))

[0071] Residual block calculation:

[0072] F res =Conv(ReLU(InstanceNorm(F modal )))+F modal

[0073] Channel attention is used to strengthen the channels related to SAR scattering characteristics, and spatial attention is used to focus on the edge details of the optical image to achieve modality-specific feature optimization. In addition, sharing the underlying convolution can reduce redundant parameters and lightweight design of the attention module to balance the computational overhead. The structure-texture features are explicitly separated to provide highly discriminative input for subsequent dynamic convolution matching. By adding a learnable cross-modal dictionary module, 64 512-dimensional feature primitives are included for both optical and SAR modalities. Cross-modal feature compensation is achieved through a multi-head attention mechanism (8 heads), and the feature generation process is as follows:

[0074] Structural feature (MSF) ⊕ texture feature (DTF) → cross-modal dictionary compensation → layer normalization → dynamic convolution fusion

[0075] The cross-modal dictionary module effectively compensates for feature space shifts caused by differences in imaging mechanisms by constructing an explicit mapping relationship between feature primitives between modalities. This module contains two learnable orthogonal sub-dictionaries: a SAR dictionary and an optical dictionary, which encode representative scattering patterns and geometric texture features, respectively. The dictionary is initialized using the He normal distribution, and the independence of entries is constrained by orthogonal regularization terms to ensure decoupling between primitives. During the feature compensation phase, the input features are flattened and then interacted with the target dictionary through an 8-head attention. The compensation strength is dynamically adjusted using a gating coefficient, and the compensation is automatically increased when the average feature-dictionary similarity falls below 0.3.

[0076] Dynamic convolution matching module: The dynamic convolution matching module reconstructs the query features into dynamic convolution kernels (dimensional transformation ), in the library feature map (reconstructed as ) performs group convolution on the traditional sliding window (O(B 2 H g W g H q W q )) complexity is reduced to (O(B 2 H g W g H q W q )) to generate a similarity heat map The heat map is then subjected to a 5×5 Gaussian filter to smooth the noise, and the sub-pixel offset is calculated using a second-order Taylor expansion. Combined peak coordinates ((x p ,y p ))Interpolation obtains normalized center coordinates Achieve end-to-end sub-pixel cross-modal registration, balancing efficiency and accuracy.

[0077] In the above step three, the present invention focuses on cross-modal feature representation learning, and adopts a hybrid loss function to jointly optimize geometric alignment and feature discriminability. During training, the filter parameters of the Gabor texture encoder (encoder.gabor.filters) are frozen, and only the Wiener decomposition module and the attention mechanism layer are fine-tuned. The input data is randomly cropped to generate a 128×128 query image block (SAR or optical modality randomly selected) and a 256×256 library image, and a hard negative sample is generated (the distance from the center of the positive sample is greater than 32 pixels). The model generates a similarity heat map through dynamic convolution matching, calculates GIoU loss, coordinate regression loss and contrast loss, among which the contrast loss extracts positive / negative sample features through ROI Align to calculate cosine similarity. The loss function of the above process is as follows:

[0078]

[0079] This paper uses the AdamW optimizer with an initial learning rate of 3e-4 for training, uses the OneCycleLR scheduler to adjust the learning rate, and adopts mixed precision training acceleration. The number of epochs in the training phase is 200, and the batch size is 16.

Claims

1. A SAR and optical image registration method based on multi-scale feature decomposition and cross-modal fusion, characterized by: include: S1, acquire synthetic aperture radar and optical images; S2. Input the synthetic aperture radar and optical images and the area to be registered into a trained multimodal registration model, and output the actual area corresponding to the area to be registered on another modality image.

2. The synthetic aperture radar and optical image registration method according to claim 1, wherein: In S2, the training process of the multimodal registration model is as follows: Step 1: Acquire registered synthetic aperture radar and optical images as a training dataset; preprocess the training dataset to eliminate modal differences and enhance feature expression; Step 2: Design a multimodal registration network, which specifically includes the following three parts: a multimodal feature decomposition module for separating the structure component (MSF) and texture component (DTF) from the original image; a cross-modal twin encoder for achieving feature semantic alignment; and a dynamic convolutional matching module for efficient matching and coordinate regression. Step 3: Use the registered synthetic aperture radar and optical images to train the multimodal registration network.

3. The synthetic aperture radar and optical image registration method according to claim 2, wherein: In the step 1, the preprocessing includes: eliminating modal differences and enhancing feature expression through a three-stage processing flow of radiation normalization, geometric alignment, and dynamic enhancement.

4. The synthetic aperture radar and optical image registration method according to claim 2, wherein: The second step specifically includes: Construct a multimodal feature decomposition module. Designing a cross-modal twin encoder. Implement dynamic convolution matching mechanism.

5. The synthetic aperture radar and optical image registration method according to claim 2, wherein: The step three specifically includes: Set the geometric alignment loss to GIoU loss; set the coordinate regression loss to smooth L1 loss; set the contrast loss to triple margin loss. The training process uses a two-stage decoupled paradigm: i) freezing the Gabor texture encoder and optimizing only the Wiener decomposition module, attention mechanism, and shared visual encoder; ii) unfreezing all parameters and introducing color jittering and local exposure enhancement. This training process significantly improves the accuracy and generalization of cross-modal remote sensing image registration by decoupling the optimization objective and dynamic training strategy. The number of epochs in the training phase is 200 and the batch size is 16.

Images