Automatic registration system and method for optical-SAR images

Abstract

This invention relates to the field of remote sensing image processing technology, and more particularly to an automatic optical-SAR image registration system and method. A dual-feature extraction module extracts features from the input optical image and its corresponding SAR image; an attention fusion module performs a weighted fusion operation on the obtained optical and SAR features based on attention weight extraction; a multi-scale radiometric module performs transformation predictions on the obtained fused enhancement features to different degrees, integrates and transforms the multiple registration matrices to obtain an affine matrix; and a spatial transformation module performs affine transformation sampling on the SAR image based on the affine matrix to obtain a registered SAR image with the same size as the optical image. This invention achieves effective alignment across modal feature spaces through the dual-feature extraction module and the attention fusion module, and robustly predicts the affine geometric transformation parameters between images through the multi-scale radiometric module and the spatial transformation module, taking into account both global registration and local detail correction.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology, and particularly relates to an automatic registration system and method for optical-SAR images. Background Technology

[0002] With the rapid development of remote sensing technology and satellite observation capabilities, a range of multi-sensor, multi-modal data can be acquired through various Earth observation satellites. Among these, optical imagery and synthetic aperture radar (SAR) imagery are two of the most representative data types. Optical imagery provides detailed texture, spectral, and color information, while SAR imagery possesses strong penetration and stable geometric structure information; the two are significantly complementary in information representation. In tasks such as multi-source remote sensing data fusion, ground feature identification, disaster monitoring, and change detection, optical-SAR image registration is the most fundamental and crucial step. The goal of this task is to precisely align two heterogeneous images from optical and SAR sensors in space, providing geometrically consistent input for subsequent fusion analysis.

[0003] However, due to the fundamental differences in imaging mechanisms between the two types of images, there are severe radiometric nonlinear distortion and geometric aberration problems between optical images and SAR images. Traditional registration methods face significant shortcomings in dealing with this cross-modal difference, mainly in the following aspects:

[0004] (1) Feature mismatch problem: Traditional feature point-based algorithms (such as SIFT, SURF, etc.) assume that the images to be registered have similar gray-scale distribution and texture structure. However, optical and SAR images differ greatly in brightness and texture features, resulting in a very low repetition rate of feature point detection and matching, and unstable registration results.

[0005] (2) Inaccurate similarity measurement: Region-based methods rely on similarity measurement (such as mutual information or normalized cross-correlation). However, under strong nonlinear gray-scale differences, mutual information and other measurements are still difficult to accurately reflect the geometric correspondence between images, leading to mismatch or local misalignment.

[0006] (3) Complex transformations are difficult to model: Traditional methods often use single-scale geometric transformation models (such as affine or rigid transformations), which cannot handle the multi-scale local deformations and structural inconsistencies between optics and SAR, especially in complex surface scenes such as cities and mountains;

[0007] (4) Limitations of deep learning methods: In recent years, deep learning methods have been gradually applied to the study of image registration, but most methods only achieve single-modal or local block matching and lack end-to-end integrated design; some methods can only predict translation parameters and have poor robustness to complex transformations such as rotation and scaling; and some methods rely on style transfer or pseudo-registration samples and have insufficient generalization performance. Summary of the Invention

[0008] In view of this, the present invention aims to provide an automatic optical-SAR image registration system and method, which achieves modality-adaptive semantic feature encoding through a two-way feature extraction architecture; enhances the key feature response through multi-scale convolution and channel weighting to improve the fusion effect of cross-modal features; performs affine transformation between predicted images of different scales to achieve global alignment and local refinement; and finally, introduces a structure-preserving interpolation strategy during spatial transformation to ensure the continuity of image edges and textures after registration.

[0009] To achieve the above objectives, the technical solution created by this invention is implemented as follows:

[0010] An automatic optical-SAR image registration system includes: a dual feature extraction module for extracting features from an input optical image and its corresponding SAR image to obtain optical features and SAR features; an attention fusion module for performing a weighted fusion operation on the optical features and SAR features based on attention weight extraction to obtain fused enhanced features; a multi-scale radiometric module for performing transformation predictions on the fused enhanced features to obtain corresponding registration matrices, and integrating and transforming multiple registration matrices to obtain an affine matrix; and a spatial transformation module for performing affine transformation sampling on the SAR image based on the affine matrix, so that the SAR image is mapped to the target space corresponding to the optical image to obtain a registered SAR image with the same size as the optical image.

[0011] Furthermore, in the dual feature extraction module, a ResNet network is used as the backbone network to extract features from optical images and SAR images respectively, thereby obtaining optical features and SAR features.

[0012] Furthermore, in the attention fusion module: after fusing optical features and SAR features, multi-scale feature extraction is performed on the fused features; after fusing the multi-scale features, attention operations are performed in the channel direction, and the obtained attention weights are divided into channels according to the number of channels of the optical features and SAR features in the fused features to obtain optical weights and SAR weights; the optical weights and SAR weights are applied to the corresponding optical features and SAR features respectively to enhance the optical features and SAR features, and the enhanced features are fused to obtain the fused enhanced features.

[0013] Furthermore, the channel-oriented attention operation includes: performing channel-oriented pooling on the input features, followed by fully connected operations and sigmoid activation operations to obtain attention weights.

[0014] Furthermore, in the multi-scale radiation module: pooling and convolution at different scales are performed on the fused enhancement features to complete the transformation prediction of the fused enhancement features to different degrees, and the corresponding registration matrix is ​​obtained; after concatenating the multiple registration matrices, the concatenated result is input into the multilayer perceptron for matrix mapping transformation to obtain the affine matrix.

[0015] Furthermore, in the spatial transformation module: branch judgment is performed based on the height dimension of the SAR image: when the height H of the SAR image is greater than or equal to a preset threshold: the SAR image is first downsampled to make its size consistent with the size of the affine matrix; a sampling grid is generated according to the affine matrix and affine transformation sampling is performed on the downsampled image; then, structure-preserving interpolation and upsampling processing are performed on the transformed image to obtain a registered SAR image; when the height H of the SAR image is less than the preset threshold, affine transformation sampling is directly performed on the SAR image according to the affine matrix, and structure-preserving interpolation processing is performed on the transformed image to obtain a registered SAR image.

[0016] An automatic registration method for optical-SAR images includes:

[0017] S1: Obtain a dataset containing optical images, matched SAR images, and corresponding affine matrices. Preprocess the optical and SAR images to obtain a training set.

[0018] S2: Construct the optical-SAR image automatic registration system as provided in this invention, and train the automatic registration system using the training set obtained in step S1 to obtain the automatic registration model;

[0019] S3: Input the SAR image to be registered and the optical image used as a reference into the automatic registration model obtained in step S2 to obtain a registered SAR image that matches the optical image.

[0020] Furthermore, the preprocessing process in step S1 includes: performing data augmentation on the optical image and the SAR image; performing random transformation on the augmented SAR image to obtain a deformed SAR image corresponding to the optical image; and integrating the optical image, its corresponding deformed SAR image, and the matched SAR image into a training set.

[0021] Furthermore, in step S2: the optical image and its corresponding deformed SAR image are used as training inputs, and the matched SAR image is used as training outputs. The automatic registration system is trained through joint optimization of affine parameter supervision, feature point consistency, cross-modal structural similarity and reference preservation constraints to obtain an automatic registration model.

[0022] Furthermore, in step S2, the model is trained using the loss function expressed in the following formula:

[0023] L total =αL matrix +βL sim +γL fp +δL ref ;

[0024] Among them, L total L represents the total loss function of the trained model. matrix L represents the supervised loss of the affine matrix parameters. sim L represents the cross-modal structural similarity loss. fp L represents the geometric consistency loss of feature points. ref The reference retention loss is represented by α, β, γ, and δ, which represent the weight loss.

[0025] Affine matrix supervision loss L matrix As shown in the following formula:

[0026] ;

[0027] Among them, A pred Let A represent the affine matrix predicted by the model. gt This represents the corresponding real affine matrix;

[0028] Feature point geometric consistency loss L fp As shown in the following formula:

[0029] ;

[0030] in, Represents based on affine matrix A pred The affine transformation operation, where N represents the number of keypoints detected by the Harris algorithm. This represents the i-th keypoint detected by the Harris algorithm in the registered SAR image. This represents the i-th key point detected by the Harris algorithm in the optical image;

[0031] Cross-modal structural similarity loss L sim As shown in the following formula:

[0032] ;

[0033] Wherein, SSIM represents structural similarity. This indicates a registered SAR image. Represents an optical image in grayscale;

[0034] Reference retention loss L ref As shown in the following formula:

[0035] ;

[0036] Where, λ mse and λ ssim They represent the weighting coefficients, This represents a deformed SAR image.

[0037] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0038] (1) The optical-SAR image automatic registration system and method described in this invention adopts a dual-branch multimodal feature extraction network to learn the semantic representation of optical and SAR respectively; an attention fusion module is introduced to highlight the common structure across modalities (such as edges and shapes), which significantly enhances the cross-modal feature expression capability, and optical and SAR images can be directly aligned in deep space, avoiding the problems of "inability to match" and "sparse feature points" in traditional methods;

[0039] (2) This invention learns implicit mapping relationships through deep networks, without directly relying on fragile gray-scale similarity such as mutual information; the feature fusion stage extracts more accurate structural information through attention mechanism, and has a wider range of applications; at the same time, this invention designs an end-to-end deep learning framework, which enables feature extraction, fusion, geometric prediction and spatial transformation to be completed in one go.

[0040] (3) The present invention uses a spatial transformation module to progressively regress transformation parameters; global positioning and local correction are achieved through coarse-to-fine prediction, resulting in higher accuracy of the final affine modeling and the ability to handle complex deformations such as rotation, scaling, and shearing; and the spatial transformation module is an improvement on the original spatial transformer (STN), adopting a structure-preserving interpolation strategy (adding gradient information) to make the edges clearer, the structure more complete, and the robustness greatly improved.

[0041] (4) The present invention establishes a unified optimization by establishing a multi-constraint joint loss (geometric, structural, reference preservation, etc.), so that the error will not be propagated or amplified at different stages, and the overall accuracy and stability are better; it can be directly put into application after training. Attached Figure Description

[0042] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0043] Figure 1 A schematic diagram of the framework of the optical-SAR image automatic registration system described in the embodiment of the present invention;

[0044] Figure 2 A schematic diagram illustrating the feature processing of the dual feature extraction module described in the embodiments of the present invention;

[0045] Figure 3A schematic diagram illustrating the feature processing of the attention fusion module as described in an embodiment of the present invention;

[0046] Figure 4 A schematic diagram illustrating the feature processing of the multi-scale radiation module described in the embodiments of the present invention;

[0047] Figure 5 A schematic diagram illustrating the feature processing of the spatial transformation module described in an embodiment of the present invention;

[0048] Figure 6 This is a schematic flowchart of the automatic optical-SAR image registration method described in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0050] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0051] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0052] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0053] like Figure 1As shown in the embodiment of the present invention, the automatic optical-SAR image registration system includes a dual-feature extraction module, an attention fusion module, a multi-scale radiometric module, and a spatial transformation module. The dual-feature extraction module extracts features from the input optical image and its corresponding SAR image, obtaining optical features and SAR features respectively. The attention fusion module performs a weighted fusion operation on the optical and SAR features based on attention weight extraction to obtain fused enhanced features. The multi-scale radiometric module performs transformation predictions on the fused enhanced features to different degrees, obtaining corresponding registration matrices. After integrating and transforming multiple registration matrices, an affine matrix is ​​obtained. The spatial transformation module performs affine transformation sampling on the SAR image based on the affine matrix, mapping the SAR image to the target space corresponding to the optical image, resulting in a registered SAR image with the same size as the optical image. This invention adopts an end-to-end integrated neural network architecture, achieving high-precision geometric alignment between optical and SAR images through the collaborative design of multi-modal feature extraction, attention enhancement, multi-scale affine transformation prediction, and an improved spatial transformation module.

[0054] The dual-feature extraction module performs deep feature encoding on both optical and SAR images, mapping the two modalities to a unified semantic feature space. The goal of this module is to extract multi-layered semantic features from optical and SAR images that are comparable, discriminative, and robust, providing a foundation for subsequent cross-modal fusion and geometric modeling. Specifically, in some embodiments, the dual-feature extraction module uses a ResNet network as the backbone to extract features from both optical and SAR images, resulting in corresponding optical and SAR features. The dual-feature extraction module provided in this embodiment is as follows: Figure 2 As shown, the optical image uses RGB three-channel input, and the SAR image is a single-channel intensity map. A ResNet-18 network is used as the backbone network to extract optical and SAR features. Furthermore, this embodiment extracts shallow features from both images through convolution, batch normalization, ReLU activation, and max pooling. These shallow features are then input into the ResNet-18 network. To adapt to different modal feature distributions, the number of input channels for the first convolution layer of the ResNet-18 network in the SAR branch is adjusted from 3 to 1, while the rest of the structure remains the same. The ResNet-18 network used in this embodiment contains a four-stage residual network structure (Layer 1–Layer 4). In each stage, semantic information is extracted layer by layer through convolution, batch normalization, and ReLU activation. Spatial downsampling is achieved between stages through convolution with a stride of 2, gradually increasing the number of channels and outputting the optical feature F. opt and SAR feature F sarAfter standardization, both images enter the attention fusion module. The dual feature extraction module, addressing the imaging differences between optical and SAR images, employs a two-path feature extraction architecture to achieve modality-adaptive semantic feature encoding.

[0055] The attention fusion module enhances key feature responses and improves cross-modal feature fusion by using multi-scale convolution and channel weighting. Specifically, it enhances discriminative features and suppresses noise and redundant information by adaptively adjusting the weights of cross-modal feature channels, thereby improving feature discrimination in registration tasks. In some embodiments, the feature processing of the attention fusion module includes: fusing optical and SAR features; extracting multi-scale features from the fused features; performing channel-oriented attention operations on the fused multi-scale features; dividing the resulting attention weights according to the number of channels in the fused features for both optical and SAR features to obtain optical and SAR weights; applying the optical and SAR weights to the corresponding optical and SAR features respectively to enhance them; and then fusing the enhanced features to obtain fused enhanced features. The channel-oriented attention operation includes: performing channel-oriented pooling on the input features, followed by fully connected operations and sigmoid activation operations to obtain attention weights.

[0056] The attention fusion module provided in the embodiments of the present invention, such as Figure 3 As shown, optical feature F opt and SAR features F sar First, perform 1×1 convolution on each feature, then concatenate the channels of the two convolutional features to complete the feature fusion of optical and SAR features, resulting in the fused feature F. cat The fused feature F is extracted in parallel through convolution operations with different kernel sizes (1×1, 3×3, 5×5). cat The local and global context information is used. After each convolution operation, the features are batch normalized and ReLU activated before being added and fused to complete multi-scale feature extraction and fusion. The feature F after multi-scale feature fusion is then... ms Attention operations are performed along the channel direction to obtain attention weights. Specifically, this involves average pooling along the channel direction on the multi-scale feature fusion, followed by fully connected operations, ReLU activation, and another fully connected operation on the pooled features. The processed features are then activated by a sigmoid operation to obtain the attention weights Att. This process is as follows:

[0057] ;

[0058] Where Att represents attention weight, σ represents sigmoid activation, W1 and W2 represent the weights of the two fully connected operations, and GAP represents average pooling.

[0059] The fused features are weighted according to the attention weight Att, generating the corresponding optical feature weights A. opt and SAR feature weights A sar Specifically, the channel attention weights are mapped to optical feature channels and SAR feature channels respectively along the channel dimension, and the attention responses of the two types of channels are normalized to obtain the corresponding optical feature weights and SAR feature weights. The optical feature weights are used to modulate the influence of the optical feature channels during the fusion process, and the SAR feature weights are used to modulate the influence of the SAR feature channels during the fusion process, so that the optical and SAR features form a relative proportional relationship during fusion. The network adaptively determines the weights of each feature based on the features of the input image.

[0060] Optical weight A opt and SAR weight A sar Optical and SAR features are applied separately after convolution to enhance them. Then, the enhanced features are fused by first concatenating channels and then convolving them. Finally, after batch normalization and ReLU activation, the fused enhanced feature F is obtained. fusion .

[0061] The multi-scale radiometric module adopts a coarse-to-fine hierarchical structure, progressively regressing the affine transformation parameters between optical and SAR images to achieve global and local alignment and local refinement. Specifically, in some embodiments, the feature processing of the multi-scale radiometric module includes: performing pooling and convolution at different scales on the fused enhancement features to predict different degrees of transformation of the fused enhancement features, obtaining the corresponding registration matrices; stitching the multiple registration matrices together, and inputting the stitched result into a multilayer perceptron for matrix mapping transformation to obtain the affine matrix.

[0062] The multi-scale radiation module provided in the embodiments of the present invention, such as Figure 4 As shown, for the fusion enhancement feature F fusion After performing average pooling at scales of 1, 4, and 8, 1×1 convolutions are then performed on the features after each of the three pooling steps to predict different degrees of transformation for the fused and enhanced features, resulting in corresponding registration matrices. The multiple registration matrices are then concatenated, and the concatenated result is input into a multilayer perceptron for matrix mapping transformation, yielding an affine matrix containing six degrees of freedom.

[0063] .

[0064] Among them, (a 11 ,a 12 ,a 21 ,a 22 () represents the rotation, scaling, and shearing parameters, (t) x ,t y ) represents the translation parameter.

[0065] The spatial transformation module provided by this invention introduces a hierarchical geometric correction mechanism and structure preservation constraints on the basis of a standard spatial transformation network to achieve high-precision resampling and geometric correction of SAR images. Based on the affine matrix, a sampling grid in the target coordinate system is generated, and the input SAR image is mapped to the target space corresponding to the optical image to ensure the continuity of structure and texture in the registered image.

[0066] In some embodiments, the processing flow of the spatial transformation module is as follows: Figure 5 As shown, branching is performed based on the height dimension of the SAR image:

[0067] When the height H of the SAR image is greater than or equal to a preset threshold (e.g., 512), the SAR image is first downsampled to make its size consistent with the size of the affine matrix; then a sampling grid is generated according to the affine matrix and affine transformation sampling is performed on the downsampled image; then structure-preserving interpolation and upsampling are performed on the transformed image to obtain the registered SAR image.

[0068] When the height H of the SAR image is less than the preset threshold, affine transformation sampling is directly performed on the SAR image according to the affine matrix, and structure-preserving interpolation is performed on the transformed image to obtain the registered SAR image. Affine transformation sampling generates a sampling grid based on the affine matrix, which is used to determine the sampling position of each pixel in the target image in the source image, and pixel resampling is completed by interpolation.

[0069] This invention also provides an automatic registration method for optical-SAR images, combined with Figure 1 and Figure 6 ,include:

[0070] S1: Obtain a dataset containing optical images, matched SAR images, and corresponding affine matrices. Preprocess the optical and SAR images to obtain a training set.

[0071] In some embodiments, the preprocessing process includes: performing data augmentation on the optical image and the SAR image; performing random transformation on the augmented SAR image to obtain a deformed SAR image corresponding to the optical image; and integrating the optical image, its corresponding deformed SAR image, and the matched SAR image into a training set.

[0072] S2: Construct the optical-SAR image automatic registration system as provided by the present invention, and train the automatic registration system using the training set obtained in step S1 to obtain the automatic registration model.

[0073] In some embodiments, in step S2, the optical image and its corresponding deformed SAR image are used as training input, and the matched SAR image is used as training output. Furthermore, to improve the training stability and physical plausibility of the model, this invention designs a multi-constraint joint loss function system to comprehensively optimize geometric accuracy and structural consistency. During the training phase, the automatic registration system is trained through joint optimization of affine parameter supervision, feature point consistency, cross-modal structural similarity, and reference-preserving constraints, resulting in an automatic registration model that improves the model's convergence stability and registration accuracy.

[0074] Specifically, the model is trained using the following loss function:

[0075] L total =αL matrix +βL sim +γL fp +δL ref ;

[0076] Among them, L total L represents the total loss function of the trained model. matrix L represents the supervised loss of the affine matrix parameters. sim L represents the cross-modal structural similarity loss. fp L represents the geometric consistency loss of feature points. ref The reference retention loss is represented by α, β, γ, and δ, which represent the weight losses. The weight losses α, β, γ, and δ can be dynamically adjusted during training.

[0077] Specifically, to ensure that the network can accurately predict geometric transformation parameters, this invention modifies the affine matrix A predicted by the model. pred Its corresponding real affine matrix A gt Applying a mean square error constraint, we obtain the affine matrix supervision loss L as follows: matrix :

[0078] ;

[0079] Affine matrix supervision loss L matrix It acts directly on the parameter space, providing a strong supervision signal to the network and ensuring, from a numerical perspective, that the predicted transformation matrix approximates the true value.

[0080] Considering the significant local structural correspondence between optical and SAR images, this invention utilizes the Harris algorithm for corner detection to construct a geometric supervision signal. Specifically, the coordinates of key points in the registered SAR image are processed by an affine matrix A.pred After transformation, calculate the geometric consistency loss L between the feature points and the corresponding keypoints in the optical image. fp :

[0081] ;

[0082] in, Represents based on affine matrix A pred The affine transformation operation, where N represents the number of keypoints detected by the Harris algorithm. This represents the i-th key point in the registered SAR image. This represents the i-th key point in the optical image.

[0083] Since optical images and SAR images have significant modal differences, traditional pixel intensity difference measures are not applicable. Therefore, this invention adopts the following cross-modal structural similarity loss L. sim Measuring cross-modal structural consistency between registered SAR and optical images:

[0084] ;

[0085] Wherein, SSIM represents structural similarity. This indicates a registered SAR image. Represents an optical image in grayscale;

[0086] This loss can constrain the registration results from three dimensions: brightness, contrast, and structure, effectively mitigating the registration bias caused by cross-modal differences.

[0087] This invention also introduces a reference retention loss L ref This is primarily to prevent the model from over-relying on the optical image structure during training, which could lead to the loss of inherent feature information in the SAR image. The constrained registration of the SAR image ensures that it aligns with the optical image while maintaining consistency with its original structure. Specifically, it is defined as follows:

[0088] ;

[0089] Where λmse and λssim represent the weighting coefficients, respectively. Represents a deformed SAR image. Reference preservation loss L ref While suppressing overfitting, it ensures the physical consistency between SAR images before and after registration.

[0090] In the early stages of model training, the overall geometric deviation is relatively large, and loss terms related to global constraints are given high weights to guide the network to prioritize learning stable global geometric relationships. As training progresses, the global registration error gradually decreases, and the weight coefficients adaptively shift towards loss terms that characterize local structural consistency, thereby strengthening the optimization of local alignment errors. Through this approach, a progressive learning process from global constraints to local optimization is achieved within the same training framework, enabling the network to balance accuracy, stability, and structural fidelity during registration.

[0091] S3: Input the SAR image to be registered and the optical image used as a reference into the automatic registration model obtained in step S2 to obtain a registered SAR image that matches the optical image.

[0092] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0093] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An automatic registration system for optical-SAR images, characterized in that, include: The dual feature extraction module extracts features from the input optical image and its corresponding SAR image, obtaining optical features and SAR features respectively. The attention fusion module performs a weighted fusion operation on optical features and SAR features based on attention weight extraction to obtain fused enhanced features; The multi-scale affine module performs transformation predictions on the fused enhancement features to different degrees to obtain the corresponding registration matrices. After integrating and transforming multiple registration matrices, the affine matrix is ​​obtained. The spatial transformation module performs affine transformation sampling on the SAR image based on the affine matrix, so that the SAR image is mapped to the target space corresponding to the optical image, and a registered SAR image with the same size as the optical image is obtained.

2. The optical-SAR image automatic registration system according to claim 1, characterized in that, In the dual feature extraction module, a ResNet network is used as the backbone network to extract features from optical images and SAR images respectively, thus obtaining optical features and SAR features.

3. The optical-SAR image automatic registration system according to claim 1, characterized in that, In the attention fusion module: After fusing optical features and SAR features, multi-scale feature extraction is performed on the fused features. After fusing multi-scale features and performing channel-oriented attention operations, the resulting attention weights are divided into channels according to the number of channels in the fused features for optical and SAR features, thus obtaining optical weights and SAR weights. Optical weights and SAR weights are applied to the corresponding optical features and SAR features respectively, and the enhanced features are then fused to obtain the fused enhanced features.

4. The automatic optical-SAR image registration system according to claim 3, characterized in that, Channel-oriented attention operations include: After channel-oriented pooling of the input features, fully connected operations and sigmoid activation operations are performed sequentially to obtain attention weights.

5. The optical-SAR image automatic registration system according to claim 1, characterized in that, In the multi-scale affine module: Pooling and convolution at different scales are performed on the fused enhanced features to perform transformation predictions of different degrees on the fused enhanced features, and the corresponding registration matrix is ​​obtained. After concatenating the multiple registration matrices, the concatenated result is input into a multilayer perceptron for matrix mapping transformation to obtain an affine matrix.

6. The optical-SAR image automatic registration system according to claim 1, characterized in that, In the space transformation module: Branching based on the height dimension of the SAR image: When the height H of the SAR image is greater than or equal to a preset threshold: first, the SAR image is downsampled to make its size consistent with the size of the affine matrix; then, a sampling grid is generated according to the affine matrix and affine transformation sampling is performed on the downsampled image; finally, structure-preserving interpolation and upsampling processing are performed on the transformed image to obtain the registered SAR image. When the height H of the SAR image is less than the preset threshold, the SAR image is directly sampled by affine transformation according to the affine matrix, and the transformed image is subjected to structure-preserving interpolation to obtain the registered SAR image.

7. An automatic registration method for optical-SAR images, characterized in that, include: S1: Obtain a dataset containing optical images, matched SAR images, and corresponding affine matrices. Preprocess the optical and SAR images to obtain a training set. S2: Construct the optical-SAR image automatic registration system as described in any one of claims 1 to 6, and train the automatic registration system using the training set obtained in step S1 to obtain an automatic registration model; S3: Input the SAR image to be registered and the optical image used as a reference into the automatic registration model obtained in step S2 to obtain a registered SAR image that matches the optical image.

8. The automatic registration method for optical-SAR images according to claim 7, characterized in that, The preprocessing process in step S1 includes: Data augmentation of optical and SAR images; The enhanced SAR image is randomly transformed to obtain a deformed SAR image corresponding to the optical image; The optical images, along with their corresponding deformed SAR images and matched SAR images, are integrated into a training set.

9. The automatic registration method for optical-SAR images according to claim 8, characterized in that, In step S2: Using optical images and their corresponding deformed SAR images as training inputs and matched SAR images as training outputs, the automatic registration system is trained through joint optimization of affine parameter supervision, feature point consistency, cross-modal structural similarity, and reference preservation constraints to obtain an automatic registration model.

10. The automatic registration method for optical-SAR images according to claim 8, characterized in that, In step S2, the model is trained using the loss function expressed in the following formula: L total =αL matrix +βL sim +γL fp +δL ref ; Among them, L total L represents the total loss function of the trained model. matrix L represents the supervised loss of the affine matrix parameters. sim L represents the cross-modal structural similarity loss. fp L represents the geometric consistency loss of feature points. ref The reference retention loss is represented by α, β, γ, and δ, which represent the weight loss. Affine matrix parameter supervision loss L matrix As shown in the following formula: ； Among them, A pred Let A represent the affine matrix predicted by the model. gt This represents the corresponding real affine matrix; Feature point geometric consistency loss L fp As shown in the following formula: ； in, Represents based on affine matrix A pred The affine transformation operation, where N represents the number of keypoints detected by the Harris algorithm. This represents the i-th keypoint detected by the Harris algorithm in the registered SAR image. This represents the i-th keypoint detected by the Harris algorithm in the optical image; Cross-modal structural similarity loss L sim As shown in the following formula: ； Wherein, SSIM represents structural similarity. This indicates a registered SAR image. Represents an optical image in grayscale; Reference retention loss L ref As shown in the following formula: ； Where, λ mse and λ ssim They represent the weighting coefficients, This represents a deformed SAR image.

Images