An optical and SAR image registration method based on geometric prior guidance

Abstract

This invention discloses a geometric prior-guided optical and SAR image registration method, belonging to the field of remote sensing image processing technology. The method includes: S1 extracting multi-scale, multi-level deep feature maps from both the original optical and SAR images; S2 estimating affine transformation parameters representing the image's relationship to global spatial transformations based on the top-level semantic feature map in the deep feature maps; S3 encoding the affine transformation parameters into geometric-aware tokens and injecting them into the cross-attention module of the fine registration network to obtain a fused feature map incorporating geometric prior information; S4 predicting the local residual displacement field based on the fused feature map, fusing the local residual displacement field with the global affine displacement field to obtain the total displacement field, and using the total displacement field to perform a resampling of the original SAR image to obtain the registered image. This invention achieves high-precision and high-stability cross-modal image registration under strong modal differences.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a method for optical and SAR image registration based on geometric prior guidance. Background Technology

[0002] High-precision registration of optical images with synthetic aperture radar (SAR) images is a core prerequisite technology for applications such as multi-source remote sensing data fusion and surface change detection. Due to fundamental differences in their imaging mechanisms, optical images, as a passive imaging method, possess clear surface texture and spectral characteristics; SAR images, as an active microwave imaging method, are susceptible to speckle noise contamination and are accompanied by geometric distortions such as overlay, perspective contraction, and shadows, resulting in a significant modal gap between the two. In actual observation scenarios, optical-SAR image pairs often exhibit large-scale, nonlinear geometric deformations, encompassing extensive rotation, translation, scaling, and local nonlinear distortions, further exacerbating the registration difficulty.

[0003] Current deep learning-based heterogeneous image registration methods generally employ a coarse-fine registration cascade architecture: first, an affine transformation network estimates global transformation parameters and performs explicit spatial transformations on the images to achieve pre-alignment; then, the coarse-registered image is input into a fine-registration network to solve for the local displacement field and complete fine registration. This architecture faces the following key technical challenges when dealing with large-scale geometric deformations and strong modal differences: when image pairs have large spatial offsets and rotations, the affine transformation in the coarse registration stage cannot completely eliminate global biases. The fine-registration network is limited by a fixed receptive field and local search window, making it easy for homonymous features to exceed the effective matching range, resulting in the inability to establish correct matches. Furthermore, modal differences interfere with geometric feature learning, and strong modal differences make it difficult for the model to directly learn modality-invariant geometric representations from the original pixel space. In the cascade architecture, coarse registration errors are directly propagated to the fine registration stage, leading to error accumulation, amplifying geometric perception biases, and significantly reducing the performance of the registration model.

[0004] In summary, under the influence of large-scale geometric deformation and modal differences, existing methods struggle to accurately estimate global spatial transformations and local nonlinear deformations, generally suffering from insufficient feature matching stability and limited registration accuracy. Therefore, embedding geometric prior depth to guide feature learning, while avoiding error accumulation caused by direct image transformation, can effectively improve the registration accuracy and applicability of optical-SAR images. Summary of the Invention

[0005] The purpose of this invention is to provide a geometric prior-guided optical and SAR image registration method to improve the registration accuracy and matching stability of optical and SAR images when facing large-scale geometric deformation and modal differences.

[0006] To achieve the above objectives, this invention provides a geometrically prior-guided optical and SAR image registration method, comprising the following steps: S1. Extract multi-scale and multi-level deep feature maps from the original optical image and the original SAR image respectively; S2. Based on the top-level semantic feature map in the deep feature map, the affine transformation parameters that characterize the relationship between the image and the global spatial transformation are estimated. S3. Encode the affine transformation parameters into geometric perception tokens, and inject the geometric perception tokens into the cross attention module of the fine registration network to obtain a fused feature map that incorporates geometric prior information. S4. Based on the fusion feature map, predict the local residual displacement field, fuse the local residual displacement field with the global affine displacement field generated by the affine transformation parameters to obtain the total displacement field, and use the total displacement field to perform a resampling on the original SAR image to obtain the registered image.

[0007] Preferably, step S1 specifically includes: S11. Construct a convolutional neural network encoder with dual-branch weights that do not share weights. The convolutional neural network encoder includes an optical branch encoder and a SAR branch encoder. S12, the optical branch encoder adopts the ResNet-18 structure, which includes four feature extraction stages. The number of output channels of the four feature extraction stages are configured as 64, 128, 256 and 512 respectively. While maintaining the same main topology as ResNet-18, the S13 and SAR branch encoders optimize the first 7×7 convolutional layer into three cascaded 3×3 convolutional layers. S14. Extract feature maps from stage 2 as spatial detail support for fine registration, and extract top-level semantic feature maps from stage 4 as guiding support for global geometric prior estimation.

[0008] Preferably, step S2 specifically includes: S21. Dimensionality reduction is performed on the optical top-level semantic feature map and the SAR top-level semantic feature map using 1×1 convolutional layers to extract key semantic information from 256 channels. S22. Perform global average pooling on the dimensionality-reduced feature maps to obtain two global descriptor vectors; S23. Concatenate the two global descriptor vectors and input them into the regression head network consisting of three fully connected layers. The output is 6-DOF affine transformation parameters. ,in This is the scaling factor in the X-axis direction. The shear coefficient is located in the X-axis direction. The shear coefficient is located in the Y-axis direction. This is the scaling factor in the Y-axis direction. This represents the horizontal translation amount. This represents the vertical translation amount; S24. Based on the affine transformation parameters Constructing the affine transformation matrix .

[0009] Preferably, S3 specifically includes: S31. Input the affine transformation parameters into the multilayer perceptron and encode them into a geometric perception token with a length of 64. S32. In the cross-attention module of the fine registration network, add sine and cosine position codes to the optical second-stage feature map and the SAR second-stage feature map respectively, and generate query vector, key vector and value vector. S33. Calculate the predicted projection offset of each pixel position using the affine transformation matrix to generate a spatial offset term; S34. Deeply integrate the spatial bias term with the geometric perception token and inject it into the calculation of the cross-attention score; S35. Introduce learnable parameters to adjust the intensity of geometric guidance and output a fused feature map that incorporates prior geometric information.

[0010] Preferably, the spatial bias term is generated by using a sub-network consisting of 1×1 convolutions to deeply fuse the predicted projection offset with the geometric perception token to obtain the spatial bias term.

[0011] Preferably, the method for calculating the cross-attention score in step S34 is one or a combination of the following: It is directly added to the attention score formula as a spatial bias term: ; in, Indicates the feature map height direction coordinate index. This represents the coordinate index along the width direction of the feature map. For spatial bias, Represents the query vector. This represents the transpose of the key vector. This represents the dimension of the key vector. Indicates attention weights, Indicates learnable parameters; Alternatively, it can be used as a modulation factor on the query vector, key vector, or value vector.

[0012] Preferably, step S4 specifically includes: S41. Construct a fine registration decoder based on the U-Net architecture, and receive the fused feature map as input; S42. By performing three upsampling operations involving convolutional layers and linear interpolation, and fusing the second-stage feature map using skip connections, the local residual displacement field is obtained through regression. S43. Generate the global affine displacement field based on the affine transformation parameters; S44. The global affine displacement field and the local residual displacement field are superimposed pixel by pixel to obtain the total displacement field; S45. Perform a bilinear interpolation resampling on the original SAR image using the total displacement field to obtain the registered image.

[0013] Preferably, global average pooling outputs two global descriptor vectors with dimensions of 1×1×256.

[0014] Therefore, this invention employs the aforementioned optical and SAR image registration method based on geometric prior guidance. By injecting global geometric priors into the feature learning process, the network can obtain spatial geometric guidance during the feature matching stage, enabling it to pre-perceive the overall displacement relationship between images at the feature level. This mechanism effectively expands the matching search range, thereby improving the model's registration capability under large-scale geometric deformation conditions and mitigating the matching offset phenomenon that easily occurs in traditional methods under large deformation scenarios.

[0015] The affine transformation parameters estimated in the global geometric prior estimation stage are not directly used for explicit transformation of image pixels. Instead, they are injected as prior information into the feature learning process to guide spatial alignment at the feature level. This strategy avoids the stepwise propagation of coarse registration errors to subsequent fine registration stages in traditional cascaded registration frameworks. Simultaneously, by integrating geometric information into the network as prior constraints, it is prevented from mixing with modal features, thereby reducing the interference of modal differences on feature matching. This allows the network to learn structural features more stably under conditions of strong modal differences, further enhancing the effectiveness of geometric guidance.

[0016] This method performs pixel-level resampling only once in the final output stage, thus effectively preserving the high-frequency information and scattering structure features of the SAR image, avoiding information degradation caused by multiple interpolations. Simultaneously, through a geometrically aware token encoding and injection mechanism, it maintains high computational efficiency while utilizing geometric priors to guide feature matching.

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] Figure 1 This is a flowchart of an optical and SAR image registration method based on geometric prior guidance according to the present invention; Figure 2This is a comparative experimental diagram showing the optical and SAR image registration effects of an embodiment of the optical and SAR image registration method based on geometric prior guidance according to the present invention. Detailed Implementation

[0019] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0020] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0021] Example 1 like Figure 1 As shown, this invention provides a geometric priori-guided optical and SAR image registration method, comprising the following steps: S1: Multi-scale heterogeneous feature extraction Construct a convolutional neural network encoder with two branches and non-shared weights, including an optical branch encoder. With SAR branch encoder .

[0022] The optical branch encoder adopts a ResNet-18 structure, which includes four feature extraction stages, with the number of output channels configured for each stage being 64, 128, 256, and 512, respectively.

[0023] While maintaining the same main topology as ResNet-18, the SAR branch encoder optimizes its first 7×7 convolutional layer into three cascaded 3×3 convolutional layers to expand the receptive field and enhance the nonlinear activation capability for SAR image scattering characteristics.

[0024] Original optical image Compared with the original SAR image Input the optical branch encoder and the SAR branch encoder separately, and extract multi-scale feature maps layer by layer: ; in, Represents the optical image. Stage-based multi-scale feature maps Indicates an optical branch encoder. Represents the SAR image of the first Stage-based multi-scale feature maps Indicates a SAR branch encoder; Feature maps are extracted from stage 2 to support spatial details for fine registration, and top-level semantic feature maps are extracted from stage 4 to guide global geometric prior estimation.

[0025] S2: Global geometric prior estimation The optical top-level semantic feature map and the SAR top-level semantic feature map are reduced in dimensionality using 1×1 convolutional layers to extract key semantic information from 256 channels.

[0026] Global average pooling (GAP) is performed on the dimensionality-reduced feature maps to obtain two global descriptor vectors. Global average pooling compresses the spatial dimension of the feature maps to 1×1, outputting two global descriptor vectors with dimensions of 1×1×256: ; in, Indicates the feature map height. Indicates the width of the feature map. Indicates the feature map height direction coordinate index. This represents the coordinate index along the width direction of the feature map. This represents the global feature vector of the original optical image. express Location feature value This represents the global feature vector of the original SAR image. Represents the real number field. The channel dimension of the global feature vector; The two global descriptor vectors are concatenated along the channel dimension to obtain a joint global feature vector with dimension 512. This joint global feature vector is then input into a regression head network consisting of three fully connected layers. A nonlinear mapping is used to output 6-DOF affine transformation parameters, as shown in the following formula: ; in, This is the scaling factor in the X-axis direction. The shear coefficient is located in the X-axis direction. The shear coefficient is located in the Y-axis direction. This is the scaling factor in the Y-axis direction. This represents the horizontal translation amount. This is the vertical translation amount. Represents the affine transformation parameter vector. This represents an affine parameter regression multilayer perceptron. Represents the joint global eigenvector; This allows us to construct an affine transformation matrix that represents the global coordinate mapping relationship between images. : ; This matrix defines a global topological mapping from the coordinates of the SAR image to be registered to the coordinates of the reference optical image: ; in, Represents the original pixel coordinates of the original SAR image. Represents the pixel coordinates of the corresponding reference optical image after transformation; S3: Geometric Token Injection and Cross-Attention Modulation affine transformation parameters Input a multilayer perceptron Encoded as a geometric perception token of length 64; In the cross-attention module (MCA) of the fine registration network, sine and cosine positional encodings are added to the feature map of the second stage, and query vector Q, key vector K, and value vector V are generated.

[0027] The predicted projection offset for each pixel position is calculated using the affine transformation matrix. For each pixel position in the query vector... The difference between its predicted projected position and the actual position is It utilizes a subnetwork consisting of 1×1 convolutions. Predict the projection offset Deeply integrate with the geometry-aware token to generate spatial bias terms: ; in, Represents pixels From the affine transformation matrix Predict the difference between the projected position and the actual position. Indicates the positional encoding of sine and cosine. Indicates fusion, This represents a multilayer perceptron; Spatial bias term This is incorporated into the calculation of the cross-attention score. The formula for calculating the attention score is: ; in, Represents the query vector. This represents the transpose of the key vector. This represents the dimension of the key vector. Indicates attention weights, This represents a learnable parameter used to adjust the intensity of geometric guidance. The injection method can also act as a modulation factor on the query vector Q, key vector K, or value vector V. The final output is a fused feature map that incorporates geometric prior information.

[0028] S4: Decoupled Flow Field Synthesis and Residual Decoding Building a fine-registration decoder based on the U-Net architecture Receive fused feature map As input.

[0029] The local residual displacement field is obtained by regressing through three upsampling operations involving convolutional layers and linear interpolation, and by fusing the second-stage feature map using skip connections. : ; Based on affine transformation parameters Generate global affine displacement field .

[0030] global affine displacement field With local residual displacement field The total displacement field is obtained by stacking pixels one by one. : ; in, Represents the pixel coordinates in the image.

[0031] The original SAR image is resampled once by bilinear interpolation using the total displacement field to obtain the registered image. Pixel-level resampling is performed only once during the entire registration process to avoid error accumulation and information degradation caused by multiple interpolations.

[0032] To verify the effectiveness of the method of this invention, registration experiments were conducted using two sets of optical and SAR remote sensing images of different land cover types. The experimental results are as follows: Figure 2 As shown in the figure, the first row of test areas represents farmland and rural scenes, and the second row represents road and building scenes. In each row, the left column (light-F) is the optical reference image (Fixed), which serves as the registration benchmark; the middle column (SAR-M) is the original SAR moving image to be registered, which, due to the SAR side-looking imaging mechanism, exhibits significant large-scale geometric distortion, rotation, and perspective shift, resulting in severe misalignment of the ground feature outlines and spatial positions with the optical reference image; the right column (SAR-reg) is the registration result image (Registered) obtained after processing using the geometric prior-guided registration method of this invention.

[0033] from Figure 2 It can be seen intuitively that: Significant global large deformation correction effect: For large-scale nonlinear geometric distortions such as rotation, scaling, and perspective in SAR-M, the method of this invention can accurately correct the spatial misalignment of SAR images through global geometric prior estimation and geometric perception token guidance, so that the overall view and ground feature outline of the registered SAR-reg image are completely aligned with the light-F of the optical reference image, effectively solving the problems of matching failure and error accumulation under large deformation in traditional cascade registration methods.

[0034] Strong adaptability to multiple scenarios: In two typical remote sensing scenarios, farmland and rural areas (first row) and roads and buildings (second row), the method of this invention can achieve high-precision registration. Whether it is a field of farmland or a linear road or building outline, the registered land feature position, orientation and boundary are highly consistent with the optical reference image, which verifies the robustness of the method to different land feature types and different modal differences.

[0035] High registration accuracy and complete information preservation: This invention adopts a decoupled flow field synthesis and single pixel resampling strategy. While completing geometric correction, it completely preserves the scattering characteristics and high-frequency details of SAR images, avoiding image information degradation caused by traditional multiple interpolation and resampling. It provides high-quality registration data for subsequent applications such as multi-source remote sensing data fusion and surface change detection.

[0036] The method of this invention can effectively overcome the problems of strong modal differences and large-scale geometric deformation between optical and SAR images, such as... Figure 2 As shown, compared with the original SAR image to be registered, the SAR image processed by the method of this invention achieves accurate spatial alignment with the optical reference image, with consistent ground feature outlines, positions, and orientations. The registration accuracy and matching stability are significantly improved. At the same time, the single resampling strategy fully preserves the scattering characteristics of the SAR image, avoiding error accumulation and information degradation, and can meet the high-precision requirements of engineering applications such as multi-source remote sensing data fusion and surface monitoring.

[0037] Therefore, this invention employs the aforementioned optical and SAR image registration method based on geometric prior guidance. By injecting global geometric priors into the feature learning process, the network can obtain spatial geometric guidance during the feature matching stage, enabling it to pre-perceive the overall displacement relationship between images at the feature level. This mechanism effectively expands the matching search range, thereby improving the model's registration capability under large-scale geometric deformation conditions and mitigating the matching offset phenomenon that easily occurs in traditional methods under large deformation scenarios.

[0038] The affine transformation parameters estimated in the global geometric prior estimation stage are not directly used for explicit transformation of image pixels. Instead, they are injected as prior information into the feature learning process to guide spatial alignment at the feature level. This strategy avoids the stepwise propagation of coarse registration errors to subsequent fine registration stages in traditional cascaded registration frameworks. Simultaneously, by integrating geometric information into the network as prior constraints, it is prevented from mixing with modal features, thereby reducing the interference of modal differences on feature matching. This allows the network to learn structural features more stably under conditions of strong modal differences, further enhancing the effectiveness of geometric guidance.

[0039] This method performs pixel-level resampling only once in the final output stage, thus effectively preserving the high-frequency information and scattering structure features of the SAR image, avoiding information degradation caused by multiple interpolations. Simultaneously, through a geometrically aware token encoding and injection mechanism, it maintains high computational efficiency while utilizing geometric priors to guide feature matching.

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for optical and SAR image registration based on geometric prior guidance, characterized in that, Includes the following steps: S1. Extract multi-scale and multi-level deep feature maps from the original optical image and the original SAR image respectively; S2. Based on the top-level semantic feature map in the deep feature map, the affine transformation parameters that characterize the relationship between the image and the global spatial transformation are estimated. S3. Encode the affine transformation parameters into geometric perception tokens, and inject the geometric perception tokens into the cross attention module of the fine registration network to obtain a fused feature map that incorporates geometric prior information. S4. Based on the fusion feature map, predict the local residual displacement field, fuse the local residual displacement field with the global affine displacement field generated by the affine transformation parameters to obtain the total displacement field, and use the total displacement field to perform a resampling on the original SAR image to obtain the registered image.

2. The optical and SAR image registration method based on geometric prior guidance according to claim 1, characterized in that, Step S1 specifically includes: S11. Construct a convolutional neural network encoder with dual-branch weights that do not share weights. The convolutional neural network encoder includes an optical branch encoder and a SAR branch encoder. S12, the optical branch encoder adopts the ResNet-18 structure, which includes four feature extraction stages. The number of output channels of the four feature extraction stages are configured as 64, 128, 256 and 512 respectively. While maintaining the same main topology as ResNet-18, the S13 and SAR branch encoders optimize the first 7×7 convolutional layer into three cascaded 3×3 convolutional layers. S14. Extract feature maps from stage 2 as spatial detail support for fine registration, and extract top-level semantic feature maps from stage 4 as guiding support for global geometric prior estimation.

3. The optical and SAR image registration method based on geometric prior guidance according to claim 1, characterized in that, Step S2 specifically includes: S21. Dimensionality reduction is performed on the optical top-level semantic feature map and the SAR top-level semantic feature map using 1×1 convolutional layers to extract key semantic information from 256 channels. S22. Perform global average pooling on the dimensionality-reduced feature maps to obtain two global descriptor vectors; S23. Concatenate the two global descriptor vectors and input them into the regression head network consisting of three fully connected layers. The output is 6-DOF affine transformation parameters. ,in This is the scaling factor in the X-axis direction. The shear coefficient is located in the X-axis direction. The shear coefficient is located in the Y-axis direction. This is the scaling factor in the Y-axis direction. This represents the horizontal translation amount. This represents the vertical translation amount; S24. Based on the affine transformation parameters Constructing the affine transformation matrix .

4. The optical and SAR image registration method based on geometric prior guidance according to claim 3, characterized in that, S3 specifically includes: S31. Input the affine transformation parameters into the multilayer perceptron and encode them into a geometric perception token with a length of 64. S32. In the cross-attention module of the fine registration network, add sine and cosine position codes to the optical second-stage feature map and the SAR second-stage feature map respectively, and generate query vector, key vector and value vector. S33. Calculate the predicted projection offset of each pixel position using the affine transformation matrix to generate a spatial offset term; S34. Deeply integrate the spatial bias term with the geometric perception token and inject it into the calculation of the cross-attention score; S35. Introduce learnable parameters to adjust the intensity of geometric guidance and output a fused feature map that incorporates prior geometric information.

5. The optical and SAR image registration method based on geometric prior guidance according to claim 4, characterized in that, The spatial bias term is generated by using a sub-network consisting of 1×1 convolutions to deeply fuse the predicted projection offset calculated in step S33 with the geometric perception token to obtain the spatial bias term.

6. The optical and SAR image registration method based on geometric prior guidance according to claim 4, characterized in that, The method for calculating the cross-attention score in step S34 is one or a combination of the following: It is directly added to the attention score formula as a spatial bias term: ； in, Indicates the feature map height direction coordinate index. This represents the coordinate index along the width direction of the feature map. For spatial bias, Represents the query vector. This represents the transpose of the key vector. The dimension of the key vector. Indicates attention weights, Indicates learnable parameters; Alternatively, it can be used as a modulation factor on the query vector, key vector, or value vector.

7. The optical and SAR image registration method based on geometric prior guidance according to claim 1, characterized in that, Step S4 specifically includes: S41. Construct a fine registration decoder based on the U-Net architecture, and receive the fused feature map as input; S42. By performing three upsampling operations involving convolutional layers and linear interpolation, and fusing the second-stage feature map using skip connections, the local residual displacement field is obtained through regression. S43. Generate the global affine displacement field based on the affine transformation parameters; S44. The global affine displacement field and the local residual displacement field are superimposed pixel by pixel to obtain the total displacement field; S45. Perform a bilinear interpolation resampling on the original SAR image using the total displacement field to obtain the registered image.

8. The optical and SAR image registration method based on geometric prior guidance according to claim 3, characterized in that, Global average pooling outputs two global descriptor vectors with dimensions of 1×1×256.

Images