Remote sensing image registration method, system and equipment based on enhanced feature fusion

Abstract

This application discloses a remote sensing image registration method, system, and device based on enhanced feature fusion. The method constructs an image registration network model comprising a deformation field generation network and a spatial transformation network; inputs an optical remote sensing image and a synthetic aperture radar image to be registered into the deformation field generation network to obtain a deformation field; inputs the synthetic aperture radar image to be registered and the deformation field into the spatial transformation network to obtain a pre-registered image; constructs a target loss function based on the deformation field, the synthetic aperture radar image to be registered, the pre-registered image, and the optical remote sensing image; trains the image registration network model according to the target loss function until the target loss function converges, obtaining a trained image registration network model; and performs remote sensing image registration on the target optical remote sensing image and the target synthetic aperture radar image to be registered using the trained image registration network model. This application can improve the accuracy of multimodal remote sensing image registration.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a remote sensing image registration method, system and device based on enhanced feature fusion. Background Art

[0002] Image registration is a key task in computer vision and remote sensing image analysis. It aims to accurately align images acquired at different times, angles, or sensors, enabling subsequent image fusion, change detection, and analysis. In remote sensing scenarios, this task is particularly complex because images often originate from different platforms (e.g., satellites, drones), different sensor types (e.g., SAR and optical), and differ in imaging conditions. This can lead to scale variations, geometric distortions, illumination differences, and even fundamental differences between modalities. Therefore, high-precision image registration technology is crucial for the comprehensive utilization of remote sensing data.

[0003] Existing feature-based methods extract salient features (e.g., points, lines, and regions) from two images and then estimate the transformation between the images using matching algorithms (e.g., brute-force matching and RANSAC). Many representative feature-based methods, such as Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and Oriented Fast and Rotated Brief (ORB), are suitable for single-modal images with rotation and scale differences due to viewpoint or temporal changes (e.g., optical-to-optical and SAR-to-SAR matching), but are relatively fragile for multimodal image matching. This is because they primarily rely on detecting highly repeatable features, which are often affected by intensity or texture differences between multimodal images. Therefore, their matching performance is very sensitive to feature differences. Due to the large differences in intensity or texture, features extracted from multimodal images are often non-repeating, which significantly affects matching accuracy.

[0004] Therefore, in existing remote sensing image registration, feature-based methods find it difficult to extract common features between optical images and SAR images because they have significant nonlinear radiation differences, which leads to relatively low accuracy of multimodal remote sensing image registration. Summary of the Invention

[0005] This application aims to propose a remote sensing image registration method, system and equipment based on enhanced feature fusion, which can improve the accuracy of multimodal remote sensing image registration.

[0006] In a first aspect, an embodiment of the present application provides a remote sensing image registration method based on enhanced feature fusion, the method comprising:

[0007] Obtain optical remote sensing images for model training and synthetic aperture radar images to be registered;

[0008] Constructing an image registration network model including a deformation field generation network and a spatial transformation network, wherein the deformation field generation network is constructed based on a U-Net network and an enhanced feature fusion module;

[0009] Inputting the optical remote sensing image and the synthetic aperture radar image to be registered into the deformation field generation network to obtain a deformation field;

[0010] Inputting the synthetic aperture radar image to be registered and the deformation field into the spatial variation network to obtain a pre-registered image;

[0011] constructing a target loss function based on the deformation field, the synthetic aperture radar image to be registered, the pre-registered image, and the optical remote sensing image;

[0012] Training the image registration network model according to the target loss function until the target loss function converges to obtain a trained image registration network model;

[0013] Remote sensing image registration is performed on the target optical remote sensing image and the target synthetic aperture radar image to be registered using the trained image registration network model.

[0014] Compared with the prior art, the first aspect of the present application has the following beneficial effects:

[0015] The method obtains an optical remote sensing image for model training and a synthetic aperture radar image to be registered; constructs an image registration network model including a deformation field generation network and a spatial transformation network, wherein the deformation field generation network is constructed based on a U-Net network and an enhanced feature fusion module; inputs the optical remote sensing image and the synthetic aperture radar image to be registered into the deformation field generation network to obtain a deformation field; inputs the synthetic aperture radar image to be registered and the deformation field into the spatial variation network to obtain a pre-registered image; constructs a target loss function based on the deformation field, the synthetic aperture radar image to be registered, the pre-registered image and the optical remote sensing image; trains the image registration network model according to the target loss function until the target loss function converges to obtain a trained image registration network model; and performs remote sensing image registration on the target optical remote sensing image and the target synthetic aperture radar image to be registered using the trained image registration network model. In this way, by combining the U-Net network and the enhanced feature fusion module to construct a deformation field generation network, it is possible to enhance feature fusion, achieve efficient cross-layer feature fusion, and improve the accuracy of remote sensing image registration; by combining the deformation field, the synthetic aperture radar image to be registered, the pre-registered image and the optical remote sensing image, a target loss function is constructed, and the network model is trained and optimized by constructing a good target loss function, which can make the registered image have high structural consistency, while ensuring the smoothness of the deformation field, avoiding overfitting, and further improving the accuracy of multimodal remote sensing image registration.

[0016] In some embodiments, inputting the optical remote sensing image and the synthetic aperture radar image to be registered into the deformation field generation network to obtain the deformation field includes:

[0017] splicing the optical remote sensing image and the synthetic aperture radar image to be registered to obtain a spliced ​​image;

[0018] The spliced ​​image is used as the input of the deformation field generation network, and the encoder in the U-Net network is used to perform convolution feature extraction and downsampling on the spliced ​​image to obtain semantic features of multiple encoding levels;

[0019] Upsampling the semantic features extracted from the last encoding layer by the decoder in the U-Net network to obtain the current upsampled features;

[0020] Performing cross-modal feature fusion on the current up-sampled feature and the semantic feature of the next up-sampled corresponding coding level through the enhanced feature fusion module to obtain a target fused feature;

[0021] The target fusion feature is upsampled by the decoder in the U-Net network, and the current upsampling result is further cross-modal fused with the semantic features of the next upsampling corresponding encoding level until the decoder completes decoding to obtain the deformation field.

[0022] In some embodiments, the enhanced feature fusion module includes a feature enhancement submodule, a channel attention submodule, and a spatial attention submodule. The enhanced feature fusion module performs cross-modal feature fusion on the current upsampled feature and the semantic feature of the next upsampled corresponding encoding level to obtain a target fusion feature, including:

[0023] Inputting the current up-sampled feature and the semantic feature of the next up-sampled corresponding coding level into the feature enhancement submodule to obtain a first fusion feature;

[0024] Performing a residual connection on the first fused feature and the semantic feature of the next up-sampled corresponding coding level to obtain a second fused feature;

[0025] Inputting the second fused feature into the channel attention submodule to obtain a third fused feature;

[0026] The third fusion feature is input into the spatial attention submodule to obtain the target fusion feature.

[0027] In some embodiments, constructing a target loss function based on the deformation field, the synthetic aperture radar image to be registered, the pre-registered image, and the optical remote sensing image includes:

[0028] constructing a mutual information loss function by calculating the mutual information between the synthetic aperture radar image to be registered and the optical remote sensing image;

[0029] constructing a structural similarity loss function by calculating the structural similarity between the pre-registered image and the optical remote sensing image;

[0030] Constructing a deformation regularization loss function according to the deformation field;

[0031] Constructing a structured distribution correlation loss function by calculating the structured distribution correlation between the synthetic aperture radar image to be registered and the optical remote sensing image;

[0032] The mutual information loss function, the structural similarity loss function, the deformation regularization loss function and the structured distribution correlation loss function are weightedly summed to obtain a target loss function.

[0033] In some embodiments, the constructing a structured distribution correlation loss function by calculating the structured distribution correlation between the synthetic aperture radar image to be registered and the optical remote sensing image includes:

[0034] constructing local pixels in the synthetic aperture radar image to be registered into a gamma distribution;

[0035] constructing local pixels in the optical remote sensing image into a Gaussian distribution;

[0036] Calculating a first structure mapping value of the synthetic aperture radar image to be registered, and calculating a second structure mapping value of the optical remote sensing image;

[0037] Calculating a correlation weight between the synthetic aperture radar image to be registered and the optical remote sensing image in a structure domain according to the first structure mapping value and the second structure mapping value;

[0038] Calculating a structured distribution correlation between the synthetic aperture radar image to be registered and the optical remote sensing image according to the gamma distribution, the Gaussian distribution, and the correlation weight;

[0039] Based on the structured distribution correlation, a structured distribution correlation loss function is constructed.

[0040] In some embodiments, calculating the first structure mapping value of the synthetic aperture radar image to be registered, and calculating the second structure mapping value of the optical remote sensing image, include:

[0041] ;

[0042] in, represents the first structure map value of the synthetic aperture radar image to be registered, Indicates that the synthetic aperture radar image to be registered is The gradient of the point, Represents the second structure mapping value of the optical remote sensing image, Indicates that the optical remote sensing image is The gradient of the point, represents the vector differential operator.

[0043] In some embodiments, calculating the structured distribution correlation between the synthetic aperture radar image to be registered and the optical remote sensing image according to the gamma distribution, the Gaussian distribution, and the correlation weight includes:

[0044] ;

[0045] in, represents structured distribution correlation, represents the sliding window size, represents the relevance weight, represents the probability density estimate of the current window calculated by gamma distribution, represents the global reference distribution of the synthetic aperture radar image to be registered, represents the probability density estimate of the current window calculated by Gaussian distribution, Represents the global reference distribution of optical remote sensing images.

[0046] In a second aspect, an embodiment of the present application further provides a remote sensing image registration system based on enhanced feature fusion, the system comprising:

[0047] A training data acquisition unit, used to acquire optical remote sensing images for model training and synthetic aperture radar images to be registered;

[0048] A network model construction unit, configured to construct an image registration network model comprising a deformation field generation network and a spatial transformation network, wherein the deformation field generation network is constructed based on a U-Net network and an enhanced feature fusion module;

[0049] a deformation field obtaining unit, configured to input the optical remote sensing image and the synthetic aperture radar image to be registered into the deformation field generating network to obtain a deformation field;

[0050] an image pre-registration unit, configured to input the synthetic aperture radar image to be registered and the deformation field into the spatial variation network to obtain a pre-registered image;

[0051] a loss function construction unit, configured to construct a target loss function based on the deformation field, the synthetic aperture radar image to be registered, the pre-registered image, and the optical remote sensing image;

[0052] A network model training unit, configured to train the image registration network model according to the target loss function until the target loss function converges, thereby obtaining a trained image registration network model;

[0053] The target image registration unit is used to perform remote sensing image registration on the target optical remote sensing image and the target synthetic aperture radar image to be registered by using the trained image registration network model.

[0054] In a third aspect, an embodiment of the present application also provides an electronic device comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute a remote sensing image registration method based on enhanced feature fusion as described above.

[0055] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute a remote sensing image registration method based on enhanced feature fusion as described above.

[0056] It can be understood that the beneficial effects of the above-mentioned second to fourth aspects compared with the relevant technologies are the same as the beneficial effects of the above-mentioned first aspect compared with the relevant technologies. Please refer to the relevant description in the above-mentioned first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0058] Figure 1 This is a flow chart of an embodiment of a remote sensing image registration method based on enhanced feature fusion provided by the present application;

[0059] Figure 2 1 is a structural diagram of an image registration network model in a preferred embodiment of a remote sensing image registration method based on enhanced feature fusion provided by the present application;

[0060] Figure 3 1 is a schematic structural diagram of a deformation field generation network in a preferred embodiment of a remote sensing image registration method based on enhanced feature fusion provided by the present application;

[0061] Figure 41 is a schematic structural diagram of an enhanced feature fusion module in a preferred embodiment of a remote sensing image registration method based on enhanced feature fusion provided by the present application;

[0062] Figure 5 1 is a schematic structural diagram of an embodiment of a remote sensing image registration system based on enhanced feature fusion provided by the present application;

[0063] Figure 6 It is a structural diagram of an embodiment of the electronic device provided by this application. DETAILED DESCRIPTION

[0064] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application.

[0065] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0066] In the description of this application, it should be understood that descriptions involving orientation, such as the orientation or positional relationship indicated by up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application.

[0067] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technical personnel in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0068] First, let’s analyze some of the terms used in this application:

[0069] Remote sensing image registration: The process of accurately aligning remote sensing images taken by different sensors, at different times, and from different perspectives into a unified spatial coordinate system to facilitate subsequent fusion and comparative analysis.

[0070] Enhanced Feature Fusion (EFF) module: A deep network module specifically designed for cross-level and cross-modal feature fusion. It contains multiple sub-modules (EAG, SA, and ECA) and achieves efficient fusion through enhanced feature representation and attention mechanism.

[0071] Enhanced Attention Group (EAG): A submodule that uses group convolution and residual connections to enhance feature expression capabilities and effectively fuse cross-modal features.

[0072] Efficient Channel Attention (ECA) module: This module calculates channel attention weights through one-dimensional convolution, which is used to strengthen important feature channels, suppress irrelevant information interference, and improve the network's attention to different channel features.

[0073] Spatial Attention submodule SA (Spatial Attention): It captures key areas in spatial positions through global average pooling and global maximum pooling, enhances the network's attention to important areas in spatial positions, and thus improves the effectiveness of feature expression.

[0074] Deformation Field: A two-dimensional vector field that describes the pixel-level correspondence between the image to be registered and the reference image. The vector at each pixel position represents the spatial displacement that the corresponding pixel needs to make.

[0075] Existing feature-based methods extract salient features (e.g., points, lines, and regions) from two images and then estimate the transformation between the images using matching algorithms (e.g., brute-force matching and RANSAC). Many representative feature-based methods, such as the Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and Oriented Fast and Rotated Brief (ORB), are suitable for single-modal images with rotation and scale differences due to viewpoint or temporal variations (e.g., optical-optical and SAR-SAR matching), but are relatively fragile for multimodal image matching. This is because they primarily rely on detecting highly repeatable features, which are often affected by intensity or texture differences between multimodal images. Therefore, their matching performance is very sensitive to feature differences. Due to the large differences in intensity or texture, features extracted from multimodal images are often non-repeating, which significantly affects matching accuracy.

[0076] In order to solve the problem of low multimodal remote sensing image registration accuracy in the existing technology, this application proposes a remote sensing image registration method, system and device based on enhanced feature fusion.

[0077] Reference Figure 1 , a flow chart of a remote sensing image registration method based on enhanced feature fusion provided in an embodiment of the present application. The remote sensing image registration method based on enhanced feature fusion is applied to an electronic device, which may be a server or a mobile terminal. Figure 1As shown, the remote sensing image registration method based on enhanced feature fusion may include the following steps:

[0078] Step S100: acquiring an optical remote sensing image for model training and a synthetic aperture radar image to be registered;

[0079] Step S200: constructing an image registration network model including a deformation field generation network and a spatial transformation network, wherein the deformation field generation network is constructed based on a U-Net network and an enhanced feature fusion module;

[0080] Step S300: inputting the optical remote sensing image and the synthetic aperture radar image to be registered into a deformation field generation network to obtain a deformation field;

[0081] Step S400: inputting the synthetic aperture radar image to be registered and the deformation field into a spatial variation network to obtain a pre-registered image;

[0082] Step S500: constructing a target loss function based on the deformation field, the synthetic aperture radar image to be registered, the pre-registered image, and the optical remote sensing image;

[0083] Step S600: training the image registration network model according to the target loss function until the target loss function converges to obtain a trained image registration network model;

[0084] Step S700: perform remote sensing image registration on the target optical remote sensing image and the target synthetic aperture radar image to be registered using the trained image registration network model.

[0085] In this embodiment, an optical remote sensing image for model training and a synthetic aperture radar image to be registered are obtained; an image registration network model including a deformation field generation network and a spatial transformation network is constructed, wherein the deformation field generation network is constructed based on a U-Net network and an enhanced feature fusion module; the optical remote sensing image and the synthetic aperture radar image to be registered are input into the deformation field generation network to obtain a deformation field; the synthetic aperture radar image to be registered and the deformation field are input into the spatial variation network to obtain a pre-registered image; a target loss function is constructed based on the deformation field, the synthetic aperture radar image to be registered, the pre-registered image, and the optical remote sensing image; the image registration network model is trained according to the target loss function until the target loss function converges to obtain a trained image registration network model; and remote sensing image registration is performed on the target optical remote sensing image and the target synthetic aperture radar image to be registered using the trained image registration network model. In this way, by combining the U-Net network and the enhanced feature fusion module to construct a deformation field generation network, it is possible to enhance feature fusion, achieve efficient cross-layer feature fusion, and improve the accuracy of remote sensing image registration; by combining the deformation field, the synthetic aperture radar image to be registered, the pre-registered image and the optical remote sensing image, a target loss function is constructed, and the network model is trained and optimized by constructing a good target loss function, which can make the registered image have high structural consistency, while ensuring the smoothness of the deformation field, avoiding overfitting, and further improving the accuracy of multimodal remote sensing image registration.

[0086] The enhanced feature fusion module can be used in the feature fusion process of the encoder and decoder in the U-Net network. By strengthening the feature fusion between different levels and modalities, the low-level texture information and high-level semantic information in the image can be better combined.

[0087] The above-mentioned target loss function is constructed based on the deformation field, the synthetic aperture radar image to be registered, the pre-registered image and the optical remote sensing image. It can be based on the deformation field, the synthetic aperture radar image to be registered, the pre-registered image and the optical remote sensing image, constructing loss functions between each other to obtain multiple loss functions, and then combining the multiple loss functions to construct the target loss function.

[0088] In some embodiments, the optical remote sensing image and the synthetic aperture radar image to be registered are input into a deformation field generation network to obtain a deformation field, including:

[0089] splicing the optical remote sensing image and the synthetic aperture radar image to be registered to obtain a spliced ​​image;

[0090] The spliced ​​image is used as the input of the deformation field generation network. The encoder in the U-Net network performs convolutional feature extraction and downsampling on the spliced ​​image to obtain semantic features at multiple encoding levels.

[0091] The semantic features extracted by the last encoding layer are upsampled through the decoder in the U-Net network to obtain the current upsampled features;

[0092] The enhanced feature fusion module is used to perform cross-modal feature fusion on the current upsampled feature and the semantic features of the next upsampled corresponding encoding level to obtain the target fused feature;

[0093] The target fusion feature is upsampled through the decoder in the U-Net network, and the current upsampling result is further cross-modal fused with the semantic features of the next upsampling corresponding encoding level until the decoder is decoded and the deformation field is obtained.

[0094] In this embodiment, an optical remote sensing image and a synthetic aperture radar image to be registered are spliced ​​to obtain a spliced ​​image; the spliced ​​image is used as the input of a deformation field generation network, and the spliced ​​image is subjected to convolution feature extraction and downsampling by an encoder in a U-Net network to obtain semantic features of multiple coding levels; the semantic features extracted at the last coding level are upsampled by a decoder in the U-Net network to obtain current upsampled features; the current upsampled features are cross-modally fused with the semantic features of the next upsampled coding level by an enhanced feature fusion module to obtain target fused features; the target fused features are upsampled by a decoder in the U-Net network, and the current upsampled results are further cross-modally fused with the semantic features of the next upsampled coding level until decoding by the decoder is completed to obtain a deformation field. In this way, by utilizing the symmetrical structure of the encoder and decoder in the U-Net network to extract and reconstruct image features, and then using the enhanced feature fusion module to perform cross-modal feature fusion on the current up-sampled features and the semantic features of the next up-sampled corresponding encoding level, feature fusion can be enhanced and efficient cross-layer feature fusion can be achieved, thereby improving the accuracy of remote sensing image registration.

[0095] In some embodiments, the enhanced feature fusion module includes a feature enhancement submodule, a channel attention submodule, and a spatial attention submodule. The enhanced feature fusion module performs cross-modal feature fusion on the current upsampled feature and the semantic feature of the next upsampled corresponding encoding level to obtain a target fused feature, including:

[0096] Input the current upsampled feature and the semantic feature of the next upsampled corresponding encoding level into the feature enhancement submodule to obtain the first fusion feature;

[0097] Perform a residual connection on the first fusion feature and the semantic feature of the next up-sampled corresponding encoding level to obtain the second fusion feature;

[0098] Input the second fused feature into the channel attention submodule to obtain the third fused feature;

[0099] The third fusion feature is input into the spatial attention submodule to obtain the target fusion feature.

[0100] In this embodiment, the first fusion feature is obtained by inputting the current upsampled feature and the semantic feature of the next upsampled encoding level into the feature enhancement submodule; the first fusion feature and the semantic feature of the next upsampled encoding level are residually connected to obtain the second fusion feature; the second fusion feature is input into the channel attention submodule to obtain the third fusion feature; the third fusion feature is input into the spatial attention submodule to obtain the target fusion feature. In this way, preliminary feature extraction and cross-layer fusion are achieved through the feature enhancement submodule; then, through the channel attention calculation of the channel attention submodule, a weight is assigned to each channel to highlight the important feature channels; then, the spatial attention submodule is entered to weight the feature map in the spatial dimension, focus on the key areas, and suppress background interference, thereby achieving efficient cross-layer feature fusion.

[0101] In some embodiments, a target loss function is constructed based on the deformation field, the synthetic aperture radar image to be registered, the pre-registered image, and the optical remote sensing image, including:

[0102] By calculating the mutual information between the synthetic aperture radar image and the optical remote sensing image to be registered, a mutual information loss function is constructed;

[0103] By calculating the structural similarity between the pre-registered image and the optical remote sensing image, a structural similarity loss function is constructed;

[0104] According to the deformation field, construct the deformation regularization loss function;

[0105] By calculating the structured distribution correlation between the synthetic aperture radar image and the optical remote sensing image to be registered, a structured distribution correlation loss function is constructed;

[0106] The mutual information loss function, structural similarity loss function, deformation regularization loss function and structured distribution correlation loss function are weightedly summed to obtain the target loss function.

[0107] In this embodiment, a mutual information loss function is constructed by calculating the mutual information between the synthetic aperture radar image to be registered and the optical remote sensing image; a structural similarity loss function is constructed by calculating the structural similarity between the pre-registered image and the optical remote sensing image; a deformation regularization loss function is constructed based on the deformation field; a structured distribution correlation loss function is constructed by calculating the structured distribution correlation between the synthetic aperture radar image to be registered and the optical remote sensing image; and a target loss function is obtained by weightedly summing the mutual information loss function, the structural similarity loss function, the deformation regularization loss function, and the structured distribution correlation loss function. In this way, by combining the mutual information loss function, the structural similarity loss function, the deformation regularization loss function, and the structured distribution correlation loss function for weighted summation to construct the target loss function, the target loss function becomes more flexible and precise, suitable for multimodal remote sensing image registration, and enables the image registration network model trained by the target loss function to more accurately perform remote sensing image registration.

[0108] In some embodiments, a structured distribution correlation loss function is constructed by calculating the structured distribution correlation between the synthetic aperture radar image and the optical remote sensing image to be registered, including:

[0109] The local pixels in the synthetic aperture radar image to be registered are constructed as a gamma distribution;

[0110] The local pixels in the optical remote sensing image are constructed as Gaussian distribution;

[0111] Calculating a first structure mapping value of the synthetic aperture radar image to be registered, and calculating a second structure mapping value of the optical remote sensing image;

[0112] Calculating a correlation weight of the synthetic aperture radar image to be registered and the optical remote sensing image in the structure domain according to the first structure mapping value and the second structure mapping value;

[0113] Calculate the structured distribution correlation between the synthetic aperture radar image and the optical remote sensing image to be registered according to the gamma distribution, Gaussian distribution and correlation weight;

[0114] Based on structured distribution correlation, a structured distribution correlation loss function is constructed.

[0115] In this embodiment, local pixels in the synthetic aperture radar image to be registered are constructed as a gamma distribution; local pixels in the optical remote sensing image are constructed as a Gaussian distribution; a first structure mapping value of the synthetic aperture radar image to be registered is calculated, and a second structure mapping value of the optical remote sensing image is calculated; based on the first structure mapping value and the second structure mapping value, a correlation weight between the synthetic aperture radar image to be registered and the optical remote sensing image in the structure domain is calculated; based on the gamma distribution, the Gaussian distribution, and the correlation weight, a structured distribution correlation between the synthetic aperture radar image to be registered and the optical remote sensing image is calculated; and based on the structured distribution correlation, a structured distribution correlation loss function is constructed. In this way, by constructing a good structured distribution correlation loss function, the structural consistency registration effect between the synthetic aperture radar image to be registered and the optical remote sensing image can be improved, and the problem that traditional loss functions are insufficient in modeling image structure distribution differences when processing multimodal remote sensing images is solved.

[0116] In some embodiments, calculating a first structure map value of a synthetic aperture radar image to be registered, and calculating a second structure map value of an optical remote sensing image, include:

[0117] ;

[0118] in, represents the first structure map value of the synthetic aperture radar image to be registered, Indicates that the synthetic aperture radar image to be registered is The gradient of the point, Represents the second structure mapping value of the optical remote sensing image, Indicates that the optical remote sensing image is The gradient of the point, represents the vector differential operator.

[0119] In some embodiments, calculating the structured distribution correlation between the synthetic aperture radar image and the optical remote sensing image to be registered according to the gamma distribution, the Gaussian distribution, and the correlation weight includes:

[0120] ;

[0121] in, represents structured distribution correlation, represents the sliding window size, represents the relevance weight, represents the probability density estimate of the current window calculated by gamma distribution, represents the global reference distribution of the synthetic aperture radar image to be registered, represents the probability density estimate of the current window calculated by Gaussian distribution, Represents the global reference distribution of optical remote sensing images.

[0122] To facilitate understanding by those skilled in the art, a set of best embodiments is provided below:

[0123] Image registration is a key task in computer vision and remote sensing image analysis. It aims to accurately align images acquired at different times, angles, or sensors, enabling subsequent image fusion, change detection, and analysis. In remote sensing scenarios, this task is particularly complex because images often originate from different platforms (e.g., satellites, drones), different sensor types (e.g., SAR and optical), and differ in imaging conditions. This can lead to scale variations, geometric distortions, illumination differences, and even fundamental differences between modalities. Therefore, high-precision image registration technology is crucial for the comprehensive utilization of remote sensing data.

[0124] Feature-based methods extract salient features (e.g., points, lines, and regions) from two images and then estimate the transformation between them using matching algorithms (e.g., brute-force matching and RANSAC). Many representative feature-based methods, such as the Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and Oriented Fast and Rotated Brief (ORB), are suitable for single-modality images with rotation and scale differences due to viewpoint or temporal variations (e.g., optical-to-optical and SAR-to-SAR matching), but are relatively vulnerable to multimodal image matching. This is because they primarily rely on detecting highly repeatable features, which are often affected by intensity or texture differences between multimodal images. Consequently, their matching performance is highly sensitive to feature differences. Due to the large differences in intensity or texture, features extracted from multimodal images are often non-repeating, which significantly affects matching accuracy. Therefore, in remote sensing image registration, feature-based methods struggle to extract common features between optical and SAR images due to their significant nonlinear radiometric differences.

[0125] SAR images, formed by the reflection of radar waves, primarily reflect the backscattering characteristics of ground objects and are subject to strong noise (such as speckle noise) and a lack of structural information. Optical images, on the other hand, are captured using visible light and possess clear textures and rich color information, making them suitable for visual analysis. Due to the different modalities of these two images, traditional registration methods (such as SIFT) perform poorly in feature matching, resulting in low image registration accuracy. Existing deep learning-based methods also suffer from insufficient cross-modal feature extraction, further leading to low image registration accuracy.

[0126] Therefore, the existing remote sensing image registration methods have the following disadvantages:

[0127] (1) Traditional feature extraction methods have poor adaptability in cross-modal registration.

[0128] Existing technologies typically rely on traditional feature extraction methods (such as SIFT, SURF, and ORB) to align images from different modalities. These methods rely on hand-crafted feature extraction techniques to extract local information (such as edges and corners) from the images. However, SAR images (i.e., synthetic aperture radar images) and optical images (i.e., optical remote sensing images) differ significantly in texture, contrast, and structure. Traditional feature extraction methods often fail to fully capture the complex feature relationships between cross-modal images, resulting in low registration accuracy. The significant difference between the noise and blur of SAR images and the high resolution and crisp texture of optical images makes it difficult for hand-crafted features to provide effective cross-modal matching.

[0129] (2) Poor feature fusion capability, that is, the feature extraction process lacks an effective multi-scale and cross-layer information fusion mechanism.

[0130] Many existing registration methods use simple convolutional neural networks (CNNs) for feature extraction, but lack effective cross-layer feature fusion mechanisms. In image registration tasks, low-level features (such as edges and textures) and high-level features (such as objects and semantic information) have different representations. While existing feature extraction modules can process certain contextual information, they are not sufficiently optimized for complex image registration tasks. In particular, when processing cross-modal images, they may not be able to fully learn higher-level semantic information. This results in an inability to effectively combine low-level and high-level features in the image, resulting in the failure to fully utilize important cross-modal information in the image registration process, affecting the accuracy of remote sensing image registration, especially in cases of large image changes, where the accuracy is even lower.

[0131] (3) Spatial transformation and registration are separated, that is, the image registration and spatial transformation modules are independent of each other and lack a joint optimization mechanism.

[0132] Existing technologies typically break down the image registration process into several stages: feature extraction, transformation estimation, and transformation application. During this process, transformation estimation (such as affine or perspective transformation) and optimization of local deformations are typically performed separately. Because geometric deformation in an image is not solely a global transformation, in many cases, non-rigid deformations may occur in local regions. Existing methods rely on separate optimization of global and local transformations, failing to effectively combine the two for joint optimization. Separately optimizing global and local transformations results in a lack of effective synergy between these two stages during the registration process. Inconsistent global and local transformations may occur during the image registration process, resulting in suboptimal image registration, particularly for remote sensing images with large terrain variations or occlusions.

[0133] (4) The loss function design is simple, that is, the loss function design is single and difficult to adapt to the differences in cross-modal image structures.

[0134] Many existing image registration methods use pixel-level error metrics (such as the L2 loss) or traditional gradient-based loss functions. These loss functions typically focus solely on overall image differences and fail to fully capture subtle structural differences (such as texture and edges) between multimodal images. For cross-modal remote sensing images, especially those between SAR and optical images, texture and structure can vary significantly. Relying solely on pixel-level error metrics (such as the L2 loss) fails to accurately reflect the structural differences between images, resulting in reduced registration quality. The registration process fails to precisely align structural information within the images, leading to low registration accuracy, especially in complex environments or with significant lighting variations.

[0135] In order to solve the shortcomings of the above-mentioned prior art, Figure 2 The image registration network model of this embodiment organically combines the U-Net structure with the efficient feature fusion module (EFF) and introduces the spatial transformer network (STN). This effectively solves the problems of insufficient cross-modal feature extraction capabilities, inadequate feature fusion mechanisms, separation of spatial transformation and registration tasks, and simplistic loss function design in existing remote sensing image registration technologies, thereby significantly improving the registration accuracy and robustness between SAR images and optical images. The technical solution of this embodiment specifically includes the following contents:

[0136] Step S1: Obtain a training image dataset containing multiple SAR images and optical images (i.e., optical remote sensing images) to be registered. Input the SAR images and optical images to be registered for model training into the registration network (i.e., deformation field generation network), and obtain the deformation field through the registration network. The deformation field represents the pixel-level position correspondence between the image to be registered and the reference image. Figure 3 , the specific processing process of the registration network is:

[0137] The registration network in this embodiment employs a U-Net-based encoder-decoder architecture and incorporates an enhanced feature fusion (EFF) module to generate a high-precision deformation field. The encoder performs convolutional feature extraction and downsampling on the input SAR image and optical image (which are concatenated in the channel dimension to form a multi-channel input) layer by layer, capturing multi-scale low-level texture and high-level semantic features. The decoder then progressively upsamples and fuses the features from the corresponding layer in the encoding phase with the currently upsampled features via skip connections. The EFF module introduced during the fusion process contains submodules such as EAG, SA, and ECA, which are used to enhance feature representation, focus on key spatial regions, and adjust channel weights, respectively, to fully integrate feature information from different modalities and levels. After encoding and decoding and multi-level feature fusion, the registration network ultimately outputs a two-dimensional deformation field (with two channels representing pixel displacements in the x and y directions, respectively) of the same size as the input image through convolution. This field describes the pixel-wise correspondence between the SAR image to be registered and the reference optical image. This deformation field directly describes the warping transformation required to spatially align the two images.

[0138] Specifically, the first step is the encoding stage. The input SAR image and the optical image are spliced ​​as the input of the registration network. The input size is , by performing convolutional layers and activation functions to extract the features of the input image, the size is Next, perform maximum pooling on the obtained feature map to retain the number of channels of the feature map , reduce the size of the image to . Then, the convolutional layer is used to extract deeper semantic information and increase the number of channels to , the feature map size is The process continues: the maximum pooling operation is performed again, and the size is further reduced to , then convolution extracts features and expands the channel to , the output is . Continue encoding downward, and after entering the deepest feature extraction stage, we get a feature map of size And send it to the EFF (Efficient Feature Fusion) module for multi-scale context fusion and feature compression.

[0139] The next step is the decoding stage. Through upsampling operation, the deep feature map is transformed from Upsample to , and the corresponding feature map in the encoding path At the same time, it is used as the input of the EFF module for feature fusion to obtain the fused features , and extract features through convolutional layers and expand the channels to The decoding path continues upwards and is restored to a size of feature maps (keep the same size as the other end of the encoder), and fuse the encoding stage The feature map is passed through the convolution EFF module to obtain a new fusion feature. This process is repeated and gradually upsampled to At each level, the current feature map is concatenated with the skip connection features in the encoding path and fused and enhanced by the EFF module. Finally, the number of channels is increased from 1 to 1 through the convolution layer. Compress to 2, output a size of The deformation field is used to estimate the corresponding offset between the SAR image and the optical image at each pixel to achieve high-precision image registration.

[0140] Among them, the enhanced feature fusion (EFF) module is a key component in the registration network of this embodiment, which is used to improve the effect of cross-modal feature extraction and fusion during the feature fusion process of the encoder and decoder. Figure 4 As shown in the figure, the EFF module receives feature maps from different layers (low-level texture features from the encoder and high-level semantic features from the decoder). It processes these features sequentially through its three internal submodules and outputs a fused and enhanced feature map. First, the input features undergo group convolution in the EAG submodule (feature enhancement submodule) for preliminary feature extraction and cross-layer fusion. Next, the ECA submodule (channel attention submodule) calculates channel attention, assigning weights to each channel and highlighting important feature channels. Finally, the SA submodule (spatial attention submodule) weights the feature maps spatially, focusing on key areas and suppressing background interference. After these three stages, the features output by the EFF module fuse multi-scale and multi-modal information, preserving low-level texture details while incorporating high-level semantic structure. This provides a rich and consistent feature representation for subsequent deformation field estimation for registration.

[0141] The EAG submodule primarily achieves enhanced cross-modal feature fusion through convolution operations and residual connections (denoted by C). The EAG submodule applies grouped convolution (Group Conv(1x1)) to the input features, grouping the channels and extracting features in parallel. This structure reduces convolution parameters and computational overhead while enabling convolution filters to learn features specifically for different groups, thereby improving adaptability to cross-modal feature differences. The features extracted by grouped convolution are batch normalized (BN) and activated with a ReLU nonlinearity to capture local combinations of feature patterns in the input. Subsequently, the parallel extracted features after BN and ReLU are summed, and then processed with a ReLU nonlinearity, convolution (Conv(1x1)), and a sigmoid activation function to obtain the convolutional features. The EAG submodule then uses residual connections to directly add the input features to the convolutional features. Through this residual path, original low-level detail information is directly integrated into high-level abstract features, avoiding the loss of key information during feature extraction and enhancing the expressive power of features. The output of the EAG submodule is a feature map that combines the original input and the newly extracted information through convolution. It not only contains enhanced cross-modal common features but also retains the original important details, providing a good feature foundation for the subsequent attention module.

[0142] The ECA (channel attention) submodule is responsible for recalibrating the importance weights of each channel in the feature map along the channel dimension. The ECA submodule first processes the output feature map from the EAG globally, typically using a global average pooling operation or an equivalent one-dimensional convolution to compress the entire feature map in the spatial dimension, generating a global description for each channel. ECA then filters this channel description vector along the channel dimension using a one-dimensional convolution. This one-dimensional convolution operation can be viewed as an efficient way to implement information exchange between channels and learn the correlations between them. The output of the convolution is passed through an activation function to form channel attention weights, with each channel corresponding to a weight coefficient between 0 and 1. These weights are then multiplied by each channel of the original feature map to complete the remodulation of the feature map channels. Functional effect: Through the ECA submodule, the network can automatically highlight important feature channels and suppress irrelevant or interfering channel information during registration. For example, in the registration of SAR and optical images, some channels may correspond to shared edges or structural features in both images, while other channels may primarily reflect noise or artifacts from a single modality. ECA assigns higher weights to the former and lower weights to the latter, allowing the feature map after ECA to focus more on the key information required for alignment. The output is a channel-weighted feature map that retains the spatial size of the original feature map but has been reordered by importance in terms of channel strength, highlighting the signals most beneficial for registration.

[0143] The SA (Spatial Attention) submodule generates attention weights for feature maps in two spatial dimensions, directing the network to focus on discriminative regions of the image and ignore irrelevant regions. The SA submodule extracts spatial importance cues from the input feature map by applying global average pooling (GAP) and global max pooling (GMP) in parallel to compress the feature map in the channel dimension. These two pooling operations yield two feature maps with only spatial dimensions (width × height): the average pooling map reflects the average activation across all channels at each spatial location, while the max pooling map reflects the strongest response across certain channels at each location. These two maps are then pixel-wise superimposed in the spatial dimension and fed into a convolutional layer followed by a sigmoid activation function to generate a spatial attention weight map of the same size as the original feature map. Each pixel value in this weight map (between 0 and 1) represents the importance of the corresponding spatial location. Finally, the SA submodule performs element-wise multiplication of the weight map with the original input feature map to spatially weight the input feature map. This operation assigns higher responses to regions of the feature map containing key features, while lowering responses to noisy or background regions irrelevant to registration. Functional Effect: The SA submodule enables the network to adaptively focus on common and salient target regions in cross-modal images, improving cross-modal feature alignment while reducing interference caused by background differences. The output is a spatially weighted feature map with the same size and number of channels as the input, but with the spatial response distribution highlighting key regions and suppressing less important regions.

[0144] Step S2: The SAR image to be registered and the deformation field are passed into the spatial transformation network to obtain the registered image (i.e., the pre-registered image).

[0145] After obtaining the deformation field between the image to be registered (SAR image) and the reference image (optical image), the deformation field and the image to be registered are used as input and passed into the spatial transformation network (STN). The spatial transformation network usually consists of three parts: a positioning network, a grid generator, and a sampler. In this embodiment, the registration network (described above) plays the role of both the positioning network and the grid generator, generating a deformation field and clarifying the coordinate mapping relationship of each output pixel in the original SAR image. The sampler then interpolates the image to be registered based on the deformation field to obtain the final registration result. Specifically,

[0146] As a differentiable image transformation module, the STN applies the deformation field predicted by the registration network in the previous step to the image to be registered, achieving spatial warping alignment of the images. Specifically, the STN consists of three parts: a localization network, a grid generator, and a sampler. The localization network predicts the corresponding spatial transformation parameters based on the input deformation field. The grid generator generates a sampling coordinate grid based on this prediction, mapping each pixel of the output image to the corresponding position in the original SAR image. The sampler resamples the SAR image onto these irregular coordinates using algorithms such as bilinear interpolation. The STN module can perform complex geometric corrections such as translation, rotation, scaling, and even projection transformations on the SAR image, accurately mapping its pixels to the coordinate system of the reference optical image. Because all components of the STN are trainable, this embodiment combines it end-to-end with the U-Net backbone network, generating alignment results in the forward propagation and jointly optimizing parameters based on a loss function in the backpropagation, thereby achieving joint optimization of the image registration process and significantly improving the accuracy and robustness of the registration.

[0147] Step S3: Calculate the loss of the obtained pre-registered image and the optical image to obtain the loss value, and back-propagate to update the network parameters. Continue iterative training until the loss function value converges to a stable level, and finally obtain the trained SAR and optical remote sensing image registration model (i.e., image registration network model). Use the trained SAR and optical remote sensing image registration model to perform remote sensing image registration on the target optical remote sensing image and the target synthetic aperture radar image to be registered. The target optical remote sensing image and the target synthetic aperture radar image to be registered are the image pairs that the user wants to register. Specifically,

[0148] The target loss function is constructed as:

[0149] ;

[0150] in, is the image after registration (i.e. the SAR image after deformation field and STN transformation), is the reference image (i.e., optical image). is the deformation field generated by the backbone of the network (U-net network). 、 、 、 The weight coefficient for each loss function is used to adjust the contribution of different loss functions. The weight coefficient of each loss is adjusted accordingly based on the experimental results.

[0151] In order to maximize the statistical correlation between SAR images and optical images, especially in cross-modal image registration, mutual information can be a good measure of the similarity between images, especially when there are illumination changes or texture differences in the images. Mutual information loss function as follows:

[0152] ;

[0153] in, is the joint probability distribution of SAR image and optical image. and are the marginal probability distributions of SAR image and optical image respectively.

[0154] In order to optimize the local structural consistency of the image, the similarity between the registered image and the reference image in local features such as structure, texture and edge is ensured. Structural similarity loss function as follows:

[0155] ;

[0156] in, Measures the similarity of image brightness, contrast, and structure. The result range is [0,1]. The closer the value is to 1, the more similar the images are.

[0157] In order to maintain the smoothness of the deformation field, avoid unnatural deformation, and ensure the geometric consistency of the registered image. Deformation regularization loss function as follows:

[0158] ;

[0159] To further enhance the structural consistency registration between SAR and optical images and address the inadequate modeling of image structural distribution differences by traditional loss functions when processing multimodal remote sensing images, this embodiment innovatively designs a Structured Distribution Correlation Loss (SDC). This loss function combines local statistical modeling with a structure-sensitive weighting mechanism, significantly improving the structural alignment of detailed regions in the registration results, particularly in the presence of speckle noise in SAR images and illumination interference in optical images.

[0160] On the one hand, SAR images have multiplicative speckle noise, blurred texture structure, and grayscale distribution generally follows a gamma distribution. On the other hand, optical images are sensitive to illumination, have sharp edges, and are rich in texture, and their grayscale distribution can be approximated by a Gaussian distribution or a Gaussian mixture model. Traditional losses such as pixel-level L2, SSIM, or mutual information (MI) struggle to fully capture the structural similarities and differences and statistical feature correspondences across modalities. Therefore, to address these issues, this embodiment proposes a new loss function that combines distribution difference modeling with structural domain correlation quantification to evaluate the structural statistical consistency between the registration result and the reference image.

[0161] This embodiment slides a fixed-size window in the image , within this local window, the following probability distribution models are established for SAR images and optical images respectively.

[0162] SAR image: Local pixel values ​​follow the Gamma distribution (i.e., gamma distribution):

[0163] ;

[0164] in, and Gamma distribution The shape and scale parameters of can be calculated from local pixels by maximum likelihood estimation.

[0165] Optical image: Local pixel values ​​follow Gaussian distribution (i.e. Gaussian distribution):

[0166] ;

[0167] in, is the local mean, is the local variance, which can also be estimated by local pixels.

[0168] In order to establish cross-modal correspondence on the structural domain, the structural mapping function is introduced , which is in the form of the local gradient magnitude of the image:

[0169] ;

[0170] in, Represents the structure mapping value of the SAR image, Represents the gradient of the SAR image at point X, represents the structure mapping value of the optical image, Represents the gradient of the optical image at point Y.

[0171] This mapping projects grayscale domain features into the structural space, enhances the response of edges and textures, and has stronger adaptability to modal differences.

[0172] in, , The calculation process represents the above formula The calculation process of this The essence of this formula is the calculation of gradients. The structure mapping function is constructed using the gradient magnitude form. By calculating the rate of change of the image in the horizontal and vertical directions, it extracts its local structural strength and is used to measure the correspondence of multimodal images in the structural domain.

[0173] In the structural space, the correspondence between the SAR pixel value x and the optical pixel value y is controlled by the following weight function:

[0174] ;

[0175] in, This is a hyperparameter that controls structural sensitivity (usually set to 2-3 times the standard deviation of the gradient). This function is inspired by Gaussian kernel similarity and can assign higher fusion weights to regions with similar structures, suppressing interference from irrelevant regions.

[0176] Based on the above modeling, this embodiment proposes the following structured distribution correlation (SDC) metric function:

[0177] ;

[0178] in, and are the probability density estimates of the current window of the SAR image and the optical image, respectively. The probability density estimates are calculated based on the Gamma distribution and the Gaussian distribution; and are the global reference distributions of SAR images and optical images respectively, and the global reference distributions are also calculated based on the Gamma distribution and Gaussian distribution; is the correlation weight on the structural domain; the numerator is the weighted distribution difference covariance, and the denominator is the normalization factor, which ensures that the output range of the function is [-1, 1], similar to the correlation coefficient. When the SDC value is closer to 1, it means that the structural distribution of the two images in this local area is more consistent, otherwise the structural distribution is inconsistent. Therefore, the structured distribution correlation loss function as follows:

[0179] ;

[0180] The specific registration steps of the SAR and optical images in this embodiment are as follows:

[0181] Step 1: Data preprocessing. Obtain a training image dataset containing multiple SAR images and optical images (i.e., optical remote sensing images) to be registered. Preprocess the image pairs consisting of the SAR and optical images to be registered, as well as the image pairs consisting of the target optical remote sensing image and the synthetic aperture radar image to be registered. Preprocessing includes image normalization, resampling, denoising, and data augmentation to ensure that the spatial size, resolution, and number of channels of the SAR and optical images match each other to ensure consistency in the neural network input dimensions.

[0182] Step 2: Model Initialization. Build a deformation field generation network based on the U-net structure and initialize the model's hyperparameters, including the learning rate, batch size, and weight initialization method. Also, set up the Adam optimizer and configure its parameters.

[0183] Step 3: Forward Propagation and Image Registration. The preprocessed SAR image and the optical image are concatenated in the channel dimension to form multi-channel input data. This data is fed into the U-Net network for forward propagation, and the network outputs a two-dimensional deformation field. A spatial transformation network is then used to generate a coordinate sampling grid based on the predicted deformation field. Bilinear interpolation is used to resample the SAR image to be registered, resulting in a registered SAR image aligned with the optical image.

[0184] Step 4: Calculate the target loss function. The registered SAR image and the reference optical image are subjected to the target loss function calculation. The target loss function uses mutual information loss, structural similarity loss, deformation regularization loss, and structured distribution correlation loss to quantitatively evaluate the registration quality and obtain the loss value.

[0185] Step 5: Backpropagation and parameter optimization. Based on the loss value, the network parameter gradient is calculated through the backpropagation algorithm. The selected Adam optimizer is used to update the network weight parameters and further optimize the network parameter configuration to improve the model's registration performance.

[0186] Step 6: Model iterative training and convergence judgment. Repeat steps 3 to 5 and continue iterative training until the loss function value converges to a stable level, and finally obtain the trained SAR and optical remote sensing image registration model.

[0187] Step 7: Use the trained SAR and optical remote sensing image registration model (i.e., the trained image registration network model) to perform remote sensing image registration on the target optical remote sensing image and the target synthetic aperture radar image to be registered.

[0188] This embodiment provides the following solutions to address the shortcomings of the prior art, specifically:

[0189] (1) Existing feature extraction methods have certain limitations when dealing with the modal differences between SAR images and optical images.

[0190] To address this issue, this embodiment uses a deep learning-based feature extraction architecture to replace handcrafted features. Specifically, a U-Net convolutional neural network is introduced as the underlying architecture for feature extraction. The U-Net's symmetric encoder-decoder structure automatically learns multi-level, multi-scale feature representations from input SAR and optical images. The encoder extracts low-level texture details and high-level semantic information through layer-by-layer convolution and pooling; the decoder gradually reconstructs features through upsampling and deconvolution and outputs the deformation field required for registration. This data-driven approach makes the extracted features adaptive to modal differences, capturing the correspondence between the backscatter pattern of the SAR image and the texture structure of the optical image, rather than relying on artificial features that are susceptible to modal variations. Furthermore, this embodiment incorporates channel attention and spatial attention mechanisms into the U-Net, enabling the network to automatically focus on discriminative regions and features in cross-modal images, further improving the adaptability of feature extraction to SAR / optical differences. In summary, the U-Net-based deep feature extraction module enables this embodiment to robustly represent features from images of different modalities, significantly improving the matching accuracy of cross-modal registration.

[0191] (2) In existing methods, feature fusion is insufficient and fails to effectively combine features at different levels of the image.

[0192] To address the problem of insufficient feature fusion capability and inability to integrate multi-level feature information, this embodiment designs an enhanced feature fusion module (EFF), which is embedded in the U-Net architecture to fully fuse features at different levels.

[0193] The schematic diagram of the registration network structure based on the attention U-Net and EFF modules proposed in this embodiment is shown in Figure 2. Figure 3The features extracted by the encoder at each layer are input to the decoder via skip connections. At each corresponding layer, the EFF module fuses the upsampled features with the decoder, and the fused features are output for step-by-step reconstruction of the deformation field. The EFF module organically combines low-level texture details with high-level semantic information through a multi-scale, cross-modal feature fusion strategy. EFF contains three submodules, each responsible for feature enhancement and fusion at different dimensions: First, the feature enhancement submodule (EAG) is introduced to extract cross-modal shared features through grouped convolution and bridge low- and high-level features using residual connections, reducing information loss while improving adaptability to modal differences. Second, the efficient channel attention submodule (ECA) is adopted to calculate channel weights using one-dimensional convolution, highlighting the importance of feature channels in multimodal images, automatically selecting feature channels that play a key role in registration, and suppressing irrelevant or noisy information. Third, the integrated spatial attention submodule (SA) obtains salient spatial regions of the feature map through global average pooling and maximum pooling, focusing on key locations corresponding to structures in SAR and optical images, avoiding wasting attention on background noise areas. Through this mechanism, the EFF module fuses features from the encoder and the current decoding layer layer by layer during the decoding phase. At each decoding level, the deep features from the previous layer are upsampled and concatenated with the features generated by the corresponding encoder. These features are then fused through the EFF module's attention, resulting in a fused feature that incorporates both detailed textures and high-level semantics. This fused feature is then fed into subsequent convolutions to recover the image deformation field. This cross-layer feature fusion design effectively overcomes the shortcomings of traditional methods that utilize only a single layer of features, enabling the registration network to simultaneously leverage multi-scale and multi-semantic information to match corresponding regions between SAR and optical images, thereby improving the accuracy and robustness of registration.

[0194] (3) Existing technologies optimize global transformation and local deformation separately, resulting in insufficient coordination during the registration process.

[0195] To address the disconnected global and local registration processes and the resulting uncoordinated optimization, this embodiment constructs an end-to-end joint optimization framework that integrates global geometric transformation and local fine-grained alignment. Architecturally, the method employs a spatial transformer network (STN) as a differentiable image transformation module, tightly coupled with a U-Net feature extraction subnetwork. This allows global and local adjustments in the registration process to be optimized simultaneously within the same network. The SAR image to be registered and the reference optical image are first concatenated in the channel dimension and fed into a U-Net network. A forward pass is then performed to obtain a two-dimensional pixel-level deformation field (i.e., a displacement vector for each pixel). This deformation field represents the combined transformation required to align the two images due to global rigid transformations (such as translation, rotation, and scale) and local nonlinear deformations (such as local offset of objects). The predicted deformation field is then applied to the SAR image to be registered via the STN module, where it undergoes geometric transformation and interpolation sampling to produce a registration result aligned with the optical image. In this process, global transformation parameters and local subtle deformations are jointly generated by the same network, avoiding the inconsistencies that may result from the traditional method of optimizing the global first and then the local independently. End-to-end back propagation is used during training: after the registration result and the reference image are combined to calculate the comprehensive loss, the gradient is passed back to the deformation field generation network through the STN, so that the network parameters representing the global and local registrations are updated synchronously. Since the registration process is regarded as a single differentiable system, this embodiment can coordinately optimize the overall pose alignment and local detail matching of the image under a unified framework, significantly improving the coordination and accuracy of the registration process. At the same time, the end-to-end joint optimization avoids the tedious feature matching and step-by-step optimization process, improves computational efficiency, and is more suitable for the registration needs of large-scale data in actual remote sensing applications.

[0196] (4) Existing methods use simple pixel-level error metrics and fail to fully capture the structural differences between cross-modal images.

[0197] To address the problem that the loss function design is single and difficult to capture the structural differences between modalities, this embodiment constructs a multivariate loss function that combines multiple metrics to evaluate and constrain the registration results from different levels to accurately capture the structural differences between SAR and optical images.

[0198] First, the mutual information (MI) loss is introduced to measure the statistical correlation between the two registered images. Mutual information quantifies the dependence of the grayscale distributions of SAR and optical images. It is robust to cross-modal registration with nonlinear intensity differences and encourages the registered images to achieve maximum correlation in their histogram distributions.

[0199] Secondly, a structural similarity (SSIM) loss is added to measure the similarity between the registered image and the reference image in terms of local brightness, contrast, and structural patterns. SSIM focuses on structural information in human visual perception. When the two images are more similar in local structures such as edges and textures, the SSIM value increases. By minimizing the SSIM loss, the registration results produced by the network are more consistent with the reference optical image in terms of detailed structure, effectively capturing the structural differences between cross-modal images.

[0200] In addition, a deformation regularization loss is added to impose a smoothness constraint on the predicted deformation field. This regularization term penalizes excessively drastic or discontinuous deformations, encouraging the estimated deformation field to vary smoothly in space, avoiding unreasonable distortions and ensuring the physical and geometric rationality of the registration transformation.

[0201] Finally, the structured distribution correlation loss models the SAR image as a Gamma distribution and the optical image as a Gaussian distribution, and performs weighted matching in the structural space, which has stronger cross-modal adaptation capabilities. The structure-sensitive weight function is introduced in the structural domain to achieve precise control of the matching of detail areas, effectively enhancing the edge and texture alignment effect. Compared with traditional MI or SSIM losses, SDC loss is insensitive to speckle noise and more sensitive to structural misalignment. It is suitable for registration task scenarios with high noise and strong structural differences. The SDC loss function is differentiable and can participate in backpropagation as part of the end-to-end training framework and be jointly optimized with registration modules such as the STN module.

[0202] The above four types of losses are combined according to appropriate weights to form the final training objective function. Through the comprehensive optimization of multiple indicators, the target loss function design of this embodiment overcomes the limitation of a single pixel difference metric (such as L2 loss) that cannot take into account significant differences between modalities, so that the model can take into account four aspects: global statistical matching, local structural alignment, deformation field smoothing, and structured distribution correlation during the training process. This more sophisticated and flexible loss function strategy ensures the quality of cross-modal registration: it maintains the consistency of the overall distribution between SAR and optical images, and retains the local structural features of the ground objects, thereby significantly improving the accuracy and robustness of the registration results in complex scenes. This embodiment proposes a new loss function that combines distribution difference modeling and structural domain correlation quantification to evaluate the structured statistical consistency between the registration results and the reference image.

[0203] Compared with the existing technology, this embodiment adopts an innovative deep learning architecture and module combination in the process of feature extraction, cross-layer fusion and image space transformation, which significantly improves the registration accuracy, robustness and computational efficiency.

[0204] Reference Figure 5The embodiment of the present application further provides a remote sensing image registration system based on enhanced feature fusion, which includes a training data acquisition unit 100, a network model construction unit 200, a deformation field acquisition unit 300, an image pre-registration unit 400, a loss function construction unit 500, a network model training unit 600, and a target image registration unit 700, wherein:

[0205] The training data acquisition unit 100 is used to acquire optical remote sensing images for model training and synthetic aperture radar images to be registered;

[0206] A network model construction unit 200 is used to construct an image registration network model including a deformation field generation network and a spatial transformation network, wherein the deformation field generation network is constructed based on a U-Net network and an enhanced feature fusion module;

[0207] The deformation field obtaining unit 300 is used to input the optical remote sensing image and the synthetic aperture radar image to be registered into the deformation field generation network to obtain the deformation field;

[0208] The image pre-registration unit 400 is used to input the synthetic aperture radar image to be registered and the deformation field into the spatial variation network to obtain a pre-registered image;

[0209] A loss function construction unit 500 is used to construct a target loss function based on the deformation field, the synthetic aperture radar image to be registered, the pre-registered image and the optical remote sensing image;

[0210] A network model training unit 600 is used to train the image registration network model according to the target loss function until the target loss function converges to obtain a trained image registration network model;

[0211] The target image registration unit 700 is used to perform remote sensing image registration on the target optical remote sensing image and the target synthetic aperture radar image to be registered using a trained image registration network model.

[0212] It should be noted that since the remote sensing image registration system based on enhanced feature fusion in this embodiment and the above-mentioned remote sensing image registration method based on enhanced feature fusion are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the system embodiment and will not be described in detail here.

[0213] Reference Figure 6 , an embodiment of the present application further provides an electronic device, the electronic device comprising:

[0214] at least one memory;

[0215] at least one processor;

[0216] at least one program;

[0217] The programs are stored in the memory, and the processor executes at least one program to implement the remote sensing image registration method based on enhanced feature fusion described above.

[0218] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0219] The electronic device according to the embodiment of the present application is described in detail below.

[0220] The processor 1600 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure.

[0221] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called by the processor 1600 to execute the remote sensing image registration method based on enhanced feature fusion in the embodiments of the present disclosure.

[0222] Input / output interface 1800, used for information input and output;

[0223] Communication interface 1900, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0224] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0225] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0226] An embodiment of the present disclosure further provides a storage medium, which is a computer-readable storage medium and stores computer-executable instructions. The computer-executable instructions are used to enable a computer to execute the above-mentioned remote sensing image registration method based on enhanced feature fusion.

[0227] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0228] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0229] Those skilled in the art will understand that the technical solutions shown in the drawings do not constitute a limitation on the embodiments of the present disclosure, and may include more or fewer steps than shown in the drawings, or a combination of certain steps, or different steps.

[0230] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0231] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0232] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0233] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0234] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0235] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0236] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0237] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in the relevant technical field without departing from the purpose of the present application.

[0238] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. A remote sensing image registration method based on enhanced feature fusion, characterized in that: The method comprises: Obtain optical remote sensing images for model training and synthetic aperture radar images to be registered; Constructing an image registration network model including a deformation field generation network and a spatial transformation network, wherein the deformation field generation network is constructed based on a U-Net network and an enhanced feature fusion module; Inputting the optical remote sensing image and the synthetic aperture radar image to be registered into the deformation field generation network to obtain a deformation field, comprising: splicing the optical remote sensing image and the synthetic aperture radar image to be registered to obtain a spliced ​​image; The spliced ​​image is used as the input of the deformation field generation network, and the encoder in the U-Net network is used to perform convolution feature extraction and downsampling on the spliced ​​image to obtain semantic features of multiple encoding levels; Upsampling the semantic features extracted from the last encoding layer by the decoder in the U-Net network to obtain the current upsampled features; Performing cross-modal feature fusion on the current up-sampled feature and the semantic feature of the next up-sampled corresponding coding level through the enhanced feature fusion module to obtain a target fused feature; The target fusion feature is upsampled by the decoder in the U-Net network, and the current upsampling result is further cross-modal fused with the semantic features of the next upsampling corresponding encoding level until the decoder completes decoding to obtain a deformation field; Inputting the synthetic aperture radar image to be registered and the deformation field into the spatial variation network to obtain a pre-registered image; constructing a target loss function based on the deformation field, the synthetic aperture radar image to be registered, the pre-registered image, and the optical remote sensing image; Training the image registration network model according to the target loss function until the target loss function converges to obtain a trained image registration network model; Remote sensing image registration is performed on the target optical remote sensing image and the target synthetic aperture radar image to be registered using the trained image registration network model.

2. The remote sensing image registration method based on enhanced feature fusion according to claim 1, characterized in that: The enhanced feature fusion module includes a feature enhancement submodule, a channel attention submodule, and a spatial attention submodule. The enhanced feature fusion module performs cross-modal feature fusion on the current upsampled feature and the semantic feature of the next upsampled corresponding encoding level to obtain the target fusion feature, including: Inputting the current up-sampled feature and the semantic feature of the next up-sampled corresponding coding level into the feature enhancement submodule to obtain a first fusion feature; Performing a residual connection on the first fused feature and the semantic feature of the next up-sampled corresponding coding level to obtain a second fused feature; Inputting the second fused feature into the channel attention submodule to obtain a third fused feature; The third fusion feature is input into the spatial attention submodule to obtain the target fusion feature.

3. The remote sensing image registration method based on enhanced feature fusion according to claim 1, characterized in that: Constructing a target loss function based on the deformation field, the synthetic aperture radar image to be registered, the pre-registered image, and the optical remote sensing image, including: constructing a mutual information loss function by calculating the mutual information between the synthetic aperture radar image to be registered and the optical remote sensing image; constructing a structural similarity loss function by calculating the structural similarity between the pre-registered image and the optical remote sensing image; Constructing a deformation regularization loss function according to the deformation field; Constructing a structured distribution correlation loss function by calculating the structured distribution correlation between the synthetic aperture radar image to be registered and the optical remote sensing image; The mutual information loss function, the structural similarity loss function, the deformation regularization loss function and the structured distribution correlation loss function are weightedly summed to obtain a target loss function.

4. The remote sensing image registration method based on enhanced feature fusion according to claim 3, characterized in that: The step of calculating the structured distribution correlation between the synthetic aperture radar image to be registered and the optical remote sensing image to construct a structured distribution correlation loss function includes: constructing local pixels in the synthetic aperture radar image to be registered into a gamma distribution; constructing local pixels in the optical remote sensing image into a Gaussian distribution; Calculating a first structure mapping value of the synthetic aperture radar image to be registered, and calculating a second structure mapping value of the optical remote sensing image; Calculating a correlation weight between the synthetic aperture radar image to be registered and the optical remote sensing image in a structure domain according to the first structure mapping value and the second structure mapping value; Calculating a structured distribution correlation between the synthetic aperture radar image to be registered and the optical remote sensing image according to the gamma distribution, the Gaussian distribution, and the correlation weight; Based on the structured distribution correlation, a structured distribution correlation loss function is constructed.

5. The remote sensing image registration method based on enhanced feature fusion according to claim 4, characterized in that: The calculating of the first structure mapping value of the synthetic aperture radar image to be registered, and the calculating of the second structure mapping value of the optical remote sensing image, include: ； in, represents the first structure map value of the synthetic aperture radar image to be registered, Indicates that the synthetic aperture radar image to be registered is The gradient of the point, Represents the second structure mapping value of the optical remote sensing image, Indicates that the optical remote sensing image is The gradient of the point, represents the vector differential operator.

6. The remote sensing image registration method based on enhanced feature fusion according to claim 4, characterized in that: The calculating, according to the gamma distribution, the Gaussian distribution, and the correlation weight, the structured distribution correlation between the synthetic aperture radar image to be registered and the optical remote sensing image comprises: ； in, represents structured distribution correlation, represents the sliding window size, represents the relevance weight, represents the probability density estimate of the current window calculated by gamma distribution, represents the global reference distribution of the synthetic aperture radar image to be registered, represents the probability density estimate of the current window calculated by Gaussian distribution, Represents the global reference distribution of optical remote sensing images.

7. A remote sensing image registration system based on enhanced feature fusion, characterized in that: The system comprises: A training data acquisition unit, used to acquire optical remote sensing images for model training and synthetic aperture radar images to be registered; A network model construction unit, configured to construct an image registration network model comprising a deformation field generation network and a spatial transformation network, wherein the deformation field generation network is constructed based on a U-Net network and an enhanced feature fusion module; A deformation field obtaining unit is configured to input the optical remote sensing image and the synthetic aperture radar image to be registered into the deformation field generation network to obtain a deformation field, comprising: splicing the optical remote sensing image and the synthetic aperture radar image to be registered to obtain a spliced ​​image; The spliced ​​image is used as the input of the deformation field generation network, and the encoder in the U-Net network is used to perform convolution feature extraction and downsampling on the spliced ​​image to obtain semantic features of multiple encoding levels; Upsampling the semantic features extracted from the last encoding layer by the decoder in the U-Net network to obtain the current upsampled features; Performing cross-modal feature fusion on the current up-sampled feature and the semantic feature of the next up-sampled corresponding coding level through the enhanced feature fusion module to obtain a target fused feature; The target fusion feature is upsampled by the decoder in the U-Net network, and the current upsampling result is further cross-modal fused with the semantic features of the next upsampling corresponding encoding level until the decoder completes decoding to obtain a deformation field; an image pre-registration unit, configured to input the synthetic aperture radar image to be registered and the deformation field into the spatial variation network to obtain a pre-registered image; a loss function construction unit, configured to construct a target loss function based on the deformation field, the synthetic aperture radar image to be registered, the pre-registered image, and the optical remote sensing image; A network model training unit, configured to train the image registration network model according to the target loss function until the target loss function converges, thereby obtaining a trained image registration network model; The target image registration unit is used to perform remote sensing image registration on the target optical remote sensing image and the target synthetic aperture radar image to be registered by using the trained image registration network model.

8. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the remote sensing image registration method based on enhanced feature fusion as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the remote sensing image registration method based on enhanced feature fusion according to any one of claims 1 to 6.

Images