A domain attention-based SAR and optical heterogeneous remote sensing image registration method

Abstract

The present invention relates to a SAR and optical heterogeneous remote sensing image registration method based on domain attention, comprising the following steps: considering the premise of structural similarity constraints, performing inter-domain confrontation on the reference image and the image to be registered, generating a domain migration image, and extracting features; performing wavelet transform operations at multiple scales, weakening inherent noise, and extracting enhanced features; performing multi-level feature fusion to obtain predicted displacement values, calculating transformation parameters, and obtaining registered images. The beneficial effects of the present invention are: through inter-domain confrontation and structural similarity constraints, modal migration to the same common subspace is achieved; on the basis of reducing modal differences, wavelet transform is used to filter out noise, pixel-level spectral matching is performed, and the modal identifiability of features is enhanced, and two-step registration is performed from coarse to fine, so as to achieve end-to-end registration of heterogeneous remote sensing images, effectively resist the influence of nonlinear radiation differences, and have strong robustness to geometric distortion.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image processing, and in particular relates to a SAR and optical heterogeneous remote sensing image registration method based on domain attention. Background Art

[0002] At present, with the launch of many remote sensing satellites and facilities, multi-sensor collaborative observation technology has made significant progress. Optical images provide high-resolution and visually rich information by capturing the reflective characteristics of the surface of objects. SAR images can reflect radar signals from ground objects and have all-weather observation and cloud penetration capabilities, but their spatial resolution is relatively low. Combining the complementary advantages of the two has important practical value. The effective registration of optical and SAR remote sensing images can lay a solid foundation for collaborative analysis and processing such as change detection, image fusion, and image stitching in subsequent remote sensing applications. Due to the significant radiation and geometric differences between optical and SAR images, how to achieve high-precision registration of heterogeneous remote sensing images remains a difficult problem that needs to be solved urgently.

[0003] Existing remote sensing image registration methods include traditional methods and deep learning methods, among which traditional methods can be divided into feature-based and region-based registration strategies. Among them, feature-based registration methods perform spatial alignment between images based on detecting feature points and constructing descriptors, and region-based registration finds the optimal transformation model through shallow information similarity. However, feature-based and region-based traditional registration methods are easily affected by nonlinear radiation differences, and it is difficult to achieve high-precision registration between SAR and optical heterogeneous remote sensing images. With the development of deep learning theory, image registration methods based on deep learning have developed rapidly. Compared with supervised learning, unsupervised learning does not require the deformation field or transformation matrix of the image to be registered to the reference image coordinate system as a label, has strong adaptability, can better capture the hidden structure of the data, and has higher development potential. However, due to factors such as the diversity of remote sensing image scenes, wide imaging range, and noise mixing, it is difficult to perform unsupervised robust registration between SAR and optical images. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a SAR and optical heterogeneous remote sensing image registration method based on domain attention.

[0005] This domain attention-based SAR and optical heterogeneous remote sensing image registration method includes the following steps:

[0006] S1. Establish a dataset of SAR and optical heterogeneous remote sensing images that have not yet been registered, including reference images and images to be registered;

[0007] S2. Considering the premise of structural similarity constraint, the reference image and the image to be registered are subjected to inter-domain confrontation to generate a domain migration image and extract features;

[0008] S3, performing wavelet transform operations at multiple scales to weaken inherent noise and extract enhanced features;

[0009] S4. Perform multi-level feature fusion to obtain the predicted displacement value, calculate the transformation parameters, and obtain the registered image.

[0010] S5. Calculate the total network loss and optimize it.

[0011] As a preferred embodiment, step S3 specifically comprises: for the features proposed in step S2, using secondary wavelet transform to decompose and generate low-frequency components F at different scales LL 、M LL and high frequency component F HH 、M HH , upsample the high-frequency component features, calculate the spectral correlation of each pixel in the high-frequency component features, and reconstruct the high-frequency and low-frequency components after filtering out inherent noise to obtain enhanced features.

[0012] Preferably, in step S4, the enhanced features obtained in step S3 are further extracted by a dense residual network, and the global features F are extracted by Trans Block. global , extracting local information F through the empty spatial convolution pooling pyramid local , displacement parameters are predicted for global features and local information respectively, and the image is registered in two steps from coarse to fine.

[0013] Preferably, in step S4, a Trans Block fused with grouped convolution is used to capture local features, realize feature interaction and global modeling, and extract global features F global .

[0014] Preferably, in step S4, multiple parallel atrous convolution layers with different sampling rates are used to capture information of different receptive fields. The features of each sampling rate are processed in an independent branch. The features of these branches are fused by superimposing different output features, and finally a 1×1 convolution is used for dimensionality reduction to generate feature extraction local information F. local .

[0015] Preferably, in step S5, the total network loss is calculated by a hybrid loss function of spatial loss and image similarity loss, the spatial loss evaluates the difference between the predicted matching point displacement and the true value, and the image similarity loss is used to measure the degree of alignment between the reference image and the registered image.

[0016] The beneficial effects of the present invention are:

[0017] 1) This paper specifically proposes a GAN model based on a dual generator, which realizes modal migration to the same common subspace through inter-domain adversarial and structural similarity constraints, while retaining key structural detail features; on the basis of reducing modal differences, a multi-scale remote sensing image registration network is designed, which uses wavelet transform to filter noise, performs pixel-level spectral matching, and enhances the modal identifiability of features, and performs two-step registration from coarse to fine to achieve end-to-end registration of heterogeneous remote sensing images. It can effectively resist the influence of nonlinear radiation differences, has strong robustness to geometric distortion, and can cope with the registration challenges of heterogeneous remote sensing images.

[0018] 2) When performing inter-domain adversarial operations, the present invention maintains the structural similarity between the intermediate image and the original image to prevent the images generated by the two generators from tending to converge to an extreme state, resulting in almost no difference between the generated images, thereby reducing the modal differences between heterogeneous remote sensing images and retaining unique features and recognizability. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram for generating unregistered SAR and optical heterogeneous remote sensing image datasets;

[0020] Figure 2 This is a flowchart of the unsupervised registration method of SAR and optical heterogeneous remote sensing images based on domain attention;

[0021] Figure 3 It is a structural diagram of the wavelet domain matching module;

[0022] Figure 4 It is the structural diagram of Trans Block;

[0023] Figure 5 This is a schematic diagram of the structure of the dilated spatial convolutional pooling pyramid module;

[0024] Figure 6 Flowchart for two-step registration;

[0025] Figure 7 Schematic diagram of the results of different methods in different registration scenarios;

[0026] Figure 8 Schematic diagram of the reference image, the image to be registered and the chessboard image in different scenes of this method;

[0027] Figure 9 is a comparison of three indicators of different methods for registering SAR images to MS images (including Figure 9a RMSE comparison chart for registering SAR images to MS images. Figure 9b MI comparison chart for registering SAR image to MS image, Fig.9cNCC comparison diagram for registering SAR images to MS images);

[0028] Figure 10 is a comparison of three indicators of different methods for registering MS images to SAR images (including Fig.10a RMSE comparison chart for registering MS images to SAR images. Fig.10b MI comparison chart for registering MS image to SAR image, Fig.10c NCC comparison diagram for registering MS images to SAR images);

[0029] Figure 11 is a comparison of three indicators of different methods for registering SAR images to RGB images (including Fig.11a RMSE comparison chart for registering SAR images to RGB images. Fig.11b MI comparison chart for registering SAR images to RGB images, Fig.11c NCC comparison chart for registering SAR images to RGB images);

[0030] Figure 12 is a comparison of three indicators of different methods for registering RGB images to SAR images (including Fig.12a RMSE comparison chart for registering RGB images to SAR images. Figure 12b MI comparison chart for registering RGB image to SAR image, Fig.12c NCC comparison chart for registering RGB images to SAR images). DETAILED DESCRIPTION

[0031] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0032] Embodiment 1

[0033] As an example, Figures 1 to 6 As shown in the figure, this domain attention-based SAR and optical heterogeneous remote sensing image registration method migrates SAR images and optical images to the same common subspace through an inter-domain adversarial network to reduce modal differences. Subsequently, the wavelet domain matching module is used to enhance the recognizability of the modality, and information fusion is performed at different scales to extract global and local information, and a two-step registration is performed from coarse to fine. The following steps are included:

[0034] Step 1: Establish a SAR and optical heterogeneous remote sensing image dataset that has not yet been registered. The dataset is divided into reference images and images to be registered.

[0035] Step 2: Perform inter-domain adversarial analysis on the reference image and the image to be registered to generate a domain migration image, thereby reducing the modality difference;

[0036] output = Conv 3 (Conv 1 (Concat(Conv 1 (input)+Conv 3 (Conv 3 (Conv 1 (input))))))

[0037] Among them, input represents the input image feature, Conv 1 It is a combination of convolution kernel 1 and ReLU activation function. 3 It is a combination of a convolution kernel of 3 and a ReLU activation function. Concat represents the concatenation operation of the input features in the channel dimension, and output represents the generator output features.

[0038] Step 3: Perform wavelet transform operations at multiple scales to weaken inherent noise and extract enhanced features. Based on the features extracted in step 2, generate low-frequency components F using secondary wavelet transform decomposition at different scales. LL ,M LL and high frequency component F HH ,M HH .

[0039] F LL ,F HH =wavelet(F);M LL ,M HH =wavelet(M)

[0040] Among them, wavelet represents the secondary wavelet decomposition operation, F and M represent the reference image and the image to be registered respectively.

[0041] Upsample the high-frequency component features, calculate the spectral correlation of each pixel in the feature, and then reconstruct the high-frequency and low-frequency components after filtering out the inherent noise to obtain the enhanced features;

[0042] F′ HH =||resample(F HH )|| 1

[0043] M′ HH =||resample(M HH )|| 1

[0044] λ=Sigmoid(Conv 1×1 (Sin(F′HH ,M′ HH ))

[0045]

[0046] Among them, F′ HH and M′ HH Represents the high frequency component F HH ,M HH The features after upsampling and l1 regularization, Sim represents the pixel-based spectral similarity measurement function, Conv 1×1 Represents a convolution operation with a convolution kernel of 1, Sigmoid is a nonlinear activation function, ⊙ represents dot product, λ represents the spectral correlation coefficient, restructure represents the reconstruction of low-frequency components and high-frequency components after filtering, and G is the module output feature.

[0047] Step 4: Perform multi-level feature fusion to obtain the predicted displacement value, calculate the transformation parameters, and obtain the registered image. The output features of the module in step 3 are further extracted through the dense residual network (Dense Path);

[0048] F 1a =Relu(Conv 1×1 (I))

[0049] F 1b =Relu(Conv 3×3 (I))

[0050]

[0051] F ia =Relu(Conv 1×1 (O i ))

[0052] F ib =Relu(Conν 3×3 (O i ))

[0053]

[0054] Among them, Conv and Relu represent convolution operation and activation function respectively, I represents input features, and F ia and F in Respectively represent the i-th layer features after 1×1 convolution and 3×3 convolution in the dense residual network, O i Represents the output of the i-th layer of the dense residual network. Symbol represents the feature addition operation, and Cat represents the feature superposition operation.

[0055] Extract global feature F through Trans Block global ;

[0056] F global =X+Concat(Linear(X+Attetion(X)))

[0057] Where X is the input of the aggregated dense residual network output features and upsampled features, Attention represents the attention module using grouped convolution, and Linear represents the linear layer.

[0058] Extract local information F through atrous spatial convolutional pyramid pooling (ASPP) local .

[0059] F local =Conv 1 (Concat(Pooling(X),Con 6,12,18 (X),Con 1 (X)))

[0060] Among them, Con i is a convolution operation with convolution kernel i, and Pooling is a pooling operation.

[0061] The displacement parameters of global features and local information are predicted respectively, and the image is registered in two steps from coarse to fine.

[0062] Matrix=STN(P)

[0063] Where P is the predicted displacement parameter, STN stands for spatial transformation network, and Matrix represents the transformation matrix parameter.

[0064] Step 5: Optimize through a hybrid loss function of spatial loss and image similarity.

[0065] Embodiment 2

[0066] As another embodiment, this embodiment 2 proposes a more specific SAR and optical heterogeneous remote sensing image registration method based on domain attention on the basis of embodiment 1:

[0067] Step 1: Establish a SAR and optical heterogeneous remote sensing image dataset that has not yet been registered. The dataset is divided into reference images and images to be registered.

[0068] Specifically, Figure 1 As shown in FIG. 1 , firstly, one of the images in the pixel-aligned image pair is selected to perform a random perspective transformation to form a distorted image, and then the image I 1 ′ and I2 Perform center cropping to generate moving image T 1 and a fixed image T 2 , and record the true value of displacement before and after the image transformation.

[0069] (Δx true ,Δy true )=(x m ,y m )-(x f ,y f )

[0070] Among them, (x m ,y m ) and (x f ,y f ) represent the position parameters of the moving point and the fixed point respectively. In this way, a large number of image pairs are generated to form the training and testing data sets.

[0071] Step 2: Perform inter-domain adversarial operations on the reference image and the image to be registered. The input image pairs are mutually true values ​​and are close to each other in the mode of the other image. If the structural similarity between the intermediate image and the original image is not considered, the images generated by the two generators will tend to converge to an extreme state. The generated images become almost indistinguishable from each other in order to be close to each other. They lose their unique features and recognizability and cannot achieve the effect of image domain migration. Therefore, the structural similarity constraint is introduced to reduce the modal difference while retaining the structural details of the original image, which facilitates the subsequent feature extraction work.

[0072] In step 3, if Figure 3 As shown, the low-frequency component F is generated by wavelet decomposition. LL ,M LL and high frequency component F HH ,M HH Rich edge details are extracted from the high-frequency components after wavelet transformation. The spectral information and deep convolution features can be combined to perform feature matching to obtain feature correlation coefficient graphs at the same scale. At the same time, the low-frequency components correspond to the global structure and overall changes of the image, contain larger-scale features, are less affected by noise, and have a certain degree of robustness to noise. Reconstructing them is conducive to reducing noise and improving quality.

[0073] Step 5: Optimize the network from two aspects: spatial loss and image similarity loss. The total loss of the network is:

[0074] L net =ω a L mse1 +ω b L mse2 +λL ncc,ω a +ω b =1

[0075] Among them, ω a ,ω b ,λ is the loss weight; L mse is the Euclidean distance, which is used to evaluate the difference between the predicted displacement of the matching point and the true value. ncc To measure the degree of alignment between the reference image and the registered image.

[0076] NCC evaluates the similarity of two image regions by calculating the cross-correlation between them, and then normalizes the result to eliminate the effects of brightness and contrast. It takes into account the correlation of the entire image region, not just a single pixel, so it can measure the similarity between a given pair of images.

[0077]

[0078] In L mse In the above example, N is the number of training samples, P is pre and P ture Represent the displacement prediction value and the true value respectively. ncc In which f(i,j) is the original image, t(i,j) is the template image, n is the number of pixels in the template, μ f and μ t Represent the mean of the original image and the template image, σ f and σ t Represent the standard deviation of the original image and the template image respectively.

[0079] It should be noted that the parts in this embodiment that are the same or similar to those in the first embodiment can be referenced to each other and will not be described in detail in this application.

[0080] Embodiment 3

[0081] As another embodiment, this embodiment 3 proposes a more specific SAR and optical heterogeneous remote sensing image registration method based on domain attention on the basis of embodiments 1 and 2, and step 4 specifically includes the following steps:

[0082] Step 4.1: Further extract the output features of the wavelet domain matching module in step 3 through a dense residual network (Dense Path). Residual learning allows information to be directly transmitted between different layers. Through cross-layer connections, it helps to better back-propagate gradients, alleviate the gradient vanishing problem, and improve the training rate. The network can also better capture the feature relationship between input and output. Dense Path, as a residual network that combines different scales, can improve the representation ability of the model, while helping to improve the performance and generalization ability of the model.

[0083] Step 4.2: Extract global feature F through Trans Block global Considering the global information, Transformer is chosen to fuse the image information. Standard Transformer often contains a large number of parameters and occupies a large amount of memory. In order to increase the running speed, reduce the number of parameters, and reduce memory consumption, such as Figure 4 As shown, in this embodiment, a Trans Block that integrates grouped convolution is proposed, which can better capture local features while achieving longer-range feature interaction and global modeling through the Transformer model, thereby improving the model's comprehensive understanding of image features.

[0084] Step 4.3: Extract local information F through atrous spatial convolutional pyramid pooling (ASPP) local The ASPP module uses multiple parallel atrous convolutional layers with different sampling rates to capture information of different receptive fields. This multi-scale feature extraction enables the model to better understand the semantic information of objects of different sizes in the image. The features of each sampling rate are processed in an independent branch, and finally a more comprehensive and accurate feature extraction result is generated by fusing the features of these branches, which significantly enhances the model's ability to capture multi-scale object information. Figure 5 As shown in the figure, in this paper, 1×1 convolution, hole convolution with rates of 6, 12, and 18, and adaptive maximum pooling are used to expand the receptive field of convolution. The continuity of information is retained by superimposing and combining different output features. Finally, 1×1 convolution is used for dimensionality reduction to output feature maps.

[0085] Step 4.4: Predict the displacement parameters of the global features and local information respectively, and perform a two-step registration of the image from coarse to fine to prevent falling into the local optimal solution. The registration diagram is shown in Figure 6 As shown. Fixed points are represented by red points, and moving points are represented by green points. Two different sets of fixed points are selected. First, a rough registration is performed by global features. The rough registration takes the fixed point on the dotted box as the target, moves from the green point to the blue point, and then projects the blue point onto the yellow point. Then, point mapping after rough registration is performed, and then the image features after rough registration are captured, local information is extracted, and fine registration is performed to accurately align the yellow point with the fixed point in the solid box. The blue point is recorded as (x′ i ,y′ i ), and the perspective matrix D is expressed as the projection of the reference point from the dotted frame coordinate system to the solid line coordinate system. The rough matching yellow point after mapping can be obtained, denoted as (x″ i ,y″ i ).

[0086] (x″i ,y″ i ,1) T =D(x′ i ,y′ i ,1) T

[0087] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiments 1 and 2 can be referenced to each other and will not be described in detail in this application.

[0088] Embodiment 4

[0089] As another embodiment, this embodiment four is proposed on the basis of embodiments one to three, and uses this domain attention-based SAR and optical heterogeneous remote sensing image registration method to perform image registration on PS-RGB (RGB), multispectral (MS) and SAR intensity (SAR) data sets from the SpaceNet data set.

[0090] The dataset has unique 0.5-meter synthetic aperture radar (SAR) images from Capella Space and 0.5-meter electro-optical images from Maxars WorldView 2 satellite. The scenes are concentrated in the largest port in Europe: Rotterdam, the Netherlands, containing thousands of buildings, vehicles, and ships of various sizes, with a variety of scenes. For SAR images, their data in the x-band have four polarizations, namely HH, HV, VH, and VV, of which HH polarization is used for training and testing. Since some images contain blank areas, the blank areas are processed as pure black in this embodiment and added to the training set and test set as difficult-to-separate samples. As an embodiment, the present invention uses the computer software Pycharm to perform affine transformation on the selected aligned image pairs to generate an unregistered dataset, and implements the automatic registration process based on the Pytorch framework of the NVIDIA GeForce RTX 3090 GPU.

[0091] In step 1, based on the existing data set, this embodiment selects the images that have been accurately aligned, uses Pycharm software to perform perspective transformation on one of the images to obtain a changed image, and then performs center cropping of the same size to form a registration data set of the reference image and the image to be registered. The prepared registration data set is normalized and preprocessed to avoid overfitting, gradient explosion and other problems caused by too large original data and small data volume during deep learning training, and the training set and test set are divided at the same time.

[0092] Step 2: Since optical and SAR images are generated by different sensors, there are inherent differences in hardware. Among them, the synthetic aperture radar coherent imaging mechanism will produce inherent coherent speckle noise, which is an unavoidable phenomenon and may cause phase discontinuity, confuse target features, and make target extraction and matching more difficult. Therefore, for the different information contained in the two images, generators with different parameters are constructed to perform inter-domain adversarial operations. The addition of maintaining the structural similarity between the intermediate image and the original image prevents the images generated by the two generators from tending to converge to an extreme state - the generated images become almost indistinguishable in order to approach each other. In this way, the effect of image domain migration is achieved, the modal differences between heterogeneous remote sensing images are reduced, and unique features and recognizability are retained.

[0093] Step 3: Construct a wavelet domain matching module. First, generate low-frequency components and high-frequency components through wavelet decomposition. By combining the spectral information of the high-frequency component with the deep convolution feature to perform feature matching, a feature correlation coefficient graph at the same scale is obtained. The low-frequency component corresponds to the global structure and overall change of the image, contains larger-scale features, is less affected by noise, and has a certain degree of robustness to noise. Its reconstruction is conducive to reducing noise and improving quality. Perform different weight addition operations for each pixel to enhance similar structural features, modal invariance and recognizability.

[0094] Step 4: Further extract the output features of the wavelet domain matching module through a dense residual network. Residual learning allows information to be directly transmitted between different layers. Through cross-layer connections, it helps to better back-propagate gradients, alleviate the gradient vanishing problem, and improve the training rate. The network can also better capture the feature relationship between input and output, improve the representation ability of the model, and help improve the performance and generalization ability of the model. Global features and local information are extracted respectively through the Transformer-based Trans Block and the dilated spatial convolution pooling pyramid. The two-step registration process from coarse to fine is performed by combining global features and local information.

[0095] Step 5: According to the unsupervised end-to-end network training strategy, the loss function for spatial and image similarity is established using mean square error and normalized cross-correlation to optimize the network, and the Adam optimizer is used for optimization. In order to focus on accurate registration, the spatial loss function is assigned more weight.

[0096] In this embodiment, comparative experiments were conducted on SAR images and RGB images, as well as SAR images and multispectral images in different scenarios to verify the results of this method:

[0097] The experiment adopted four traditional registration methods (SIFT, PSO-SIFT, CFOG and RIFT) and two deep learning-based registration methods (D2-Net and LoFTR) for comparison with the method proposed in this invention. Figure 7 Schematic diagram of the results of different methods in different registration scenarios. Figure 8 The following are the reference images, images to be registered, and chessboard images of this method in different scenes. In order to more intuitively show the accuracy of the registration results, green boxes and red boxes are used to represent the ranges formed by fixed points and displacement points respectively. The larger the area covered by the two boxes, the more significant the registration effect.

[0098] Figures 9a to 12c The three indicators are compared by different methods: registering SAR images to MS images, registering SAR images to MS images, registering SAR images to RGB images, and registering RGB images to SAR images.

[0099] from Figure 7 It is not difficult to find that in the scenario of MS and SAR image registration, the SIFT and PSO-SIFT methods cannot perform effective registration due to the existence of nonlinear radiation differences. RIFT can effectively capture some significant features to improve the registration accuracy, and CFOG performs registration by matching structural features. The registration effects of both are relatively unstable. In view of the good generalization ability of deep learning, the registration effects of D2-Net and LoFTR are relatively stable. In horizontal comparison, the method proposed in the present invention has the highest box overlap and the best registration effect. In the scenario of RGB and SAR image registration, SIFT and PSO-SIFT are much better than those in MS and SAR scenarios, but they still cannot perform effective registration. Due to the existence of modal differences, the box overlap of other comparison methods is not as high as that of the method proposed in the present invention, and the registration accuracy is slightly insufficient.

[0100] like Figure 8 As shown, the scene includes RGB-SAR, SAR-RGB, MS-SAR, and SAR-MS. The method proposed in the present invention superimposes the matching results to form a checkerboard image, which is convenient for better displaying the degree of alignment. Although there are some deviations in the alignment of some buildings, from the perspective of global pixels, the selected areas can be effectively registered. Comparing different scenes, when the impact is less missing, the impact on the accuracy of registration is not obvious; however, when the image is missing more, the accuracy of registration will decrease. The addition of difficult-to-separate samples can increase the robustness of the model to a certain extent, so that the model can be effectively registered even when the amount of information is less.

[0101] Figures 9a to 12cThe comparison charts of the indicators of 10 groups of image pairs in different registration scenarios are shown respectively. For SIFT and PSO-SIFT, due to the differences between optical and SAR, SIFT may not be able to capture the key feature points in SAR images, resulting in the inability of both methods to perform heterogeneous image registration well. RIFT, as a feature matching method for multi-modal remote sensing images, has a relatively stable registration capability for different types of optical and SAR image data sets, and has good translation and rotation invariance. CFOG is a template matching method with fast computational efficiency and strong matching performance. It is relatively robust to radiation differences, but is sensitive to rotation and scale differences, resulting in slightly worse results on some data. D2-Net and LoFTR, as deep learning methods, have good robustness for feature point extraction in complex environments, laying a good foundation for the subsequent feature matching stage, and the obtained registration results are relatively stable. However, due to the small size of the image pairs to be registered, the existence of ground object changes and nonlinear radiation differences between optical and SAR heterogeneous remote sensing images, the quality of the feature points they extract is poor, and the RMSE is ≥ 3 on the test data set, and the registration accuracy is poor.

[0102] In summary, the method proposed in the present invention is significantly superior to the compared methods. It can effectively resist the influence of nonlinear radiation differences, has strong robustness to geometric distortion, and can cope with the challenges of registration of heterogeneous remote sensing images.

[0103] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiments 1 to 3 can be referenced to each other and will not be described in detail in this application.

[0104] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

Claims

1. A domain attention-based SAR and optical heterogeneous remote sensing image registration method, characterized in that: The following steps are involved: S1. Establish a dataset of SAR and optical heterogeneous remote sensing images that have not yet been registered, including reference images and images to be registered; S2. Considering the premise of structural similarity constraint, the reference image and the image to be registered are subjected to inter-domain confrontation to generate a domain migration image and extract features; in, Represented as input image features, It is a combination of convolution kernel 1 and ReLU activation function. It is a combination of convolution kernel 3 and ReLU activation function. represents the concatenation operation of the input features in the channel dimension, Represents the output features of the generator; S3, perform wavelet transform operations at multiple scales to weaken inherent noise and extract enhanced features; for the features proposed in step S2, use secondary wavelet transform to decompose and generate low-frequency components at different scales , and high frequency components , , upsample the high-frequency component features, calculate the spectral correlation of each pixel in the high-frequency component features, and reconstruct the high-frequency and low-frequency components after filtering out inherent noise to obtain enhanced features; S4, perform multi-level feature fusion to obtain the predicted displacement value, calculate the transformation parameters, and obtain the registered image; further extract the enhanced features obtained in step S3 through the dense residual network, and extract the global feature F through Trans Block global , extracting local information F through the empty spatial convolution pooling pyramid local , respectively predict the displacement parameters of global features and local information, and perform a two-step registration of the image from coarse to fine; S5. Calculate the total network loss and optimize it.

2. The domain attention-based SAR and optical heterogeneous remote sensing image registration method according to claim 1, characterized in that: In step S4, the Trans Block fused with grouped convolution is used to capture local features, realize feature interaction and global modeling, and extract the global feature F global .

3. The domain attention-based SAR and optical heterogeneous remote sensing image registration method according to claim 1, characterized in that: In step S4, multiple parallel atrous convolution layers with different sampling rates are used to capture information of different receptive fields. The features of each sampling rate are processed in an independent branch. The features of these branches are fused by superimposing different output features, and finally a 1×1 convolution is used to perform dimensionality reduction to generate feature extraction local information F local .

4. The domain attention-based SAR and optical heterogeneous remote sensing image registration method according to claim 1, characterized in that: In step S5, the total network loss is calculated by a hybrid loss function of spatial loss and image similarity loss. The spatial loss evaluates the difference between the predicted matching point displacement and the true value, and the image similarity loss is used to measure the degree of alignment between the reference image and the registered image.

Images