Multi-modal remote sensing image matching method based on modal transformation and comparative learning

Abstract

The invention discloses a multi-modal remote sensing image matching method based on modal transformation and comparative learning. The method comprises the following steps: S1, constructing a training data set; s2, constructing a generation path of an image continuous domain; s3, designing a shortest path constraint; s4, defining the total loss of image conversion; s5, extracting rotation invariance feature expression of comparative learning; and S6, establishing a feature matching framework, and outputting a matching result. According to the method, potential sharing features are mined, the generation process of image translation is converted into a step-by-step generation mode based on a network path, the shortest path constraint is realized by constructing an intermediate domain, and the significant nonlinear radiation difference and noise between a reference image and an image to be registered are effectively eliminated; and meanwhile, by utilizing an EfficientNet architecture optimized by an expansion convolution and an attention mechanism, the representation quality of deep features is remarkably improved, and a training system of sample augmentation is designed in combination with twinning and pseudo-twinning networks to obtain better rotation consistency feature expression, so that the matching robustness of a descriptor to a weak texture region is effectively enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing, and in particular relates to a multimodal remote sensing image matching method based on modality conversion and contrast learning. Background Art

[0002] The gradual deployment of remote sensing satellites and related facilities has promoted the development of integrated, multi-sensor, space-ground, stereoscopic observation technology. Remote sensing imaging systems have entered a diversified era characterized by multimodality, multispectrality, and multiresolution. Multimodal remote sensing images of varying resolutions can reflect the diverse characteristics of surface targets. Leveraging these images facilitates multi-layered, all-encompassing, and all-weather Earth observation and provides more comprehensive and accurate information support for real-time intelligent geometric processing of multimodal remote sensing data.

[0003] Data-driven methods based on deep learning, leveraging their "learning-feedback-adjustment" strategy to extract high-level semantic information from large amounts of data, have become a research hotspot in the field of multimodal remote sensing image registration. Many studies have applied deep learning to image registration, improving its accuracy by extracting more refined common features than traditional handcrafted features, particularly in addressing geometric distortion and radiometric differences. Deep learning-driven registration methods can be broadly divided into two categories: modular registration methods, which embed deep learning technology into a specific step of the traditional registration process, replacing certain modules with neural networks (such as convolutional neural networks and siamese networks); and end-to-end registration methods, which directly implement image registration by constructing an end-to-end neural network. While end-to-end methods can avoid local optima and achieve global optimal solutions, their training process is resource-intensive, the models are complex, and they lack interpretability.

[0004] A common strategy for modular registration methods is to integrate deep learning networks into traditional feature- or region-based registration methods. Leveraging deep learning's data-driven nature and high-dimensional feature extraction, they generate more robust features and descriptors, improving the robustness of image registration. Some methods employ image transformation techniques to unify the textures of multimodal images and eliminate radiometric differences, transforming the multimodal registration problem into a unimodal one and applying traditional unimodal matching methods (such as SIFT). While learning-based template matching methods have improved in handling radiometric differences in multimodal remote sensing images, they still struggle to cope with significant scale and rotation variations between images. Learning-based feature matching methods, through deep feature extraction and high-dimensional feature representation, improve the stability of feature detection and the robustness of descriptors. However, existing research on multimodal remote sensing image registration has largely focused on feature description, with limited exploration of feature detection. Furthermore, some learning-driven feature matching methods fail to consider scale and rotation differences and therefore lack scale and rotation invariance.

[0005] Image translation-based registration methods first perform image-to-image translation through training, ensuring that the multimodal remote sensing images to be registered have similar radiometric information and structural features. Although architectures such as generative adversarial networks (GANs) have achieved significant results in multimodal remote sensing image translation, the connection between the target image and the generated image remains weak, and they lack comprehensive consideration of multimodal image features such as detail and texture, resulting in incorrect grayscale value assignments or blurring in the generated image. Furthermore, although multimodal images generally reflect the geometry of ground targets and possess potential shared features, existing methods often overlook these potential shared features. Therefore, leveraging these potential shared features to generate new information that is closer to the true target image through knowledge learning, and combining them with contrastive learning techniques to extract local deep common features that are radiometrically and rotationally invariant, remains an important research direction.

[0006] In view of this, the present invention is proposed. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to overcome the shortcomings of the existing technology and provide a multimodal remote sensing image automatic matching method with high translation fidelity and strong matching robustness. By mining potential shared features, the image translation generation process is converted into a step-by-step generation method based on network paths, and the shortest path constraint is realized by constructing an intermediate domain, thereby effectively eliminating the significant nonlinear radiation differences and noise between the reference image and the image to be registered. At the same time, the present invention uses the EfficientNet architecture optimized by dilated convolution and attention mechanism to significantly improve the representation quality of deep features, and combines the twin and pseudo-twin network design sample augmentation training system to obtain better rotation consistency feature expression, effectively enhancing the robustness of the descriptor to matching of weak texture areas.

[0008] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is:

[0009] The multimodal remote sensing image matching method based on modality conversion and contrastive learning includes the following steps:

[0010] S1, build training data set:

[0011] Obtain a reference image and an image to be registered that contain the same area, and pair them to form multiple multimodal image pairs; combine all the multimodal image pairs into a multimodal image pair dataset, and divide the dataset into a training dataset and a test dataset;

[0012] S2, constructs the generation path of the image continuous domain, specifically:

[0013] S21 uses a shared network encoder to extract the potential shared features of the training dataset in S1 and embeds them into a shared latent space;

[0014] S22 designs an image generation path from one domain to another, trains the decoder G to reconstruct the input image x (Reconstruction term) and finds the end point of the image generation path;

[0015] S23 uses the generative adversarial network GAN to construct the loss function and train the discriminator D to distinguish the generated image G(f x ,1) with the real image y∈Y;

[0016] S3, design the shortest path constraints, specifically:

[0017] S31, to find the minimum path length, the energy functional of the curve is optimized;

[0018] S32, penalizes the path length of multi-layer features:

[0019] S33, given the features of layer L, obtain the final shortest path length regularization loss;

[0020] S4, overall loss of image conversion:

[0021] The overall loss also introduces reconstruction loss (identity term) on the Y domain to further regularize the network; in order to retain more details, the structural similarity SSIM loss is introduced into the reconstruction loss and the reconstruction loss;

[0022] Using the above different loss functions to form the overall loss function, we have the following formula:

[0023]

[0024] Where λ kl ,λ rec ,λ sp ,λ idt They are the hyperparameters for balancing different loss functions.

[0025] The ultimate goal is to achieve the following:

[0026]

[0027] Training the image translation network until the loss function LOSS converges to obtain a trained multimodal image translation model;

[0028] S5, the rotation invariant feature expression of contrastive learning, specifically:

[0029] S51, feature extraction architecture:

[0030] The EfficientNet-B1 network with dilated convolution and attention mechanism is selected as the basic backbone network, and its input and output are adaptively adjusted to adapt to the task of local deep feature extraction.

[0031] S52 sample augmentation training system:

[0032] The training dataset in S1 is transformed using the multimodal image conversion model trained in S4, and image pairs with similar radiometric information are output. Then, affine transformations and circular masks with different rotation angles (0 to 360 degrees) are added to the transformed images, and the corresponding pixel relationships between these transformed images and the reference image are obtained. Finally, the Triplet Loss is used to improve the difference between matching image patches and non-matching image patches by combining the Siamese and Pseudo-Siamese networks and minimizing the distance between matching descriptors and maximizing the distance between non-matching descriptors. The loss is implemented as follows:

[0033]

[0034] In the above formula, α is the regularization parameter, θ i + and θ i - represent positive samples and negative samples respectively, and x i is a pair of positive sample descriptors before normalization. S H The calculation formula for (θ) is as follows:

[0035]

[0036] where a is a hyperparameter that balances the distance metric and the correlation metric, and z is a normalization factor to ensure that the maximum value is 1.

[0037] S6 establishes a feature matching framework:

[0038] The multimodal image conversion model trained in S4 is used to convert the test dataset in S1 and output image pairs with similar radiation information. The FAST operator and adaptive non-maximum suppression strategy are then combined to extract evenly distributed feature points. A circular neighborhood of 128×128 pixels is selected around each feature point to obtain a 256-dimensional feature vector using the rotation-invariant feature expression model trained in S5. Finally, the fast nearest neighbor matching strategy is used to identify the initial same-name points between the reference image and the image to be registered, and the DEGENSAC algorithm is used to remove outliers to determine the final matching points.

[0039] Furthermore, the scale difference between the acquired reference image and the image to be registered is within 0.8-1.2 times, and the rotation angle difference is within 10°.

[0040] Furthermore, S21 includes the following steps:

[0041] S211, use the shared encoder E to extract potential shared features between multimodal images:

[0042]

[0043] where f∈F is a potential shared feature.

[0044] S212, in order to ensure that the potential shared features are in the same space, their distribution q(f x )、q(f y ) matches the Kullback-Leibler (KL) divergence:

[0045] L kl =KL(q(f x )||p(f))+KL(q(f y )||p(f))

[0046] Where p(f) is the prior distribution, which is assumed to obey the isotropic Gaussian ￥(0,Ι).

[0047] Furthermore, S22 includes the following steps:

[0048] S221, using the commonly used image reconstruction L1 loss to train the decoder G to reconstruct the input image x (Reconstruction item):

[0049]

[0050] Among them, G(f x ,0) is the starting point of the path.

[0051] S222, in order to retain more details, introduces the structural similarity (SSIM) loss that focuses more on the structural information between images:

[0052]

[0053] Furthermore, S3 includes the following steps:

[0054] S31, by minimizing the energy functional to obtain a smoothly varying path, the Jacobian vector is approximated using the classical central finite difference method:

[0055]

[0056] where h is a hyperparameter that controls the granularity of the estimated Jacobian matrix.

[0057] S32, select the L layers of interest, you can get the feature sets of the two domains, and calculate the Jacobian matrix of the lth layer:

[0058]

[0059] where l is the l-th layer of decoder G.

[0060] S33, shortest path loss:

[0061] The shortest path constraint is calculated using the following formula

[0062]

[0063] This constraint encourages features of two close domains to be close by penalizing the Jacobian norm.

[0064] Furthermore, the adaptively adjusted EfficientNet-B1 network architecture in S51 takes an image of size 128×128×1 pixels as input and outputs a feature vector (1×1×256 dimensions) describing the deep consistency features of the image. The feature extraction architecture based on EfficientNet-B1 consists of: a first regular convolution with a batch normalization (BN) layer and a SiLU activation function layer, a middle stack of seven MobileInvertedBottleneckConv (MBConv) convolutional structures, and a final regular convolution with L2 normalization.

[0065] Furthermore, the training system of sample augmentation in S52 includes two computational homologous image blocks (y r and y' r 、y f and y' f ) rotation consistency twin network and 2 calculated reference image blocks and the transformed image blocks to be registered (y r and y f ,y' r and y' f ) Pseudo-twin network with weak radiation consistency; the total loss function of the training system can be expressed as:

[0066] L t =λ1L[F(y r ),F(y' r )]+λ1L[F(y f ),F(y' f )]+λ2L[F(y r ),F(y f )]+λ2L[F(y' r),F(y' f )]

[0067] where λ1 and λ2 are hyperparameters for balancing the rotational consistency loss and the weak radiation consistency loss, respectively.

[0068] Furthermore, the hyperparameters for balancing different loss functions in S4 are λ kl =0.01,λ rec =5,λ sp =0.1,λ idt =5.

[0069] Furthermore, the hyperparameters λ1 and λ2 in S52 are 0.4 and 0.1, respectively.

[0070] After adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art.

[0071] (1) The method of the present invention establishes the shortest path constraint by constructing an intermediate domain and utilizing potential shared features, which effectively improves the semantic consistency between translated images and eliminates the significant nonlinear radiation differences and noise between multimodal image pairs.

[0072] (2) The present invention adopts a network architecture with dilated convolution and attention mechanism for feature extraction, which significantly improves the quality of feature representation.

[0073] (3) In the present invention, a sample augmentation training system is designed by combining contrastive learning with twin network and pseudo-twin network to obtain a better rotation consistency feature expression, which effectively improves the robustness of multimodal image feature matching.

[0074] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The accompanying drawings are part of this application and are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but do not constitute an undue limitation of the present invention. Obviously, the drawings described below are only some embodiments. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without inventive effort. In the accompanying drawings:

[0076] Figure 1 It is a schematic flow chart of the method of the present invention;

[0077] Figure 2 is a multimodal image of the matching results of different methods in the embodiment.

[0078] It should be noted that these drawings and textual descriptions are not intended to limit the conceptual scope of the present invention in any way, but rather to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The following embodiments are used to illustrate the present invention but are not used to limit the scope of the present invention.

[0080] like Figure 1 As shown, the multimodal remote sensing image matching method based on modality conversion and contrast learning of the present invention includes the following steps:

[0081] S1, build training data set:

[0082] A reference image and an image to be registered containing the same area are obtained and paired to form multiple multimodal image pairs; all multimodal image pairs are combined into a multimodal image pair dataset, and the dataset is divided into a training dataset and a test dataset.

[0083] S2, constructs the generation path of the image continuous domain, specifically:

[0084] S21 uses a shared network encoder to extract the latent shared features of the training dataset in S1 and embeds them into a shared latent space, including the following steps:

[0085] S211 uses the shared network encoder E to extract the potential shared features of the training dataset in S1 and embed them into the shared latent space:

[0086]

[0087] where f∈F is a potential shared feature.

[0088] S212 To ensure that the potential shared features are in the same space, their distribution q(f x )、q(f y ) matches the Kullback-Leibler (KL) divergence:

[0089] L kl =KL(q(f x )||p(f))+KL(q(f y )||p(f))

[0090] Where p(f) is the prior distribution, which is assumed to obey the isotropic Gaussian ￥(0,Ι).

[0091] S22 designs an image generation path from one domain to another, trains the decoder G to reconstruct the input image x (Reconstruction term) and find the end point of the image generation path, specifically:

[0092] S221, using the commonly used image reconstruction L1 loss to train the decoder G to reconstruct the input image x (Reconstruction item):

[0093]

[0094] Among them, G(f x ,0) is the starting point of the path.

[0095] S222, in order to retain more details, introduces the structural similarity (SSIM) loss that focuses more on the structural information between images:

[0096]

[0097] S23 uses the generative adversarial network GAN to construct the loss function and train the discriminator D to distinguish the generated image G(f x ,1)with the real image y∈Y.

[0098] S3, design the shortest path constraint, which specifically includes the following steps:

[0099] S31, to find the minimum path length, the energy functional of the curve is optimized: by minimizing the energy functional to obtain a smoothly changing path, the classic central finite difference method is used to approximate the Jacobian vector:

[0100]

[0101] where h is a hyperparameter that controls the granularity of the estimated Jacobian matrix.

[0102] S32, penalize the path length of multi-layer features: select L layers of interest, you can get the feature sets of the two domains, and calculate the Jacobian matrix of the lth layer:

[0103]

[0104] where l is the l-th layer of decoder G.

[0105] S33, given the features of the L layer, use the following formula to calculate the final shortest path length regularization loss

[0106]

[0107] This constraint encourages features of two close domains to be close by penalizing the Jacobian norm.

[0108] S4, overall image conversion loss: The overall loss also introduces reconstruction loss (identity term) on the Y domain to further regularize the network; in order to retain more details, the structural similarity SSIM loss is introduced into the reconstruction loss and the reconstruction loss;

[0109] Using the above different loss functions to form the overall loss function, we have the following formula:

[0110]

[0111] Where λ kl ,λ rec ,λ sp ,λ idt They are the hyperparameters for balancing different loss functions.

[0112] The ultimate goal is to achieve the following:

[0113]

[0114] In the above embodiment, the preferred hyperparameter is λ kl =0.01,λ rec =5,λ sp =0.1,λ idt =5.

[0115] The image translation network is trained until the loss function LOSS converges to obtain a trained multimodal image conversion model.

[0116] The rotation invariant feature expression of S5 contrastive learning is as follows:

[0117] S51, Feature Extraction Architecture: We selected the EfficientNet-B1 network with dilated convolution and attention mechanism as the basic backbone network, and adaptively adjusted its input and output to adapt to the task of local deep feature extraction;

[0118] In the above embodiment, the adaptively adjusted EfficientNet-B1 network architecture inputs an image size of 128×128×1 pixels and outputs a feature vector (1×1×256 dimensions) used to describe the deep consistency features of the image. The feature extraction architecture based on EfficientNet-B1 includes: a first regular convolution with a batch normalization (BN) layer and a SiLU activation function layer, a middle 7-stacked MobileInvertedBottleneckConv (MBConv) convolution structure, and a final regular convolution with L2 normalization.

[0119] S52 sample augmentation training system:

[0120] The training dataset in S1 is transformed using the multimodal image conversion model trained in S4, and image pairs with similar radiometric information are output. Then, affine transformations and circular masks with different rotation angles (0 to 360 degrees) are added to the transformed images, and the corresponding pixel relationships between these transformed images and the reference image are obtained. Finally, the Triplet Loss is used to improve the difference between matching image patches and non-matching image patches by combining the twin and pseudo-twin networks to minimize the distance between matching descriptors and maximize the distance between non-matching descriptors. The loss is implemented as follows:

[0121]

[0122] In the above formula, α is the regularization parameter, θ i + and θ i - represent positive samples and negative samples respectively, and x i is a pair of positive sample descriptors before normalization. S H The calculation formula for (θ) is as follows:

[0123]

[0124] where a is a hyperparameter that balances the distance metric and the correlation metric, and z is a normalization factor to ensure that the maximum value is 1.

[0125] The training system of sample augmentation includes two computational homologous image patches (y r and y′ r 、y f and y′ f ) rotation consistency twin network and 2 calculated reference image blocks and the transformed image blocks to be registered (y r and y f , y′ r and y′ f ) Pseudo-twin network with weak radiation consistency; the total loss function of the training system can be expressed as:

[0126] L t =λ1L[F(y r ),F(y' r )]+λ1L[F(y f ),F(y' f )]+λ2L[F(y r ),F(y f )]+λ2L[F(y' r),F(y' f )]

[0127] Where λ1 and λ2 are hyperparameters that balance the rotational consistency loss and the weak radiation consistency loss, respectively;

[0128] In the above embodiment, the hyperparameters λ1 and λ2 are preferably 0.4 and 0.1 respectively.

[0129] The image feature expression network is trained until the loss function LOSS converges to obtain a trained rotation consistency feature expression model.

[0130] S6, establish feature matching framework:

[0131] The multimodal image conversion model trained in S4 is used to convert the test dataset in S1, and an image pair with similar radiation information is output. Then, the FAST operator and the adaptive non-maximum suppression strategy are combined to extract uniformly distributed feature points, and a circular neighborhood of 128×128 pixels is selected around each feature point to obtain a 256-dimensional feature vector using the rotation-invariant feature expression model trained in S5. Finally, the fast nearest neighbor matching strategy is used to identify the initial same-name points between the reference image and the image to be registered, and the DEGENSAC algorithm is used to remove outliers to determine the final matching points and output the matching results.

[0132] Example

[0133] To validate the performance of the proposed multimodal image matching method, we selected multimodal images from a test dataset encompassing various landform types, including cities, suburbs, farmland, and ports, for quantitative and qualitative evaluation. The quantitative evaluation compared the matching results of the proposed method with those of six existing matching methods to verify its effectiveness.

[0134] In Table 1, the matching performance of the proposed method is evaluated using two indicators: the number of correct matches (NCM) and the registration accuracy (RMSE) by comparing it with three methods based on manual features (RIFT2, HOWP, and WSSF) and three methods based on deep learning (Superglue, CMMNet, and LightGlue). The results show that the proposed method exhibits significant advantages in both NCM and RMSE values. Specifically, the proposed method achieves the highest NCM value in all test scenarios, with an average NCM value of 868 and an average RMSE of only 1.43, fully demonstrating the superiority and applicability of the proposed method in handling multimodal remote sensing image matching tasks with nonlinear geometry and radiometric differences.

[0135]

[0136] Qualitative evaluation further verifies the matching performance of the present invention by displaying correctly matched homonymous points and performing visual inspection. Figure 2 Visual matching results for multimodal image pairs from various scenarios are presented. The proposed method successfully identified a large number of correct homonymous points in all tested multimodal image pairs, and these homonymous points were evenly distributed across the images. In particular, the proposed method was able to identify a large number of evenly distributed homonymous points in pairs of weakly textured farmland scenes, fully demonstrating its superior scalability and robustness to multimodal differences.

[0137] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as above in terms of a preferred embodiment, it is not intended to limit the present invention. Any technician familiar with the present invention can make slight changes or modifications to equivalent embodiments using the above-mentioned technical contents without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the solution of the present invention.

Claims

1. A multimodal remote sensing image matching method based on modality conversion and contrastive learning, characterized in that: The following steps are involved: S1, build training data set: Obtain a reference image and an image to be registered that contain the same area, and pair them to form multiple multimodal image pairs; combine all the multimodal image pairs into a multimodal image pair dataset, and divide the dataset into a training dataset and a test dataset; S2, construct the generation path of the continuous domain of the image: S21, uses the shared network encoder to extract the potential shared features of the training dataset in S1 and embeds them into the shared latent space; S22, design an image generation path from one domain to another, train the decoder G to reconstruct the input image x and find the end point of the image generation path; S23, using the generative adversarial network GAN to construct the loss function, training the discriminator D to distinguish the generated image G(f x ,1) with the real image y∈Y; S3, design the shortest path constraint: S31, to find the minimum path length, the energy functional of the curve is optimized; S32, penalizes the path length of multi-layer features: S33, given the features of layer L, obtain the final shortest path length regularization loss; S4, overall loss of image conversion: The overall loss also introduces reconstruction loss in the Y domain to further regularize the network; in order to retain more details, the structural similarity SSIM loss is introduced into the reconstruction loss and the reconstruction loss; Using the above different loss functions to form the overall loss function, we have the following formula: Where λ kl ,λ rec ,λ sp ,λ idt They are the hyperparameters for balancing different loss functions; The ultimate goal is to achieve the following: Train the image translation network until the loss function LOSS converges to obtain a trained multimodal image translation model; S5, rotation-invariant feature representation for contrastive learning: S51, feature extraction architecture: The EfficientNet-B1 network with dilated convolution and attention mechanism was selected as the basic backbone network, and its input and output were adaptively adjusted to adapt to the task of local deep feature extraction; S52, sample augmentation training system: The training dataset in S1 is transformed using the multimodal image conversion model trained in S4, and image pairs with similar radiometric information are output. Then, affine transformations and circular masks with different rotation angles are added to the transformed images, and the corresponding pixel relationships between these transformed images and the reference image are obtained. Finally, the Triplet Loss is used to improve the difference between matching and non-matching image patches by combining the Siamese and Pseudo-Siamese networks and minimizing the distance between matching descriptors and maximizing the distance between non-matching descriptors. The loss is implemented as follows: In the above formula, α is the regularization parameter, and represent positive samples and negative samples respectively, and x i is a pair of positive sample descriptors before normalization; S H The calculation formula for (θ) is as follows: Where a is a hyperparameter that balances the distance metric and the correlation metric, and z is a normalization factor to ensure that the maximum value is 1; S6 establishes a feature matching framework: The multimodal image conversion model trained in S4 is used to convert the test dataset in S1 and output image pairs with similar radiation information. The FAST operator and adaptive non-maximum suppression strategy are then combined to extract evenly distributed feature points. A circular neighborhood of 128×128 pixels is selected around each feature point to obtain a 256-dimensional feature vector using the rotation-invariant feature expression model trained in S5. Finally, the fast nearest neighbor matching strategy is used to identify the initial same-name points between the reference image and the image to be registered, and the DEGENSAC algorithm is used to remove outliers to determine the final matching points.

2. The multimodal remote sensing image matching method based on modality conversion and contrastive learning according to claim 1, characterized in that: The scale difference between the acquired reference image and the image to be registered is within 0.8-1.2 times, and the rotation angle difference is within 10°.

3. The multimodal remote sensing image matching method based on modality conversion and contrastive learning according to claim 1, characterized in that: S21 specifically includes the following steps: S211, use the shared encoder E to extract potential shared features between multimodal images: Among them, f∈F is the potential shared feature; S212, in order to ensure that the potential shared features are in the same space, their distribution q(f x )、q(f y ) matches the Kullback-Leibler (KL) divergence: L kl =KL(q(f x )||p(f))+KL(q(f y )||p(f)) Where p(f) is the prior distribution, which is assumed to obey the isotropic Gaussian ￥(0,Ι).

4. The multimodal remote sensing image matching method based on modality conversion and contrastive learning according to claim 3, characterized in that: S22 specifically includes the following steps: S221, using the commonly used image reconstruction L1 loss to train the decoder G to reconstruct the input image x (Reconstruction item): Among them, G(f x ,0) is the starting point of the path; S222, in order to retain more details, introduces the structural similarity (SSIM) loss that focuses more on the structural information between images:

5. The multimodal remote sensing image matching method based on modality conversion and contrastive learning according to claim 4, characterized in that: S3 specific The following steps are involved: S31, by minimizing the energy functional to obtain a smoothly varying path, the Jacobian vector is approximated using the classical central finite difference method: Where h is a hyperparameter that controls the granularity of the estimated Jacobian matrix; S32, select the L layers of interest, you can get the feature sets of the two domains, and calculate the Jacobian matrix of the lth layer: Where l is the lth layer of decoder G; S33, shortest path loss: The shortest path constraint is calculated using the following formula This constraint encourages features of two close domains to be close by penalizing the Jacobian norm.

6. The multimodal remote sensing image matching method based on modality conversion and contrastive learning according to claim 1, characterized in that: The adaptively adjusted EfficientNet-B1 network architecture in S51 takes as input an image of size 128×128×1 pixels and outputs a feature vector (1×1×256 dimensions) describing the deep consistency features of the image. The feature extraction architecture based on EfficientNet-B1 consists of the first regular convolution with a batch normalization layer and a SiLU activation function layer, the middle 7 stacked MobileInvertedBottleneckConv convolution structures, and the last regular convolution with L2 normalization.

7. The multimodal remote sensing image matching method based on modality conversion and contrastive learning according to claim 1, characterized in that: The training system of sample augmentation in S52 includes: 2 calculations of homologous image blocks (y r and y' r 、y f and y' f ) rotation consistency twin network, 2 calculated reference image patches and the transformed image patch to be registered (y r and y f 、y' r and y' f ) Pseudo-twin networks with weak radiative consistency; The total loss function of the training system can be expressed as: L t =λ1L[F(y r ),F(y' r )]+λ1L[F(y f ),F(y' f )]+λ2L[F(y r ),F(y f )]+λ2L[F(y' r ),F(y' f )] where λ1 and λ2 are hyperparameters for balancing the rotational consistency loss and the weak radiation consistency loss, respectively.

8. The multimodal remote sensing image matching method based on modality conversion and contrastive learning according to claim 1, characterized in that: The hyperparameters for balancing different loss functions in S4 are λ kl =0.01,λ rec =5,λ sp =0.1,λ idt =5.

9. The multimodal remote sensing image matching method based on modality conversion and contrastive learning according to claim 7, characterized in that: The hyperparameters λ1 and λ2 in S52 are 0.4 and 0.1 respectively.

Images