A Multimodal Remote Sensing Image Matching Method Based on Modality Reconstruction and Feature Perturbation Learning

Abstract

This invention belongs to the field of remote sensing image matching technology, specifically involving a multimodal remote sensing image matching method based on modal reconstruction and feature perturbation learning. First, this invention constructs a cross-modal local region feature representation model, using matching and non-matching regions to construct positive and negative samples respectively, optimizing the correlation of positive sample descriptors and the difference of negative sample descriptors. Then, based on the original radiometric difference supervision signal, it introduces the difference supervision signal before and after sample perturbation, constructing an enhanced supervision mechanism for feature perturbation learning. Subsequently, through modal reconstruction, the local features of one modality image are reconstructed into a pseudo-image of another modality, providing additional information for descriptor learning through reconstruction supervision. Finally, the model is trained through multi-objective supervision, enabling the model to extract the radiometric and geometrically invariant local common features of feature points. This invention can effectively extract the common features of radiometric resistance differences and geometric distortions, providing strong robust technical support for multi-source remote sensing collaborative tasks.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote sensing image matching, and particularly relates to a multi-modal remote sensing image matching method based on modal reconstruction and feature disturbance learning. BACKGROUND

[0002] Multi-modal image matching aims to identify the same name matching points from two or more images collected by different sensors, at different times or from different perspectives, to realize image accurate alignment and registration, and provide unified spatial coordinate basis data for multi-modal collaborative tasks such as image fusion, change detection and visual navigation.

[0003] There are currently two main technologies for multi-modal image matching: (1) Template matching, which defines a template window in the reference image, and then searches for the extreme value of similarity measure in the search area of the image to be matched to identify the same name points. This technology usually has high computational efficiency and matching accuracy, but is very sensitive to image geometric transformation and highly dependent on prior geographical information to correct global geometric distortion, and is suitable for image matching scenarios with almost translational transformation relationship. (2) Feature matching, which usually follows the paradigm of "detection-description-matching": first, extract significant feature points such as corner points, edge points, and line segment intersection points in the image, then perform feature description in the neighborhood of the feature points, and finally identify the same name points through the similarity measure of the descriptors. Compared with the template matching method, the feature matching method has better geometric distortion resistance.

[0004] However, due to the different imaging principles of multi-source sensors, there are significant geometric distortion and nonlinear radiation differences between multi-modal images, making it difficult to express common features, and existing learning-based feature matching methods usually force the representation of different modal images to be similar through feature space alignment, without fully exploiting the transformation mapping relationship between modalities, lacking decoupling learning of unique features and common features of multi-modal; In addition, during remote sensing imaging, image quality degradation may occur due to changes in external environment, for example, optical images may be obscured by clouds and fog due to weather, SAR images may have noise due to interference during active imaging, resulting in large fluctuations in common features under complex interference, and in the descriptor learning process, although the matching region information of paired multi-modal images can be used, only limited samples are used for supervision, lacking comprehensive consideration of complex interference factors, resulting in feature interference phenomenon, that is, the robustness of common features to complex interference is insufficient, and the generalization ability of the method is limited. SUMMARY

[0005] To overcome the shortcomings of existing technologies, this invention proposes a matching method based on modal reconstruction and feature perturbation learning. By constructing a cross-modal local region feature representation model, perturbation samples are introduced to enhance supervision on the basis of radiation difference supervision. At the same time, a modal reconstruction decoding module is designed to use the correlation of reconstructed images as an additional supervision signal. Through multi-target supervised training, the common feature representation of local regions that are both radiation- and geometrically invariant is realized.

[0006] The technical solution adopted in this invention is as follows:

[0007] A multimodal remote sensing image matching method based on modal reconstruction and feature perturbation learning takes visible light-SAR multimodal images as the processing object, defining visible light images as I... OPT SAR image is I SAR This includes the following steps:

[0008] S1. Construct a multi-source remote sensing image registration dataset, specifically by collecting images containing I... OPT with I SAR Multimodal remote sensing image pairs, and then for I OPT with I SAR Preprocessing is performed to obtain the training set and validation set;

[0009] S2. Construct a multimodal remote sensing image matching network for image matching. The network consists of several parts: feature point detection, feature point description, and descriptor matching.

[0010] The feature point detection is for I OPT and I SAR Feature extraction is performed to obtain a set of feature points, and a corresponding set of local regions is constructed. Specifically, I is extracted using a pre-trained SuperPoint detector. OPT and I SAR The feature point set is defined as P. OPT ={p1,p2,...,p m} and P SAR ={q1,q2,...,q n}, where m is I OPT The number of feature points, n is I SAR The number of feature points extracted from P; OPT and P SAR Then, a local region set is obtained by sampling the local regions of feature points, specifically using P... OPT Each feature point p i Centered on a local area of ​​a sampling square with a sampling side length D, an I is constructed. OPT The set of local regions X OPT ={x1,x2,...,x m}, xi For p i The corresponding local region; similarly, with P SAR Each feature point q j Centered on a local region using the same side length D, an I is constructed. SAR The set of local regions X SAR ={y1,y2,...,y n}, y j For q j The corresponding local area;

[0011] The feature point description is based on the constructed X OPT and X SAR To obtain the feature descriptor set and perform modality reconstruction, feature descriptors are extracted by constructing a cross-modal local region feature representation model. This model employs a pseudo-Twin architecture with two vertical branches that do not share weights, and a symmetrical encoder-decoder structure horizontally. One branch in the architecture is defined as the left branch, used to process X. OPT The other branch is the right branch, used to process X. SAR Cross-modal local region feature representation model for X OPT and X SAR The specific processing procedure is: to process X OPT Input the left branch encoder, extract feature descriptors, and construct I OPT The descriptor subset F OPT ={f1, f2, ..., f m}; X SAR Input the right branch encoder, extract feature descriptors, and construct I. SAR The descriptor subset F SAR ={g1 ,g2,...,g n}, and ensure P OPT With F OPT P SAR With F SAR One-to-one correspondence; then, through the left branch decoder, the descriptor subset is reconstructed into a pseudo-SAR image y. SAR Similarly, the descriptor is constructed into a pseudo-visible light image y using the right-branch decoder. OPT ;

[0012] The descriptor matching is based on the obtained descriptor set for image registration, specifically by calculating the descriptor set F. OPT and F SAR Similarity of descriptors Then, the correct matching pairs are filtered through the Lowe ratio test: for each Take its two nearest neighbor descriptors and The subscripts j and k are descriptor numbers, if / Preserve matching pairs ( and (These are the feature points corresponding to indices i and j, respectively); otherwise, they are not considered correct matches and are discarded. All correct matches are then used to estimate I. OPT and I SAR The homography matrix H between the reference and the image is used to register the image to be matched with the reference image.

[0013] S3. The constructed multimodal remote sensing image matching network is trained using the training set. During training, an enhanced supervision mechanism based on feature perturbation learning is employed, specifically for X... OPT With X SAR Applying composite perturbations generates perturbation data. and The perturbation loss is obtained by calculating the feature loss of the original data and the perturbation data within the same mode. disturb−OPT With Loss disturb−SAR Thus, the encoder loss function is constructed:

[0014] Loss encoder =Loss radiation +α×(Loss disturb−OPT +Loss disturb−SAR ),

[0015] Among them, Loss radiation It is radiation loss, disturbance loss. disturb−OPT With Loss disturb−SAR Calculation method and radiation loss Loss radiation The calculation method is the same:

[0016] ,

[0017] in, It is the marginal loss of the triple. It is an L2 regularization term:

[0018] ,

[0019] ,

[0020] in, , In training batch N, from F OPT With F SAR The positive samples obtained from the N pairs of matching descriptors selected in the middle At the same time, all mismatched descriptors are treated as negative samples. The correlation of descriptors is calculated using a hybrid similarity metric. ,fixed ,exist The minimum value corresponds to Obtain hard negative samples , , For positive sample descriptors before L2 normalization, M is the adaptively adjusted boundary value:

[0021] ;

[0022] The loss function for constructing the decoder is:

[0023] ,

[0024] in, For hyperparameters, It is the pixel-level L1 loss between the original image patch and the pseudo image patch. It is feature-level L2 loss:

[0025] ,

[0026] ,

[0027] Among them, X OPT (x,y),X SAR (x, y) represents the pixel values ​​of the original image patch, where y OPT (x,y), y SAR (x,y) represents the pixel values ​​of the pseudo-image patch; Φ OPT (y OPT ) and Φ SAR (y SAR ) is to use y OPT Input left branch encoder Φ OPT y SAR Input right branch encoder Φ SAR Extracted reconstructed features;

[0028] Therefore, the total loss function used during training is:

[0029] ,

[0030] in, For hyperparameters;

[0031] After training the network to convergence or reaching the set maximum number of iterations on the training set, the convergence of the training is monitored and adjusted using the validation set, and finally a well-trained multimodal remote sensing image matching network is obtained.

[0032] S4. Use the trained multimodal remote sensing image matching network to match the target image.

[0033] Furthermore, in step S1, the weights of the pre-trained SuperPoint detector are pre-trained based on the COCO dataset, and the input image needs to undergo channel adaptation: I OPT It is an RGB three-channel, I SAR Convert to three channels by channel copying; after feature point extraction, isolated points with response values ​​below the threshold need to be filtered to ensure that the feature point density meets the requirement of no less than 5 per 100×100 pixel area.

[0034] Furthermore, in step S2, the adaptive adjustment rule for the sampling side length D is: when I OPT with I SAR When the resolution is greater than 1024×1024 pixels, D=48 pixels; when the resolution is between 512×512 and 1024×1024 pixels, D=32 pixels; when the resolution is less than 512×512 pixels, D=24 pixels; when the sampling area exceeds the image boundary, reflection filling is used to supplement pixels.

[0035] Furthermore, in step S2, the encoder of the cross-modal local region feature representation model includes 5 convolutional layers, each using a 3×3 convolutional kernel, a stride of 2, and equal-length padding, sequentially performing batch normalization, instance normalization, and ReLU activation. The corresponding decoder includes 5 deconvolutional layers, symmetrical to the encoder. The output features of each deconvolutional layer are element-wise added to the corresponding layer features of the encoder. The last layer uses the Tanh activation function to output a pseudo-image. The size and number of channels of the output feature maps of each layer of the encoder are as follows: Layer 1 is 64×64, 16 channels; Layer 2 is 32×32, 32 channels; Layer 3 is 16×16, 64 channels; Layer 4 is 8×8, 128 channels; and Layer 5 is 4×4, 256 channels. The size and number of channels of the output feature maps of each layer of the decoder are as follows: Layer 1 is 8×8, 128 channels; Layer 2 is 16×16, 64 channels; and so on. The first layer has 32×32 channels, the third layer has 32 channels, the fourth layer has 64×64 channels, the fifth layer has 16 channels, and the fifth layer has 128×128 channels.

[0036] Furthermore, in step S2, the similarity of descriptors is calculated by constructing a KD-tree index structure using the FLANN algorithm, with L2 distance as the calculation method, and the construction time of the KD-tree index does not exceed 30% of the total matching time.

[0037] Furthermore, in step S2, the threshold for the Lowe ratio test is adaptively adjusted according to the image noise level: when the image noise variance > 0.03, the threshold is set to 0.75; when the noise variance ≤ 0.03, the threshold is set to 0.8; after adjustment, it is necessary to ensure that the correct matching rate is ≥ 90%.

[0038] Furthermore, the formula for calculating descriptor relevance is:

[0039] ,

[0040] in, For vector dot product operation, Let L2 be the norm of the vector;

[0041] , The normalization formula is:

[0042] ,

[0043] Furthermore, the statistical properties of the normalized descriptors must meet the following requirements: mean within the range of 0 ± 0.1 and standard deviation within the range of 0.3 ± 0.05.

[0044] Furthermore, the feature-level L2 loss is modified by performing L2 normalization on the features before calculating the L2 loss between the reconstructed features and the original features:

[0045] ,

[0046] in, , X represents the L2 normalized output features of the left and right encoders, respectively. SAR,i y SAR,i X SAR y SAR The i-th local region in X SAR,j y SAR,j X SAR y SAR The j-th local region in.

[0047] The beneficial effect of this invention is that it proposes a robust multimodal remote sensing image matching framework, which solves the problem of difficulty in expressing common features of multimodal images.

[0048] (1) The method of this framework can effectively extract the common features of radiation resistance differences and geometric distortion, and solve the problem of difficulty in expressing the common features of multimodal images.

[0049] (2) The proposed framework method outperforms other advanced methods in terms of reprojection error, success rate and area under curve.

[0050] (3) This framework is applicable to scenarios with different rotation angles from 0 to 360° and can provide robust technical support for multi-source remote sensing collaborative tasks. Attached Figure Description

[0051] Figure 1 The diagram shows a matching framework based on modality reconstruction and feature perturbation learning.

[0052] Figure 2 The diagram shows the network structure for local region common feature extraction and modality reconstruction.

[0053] Figure 3 The diagram shown illustrates the feature encoder training process.

[0054] Figure 4 The diagram shows the feature decoder training process. Detailed Implementation

[0055] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0056] Example:

[0057] The overall framework of this example is as follows: Figure 1 As shown, this embodiment uses a visible light image (denoted as I) OPT ) and SAR images (denoted as I SAR The specific execution process in this example is as follows: (The object being processed is )

[0058] S1. Construct a multi-source remote sensing image registration dataset:

[0059] S1-1. Collect multimodal remote sensing image pairs that cover scenes with 0-360° rotation angle differences, 5%-20% radiation intensity differences, and Gaussian noise interference, and each image pair contains no less than 100 matching point pairs with the same name.

[0060] S1-2, for I OPT with I SAR Preprocessing: crop to a fixed size of 512×512 pixels; normalize pixel values ​​to the range of [-1,1]; remove invalid samples that are blurry or have an occlusion area of ​​more than 30%.

[0061] S1-3. Divide the training set, validation set, and test set into a 7:2:1 ratio. The training set is used for learning model parameters, the validation set is used to monitor training convergence, and the test set is used to evaluate the final matching performance.

[0062] S2, Feature Point Detection (based on a pre-trained SuperPoint detector):

[0063] S2-1, The weights of the pre-trained SuperPoint detector are obtained based on the COCO dataset, and the input image needs to be channel-adapted: I OPT For RGB three-channel direct input, I SAR It was initially a single grayscale channel, which was converted to a three-channel format via channel duplication. The feature point response threshold was set to 0.01, and the non-maximum suppression radius was set to 3 pixels.

[0064] S2-2, Extract the feature point set P of IOPT OPT ={p1,p2,...,p m}, extract I SAR Feature point set P SAR ={q1,q2,...,q n}; Filter out isolated points with response values ​​below the threshold to ensure that the feature point density meets the requirement of "no less than 5 per 100×100 pixel region".

[0065] S3. Local region sampling of feature points:

[0066] S3-1, with P OPT Each feature point p i Centered on a sampled square with a side length D, a local area of ​​the sampled square is constructed using the sampled square. OPT The set of local regions X OPT ={x1,x2,...,x m}

[0067] S3-2, with P SAR Each feature point q j Centered on a local region using the same side length D, an I is constructed. SAR The set of local regions X SAR ={y1,y2,...,y n}

[0068] S3-3. The sampling side length D follows an adaptive adjustment rule: when the image resolution is greater than 1024×1024 pixels, D=48 pixels; when the resolution is between 512×512 and 1024×1024 pixels, D=32 pixels; when the resolution is less than 512×512 pixels, D=24 pixels. When the sampling area exceeds the image boundary, reflection filling is used to supplement pixels.

[0069] S4. Construct a cross-modal local region feature representation model and extract feature descriptors:

[0070] S4-1, such as Figure 2As shown, the model adopts a pseudo-twin architecture with two vertical branches that do not share weights. The left branch processes X. OPT Right branch processing X SAR ; Horizontally, it has a symmetrical "encoding-decoding" structure.

[0071] S4-2, The encoder section consists of 5 convolutional layers, each using a 3×3 convolutional kernel, a stride of 2, and equal-length padding. Batch normalization, instance normalization, and ReLU activation are performed sequentially. The size and number of channels of the output feature maps of each layer are as follows: Layer 1 (64×64, 16 channels), Layer 2 (32×32, 32 channels), Layer 3 (16×16, 64 channels), Layer 4 (8×8, 128 channels), and Layer 5 (4×4, 256 channels), ultimately increasing the number of channels in local regions from 3 to 256.

[0072] S4-3. The decoder consists of 5 deconvolutional layers, symmetrical to the encoder. The size and number of channels of the output feature maps of each layer are as follows: Layer 1 (8×8, 128 channels), Layer 2 (16×16, 64 channels), Layer 3 (32×32, 32 channels), Layer 4 (64×64, 16 channels), and Layer 5 (128×128, 3 channels). The output features of each deconvolutional layer are added element-wise to the corresponding layer features of the encoder (skip connections). The last layer uses the Tanh activation function to output a pseudo-image.

[0073] S4-4, X OPT Input the left branch encoder to extract 256-dimensional feature descriptors, which constitute I. OPT The descriptor subset F OPT ={f1,f2,...,f m}; X SAR Input the right branch encoder to extract 256-dimensional feature descriptors, forming I SAR The descriptor subset F SAR ={g1,g2,...,g n The feature descriptors need to be L2 normalized. After normalization, the mean of the descriptors should be within the range of 0 ± 0.1 and the standard deviation should be within the range of 0.3 ± 0.05.

[0074] S5. Construct an enhanced supervision mechanism for feature perturbation learning:

[0075] S5-1, such as Figure 3 As shown, training batches N=4 are selected, starting from F OPT With F SAR N pairs of matching descriptors are selected as positive samples, and all non-matching descriptors are selected as negative samples.

[0076] S5-2. Calculate descriptor relevance using a hybrid similarity metric, with the following formula:

[0077]

[0078] Among them, f a g b Let be the feature descriptor whose correlation is to be calculated, ⋅ denotes the vector dot product operation, and ∥⋅∥ denotes the L2 norm of the vector.

[0079] S5-3. Calculate the marginal loss of the triplet (including the selection of hard negative samples): First, adaptively adjust the boundary value m, then select the hard negative sample g. neg (fixed f) i ,exist The minimum value of g is taken from the middle. j The marginal loss formula for triplet is calculated according to the corresponding rules.

[0080] S5-4. Introduce an L2 regularization term for the original feature space differences, the formula is:

[0081]

[0082] in, , These are the positive sample descriptors before L2 normalization.

[0083] S5-5, By combining the triplet marginal loss and the L2 regularization term, the radiation loss Loss is obtained. radiation .

[0084] S5-6, Regarding X OPT With X SAR Apply composite perturbations: geometric distortion (random rotation of -10° to 10°, random translation of ±5 pixels), radiative perturbation (intensity scaling of 0.8 to 1.2 times), and Gaussian noise (variance of 0.01 to 0.05) to generate perturbation data. and .

[0085] S5-7. Calculate the feature loss of the original data and the perturbation data within the same mode (the calculation process is the same as Loss). radiation ), and obtain the disturbance loss Loss disturb−OPT With Loss disturb−SAR .

[0086] S5-8. Constructing the encoder loss function:

[0087] Loss encoder =Loss radiation +α×(Loss disturb−OPT +Loss disturb−SAR )

[0088] Where α=0.3 is the hyperparameter for balancing radiation loss and disturbance loss.

[0089] S6. Design the modal reconstruction decoding module:

[0090] S6-1 as follows Figure 4 As shown, the I output from the left encoder... OPT The features are input into the decoder and reconstructed into a pseudo-SAR image y. SAR ; the I output of the right branch encoder SAR The features are input into the decoder and reconstructed into a pseudo-visible light image y. OPT .

[0091] S6-2. Calculate the pixel-level L1 loss for the original image patch and the pseudo image patch:

[0092]

[0093] Among them, X OPT (x,y),X SAR (x, y) represents the pixel values ​​of the original image patch, where y OPT (x,y), y SAR (x,y) represents the pixel values ​​of the pseudo-image patch.

[0094] S6-3, y OPT Input left branch encoder Φ OPT y SAR Input right branch encoder Φ SAR Extract and reconstruct features Φ OPT (y OPT ), Φ SAR (y SAR Before calculating the feature-level L2 loss, the features need to be L2 normalized, and the corrected L2 loss formula is calculated according to the corresponding rules.

[0095] S6-4. Constructing the decoder loss function:

[0096]

[0097] Where β=0.8 is the hyperparameter that balances L1 loss and L2 loss.

[0098] S7. Multi-objective supervised training and model optimization:

[0099] S7-1. Construct the total loss function and balance the encoder and decoder training:

[0100] Loss total =Loss encoder +λ×Loss decoder

[0101] Where λ=0.2 is a hyperparameter.

[0102] S7-2 uses the AdamW optimizer with an initial learning rate of 0.001 and a cosine annealing strategy (the learning rate decays to 0.9 of the current value every 10 epochs). The training batch size is 4, and the total training period is 60 epochs.

[0103] S7-3. Adopt an early stopping strategy: when the loss of the validation set decreases by less than 1e-5 over 10 consecutive epochs, stop training and save the optimal model parameters.

[0104] S8. Feature descriptor matching and image registration:

[0105] S8-1. Construct a KD-tree index structure using the FLANN algorithm (number of trees checked = 5, number of iterations = 10), and calculate F using L2 distance. OPT With F SAR Sim(f) similarity of descriptors i ,g j The matching time for a single image pair is controlled within 0.5 seconds, and the construction time of the KD tree index accounts for no more than 30% of the total matching time.

[0106] S8-2. Filtering correct matching pairs using the Lowe ratio test: For each f i Take its two nearest neighbor descriptors g j (First nearest neighbor) and g k (2nd nearest neighbor), if / <threshold, retain matching pairs (p i ,q j The threshold is adaptively adjusted based on the image noise level: when the noise variance is >0.03, the threshold is set to 0.75; when the noise variance is ≤0.03, the threshold is set to 0.8; after adjustment, the correct matching rate must be ≥90%.

[0107] S8-3. Remove outliers using the RANSAC algorithm: Set the inlier threshold to 1.5 pixels and the number of iterations to 1000. Estimate I... OPT with I SAR The homography matrix H between the reference and the image is used to achieve accurate registration between the image to be matched and the reference image.

Claims

1. A multimodal remote sensing image matching method based on modal reconstruction and feature perturbation learning, using visible light-SAR multimodal images as the processing object, defining visible light images as I... OPT SAR image is I SAR Its characteristics are, Includes the following steps: S1. Construct a multi-source remote sensing image registration dataset, specifically by collecting images containing I... OPT with I SAR Multimodal remote sensing image pairs, and then for I OPT with I SAR Preprocessing is performed to obtain the training set and validation set; S2. Construct a multimodal remote sensing image matching network for image matching. The network consists of several parts: feature point detection, feature point description, and descriptor matching. The feature point detection is for I OPT and I SAR Feature extraction is performed to obtain a set of feature points, and a corresponding set of local regions is constructed. Specifically, I is extracted using a pre-trained SuperPoint detector. OPT and I SAR The feature point set is defined as P. OPT ={p1,p2,...,p m } and P SAR ={q1,q2,...,q n }, where m is I OPT The number of feature points, n is I SAR The number of feature points extracted from P; OPT and P SAR Then, a local region set is obtained by sampling the local regions of feature points, specifically using P... OPT Each feature point p i Centered on a local area of ​​a sampling square with a sampling side length D, an I is constructed. OPT The set of local regions X OPT ={x1,x2,...,x m }, x i For p i The corresponding local region; similarly, with P SAR Each feature point q j Centered on a local region using the same side length D, an I is constructed. SAR The set of local regions X SAR ={y1,y2,...,y n }, y j For q j The corresponding local area; The feature point description is based on the constructed X OPT and X SAR To obtain the feature descriptor set and perform modality reconstruction, feature descriptors are extracted by constructing a cross-modal local region feature representation model. This model employs a pseudo-Twin architecture with two vertical branches that do not share weights, and a symmetrical "encoder-decoder" structure horizontally. One branch in the architecture is defined as the left branch, used to process X. OPT The other branch is the right branch, used to process X. SAR Cross-modal local region feature representation model for X OPT and X SAR The specific processing procedure is: to process X OPT Input the left branch encoder, extract feature descriptors, and construct I OPT The descriptor subset F OPT ={f1, f2, ..., f m }; X SAR Input the right branch encoder, extract feature descriptors, and construct I. SAR The descriptor subset F SAR ={g1 ,g2,...,g n }, and ensure P OPT With F OPT P SAR With F SAR One-to-one correspondence; then, through the left branch decoder, the descriptor subset is reconstructed into a pseudo-SAR image y. SAR Similarly, the descriptor is constructed into a pseudo-visible light image y using the right-branch decoder. OPT ; The descriptor matching is based on the obtained descriptor set for image registration, specifically by calculating the descriptor set F. OPT and F SAR Similarity of descriptors Then, the correct matching pairs are filtered using the Lowe ratio test: for each Take its two nearest neighbor descriptors and The subscripts j and k are descriptor numbers, if / Preserve matching pairs pi and qj are the feature points corresponding to indices i and j, respectively; otherwise, they are discarded. After obtaining all correct matching pairs, they are used to estimate I. OPT and I SAR The homography matrix H between the reference and the image is used to register the image to be matched with the reference image. S3. The constructed multimodal remote sensing image matching network is trained using the training set. During training, an enhanced supervision mechanism based on feature perturbation learning is employed, specifically for X... OPT With X SAR Applying composite perturbations generates perturbation data. and The perturbation loss is obtained by calculating the feature loss of the original data and the perturbation data within the same mode. disturb−OPT With Loss disturb−SAR Thus, the encoder loss function is constructed: Loss encoder =Loss radiation +α×(Loss disturb−OPT +Loss disturb−SAR ), Among them, Loss radiation It is radiation loss, disturbance loss. disturb−OPT With Loss disturb−SAR Calculation method and radiation loss Loss radiation The calculation method is the same: ， in, It is the marginal loss of the triple. It is an L2 regularization term: ， ， in, , In training batch N, from F OPT With F SAR The positive samples obtained from the N pairs of matching descriptors selected in the middle At the same time, all mismatched descriptors are treated as negative samples. The correlation of descriptors is calculated using a hybrid similarity metric. ,fixed ,exist The minimum value corresponding to Obtain hard negative samples , , For positive sample descriptors before L2 normalization, M is the adaptively adjusted boundary value: ； The loss function for constructing the decoder is: ， in, For hyperparameters, It is the pixel-level L1 loss between the original image patch and the pseudo image patch. It is feature-level L2 loss: ， ， Among them, X OPT (x,y),X SAR (x, y) represents the pixel values ​​of the original image patch, where y OPT (x,y), y SAR (x,y) represents the pixel values ​​of the pseudo-image patch; Φ OPT (y OPT ) and Φ SAR (y SAR ) is to use y OPT Input left branch encoder Φ OPT y SAR Input right branch encoder Φ SAR Extracted reconstructed features; Therefore, the total loss function used during training is: ， in, For hyperparameters; After training the network to convergence or reaching the set maximum number of iterations on the training set, the convergence of the training is monitored and adjusted using the validation set, and finally a well-trained multimodal remote sensing image matching network is obtained. S4. Use the trained multimodal remote sensing image matching network to match the target image.

2. The multimodal remote sensing image matching method based on modal reconstruction and feature perturbation learning according to claim 1, characterized in that, In step S1, the weights of the pre-trained SuperPoint detector are pre-trained based on the COCO dataset, and the input image needs to undergo channel adaptation: I OPT It is an RGB three-channel, I SAR Convert to three channels by channel copying; after feature point extraction, isolated points with response values ​​below the threshold need to be filtered to ensure that the feature point density meets the requirement of no less than 5 per 100×100 pixel area.

3. The multimodal remote sensing image matching method based on modal reconstruction and feature perturbation learning according to claim 1, characterized in that, In step S2, the adaptive adjustment rule for the sampling side length D is: when I OPT with I SAR When the resolution is greater than 1024×1024 pixels, D=48 pixels; when the resolution is between 512×512 and 1024×1024 pixels, D=32 pixels; when the resolution is less than 512×512 pixels, D=24 pixels; when the sampling area exceeds the image boundary, reflection filling is used to supplement pixels.

4. The multimodal remote sensing image matching method based on modal reconstruction and feature perturbation learning according to claim 1, characterized in that, In step S2, the encoder of the cross-modal local region feature representation model includes 5 convolutional layers, each with a 3×3 convolutional kernel, stride 2, and equal-length padding. Batch normalization, instance normalization, and ReLU activation function are performed sequentially. The corresponding decoder includes 5 deconvolutional layers, symmetrical to the encoder. The output features of each deconvolutional layer are added element-wise to the corresponding layer features of the encoder. The last layer uses the Tanh activation function to output a pseudo-image. The size and number of channels of the output feature maps of each layer of the encoder are as follows: Layer 1 is 64×64 with 16 channels, Layer 2 is 32×32 with 32 channels, Layer 3 is 16×16 with 64 channels, Layer 4 is 8×8 with 128 channels, and Layer 5 is 4×4 with 256 channels. The size and number of channels of the output feature maps of each layer of the decoder are as follows: Layer 1 is 8×8 with 128 channels, Layer 2 is 16×16 with 64 channels, Layer 3 is 32×32 with 32 channels, Layer 4 is 64×64 with 16 channels, and Layer 5 is 128×128 with 3 channels.

5. The multimodal remote sensing image matching method based on modal reconstruction and feature perturbation learning according to claim 1, characterized in that, In step S2, the similarity of descriptors is calculated by constructing a KD-tree index structure using the FLANN algorithm. The calculation method uses L2 distance, and the construction time of the KD-tree index accounts for no more than 30% of the total matching time.

6. The multimodal remote sensing image matching method based on modal reconstruction and feature perturbation learning according to claim 1, characterized in that, In step S2, the threshold for the Lowe ratio test is adaptively adjusted according to the image noise level: when the image noise variance > 0.03, the threshold is set to 0.75; when the noise variance ≤ 0.03, the threshold is set to 0.8; after adjustment, the correct matching rate must be ensured to be ≥ 90%.

7. The multimodal remote sensing image matching method based on modal reconstruction and feature perturbation learning according to claim 1, characterized in that, The formula for calculating descriptor relevance is: ， in, For vector dot product operation, Let L2 be the L2 norm of the vector; , The normalization formula is: ， Furthermore, the statistical properties of the normalized descriptors must meet the following requirements: mean within the range of 0 ± 0.1 and standard deviation within the range of 0.3 ± 0.

05.

8. The multimodal remote sensing image matching method based on modal reconstruction and feature perturbation learning according to claim 7, characterized in that, The feature-level L2 loss is modified by performing L2 normalization on the features before calculating the L2 loss between the reconstructed features and the original features: ， in, , X represents the L2 normalized output features of the left and right encoders, respectively. SAR,i y SAR,i X SAR y SAR The i-th local region in X SAR,j y SAR,j X SAR y SAR The j-th local region in.

Images