Optical-sar image matching method and system for significant geometric difference

Abstract

The application provides an optical-SAR image matching method and system facing significant geometric differences, and the method comprises the following steps: inputting an optical-SAR image pair into a mixed feature extraction network composed of MaRS and RepVGG for processing, to generate multi-scale high-resolution feature maps and multi-scale effective feature maps of each modality; outputting a first co-visible region mask of each modality by using a co-visible region extraction network based on an attention mechanism, and determining a one-way matching direction; performing coarse matching according to the first co-visible region mask of each modality and the multi-scale effective feature maps; performing fine matching according to the coarse matching result and the multi-scale effective feature maps of each modality; when the number of inlier image feature point pairs in the fine matching result exceeds a preset number, performing convolution final matching according to the fine matching result and the multi-scale high-resolution feature maps of each modality, and outputting a final matching result of the optical-SAR image pair, which comprises dense high-precision matching point pairs and a final homography transformation matrix covering the whole image.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and remote sensing image processing technology, specifically relating to an optical-synthetic aperture radar (SAR) image matching method (CASGM) and system for significant geometric differences. Background Technology

[0002] Matching optical images with SAR (Synthetic Aperture Radar) images is a key technology for multi-source remote sensing data applications. However, due to the fundamental differences in the imaging mechanisms of the two sensors, severe nonlinear radiometric distortion and inherent speckle noise exist between the images. In practical applications, the collaboration of heterogeneous sensors often leads to multi-resolution mismatches, as well as large-scale geometric transformations such as significant scaling and full-range rotation caused by differences in viewing angles, posing a serious challenge to existing matching methods.

[0003] Existing methods are mainly divided into two categories: those based on handcrafted features and those based on deep learning. Methods based on handcrafted features struggle to adapt to significant radiometric differences and weakly textured regions, and feature repeatability drops sharply under large-scale geometric transformations. While deep learning-based dense matching methods (such as RoMa) have established new benchmarks, the sharp contraction of co-visible overlapping regions between images leads to the failure of the nearest neighbor supervision strategy when dealing with large geometric differences, resulting in extreme scarcity of positive samples and even matching collapse. Furthermore, the large disparity in cross-modal feature distribution makes it difficult for the model to obtain correct correspondences in the early training stages without effective guidance from co-visible regions. Therefore, there is an urgent need for an optical-SAR image matching method that can efficiently identify co-visible regions, is robust to multi-scale geometric distortions, and has high accuracy. Summary of the Invention

[0004] To overcome the limitations of existing deep learning-based dense matching methods when handling significant geometric differences, where the sharp contraction of the co-visible overlapping region between images leads to the failure of the nearest neighbor supervision strategy, resulting in an extreme scarcity of positive samples or even matching collapse, and the model struggles to obtain correct correspondences in the early training stages without effective co-visible region guidance, this invention provides an optical-SAR image matching method and system for significant geometric differences. It achieves accurate extraction of source regions and adaptive matching direction through a co-visible region identification module, and achieves dense, pixel-level high-precision localization through a convolutional final matching module. This overcomes the matching bottleneck under large-scale geometric deformation, organically combining CVM with a coarse-to-fine matching module into a complete robust matching link. Even in scenarios with significant radiation differences, speckle noise, and large-scale geometric distortion, it can stably establish high-quality dense correspondences, significantly improving matching success rate and accuracy.

[0005] According to one aspect of the present invention, an optical-SAR image matching method for significant geometric differences is provided, comprising:

[0006] Obtain the optical-SAR image pairs to be matched;

[0007] Optical-SAR image pairs are input into a hybrid feature extraction network composed of MaRS and RepVGG to generate multi-scale high-resolution feature maps and multi-scale effective feature maps for each modality.

[0008] A co-visible region extraction network based on an attention mechanism is used to process multi-scale high-resolution feature maps of each modality, output the first co-visible region mask of each modality, and determine the unidirectional matching direction of optical-SAR image pairs;

[0009] Coarse matching is performed based on the first common visible region mask of each modality and the effective feature map at multiple scales to obtain the coarse matching result of optical-SAR image pairs; wherein, the coarse matching result is a pair of image feature points;

[0010] Based on the coarse matching results and the multi-scale effective feature maps of each modality, fine matching is performed to obtain the fine matching results of optical-SAR image pairs, including the sub-pixel level coordinates of the feature point pairs of the in-point image, the unified homography transformation matrix, and the second common visible region mask of each modality.

[0011] When the number of inlier image feature point pairs in the fine matching result exceeds the preset number, convolutional final matching is performed based on the fine matching result and the multi-scale high-resolution feature maps of each modality, and the final matching result of the optical-SAR image pair is output, including dense high-precision matching point pairs covering the entire image and the final homography transformation matrix.

[0012] As a further technical solution, the hybrid feature extraction network includes a first feature extraction branch for processing optical modal images and a second feature extraction branch for processing SAR modal images; wherein, each feature extraction branch processes the input image of the target modality and extracts features using MaRS. and Feature maps at resolution scales; extracted using RepVGG , and Multi-scale high-resolution feature maps of three target modalities with higher resolution scales; the feature maps extracted by MaRS are enhanced with long-connection information using the Transformer attention mechanism to output multi-scale effective feature maps of the target modalities.

[0013] As a further technical solution, an attention-based co-visible region extraction network is used to process multi-scale effective feature maps of various modalities. The steps include:

[0014] The effective feature maps of any modality at multiple scales are adaptively concatenated along the channel dimension and then convolved to obtain the fused feature map of that modality.

[0015] A gated fusion mechanism using self-attention and cross-attention Transformer layers is adopted to generate a co-visibility probability map of each modality based on the obtained fusion feature maps of each modality.

[0016] Based on a preset confidence threshold, the first common visible region mask for each mode is generated using the common visible probability map of each mode, and the unidirectional matching direction of the optical-SAR image pair is determined according to the number of first common visible region masks in the first common visible region mask of each mode.

[0017] As a further technical solution, the gated fusion mechanism of the self-attention and cross-attention Transformer layers takes the flattened sequence of the fusion feature maps of each modality as input, generates self-attention features of each modality through several Transformer layers equipped with self-attention, and then performs cross-modal interaction of self-attention features through the cross-attention Transformer layer to generate cross-attention features of each modality. Finally, for any modality, the cross-attention features and self-attention features of that modality are dynamically integrated using a learning gating mechanism, and the common visible region probability map of that modality is generated by the probability prediction head.

[0018] As a further technical solution, the coarse matching results of optical-SAR image pairs are obtained through the following steps:

[0019] Based on the first common visible region mask of each modality, respectively for the same modality The effective feature maps at the resolution scale are subjected to non-common visible region occlusion processing to obtain the mask features of each modality;

[0020] The global similarity matrix between the feature sets of the two modal masks is calculated by dot product, and the global similarity matrix is ​​scaled by temperature hyperparameter.

[0021] A one-way Softmax function is applied to the scaled global similarity matrix in the one-way matching direction to generate a matching confidence matrix. Image feature point pairs exceeding the preset matching confidence threshold are retained. For many-to-one mapping conflicts, only the image feature point pairs with the highest matching confidence are retained. The retained image feature point pairs are used as the coarse matching results of optical-SAR image pairs.

[0022] As a further technical solution, fine-tuning involves the following steps:

[0023] Using the image feature point pairs output by global coarse matching as seed points, from each modality Local features are extracted from the effective feature map at the resolution scale according to a window of a preset size; the local correlation volume between the local features of the image feature point pair is calculated, a probability distribution is generated by the spatial Softmax operation, and the sub-pixel level coordinates of the image feature point pair are obtained by calculating the mathematical expectation of the probability distribution;

[0024] RANSAC is applied to filter out subpixel-level coordinate outliers, retain inlier image feature point pairs, and use the subpixel-level coordinates of the inlier image feature point pairs to calculate the transformation from the source image to the target image. Combined with the unidirectional matching direction, a unified homography transformation matrix is ​​calculated.

[0025] The image boundaries of optical-SAR image pairs are deformed using a unified homography transformation matrix, and the first common visible region mask of each mode is dynamically updated to obtain the second common visible region mask of each mode.

[0026] The subpixel-level coordinates of the feature point pairs in the output image, the homography transformation matrix, and the second common visible region mask for each modality are used as fine matching results.

[0027] As a further technical solution, the final matching step of convolution includes:

[0028] At 1 / 8 and 1 / 4 resolution scales, initial points are randomly sampled from the effective pixels of the source image using a second common-visible mask, and the initial points are projected onto the target image using a unified homography transformation matrix to obtain initial matching coordinates;

[0029] The projected reference features, geometrically deformed target features, coordinate displacement embeddings, and local correlation volumes are obtained. These four features are then concatenated along the channel dimension, and the resulting joint features are input into the thinner network to output the residual coordinate offset.

[0030] The matching coordinates are updated based on the residual coordinate offset, and the second homography transformation matrix is ​​calculated using the RANSAC algorithm. The second common-visible mask is then dynamically updated using the second homography transformation matrix.

[0031] At a 1 / 2 resolution scale, the high-resolution feature maps at the 1 / 2 resolution scale and the 1 / 4 resolution scale are fused, and the pixel-level high-precision coordinate offset is output through the final convolutional thinner. The high-precision coordinate offset is added to the initial matching coordinates, and the final output is a dense high-precision matching point pair covering the entire image. The second homography transformation is output as the final homography transformation matrix.

[0032] According to another aspect of this specification, an optical-SAR image matching system for significant geometric differences is provided, comprising:

[0033] The input module is used to acquire optical-SAR image pairs to be matched;

[0034] The hybrid feature extraction module is used to input optical-SAR image pairs into a hybrid feature extraction network composed of MaRS and RepVGG to generate multi-scale high-resolution feature maps and multi-scale effective feature maps for each modality.

[0035] The common visible region extraction module is used to process multi-scale high-resolution feature maps of each modality using an attention-based common visible region extraction network, output the first common visible region mask of each modality, and determine the unidirectional matching direction of optical-SAR image pairs.

[0036] The coarse matching module is used to perform coarse matching based on the first common visible region mask of each modality and the effective feature map at multiple scales to obtain the coarse matching result of optical-SAR image pairs; wherein, the coarse matching result is a pair of image feature points;

[0037] The fine matching module is used to perform fine matching based on the coarse matching results and the multi-scale effective feature maps of each modality to obtain the fine matching results of optical-SAR image pairs, including the sub-pixel level coordinates of the feature point pairs of the in-point image, the unified homography transformation matrix, and the second common visible region mask of each modality.

[0038] The convolutional final matching module is used to perform convolutional final matching based on the fine matching results and the multi-scale high-resolution feature maps of each modality when the number of inlier image feature point pairs in the fine matching results exceeds a preset number. It outputs the final matching results of optical-SAR image pairs, including dense high-precision matching point pairs covering the entire image and the final homography transformation matrix.

[0039] According to another aspect of this specification, an electronic device is provided, including a memory and a processor, the memory storing program instructions executed by the processor, the processor invoking the program instructions to perform an optical-SAR image matching method for significant geometric differences.

[0040] According to another aspect of this specification, a non-transitory computer-readable storage medium is provided that stores computer instructions that cause the computer to perform an optical-SAR image matching method oriented towards significant geometric differences.

[0041] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0042] The Common Visible Region (CVM) module enables accurate extraction and adaptive matching of homologous regions. It accurately identifies overlapping regions between optical and SAR images and adaptively selects the optimal unidirectional matching direction, eliminating invalid search space at the source and alleviating the problem of scarce positive samples under large-scale geometric distortion.

[0043] The Convolutional Final Matching (CFM) module achieves dense, pixel-level high-precision localization. It iteratively refines coordinates, homography matrices, and co-visible masks on multi-scale high-resolution features, overcoming the limitations of sparse keypoint frameworks and outputting dense, high-precision pixel-level matching results covering the entire image.

[0044] Overcoming the matching bottleneck under large-scale geometric deformation. By organically combining CVM with a coarse-to-fine matching module into a complete robust matching link, it can still stably establish high-quality dense correspondences in scenarios with significant radiation differences, speckle noise, and large-scale geometric distortion, significantly improving the matching success rate and accuracy. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart illustrating an optical-SAR image matching method for significant geometric differences, provided in an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of the common visible area identification module in an embodiment of the present invention;

[0048] Figure 3 This is a visual comparison chart of the matching effect of this method and some other methods in the embodiments of the present invention;

[0049] Figure 4 This is a visual comparison chart of the generalization matching of the method in this embodiment of the invention and some other methods on different datasets;

[0050] Figure 5 A schematic diagram of the structure of an optical-SAR image matching system for significant geometric differences provided in an embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0052] It should be noted that:

[0053] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0054] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be decomposed, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0056] like Figure 1 As shown, an optical-SAR image matching method (CASGM) for significant geometric differences includes:

[0057] Step 1: Obtain the optical-SAR image pairs to be matched;

[0058] Step 2: Input the optical-SAR image pairs into a hybrid feature extraction network composed of MaRS and RepVGG to generate multi-scale high-resolution feature maps and multi-scale effective feature maps for each modality;

[0059] Step 3: Use the attention-based common visible region extraction network to process the multi-scale high-resolution feature maps of each modality, output the first common visible region mask of each modality, and determine the unidirectional matching direction of the optical-SAR image pair.

[0060] Step 4: Perform coarse matching based on the first common visible region mask of each modality and the multi-scale effective feature map to obtain the coarse matching result of the optical-SAR image pair; wherein, the coarse matching result is the image feature point pair;

[0061] Step 5: Perform fine matching based on the coarse matching results and the multi-scale effective feature maps of each modality to obtain the fine matching results of optical-SAR image pairs, including the sub-pixel level coordinates of the feature point pairs of the in-point image, the unified homography transformation matrix, and the second common visible region mask of each modality.

[0062] Step 6: When the number of inlier image feature point pairs in the fine matching result exceeds the preset number, perform convolutional final matching based on the fine matching result and the multi-scale high-resolution feature maps of each modality, and output the final matching result, including dense high-precision matching point pairs covering the entire image and the final homography transformation matrix.

[0063] In step 2, the hybrid feature extraction network includes a first feature extraction branch for processing optical modal images and a second feature extraction branch for processing SAR modal images; wherein each feature extraction branch processes the input image of the target modality and extracts features using MaRS. and Multi-scale feature maps of varying resolution were extracted using RepVGG. , and Three higher resolution scales of multi-scale high-resolution feature maps; the multi-scale feature maps extracted by MaRS are enhanced with long connection information using the Transformer attention mechanism to output the multi-scale effective feature maps of the target modality.

[0064] Specifically, the optical image and SAR image to be matched are input into a hybrid feature extraction network consisting of MaRS (Multi-Modality Remote Sensing) and RepVGG (Re-parameterization VGG). The network utilizes the rich global semantics extracted by MaRS. and Feature maps at resolution scales (denoted as) , This is used for subsequent coarse matching, common visible region identification, and intermediate refinement stages; RepVGG is used to extract discriminative local structural details. , and Feature maps at three higher resolution scales (denoted as) , , This is used in the subsequent local convolution final matching stage. Finally, the Transformer attention mechanism is used to... and Long-connection information enhancement is performed to output multi-scale effective feature maps.

[0065] As a supplement, the hybrid feature extraction network adopts a multi-branch structure during training and is integrated into a single-path network during inference through reparameterization technology, so that the image is propagated forward to the input network.

[0066] In step 3, a network based on an attention mechanism for extracting co-visible regions is used to process multi-scale effective feature maps of each modality. The steps include:

[0067] Step 3-1: Adaptively stitch the multi-scale effective feature maps of any modality along the channel dimension, and then convolve them to obtain the fused feature map of that modality;

[0068] Step 3-2: Using a gated fusion mechanism of self-attention and cross-attention Transformer layers, generate a co-visibility probability map of each modality based on the obtained fusion feature maps of each modality;

[0069] Step 3-3: Based on the preset confidence threshold, generate the first common visible region mask for each mode using the common visible probability map of each mode, and determine the unidirectional matching direction of the optical-SAR image pair according to the number of first common visible region masks in the first common visible region mask of each mode.

[0070] Specifically, in step 3-1, each mode Effective feature maps at resolution scales Bilinear interpolation downsampling to Resolution, and with Effective feature maps at resolution scales The features are concatenated along the channel dimension and then convolved to obtain the fused feature maps of each modality. ,in, Represents a set of modes. These represent optical and SAR modes, respectively.

[0071] In step 3-2, the gated fusion mechanism of the self-attention and cross-attention Transformer layers takes the flattened sequence of the fusion feature maps of each modality as input, generates self-attention features of each modality through several Transformer layers equipped with self-attention, and then performs cross-modal interaction of self-attention features through the cross-attention Transformer layer to generate cross-attention features of each modality. Finally, for any modality, the cross-attention features and self-attention features of that modality are dynamically integrated using the learning gating mechanism, and the common visible region probability map of that modality is generated by the probability prediction head.

[0072] Specifically, the mathematical representation of the learning gating mechanism is as follows:

[0073]

[0074]

[0075] In the formula, , These are the outputs obtained from the fused feature maps through the self / cross attention Transformer layers, respectively. , For learnable linear projection; Represents the gate vector; For the Sigmoid function; This indicates concatenation along the channel dimension. This is element-wise multiplication; Indicates gated output, After processing by the prediction head, a common visibility probability map is obtained. .

[0076] In step 3-3, pixels with probability values ​​greater than a preset confidence threshold in the common visible region probability map of each modality are identified as common visible region pixels, forming the first common visible region mask for each modality, and the unidirectional matching direction is determined based on the number of common visible region pixels in the first common visible region masks of the two images.

[0077] In this context, the unidirectional matching direction is set to start matching from the modality with fewer pixels in the common visible area, that is, the image with fewer pixels in the common visible area is used as the source image.

[0078] Specifically, in step 3-3, With preset confidence threshold Binarization yields the first visible region mask. Based on the total number of effective pixels in the mask, a unique unidirectional matching direction is dynamically determined using the image with the smaller coverage area as the source image. Mathematically, it is represented as:

[0079]

[0080] Step 4, obtaining the coarse matching results of the optical-SAR image pairs, includes the following steps:

[0081] Step 4-1: Based on the first common visible region mask of each modality, analyze the multi-scale effective feature maps of the same modality. The effective feature maps at the resolution scale are subjected to non-common visible region occlusion processing to obtain the mask features of each modality;

[0082] Step 4-2: Calculate the global similarity matrix between the feature sets of the two modal masks by dot product, and scale the global similarity matrix using the temperature hyperparameter.

[0083] Step 4-3: Apply the one-way Softmax function to the scaled global similarity matrix in the one-way matching direction to generate a matching confidence matrix. Retain image feature point pairs that exceed the preset matching confidence threshold. For many-to-one mapping conflicts, only retain the image feature point pairs with the highest matching confidence. Use the retained image feature point pairs as the coarse matching results of optical-SAR image pairs.

[0084] Preferably, step 4-1 further includes processing different modes. The effective feature maps at the resolution scale are normalized.

[0085] Specifically, in step 4-2, the scaling process uses the temperature hyperparameter. The similarity score of image feature point pairs falling outside the first common visible region mask is penalized to negative infinity.

[0086] In step 4-3, the matching confidence matrix is ​​calculated from the global similarity matrix after global scaling, and its matrix elements correspond to the matching confidence scores of each candidate image feature point pair.

[0087] Specifically, the global coarse matching in step 4 is essentially performed on the common visible mask. Constrained unidirectional Softmax similarity filtering sensitive to execution direction:

[0088] First After normalization Obtain mask features by occlusion : Calculate the similarity matrix, and then calculate the temperature hyperparameter. Scaling will penalize items that fall outside the shared visible mask. Mathematically, it is represented as:

[0089]

[0090] In the formula, This represents the similarity matrix after scaling. Indicates the index of feature points in a visible light modal image; This represents the feature point index in the visible light modal image.

[0091] Along the unidirectional matching direction The confidence matrix is ​​obtained by applying Softmax. : Select those with a confidence level exceeding the matching confidence threshold. Furthermore, for many-to-one mapping conflicts, only the image feature point pair with the highest confidence in each unique image pair is retained, thereby establishing a set of highly reliable initial matching relationships as coarse matching results, providing initial seed points for subsequent refinement.

[0092] In step 5, the fine matching process includes:

[0093] Using the image feature point pairs output by global coarse matching as seed points, from each modality Local features are extracted from the effective feature map at the resolution scale according to a window of a preset size; the local correlation volume between the local features of the image feature point pair is calculated, a probability distribution is generated by the spatial Softmax operation, and the sub-pixel level coordinates of the image feature point pair are obtained by calculating the mathematical expectation of the probability distribution;

[0094] RANSAC is applied to filter out subpixel-level coordinate outliers, retain inlier image feature point pairs, and use the subpixel-level coordinates of the inlier image feature point pairs to calculate the transformation from the source image to the target image. Combined with the unidirectional matching direction, a unified homography transformation matrix is ​​calculated.

[0095] The image boundaries of optical-SAR image pairs are deformed using a unified homography transformation matrix, and the first common visible region mask of each mode is dynamically updated to obtain the second common visible region mask of each mode.

[0096] Specifically, step 5 is the intermediate refinement matching process, which uses the coarse matching result as the seed point. Coordinate refinement is achieved at the scale using the spatial expectation of locally correlated volumes: Extraction at scale Local features within the window (w represents the window width); calculate the local correlation volume between the reference feature and the target feature, generate a probability distribution through a spatial Softmax operation, and obtain more refined coordinates, i.e., sub-pixel level coordinates, by calculating the mathematical expectation of this probability distribution.

[0097] The refined coordinates are filtered for outliers using RANSAC (RANdom Sample Consensus), and the transformation matrix from the source image to the target image is robustly estimated using the retained inlier image feature points. And based on the matching direction Mapped to a unified homography transformation matrix Mathematically, it is represented as:

[0098]

[0099] Finally utilize Deform the image boundaries of the input optical-SAR image pair and update them dynamically. Output the updated homography transformation matrix and the common-visible mask. This provides more precise geometry and mask guidance for subsequent stages.

[0100] In step 6, it is first necessary to determine whether the number of inliers in the inlier image feature point pairs obtained in the intermediate refinement stage exceeds the preset inlier number threshold. If it does, the convolution final matching is activated.

[0101] Furthermore, in step 6, the final convolutional matching includes the following steps:

[0102] Step 6-1: At 1 / 8 and 1 / 4 resolution scales, use the second common visible mask to randomly sample initial points from the effective pixels of the source image, and use the unified homography transformation matrix to project the initial points onto the target image to obtain the initial matching coordinates;

[0103] Step 6-2: At a 1 / 2 resolution scale, calculate the residual coordinate offset based on the obtained initial matching coordinates;

[0104] Step 6-3: Update the matching coordinates according to the residual coordinate offset, and calculate the second homography transformation matrix using the RANSAC algorithm, and dynamically update the second common-visible mask using the second homography transformation matrix;

[0105] Step 6-4: At the 1 / 2 resolution scale, fuse the high-resolution feature maps at the 1 / 2 resolution scale and the 1 / 4 resolution scale, and output the pixel-level high-precision coordinate offset through the final convolutional thinner. Add the high-precision coordinate offset to the initial matching coordinates, and finally output dense high-precision matching point pairs covering the entire image, and output the second homography transformation as the final homography transformation matrix.

[0106] Preferably, step 6-2 can also employ a feature splicing and embedding process to obtain the residual coordinate offset considering multiple factors. The steps include:

[0107] First, the projected reference features, geometrically deformed target features, coordinate displacement embeddings, and local correlation volumes are obtained. These include: reference features (high-resolution feature maps of the source image at a 1 / 2 resolution scale) and dense feature maps of the target image (high-resolution feature maps of the target image at a 1 / 2 resolution scale). These are first compressed using a linear projection layer, deforming the projected target features to the reference coordinate system, resulting in the projected reference features and geometrically deformed target features. Simultaneously, convolution is applied to the relative offset between the current initial matching coordinates and the normalized uniform grid to generate coordinate displacement embeddings, and local correlation volumes are calculated within a defined local search radius.

[0108] The acquired projected reference features, geometrically deformed target features, coordinate displacement embeddings, and local correlation volumes are concatenated along the channel dimension. The resulting joint features are then input into a thinner network, which outputs the residual coordinate offset. The thinner network consists of multiple depthwise separable convolutional blocks. Each convolutional block sequentially comprises depthwise convolution, batch normalization, ReLU activation, and pointwise convolution, and uses two independent convolutional heads to regress the residual coordinate offset.

[0109] Step 6-3 is and Iterative update process: In After updating the dense coordinates by scale, the RANSAC algorithm is used to recalculate a more accurate result. And then used again to update the common-visible mask; subsequently in The above "sampling-refinement-RANSAC update" cycle is repeated in scale order. and After updating the coordinates by scale, the RANSAC algorithm is used to recalculate a more accurate second homography transformation matrix, which is then used again to update the co-visible mask.

[0110] Step 6-4 is The process of scale blending and final output: in the final In terms of scale, and The feature maps at the level are fused to fully integrate high-resolution discriminative local details; the fused features are then processed by a final convolutional thinner to determine pixel-level high-precision coordinate offsets. At this stage, the homography transformation matrix or common visible mask is no longer updated. Instead, the coordinate offsets are directly added to the initial grid coordinates to output a dense, high-precision correspondence covering the entire image and the final homography transformation matrix.

[0111] Example 1: Overall Network Architecture and Workflow:

[0112] The CASGM method of this invention adopts an end-to-end network architecture, consisting of four parts: data preparation, multi-scale feature extraction, common visible region identification and extraction (CVM), and coarse-to-fine matching module, to achieve efficient and high-precision optical-SAR image matching.

[0113] The specific workflow is as follows:

[0114] S1. Data Preparation: Construct a strictly pixel-aligned multi-resolution dataset, covering scaling up to 4x. The ground truth image pairs with full-range rotation and translation transformations up to 0.4 times the image patch size provide accurate pixel-level supervision for subsequent network training.

[0115] S2. Multi-scale Feature Extraction: Input optical and SAR images to be matched, and construct a hybrid feature extraction network composed of MaRS and RepVGG. MaRS is used to extract feature maps containing rich global semantics; at the same time, RepVGG is used to extract high-resolution feature maps containing discriminative local structural details. , , Finally, the feature maps extracted by MaRS are enhanced with long-connection information using the Transformer attention mechanism to obtain multi-scale effective feature maps. , ).

[0116] S3. Common Visible Region Recognition (CVM): After fusing the input multi-scale effective feature maps through self-attention and cross-attention Transformer layers and learning gating, predict the common visible probability map pixel by pixel. and with threshold Binarization yields the first commonly visible mask. Based on the total number of effective pixels covered by the first common-visible mask of the two modalities, and using the image with the smaller coverage as the source image, the unidirectional matching direction is dynamically determined. .

[0117] S4. Coarse-to-fine matching module: Guided by the common-visible mask output by CVM and the unidirectional matching direction, it sequentially performs three stages of coarse-to-fine matching:

[0118] (1) Coarse matching: In Firstly in terms of scale Occlusion normalized features Then, according to the temperature over-parameter Calculate the global similarity matrix with mask penalty by scaling along the direction. The confidence matrix is ​​obtained by applying one-way Softmax. According to the preset confidence threshold After filtering and resolving many-to-one conflicts, a highly reliable initial correspondence (image feature point pairs) is established as seed points for subsequent stages.

[0119] (2) Intermediate refinement: In Extraction at scale centered on seed point The local related volumes within the window are used to obtain more refined coordinates through the mathematical expectation of the spatial Softmax probability distribution; RANSAC is applied to the refined coordinates to filter outgoing points and estimate the source-to-target transformation matrix. Combined with direction Mapped to a uniform homography matrix Based on this, the image boundaries are deformed to dynamically update the common visible mask. This provides more precise geometry and mask guidance for the next stage.

[0120] (3) Convolutional Final Matching (CFM): Activated only when the number of points exceeds a preset threshold during the intermediate refinement stage; sequentially in and The initial point in the effective pixels of the dynamic mask is resampled at scale. Projected onto the target image, the reference features, deformed target features, coordinate displacement embeddings, and local correlation volumes are concatenated along the channel dimension. This concatenation is then input into a convolutional thinner consisting of multiple depthwise separable convolutional hidden blocks, regressing the residual coordinate offsets and confidence scores respectively. After each scale is completed, the results are updated using RANSAC. And synchronized updates ; Ultimately Scale will and Feature fusion, through convolution thinner, directly outputs dense, high-precision pixel-level matching point pairs covering the entire image range. At this stage, the transformation matrix or mask is no longer updated.

[0121] The network takes as input pairs of optical and SAR images to be matched and outputs dense, high-precision matching point pairs covering the entire map area and the final homography transformation matrix. It is suitable for multi-source, multi-modal data analysis tasks in scenarios such as remote sensing image fusion and change monitoring.

[0122] Example 2: Multi-scale feature extraction process:

[0123] The core of multi-scale feature extraction is to balance the global expressive power of cross-modal features with the computational efficiency of local localization. The specific implementation is as follows:

[0124] S21. Accelerated Network Construction and Inference: The RepVGG network adopts a multi-branch structure during the training phase to enhance feature learning capabilities; during the inference phase, it integrates the network into a single-path forward propagation through structural reparameterization technology, thereby significantly improving computational and inference efficiency.

[0125] S22. Hybrid Feature Map Extraction: The optical image to be matched is scaled down to the SAR image (input size uniformly). The pixel is input into a hybrid feature extraction network consisting of MaRS and RepVGG: the multimodal base model (MaRS) is used to extract features containing rich global semantics. and Feature maps at resolution scales (denoted as) , This is mainly used for subsequent coarse matching, common visible region identification, and intermediate refinement stages; RepVGG is used to extract features containing discriminative local structural details. , , Feature maps at three high-resolution scales (denoted as follows) , , ), mainly used in the subsequent local convolution final matching (CFM) stage.

[0126] S23. Feature Information Enhancement: Enhancement of Feature Maps Extracted from MaRS and It integrates a Transformer-based feature interaction module. It captures global contextual information within a single modality through a self-attention mechanism, and then enhances the cross-modal feature correlation between optical and SAR images through a cross-attention mechanism. Finally, it outputs an enhanced multi-scale effective feature map for subsequent modules to use.

[0127] Example 3: Common Visible Region Identification and Matching Direction Determination:

[0128] like Figure 2 As shown, the core of the Common Visible Region (CVM) module is to eliminate invalid regions and determine the matching direction. The specific implementation steps are as follows:

[0129] S31. Feature Preprocessing: Preprocessing the features obtained from MaRS during the feature extraction stage. and ( , Derived from optical images, (Source: SAR imagery). downsampling to Resolution, and splicing along the channel dimension, via The convolutional layers are compressed and fused, and then subjected to batch normalization and ReLU activation to obtain a unified feature representation. .

[0130] S32, Feature Interaction and Fusion: [This section appears to be incomplete and requires further context.] The flattened sequence input is processed by a self-attention multi-layer Transformer encoder to generate self-attention features. Subsequently, multiple cross-attention Transformer layers are used to facilitate cross-image interaction, resulting in cross-attention features. By dynamically integrating the two feature representations mentioned above using a learning gating mechanism, we obtain... .

[0131] S33, Common Visible Area Mask Generation: [This part is incomplete and likely refers to a separate process.] After restoring to the original two-dimensional spatial dimensions, the final co-visible region probability map is generated using a probability prediction head. The probability value is greater than a set threshold (e.g., ...). The pixels of the first common-visible region are determined to be the common-visible region, forming a binary first common-visible mask. .

[0132] S34. Matching Direction Determination: The matching direction is dynamically determined based on the total number of valid pixels (value 1) in the first common visible mask for each modality. (Comparison) and The size of the image determines the source image, using an image with fewer effective pixels to form a unique unidirectional matching direction. This strategy of matching from images with a small overlap ratio to larger images ensures that all source feature points have corresponding visible target points.

[0133] Example 4: Global coarse matching:

[0134] The goal of global coarse matching is to establish initial correspondences at a coarse scale and effectively suppress outliers under the constraint of a shared visible region mask. The specific implementation steps are as follows:

[0135] S41. Feature Preprocessing and Masking / Occlusion: Obtaining the output of the feature extraction stage. Scale feature map (in (Representing optical and SAR images respectively). First, regarding... Normalization is then performed. Subsequently, the common visible area mask generated in Example 3 is used. Occlude the non-co-visible regions of the normalized features (i.e., set the feature values ​​of the non-visible regions to zero) to obtain a coarse feature map of the mask. .

[0136] S42. Global Similarity Matrix Calculation: Calculate the global similarity matrix between mask feature sets using dot product. and with temperature hyperparameter Scaling is performed.

[0137] S43. Matching confidence generation and filtering: Based on the unidirectional matching direction dynamically determined in Example 3. Apply the one-way Softmax function to the similarity matrix to generate a matching confidence matrix. Finally, select those in the confidence matrix that exceed the preset matching confidence threshold (e.g., Image feature point pairs. To address potential many-to-one mapping conflicts, only the match with the highest confidence in each unique image feature point pair is retained, thereby establishing a set of highly reliable and unique initial correspondences, providing initial seed points for subsequent local refinement.

[0138] Example 5: Intermediate Refinement Matching:

[0139] The intermediate refinement matching stage aims to further improve spatial positioning accuracy and estimate a reliable transformation matrix based on the global coarse matching. The specific implementation steps are as follows:

[0140] S51. Local Feature Extraction and Coordinate Refinement: Using the highly reliable initial correspondence obtained in the global coarse matching stage as seed points, refine the feature map at a higher resolution. Local features are extracted. Subsequently, the local correlation volume between the reference features and the target features is calculated, and a probability distribution is generated through a spatial softmax operation. By calculating the mathematical expectation of this distribution, more refined coordinates (sub-pixel level coordinates) are obtained.

[0141] S52. Outlier Filtering and Homography Matrix Estimation: After obtaining the refined coordinates, the RANSAC algorithm is applied to filter outliers. Using the retained inlier image feature point pairs, the initial transformation matrix from the source image to the target image is robustly estimated. .

[0142] S53. Calculation of the unified homography transformation matrix: To maintain a geometrically consistent mapping from the optical image to the SAR image, the unified homography transformation matrix is ​​calculated based on the optimal unidirectional matching direction determined in Example 3. Defined as:

[0143]

[0144] S54. Dynamic Update of Common Visible Mask: The effective common visible mask in this stage is not constant. A new mask is calculated each time. Immediately apply it to the boundary of the deformed image and dynamically update the binary co-visible mask of the two images. The final output is the updated homography transformation matrix and the co-visible mask. This provides more accurate geometric and mask guidance for the subsequent Convolution Final Matching (CFM) stage.

[0145] Example 6: CFM Convolution Final Matching:

[0146] The Convolutional Final Matching (CFM) stage aims to acquire dense matching points covering the entire image and continuously refine the alignment accuracy between images. Since this stage is highly dependent on the accuracy of preceding geometric estimations, its implementation must follow an iterative refinement process from coarse to fine.

[0147] S61. Stage Activation Judgment: First, determine whether the number of interior points obtained in the intermediate refinement stage exceeds a preset threshold (e.g., 100). This stage is activated only when the number of interior points is sufficient to avoid relying on unreliable data. Incorrect projection coordinates caused training and matching to fail.

[0148] S62, Multi-scale Iterative Refinement and Final Output: (Sequentially in...) , , The convolutional refiner is executed on three scales to achieve coarse-to-fine dense matching.

[0149] like Figure 3 The image shown is a visual comparison of the matching effects of the method of this invention with GLIFT, LoFTR, GeoFormer, and RoMa at different scaling scales. Figure 3 (a) in the text represents a scaling factor of 1 to 2. Figure 3 (b) in the text represents a scaling factor of 2 to 4.

[0150] like Figure 4 The figure shown is a visualization comparison of the generalization matching of the method of the present invention with GLIFT, LoFTR, GeoFormer, and RoMa on different datasets: other SEN1 and SEN2 satellite images are used to evaluate the cross-scene generalization ability under the same satellite and resolution conditions, GF1 and GF3 satellite images are used to evaluate the cross-platform and cross-SAR imaging mode generalization ability under the same resolution, and Google Maps and satellite images are used to evaluate the cross-modal generalization ability. Figure 4 (a) in the text represents a scaling factor of 1 to 2. Figure 4 (b) in the text represents a scaling factor of 2 to 4.

[0151] Example 7: Loss Function:

[0152] The total training loss consists of the common visible area loss. and matching loss composition: .

[0153] (1) Loss in the visible area FocalLoss is used to address the inherent class imbalance between visible and invisible regions. The predicted co-visibility probability map is supervised against its corresponding ground truth value. The calculation formula is as follows:

[0154]

[0155] In the formula, , These are the predicted probability maps of the co-visible area from optical and SAR images, respectively. , These are the corresponding truth masks.

[0156] (2) Overall matching loss It provides supervision for estimating the correspondences at various scales of the network, and consists of a three-stage loss function, the formula of which is:

[0157]

[0158] In the formula, , and These are predefined weight coefficients for each stage, which are empirically set to 1 during training. Specifically, they include:

[0159] coarse matching loss FocalLoss is used to supervise the matching confidence matrix. To address the class imbalance problem between positive and negative matching pairs, the formula is:

[0160]

[0161] In the formula, Represents the set of valid dense matching pairs. It is the predicted matching probability when the image pair is a true match (otherwise it is...). ), It is a balancing factor. It is the focusing parameter.

[0162] Intermediate refinement loss :right The scaled coordinates are weighted using uncertainty. Distance is used for supervision, and the formula is:

[0163]

[0164] In the formula, It is the number of effectively refined matches. and These are the predicted and the ground truth normalized refined coordinates, respectively. This represents the spatial variance (i.e., uncertainty) of network-based predictions. The weights are calculated dynamically.

[0165] Convolutional Final Matching (CFM) Loss : Calculate three refinement scales in normalized coordinate space ( The sum of endpoint errors (EPE) is given by the formula:

[0166]

[0167] In the formula, Includes scale The effective intra-frame pixels below, and These represent the predicted and ground truth normalized deformable coordinates on the target image, respectively.

[0168] The implementation of the various embodiments of the present invention is based on programmed processing through a system with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of the present invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of the present invention provide an optical-SAR image matching system for significant geometric differences. This system is used to execute an optical-SAR image matching method for significant geometric differences from the above method embodiments.

[0169] See Figure 5 The system includes:

[0170] The input module is used to acquire optical-SAR image pairs to be matched;

[0171] The hybrid feature extraction module is used to input optical-SAR image pairs into a hybrid feature extraction network composed of MaRS and RepVGG to generate multi-scale high-resolution feature maps and multi-scale effective feature maps for each modality.

[0172] The common visible region extraction module is used to process multi-scale high-resolution feature maps of each modality using an attention-based common visible region extraction network, output the first common visible region mask of each modality, and determine the unidirectional matching direction of optical-SAR image pairs.

[0173] The coarse matching module is used to perform coarse matching based on the first common visible region mask of each modality and the effective feature map at multiple scales to obtain the coarse matching result of optical-SAR image pairs; wherein, the coarse matching result is a pair of image feature points;

[0174] The fine matching module is used to perform fine matching based on the coarse matching results and the multi-scale effective feature maps of each modality to obtain the fine matching results of optical-SAR image pairs, including the sub-pixel level coordinates of the feature point pairs of the in-point image, the unified homography transformation matrix, and the second common visible region mask of each modality.

[0175] The convolutional final matching module is used to perform convolutional final matching based on the fine matching results and the multi-scale high-resolution feature maps of each modality when the number of inlier image feature point pairs in the fine matching results exceeds a preset number. It outputs the final matching results of optical-SAR image pairs, including dense high-precision matching point pairs covering the entire image and the final homography transformation matrix.

[0176] It should be noted that the system embodiments provided by the present invention are used not only to implement the methods in the above method embodiments, but also to implement the methods in other method embodiments provided by the present invention. The only difference is that corresponding functional modules are set. The principle is basically the same as that of the above system embodiments provided by the present invention. As long as those skilled in the art can improve the modules in the above system embodiments by referring to the specific technical solutions in other method embodiments and combining technical features to obtain corresponding technical means and technical solutions composed of these technical means, on the basis of the above system embodiments, and on the premise of ensuring the practicality of the technical solutions, they can obtain corresponding system-like embodiments for implementing the methods in other method-like embodiments.

[0177] The method in this embodiment of the invention is implemented using an electronic device; therefore, it is necessary to introduce the relevant electronic device. For this purpose, embodiments of the present invention provide an electronic device, such as... Figure 6 As shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, wherein the at least one processor, the communication interface, and the at least one memory communicate with each other via the communication bus. The at least one processor invokes logical instructions stored in the at least one memory to execute all or part of the steps of the methods provided in the foregoing method embodiments.

[0178] Furthermore, when the logical instructions in at least one of the aforementioned memories are implemented as software functional units and sold or used as independent products, they are stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, is embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (a personal computer, server, or network device) to execute all or part of the steps of the methods described in the various method embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks—various media for storing program code.

[0179] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, located in one place, or distributed across multiple network units. The purpose of this embodiment is achieved by selecting some or all of the modules according to actual needs. Those skilled in the art will understand and implement this without any inventive effort.

[0180] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0181] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0182] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0183] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0184] Based on the same technical concept as the foregoing embodiments, the present invention provides a non-transitory computer-readable storage medium that stores computer instructions that cause the computer to execute a method.

[0185] This invention proposes a CASGM (Optical-SAR Image Matching Method) for Optical-SAR Images with Significant Geometric Differences. The method employs an end-to-end forward architecture, sequentially performing three main modules on the input optical-SAR image pairs: multi-scale feature extraction, common visible region identification and extraction (CVM), and coarse-to-fine matching.

[0186] (1) Multi-scale feature extraction: Construct a hybrid feature extraction network consisting of the multimodal basic model MaRS and RepVGG to extract multi-scale feature maps from the input image that take into account both cross-modal global semantics and local discriminative details;

[0187] (2) Common Visible Region Recognition (CVM): Using multi-scale features as input, the common visibility probability map is predicted pixel by pixel and binarized to obtain the common visibility mask. Then, based on the total number of effective pixels in the mask, the optimal unidirectional matching direction is adaptively selected to eliminate invalid search space at the source.

[0188] (3) Coarse to fine matching module: Under the guidance of the mask and direction output by CVM, the three stages of coarse matching, intermediate refinement and convolutional final matching (CFM) are completed in sequence. The unified homography transformation matrix and common visible mask are updated in a linked iterative manner, and the final output is a dense high-precision matching point pair covering the entire map range and the final spatial transformation matrix.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for matching optical-SAR images with significant geometric differences, characterized in that, include: Obtain the optical-SAR image pairs to be matched; Optical-SAR image pairs are input into a hybrid feature extraction network composed of MaRS and RepVGG to generate multi-scale high-resolution feature maps and multi-scale effective feature maps for each modality. A co-visible region extraction network based on an attention mechanism is used to process multi-scale high-resolution feature maps of each modality, output the first co-visible region mask of each modality, and determine the unidirectional matching direction of optical-SAR image pairs; Coarse matching is performed based on the first common visible region mask of each modality and the effective feature map at multiple scales to obtain the coarse matching result of optical-SAR image pairs; wherein, the coarse matching result is a pair of image feature points; Based on the coarse matching results and the multi-scale effective feature maps of each modality, fine matching is performed to obtain the fine matching results of optical-SAR image pairs, including the sub-pixel level coordinates of the feature point pairs of the in-point image, the unified homography transformation matrix, and the second common visible region mask of each modality. When the number of inlier image feature point pairs in the fine matching result exceeds the preset number, convolutional final matching is performed based on the fine matching result and the multi-scale high-resolution feature maps of each modality, and the final matching result of the optical-SAR image pair is output, including dense high-precision matching point pairs covering the entire image and the final homography transformation matrix.

2. The optical-SAR image matching method for significant geometric differences as described in claim 1, characterized in that, The hybrid feature extraction network includes a first feature extraction branch for processing optical modal images and a second feature extraction branch for processing SAR modal images; wherein each feature extraction branch processes the input image of the target modality and extracts features using MaRS. and Feature maps at resolution scales; extracted using RepVGG , and Multi-scale high-resolution feature maps of three target modalities with higher resolution scales; the feature maps extracted by MaRS are enhanced with long-connection information using the Transformer attention mechanism to output multi-scale effective feature maps of the target modalities.

3. The optical-SAR image matching method for significant geometric differences as described in claim 1, characterized in that, The process involves using an attention-based co-visible region extraction network to process multi-scale effective feature maps for each modality. The steps include: The effective feature maps of any modality at multiple scales are adaptively concatenated along the channel dimension and then convolved to obtain the fused feature map of that modality. A gated fusion mechanism using self-attention and cross-attention Transformer layers is adopted to generate a co-visibility probability map of each modality based on the obtained fusion feature maps of each modality. Based on a preset confidence threshold, the first common visible region mask for each mode is generated using the common visible probability map of each mode, and the unidirectional matching direction of the optical-SAR image pair is determined according to the number of first common visible region masks in the first common visible region mask of each mode.

4. The optical-SAR image matching method for significant geometric differences as described in claim 1, characterized in that, The gated fusion mechanism of the self-attention and cross-attention Transformer layers takes the flattened sequence of the fusion feature maps of each modality as input, generates self-attention features of each modality through several Transformer layers equipped with self-attention, and then performs cross-modal interaction of self-attention features through cross-attention Transformer layers to generate cross-attention features of each modality. Finally, for any modality, the cross-attention features and self-attention features of that modality are dynamically integrated using a learning gating mechanism, and the co-visible region probability map of that modality is generated by a probability prediction head.

5. The optical-SAR image matching method for significant geometric differences as described in claim 1, characterized in that, The steps for obtaining the coarse matching result of the optical-SAR image pair include: Based on the first common visible region mask of each modality, respectively for the same modality The effective feature maps at the resolution scale are subjected to non-common visible region occlusion processing to obtain the mask features of each modality; The global similarity matrix between the feature sets of the two modal masks is calculated by dot product, and the global similarity matrix is ​​scaled by temperature hyperparameter. A one-way Softmax function is applied to the scaled global similarity matrix in the one-way matching direction to generate a matching confidence matrix. Image feature point pairs exceeding the preset matching confidence threshold are retained. For many-to-one mapping conflicts, only the image feature point pairs with the highest matching confidence are retained. The retained image feature point pairs are used as the coarse matching results of optical-SAR image pairs.

6. The optical-SAR image matching method for significant geometric differences as described in claim 1, characterized in that, The fine matching includes: Using the image feature point pairs output by global coarse matching as seed points, from each modality Local features are extracted from the effective feature map at the resolution scale according to a window of a preset size; the local correlation volume between the local features of the image feature point pair is calculated, a probability distribution is generated by the spatial Softmax operation, and the sub-pixel level coordinates of the image feature point pair are obtained by calculating the mathematical expectation of the probability distribution; RANSAC is applied to filter out subpixel-level coordinate outliers, retain inlier image feature point pairs, and use the subpixel-level coordinates of the inlier image feature point pairs to calculate the transformation from the source image to the target image. Combined with the unidirectional matching direction, a unified homography transformation matrix is ​​calculated. The image boundaries of optical-SAR image pairs are deformed using a unified homography transformation matrix, and the first common visible region mask of each mode is dynamically updated to obtain the second common visible region mask of each mode. The subpixel-level coordinates of the feature point pairs in the output image, the homography transformation matrix, and the second common visible region mask for each modality are used as fine matching results.

7. The optical-SAR image matching method for significant geometric differences as described in claim 1, characterized in that, The final matching of the convolution includes the following steps: At 1 / 8 and 1 / 4 resolution scales, initial points are randomly sampled from the effective pixels of the source image using a second common-visible mask, and the initial points are projected onto the target image using a unified homography transformation matrix to obtain initial matching coordinates; The projected reference features, geometrically deformed target features, coordinate displacement embeddings, and local correlation volumes are obtained. These four features are then concatenated along the channel dimension, and the resulting joint features are input into the thinner network to output the residual coordinate offset. The matching coordinates are updated based on the residual coordinate offset, and the second homography transformation matrix is ​​calculated using the RANSAC algorithm. The second common-visible mask is then dynamically updated using the second homography transformation matrix. At a 1 / 2 resolution scale, the high-resolution feature maps at the 1 / 2 resolution scale and the 1 / 4 resolution scale are fused, and the pixel-level high-precision coordinate offset is output through the final convolutional thinner. The high-precision coordinate offset is added to the initial matching coordinates, and the final output is a dense high-precision matching point pair covering the entire image. The second homography transformation is output as the final homography transformation matrix.

8. An optical-SAR image matching system for significant geometric differences, characterized in that, include: The input module is used to acquire optical-SAR image pairs to be matched; The hybrid feature extraction module is used to input optical-SAR image pairs into a hybrid feature extraction network composed of MaRS and RepVGG to generate multi-scale high-resolution feature maps and multi-scale effective feature maps for each modality. The common visible region extraction module is used to process multi-scale high-resolution feature maps of each modality using an attention-based common visible region extraction network, output the first common visible region mask of each modality, and determine the unidirectional matching direction of optical-SAR image pairs. The coarse matching module is used to perform coarse matching based on the first common visible region mask of each modality and the effective feature map at multiple scales to obtain the coarse matching result of optical-SAR image pairs; wherein, the coarse matching result is a pair of image feature points; The fine matching module is used to perform fine matching based on the coarse matching results and the multi-scale effective feature maps of each modality to obtain the fine matching results of optical-SAR image pairs, including the sub-pixel level coordinates of the feature point pairs of the in-point image, the unified homography transformation matrix, and the second common visible region mask of each modality. The convolutional final matching module is used to perform convolutional final matching based on the fine matching results and the multi-scale high-resolution feature maps of each modality when the number of inlier image feature point pairs in the fine matching results exceeds a preset number. It outputs the final matching results of optical-SAR image pairs, including dense high-precision matching point pairs covering the entire image and the final homography transformation matrix.

9. An electronic device, characterized in that, The method includes a memory and a processor, the memory storing program instructions that are executed by the processor, the processor invoking the program instructions to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the method described in any one of claims 1 to 7.

Images