Image matching method and device based on state space modeling and sparse correlation

Abstract

The application belongs to the technical field of image registration, and discloses an image matching method and device based on state space modeling and sparse correlation, which extracts feature maps of a first image and a second image to be registered; determines a candidate feature point set based on L2 norm of feature points in the feature map under a third resolution; determines a candidate feature point set based on L2 norm of feature points in the feature map under the third resolution; generates a first feature point pair composed of a feature point in the first feature map and a feature point in the second feature map based on local correlation; performs local window correlation on the feature map under the first resolution based on the first feature point pair, and obtains a second feature point pair; determines a matching result of the first image and the second image based on the second feature point pair; the application introduces L2 norm as discriminative and saliency prior, utilizes the phenomenon that L2 norm of features in an image is amplified, and can significantly accelerate the image registration process while maintaining accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image registration, and particularly relates to an image matching method and device based on state space modeling and sparse correlation. BACKGROUND

[0002] Image registration is a fundamental pillar of 3D computer vision, supporting key tasks such as Structure from Motion (SfM), Novel View Synthesis (NVS), Simultaneous Localization and Mapping (SLAM), and Visual Localization. Historically, image matching has experienced the development of two dominant paradigms. Manually designed feature-based methods rely on descriptors such as SIFT and ORB for keypoint detection and description, but perform poorly in challenging scenarios involving extreme viewpoint changes, textureless regions, and repetitive patterns. The deep learning revolution subsequently shifted the field towards data-driven feature detection learning methods, but the decoupling of processing stages led to increased model complexity and decreased recall.

[0003] To overcome these limitations, detector-free methods emerged by directly modeling dense feature correlation to bypass explicit keypoint detection. However, their generalization ability is constrained by the feature extraction network, as the lack of cross-view feature interaction faces a fundamental trade-off between localization accuracy and feature discriminability. Recent Transformer-based architectures address this issue by modeling cross-view and cross-scale contextual interactions, achieving state-of-the-art accuracy. Subsequent optimizations further refine the accuracy, efficiency, and robustness dimensions of these mechanisms.

[0004] Despite these advances, the focus on efficiency optimization has been limited compared to efforts to improve accuracy. Reviewing contemporary frameworks, it can be found that feature matching relies on full correlation computation, whose memory consumption grows quadratically with spatial resolution. Moreover, the quadratic complexity of Transformers remains a bottleneck for high-resolution matching, prompting the exploration of efficient alternatives. Furthermore, empirical studies show that matching accuracy strongly depends on high input resolution, with computational latency and memory requirements exhibiting a highly increasing pattern. These efficiency bottlenecks, particularly in feature extraction and global correlation operations, are serious limiting factors for image matching. SUMMARY

[0005] The purpose of the present application is to provide an image matching method and device based on state space modeling and sparse correlation, which optimizes the global correlation method, significantly reduces the amount of computation, and reduces memory consumption while ensuring matching accuracy.

[0006] The present application adopts the following technical solution: an image matching method based on state space modeling and sparse correlation, comprising the following steps:

[0007] extracting features of a first image and a second image to be registered respectively; the feature maps include feature maps at different resolutions;

[0008] determining a candidate feature point set based on L2 norm of feature points in the feature map at the third resolution ; determining a candidate feature point set based on L2 norm of feature points in the feature map at the third resolution ;

[0009] generating a first feature point pair composed of a feature point in and a feature point in based on local correlation;

[0010] performing local window correlation on the feature map at the first resolution based on the first feature point pair, to obtain a second feature point pair; wherein the feature map at the first resolution is extracted from the first image and the second image , and the first resolution is greater than the third resolution;

[0011] determining a matching result of the first image and the second image based on the second feature point pair.

[0012] The present application has the beneficial effect that: the present application introduces L2 norm as discriminative and saliency prior, which takes advantage of the phenomenon that L2 norm of intra-image distributed features is amplified, and this mechanism reduces the computational load by selectively identifying geometric salient regions, while maintaining accuracy, which can significantly speed up the image registration process. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a flowchart of the image matching method based on state space modeling and sparse correlation of the embodiment of the present application;

[0014] Figure 2 is a structural diagram of the feature extraction network in the embodiment of the present application;

[0015] Figure 3 is a structural diagram of the SS2D block and the SS2D layer in the embodiment of the present application;

[0016] Figure 4 is a diagram of the recursive coordinate refinement process in the embodiment of the present application;

[0017] Figure 5 is a diagram of the qualitative comparison on the MegaDepth dataset in the embodiment of the present application;​

[0018] Figure 6 A comparison diagram of peak GPU memory consumption of different input resolutions in the verification example of the present application;

[0019] Figure 7 A comparison diagram of inference time measurement of different input resolutions in the verification example of the present application;

[0020] Figure 8 A comparison diagram of matching accuracy under different input resolutions in the verification example of the present application;

[0021] Figure 9 A raw distribution diagram based on feature mean in the verification example of the present application;

[0022] Figure 10 A raw distribution diagram based on feature variance in the verification example of the present application;

[0023] Figure 11 A raw distribution diagram based on feature L2 norm in the verification example of the present application;

[0024] Figure 12 A significant region identified by mean threshold in the verification example of the present application;

[0025] Figure 13 A significant region identified by variance threshold in the verification example of the present application;

[0026] Figure 14 A significant region identified by L2 norm threshold in the verification example of the present application;

[0027] Figure 15 A comparison diagram of accuracy and efficiency trade-off of the method of the present application and existing methods on the MegaDepth dataset. DETAILED DESCRIPTION

[0028] The present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0029] Early visual-based matching methods rely on hand-designed descriptors such as SIFT, SURF, and ORB, which detect keypoints and compute descriptors through pre-defined gradient or intensity patterns. These methods perform well in terms of efficiency and interpretability, but due to the reliance on hand-designed heuristic methods, they perform poorly in handling textureless regions or extreme view changes.

[0030] The rise of deep learning has shifted the paradigm to data-driven detector-based methods. Methods like D2Net, R2D2, and SuperPoint employ CNNs to jointly learn keypoint detection and descriptor extraction, while SuperGlue introduces graph neural networks to model matching as a partial assignment problem. While robustness has improved, these methods inherit a segmented pipeline where detection, descriptor, and matching are optimized separately. This decoupling introduces inefficiency, leading to the loss of valid correspondences due to imperfect keypoint localization. Furthermore, the segmented stages result in information loss and suboptimal performance in challenging scenarios, such as low-texture scenes or fuzzy matching.

[0031] To overcome these limitations, detector-free methods have emerged, bypassing explicit keypoint detection by directly modeling dense feature correlations. NC-Net pioneered this direction by constructing a 4D correlation tensor and applying 4D convolutions to enforce neighborhood consensus, while DRC-Net further refined this approach through multi-resolution feature fusion. Recent work has integrated the Transformer architecture to enhance context modeling, with COTR utilizing cross-attention for cross-view correspondence search. Subsequent innovations, such as MatchFormer and ASpanFormer, have optimized the attention mechanism for cross-scale fusion, while Efficient LoFTR reduces computational overhead through label pruning.

[0032] This invention discloses an image matching method based on state-space modeling and sparse correlation, comprising the following steps: extracting the first image to be registered. Second image Feature maps; feature maps include feature maps at different resolutions; based on Feature map at third resolution The L2 norm of the feature points generates a set of candidate feature points. ;based on Feature map at third resolution The L2 norm of the feature points generates a set of candidate feature points. ; Generate based on local correlation Feature points and The first feature point pair is formed by the feature points in the first image; a second feature point pair is obtained by performing local window correlation on the feature map at the first resolution based on the first feature point pair; wherein, the feature map at the first resolution is obtained through the first image. Second image The extracted data shows that the first resolution is greater than the third resolution; the first image is determined based on the second feature point pairs. Second image Subpixel displacement map.

[0033] This invention introduces the L2 norm as a discriminative and saliency prior, taking advantage of the phenomenon of norm growth in the distribution features within an image. By selecting geometrically salient regions, it utilizes the phenomenon that the L2 norm of the distribution features within an image is amplified. This mechanism reduces the amount of computation by selectively identifying geometrically salient regions, and can significantly accelerate the image matching process while maintaining accuracy.

[0034] It should be noted that the first image and the second image in this invention refer to different images, the first resolution, the second resolution and the third resolution represent image resolutions of different sizes, and the first feature point pair and the second feature point pair represent feature point pairs obtained at different stages.

[0035] like Figure 1 As shown, the method of the present invention consists of three parts: (1) Feature extraction adopts Conv-Mamba backbone network and feature pyramid network (FPN) to fuse cross-view and cross-scale context modeling; (2) Two-stage coarse-to-fine sparse correlation, firstly establishes an initial correspondence at the 1 / 8 scale, and then performs high-resolution correlation at the 1 / 2 scale; (3) Performs recursive coordinate refinement operation on the original scale feature window through a lightweight recursive network to predict the sub-pixel offset between the reference and target feature windows.

[0036] Given image pairs ( , The method of this invention establishes semi-dense correspondences through three stages. First, multi-scale features are extracted using a feature extraction network, while simultaneously fusing cross-scale and cross-view contextual information. Then, a two-stage association process is performed: at 1 / 8 resolution (… A rough match is performed under ( ). This represents the stage number of the feature extraction network, and the initial correspondence is generated by sparsely associating the filtered features. (Contains several pairs of first feature points), then at 1 / 2 resolution ( Fine-grained matching is performed under this condition, and refined correspondences are generated through local window association. (This includes several pairs of second feature points). Finally, a lightweight recursive refinement unit iteratively predicts sub-pixel offsets between the original scale feature windows.

[0037] This invention develops a lightweight feature extraction network, which extracts features from the first image using the constructed feature extraction network. Second image The feature extraction network includes a three-stage Conv-Mamba backbone network and a feature pyramid network. The three-stage Conv-Mamba backbone network is used to extract feature maps at different resolutions. The three-stage Conv-Mamba backbone network combined with the FPN is used for cross-view and cross-scale feature interaction with linear complexity. The present application also proposes a lightweight recursive refinement unit (RRU) containing only 816K parameters, which is used for iterative spatial offset optimization.

[0038] As shown in Figure 2 The feature extraction network adopts a three-stage hierarchical architecture for multi-scale feature extraction. The first stage uses a single ConvNeXt block with a 7*7 kernel to process the input grayscale image, generating a feature map at the first resolution, preserving high-frequency details.

[0039] The second stage is connected in parallel with two ConvNeXt blocks sharing parameters; the output information of the two ConvNeXt blocks is input into the same SS2D block; the two output information of the SS2D block is input into the third stage of the three-stage Conv-Mamba backbone network, finally obtaining the feature map at the second resolution; wherein the SS2D block realizes cross-view context fusion through four-direction scanning.

[0040] The third stage combines two ConvNeXt blocks sharing parameters, an SS2D block, and two aggregation modules (each containing an Inception-NeXt and an SS2D block), generating a feature map at the third resolution, i.e., high-level features. That is, the third stage includes two ConvNeXt blocks sharing parameters; the output information of the two ConvNeXt blocks is input into an SS2D block; the two output information of the SS2D block is input into two aggregation modules connected in turn; wherein the aggregation module includes two Inception-NeXt blocks and an SS2D block, the two Inception-NeXt blocks are used to receive the output information of the previous SS2D block, and the output of the two Inception-NeXt blocks is used as the input of the next SS2D block; the output information of the aggregation module is used as the input information of the feature pyramid network; wherein the input information of the first and second stages of the three-stage Conv-Mamba backbone network is also used as the input information of the feature pyramid network.

[0041] It should be noted that in the aggregation module, the features are first down-sampled by 4 times through convolution with a step of 4, then processed through the Inception-NeXt block with parallel branches, which include 3*3, 1*7, and 7*1 depth separable convolutions, followed by an SS2D block. The processed features are up-sampled by 4 times through bilinear interpolation. This design effectively balances the receptive field expansion and computational efficiency.

[0042] As Figure 3 (a) is a structural schematic diagram of the SS2D block in the application, as Figure 3 (b) is a structural schematic diagram of the SS2D layer in the SS2D block, and the cross-view fusion in the SS2D layer is operated through a four-stage process: first, the paired features are connected along the width dimension, then four independent directional scans are applied, followed by four parallel Mamba blocks. Finally, the segmented and processed features are averaged to obtain enhanced representations, so that the final features contain contextual information.

[0043] The cross-scale fusion is realized through a feature pyramid network, which combines features of different resolution levels using lateral connections. The fusion process at each level can be represented as:

[0044] (1)

[0045] wherein, represents the final output of the first stage of the feature extraction network; represents the direct output of the first stage of the three-stage Conv-Mamba backbone network; represents the final output of the first stage of the feature extraction network, and it should be noted that when is equal to 3, is considered as a zero tensor and is ignored in the calculation. Therefore, the feature pyramid network can combine deep semantic information and shallow spatial details to achieve robust multi-scale representation learning. It should be noted that the loss function of the feature extraction network in the application is composed of the first feature point pair matching loss, the second feature point pair matching loss and the coordinate refinement loss.

[0046] Matching loss: Given the true coarse correspondence

[0047] and the fine correspondence , the negative log-likelihood loss on these two score matrices is calculated:

[0048] (2)

[0049] (3)

[0050] (4)

[0051] wherein, represents the coarse matching loss, represents the fine matching loss.

[0052] ​​Coordinate refinement loss: for recursive coordinate refinement, use the reference coordinates (i.e. the feature point coordinates) paired true correspondence (i.e. the other feature point coordinate in the feature point pair) Supervise all intermediate predictions with an exponentially decaying L2 loss:

[0053] (4)

[0054] where, is the decay factor (empirically set to 0.8), denotes the L2 norm of a vector, t denotes the iteration number of RRU, and T denotes the total iteration number of RRU.

[0055] Finally, the above three losses are combined according to different weights, and the weights are determined by cross-validation to balance the scale difference between matching probability and coordinate offset.

[0056] Regarding the two-stage sparse correlation, the present application proposes a feature filtering strategy, which uses the L2 norm of the feature as a discriminative and saliency prior. This mechanism reduces the computational complexity by selectively identifying geometrically salient regions, significantly accelerating the inference process while maintaining accuracy, reducing the computational complexity in the registration process, and reducing the memory usage of the network, which is superior to the full correlation calculation used in other methods.

[0057] It is observed that the inner distribution features gradually develop larger norms during the training process, thereby forming a distinguishable separation from the outlier features. This phenomenon occurs because the classification output becomes more polarized (harder probability distribution) under softmax normalization, indirectly amplifying the L2 norm of the inner distribution features. Some detector-based matching methods associate feature saliency with entropy and variance, demonstrating the effectiveness of these metrics as keypoint detection indicators.

[0058] Extending this mechanism to detector-free feature matching, the present application infers that effective and salient feature matching naturally exhibits the growth of norms and variances during the training process.

[0059] The present application method only implements sparse correlation and double Softmax on features with high norms. and are determined by the threshold :

[0060] (5)

[0061] In the coarse stage, local correlation is performed on the feature map at 1 / 8 scale and Internally computed, i.e. generated based on local correlations Feature points and The first feature point pair, composed of feature points in the data, includes: selection Feature points and The feature points in the data form a first initial feature point pair; the local correlation of the first initial feature point pair is calculated, and the local correlation is double Softmax normalized to obtain a first score; the first initial feature point pair corresponding to the first score that is greater than the coarse confidence threshold is selected as the first feature point pair.

[0062] Specifically, calculating the local correlation of the first initial feature point pair includes:

[0063] (6)

[0064] in, Indicates by The first in Feature points and The first in Feature points The local correlation of the initial feature point pairs is represented by d, which represents the feature dimension of the feature point and is set to 192 in this embodiment of the invention.

[0065] Double Softmax normalization produces a coarse score matrix; the method for determining the first score is as follows:

[0066] (7)

[0067] in, This represents the first score of the first initial feature point pair. Indicates For dimension pairs Perform softmax operation. Indicates For dimension pairs Perform a softmax operation, where the superscript 'c' indicates a coarse correspondence.

[0068] Finally, the first initial feature point pair corresponding to the first score greater than the coarse confidence threshold is selected as the first feature point pair, thus obtaining the initial coarse correspondence. .

[0069] In the fine-grained stage, the initial coarse correspondence is... For feature point pairs, local window correlation operations are performed on the 1 / 2 scale feature map centered on the feature point locations. (Feature map at the first resolution) and Select the feature point corresponding to the feature point of the first feature point pair; generate a feature window of a predetermined size on the feature map at the first resolution with the selected feature point as the center, then:

[0070] (8)

[0071] (9)

[0072] in, Represents the feature map at the first resolution. The selected feature points correspond to the feature points in the first feature point pair. Indicates The central feature window, Represents the feature map at the first resolution. The selected feature points correspond to the feature points in the first feature point pair. Indicates The central feature window.

[0073] Next, a second initial feature point pair is generated based on the feature points in the feature window; then, the local correlation of the second initial feature point pair is calculated, and double Softmax normalization is applied to the local correlation to obtain the second score. Similarly, the local correlation of the second initial feature point pair is calculated as follows:

[0074] (10)

[0075] in, express Offset the feature point corresponding to p in its feature window and Offset the feature point corresponding to q in its feature window The local correlation of the second initial feature point pair is represented by a feature dimension of 48.

[0076] Next, the second score for the second initial feature point pair is calculated:

[0077] (11)

[0078] in, This represents the second score of the second initial feature point pair. Indicates a pair with dimension p Perform softmax operation. Indicates pairs with q as the dimension Perform a softmax operation, where the superscript f indicates fine-grained correspondence.

[0079] Finally, the second initial feature point pair corresponding to the second score that is greater than the fine confidence threshold is selected as the second feature point pair.

[0080] After the two-stage sparse correlation, a recursive coordinate refinement process is required. Existing feature matching frameworks usually employ expectation-based coordinate regression in the fine matching stage. Although this method is computationally efficient, it has two inherent limitations: (1) the expectation operation assumes a unimodal probability distribution within the local window, making it difficult to handle matching ambiguity caused by repetitive textures or noise; (2) direct regression lacks explicit modeling of spatial consistency between adjacent pixels, which may amplify the localization error.

[0081] To overcome these limitations, drawing inspiration from the recursive architecture in optical flow estimation, sub-pixel thinning is reformulated as an iterative coordinate optimization task. For example... Figure 4 As shown, this lightweight design (816K parameters) can achieve high-precision positioning in just 3-5 iterations while maintaining real-time performance.

[0082] For each second feature point pair obtained through two-stage matching, in this embodiment, it is assumed that the second feature point pair is... First, according to and Generate original scale feature maps W respectively A and W B The preferred method is to use interpolation.

[0083] RRU first starts from the original scale feature map W A and W B A k×k feature window is extracted around the second feature point pair. The core refinement process updates the hidden state through efficient deep convolutions and multilayer perceptrons (MLPs), thereby computing offsets and context updates. These states are then decoded into subpixel displacement maps. and confidence weight Assuming W remains constant... A The positions of the feature points in the image remain unchanged, so W is updated iteratively. B Feature points in Let t represent the number of iterations. The coordinates are then adjusted using a weighted average of local displacements. Where z represents the ordinal number of the feature points in a k×k feature window, and h t-1 h represents the parameters after t-1 iterations. t This represents the parameters after t iterations. After T iterations, sub-pixel accuracy is achieved in the final match. This design achieves adaptive accuracy and speed control by dynamically adjusting the number of iterations during inference.

[0084] In addition, verification experiments were conducted to verify the effectiveness of the method of the present invention.

[0085] Training uses the MegaDepth dataset, which contains 130,000 internet photos from 196 scenes reconstructed using COLMAP. Following the SuperGlue experimental protocol, the present invention uses 38,300 image pairs from 368 sub-scenes, using pre-computed depth maps and camera intrinsics. All input images are resized to 1024x1024 resolution while preserving the original aspect ratio.

[0086] The training configuration uses the AdamW optimizer (weight decay: 0.1, initial learning rate: 1x10 -3 ), coarse matching loss weight a = 0.25, fine matching loss weight b = 0.2, coordinate refinement loss weight l = 1.0, and a batch size of 6 distributed across 6 RTX 4090 GPUs (24 GB VRAM each). The network converges in 19 epochs, taking 21 hours. The trained model is evaluated on other datasets without fine-tuning, following the training protocol of Efficient LoFTR to demonstrate generalization capabilities.

[0087] Indoor relative pose estimation is evaluated on ScanNet using 1500 challenging monocular image pairs at 640x480 resolution, preserving the aspect ratio. Outdoor validation uses 1500 MegaDepth dataset image pairs from the “Sacré Coeur” and “St. Peter's Square” scenes, resized to 1200x1200 with edge padding.

[0088] Pose accuracy metrics measure the AUC of the maximum angular pose error (i.e., matching point pixel position error) at 5° / 10° / 20° thresholds, with fundamental matrix extraction from predicted correspondences using basic RANSAC. The present invention is compared to sparse methods (SuperPoint + SuperGlue / LightGlue / MambaGlue) and semi-dense methods (DRCNet, LoFTR, QuadTree, ASpanFormer, MatchFormer, Efficient LoFTR, JamMa) using a single RTX 4090 GPU for timing. For SuperGlue, LightGlue, and MambaGlue, their evaluation protocols are followed, with up to 2048 keypoints extracted using SuperPoint.

[0089] As shown in Table 1 below, our method (i.e., the method of the present application) achieves state-of-the-art performance on all thresholds of MegaDepth, obtaining an AUC@5° of 57.9, outperforming EfficientLoFTR by 1.4 percentage points, using only 5.9M parameters, which is only 50.9% of the LoFTR model size, consuming 48% less memory than JamMa, and being 1.8x faster than Efficient LoFTR at high resolution. In terms of computational efficiency, the single-iteration variant (†) processes ScanNet pairs in only 23.9ms under PyTorch AMP, outperforming JamMa's 26.6ms while providing higher accuracy. Although the method of the present application has higher latency than sparse methods (LightGlue, MambaGlue) at high resolution, it provides substantially improved accuracy. Despite not reaching the highest accuracy on ScanNet, the method of the present application shows relatively good generalization capabilities across different scenes.

[0090] As shown in Table 1 below, relative pose estimation results on MegaDepth (1184x1184) and ScanNet (640x480). AUC scores at different thresholds (5° / 10° / 20°) and matching times (in milliseconds). † denotes refinement using 1 iteration. Parameter counts include only trainable weights. Feature filtering is used in both variants of the method of the present application.

[0091] Table 1

[0092]

[0093] Homography estimation experiments were evaluated on the HPatches dataset, which contains 108 sequences (56 viewpoint changes, 52 illumination changes), rescaled the shorter side to 480 pixels while preserving aspect ratio, resulting in 540 test pairs. Homography accuracy was measured by the AUC of the mean angular re-projection error at 3 / 5 / 10px thresholds, using RANSAC to estimate homographies, using up to 1000 correspondences.

[0094] Table 2

[0095]

[0096] Comparisons include sparse methods (D2Net+NN, R2D2+NN, DISK+NN, SP+SG) and semi-dense methods (NC-Net, DRC-Net, LoFTR, Efficient LoFTR, JamMa). The method of the present application (i.e., our method) achieves state-of-the-art AUC performance on all thresholds, showing statistically significant improvements over all comparison methods.

[0097] The homography estimation results on HPatches are shown in Table 2. The AUC score of the average corner re-projection error under 3 / 5 / 10 pixel threshold demonstrates that the method of the present application has higher matching accuracy.

[0098] As shown in Figure 5 the qualitative comparison on MegaDepth. The method of the present application generates denser and more accurate correspondences than Efficient LoFTR and MatchFormer. The improved matching quality directly contributes to the improvement of pose estimation accuracy as shown in Table 2.

[0099] To evaluate the scalability of the method of the present application at different resolutions, some resolution scaling tests are performed on the MegaDepth dataset. All evaluations are performed in mixed precision mode with gradient computation disabled. The horizontal axis represents the input resolution, and the vertical axis represents the measurement: Figure 6 The peak GPU memory consumption, Figure 7 The inference latency per pair, and Figure 8 The matching accuracy (AUC@5°).

[0100] As shown in Figure 6 the method of the present application achieves a 48% memory reduction compared to JamMa at 1184x1184 resolution while maintaining near-linear scaling characteristics. As shown in Figure 7 the inference speedup reaches 1.8 times that of Efficient LoFTR. As shown in Figure 8 there is a consistent accuracy advantage at high resolution.

[0101] Specifically, the resolution scaling evaluation: Figure 6 The peak GPU memory consumption comparison for different input resolutions; Figure 7 The inference time measurement comparison for different input resolutions; Figure 8 The matching accuracy comparison (AUC@5° on the MegaDepth dataset) at different input resolutions. All measurements are performed in mixed precision mode with gradient computation disabled.

[0102] The present application also conducts qualitative analysis to compare the differences between mean, variance and L2 norm as correspondence point filtering indicators. At the 1 / 8 resolution scale, the distributions of variance and L2 norm can accurately identify significant areas such as building structures. After threshold filtering (normalized value > threshold, where the threshold is actually set to 0.65, shown in red), the coverage is highly consistent with the visually salient areas. Specifically, the original distribution of features: Figure 9 The original distribution based on feature mean in the verification embodiment of the present application, Figure 10 The original distribution based on feature variance in the verification embodiment of the present application,Figure 11 Fig. 3 is a schematic diagram of the original distribution of feature L2 norms in the validation examples of the present application; the threshold version shows the feature regions: Figure 12 Fig. 4 is a schematic diagram of the salient regions identified with the mean threshold in the validation examples of the present application, Figure 13 Fig. 5 is a schematic diagram of the salient regions identified with the variance threshold in the validation examples of the present application, Figure 14 Fig. 6 is a schematic diagram of the salient regions identified with the L2 norm exceeding the normalized threshold of 0.65 (red regions). The L2 norm shows stronger discriminative power for salient features while suppressing background responses.

[0103] The quantitative analysis in Table 3 and Table 4 shows that the L2 norm filtering significantly improves the efficiency and accuracy on the MegaDepth and ScanNet datasets, outperforming the variance-based and mean-based filtering methods.

[0104] Table 3

[0105]

[0106] Table 4

[0107]

[0108] In addition, the accuracy and efficiency trade-off comparison of the semi-dense feature matcher was conducted on the MegaDepth dataset (resolution of 1184x1184). As shown in Fig. 7, the circle diameter represents the peak memory consumption of each method. The present method achieved the state-of-the-art performance in all indicators, demonstrating an excellent balance between matching accuracy and computational efficiency. Figure 15

[0109] The experimental results show that the present method outperforms the state-of-the-art (SOTA) method on indoor and outdoor datasets, with higher accuracy and efficiency performance. Notably, in the resolution scaling test, compared with JamMa, the present method achieved a 48% reduction in memory usage and a 2.4-fold inference speedup, while at 1184x1184 resolution, it outperformed Efficient LoFTR by 60% memory reduction and 1.8-fold speedup, verifying the scalability efficiency of the present method relative to the leading method.

[0110] The present application also discloses an image matching device based on state space modeling and sparse correlation, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the above-mentioned method when executing the computer program.

[0111] ​The application further discloses a computer readable storage medium, which stores a computer program.

[0112] The application further provides a computer program product, which, when running on a data storage device, enables the data storage device to perform the steps in the above-mentioned method embodiments.

[0113] The integrated unit module, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the application can realize all or part of the processes in the above-mentioned embodiment methods by a computer program to instruct relevant hardware, and the computer program can be stored in a computer readable storage medium. The computer program, when executed by a processor, can realize the steps in the above-mentioned method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, an executable file or some intermediate form. The computer readable medium at least includes any entity or device capable of carrying the computer program code to a storage device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal and a software distribution medium. For example, a U disk, a mobile hard disk, a magnetic disk or an optical disk.

[0114] In the above-mentioned embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0115] Those skilled in the art can realize that the algorithm steps of each example described in combination with the embodiments disclosed in the application can be realized by electronic hardware or a combination of electronic hardware and computer software. Whether the functions are realized by hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

Claims

1. An image matching method based on state space modeling and sparse correlation, characterized in that, The method comprises the following steps: extracting features maps of a first image and a second image to be registered; the feature maps comprising feature maps at different resolutions; based on feature maps at the third resolution l2-norm of the feature points in the feature map ; based on feature maps at the third resolution l2-norm of the feature points in the feature map ; generating a first feature point pair composed of a feature point in and a feature point in based on local correlation perform local window correlation on the feature map at the first resolution based on the first feature point pair, to obtain a second feature point pair; wherein the feature map at the first resolution is obtained through the first image and the second image extraction, the first resolution is greater than the third resolution; determining a matching result of the first image and the second image based on the second feature point pair.

2. The image matching method based on state space modeling and sparse correlation according to claim 1, wherein, generating pairs of feature points consisting of a feature point in and a feature point in based on local correlation includes: choose Feature points and The feature points in the data form the first initial feature point pair; calculating a local correlation of the first initial feature point pair and performing double Softmax normalization on the local correlation to obtain a first score; selecting the first initial feature point pair corresponding to the first score greater than a coarse confidence threshold as a first feature point pair.

3. The image matching method based on state space modeling and sparse correlation according to claim 2, wherein, calculating a local correlation of the first initial feature point pair includes: , wherein, denotes the local correlation of an initial feature point pair consisting of the th feature point in and the th feature point in , d denotes the feature dimension of the feature points. denotes the local correlation of an initial feature point pair consisting of the​ 4. The image matching method based on state space modeling and sparse correlation according to claim 3, wherein, The determination method of the first score comprises: ， wherein, denotes a first score of the first initial feature point pair, denotes a softmax operation on with dimension , denotes a softmax operation on with dimension .

5. The image matching method based on state space modeling and sparse correlation according to any one of claims 2-4, characterized in that, performing local window correlation based on the feature map at the first resolution of the first feature point pair comprises: selecting feature points corresponding to the feature points in the first feature point pair on the feature map at the first resolution; generating a feature window of a predetermined size on the feature map at the first resolution with the selected feature points as the center; generating a second initial feature point pair based on the feature points in the feature window; calculating a local correlation of the second initial feature point pair and performing double Softmax normalization on the local correlation to obtain a second score; selecting the second initial feature point pair corresponding to the second score greater than a fine confidence threshold as a second feature point pair.

6. The image matching method based on state space modeling and sparse correlation according to claim 5, wherein, extracting features of a first image and a second image to be registered, respectively, comprises: extract features of the first image and the second image respectively by using the constructed feature extraction network and second image ​ The feature extraction network comprises a three-stage Conv-Mamba backbone network and a feature pyramid network, and the three-stage Conv-Mamba backbone network is used to extract feature maps at different resolutions.

7. The image matching method based on state space modeling and sparse correlation according to claim 6, wherein, The second stage of the three-stage Conv-Mamba backbone network is connected in parallel with two ConvNeXt blocks sharing parameters; the output information of the two ConvNeXt blocks sharing parameters is input into an SS2D block; the two output information of the SS2D block is used as the input information of the third stage of the three-stage Conv-Mamba backbone network.

8. The image matching method based on state space modeling and sparse correlation according to claim 7, wherein, The third stage comprises two ConvNeXt blocks sharing parameters; the output information of the two ConvNeXt blocks sharing parameters is input into an SS2D block; the two output information of the SS2D block is input into two aggregation modules connected in sequence; The aggregation module comprises two Inception-NeXt blocks and an SS2D block, the two Inception-NeXt blocks are used to receive the output information of the previous SS2D block, and the output of the two Inception-NeXt blocks is used as the input of the next SS2D block. The output information of the aggregation module is used as the input information of the feature pyramid network; wherein the input information of the first stage and the second stage of the three-stage Conv-Mamba backbone network is also used as the input information of the feature pyramid network.

9. The image matching method based on state space modeling and sparse correlation according to any one of claims 6-8, characterized in that, The loss function of the feature extraction network comprises a first feature point pair matching loss, a second feature point pair matching loss and a coordinate refining loss.

10. An image matching apparatus based on state space modeling and sparse correlation, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, The processor executes the computer program to realize the method of any one of claims 1-9. The processor executes the computer program to realize the method of any one of claims 1-9.

Images