An image feature matching method based on cross-directional dual-stream interaction and spatial enhancement

Abstract

This invention discloses an image feature matching method based on cross-directional dual-stream interaction and spatial enhancement, comprising: extracting coarse-scale feature maps and fine-scale feature maps from the input source image and target image, respectively; concatenating the two coarse-scale feature maps and extracting features along multiple scanning directions to generate a sequence tensor; updating the state-space model of the first part of the sequence tensor; concatenating and fusing the second part of the sequence tensor with the inverse features of the updated sequence to generate a compensation sequence; updating the state-space model of the compensation sequence; constructing a multi-directional output sequence from the two updated sequences; reconstructing a two-dimensional coarse-scale feature map based on the multi-directional output sequence, and performing coarse-to-fine matching optimization in conjunction with the fine-scale feature map to obtain the final matching result. This method can effectively achieve multi-directional structural compensation, local geometric consistency enhancement, and state update driven by multiple source dynamic terms under low computational conditions.

Description

Technical Field

[0001] This invention relates to the field of image feature matching technology, and more specifically to an image feature matching method based on cross-directional dual-stream interaction and spatial enhancement. Background Technology

[0002] 3D vision technology is being widely applied in both the virtual and real worlds. In the virtual world, 3D vision technology supports scenarios such as games, shopping, social interaction, and film and television. It helps games build immersive environments, allows consumers to view product details using AR / VR, enables users to reconstruct and share 3D scenes using their mobile phones, and provides realistic models and rendering capabilities for film and television special effects and digital characters. In the real world, 3D vision technology provides intelligent robots with environmental perception and navigation capabilities, supports LiDAR and depth estimation for autonomous driving, and improves traffic safety. It is also used for the 3D digital preservation of cultural relics and street view reconstruction, serving cultural protection, map navigation, and urban planning. To use 3D vision technology, 3D information needs to be acquired, mainly in two ways: one is to use 3D sensors directly (such as LiDAR, millimeter-wave radar, etc.), which has good results but is expensive and has limitations; the other is to recover 3D information from 2D images. Because 2D camera technology is mature, inexpensive, and adaptable to various environments, this method is being widely explored. In 3D information restoration technology, feature matching technology is crucial. Its core is to find matching points from two images of the same object from different perspectives that may have differences in lighting and scale. By using the matching points and camera imaging parameters, and through algorithms such as RANSAC and the eight-point method, the relative position of the camera and the depth information of the points can be calculated. The 2D pixels are then back-projected onto the 3D coordinate system to obtain key 3D information, which helps to complete 3D vision tasks with inexpensive 2D cameras.

[0003] Due to widespread attention, many feature matching methods have been proposed, which can be broadly classified into two categories: detector-based methods and detector-free methods.

[0004] 1. Detector-based matching method:

[0005] Detector-based feature matching, also known as sparse matching, is the earliest developed type of feature matching. Its core process includes three stages: key point detection, feature description (generating descriptors), and matching computation. The overall performance is highly dependent on the first two stages. Therefore, developing efficient, robust, and highly discriminative detection and description schemes is the core of research in this field.

[0006] Early keypoint detection relied heavily on manual design. Harris corner detection detects sharp changes in pixel grayscale within a local window, calculates the corner response function using differential operations, and determines the location through thresholding or non-maximum suppression. Forstner corner detection employs a similar approach, locating keypoints through grayscale covariance matrix and eigenvalue analysis. However, both methods struggle to handle drastic changes in angle and scale. The subsequent SIFT algorithm introduced a Gaussian difference pyramid and scale space to ensure scale invariance, and defined a principal direction to achieve rotation invariance, laying the foundation for further research, but it incurs significant computational costs. To address this, SURF uses Haar wavelets to replace gradient calculations for acceleration, FAST simplifies the process by comparing a small number of pixel grayscale values, BRIEF constructs binary descriptors using random point pair sampling and grayscale comparison, and ORB integrates and optimizes FAST and BRIEF, gaining widespread application in SLAM. However, these manually designed feature points are prone to failure in weakly textured or repetitive textured scenes; for example, ORB is completely unable to detect keypoints in SLAM scenes with smooth surfaces.

[0007] Deep learning technology offers a new approach to solving these problems. GCNv2 can still detect valid keypoints even when ORB fails, LF-Net can learn highly discriminative local features, LIFT improves robustness to illumination changes through joint training, GCN and GCNv2 achieve detection by learning the geometric structure of the feature space, SuperPoint optimizes the model using synthetic datasets and pseudo-labels, and R2D2 focuses on the repeatability and reliability of keypoints, significantly improving detection quality.

[0008] Some studies attempt to integrate the detection and description stages, with D2-Net being a typical example. SuperGlue performs both tasks simultaneously using a neural network. COTR introduces a graph neural network based on an attention mechanism, adding an update layer between feature description and matching computation. ClusterGNN first clusters descriptors, applying self-attention only within each cluster and pruning redundant message passing to accelerate the network. In terms of matching computation, classic nearest neighbor matching selects the optimal match based on similarity, while mutual nearest neighbor matching adds bidirectional verification to improve accuracy. NCN utilizes local neighborhood consistency to optimize prediction, SuperGlue introduces optimal transmission problem modeling, and NCNet and DISK use the Dual-Softmax algorithm instead of Sinkhorn, significantly reducing computational overhead. However, this detector-based method is limited by keypoint detection, struggling to obtain reliable keypoints in scenes with weak textures, large viewpoint changes, and drastic lighting changes. Local descriptors in repetitive texture regions cannot be distinguished using global information; for example, SuperGlue cannot establish effective matches in ground and shadow areas. Therefore, detector-free feature matching (also known as dense matching) has been proposed.

[0009] 2. Detectorless matching method:

[0010] Detectorless matching methods do not require keypoint detection and directly establish pixel-scale dense correspondences. For example, LoFTR can achieve high-quality matching in ground and shadow areas.

[0011] Classical detector-free methods, represented by SIFTFlow and Lucas and Horn-Schunck, describe pixel correspondences through optical flow, but their performance is inferior to contemporary detector-based methods. Deep learning, especially the introduction of Transformer technology, has driven a breakthrough in performance. Choy et al. and Schmidt et al. pioneered the use of learning-based pixel-scale descriptors, combining contrastive loss with nearest neighbor matching to obtain pixel-level matching. Subsequent methods have diverged into two categories: cost volume-based and Transformer-based.

[0012] The groundbreaking work NCNet, based on the cost volume, constructs a 4D cost volume by computing all pixel-to-pixel matches and achieves end-to-end training through 4D convolutional regularization. However, it incurs huge memory and computational costs, limiting the input image size. Subsequent methods such as Sparse NCNet use sparse convolutions to improve efficiency, DRCNet integrates multi-scale cost volumes, ANC-Net introduces an adaptive domain consistency module, GLU-Net combines adaptive multi-resolution strategies with feature pyramids, and GOCor optimizes feature-related layers to handle matching ambiguities. However, these methods rely on convolutions, have limited receptive fields, and are insufficient in capturing global information.

[0013] Transformer has become a core framework due to its excellent long-range association capture capabilities; a single attention layer is sufficient to establish cross-image associations, whereas convolution requires three layers. COTR constructs a functional matching module and combines iterative scaling techniques to obtain high-precision matches, but its feature resolution needs to be compressed due to computational limitations. LoFTR adopts a coarse-to-fine strategy, extracting multi-scale features through FPN, refining the matching results through attention updates and a Dual-Softmax algorithm, and using LinearAttention to reduce overhead. MatchFormer adds a feature scale level, OETR first estimates the overlapping parts of the images, ASpanFormer dynamically adjusts the attention window size, and ASTR uses high-confidence matching to guide low-confidence matching. These Transformer-based methods have become the mainstream.

[0014] In summary, detector-based methods offer high computational efficiency and low memory consumption, while manually designed algorithms (such as ORB) have low deployment costs, making them suitable for scenarios with high real-time requirements (such as SLAM front-ends). Deep learning-driven methods exhibit better robustness in complex scenes. However, these methods rely on the quality of keypoint detection and are prone to failure in scenarios with weak textures and large viewpoint changes. Keypoints are difficult to distinguish in repetitive texture regions, and sparse matching cannot provide dense pixel-level correspondences. Detector-free methods do not rely on keypoint detection and perform better in challenging scenarios, providing rich 3D reconstruction information. Transformer-based methods offer higher matching accuracy. However, cost-volume-based methods have high computational and memory overhead, limiting image input size; Transformer methods have stringent hardware requirements, and some methods lack accuracy in small-scale detail matching.

[0015] Detector-free matching methods include semi-dense matching and VisionMamba (VMamba for short).

[0016] Semi-dense matching is a detector-free local feature matching technique widely used in 3D computer vision tasks such as Structure for Motion (SfM), Simultaneous Localization and Mapping (SLAM), and visual localization. Its core objective is to establish pixel-level semi-dense correspondences between two images without prior keypoint detection, leveraging the global receptive field of the Transformer architecture and a coarse-to-fine matching strategy. This is particularly suitable for challenging scenarios with weak or repetitive textures. The LoFTR algorithm is a semi-dense matching algorithm that assumes geometric constraints (such as epipolar geometry) between image pairs. Through an iterative process of feature extraction, transformation, coarse matching, and fine optimization, it gradually improves matching accuracy, ultimately outputting a high-confidence semi-dense matching result. The existing literature “J. Sun, Z. Shen, Y. Wang, H. Bao, and X. Zhou. LoFTR: Detector-Free Local Feature Matching With Transformers. In Proceedings of the CVPR, pages 8922–8931, 2021. 1, 2, 6, 7, 8, 3” involves the application of semi-dense matching algorithms.

[0017] VisionMamba is a state-space model-based visual backbone network widely used in computer vision tasks such as image classification, object detection, and semantic segmentation. Its core objective is to transfer the linear computational complexity and global modeling capabilities of state-space models to the visual domain through an innovative 2D selective scanning mechanism, achieving efficient inference while capturing global dependencies in images. The existing literature "Yue Liu, Yunjie Tian, ​​Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166, 2024. 2, 3" discusses the application of the VisionMamba architecture.

[0018] However, the existing literature mentioned above has the following shortcomings:

[0019] 1. Performance Limitations of Transformer's High Complexity and Mamba's Unidirectional Modeling in Feature Matching. Traditional Transformer feature matching methods rely on O(N²) attention computation, which is insufficient to support the matching requirements of high-resolution feature maps. The high training and inference costs make them unsuitable for real-time or resource-constrained scenarios. While Mamba offers the advantage of linear complexity, its unidirectional scanning and orientation-independent modeling mechanism still cannot effectively capture multi-directional structural relationships in two-dimensional images, limiting its global modeling capabilities in high-resolution matching.

[0020] 2. Insufficient 2D structural representation due to multi-directional independent scanning. Existing multi-directional sequential feature matching methods (such as 4-way / 8-way) typically split a 2D image into multiple independent 1D sequences, perform independent scanning in each direction first, and then aggregate them at the end. Since the directions do not share information at all in the early stages, this model makes it difficult to explicitly establish structural dependencies between directions, and directional coupling can only be passively learned in the later stages. At the same time, due to the influence of Mamba's unidirectional causality, the sequences in each direction are prone to directional bias, resulting in insufficient 2D spatial correlation representation, and causing increased mismatch rates and missing structural information in weak texture and repetitive texture scenes.

[0021] 3. Insufficient geometric consistency in matching due to sequence modeling dependency. Traditional Mamba feature matching methods rely entirely on sequence modeling, lacking targeted optimization of local spatial structure. They are weak in feature point geometric alignment and preservation of local detail consistency, and have poor robustness to local noise. This results in insufficient discriminability of the matching region, low detail fidelity, and difficulty in establishing stable feature correspondences in low-texture and noisy scenes.

[0022] 4. The problem of missing matching direction information caused by the single dynamics of the state equation. In the core state equation of the traditional Mamba feature matching method, the time constant dt and the multiplication gates C, B, and skip term D are only generated by the current sequence token. They can only represent the local dynamic weights in a single direction and cannot incorporate multi-directional structural associations and semantic compensation information. As a result, the state update process lacks consideration of the multi-directional feature relationships of the two-dimensional image and it is difficult to capture the matching dependence of feature points in different directions.

[0023] Therefore, how to overcome the inherent limitations of traditional Transformer and Mamba architectures in feature matching while maintaining low computational complexity and efficient inference capabilities, in order to achieve a high-precision, high-robust image feature matching method applicable to challenging scenarios such as high resolution and weak texture, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0024] In view of the above problems, the present invention proposes an image feature matching method based on cross-directional dual-stream interaction and spatial enhancement to overcome or at least partially solve the above problems.

[0025] To achieve the above objectives, the present invention adopts the following technical solution:

[0026] This invention provides an image feature matching method based on cross-directional dual-stream interaction and spatial enhancement, comprising the following steps:

[0027] S1. Extract coarse-scale feature maps and fine-scale feature maps from the input source image and target image, respectively;

[0028] S2. The coarse-scale feature maps of the source image and the target image are concatenated, and features are extracted from the concatenated feature maps in multiple scanning directions to generate a sequence tensor.

[0029] S3. Update the state-space model of the first part of the sequence tensor to obtain the first part update sequence; concatenate and fuse the inverse features of the second part of the sequence tensor with the first part update sequence, and generate a compensation sequence through gated linear transformation; update the state-space model of the compensation sequence to obtain the second part update sequence; the first part update sequence and the second part update sequence together constitute the multi-directional output sequence of the current layer.

[0030] S4. Reconstruct a two-dimensional coarse-scale feature map based on the multi-directional output sequence, and perform coarse-to-fine matching optimization in combination with the fine-scale feature map to obtain the final matching result at the sub-pixel level.

[0031] Furthermore, in S1, a ConvNeXt V2 network is used as an encoder to extract the coarse-scale feature map and the fine-scale feature map.

[0032] Furthermore, S2 specifically includes:

[0033] The coarse-scale feature maps of the source image and the target image are stitched together along the width and height dimensions, respectively, to obtain the corresponding horizontal stitched feature map and vertical stitched feature map.

[0034] Based on the horizontally spliced ​​feature map, the corresponding left-right scanning sequence and right-left scanning sequence are extracted in the left-right and right-left directions;

[0035] Based on the vertical stitching feature map, the corresponding up-down scanning sequence and down-up scanning sequence are extracted in the up-down and down-up directions;

[0036] The left-to-right scan sequence, the right-to-left scan sequence, the up-to-down scan sequence, and the down-to-up scan sequence constitute a sequence tensor with four scan directions.

[0037] Furthermore, S3 specifically includes:

[0038] For a sequence tensor containing four scanning directions, in each layer of computation, the state space model is updated independently for the sequences in two scanning directions to obtain the corresponding first update sequence and second update sequence.

[0039] For the sequences in the remaining two scanning directions, one sequence is concatenated and fused with the reverse feature of the first update sequence to obtain the first fused sequence; the other sequence is concatenated and fused with the reverse feature of the second update sequence to obtain the second fused sequence.

[0040] The first fusion sequence is passed sequentially through a linear transformation layer and an activation function layer, multiplied with learnable weights, and then residually connected with the first sequence to generate a first compensation sequence.

[0041] The second fusion sequence is passed sequentially through a linear transformation layer and an activation function layer, multiplied with learnable weights, and then residually connected with the other sequence to generate a second compensation sequence.

[0042] The state-space model is updated for the first compensation sequence and the second compensation sequence respectively to obtain the corresponding third update sequence and fourth update sequence;

[0043] The first update sequence, the second update sequence, the third update sequence, and the fourth update sequence together constitute the multi-directional output sequence of the current layer.

[0044] Furthermore, the state equation corresponding to the state-space model is represented in the form of the update sequence at each time step as follows:

[0045]

[0046]

[0047] in, Indicates the first k The sequence at time step i The hidden state; Indicates the first k The sequence at time step i Input; Indicates input The corresponding output; Representing the matrix of a continuous system A Discretized state transition matrix; This represents the discretized input projection matrix; Indicates the first k The sequence at time step i The projection matrix; Indicates the first k The sequence at time step i The jump link parameter.

[0048] Furthermore, the state equation corresponding to the state-space model is expressed in the form of updating the compensation sequence at each time step as follows:

[0049]

[0050]

[0051] in, Indicates the first l Each compensation sequence at time step i The hidden state; Indicates the first l Each compensation sequence at time stepi Input; Indicates input The corresponding output; Representing the matrix of a continuous system A Discretized state transition matrix; This represents the discretized input projection matrix; Indicates the first l Each compensation sequence at time step i The jump link parameter.

[0052] Furthermore, when performing a state-space model update on the sequence in the sequence tensor, the parameters of the dynamic terms involved in its state equations are generated as follows:

[0053] Time step i input signal Taken from sequence tensors;

[0054] Discretized state transition matrix Discretized input projection matrix Time step i projection matrix and time step i jump link parameters Each of these is generated by the corresponding sequence in the sequence tensor through an independent linear projection layer.

[0055] The time step parameter on which the discretization process depends Obtained through a multi-source fusion mechanism, and represented as:

[0056]

[0057] in, Indicates the first k The sequence at time step i The step size is represented by linear projection of the sequence features; This indicates that the first element in the sequence tensor will be... k The enhancement stride is represented by a sequence input spatial enhancement module, which is enhanced by local convolution and then linearly projected. and All represent learnable fusion coefficients; This represents the activation function.

[0058] Furthermore, when performing a state-space model update on the first or second compensation sequence, the parameters of the dynamic terms involved in its state equations are generated in the following manner:

[0059] Time step i input signal Taken from sequence tensors;

[0060] Discretized state transition matrix Discretized input projection matrix Time step i projection matrix and time step i jump link parameters Each of these is generated by the corresponding compensation sequence through an independent linear projection layer;

[0061] The time step parameter on which the discretization process depends Obtained through a multi-source fusion mechanism, and represented as:

[0062]

[0063] in, This indicates the compensation sequence at time step i The step size is represented by linear projection of the sequence features; This represents the enhancement stride obtained by linear projection after local convolution enhancement of the sequence input space enhancement module in the sequence tensor; and All represent learnable fusion coefficients; This represents the activation function.

[0064] Furthermore, the spatial enhancement module includes a sequence-feature transformation layer, a 3×3 depthwise separable convolutional layer, a first 1×1 convolutional channel compression layer, an activation function, and a second 1×1 convolutional channel expansion layer;

[0065] In the sequence-feature transformation layer, the input sequence is converted into a two-dimensional feature map by a tensor rearrangement operator;

[0066] The two-dimensional feature map is sequentially passed through the 3×3 depth separable convolutional layer, the first 1×1 convolutional channel compression layer, the activation function, and the second 1×1 convolutional channel expansion layer to obtain enhanced features.

[0067] Furthermore, S4 specifically includes:

[0068] The multi-directional output sequence is sequentially subjected to dimension reshaping, feature reconstruction, and inverse scanning operations to reconstruct the multi-directional output sequence into a two-dimensional coarse-scale feature map of the source image and the target image;

[0069] Calculate the coarse feature similarity matrix from the two-dimensional coarse-scale feature maps of the source and target images, and perform Softmax normalization on the rows and columns of the coarse feature similarity matrix to obtain the matching confidence matrix.

[0070] Elements in the matching confidence matrix that are higher than a preset threshold are selected as candidate matching pairs, and the candidate matching pairs are further filtered based on the nearest neighbor criterion to obtain coarse matching pairs.

[0071] The coarse matching pairs are mapped to the corresponding positions in the fine-scale feature maps, and local feature windows centered on the mapping points are cropped on the fine-scale feature maps of the source and target images, respectively.

[0072] The correlation between the center point features of the local feature window of the source image and all features within the local feature window of the target image is calculated, and the fine similarity matrix of features is obtained through the inner product.

[0073] The rows and columns of the feature fine similarity matrix are subjected to Softmax normalization to obtain fine matching pairs, which serve as the final matching results at the sub-pixel level.

[0074] As can be seen from the above technical solution, compared with the prior art, the present invention discloses an image feature matching method based on cross-directional dual-stream interaction and spatial enhancement, which has the following beneficial effects:

[0075] This invention, based on the linear efficiency of Mamba, introduces a cross-directional dual-stream interaction mechanism. By establishing explicit information compensation and structural fusion pathways between multi-directional sequences, the model can maintain linear complexity while possessing wide-area structural modeling capabilities close to those of Transformer. This significantly improves the matching efficiency and stability of high-resolution feature maps, making it more suitable for low-latency scenarios such as real-time SLAM and UAV vision.

[0076] This invention proposes a "cross-directional dual-flow interaction mechanism," fundamentally breaking through the aforementioned "end-to-end aggregation" modeling approach. In each layer, dual-flow interaction is performed on adjacent directional sequences. Through operations such as cross-directional alignment (flip), information compensation and structural coupling of multi-directional features are achieved at an early stage, followed by sequence scanning on the fused features. Compared to the traditional "scan first, then aggregate" approach, this invention's "fusion first, then scan, then aggregate" design explicitly constructs stable multi-directional structural associations while maintaining linear complexity, significantly enhancing the expressive power of two-dimensional spatial relationships, effectively reducing directional bias, and improving the accuracy and robustness of feature matching.

[0077] This invention designs a "spatial enhancement module" to construct a "global + local" dual-scale feature matching structure. This module employs a design combining depthwise separable convolution and lightweight projection to preprocess features before Mamba sequence processing, achieving local smoothing, enhanced geometric consistency, and improved robustness to local noise. This provides more stable foundational features for cross-directional interactions, freeing the model from a single reliance on sequence modeling and forming a fusion system of "stable local spatial priors + efficient global sequence inference." This significantly improves the discriminability and detail fidelity of matching regions, enabling accurate feature correspondences even in low-texture, noisy scenes.

[0078] This invention introduces a "fusion sequence projection" mechanism to the state equation structure for the first time, fundamentally changing the internal dynamics of the state-space model. Specifically, it utilizes the cross-directional fused sequence (which already contains multi-directional compensation information) to project the dynamic term parameters, while retaining the original sequence as model input. This allows the calculation of the dynamic term in the state equation to no longer be limited to a single-directional input, but to reflect the joint structural dynamics of multiple directions. From a mathematical perspective, it provides a new form for multi-directional feature matching reasoning, allowing the state update process to naturally incorporate directional relationships and semantic compensation information, thereby improving the matching accuracy of multi-directional feature points. Attached Figure Description

[0079] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0080] Figure 1 This is a schematic diagram of the image feature matching method based on cross-directional dual-stream interaction and spatial enhancement provided in an embodiment of the present invention.

[0081] Figure 2(a) is a schematic diagram of the results of the ELOFIR method provided in the embodiment of the present invention while maintaining the same high matching accuracy (98.5%).

[0082] Figure 2(b) is a schematic diagram of the results of the present invention provided in an embodiment of the present invention, while maintaining the same high matching accuracy (98.5%).

[0083] Figure 3(a) is a schematic diagram of the results of the ELOFIR method in a complex scenario provided in the embodiment of the present invention.

[0084] Figure 3(b) is a schematic diagram of the results of the present invention in a complex scenario provided in an embodiment of the present invention.

[0085] Figure 4(a) is a schematic diagram of the results of the ELOFIR method provided in the embodiment of the present invention, while maintaining the same high matching accuracy (100.0%).

[0086] Figure 4(b) is a schematic diagram of the results of the present invention provided in an embodiment of the present invention, under the premise of maintaining the same high matching accuracy (100.0%). Detailed Implementation

[0087] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0088] This invention discloses an image feature matching method based on cross-directional two-stream interaction and spatial enhancement, such as... Figure 1 As shown, it includes the following steps:

[0089] S1. Extract coarse-scale feature maps and fine-scale feature maps from the input source image and target image, respectively;

[0090] S2. The coarse-scale feature maps of the source image and the target image are concatenated, and features are extracted from the concatenated feature maps in multiple scanning directions to generate a sequence tensor.

[0091] S3. Update the state-space model of the first part of the sequence tensor to obtain the first part of the updated sequence; concatenate and fuse the inverse features of the second part of the sequence tensor with the first part of the updated sequence, and generate a compensation sequence through a gated linear transformation; update the state-space model of the compensation sequence to obtain the second part of the updated sequence; the first part of the updated sequence and the second part of the updated sequence together constitute the multi-directional output sequence of the current layer.

[0092] S4. Reconstruct a two-dimensional coarse-scale feature map based on the multi-directional output sequence, and perform coarse-to-fine matching optimization in combination with the fine-scale feature map to obtain the final matching result at the sub-pixel level.

[0093] Next, each of the above steps will be explained in detail.

[0094] In step S1 above, a ConvNeXt V2 network is used as the encoder. Encoder ), from the input source image I A and target image I B Extract coarse-scale and fine-scale feature maps. Represented as:

[0095]

[0096] in, Indicates from the source image I A The coarse-scale feature map extracted from it, and ; Indicates from the source image I A The fine-scale feature map extracted from it, and ; Indicates from the target image I B The coarse-scale feature map extracted from it, and ; Indicates from the target image I B The fine-scale feature map extracted from it, and ; H c , W c and C 1 represents the height, width, and number of channels of the coarse-scale feature map, respectively; H f , W f and C 2 represents the height, width, and number of channels of the fine-scale feature map, respectively.

[0097] In step S2 above, a joint multi-direction scanning feature sequence is constructed: coarse-scale feature maps of the source and target images are concatenated, and features are extracted along multiple scanning directions based on the concatenated feature maps to generate a sequence tensor; specifically including:

[0098] S21. Calculate the coarse-scale feature map of the source image. Coarse-scale feature map of the target image By concatenating the features along both the width (horizontal) and height (vertical) dimensions, the corresponding horizontally concatenated feature maps are obtained. and vertically stitched feature maps ;

[0099] The current approach uses coarse-scale feature maps of the source image. Coarse-scale feature map of the target image Directly stitching along the spatial dimension before scanning; in other embodiments, this can be adjusted to stitching along the sequence dimension: first, separately... and Scan individually in the same direction to obtain the source image. I A and target imageI B Each has its own multi-directional sequence, and then the sequences of corresponding directions of the two are concatenated along the sequence dimension (e.g., ...). left→right sequence and (Left-to-right sequence concatenation) strengthens the feature association between corresponding directions of the two images; 1×1 convolution or linear projection layers are used to... and Perform feature fusion (such as element-wise addition, convolution after concatenating channel dimensions) to generate a fused joint feature map.

[0100] S22. Extract sequences based on the spliced ​​feature maps according to multiple scanning directions:

[0101] 1) Based on the horizontally stitched feature map, the corresponding left-right scanning sequences are extracted in the left→right and right→left directions. and right → left scan sequence ;

[0102] 2) Based on the vertically stitched feature map, the corresponding up-down scan sequences are obtained by extracting them in the up-down and down-up directions. and bottom → top scan sequence .

[0103] In this embodiment of the invention, four basic directions (left→right, right→left, up→down, down→up) are used to cover the main structural information of the two-dimensional image. For some resource-limited or scene-structured scenes (such as those dominated by horizontal textures), only two mutually inverse directions can be used (e.g., only the horizontal directions left→right and right→left, or the vertical directions up→down and down→up), reducing the computational load while ensuring directional complementarity. In scenes with high matching accuracy requirements for complex geometric structures (such as diagonal textures and irregular contours), it can be extended to more complex paths such as 8-directional scanning (adding diagonal directions such as left-up-right-down and right-up-left-down), spiral scanning (scanning layer by layer from the center outward to strengthen the feature association of the central region), and checkerboard scanning (scanning in the order of grid blocks to improve the structural consistency between local blocks). As long as the two-dimensional feature map can be transformed into a multi-directional sequence with directional discrimination, it can meet the subsequent modeling requirements.

[0104] S23. A multi-directional sequence tensor composed of the above four directional sequences. X :

[0105]

[0106] In step S3 above, a cross-direction bi-stream interaction mechanism is constructed: the state-space model of the first part of the sequence tensor is updated to obtain the first part of the updated sequence; the inverse features of the second part of the sequence tensor and the first part of the updated sequence are concatenated and fused, and a compensation sequence is generated through a gated linear transformation; the state-space model of the compensation sequence is updated to obtain the second part of the updated sequence; the first part of the updated sequence and the second part of the updated sequence together constitute the multi-directional output sequence of the current layer; this step will be explained in detail below:

[0107] (1) Step S3 specifically includes:

[0108] S31. For a sequence tensor containing four scanning directions, in each layer of computation, the state space model (SSM) is updated independently for the sequences in two scanning directions to obtain the corresponding first update sequence and second update sequence, which are used as the first part of the update sequence.

[0109] S32. For the sequences in the remaining two scanning directions, one sequence is concatenated and fused with the reverse feature of the first update sequence to obtain the first fused sequence; the other sequence is concatenated and fused with the reverse feature of the second update sequence to obtain the second fused sequence.

[0110] S33. Perform gated linear transformation on the fused sequence:

[0111] 1) The first fusion sequence is passed through a linear transformation layer and an activation function layer in sequence, and after being multiplied with learnable weights, it is residually connected with one of the sequences mentioned above to generate the first compensation sequence;

[0112] 2) Pass the second fusion sequence through a linear transformation layer and an activation function layer in sequence, multiply it with the learnable weights, and then perform a residual connection with the other sequence mentioned above to generate the second compensation sequence;

[0113] S34. Perform state-space model (SSM) updates on the first compensation sequence and the second compensation sequence respectively to obtain the corresponding third update sequence and fourth update sequence, which are used as the second part of the update sequence;

[0114] S35. The first update sequence, the second update sequence, the third update sequence and the fourth update sequence together constitute the multi-directional output sequence of the current layer.

[0115] (2) Using sequence tensors For example, let's explain S31-S35 above:

[0116] 1) For sequence tensors In each layer of computation, the right-to-left scan sequence is first selected. and up-down scan sequence Independently perform state-space model (SSM) updates to obtain the corresponding first update sequence. Second update sequence ; indicates as: = SSM ( ); SSM ( );

[0117] 2) Scan the sequence from bottom to top With the first update sequence The reverse features are spliced ​​and fused to obtain the first fused sequence. ; Scan the sequence from left to right With the second update sequence The reverse features are spliced ​​and fused to obtain the second fused sequence. ; indicates as: ; ;in, flip This indicates a space flip operation; cat Indicates a splicing operation;

[0118] 3) The first fusion sequence Passing through the linear transformation layer in sequence W and activation function layer σ and with learnable weights γ After multiplication, it is combined with the bottom-to-top scan sequence. Perform residual connection to generate the first compensation sequence. ; indicates ; the second fusion sequence Passing through the linear transformation layer in sequence W and activation function layer σ and with learnable weights γ After multiplication, it is compared with the left-to-right scan sequence. Perform residual connection to generate a second compensation sequence. ; indicates ;

[0119] 4) For the first compensation sequence Second compensation sequence Perform state-space model (SSM) updates separately to obtain the corresponding third update sequence. and the fourth update sequence ; indicates ; ;

[0120] 5) First update sequence Second update sequence Third update sequence and the fourth update sequence Together they constitute the multi-directional output sequence of the current layer. Y ={ , , , }

[0121] (3) Explanation of the relevant content of the above state-space model:

[0122] 1) When independently updating the state-space model for the sequences in two scanning directions during each layer of computation, the parameters of the dynamic terms involved in the state equations are generated as follows:

[0123] ① Input signal of state equation The original direction sequence in the sequence tensor; represented as ;in, Indicates time step i The first in the sequence tensor k A sequence;

[0124] ② Discretized state transition matrix Discretized input projection matrix Time step i projection matrix and time step i jump link parameters All are composed of sequence tensors X The corresponding sequences are generated through independent linear projection layers; the right-to-left scan sequence is used. and up-down scan sequence Let's take updating the state-space model as an example to illustrate:

[0125] Right-to-left scan sequence and up-down scan sequence The parameters of the dynamic terms that generate the state equations are the sole source of characteristics. , , and ; indicates as:

[0126]

[0127]

[0128]

[0129]

[0130] in, Indicates time step i The next k A sequence; Proj (·) denotes the projection function; Indicates the first k The sequence at time step i Dynamic parameters under; Indicates the first k The sequence at time step i The influence coefficient of the state; Indicates the first k The sequence at time step i The projection matrix below; Indicates the first k The sequence at time step i The jump link parameters below;

[0131] ③ Traditional Mamba's state updates depend on a single source. This invention addresses the issue of local dependency bias by proposing a multi-source approach. dt The integration mechanism will and The dynamic items from the two sources are unified and integrated, and represented as follows:

[0132]

[0133] in, Indicates the first k The sequence at time step i The step size is represented by linear projection of the sequence features; This indicates that the first element in the sequence tensor will be... k Each input spatial enhancement module, after local convolution enhancement, performs linear projection to obtain an enhancement stride representation, which is used to enhance the geometric features of edge regions, smooth regions, and curved surface regions. dt It can better reflect local structural changes; and All represent learnable fusion coefficients; This represents the activation function.

[0134] 2) When performing state-space model updates on the first and second compensation sequences, the parameters of the dynamic terms involved in the state equations are generated as follows:

[0135] ① The aforementioned dynamic parameters carry complementary information in multiple directions, enabling state updates to be guided by multi-directional structural constraints; simultaneously, input signals are applied to the state equations. The original direction sequence is preserved; represented as ;in, ;

[0136] ② Discretized state transition matrix Discretized input projection matrix Time step i projection matrix and time step i jump link parameters Each is generated from its corresponding compensation sequence through an independent linear projection layer; specifically:

[0137] With the first compensation sequence Second compensation sequence The parameters of the dynamic terms that generate the state equations are the sole source of characteristics. , , and ; indicates as:

[0138]

[0139]

[0140]

[0141]

[0142] in, Indicates time step i The next l One compensation sequence; Proj (·) denotes the projection function; Indicates the first l Each compensation sequence at time step i Dynamic parameters under; Indicates the first l Each compensation sequence at time step i The influence coefficient of the state; Indicates the first l Each compensation sequence at time step i The projection matrix below; Indicates the first l Each compensation sequence at time step i The jump link parameters below;

[0143] ③ Traditional Mamba's state updates depend on a single source. This invention addresses the issue of local dependency bias by proposing a multi-source approach. dt The integration mechanism will and The dynamic items from the two sources are unified and integrated, and represented as follows:

[0144]

[0145] in, This indicates the compensation sequence at time step i The step size is represented by linear projection of the sequence features; This represents the enhancement stride, obtained by linear projection after local convolution enhancement of the sequence input space enhancement module in the sequence tensor. It is used to enhance the geometric features of edge regions, smooth regions, and curved regions. dt It better reflects local structural changes; continuing with the above case, here... l =1 and k =3 corresponds to l =2 and k =0 corresponds to. and All represent learnable fusion coefficients; This represents the activation function.

[0146] 3) The aforementioned spatial enhancement module includes a sequence-feature transformation layer, a 3×3 depthwise separable convolutional layer, a first 1×1 convolutional channel compression layer, an activation function, and a second 1×1 convolutional channel expansion layer; the input to the spatial enhancement module is a feature map, which can be denoted as... ,in, For entities, B For the sample size, C 1 represents the number of channels. H C For feature map height, W C The width of the feature map; the sequence tensor X Left-to-right scan sequence Right → Left Scan Sequence Up-down scan sequence and bottom → top scan sequence Record ,in, L Let the dimension of the sequence in each direction be denoted; at this point, the tensor rearrangement operator in the sequence-feature transformation layer can be used. The sequence is converted into a two-dimensional feature map, represented as:

[0147]

[0148] in, This can be achieved using reshape / permute functions, etc. Then, the obtained 2D feature map is sequentially passed through a 3×3 depthwise separable convolutional layer, a first 1×1 convolutional layer, an activation function, and a second 1×1 convolutional layer to obtain enhanced features. , is represented as:

[0149]

[0150] in, This represents a 3×3 depthwise separable convolution; This represents a 1×1 pointwise convolution with channel compression. This represents a 1×1 pointwise convolution with channel expansion; This represents the activation function. Subsequent sequences need to be in reverse order. Will Restored to a sequence and used to generate enhanced step size representations .

[0151] 4) The state equation corresponding to the state-space model is updated in two parts at each time step. One part is used to update the sequence in the sequence tensor, expressed as:

[0152]

[0153]

[0154] in, Indicates the first k The sequence at time step i The hidden state; Indicates the first k The sequence at time step i Input; Indicates input The corresponding output; Representing the matrix of a continuous system A The discretized state transition matrix, and , , This represents the effective time interval used for discretization, and the state evolution is jointly controlled by multi-directional information. Let represent the discretized input projection matrix, and ; Indicates the first k The sequence at time step i The projection matrix; Indicates the first k The sequence at time step i The jump link parameter.

[0155] A portion is used to update the compensation sequence, represented as:

[0156]

[0157]

[0158] in, Indicates the first l Each compensation sequence at time step i The hidden state; Indicates the first l Each compensation sequence at time step i Input; Indicates input The corresponding output; Representing the matrix of a continuous system A The discretized state transition matrix, and , , This represents the effective time interval used for discretization, and the state evolution is jointly controlled by multi-directional information. Let represent the discretized input projection matrix, and ; Indicates the first l Each compensation sequence at time step i The projection matrix; Indicates the first l Each compensation sequence at time step i The jump link parameter.

[0159] In step S4 above, a two-dimensional coarse-scale feature map is reconstructed based on the multi-directional output sequence, and a coarse-to-fine matching optimization is performed in conjunction with the fine-scale feature map to obtain a sub-pixel-level final matching result. This includes:

[0160] S41, Multi-directional output sequence Y ={ , , , The process sequentially performs reshape, feature reconstruction, and merge operations. By deserializing, the four-directional information is fused back into a two-dimensional feature map, ultimately yielding a reconstructed two-dimensional coarse-scale feature map of the source image. and the two-dimensional coarse-scale feature map of the target image , is represented as: , , , ;

[0161] S42, Two-dimensional coarse-scale feature map of the source image and the two-dimensional coarse-scale feature map of the target image Establish initial matching relationships and calculate the coarse similarity matrix of features. S c The formula is as follows:

[0162]

[0163] in, Represents the feature points in the two-dimensional coarse-scale feature map of the source image. Index in; Represents the feature points in the two-dimensional coarse-scale feature map of the target image. Index in; This represents the inner product operation. A dual-softmax strategy is employed; subsequently, the coarse similarity matrix of features is... S c After performing Softmax normalization on the rows and columns, we get:

[0164]

[0165]

[0166] in, and The matching confidence matrix represents the matching of two points; Represents the coarse similarity matrix of features S c The rows are subjected to Softmax normalization. Represents the coarse similarity matrix of features S c The columns are subjected to Softmax normalization.

[0167] S43. Select the matches with confidence scores higher than the preset threshold from the matching confidence matrix. The elements are selected as candidate matching pairs, and the candidate matching pairs are further filtered based on the nearest neighbor (MNN) criterion to obtain coarse matching pairs. M c :

[0168]

[0169] in, Indicates feature points at Index in; Indicates feature points at Index in; r This represents a dummy variable used to iterate through all possible matches in the corresponding set;

[0170] S44. Perform coarse matching pairs M c Fine-scale feature map of the source image and fine-scale feature maps of the target image Local window optimization is performed to improve matching accuracy to the sub-pixel level; specifically: the coarse matching pairs... M c Mapping to fine-scale feature maps and The corresponding positions, and respectively in the fine-scale feature maps of the source image. and fine-scale feature maps of the target image The top cropping is a local feature window centered on the mapping point, and the cropping size can be set to 5×5;

[0171] S45. Calculate the correlation between the center point features of the local feature window of the source image and all features within the local feature window of the target image, and obtain the fine feature similarity matrix through the inner product. S f ;

[0172] S46. Perform Softmax normalization on the rows and columns of the feature fine similarity matrix to obtain fine matching pairs. M f This serves as the final matching result at the sub-pixel level; fine-grained matching pairs M f Represented as:

[0173]

[0174] in, Represents the fine similarity matrix S f The rows are subjected to Softmax normalization. Represents the fine similarity matrix S f The columns are subjected to Softmax normalization.

[0175] To further demonstrate the effectiveness of the image feature matching method based on cross-directional dual-stream interaction and spatial enhancement provided in the above embodiments of the present invention, a series of experiments were conducted. The experimental process and results are described below.

[0176] The image feature matching method based on cross-directional dual-stream interaction and spatial enhancement provided in this invention is implemented using the PyTorch framework. In the feature extraction stage, we employ the first two stages of ConvNeXt V2-N, a module containing 0.65M (millions) parameters. The channel dimension of the coarse features... C 1=256, channel dimension of fine features C 2=64; the resolution of coarse features is 1 / 8 (i.e., 1 / 8 of the original image resolution), and the resolution of fine features is 1 / 2 (i.e., 1 / 2 of the original image resolution). Temperature parameter. Set to 0.1, coarse matching threshold Set it to 0.2.

[0177] The model was trained on the MegaDepth dataset for 30 epochs with a batch size of 2 and the AdamW optimizer. During training, images were scaled and padded to 832×832 pixels. The initial learning rate was set to 0.0002, and a cosine decay learning rate scheduling strategy was used, along with one epoch of linear warm-up. All network training and evaluation experiments were performed on a single NVIDIA 3090 GPU.

[0178] We evaluate the pose estimation performance of the matcher on the MegaDepth dataset. Test images are scaled and padded to a uniform size of 832×832, consistent with the training image size. We use the LO-RANSAC algorithm to estimate the essential matrix for all methods—this algorithm is more robust than the basic RANSAC (vanilla RANSAC) and improves the reliability of the evaluation results.

[0179] In Table 1 below, we report the cumulative area under the curve (AUC) of pose error at three thresholds (5°, 10°, 20°). Among all the semi-dense and sparse matchers evaluated, our model has the best pose estimation results, only inferior to the dense matchers DKM and RoMa, which are designed with accuracy as the core objective.

[0180] Table 1: Cumulative area under the curve for pose error at three thresholds (5°, 10°, 20°)

[0181]

[0182] The methods listed in Table 1 above represent different technical approaches in the field of image feature matching, among which:

[0183] 1) Sparse methods (such as XFeat, SP+SG, SP+LG, DeDoDeB, DeDoDeG) mainly detect and match a small number of key points (such as corner points or salient areas), emphasizing positioning accuracy and efficiency;

[0184] 2) Dense methods (such as DKM and RoMa) match each pixel or dense grid in the image, aiming for high coverage but with high computational cost;

[0185] 3) Semi-dense methods (such as XFeat) XFeat (RCM, LoFTR, MatchFormer, ASpanFormer, ELoFTR) falls between these two approaches, employing attention mechanisms or local-global fusion strategies to maintain high matching density while balancing efficiency and robustness. LoFTR and its improved versions (such as ELoFTR, MatchFormer, and ASpanFormer) utilize the Transformer architecture to achieve fine-grained matching without explicit keypoint detection, while XFeat... This is a denser extension of XFeat. These methods collectively drive the advancement of tasks such as visual localization, 3D reconstruction, and motion estimation.

[0186] In addition, we provide additional qualitative comparison results between the model provided in this invention and ELoFTR. ELoFTR is an efficient detector-free feature matching method proposed by the original LoFTR team at ECCV 2024. By introducing linear attention and a memory-optimized coarse-to-fine architecture, it significantly improves inference speed and reduces memory consumption while maintaining high matching accuracy, achieving faster, lighter, and deployable Transformer-based image matching.

[0187] The indicators used in the qualitative results comparison include:

[0188] Matches: The total number of valid matching pairs in the final output of the algorithm. Precision@1e-4: The proportion of all output matching pairs that satisfy the condition that the reprojection error is less than (usually in normalized coordinates or meters), measuring the accuracy or reliability of the matching.

[0189] R_errs: The angular deviation between the estimated camera (or object) rotation and the actual rotation, used to assess the accuracy of orientation in pose estimation.

[0190] t_errs: The deviation between the estimated translation vector and the actual translation: a measure of the accuracy of the position estimate.

[0191] runtime: The time taken for the algorithm to complete the entire matching and pose estimation process;

[0192] Figures 2(a) and 2(b) show the results of the ELOFIR method and the present invention, respectively, while maintaining the same high matching accuracy (98.5%). Compared with ELOFIR, the method provided in this embodiment of the invention significantly improves computational efficiency while maintaining the same high matching accuracy (98.5%)—the running time is reduced from 152.4 ms to 82.9 ms, a speedup of nearly 46%. This makes it more suitable for real-time applications.

[0193] Figures 3(a) and 3(b) show the results of the ELOFIR method and the present invention, respectively, in complex scenes. Clearly, the method provided in this embodiment exhibits significant advantages: although the method on the left has a smaller number of matches (196 pairs), its accuracy is only 41.8%, and numerous mismatches (red lines) severely interfere with geometric consistency, resulting in rotation and translation errors as high as 2.7° and 3.6°, respectively. In contrast, the method in this paper increases the total number of matches to 263 pairs while significantly improving the matching accuracy to 71.1% (187 / 263), effectively suppressing noise interference, and achieving stronger structural consistency through a more robust feature association strategy. More importantly, the runtime is reduced from 147.3 ms to 65.2 ms, nearly doubling the efficiency, demonstrating its efficient, stable, and scalable matching capabilities even under complex lighting, texture loss, and occlusion conditions.

[0194] Figures 4(a) and 4(b) are schematic diagrams of the results obtained by the ELOFIR method and the present invention, respectively, while maintaining the same high matching accuracy (100.0%). Compared with ELOFIR, the method provided in this embodiment of the invention significantly improves computational efficiency while maintaining extremely high matching accuracy (100.0%)—the running time is reduced from 153.8 ms to 85.2 ms, a speedup of nearly 45%, while achieving better geometric consistency. Even in scenarios with complex lighting and texture variations, this method can still stably output high-confidence correspondences, demonstrating stronger robustness and practicality.

[0195] Our model consistently delivers robust matching results with shorter runtime and lower pose estimation error.

[0196] In summary, the image feature matching method based on cross-directional dual-stream interaction and spatial enhancement provided by this invention has the following advantages:

[0197] (1) Advantages compared to LoFTR (feature matching based on Transformer):

[0198] Compared to LoFTR, which relies on Transformer self-attention, this invention offers superior computational and memory efficiency and stronger engineering adaptability in high-resolution dense matching scenarios. LoFTR's computational / memory complexity increases quadratically with the number of tokens, leading to a surge in resource consumption under high-resolution or dense matching conditions, making efficient deployment difficult. In contrast, this invention employs O(N) linear complexity Mamba state space sequence modeling, combined with cross-directional dual-stream interaction, eliminating the need to construct a large-scale attention matrix. This significantly reduces resource overhead at the same resolution and matching density, making it more suitable for engineering applications involving large-size images and massive feature points. Furthermore, its linear complexity structure maintains good inference speed and memory control, making it more suitable for latency-sensitive or resource-constrained deployment scenarios such as real-time localization, SLAM, and 3D reconstruction. Meanwhile, the present invention has superior directional modeling and structure perception capabilities. LoFTR implicitly blends directional structures and cannot distinguish multi-directional features of two-dimensional images. In contrast, the present invention explicitly achieves information compensation and fusion in each direction through multi-directional serialization and cross-directional dual-stream interaction mechanism, and drives the dynamics of the state equation with cross-directional fusion sequence, so that the state update naturally includes directional relationship and structural compensation information, resulting in more robust matching performance in scenarios with large viewpoint changes, rotation and other strong geometric transformations. In addition, in view of the shortcomings of LoFTR, which lacks a dedicated local geometric consistency enhancement module and focuses on global correlation modeling, the present invention introduces the SpatialEnhancer module before multi-directional scanning. It enhances the geometric consistency of local structure and detailed regions through depthwise separable convolution and 1×1 channel reconstruction, and then performs global / multi-directional modeling to form a "global + local" dual-scale design. This provides a more stable matching response for regions with blurred contours and weak texture boundaries, which is more in line with the core requirements of geometric matching.

[0199] (2) Advantages compared to VMamba (a Mamba vision model based on multi-directional scanning):

[0200] Compared to VMamba, a general-purpose visual backbone, this invention achieves substantial innovation and performance leap in image feature matching scenarios: VMamba only expands two-dimensional features independently along multiple directions, lacks information compensation in different directions, the state equation parameters still depend on a single-directional sequence, the dynamics are essentially "one-dimensional unidirectional" expansion, and it focuses on macroscopic direction and long-range dependency processing without dedicated local geometric enhancement design. As a general-purpose model, it does not closely meet the core requirements of feature matching such as dual-image alignment and multi-view geometric consistency. It is only used for transfer purposes and requires increasing the network width / depth to improve expressiveness, resulting in an increase in the number of parameters and latency. This invention upgrades from "multi-directional independent modeling" to "cross-directional interaction + state dynamics reconstruction." It achieves information compensation and fusion in each direction through a cross-directional dual-stream interaction mechanism, and uses the cross-directional fusion sequence as the sole source of the dynamic terms of the state equation dt / B / C / D, introducing multi-directional joint structural dynamics from a mathematical perspective. Simultaneously, a Spatial Enhancer module is introduced before multi-directional scanning. It obtains local geometric information through depthwise 3×3 convolution and achieves channel reconstruction and noise reduction through 1×1 convolution, before feeding it into cross-directional Mamba modeling, forming a dual optimization of "local geometric stability + global structural enhancement." Its multi-directional scanning, cross-directional interaction, and local enhancement designs are all customized for image feature matching / mapping / registration scenarios, optimized around matching stability and reconstructability. Moreover, through structural innovation, it improves expressiveness and matching accuracy while maintaining controllable model size and complexity, without blindly stacking parameters. Its performance potential, applicability, engineering deployment, and real-time application adaptability in geometric tasks are significantly better than VMamba.

[0201] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0202] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image feature matching method based on cross-directional dual-flow interaction and spatial enhancement, characterized in that, The method comprises the following steps: S1, extracting coarse scale feature maps and fine scale feature maps from input source images and target images respectively; S2, splicing the coarse scale feature maps of the source images and the target images, and based on the spliced feature maps, performing feature extraction in multiple scanning directions to generate a sequence tensor; S3, updating the first part of the sequence in the state space model to obtain the first part of the updated sequence; splicing and fusing the second part of the sequence with the reverse features of the first part of the updated sequence, and generating a compensation sequence through a gated linear transformation; updating the state space model of the compensation sequence to obtain the second part of the updated sequence; the first part of the updated sequence and the second part of the updated sequence jointly constitute the multi-direction output sequence of the current layer; S4, reconstructing the two-dimensional coarse scale feature map based on the multi-direction output sequence, and performing coarse-to-fine matching optimization combined with the fine scale feature map to obtain the final matching result at the sub-pixel level.

2. The image feature matching method based on cross-directional dual-flow interaction and spatial enhancement of claim 1, wherein, In the S1, a ConvNeXt V2 network is used as an encoder to extract the coarse scale feature maps and the fine scale feature maps.

3. The image feature matching method based on cross-directional dual-flow interaction and spatial enhancement of claim 1, wherein, The S2 specifically comprises: Splicing the coarse scale feature maps of the source images and the target images along the width dimension and the height dimension respectively to obtain the corresponding horizontal splicing feature map and the vertical splicing feature map; Based on the horizontal splicing feature map, the left→right and right→left directions are extracted to obtain the corresponding left→right scanning sequence and right→left scanning sequence; Based on the vertical splicing feature map, the up→down and down→up directions are extracted to obtain the corresponding up→down scanning sequence and down→up scanning sequence; The left→right scanning sequence, the right→left scanning sequence, the up→down scanning sequence and the down→up scanning sequence constitute a sequence tensor containing four scanning directions.

4. The image feature matching method based on cross-directional dual-flow interaction and spatial enhancement of claim 3, wherein, The S3 specifically comprises: For the sequence tensor containing four scanning directions, in each layer calculation, the state space model update is independently performed on the sequences of two scanning directions to obtain the corresponding first updated sequence and the second updated sequence; For the sequences of the remaining two scanning directions, one of the sequences is spliced and fused with the reverse features of the first updated sequence to obtain a first fused sequence; the other sequence is spliced and fused with the reverse features of the second updated sequence to obtain a second fused sequence; The first fused sequence is sequentially passed through a linear transformation layer and an activation function layer, multiplied by a learnable weight, and then residual connected with the one sequence to generate a first compensation sequence; The second fused sequence is sequentially passed through a linear transformation layer and an activation function layer, multiplied by a learnable weight, and then residual connected with the other sequence to generate a second compensation sequence; The state space model update is performed on the first compensation sequence and the second compensation sequence respectively to obtain the corresponding third updated sequence and the fourth updated sequence; The first updated sequence, the second updated sequence, the third updated sequence and the fourth updated sequence jointly constitute the multi-direction output sequence of the current layer.

5. The image feature matching method based on cross-directional dual-flow interaction and spatial enhancement of claim 4, wherein, The state equation corresponding to the state space model is expressed in the form of each time step update sequence as: wherein, denotes the hidden state of the k th sequence at time step i ; denotes the input of the k th sequence at time step i ; denotes the input corresponding output; denotes the continuous system matrix A discretized state transition matrix; denotes the discretized input projection matrix; denotes the projection matrix of the k th sequence at time step i ; denotes the jump link parameter of the k th sequence at time step i .

6. The image feature matching method based on cross-directional dual-flow interaction and spatial enhancement of claim 4, wherein, The state equation corresponding to the state space model is updated in the form of a compensation sequence at each time step as follows: wherein, denotes the hidden state of the l th compensation sequence at time step i ; denotes the input of the l th compensation sequence at time step i ; denotes the input corresponding output; denotes the continuous system matrix A discretized state transition matrix; denotes the discretized input projection matrix; denotes the jump link parameter of the l th compensation sequence at time step i .

7. The image feature matching method based on cross-directional dual-flow interaction and spatial enhancement of claim 5, wherein, When performing state space model updating on the sequence in the sequence tensor, the dynamic term parameters involved in the state equation are generated in the following manner: time step i input signal taken from a sequence tensor; Discretized state transition matrix Discretized input projection matrix Time step i projection matrix and time step i jump link parameters Each of these is generated by the corresponding sequence in the sequence tensor through an independent linear projection layer. The time step parameter on which the discretization process relies Obtained by a multi-source fusion mechanism, expressed as: wherein, represents the step length representation obtained by linearly projecting the sequence feature of the k th sequence at the time step i ; represents the enhanced step length representation obtained by linearly projecting the sequence feature of the k th sequence input into the spatial enhancement module and then enhanced by local convolution; and both represent learnable fusion coefficients; represents an activation function.

8. The image feature matching method based on cross-directional dual-flow interaction and spatial enhancement of claim 6, wherein, When performing state space model updating on the first compensation sequence or the second compensation sequence, the dynamic term parameters involved in the state equation are generated in the following manner: time step i input signal taken from a sequence tensor; discretized state transition matrix , discretized input projection matrix , projection matrix i of time step , and hop link parameter i of time step , are generated by independent linear projection layers through corresponding compensation sequences; The time step parameter on which the discretization process relies Obtained by a multi-source fusion mechanism, expressed as: wherein, represents a step representation obtained by linearly projecting sequence features of the compensation sequence at time step i represents an enhanced step representation obtained by linearly projecting the sequence input space enhanced module in the sequence tensor after local convolution enhancement; and both represent learnable fusion coefficients; represents an activation function.​ 9. The image feature matching method based on cross-directional dual-flow interaction and spatial enhancement of claim 7 or claim 8, wherein, The spatial enhancement module includes a sequence-feature conversion layer, a 3x3 depth separable convolution layer, a first 1x1 convolution channel compression layer, an activation function, and a second 1x1 convolution channel expansion layer; In the sequence-feature conversion layer, the input sequence is converted into a two-dimensional feature map by a tensor rearrangement operator; After sequentially passing the two-dimensional feature map through the 3x3 depth separable convolution layer, the first 1x1 convolution channel compression layer, the activation function, and the second 1x1 convolution channel expansion layer, the enhanced feature is obtained.

10. The image feature matching method based on cross-directional dual-flow interaction and spatial enhancement of claim 1, wherein, The S4 specifically includes: Performing dimension reshaping, feature reconstruction, and inverse scanning operations on the multi-direction output sequence in sequence to reconstruct the multi-direction output sequence into two-dimensional coarse-scale feature maps of the source image and the target image; Calculating a feature coarse similarity matrix for the two-dimensional coarse-scale feature maps of the source image and the target image, and performing Softmax normalization processing on the rows and columns of the feature coarse similarity matrix to obtain a matching confidence matrix; Selecting elements in the matching confidence matrix that are higher than a preset threshold as candidate matching pairs, and performing secondary screening on the candidate matching pairs based on the mutual nearest neighbor criterion to obtain coarse matching pairs; Mapping the coarse matching pairs to corresponding positions of the fine-scale feature maps, and respectively cropping local feature windows centered on the mapping points on the fine-scale feature maps of the source image and the target image; Calculating the correlation between the center point feature of the local feature window of the source image and all features within the local feature window of the target image, and obtaining a feature fine similarity matrix through inner product; Performing Softmax normalization processing on the rows and columns of the feature fine similarity matrix to obtain fine matching pairs as the final matching results at the sub-pixel level.

Images