Dual-view feature matching method based on geometric context modulation and consensus distillation

Abstract

The invention provides a double-view feature matching method based on geometric context modulation and consensus distillation, which comprises the following steps: acquiring an initial corresponding point set between two views, determining a motion vector and a space coordinate of each corresponding point, and constructing initial feature representation and position representation; iteratively refining the feature representation and the location representation by a cascade network comprising at least one refining layer; wherein each refining layer comprises the following steps: modulating input characteristics based on a motion vector and a space coordinate; performing processing on the modulated features: generating anchor point representations from all corresponding points through a learnable aggregation operation, and interacting among the anchor point representations by utilizing a position-guided attention mechanism to distill out a global motion consensus representation; propagating the global motion consensus representation from the anchor representation to all corresponding points by using a position-guided attention mechanism, and generating refined feature representation; and on the basis of the feature representation finally output by the cascade network, predicting the inner point probability of the corresponding point and estimating the geometrical relationship between the two views.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a dual-view feature matching method based on geometric context modulation and consensus distillation. Background Technology

[0002] In computer vision, establishing accurate correspondences between two views is a fundamental and crucial task, providing essential input for a range of upstream applications such as image registration, Simultaneous Localization and Mapping (SLAM), Structure-of-Motion (SfM), and 3D reconstruction. However, due to limitations imposed by complex scene conditions such as image quality variations, lighting changes, drastic viewpoint changes, weak textures, and repetitive structures, even classic feature matching methods like SIFT often produce initial matching sets containing a large number of erroneous correspondences, i.e., outliers. These outliers severely interfere with the estimation accuracy of subsequent geometric models (such as the fundamental matrix, essential matrix, and homography matrix). Therefore, developing robust and efficient outlier removal algorithms is of significant practical importance.

[0003] Existing outlier removal algorithms can be broadly categorized into two types: traditional methods and deep learning-based methods. Traditional methods, represented by Random Sample Consensus (RANSAC) and its many variants, rely on the core idea of ​​assuming a geometric model by randomly sampling a minimum set of points and evaluating its consistency. However, when the proportion of outliers is high, the number of samples required to find a clean subset increases exponentially, resulting in low computational efficiency and a significant drop in stability and success rate. With the development of deep learning, researchers have begun to explore data-driven approaches to outlier removal. Pioneering work such as LFGC (often referred to as PointCN) first constructed the correspondence filtering problem as a point-by-point binary classification (interior / outterior) task, directly processing unordered sets of correspondence points through an end-to-end network. Despite significant progress, PointCN-style encoders typically employ a shared multilayer perceptron (MLP) to perform independent and homogeneous feature transformations on each correspondence point, ignoring the local geometric context. When there are boundary regions of significantly discontinuous depths or independently moving objects in the scene, this processing method can easily lead to the "smoothing" or homogenization of the features of points corresponding to different motion modes, thereby blurring the feature distinction between inliers and outliers, as well as between different clusters of inliers.

[0004] To capture contextual information, subsequent methods such as OANet introduced modules like self-attention or graph neural networks to model global or local relationships between corresponding points. However, these methods still essentially rely on directly learning a consensus function from all corresponding points, which may contain a lot of noise. When outliers dominate or have a complex distribution, the global attention mechanism or graph propagation process is easily contaminated or misled by noise signals, making it difficult for the network to distill a pure and consistent geometric consensus. Another approach, such as ConvMatch and DeMo, attempts to explicitly fit a continuous motion field (e.g., predicting a flow field or spatial transformation field through a neural network), hoping that the field will exhibit consistency in the interior point region. The potential drawback of this "global fitting" strategy is that it implicitly assumes the existence of a single global motion that can be well described by a smooth function. In scenarios with extremely high noise or multiple independent motions (multi-motion), the real, potentially discontinuous or sparse geometric signals are easily overwhelmed by the majority of outlier noise, causing the fitted "consensus" field to deviate significantly from the real geometry, resulting in consensus extraction failure.

[0005] Therefore, the current challenges facing outlier removal techniques can be summarized in two aspects: first, how to design an effective feature extraction mechanism that maintains the discriminative power of features at complex motion boundaries and avoids over-smoothing; second, how to design a robust architecture that can stably and reliably distill potential geometric consensus without being overwhelmed by noise when the proportion of outliers is extremely high and noise dominates. Solving these two challenges is crucial for improving the robustness and practicality of feature matching in extremely realistic scenarios. Summary of the Invention

[0006] To address the shortcomings and deficiencies of existing technologies, such as over-smoothing of features and difficulty in extracting global consensus under high outlier ratios, this invention provides a dual-view feature matching method based on geometric context modulation and consensus distillation, aiming to improve the robustness of dual-view feature matching and pose estimation. This method first obtains the initial set of corresponding points between two views, calculates the motion vector and spatial coordinates of each corresponding point, and constructs the initial feature representation and position representation. Then, iterative refinement is performed through a cascaded network containing multiple refinement layers. Each refinement layer integrates three core modules working together: the Geometric Context Modulation (GCM) module generates gated weights based on local motion vectors and spatial coordinates, and adaptively modulates feature transformation through parallel processing of two branches, effectively preserving the feature discriminativeness at the motion boundary and alleviating the problem of feature oversmoothing; the Position Guided Attention (PGA) mechanism explicitly fuses appearance feature affinity and position encoding affinity when calculating attention weights, directly embedding spatial geometric constraints into the feature interaction process, and strengthening the capture of long-range dependencies; the Consensus Distillation and Propagation (CDP) module adopts a two-stage strategy of "distillation before propagation", generating fewer anchor points from all corresponding points through learnable soft aggregation operations, naturally suppressing the influence of outliers, and then distilling a pure global consensus field representation through self-attention interaction between anchor points. Finally, consensus information is propagated to all original corresponding points through cross-attention calculation to generate refined features. Through progressive refinement using a multi-layered cascaded architecture, the network is optimized by combining a classification loss and a geometric loss based on Sampson distance, ultimately outputting the inlier probability of corresponding points and estimating the essential matrix between two views. Experimental results demonstrate that this method performs exceptionally well on multiple public datasets. In the relative pose estimation task, the AUC@5° reaches 38.32%, representing a 17.6% improvement over the current state-of-the-art method. In the outlier removal task, the F-score reaches a maximum of 74.71%, significantly enhancing the robustness and accuracy of feature matching in complex scenarios.

[0007] The specific technical solution adopted by this invention to solve its technical problem is as follows:

[0008] A dual-view feature matching method, comprising:

[0009] Obtain the initial set of corresponding points between the two views, determine the motion vector and spatial coordinates of each corresponding point, and construct the initial feature representation and position representation;

[0010] The feature representation and the location representation are iteratively refined using a cascaded network containing at least one refinement layer;

[0011] Each of the refining layers includes:

[0012] The geometric context modulation step modulates the input features based on the motion vector and spatial coordinates;

[0013] The consensus distillation and propagation steps perform two-stage processing on the modulated features: In the first stage, anchor representations are generated from all corresponding points through learnable aggregation operations, and the position-guided attention mechanism is used to interact between the anchor representations to distill the global motion consensus representation; In the second stage, the position-guided attention mechanism is used to propagate the global motion consensus representation from the anchor representations to all corresponding points to generate a refined feature representation.

[0014] Based on the feature representation of the final output of the cascaded network, the inlier probability of the corresponding point is predicted and the geometric relationship between the two views is estimated.

[0015] Furthermore, the method for constructing the initial feature representation and position representation specifically includes: mapping the motion vector and the spatial coordinates to a higher dimension, and then processing them through the geometric context modulation step.

[0016] Furthermore, the geometric context modulation step is implemented through a gating mechanism: channel modulation weights are generated based on the input information, and the extracted features to be modulated are weighted and modulated using the channel modulation weights.

[0017] Furthermore, the location-guided attention mechanism calculates attention weights by fusing the feature affinity matrix obtained from the feature representation with the location affinity matrix obtained from the location representation.

[0018] Furthermore, the learnable aggregation operation is a soft aggregation operation, whose aggregation weight is determined by the input features, and is used to suppress the influence of abnormal corresponding points when generating anchor points.

[0019] Furthermore, the number of anchor points is less than the number of corresponding points in the initial set of corresponding points.

[0020] Furthermore, during training, the cascaded network calculates a classification loss and a geometric loss based on Sampson distance for the output of each refined layer, and optimizes the total loss by weighted summation.

[0021] Furthermore, the interior point probability is predicted by an interior point predictor based on a multilayer perceptron; the geometric relationship between the two views is represented by an essential matrix, which is estimated using a weighted eight-point algorithm.

[0022] Furthermore, the location-guided attention mechanism includes multiple attention heads.

[0023] And a computer device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described above.

[0024] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0025] Compared to existing technologies, this invention and its preferred solution utilize an innovative geometric context modulation module. By generating gating weights from local motion vectors and spatial coordinates, it adaptively adjusts feature transformations, effectively mitigating the oversmoothing problem that traditional methods often encounter in regions of discontinuous motion. This ensures feature discriminability at motion boundaries and provides a foundation for accurately distinguishing interior points from outliers. Based on a position-guided attention mechanism, it explicitly fuses appearance feature affinity and spatial location affinity, deeply embedding geometric constraints into the feature interaction process. This significantly enhances the spatial consistency and long-range dependency capture capability of feature matching, improving the accuracy of feature association in complex scenarios. Through a two-stage strategy involving consensus distillation and propagation modules, anchor points are first generated and outlier influence is suppressed through learnable soft aggregation operations. Then, global consensus information is distilled and propagated, avoiding the fragility of directly fitting consensus from high-noise data and achieving robust extraction of global geometric consensus even with a high outlier ratio. By combining a multi-layered, progressively refined architecture with a joint optimization strategy of classification loss and geometric loss, feature representations are gradually purified. This approach balances the accuracy of interior point prediction with the precision of geometric relationship estimation between views, significantly improving the robustness and reliability of dual-view feature matching and pose estimation. This provides stronger support for upstream computer vision tasks such as image registration, simultaneous localization and mapping, and 3D reconstruction. Attached Figure Description

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0027] Figure 1 This is a schematic diagram of the GMC-Net framework and workflow according to an embodiment of the present invention;

[0028] Figure 2 This is a test result diagram of the solution of the present invention on the YFCC100M dataset;

[0029] Figure 3 The image shows the test results of the solution in this embodiment of the invention on the SUN3D dataset. Detailed Implementation

[0030] To make the features and advantages of the present invention more apparent and understandable, specific embodiments are described below in detail:

[0031] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0032] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0033] To alleviate the aforementioned problems, this invention proposes a dual-view feature matching method based on geometric context modulation and consensus distillation. Based on the newly proposed GMC-Net framework, a Geometric Context Modulation (GCM) module is introduced to dynamically modulate feature transformations using local geometric information, preserving motion boundary information. Simultaneously, a Consensus Distillation and Propagation (CDP) module is designed, employing a two-stage strategy of "distillation before propagation." This is combined with a Position-Guided Attention (PGA) mechanism to explicitly fuse spatial position constraints, guiding the network to exhibit stronger expressive characteristics for interior points.

[0034] The implementation of this scheme follows these steps: A. Extract the initial corresponding point set and initialize the high-dimensional feature tensor and position encoding. B. Utilize the Geometric Context Modulation (GCM) module to adjust the feature transformation through an adaptive gating mechanism based on local motion vectors and coordinate information, preserving motion boundary information. C. Utilize the Position Guided Attention (PGA) mechanism to explicitly fuse features and position affinity to capture spatial geometric constraints. D. Utilize the Consensus Distillation and Propagation (CDP) module to generate anchor points using learnable soft aggregation operations, and distill and propagate the global motion consensus signal from the anchor points in two stages. E. Through... The cascaded architecture of the refining layers performs iterative processing, and the classification result is output by the interior point predictor. F estimates the essential matrix based on the classification result and optimizes the network using a joint loss of classification and geometry. This invention effectively alleviates the problem of consensus extraction in environments with overly smooth features at motion boundaries and high noise, and significantly improves the robustness of feature matching and pose estimation.

[0035] like Figure 1 As shown, the Geometric Modulation Consensus Network (GMC-Net) proposed in this embodiment of the invention purifies the features of corresponding points step by step through cascaded refinement layers. The specific implementation steps and process are as follows:

[0036] A. Given image pairs, construct a structure of size... initial corresponding point set Calculate the initial motion vector set and the original image coordinate set Among them, the motion vector set It naturally contains the local displacement context and coordinate set of each corresponding point. The spatial location context is provided, and both serve as sources of local geometric information. These are then mapped to a high-dimensional space through 1×1 convolutional layers before being input into the GCM module for context-aware initialization, generating initial feature tensors. and position encoding .

[0037]

[0038] in, This represents a convolutional layer; GCM stands for Geometric Context Modulation module, whose input is the motion vector or coordinate information of each corresponding point, and whose output is the corresponding... High-dimensional embedding is used to embed local geometric context into the feature representation.

[0039] B. To alleviate the feature oversmoothing problem, a GCM module is used for dynamic feature modulation in each refining layer. The GCM module receives feature input from the previous layer and processes it through two parallel branches: one branch generates channel modulation weights based on local geometric context, and the other branch extracts features. In this embodiment, the inputs to both the gate branch and the feature branch are the original input features F of the GCM module. in That is, F gate =F in By directly processing the module's input features, it is ensured that the gating weights accurately reflect the local geometric context characteristics of the input features. For the gating branch, based on the input features... Generate channel modulation weights :

[0040]

[0041] Utilizing this weighted modulation feature branch Output modulated features :

[0042]

[0043] in, Indicates instance normalization, Indicates the activation function; Represents element-wise product; This design allows the network to adaptively adjust its feature responses based on the local motion and coordinate context of each point, thereby maintaining feature discriminative power at motion boundaries.

[0044] C. Employ a location-guided attention PGA mechanism to capture long-range dependencies. This involves calculating the attention distribution. When considering the appearance feature affinity matrix and positional encoding affinity matrix and in Before normalization, the two are added together, thus embedding spatial geometric constraints directly into the core of attention weight calculation:

[0045]

[0046] in, , , All are position-coded The transformation yields, This is the scaling factor. In this way, spatial geometric constraints are directly embedded into the attention map.

[0047] D. Perform two-stage inference using the Consensus Distillation and Propagation (CDP) module. First, perform consensus distillation: using a learnable soft aggregation operation, from all... Generate by weighted aggregation of corresponding points Anchor point features and location ( In this embodiment, the learnable soft aggregation operation calculates the aggregation weights as follows: First, the features of each corresponding point are input into a lightweight multilayer perceptron (MLP), and the importance score of that point is output. Then, the importance scores of all corresponding points are Softmax normalized to obtain the normalized aggregation weight generated by each corresponding point for the anchor point. Inlier points receive higher weights due to their high feature consistency, while the weights of outlier points are suppressed, thereby weakening noise interference during the aggregation process. This weight is determined by the input features themselves, thus naturally suppressing the influence of outliers during aggregation. Next, a PGA mechanism is used to perform self-attention interactions between these anchor points, distilling a pure global consensus field representation. :

[0048]

[0049] Consensus propagation is then performed: using the characteristics of the consensus field obtained through distillation. and anchor point position As keys and values, based on the characteristics of all original points and location As a query, cross-attention is calculated using PGA, and global consensus information is propagated back to all corresponding points to generate refined features. :

[0050]

[0051] This invention effectively avoids the fragility of directly fitting consensus from data containing a lot of noise through a two-stage strategy of "distillation before propagation".

[0052] E. This invention employs the following: The cascading architecture of the refining layers allows for iterative execution of steps B through D. (Preferred in this embodiment) =8), each layer progressively refines the output of the previous layer. Finally, an interior point predictor based on a multilayer perceptron is used to classify the features of the last layer, outputting a classification probability vector for each point belonging to an interior point. .

[0053] F. Based on predicted probabilities The essential matrix is ​​estimated using a weighted eight-point algorithm. During the training phase, classification loss is used. and geometric loss Weighted and co-optimized networks:

[0054]

[0055] in, For the regression loss based on Sampson distance, To balance the coefficients, a high-performance feature matching model is finally constructed.

[0056] As a preferred implementation of this embodiment, the following configuration is adopted:

[0057] (1) Network structure: number of refined layers Feature Dimension The channel segmentation ratio of the GCM module is 0.5; the number of anchor points in the CDP module. The PGA attention head count is 8.

[0058] (2) Training configuration: YFCC100M and SUN3D datasets were used for training and evaluation; the optimizer was Adam; the initial learning rate was set to The learning rate is adjusted using an exponential decay strategy during training; the batch size is 32; and the total number of training iterations is [number missing]. All training and evaluation were performed on a single NVIDIA RTX 3090 GPU.

[0059] Compared to existing technologies, the core innovations of this invention include:

[0060] 1. A Geometric Context Modulation (GCM) module is proposed. This module utilizes a gating mechanism to dynamically modulate features based on local geometric context, directly alleviating the feature oversmoothing problem caused by isotropic processing in traditional PointCN-like methods. Ablation experiments show that replacing the GCM module with the standard PointCN block results in a significant drop in AUC@5° performance on the YFCC100M dataset from 38.32% to 34.99%.

[0061] 2. A consensus distillation and propagation (CDP) module based on Position Guided Attention (PGA) was designed. This module adopts a two-stage strategy of "distillation before propagation" and explicitly integrates position affinity in the attention calculation, achieving robust global consensus establishment even with a high outlier ratio. Ablation experiments show that removing the position guidance from PGA or skipping the consensus distillation stage significantly reduces performance (AUC@5° drops to 36.82% and 35.30%, respectively).

[0062] Tests showed that the proposed method achieved state-of-the-art results in both relative pose estimation and outlier removal on several challenging public datasets, including YFCC100M, SUN3D, and MegaDepth. For example, on the YFCC100M dataset, the AUC@5° for pose estimation using the weighted eight-point method reached 38.32%, representing a 17.6% improvement over the previous best method (DeMo, 32.57%). In the outlier removal task, the proposed method also achieved the highest F-score (74.71%) on the YFCC100M dataset. Figure 2 and Figure 3 The registration results of this method on the YFCC100M dataset and the SUN3D dataset are shown respectively.

[0063] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0064] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0065] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0066] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0067] This invention is not limited to the preferred embodiment described above. Anyone inspired by this invention can derive other forms of dual-view feature matching methods based on geometric context modulation and consensus distillation. All equivalent variations and modifications made within the scope of the claims of this invention shall fall within the scope of this invention.

Claims

1. A dual-view feature matching method, characterized in that, include: Obtain the initial set of corresponding points between the two views, determine the motion vector and spatial coordinates of each corresponding point, and construct the initial feature representation and position representation; The feature representation and the location representation are iteratively refined using a cascaded network containing at least one refinement layer; Each of the refining layers includes: The geometric context modulation step modulates the input features based on the motion vector and spatial coordinates; The consensus distillation and propagation steps perform two-stage processing on the modulated features: In the first stage, anchor representations are generated from all corresponding points through learnable aggregation operations, and the position-guided attention mechanism is used to interact between the anchor representations to distill the global motion consensus representation; In the second stage, the position-guided attention mechanism is used to propagate the global motion consensus representation from the anchor representations to all corresponding points to generate a refined feature representation. Based on the feature representation of the final output of the cascaded network, the inlier probability of the corresponding point is predicted and the geometric relationship between the two views is estimated.

2. The dual-view feature matching method according to claim 1, characterized in that: The method for constructing the initial feature representation and position representation specifically includes: mapping the motion vector and the spatial coordinates to a higher dimension, and then processing them through the geometric context modulation step.

3. The dual-view feature matching method according to claim 2, characterized in that: The geometric context modulation step is implemented through a gating mechanism: channel modulation weights are generated based on the input information, and the extracted features to be modulated are weighted and modulated using the channel modulation weights.

4. The dual-view feature matching method according to claim 1, characterized in that: The location-guided attention mechanism calculates attention weights by fusing the feature affinity matrix obtained from feature representation with the location affinity matrix obtained from location representation.

5. The dual-view feature matching method according to claim 1, characterized in that: The learnable aggregation operation is a soft aggregation operation, and its aggregation weight is determined by the input features, which is used to suppress the influence of abnormal corresponding points when generating anchor points.

6. The dual-view feature matching method according to claim 1, characterized in that: The number of anchor points is less than the number of corresponding points in the initial set of corresponding points.

7. The dual-view feature matching method according to claim 1, characterized in that: During training, the cascaded network calculates a classification loss and a geometric loss based on Sampson distance for the output of each refined layer, and optimizes the total loss by weighted summation.

8. The dual-view feature matching method according to claim 1, characterized in that: The interior point probability is predicted by an interior point predictor based on a multilayer perceptron; the geometric relationship between the two views is represented by an essential matrix, which is estimated using a weighted eight-point algorithm.

9. The dual-view feature matching method according to claim 1, characterized in that: The location-guided attention mechanism includes multiple attention heads.

10. A computer device, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor, when executing the computer program, implements the method of any one of claims 1 to 9.

Images