Knowledge-guided hypergraph network-based high-quality image feature matching method

Abstract

The invention discloses a high-quality image feature matching method based on a knowledge-guided hypergraph network, and the method comprises the steps: 1, constructing an initial matching set: extracting feature descriptors of similar images, carrying out the similarity comparison, and carrying out the normalization, thereby obtaining the initial matching set; step 2, setting a structure of a knowledge-guided hypergraph network EGH-Net, namely performing two rounds of pruning through an energy function guiding module, namely an EFG module, so as to eliminate wrong matching, namely an exterior point, calculating an essential matrix and verifying an epipolar distance; step 3, training the EGH-Net until a preset number of times is reached; and step 4, inputting a to-be-matched image pair into the trained EGH-Net to obtain a corresponding relation of feature points. According to the method, the high-order consistency between the nodes and the dependency relationship between the sub-graphs can be effectively captured, the graph node features are dynamically optimized, and mismatching points are accurately eliminated.

Description

Technical Field

[0001] This invention belongs to the field of image matching technology, and in particular relates to a high-quality image feature matching method based on knowledge-guided hypergraph networks. Background Technology

[0002] In computer vision, image feature matching is a fundamental and crucial technique. Its purpose is to establish correct correspondences (inliers) between feature points in two images and to estimate camera pose accordingly. High-quality feature matching is a vital prerequisite for many high-level vision tasks, such as image registration, 3D reconstruction, loop closure detection, image stitching, and 3D object recognition, and directly impacts the accuracy and robustness of these tasks. Although matching methods based on local feature descriptors (e.g., SIFT, SuperPoint) have made significant progress in cross-scene adaptability and general robustness, they still commonly suffer from declining matching quality when faced with complex deformations, non-rigid motion, or extreme viewpoint changes, generating numerous false matches (outliers). This severely affects the accuracy of subsequent geometric estimation and system stability. Therefore, the mechanism for eliminating false matches is a key step in improving feature matching quality and plays a crucial role in ensuring the performance of downstream vision tasks.

[0003] In recent years, graph neural networks have been increasingly applied to mismatch removal tasks in image matching due to their powerful modeling capabilities for structured data. Typical methods already exist, such as CLNet and MS. 2 DGNet, NCMNet, MGNet, and BCLNet typically model the initial matching set as a graph structure, treating each matching pair as a node in the graph, and using the relationships between nodes and edges in the graph for outlier detection and filtering. However, most of these methods are still limited to pairwise connectivity discrimination patterns, making it difficult to effectively capture the potential high-order local consistency between nodes. Therefore, their ability to model structural consistency in complex scenarios is significantly insufficient. In addition, existing methods usually adopt a strategy of first independently modeling local subgraphs and then aggregating features through attention mechanisms or graph convolutional networks. This approach often ignores the structural alignment relationships between subgraphs, resulting in the inability to fully utilize global structural dependencies and consistency constraints, thus limiting further improvements in match removal performance.

[0004] Therefore, determining a new method that can more fully explore the consistency relationship between matching points and achieve effective fusion at the local and global levels to eliminate mismatches in order to capture higher-order constraints and thus significantly improve the robustness and accuracy of the matching process under complex imaging conditions has become an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to solve the problems mentioned in the background art and to propose a high-quality image feature matching method based on knowledge-guided hypergraph networks.

[0006] A high-quality image feature matching method based on knowledge-guided hypergraph networks, the method comprising:

[0007] Step 1, Construct the initial matching set: Extract feature descriptors of similar images, compare their similarity, and normalize them to obtain the initial matching set;

[0008] Firstly, based on the Scale Invariant Feature Transform (SIFT) algorithm and the SuperPoint deep learning feature extraction method, given two similar image pairs... and Detect key points and generate corresponding feature descriptors; image pairs and Two images of the same object from different angles;

[0009] Subsequently, the feature descriptors of the two images are compared for similarity using a nearest neighbor matching strategy to obtain an initial feature matching set. , where the initial matching set Includes For the initial matching point; for each matching point pair Indicates the first Initial feature matching, Representing an image Normalized coordinates of feature points on the surface; Representing an image The normalized coordinates of the corresponding feature points are obtained, and the normalization operation is performed based on the camera's intrinsic parameters.

[0010] Step 2: Set the structure of the knowledge-guided hypergraph network EGH-Net, that is, perform two rounds of pruning through the energy function guidance module, i.e., the EFG module, to remove erroneous matches, i.e., outliers, calculate the essential matrix and verify the epipolar distance;

[0011] The knowledge-guided hypergraph network EGH-Net includes two EFG modules, a model estimation module, and a full-size validation module.

[0012] Due to the fuzziness of feature descriptors, the initial matching set Inevitably, a large number of incorrect matches exist in the graph. To identify correct matches (interior points) and eliminate outterior points, this application proposes an Energy Guided Hypergraph Network (EGH-Net, e.g., ...). Figure 1 (As shown). Specifically, the initial matching set As input, the EFG module calculates the probability value of whether each pair of matches is an interior point. Then based on probability values For the initial matching set Perform the first round of pruning to obtain the matching subset of the first round of pruning. ,in The matching subset for the first round of pruning The number of matching pairs;

[0013] This result is then fed into a second EFG module to calculate more precise probability values ​​and perform a second round of pruning, yielding a matching subset from the second round of pruning. ,in Matching subsets for the second round of pruning The number of matching pairs;

[0014] The data is fed into the model estimation module to calculate the interior point weights. Next, the weighted eight-point algorithm is applied. To calculate the essential matrix :

[0015] ;

[0016] Finally, the predictions obtained using EGH-Net were... Perform full-size validation on the initial matching set S to obtain the epipolar distance y of the initial matching set S:

[0017] ;

[0018] Where the function Used to calculate epipolar distance, the output set This indicates the correctness of each match.

[0019] The core of EGH-Net is the EFG module. The goal of the EFG module is to model intra- and inter-graph energies based on energy functions to update feature representations, thereby effectively modeling the consistency between corresponding features (e.g., ...). Figure 2 (As shown). Specifically, the total energy in the energy function guiding module. Including energy within the diagram Energy between graphs Energy within the diagram Hypergraph modeling is used to model consistency relationships between nodes, and structural relationships within hyperedges are captured by weighted summation of node feature differences; inter-graph energy. The graph kernel module is used to model the structural correspondences across subgraphs, thereby achieving global structural alignment. Based on these two energy terms, the EFG module dynamically updates node features through gradient descent and updates the graph representation. Merging into a global graph representation ;

[0020] The EFG module includes a hypergraph construction module, an intra-graph energy calculation module, an inter-graph energy calculation module, and a prediction layer; the execution flow of the EFG module is as follows:

[0021] Step 21: Use a hypergraph to model intragraph relationships;

[0022] Using a multilayer perceptron (MLP) to connect each matching pair After mapping to a high-dimensional space to obtain feature representations, a hypergraph of matching pairs is constructed by searching a predetermined number of nearest neighbor points within its local neighborhood. ,in , Indicates the number of superedges; Represents a set of nodes, where each node is a neighboring node. Defined as , representing a superedge set The nodes, among which Indicates the superedge The number of nodes included;

[0023] Step 22, calculate the energy within the graph. High-order consistency between modeling nodes;

[0024] The energy terms in the diagram Defined as a hyperedge Weighted feature differences among all nodes:

[0025] ;

[0026] in They represent hyperedges respectively. Middle node and The feature representation, where C represents the number of channel dimensions and D represents the number of feature dimensions;

[0027] To update the node representation, calculate the energy term within the computational graph. Regarding the characteristics of hyperedge nodes The gradient; the squared Euclidean distance of the energy term within the graph:

[0028] ;

[0029] Therefore, we can conclude that:

[0030] ;

[0031] Based on the definition of the trace of a matrix, the simplified formula is:

[0032] ;

[0033] in It is the identity matrix. It is a matrix whose elements are all 1s; It is a smoothing constraint matrix designed to capture hyperedges. Higher-order consistency among internal nodes; computation of graph energy to dynamically update node representations. Regarding the characteristics of hyperedge nodes gradient:

[0034] ;

[0035] Step 23, calculate inter-graph energy Used to capture structural correspondences across subgraphs;

[0036] Given two hyperedges and , The inter-graph energy term is defined as follows:

[0037] ;

[0038] in It is the Frobenius norm. Represents a similarity function based on the Frobenius norm; to progressively align feature structures, ... Calculate the gradient:

[0039] ;

[0040] because Often nonlinear and difficult to define precisely, the derivative is approximated using a graphical kernel module. Specifically, firstly, the input subgraph... and A max-pooling layer (MaxPool) and a multilayer perceptron (MLP) are used to perform multi-scale sampling along the channel dimension C, resulting in feature representations with T different scales along the channel dimension. and ,in And the number of channels in the t-th layer satisfies The operation is defined as follows:

[0041] ;

[0042] ;

[0043] Next, we will use top-down path convolution. Extract global structural features, using The convolution kernel has a channel dimension relative to the (t-1)th layer. Perform transformations to update the features. :

[0044] ;

[0045] Among them, when hour, Meanwhile, bottom-up path convolution Through reverse transmission and combination The convolution kernel completes the transformation from the (t+1)th layer channel dimension up to the t-th channel dimension The mapping is used to enhance local structural details and obtain features. :

[0046] ;

[0047] Among them, when hour, At each layer, features from bidirectional paths are connected, and convolutional kernels are used to... Point convolution To update features; approximate calculation derivative :

[0048] ;

[0049] Therefore, the final inter-graph energy The gradient is expressed as:

[0050] ;

[0051] Step 24, determine the energy within the diagram. Energy between graphs Joint, gradient descent update;

[0052] The total gradient is:

[0053] ;

[0054] The hyperedge features are updated iteratively using gradient descent until the k-th iteration converges:

[0055] ;

[0056] in, and Each subgraph represents the features at the k-th and (k-1)-th iterations. The corresponding feature representation is obtained after k iterations. When k=0, ; It is a learnable parameter used to control the step size of each update; after iterating to the Kth time, the subgraphs are merged to obtain the global graph representation. :

[0057] ;

[0058] in Includes feature concatenation operations and a ResNet block; arrive These represent from all subgraphs. arrive The feature set is used; finally, MLP, tanh, and ReLU activation functions are used to predict the inlier probability. ;

[0059] Step 25: Optimize the proposed network using a hybrid loss function;

[0060] The hybrid loss function is:

[0061] ;

[0062] in, Describes a cross-entropy loss function used for inlier-outlier classification; using Regression is performed on the basic matrix, where For the true essential matrix, The essential matrix for prediction; These are the weighting parameters that balance these two types of losses;

[0063] Thus, we obtain EGH-Net, which can take a set of input features in an end-to-end manner and, through learning by EGH-Net, give the final probability set representing the outer pole distance of the corresponding set of features.

[0064] Step 3: Train EGH-Net until the preset number of iterations is reached;

[0065] Step 4: Pair the image to be matched with the trained EGH-Net to obtain the correspondence of feature points.

[0066] The significant advancement of this invention compared to existing technologies lies in:

[0067] 1. The graph energy function proposed in this invention models the consistency constraints of internal nodes by calculating the feature differences among multiple nodes within a subgraph. This design overcomes the limitations of traditional methods that only consider point-to-point relationships or low-order edge constraints, enabling the hypergraph to more accurately express high-order dependencies between nodes within the subgraph. It directly links feature differences with consistency constraints and distinguishes internal and external nodes by the magnitude of the energy value: lower energy values ​​indicate that nodes satisfy consistency constraints, while significantly higher energy values ​​correspond to external nodes that violate the constraints. Therefore, this energy function can achieve effective constraint modeling and external node detection at the subgraph level, significantly improving the model's ability to perceive local consistency and suppress external nodes in complex scenarios.

[0068] 2. This invention introduces an inter-graph energy function between subgraphs and combines it with a graph kernel module to achieve structural relationship alignment across subgraphs. Inter-graph energy is calculated through a multi-scale bidirectional mechanism: the bottom-up path captures the global structural features of the subgraph, while the top-down path preserves local details, ultimately achieving structural similarity modeling between subgraphs at multiple scales. This design avoids the shortcomings of insufficient information exchange between subgraphs or single-scale dependence in traditional methods. This mechanism can fully utilize global and local structural information, making the hypergraph more comprehensive and accurate in subgraph-level structural alignment and similarity modeling, thereby enhancing the consistency modeling capability across subgraphs and improving the interior point recognition rate and overall matching accuracy in complex scenarios.

[0069] 3. This invention proposes a gradient update mechanism based on energy functions. By jointly optimizing intra-graph energy and inter-graph energy functions, dynamic iterative updates of node features are achieved. Combining energy functions with gradient descent allows node features to continuously and adaptively adjust under energy constraints. Intra-point relationships are gradually strengthened during optimization, while the influence of out-points that disrupt consistency is effectively weakened, resulting in more accurate and robust image feature matching.

[0070] To more clearly illustrate the functional characteristics and structural parameters of the present invention, further explanation is provided below in conjunction with the accompanying drawings and specific embodiments. Attached Figure Description

[0071] Figure 1 This is the EGH-Net diagram provided in the embodiments of this application;

[0072] Figure 2 This is an architecture diagram of the core module provided in the embodiments of this application;

[0073] Figure 3 These are visualization results of different methods provided in the embodiments of this application. Detailed Implementation

[0074] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] This invention proposes a knowledge-guided hypergraph network (EGH-Net). This network constructs subgraphs using a hypergraph and models intra- and inter-graph knowledge through an energy function to update feature representations. This effectively captures higher-order consistency between nodes and dependencies between subgraphs, dynamically optimizes graph node features, and accurately eliminates mismatched points, thereby improving the accuracy and robustness of feature matching.

[0076] The goal of camera pose estimation is to accurately compute the relative rotation and displacement matrices between different camera views using identified inliers. First, EGH-Net was trained on the YFCC100M dataset, and camera poses were estimated using SIFT and SuperPoint descriptors in both known and unknown outdoor scenes. The camera poses of image pairs were encoded using rotation and displacement vectors computed from the estimated essential matrix. The angular difference between the ground truth essential matrix and the estimated essential matrix was selected as the error metric. The mean accuracy (mAP) at different thresholds was used as the evaluation metric for dual-view geometry estimation, with mAP at 5° (i.e., mAP5°) being the default. Furthermore, mAP at a 20° threshold (i.e., mAP20°) was also computed to further evaluate the model's performance within a more forgiving error range. Experimental results are shown in Table 1. When using the SIFT descriptor, EGH-Net improved mAP5° by 4.76% and 4.67% compared to BCLNet in known and unknown outdoor scenes, respectively, while reducing the number of training parameters by 42.30%. When using the SuperPoint descriptor, EGH-Net also achieved leading performance, improving mAP5° by 3.69% and 2.57% compared to BCLNet in known and unknown scenarios, respectively.

[0077] Figure 3 The visualization results of the predicted matching are presented. Match quality is quantitatively evaluated using inlier ratio, precision (P), recall (R), and F-score (F). Figure 3In the image, each column forms a group, from left to right, showing the visualization results for CLNet, NCMNet, BCLNet, and EGH-Net. The green and red lines represent inliers and mismatches, respectively. It is observed that EGH-Net removes more mismatches than CLNet, NCMNet, and BCLNet, thus significantly improving matching accuracy.

[0078] Table 1. Evaluation results of camera pose estimation on the YFCC100M dataset using different descriptors.

[0079]

[0080] Furthermore, the camera pose estimation capability of EGH-Net was evaluated on the more complex indoor dataset SUN3D, under both RANSAC-enabled and non-RANSAC-enabled conditions, including both known and unknown scenes. The experimental results are shown in Table 2. EGH-Net performed best at mAP5°, outperforming all baseline methods.

[0081] Table 2: Evaluation results of camera pose estimation using SIFT features on the SUN3D dataset

[0082]

[0083] The goal of homography estimation is to determine the transformation matrix between two images. Experiments were conducted on the HPatches dataset, which contains 116 different scenes, each with 6 images and a corresponding homography reference. First, keypoints were extracted and descriptors were computed using the SIFT algorithm to construct initial matches. Subsequently, EGH-Net was trained on the YFCC100M dataset and applied to remove mismatches from HPatches image pairs. The percentage of matching image pairs was calculated when the reprojection error was less than 3 pixels, 5 pixels, and 10 pixels, and compared with existing methods. The results are shown in Table 3. EGH-Net outperforms the baseline method at all error thresholds, with the most significant improvement at the strict 3-pixel threshold.

[0084] Table 3: Homography matrix evaluation results on the HPatches dataset

[0085]

[0086] In the embodiment, the input to EGH-Net in step 21 is of size [size missing]. The initial matching point pairs are used to extract feature points using the SIFT or SuperPoint algorithm. During the two rounds of iterative pruning, To ensure computational stability and efficiency, the number of nodes in the graph was set to 18, and the feature channel dimension was set to 128. In the graph kernel module, the scale T was set to 4. EGH-Net was trained using the Adam optimizer for a total of 500,000 iterations. A warm-up strategy was employed to adjust the learning rate: in the first 10,000 iterations, the learning rate was linearly increased from 0 to 10. −3 The loss function then decays every 20,000 iterations with a decay factor of 0.4. The parameters in the loss function... The value was set to 0.5. All experiments were conducted on an Ubuntu 18.04 system and an NVIDIA RTX 4090 GPU.

[0087] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A high-quality image feature matching method based on knowledge-guided hypergraph networks, characterized in that, The methods include: Step 1, Construct the initial matching set: Extract feature descriptors of similar images, compare their similarity, and normalize them to obtain the initial matching set; Step 2: Set the structure of the knowledge-guided hypergraph network EGH-Net, that is, perform two rounds of pruning through the energy function guidance module, i.e., the EFG module, to remove erroneous matches, i.e., outliers, calculate the essential matrix and verify the epipolar distance; The Energy Guided Hypergraph Network (EGH-Net) includes two EFG modules, a model estimation module, and a full-size validation module. Step 3: Train EGH-Net until the preset number of iterations is reached; Step 4: Pair the image to be matched with the trained EGH-Net to obtain the correspondence of feature points.

2. The method according to claim 1, characterized in that, Step 1, construct the initial matching set, including: Firstly, based on the Scale Invariant Feature Transform (SIFT) algorithm and the SuperPoint deep learning feature extraction method, given two similar image pairs... and Detect key points and generate corresponding feature descriptors; image pairs and Two images of the same object from different angles; Subsequently, the feature descriptors of the two images are compared for similarity using a nearest neighbor matching strategy to obtain an initial feature matching set. , where the initial matching set Includes For the initial matching point; for each matching point pair Indicates the first Initial feature matching, Representing an image Normalized coordinates of feature points on the surface; Representing an image The normalized coordinates of the corresponding feature points are obtained, and the normalization operation is performed based on the camera's intrinsic parameters.

3. The method according to claim 2, characterized in that, Step 2: Define the structure of the knowledge-guided hypergraph network EGH-Net, which involves two rounds of pruning using the energy function-guided module (EFG module) to remove incorrect matches (outside points), calculate the essential matrix, and verify the epipolar distance; including: The initial matching set As input, the EFG module calculates the probability value of whether each pair of matches is an interior point. Then based on probability values For the initial matching set Perform the first round of pruning to obtain the matching subset of the first round of pruning. ,in Matching subsets for the first round of pruning The number of matching pairs; This result is then fed into a second EFG module to calculate more precise probability values ​​and perform a second round of pruning, yielding a matching subset from the second round of pruning. ,in Matching subsets for the second round of pruning The number of matching pairs; The data is fed into the model estimation module to calculate the interior point weights. Next, the weighted eight-point algorithm is applied. To calculate the essential matrix : ； Finally, the predictions obtained using EGH-Net were... Perform full-size validation on the initial matching set S to obtain the epipolar distance y of the initial matching set S: ； Where the function Used to calculate epipolar distance, the output set This indicates the correctness of each match.

4. The method according to claim 3, characterized in that, Total energy in the energy function bootstrap module Including energy within the diagram Energy between graphs Energy within the diagram Hypergraph modeling is used to model consistency relationships between nodes, and structural relationships within hyperedges are captured by weighted summation of node feature differences; inter-graph energy. Then, the graph kernel module is used to model the structural correspondences across subgraphs, thereby achieving global structural alignment; Based on these two energy terms, the EFG module dynamically updates node features through gradient descent and then updates the graph representation. Merging into a global graph representation ; The EFG module includes a hypergraph construction module, an intra-graph energy calculation module, an inter-graph energy calculation module, and a prediction layer.

5. The method according to claim 4, characterized in that, The execution flow of the EFG module is as follows: Step 21: Use a hypergraph to model intragraph relationships; Using a multilayer perceptron (MLP) to match each pair After mapping to a high-dimensional feature space to obtain feature representations, a hypergraph of matching pairs is constructed by searching for a predetermined number of nearest neighbor points in the local neighborhood. ,in , Indicates the number of superedges; Represents a set of nodes, where each node is a neighboring node. Defined as , representing a superedge set The nodes, among which Indicates the superedge The number of nodes included; Step 22, calculate the energy within the graph. High-order consistency between modeling nodes; The energy terms in the diagram Defined as a hyperedge Weighted feature differences among all nodes: ； in They represent hyperedges respectively. Middle node and The feature representation, where C represents the number of channel dimensions and D represents the number of feature dimensions; To update the node representation, compute the in-graph energy term. Regarding the characteristics of hyperedge nodes The gradient; the squared Euclidean distance of the energy term within the graph: ； Therefore, we can conclude that: ； Based on the definition of the trace of a matrix, the simplified formula is: ； in It is the identity matrix. It is a matrix whose elements are all 1s; It is a smoothing constraint matrix designed to capture hyperedges. Higher-order consistency among internal nodes; computation of graph energy to dynamically update node representations. Regarding the characteristics of hyperedge nodes gradient: ； Step 23, calculate inter-graph energy Used to capture structural correspondences across subgraphs; Given two hyperedges and , The inter-graph energy term is defined as follows: ； in It is the Frobenius norm. Represents a similarity function based on the Frobenius norm; to progressively align feature structures, [the following is used]... Calculate the gradient: ； because For nonlinear problems, the derivative is often approximated using a kernel module. First, the input subgraph and MaxPool and MLP are used to perform multi-scale sampling along the channel dimension C, resulting in feature representations with T different scales along the channel dimension. and ,in And the number of channels in the t-th layer satisfies The operation is defined as follows: ； ； Next, we will use top-down path convolution. Extract global structural features, using The convolution kernel has a channel dimension relative to the (t-1)th layer. Perform transformations to update and obtain features. : ； Among them, when hour, Meanwhile, using bottom-up convolutional kernels for convolution Complete the process from the (t+1)th layer channel dimension up to the t-th channel dimension The mapping is used to enhance local structural details and obtain features. : ； Among them, when hour, At each layer, features from bidirectional paths are connected, and convolutional kernels are used to... Point convolution To update features; approximate calculation derivative : ； Therefore, the final inter-graph energy The gradient is expressed as: ； Step 24, determine the energy within the diagram. Energy between graphs Joint, gradient descent update; The total gradient is: ； The hyperedge features are updated iteratively using gradient descent until the k-th iteration converges: ； in, and Each subgraph represents the features at the k-th and (k-1)-th iterations. The corresponding feature representation is obtained after k iterations. When k=0, ; It is a learnable parameter used to control the step size of each update; after iterating to the Kth time, the subgraphs are merged to obtain the global graph representation. : ； in Includes feature concatenation operations and a ResNet block; arrive These represent from all subgraphs. arrive The feature set is then used; finally, it is mapped to interior point probabilities using MLP, tanh, and ReLU activation functions. ; Step 25: Optimize the proposed network using a hybrid loss function; The hybrid loss function is: ； in, Describes a cross-entropy loss function used for inlier-outlier classification; using Regression is performed on the basic matrix, where For the true essential matrix, The essential matrix for prediction; These are the weighting parameters that balance these two types of losses; Thus, EGH-Net is obtained.

Images