Robust guarantee method for graph matching

Abstract

The application relates to a robust guarantee method for graph matching, which is used for solving the problem that existing image matching produces obvious matching errors under external disturbance. In the method, joint Gaussian distribution is constructed based on key point position information in a pair of images to be matched, and N1 joint sampling noises and N2 joint sampling noises are obtained by sampling the joint Gaussian distribution; a first matching result set Y1 is obtained based on the N1 joint sampling noises and the pair of images to be matched, and a second matching result set Y2 is obtained based on the N2 joint sampling noises and the pair of images to be matched; a key point matching matrix of the first matching result set Y1 is smoothed to obtain a smoothed key point matching matrix y, the similarity between the smoothed key point matching matrix y and elements in the second matching result set Y2 is calculated, and a robust guarantee range is calculated based on the similarity and a preset similarity threshold value, so that when the disturbance of the input key point position is within the range, the output matching result is unchanged or the result matching similarity is higher than the set threshold value.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more particularly to a robustness guarantee method for graph matching problems. Background Technology

[0002] Graph matching aims to find correspondences between nodes in a graph and has wide applications in computer vision and pattern recognition, such as image keypoint matching and visual-text retrieval. Like most deep learning algorithms, graph matching algorithms are highly vulnerable to adversarial attacks, leading to matching failures. This is especially true in fields with high security requirements, where adversarial attacks can severely impact the usability and reliability of the algorithm.

[0003] Most existing methods for improving the robustness of graph matching algorithms are empirical, often only able to defend against fixed types of attacks and usually broken by more advanced attack types. The few robustness guarantee methods that can defend against all types of attacks also suffer from limited robustness guarantees when applied to graph matching problems, making them difficult to apply in real-world scenarios. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, this invention aims to propose a robustness guarantee method for graph matching. By calculating the robustness guarantee range, the method ensures that when the perturbation of the input key point position falls within this range, the output matching result remains unchanged or the similarity of the result matching is higher than a certain threshold, thereby achieving complete robustness guarantee.

[0005] To achieve the above objectives, the technical solution of this case is as follows.

[0006] Firstly, this case proposes a robustness guarantee method for graph matching, the method comprising the following steps:

[0007] A joint Gaussian distribution is constructed using the location information of key points in the image to be matched, and the joint Gaussian distribution is sampled to obtain N1 joint sampling noise and N2 joint sampling noise, where N2 > N1.

[0008] Based on N1 joint sampling noise and the image pair to be matched, the first matching result set Y1 is obtained, and based on N2 joint sampling noise and the image pair to be matched, the second matching result set Y2 is obtained.

[0009] The keypoint matching matrix of the first matching result set Y1 is smoothed to obtain a smoothed keypoint matching matrix y. The similarity between y and the elements in the second matching result set Y2 is calculated. Based on the similarity and a preset similarity threshold, the robustness guarantee range is calculated.

[0010] In the above technical solution, a similarity matrix is ​​constructed using the key point location information of the image pair to be matched, and a joint Gaussian distribution is constructed using the similarity matrix as the covariance.

[0011] In one embodiment of the above technical solution, the spatial distance between key points is calculated using the Euclidean distance formula, thereby obtaining a similarity matrix.

[0012] In the above technical solution, the joint sampling noise is obtained by Monte Carlo sampling of the joint Gaussian distribution.

[0013] In the above technical solution, the steps for obtaining the key point matching matrix y include:

[0014] The N1 joint sampling noises are added to the image pairs to be matched to obtain the set of image pairs to be matched, and then the first matching result set Y1 containing N1 key point matching matrices is obtained;

[0015] The keypoint matching matrices in the first matching result set Y1 are summed and then smoothed to obtain a smoothed keypoint matching matrix y.

[0016] In the above technical solution, the robustness guarantee range calculation steps include:

[0017] The number of elements with similarity higher than the similarity threshold is counted, and the probability p of elements with similarity higher than the similarity threshold in the second matching result set Y2 is obtained by using the p-value of the two-sided hypothesis.

[0018] The robustness guarantee range is calculated using the following formula:

[0019]

[0020] In the formula: λ max It is the largest eigenvalue of the inverse variance of the joint Gaussian distribution, Φ -1 It is the inverse of the cumulative distribution function of the standard Gaussian distribution.

[0021] In one embodiment of the above technical solution, the Sinkhorn algorithm and the Hungarian algorithm are used for smoothing.

[0022] In the above technical solution, the L2 norm perturbation at the key point location is less than 5.

[0023] In the above technical solution, the set of image pairs to be matched is input into the depth map matching model NGMv2 to obtain the key point matching matrix.

[0024] Secondly, this case proposes a graph matching method that uses any of the above methods to obtain the robustness guarantee range r. When the joint perturbation of the key point positions of the image pair to be matched is within the robustness guarantee range r, a matching image that meets the preset similarity threshold can be obtained, thus achieving successful matching. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram of an implementation method for ensuring robustness of graph matching based on key point locations;

[0027] Figure 2 A schematic diagram of the smooth matching model structure;

[0028] Figure 3 A schematic diagram of the robust computational model data flow framework. Detailed Implementation

[0029] Common depth map matching algorithms utilize convolutional neural networks to extract node features and build end-to-end models with spectral matching. However, similar to their deep learning models, depth map matching algorithms are also fragile and easily affected by external disturbances, resulting in significant matching errors.

[0030] Existing defenses against adversarial attacks applicable to graph matching models mainly include empirical adversarial training methods and robustness guarantee methods based on smoothing models. Adversarial training involves adding adversarial examples to the training set and performing targeted training to increase the predictive model's resistance to perturbations. Robustness guarantee methods based on smoothing models construct smoothing models using Gaussian distributions, ensuring that within a fixed perturbation range, the graph matching result remains unchanged or the matching similarity exceeds a certain threshold. The former cannot fully guarantee the model's robustness against a fixed range of perturbations and is easily compromised by newly developed attack algorithms, resulting in very poor matching results. Furthermore, the output only contains matching results without a quantitative description of the model's robustness, limiting its practical application due to a lack of reliability. The latter method can only handle classification problems. Since the output formats of classification and graph matching problems differ, using this method to guarantee the robustness of graph matching algorithms underestimates the model's true robustness. Moreover, this method does not consider the structural information of image keypoints, leading to a small range of calculated robustness guarantees.

[0031] This case considers the structural uniqueness of graph matching. It constructs a joint Gaussian distribution based on the structure of keypoints in the image pair to be matched. Based on the joint Gaussian distribution and a depth matching model, two sets of keypoint matching matrices are obtained. One set of keypoint matching matrices is smoothed, and the similarity between the two sets is calculated. A robustness guarantee range for the image pair to be matched is obtained based on probability calculation. Attack samples within this robustness guarantee range (regardless of the attack type) will all yield a matching result that is the same as or higher than a certain fixed threshold, thus resisting any attack type within a fixed range.

[0032] The following description, in conjunction with the accompanying drawings, clearly and completely describes how the technical solution of this case is implemented. Obviously, the described embodiments are only a part of the embodiments of this case, and not all of them. Based on the embodiments in this case, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0033] See Figure 1 The steps for calculating the robustness guarantee range of the image pairs to be matched in this scheme include:

[0034] S10. Construct a joint Gaussian distribution based on the location information of key points in the image to be matched, and sample the joint Gaussian distribution to obtain N1 joint sampling noise and N2 joint sampling noise, respectively, where N2 > N1, and N2 and N1 are both natural numbers.

[0035] First, distance metrics are calculated based on the position coordinates of keypoints in the image to be matched. Then, a similarity matrix between keypoints is constructed using the distance metric calculation results. For example, the spatial distance between keypoints can be calculated using the Euclidean distance formula, and this spatial distance is used as the distance metric. In the similarity matrix, the similarity between keypoints is negatively correlated with the spatial distance, and a joint Gaussian distribution is constructed using the similarity matrix as the covariance.

[0036] For the obtained joint Gaussian distribution, two sets of samples are performed using Monte Carlo sampling. The first set yields N1 sets of joint sampled noise, and the second set yields N2 sets of joint sampled noise. For ease of calculation, N2 = 10N1 is used as an example.

[0037] S20. Based on N1 joint sampling noise and the image pair to be matched, obtain the first matching result set Y1, and based on N2 joint sampling noise and the image pair to be matched, obtain the second matching result set Y2.

[0038] Two sets of joint Gaussian noise are added to the pairs of images to be matched to obtain two sets of images to be matched. These two sets of images to be matched are then input into the depth map matching model: for the first set of images to be matched, a first matching result set Y1 containing N1 keypoint matching matrices is obtained; for the second set of images to be matched, a second matching result set Y2 containing N2 keypoint matching matrices is obtained. For example, the depth map matching model is NGMv2.

[0039] S30. Smooth the keypoint matching matrix of the first matching result set Y1 to obtain a smoothed keypoint matching matrix y. Calculate the similarity between y and the elements in the second matching result set Y2. Based on the similarity and a preset similarity threshold, calculate the robustness guarantee range.

[0040] The keypoint matching matrices in the first set of matching results Y1 are summed and smoothed to obtain a smoothed keypoint matching matrix y. For example, the Sinkhorn algorithm and the Hungarian algorithm are used for smoothing.

[0041] The above process combines the establishment of a joint Gaussian distribution using keypoint locations with the depth map matching model to construct a smooth matching model. This results in a model with stronger perturbation resistance and guaranteed robustness than the depth map matching model. When the image to be matched is input into the smooth matching model, the corresponding keypoint matching matrix can be directly output based on the set sampling noise level. See also... Figure 2 As shown.

[0042] Calculate the similarity between elements in the second matching result set Y2 and the keypoint matching matrix y. Count the number of elements with similarity higher than the similarity threshold, denoted as Nd. Use the p-value of the two-sided hypothesis to detect the probability p that the similarity of elements in the second matching result set Y2 is higher than the similarity threshold. Calculate the robustness guarantee range according to the following formula:

[0043]

[0044] In the formula: λ max It is the largest eigenvalue of the inverse variance of the joint Gaussian distribution, Φ -1 It is the inverse of the cumulative distribution function of the standard Gaussian distribution.

[0045] Following the above implementation steps, tests were conducted on the Pascal VOC dataset with Berkeley annotations. It can be seen that, when the L2 norm perturbation at the keypoint location is less than 5, the similarity of the keypoint matching matrix is ​​always higher than the threshold, regardless of the attack type.

[0046] The above processes S10-S30 can be implemented as a robustness calculation model. When the image pairs to be matched are input, combined with a preset similarity threshold, the robustness guarantee range can be directly output. Compared to the robustness model trained adversarially, this robustness calculation model can provide a larger quantitative robustness guarantee range; and due to considering the structural specialities of graph matching, it can obtain an even larger robustness guarantee range compared to previous robustness guarantee methods. See [link to relevant documentation] Figure 3 As shown.

[0047] In the above process, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0048] The operations described above can be performed out of order. Instead, they can be performed in reverse order or simultaneously. Furthermore, one or more additional operations can be added to the process. One or more operations can be removed from the process.

[0049] In summary, this scheme can guarantee the robustness of the graph matching algorithm, improve the graph matching model's ability to cope with adversarial attacks, and enhance the reliability of the graph matching algorithm in real-world applications. When the joint perturbation of the keypoint positions of the image pair to be matched is within the robustness guarantee range r, the similarity of the keypoint matching matrix is ​​always higher than the preset similarity threshold used to calculate the robustness guarantee range, thus achieving successful matching; when the preset similarity threshold is 1, the keypoint matching result of the image pair to be matched remains unchanged.

[0050] In other embodiments, the above method may be implemented as a corresponding system or apparatus.

[0051] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods or systems disclosed herein can be implemented using software plus necessary general-purpose hardware, or they can be implemented using dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memory, dedicated components, etc. Generally, any function performed by a computer program can be easily implemented using corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the purposes of this disclosure, software program implementation is more often a preferred implementation method.

[0052] Although embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art can make many other forms based on the guidance of this specification and without departing from the scope of protection of the claims of the present invention, and all of these are within the scope of protection of the present invention.

Claims

1. A robustness guarantee method for graph matching, characterized in that, The method includes the following steps: A similarity matrix is ​​constructed using the key point location information of the image to be matched. A joint Gaussian distribution is constructed using the similarity matrix as the covariance. The joint Gaussian distribution is sampled to obtain N1 joint sampling noise and N2 joint sampling noise, where N2>N1. Based on N1 joint sampling noise and the image pairs to be matched, the N1 joint sampling noise and the image pairs to be matched are added to obtain the set of image pairs to be matched, and the first matching result set Y1 containing N1 key point matching matrices is obtained. Based on N2 joint sampling noise and the image pairs to be matched, the second matching result set Y2 containing N2 key point matching matrices is obtained. The keypoint matching matrix of the first matching result set Y1 is smoothed to obtain a smoothed keypoint matching matrix y. The similarity between y and elements in the second matching result set Y2 is calculated. Based on the similarity and a preset similarity threshold, the robustness guarantee range is calculated, including: counting the number of elements with similarity higher than the similarity threshold, and using the p-value of the two-sided hypothesis to detect the probability that the similarity of elements in the second matching result set Y2 is higher than the similarity threshold. The robustness guarantee range is calculated using the following formula: In the formula: It is the largest eigenvalue of the inverse variance of the joint Gaussian distribution. It is the inverse of the cumulative distribution function of the standard Gaussian distribution.

2. The method according to claim 1, characterized in that: The spatial distance between key points is calculated using the Euclidean distance formula, and then a similarity matrix is ​​obtained.

3. The method according to claim 1, characterized in that: The joint sampling noise was obtained by Monte Carlo sampling of the joint Gaussian distribution.

4. The method according to claim 1, characterized in that, Smoothing is achieved using the Sinkhorn algorithm and the Hungarian algorithm.

5. The method according to claim 1, characterized in that, The L2 norm perturbation at key points is less than 5.

6. The method according to claim 1, characterized in that, The set of image pairs to be matched is input into the depth map matching model NGMv2 to obtain the key point matching matrix.

7. A graph matching method, characterized in that: The robustness guarantee range r is obtained by any one of the methods in claims 1 to 6; When the joint perturbation of the key point positions of the image pair to be matched is within the robustness guarantee range r, a matching image that meets the preset similarity threshold can be obtained, thus achieving successful matching.

Images