A false match elimination method based on improved vector field consistency

Abstract

The present invention provides a method for eliminating false matches based on improved vector field consistency. The method achieves the purpose of improving the accuracy of feature matching and the stability of the reconstruction algorithm by eliminating feature points that are incorrectly matched during the image stereo reconstruction process, and provides more accurate parameter information for subsequent stereo reconstruction tasks. In addition, the elimination method combines the global semantic information extracted by the visual Transformer model with the feature descriptors that characterize local details to form a high-dimensional feature vector containing local and global information. Under the Bayesian framework, the improved VFC algorithm converts the interpolation problem into a solution process for maximum a posteriori probability estimation. A hidden variable is associated with each vector field sample to identify whether the sample is an inlier or an outlier, and matching expansion is performed through the Bayesian formula to retain the correct feature matching pairs as much as possible and find potential correct matching pairs.

Description

Technical Field

[0001] The invention relates to the technical field of feature matching, and in particular to a method for eliminating false matches based on improving vector field consistency. Background Art

[0002] The core task of image feature matching is to find the same-name image points between different images, which is an important research topic in photogrammetry and computer vision. Feature matching is one of the key technologies in computer vision and image processing, and is widely used in stereo vision, image recognition, 3D reconstruction and other fields.

[0003] However, in practical applications, image feature matching in the prior art is prone to mismatching problems due to factors such as changes in ambient lighting, object occlusion, and image noise. These mismatches not only reduce the accuracy of feature matching, but may also affect the stability and reliability of subsequent processing flows. Although traditional feature matching algorithms, such as descriptor-based matching methods and geometric consistency-based algorithms, can achieve good results in some cases, they still face the challenge of high mismatch rates in complex environments. Therefore, studying an algorithm that can effectively eliminate mismatched features is of great significance for improving the accuracy and robustness of feature matching. Summary of the invention

[0004] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a method for eliminating false matches based on improving the consistency of vector fields to solve the problems raised in the above-mentioned background technology. The present invention eliminates feature points that are incorrectly matched during the image stereo reconstruction process to achieve the purpose of improving the accuracy of feature matching and the stability of the reconstruction algorithm, thereby providing more accurate parameter information for subsequent stereo reconstruction tasks.

[0005] In order to achieve the above-mentioned object, the present invention is implemented by the following technical scheme: a method for eliminating mismatches based on improving the consistency of vector fields, comprising the following steps: S1: using a DPT network to generate a semantic feature vector representing global information for each pixel in an input image, and upgrading the one-dimensional mark to a two-dimensional space through a decoder to restore the spatial structure of the image;

[0006] S2: Use the visual Transformer model to extract the global semantic information of the image and combine it with the local feature descriptors generated by the SIFT algorithm to form a higher-dimensional feature vector containing local and global information;

[0007] S3: Calculate the distribution density of key points relative to the initial matching set using the kernel density estimation method to determine candidate expandable key points;

[0008] S4: Use the Bayesian formula to expand the matching, and gradually build and optimize the matching set in an iterative manner;

[0009] S5: Use the EM algorithm to solve the optimization problem by alternating between expectation steps and maximization steps until the objective function converges and finally outputs a vector field.

[0010] Furthermore, the solution optimization steps included in the EM algorithm are as follows:

[0011] S5.1: Compute the expectation, based on the current parameter estimates, and compute the posterior distribution of the latent variable;

[0012] S5.2: Perform a maximization operation and use the posterior distribution of the sample points as inliers to update the parameter estimates;

[0013] S5.3: Iterate the above two steps until the EM algorithm converges and finally outputs the vector field f.

[0014] Furthermore, the encoding operation process of the decoder in S1 is as follows: the input image is divided into several non-overlapping image blocks, each of which is converted into a token with a position code. This process is similar to the word segmentation of text in the natural language processing (NLP) task, where the position code is used to identify the position of the vocabulary. The visual Transformer uses a multi-head self-attention mechanism to process the token, so that each token is associated with all other tokens, so that it has a global receptive field at all stages of the network. The decoder performs three operations of reading, splicing and resampling in sequence, with the aim of upgrading the one-dimensional token to a two-dimensional space to better restore the spatial structure of the image; first, by connecting the position code token with all other tokens, the following formula is used to convert The high-dimensional vector is mapped back indivual:

[0015]

[0016] In the formula, is the identification feature, Records the location information of each mark, splicing operation Used to connect these identification features together, is a fully connected layer.

[0017] Furthermore, after the reading operation, the obtained The spatial connection of the markers generates a , the number of channels is The feature map is reshaped into an image representation, and finally, through the resampling layer, the feature map is adjusted to :

[0018]

[0019]

[0020] in Represents the feature dimension after resampling, after spatial connection ( ) and resampling ( ) operation, the feature fusion module then fuses the feature bodies of different sizes and gradually upsamples them by two times in each fusion stage, and finally outputs the corresponding semantic feature vector for each pixel.

[0021] Furthermore, the operation of combining the global semantic information of the image extracted by the visual Transformer model with the local feature descriptor generated by the SIFT algorithm in S2 is:

[0022] The SIFT algorithm is used to construct local feature descriptors. In order to ensure that the feature points are rotationally invariant, the gradients and directions of all pixels in a certain area around the feature points are calculated, and a gradient histogram is constructed based on this. The highest peak in the histogram represents the main direction of the feature point. For each detected feature point, The pixel neighborhood is divided into sub-regions, calculate the pixel gradient direction of each sub-region, and quantize it according to 8 directions to form a 128-dimensional feature descriptor vector. The calculation of gradient and direction can be performed by the following formula:

[0023]

[0024]

[0025] Furthermore, the matching extension based on the Bayesian formula in S4 is:

[0026] Through a progressive strategy that uses the Bayesian formula for matching expansion, each sampling and model estimation can benefit from the results of the previous iteration, and the correctness of the matching points is estimated and continuously updated based on prior knowledge and observed data;

[0027] First, a strict threshold nearest neighbor distance ratio (NNDR) is used to obtain a small but accurate and reliable initial matching set. , and then gradually build and optimize the matching set in an iterative manner. This process comprehensively considers the spatial distribution of feature points, feature descriptors, and semantic vectors to form a larger extended matching set .

[0028] Furthermore, the goal of match expansion is to use the initial match set and the initial transformation matrix For prior information, find a new matching set , this process can be achieved by maximizing the conditional probability , find the most likely correct set of matching points. According to the Bayesian formula, this conditional probability can be decomposed into two terms. For the first probability Represents a given initial matching set and the initial transformation matrix In the case of images The key points in are selected as extended key points The probability of the key point relative to The distribution density is used to estimate, that is, the position of new potential matching points is predicted by the distribution of existing matching points:

[0029]

[0030] The second probability Indicates that the image is found The corresponding extended key point in The probability of is calculated by considering descriptor similarity, semantic vector similarity and geometric constraints, and the input Unmatched key points in , according to the initial vector field Make sure it is in the image In the corresponding projection area, find The most similar eigenvector , using cosine similarity as a measure to measure the similarity between two feature vectors, and the new matching pairs Perform reverse verification, that is, in the image Search The corresponding projection area of ​​​​ in agreement;

[0031]

[0032]

[0033] The above matching expansion process is repeated until no new matching pairs are generated or the number of newly added matching pairs is lower than a certain threshold, or the set maximum number of iterations is reached.

[0034] Furthermore, in S5.1, the expected value is calculated as follows:

[0035] Based on the current parameter estimates , calculate the posterior distribution of the latent variable. Specifically, define a diagonal matrix , the posterior probability of each element is , indicating that given the observed data and the current parameter estimates Next, The hidden variable corresponding to the sample The probability of being 1 reflects the The consistency between the samples and the current estimated vector field f, the posterior probability It can be calculated by Bayesian criterion:

[0036]

[0037] Furthermore, the maximization step performed in S5.2 is as follows:

[0038] Using the posterior probability of the sample point as an internal point To update the parameter estimates , the goal is to maximize the function ,in , considering that P is a diagonal matrix, about and Derivatives are taken separately and the derivatives are set to zero to obtain the update, where is the trace of the matrix:

[0039] σ 2 = tr[(YV ) T P(YV)] D⋅tr P ,γ= tr P N #

[0040] in, and Represents the matrix form of all observations and corresponding model output values, represents the trace of a matrix, which is the sum of the diagonal elements of the matrix.

[0041] Beneficial effects of the present invention:

[0042] 1. The false match elimination method based on improved vector field consistency improves robustness: The elimination algorithm shows significant robustness when facing images with changing illumination and perspective. The algorithm can effectively identify and eliminate false matching point pairs while retaining the correct matching point pairs to the maximum extent, ensuring high matching quality under various environmental conditions.

[0043] 2. The false match elimination method based on improved vector field consistency enhances matching performance: By integrating the high-dimensional features of semantic information and the matching extension module based on Bayesian inference, the present invention improves the matching ratio while ensuring the accuracy of matching. This combination not only improves the accuracy of feature matching, but also can find and retain potential correct matching point pairs while eliminating false matches, thereby further improving the overall matching performance on the basis of expanding the number of matching pairs. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1A flow chart of a method for eliminating mismatches based on improving the consistency of a vector field according to the present invention;

[0045] Figure 2 This is a visualization effect diagram of the characteristic points of the present invention;

[0046] Figure 3 This is a structural diagram of the DPT network model of the present invention;

[0047] Figure 4 Constructing a high-dimensional feature vector graph for the present invention by fusing semantic information;

[0048] Figure 5 Graphs showing some experimental results in the embodiments of the present invention. DETAILED DESCRIPTION

[0049] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0050] See also Figures 1 to 5 The present invention provides the following technical solution: a method for eliminating mismatches based on improving the consistency of a vector field, comprising the following steps:

[0051] Step S1: Use the DPT network to generate a semantic feature vector representing global information for each pixel in the input image, and use the decoder to upgrade the one-dimensional mark to a two-dimensional space to restore the spatial structure of the image.

[0052] The DPT network without the downstream task prediction head is used as the backbone to generate a semantic feature vector representing global information for each pixel in the input image. In the encoder stage, the input image is divided into several non-overlapping image blocks, and each image block is converted into a token with a positional encoding. This process is similar to the word segmentation of text in natural language processing (NLP) tasks.

[0053] The position encoding is used to identify the position of the word, and the visual Transformer uses a multi-head self-attention mechanism to process the identification, so that each identification is associated with all other identifications, thereby having a global receptive field at all stages of the network.

[0054] The decoder performs three operations in sequence: reading, splicing, and resampling, with the goal of upgrading the one-dimensional mark to a two-dimensional space to better restore the spatial structure of the image.

[0055] In this embodiment, firstly, the position code identifier is connected with all other identifiers, and the following formula is used to convert The high-dimensional vector is mapped back indivual;

[0056]

[0057] In the formula, is the identification feature, Records the location information of each mark, splicing operation Used to connect these identification features together, is a fully connected layer.

[0058] After the read operation, the obtained The spatial connection of the markers generates a , the number of channels is The feature map is reshaped into an image representation. Finally, through the resampling layer, the feature map is adjusted to :

[0059]

[0060]

[0061] in Represents the feature dimension after resampling, after spatial connection ( ) and resampling ( ) operation, the feature fusion module then fuses the feature bodies of different sizes and gradually upsamples them by two times in each fusion stage, and finally outputs the corresponding semantic feature vector for each pixel.

[0062] Step S2: Use the visual Transformer model to extract the global semantic information of the image and combine it with the local feature descriptors generated by the SIFT algorithm to form a higher dimensional feature vector containing local and global information.

[0063] The SIFT algorithm is used to construct local feature descriptors. In order to ensure that the feature points are rotationally invariant, the gradients and directions of all pixels in a certain area around the feature points are calculated, and a gradient histogram is constructed based on this. The highest peak in the histogram represents the main direction of the feature points.

[0064] For each detected feature point, The pixel neighborhood is divided into sub-regions, calculate the pixel gradient direction of each sub-region, and quantize it according to 8 directions to form a 128-dimensional feature descriptor vector. The calculation of gradient and direction can be performed by the following formula:

[0065]

[0066]

[0067] Step S3: Calculate the distribution density of key points relative to the initial matching set using the kernel density estimation method to determine candidate expandable key points.

[0068] For each key point in the data set, the point is taken as the center and a Gaussian kernel is used for smoothing. After normalizing the result, an estimate of the true probability density function is obtained. If the image Probability density of key points in If it is greater than a given threshold, it will be added to the candidate extended key point set. middle;

[0069]

[0070] in, Is an existing matching set The number of feature points in is the Gaussian kernel;

[0071]

[0072] Step S4: Use the Bayesian formula to perform matching expansion, and gradually build and optimize the matching set in an iterative manner.

[0073] In this embodiment, a progressive strategy using the Bayesian formula for matching expansion is adopted so that each sampling and model estimation can benefit from the results of the previous iteration, and the correctness of the matching points is probabilistically estimated and continuously updated based on prior knowledge and observed data.

[0074] First, a strict threshold nearest neighbor distance ratio (NNDR) is used to obtain a small but accurate and reliable initial matching set. , but in the process many potentially correct matches are excluded, such as those with relatively low NNDR values ​​or non-nearest neighbor relationships.

[0075] Then the matching set is gradually constructed and optimized in an iterative manner. This process comprehensively considers the spatial distribution of feature points, feature descriptors, and semantic vectors to form a larger extended matching set. The specific algorithm ideas are as follows:

[0076] Table 1

[0077]

[0078] The goal of match expansion is to use the initial match set and the initial transformation matrix For prior information, find a new matching set , this process can be achieved by maximizing the conditional probability , find the most likely correct set of matching points. According to the Bayesian formula, this conditional probability can be decomposed into two terms. For the first probability Represents a given initial matching set and the initial transformation matrix In the case of images The key points in are selected as extended key points The probability of the key point relative to The distribution density of the matching points is used to estimate, that is, the position of new potential matching points is predicted by the distribution of existing matching points;

[0079]

[0080] In this embodiment, the second probability Indicates that the image is found The corresponding extended key point in The probability of is calculated by considering descriptor similarity, semantic vector similarity and geometric constraints. Unmatched key points in , according to the initial vector field Make sure it is in the image In the corresponding projection area, find The most similar eigenvector , using cosine similarity as a measure to measure the similarity between two feature vectors, see formula 9. Perform reverse verification, that is, in the image Search The corresponding projection area of ​​​​ in agreement;

[0081]

[0082]

[0083] In this embodiment, the above matching expansion process is repeated until no new matching pairs are generated or the number of newly added matching pairs is lower than a certain threshold, or a set maximum number of iterations is reached.

[0084] Step S5: Solve the optimization problem using the EM algorithm, by alternating between the expectation step and the maximization step until the objective function converges, and finally output the vector field. Finding the maximum a posteriori solution can be equivalently converted into an energy function minimization problem, by using the standard EM algorithm and ignoring the parameters The irrelevant terms give the logarithmic posterior function of the complete data:

[0085]

[0086]

[0087] Step S5.1: Compute the expectation, based on the current parameter estimates , calculate the posterior distribution of the latent variable. Specifically, define a diagonal matrix , the posterior probability of each element is , indicating that given the observed data and the current parameter estimates Next, The hidden variable corresponding to the sample is 1. This reflects the The consistency between the samples and the current estimated vector field f, the posterior probability It can be calculated by Bayesian criterion:

[0088]

[0089] Step S5.2: Maximize the posterior probability of the sample point being an inlier point To update the parameter estimates The goal is to maximize the function ,in , considering that P is a diagonal matrix, about and Derivatives are taken separately and the derivatives are set to zero to obtain the update, where is the trace of the matrix;

[0090] σ 2 = tr[(YV ) T P(YV)] D⋅tr P ,γ= tr P N #

[0091] in, and Represents the matrix form of all observations and corresponding model output values, represents the trace of a matrix, which is the sum of the diagonal elements of the matrix.

[0092] Step S5.3: Iterate the above two steps until the EM algorithm converges and finally outputs the vector field f. Set a threshold , the sample points whose posterior probability is greater than the threshold are the inliers, forming the inlier set As the final correspondence.

[0093] The results of the present invention for removing mismatches from some experimental data are shown in the following example: Figure 5 As shown, it can be seen that the present invention can stably and accurately output the correspondence between image feature points.

[0094] The basic principles and main features of the present invention and the advantages of the present invention are shown and described above. For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention.

[0095] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for eliminating false matches based on improving the consistency of vector fields, characterized in that: The following steps are involved: S1: The DPT network is used to generate a semantic feature vector representing global information for each pixel in the input image, and the decoder is used to upgrade the one-dimensional mark to a two-dimensional space to restore the spatial structure of the image; S2: Use the visual Transformer model to extract the global semantic information of the image and combine it with the local feature descriptors generated by the SIFT algorithm to form a higher-dimensional feature vector containing local and global information; S3: Calculate the distribution density of key points relative to the initial matching set using the kernel density estimation method to determine candidate expandable key points; S4: Use the Bayesian formula to expand the matching, and gradually build and optimize the matching set in an iterative manner; S5: Use the EM algorithm to solve the optimization problem by alternating between expectation steps and maximization steps until the objective function converges and finally outputs a vector field.

2. The method for eliminating false matches based on improving vector field consistency according to claim 1, characterized in that: The solution optimization steps included in the EM algorithm are as follows: S5.1: Compute the expectation, based on the current parameter estimates, and compute the posterior distribution of the latent variable; S5.2: Perform a maximization operation and use the posterior distribution of the sample points as inliers to update the parameter estimates; S5.3: Iterate the above two steps until the EM algorithm converges and finally outputs the vector field f.

3. The method for eliminating false matches based on improving vector field consistency according to claim 1, characterized in that: The encoding operation process of the decoder in S1 is as follows: the input image is divided into several non-overlapping image blocks, and each image block is converted into a mark with a position code. This process is similar to the word segmentation of text in natural language processing tasks, where the position code is used to identify the position of the vocabulary. The visual Transformer uses a multi-head self-attention mechanism to process the mark, so that each mark is associated with all other marks, so that it has a global receptive field at all stages of the network. The decoder performs three operations of reading, splicing and resampling in sequence, with the purpose of upgrading the one-dimensional mark to a two-dimensional space to better restore the spatial structure of the image; first, by connecting the position code mark with all other marks, the following formula is used to convert N p +1 high-dimensional vector mapped back to N p indivual: In the formula, is the identification feature, where t0 records the position information of each identifier, the concatenation operation cat(·) is used to connect these identification features together, and mlp(·) is the fully connected layer.

4. The method for eliminating false matches based on improving vector field consistency according to claim 3, characterized in that: After the read operation, the obtained N is compared according to the position of the initial block in the image. p The labels are spatially connected to generate a feature map of size H / p×W / p and number of channels D, which is reshaped into the representation of the image. Finally, the feature map is adjusted to H / s×W / s through the resampling layer: in Represents the feature dimension after resampling, after spatial connection (F concat ) and resampling (F rsmp ) operation, the feature fusion module then fuses the feature bodies of different sizes and gradually upsamples them by two times in each fusion stage, and finally outputs the corresponding semantic feature vector for each pixel.

5. The method for eliminating false matches based on improving vector field consistency according to claim 1, characterized in that: The operation of combining the global semantic information of the image extracted by the visual Transformer model with the local feature descriptor generated by the SIFT algorithm in S2 is: The SIFT algorithm is used to construct local feature descriptors. In order to ensure that the feature points are rotationally invariant, the gradients and directions of all pixels in a certain area around the feature points are calculated, and a gradient histogram is constructed based on this. The highest peak in the histogram represents the main direction of the feature point. For each detected feature point, its 16×16 pixel neighborhood is divided into 4×4 sub-regions, and the pixel gradient direction of each sub-region is calculated and quantized according to 8 directions to form a 128-dimensional feature descriptor vector. The calculation of gradient and direction is performed using the following formula:

6. The method for eliminating false matches based on improving vector field consistency according to claim 2, characterized in that: The matching extension based on the Bayesian formula in S4 is: Through a progressive strategy that uses the Bayesian formula for matching expansion, each sampling and model estimation can benefit from the results of the previous iteration, and the correctness of the matching points is estimated and continuously updated based on prior knowledge and observed data; Firstly, the nearest neighbor distance ratio with a strict threshold is used to obtain an initial matching set M0 with a small amount of data but accurate and reliable. Then, the matching set is gradually constructed and optimized in an iterative manner. This process comprehensively considers the spatial distribution of feature points, feature descriptors and semantic vector information, thereby forming a larger extended matching set M'.

7. The method for eliminating false matches based on improving vector field consistency according to claim 6, characterized in that: The goal of matching expansion is to find a new matching set M based on the initial matching set M0 and the initial transformation matrix E0 as prior information. add , this process maximizes the conditional probability p(M add |E0,M0), find the most likely correct set of matching points. According to the Bayesian formula, this conditional probability is decomposed into two terms. For the first probability Indicates that given the initial matching set M0 and the initial transformation matrix E0, the key points in image A are selected as extended key points The probability is estimated by calculating the distribution density of key points relative to M0, that is, predicting the position of new potential matching points by the distribution of existing matching points: The second probability Indicates finding the corresponding extended key point in image B The probability of K is calculated by considering the descriptor similarity, semantic vector similarity and geometric constraints. A Unmatched key points in According to the initial vector field f0, determine its corresponding projection area in image B, and find the corresponding The most similar eigenvector The cosine similarity is used as a measure to measure the similarity between two feature vectors. add Perform reverse verification, that is, search in image A The corresponding projection area of ​​​​ in agreement; The above matching expansion process is repeated until no new matching pairs are generated or the number of newly added matching pairs is lower than a certain threshold, or the set maximum number of iterations is reached.

8. The method for eliminating false matches based on improving vector field consistency according to claim 2, characterized in that: In S5.1, the expected value is calculated as: Based on the current parameter estimates θ odd , calculate the posterior distribution of the latent variable. Specifically, define a diagonal matrix P = diag(p1,…p N ), the posterior probability of each element is p n =P(z n =1|x n ,y n ,θ old ), which means that given the observed data (x n ,y n ) and the current parameter estimate θ odd Next, the hidden variable z corresponding to the nth sample n The probability of being 1 reflects the consistency between the nth sample and the currently estimated vector field f. The posterior probability p n Calculated by Bayesian criterion:

9. The method for eliminating false matches based on improving the consistency of vector fields according to claim 8, characterized in that: The maximization steps performed in S5.2 are as follows: Using the posterior probability p of the sample point as an internal point n To update the parameter estimate θ new , the goal is to maximize the function in Considering that P is a diagonal matrix, About σ 2 We take the derivatives of and γ and set them equal to zero to get the update, where tr(·) is the trace of the matrix: Among them, Y and V represent the matrix forms of all observations and the corresponding model output values, respectively, and tr(·) represents the trace of the matrix, that is, the sum of the diagonal elements of the matrix elements.

Images