Multi-modal image point feature fast and accurate matching method based on auxiliary navigation information

Abstract

The application discloses a kind of multi-modal image point feature fast and accurate matching method based on auxiliary navigation information, comprising: the coarse matching result of feature point on real-time graph and reference graph is obtained, including the feature point set corresponding to real-time graph, reference graph, the mixed distance of two row and column direction is calculated, the mixed distance and mixed direction of each feature point pair are determined;In combination with preset threshold, construct parallel degree criterion;Initialize feature screening array, traverse between different feature point pairs;In the process of traversal, screening is carried out using parallel degree criterion, the number of feature point pairs meeting parallel degree criterion is recorded in the corresponding position of feature screening array;Determine the serial number at the maximum value in feature screening array, and re-traverse other feature point pairs with the serial number, the serial number of all feature point pairs meeting parallel degree criterion is stored in a set;Then, after traversal is completed, the corresponding feature point pair in the feature point set of real-time graph and reference graph in all serial numbers in the set is accurate matching result.

Description

Technical Field

[0001] This invention relates to the field of scene matching technology, and specifically to a fast and accurate matching method for multimodal image point features based on assisted navigation information. Background Technology

[0002] With the rapid development of visual scene matching navigation and multimodal remote sensing image fusion, research on multimodal image matching is gaining momentum both domestically and internationally. Examples include "real-time infrared image + visible light reference image" in scene matching navigation and "SAR remote sensing image + visible light reference image" in remote sensing image fusion. For these engineering problems, point features + region descriptors are currently the mainstream solution. This method completes the matching task through three sequential steps: point feature and region descriptor extraction, coarse matching, and fine matching. However, due to differences in image sensor information sources and imaging weather conditions, the similarity of point feature descriptors is often difficult to reach the level of images from the same source. This further leads to a high proportion of mismatches in the point pairs obtained from coarse matching, significantly increasing the time consumption of the fine matching process.

[0003] However, for engineering tasks such as scene matching navigation, the time spent on scene matching directly determines the robustness of the navigation task. If the update frequency is too low, it will lead to serious divergence in the navigation system. Summary of the Invention

[0004] The purpose of this invention is to provide a fast and accurate multimodal image point feature matching method based on auxiliary navigation information for point feature matching of heterogeneous images, so as to ensure the real-time performance of the matching.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] A fast and accurate matching method for multimodal image point features based on assisted navigation information includes:

[0007] Obtain coarse matching results of feature points on the real-time image and the baseline image, including the feature point sets corresponding to the real-time image and the baseline image;

[0008] Based on the feature point sets corresponding to the real-time image and the baseline image, the mixed distance in the row and column directions of the two is calculated.

[0009] The mixing distance and mixing direction of each feature point pair are calculated using the row and column mixing distance.

[0010] By combining distance and direction with a preset threshold, a parallelism criterion is constructed.

[0011] Initialize the feature filtering array, the length of which is the same as the number of feature point pairs;

[0012] Perform traversal between different feature point pairs; during the traversal, use the parallelism criterion to filter and record the number of feature point pairs that meet the parallelism criterion in the corresponding position of the feature filtering array.

[0013] Determine the index of the maximum value in the feature filtering array, and then re-traverse other feature point pairs using that index. Store the indices of all feature point pairs that satisfy the parallelism criterion in a set. After the traversal is completed, the feature point pairs corresponding to all indices in the set in the real-time and baseline images are the fine matching results.

[0014] Furthermore, based on the feature point sets corresponding to the real-time image and the baseline image, the mixed distance in the row and column directions of the two is calculated, expressed as:

[0015]

[0016] Where lx[i] and ly[i] are the mixed distances of the i-th feature point in the row and column directions, respectively, and C is the number of columns in the real-time graph; ptRt[i].x and ptBenc[i].x represent the row positions of feature points ptRt[i] and ptBenc[i] in the feature point sets of the real-time graph and the baseline graph, respectively; ptRt[i].y and ptBenc[i].y represent the column positions of feature points ptRt[i] and ptBenc[i], respectively.

[0017] Furthermore, the mixing distance and mixing direction of each feature point pair are calculated using the row and column mixing distance, and are expressed as follows:

[0018]

[0019] Where L[i] and A[i] represent the mixing distance and mixing direction of feature point pair i, respectively.

[0020] Furthermore, by combining the distance and direction of the mixture with a preset threshold, a parallelism criterion is constructed, including:

[0021] The parallelism criterion includes two parts, the first being the proximity criterion:

[0022] |L[i]-L[j]|<t1

[0023] The second is the principle of similar angles:

[0024] |A[i]-A[j]|<t2

[0025] Where t1 and t2 are preset thresholds, and L[j] and A[j] are the mixing distance and mixing direction of feature point pair i.

[0026] Furthermore, the accuracy of the fine matching results can be adjusted by adjusting the preset threshold.

[0027] Furthermore, the process involves traversing different feature point pairs; during the traversal, a parallelism criterion is used for filtering, and the number of feature point pairs that satisfy the parallelism criterion is recorded in the corresponding position of the feature filtering array, including:

[0028] Execute a double loop, with the outer loop iterating through i∈[1,N] and the inner loop iterating through j∈[1,N]. After completion, assign values ​​to the positions of each element in the feature filtering array Π. The value of the i-th element Π(i) in array Π is incremented by 1 if and only if the proximity criterion and the angle proximity criterion are satisfied simultaneously. N is the number of feature point pairs.

[0029] Furthermore, the index of the maximum value in the feature filtering array is determined, and other feature point pairs are re-traversed using this index. The indices of all feature point pairs that satisfy the parallelism criterion are stored in a set, including:

[0030] Perform a maximum value determination to obtain the index of the feature point pair that makes the array Π the maximum value, and assign the result to the variable I; let i = I, and re-traverse j∈[1,N], and make a determination according to the parallelism criterion, and put the j that satisfy the criterion into the set K.

[0031] Furthermore, if there are multiple maximum values, the smallest index among the multiple maximum values ​​is assigned to I.

[0032] A terminal device includes a processor, a memory, and a computer program stored in the memory; when the processor executes the computer program, it implements the fast and precise matching method for multimodal image point features based on assisted navigation information.

[0033] A computer-readable storage medium storing a computer program; when executed by a processor, the computer program implements the fast and precise matching method for multimodal image point features based on assisted navigation information.

[0034] Compared with the prior art, the present invention has the following technical features:

[0035] 1. This invention provides a novel fast and precise matching method for multimodal image point features, which can achieve rapid matching of point features. Compared with the classic RANSAC and FSC precise matching methods, its advantage lies in its extremely fast speed.

[0036] 2. This invention has the function of adjusting the precision of fine matching. The precision of fine matching results can be adjusted by two thresholds of the parallelism criterion, which is equivalent to the threshold design of the number of iterations in fine matching methods such as RANSAC and FSC.

[0037] 3. This invention has undergone extensive experimental verification. Regarding the matching problem between 250*250 pixel infrared and 650*650 pixel light, on the Nvidia-NX computing platform, the average fine matching time of this invention is 3ms and the matching accuracy is 1.1 pixels; while the matching time and accuracy of RANSAC are 97ms and 1.7 pixels, respectively, and the matching time and accuracy of FSC are 114ms and 1.4 pixels, respectively, demonstrating the advantages of this invention in terms of speed and accuracy. Attached Figure Description

[0038] Figure 1 This is an overall flowchart of the present invention;

[0039] Figure 2 A set of examples for the input of this invention;

[0040] Figure 3 The precise matching result of the present invention for the example described above;

[0041] Figure 4 The result of using the FSC fine matching method for the example described;

[0042] Figure 5 The result is the result of using the RANSAC fine matching method for the example described.

[0043] Figure 6 This is the second example of inputting data for this invention;

[0044] Figure 7 This is the result of a precise match for Sample 2. Detailed Implementation

[0045] In multimodal image matching tasks, due to differences in image sensor information sources, the similarity of point feature descriptors is often difficult to reach the level of source images. This results in a relatively high proportion of incorrectly matched point pairs in the input image point pairs for the fine matching step. Traditional point feature fine matching methods rely on iterative loops to filter out correct point pairs, such as RANSAC (Random Sample Consensus Algorithm) and FAC (Fast Sample Consensus Algorithm). A high proportion of incorrectly matched point pairs leads to a large number of iterations, significantly increasing the time consumption of the fine matching process. To address this problem, this invention designs a fast fine matching method for multimodal image point features based on auxiliary navigation information. The method uses auxiliary navigation information to correct the real-time image, ensuring that its rotation and scale are the same as the reference image. Subsequently, a coarse matching of feature points is performed between the corrected real-time image and the reference image to obtain the corresponding feature point set. Then, the following fine matching process is performed:

[0046] Algorithm input: The set of feature points ptRt on the real-time graph and the set of feature points ptBenc on the baseline graph. The correspondence between the two is as follows:

[0047] (ptRt[i](x,y),ptBenc[i](x,y)),i∈[1,N] (1)

[0048] Where ptRt[i] is the i-th feature point in the feature point set ptRt, and ptBenc[i] is the i-th feature point in the feature point set ptBenc; ptRt[i](x,y) represents the position of feature point ptRt[i] on the real-time graph, ptBenc[i](x,y) represents the position of feature point ptBenc[i] on the baseline graph, (x,y) are the row position and column position respectively, and N is the number of feature point pairs.

[0049] Based on the above input feature point sets ptRt and ptBenc, the mixing distance between them in the row and column directions is first calculated:

[0050]

[0051] Where lx[i] and ly[i] are the mixed distances of the i-th feature point in the row and column directions, respectively, and C is the number of columns in the real-time graph; ptRt[i].x and ptBenc[i].x represent the row positions of feature points ptRt[i] and ptBenc[i], respectively; ptRt[i].y and ptBenc[i].y represent the column positions of feature points ptRt[i] and ptBenc[i], respectively.

[0052] Formula (2) can also be replaced by:

[0053]

[0054] Where D represents the number of columns in the baseline chart.

[0055] Calculate the mixing distance L[i] and mixing direction A[i] of the i-th feature point pair based on the row and column mixing distances lx[i] and ly[i].

[0056]

[0057] Initialize the feature filtering array Π; array Π is a one-dimensional array, and its length is the same as the number of feature point pairs, used to store the number of feature point pairs that satisfy the parallelism criterion:

[0058] Π(i)=0,i∈[1,N] (5)

[0059] Where Π(i) represents the i-th element in the array Π; N represents the number of feature point pairs in the feature point sets ptRt and ptBenc.

[0060] For the i-th feature point pair, traverse the 1st to Nth feature point pairs, denoted by the traversal index j; the judgment is based on the parallelism criterion; the parallelism criterion includes two elements, the first being the proximity criterion:

[0061] |L[i]-L[j]|<t1 (6) The second criterion is the similarity of angles:

[0062] |A[i]-A[j]|<t2 (7)

[0063] Where t1 and t2 are preset thresholds, which are set to 4 and 0.3 respectively in this scheme; the accuracy of the fine matching result can be adjusted by adjusting the preset thresholds.

[0064] The value of the i-th element Π(i) in array Π is incremented by 1 if and only if both criteria are satisfied simultaneously:

[0065] Π(i)=Π(i)+1 (8)

[0066] As described above, a double loop is executed, with the outer loop iterating through i∈[1,N] and the inner loop iterating through j∈[1,N]. After completion, the positions of each element in the feature filtering array Π are assigned.

[0067] Then, the maximum value is determined, the index of the feature point pair that results in the maximum value of array Π is obtained, and the result is assigned to variable I:

[0068]

[0069] Here, argmax is a function that finds the position of the maximum value.

[0070] If there are multiple maximum values, then the smallest index among the multiple maximum values ​​is assigned to I.

[0071] Let i = I, and re-traverse j∈[1,N]. Determine the parallelism criteria according to equations (6) and (7), and place the j that satisfy the criteria in set K. After the traversal is completed, the feature point pairs corresponding to all indices in the feature point sets ptRt and ptBenc in the final set K are the fine matching results, as shown below:

[0072] (ptRt_r[i](x,y),ptBenc_r[i](x,y)),i∈K (10)

[0073] Where ptRt_r[i](x,y) represents the position of the feature points after coarse matching and filtering of the feature point set ptRt on the real-time graph, and ptBenc_r[i](x,y) represents the position of the feature points after coarse matching and filtering of the feature point set ptBenc on the baseline graph. The two correspond one-to-one, that is, they match each other. Finally, the homography matrix H is calculated based on the feature point pairs to complete the fast matching.

[0074] Example:

[0075] In this embodiment, Figure 6 The given example illustrates this: the real-time graph and the baseline graph have row and column dimensions of 200*200 and 500*500 respectively. Figure 6 The numbers 1 to 8 in the table represent 1 to 8 pairs of matching points, i.e., N = 8. The coordinates (x, y) of the matching point pairs are as follows:

[0076] (50,180)—(60,450); (70,180)—(80,450); (50,35)—(75,103); (70,35)—(212,220); (14,77)—(156,362); (170,150)—(312,335); (170,4)—(312,189); (20,152)—(398,76).

[0077] The left side shows the coordinates of points on the real-time map, and the right side shows the coordinates of points on the baseline map.

[0078] Using the method of this invention, the obtained feature filtering array Π is [2,2,1,4,4,4,4,1], and the result of I is 4; the K set is [4,5,6,7], that is, the 4th, 5th, 6th, and 7th matching point pairs are the feature point pairs obtained by the fine matching result of this scheme. The final fine matching result is as follows. Figure 7 As shown. Finally, the homography matrix H is calculated based on the feature point pairs, thus obtaining the fine matching result; in addition, the accuracy of the fine matching result can be adjusted by adjusting the preset thresholds t1 and t2 in formulas (6) and (7).

[0079] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A fast and precise matching method for multimodal image point features based on auxiliary navigation information, characterized in that, include: Obtain coarse matching results of feature points on the real-time image and the baseline image, including the feature point sets corresponding to the real-time image and the baseline image; Based on the feature point sets corresponding to the real-time image and the baseline image, the mixed distance in the row and column directions of the two is calculated. The mixing distance and mixing direction of each feature point pair are calculated using the row and column mixing distance. By combining distance and direction with a preset threshold, a parallelism criterion is constructed, including: The parallelism criterion includes two parts, the first being the proximity criterion: The second is the principle of similar angles: in, and It is a preset threshold. and For the first i The mixing distance and mixing direction of the nth feature point pair; for the nth feature point pair... i For each pair of feature points, iterate through the first to... N There are 1 pair of feature points, and their traversal index is denoted as . j ; Initialize the feature filtering array, the length of which is the same as the number of feature point pairs; Perform traversal between different feature point pairs; during the traversal, use the parallelism criterion to filter, and record the number of feature point pairs that meet the parallelism criterion in the corresponding position of the feature filtering array, including: Execute a nested loop, with the outer loop iterating through... Inner loop traversal After completion, the feature filtering array is finished. Assigning values ​​to the positions of each element; the array is set only if both the proximity criterion and the angle proximity criterion are satisfied simultaneously. The first in element Increment the value by 1; N The number of feature point pairs; Determine the index of the maximum value in the feature filtering array, and use this index to re-traverse other feature point pairs. Store the indices of all feature point pairs that satisfy the parallelism criterion in a set, including: Perform a maximum value determination to obtain the array The index of the feature point pair at the maximum value is assigned to the variable. I If multiple maximum values ​​exist, then the smallest index among the multiple maximum values ​​is assigned to... I ;make i = I and re-traverse The determination is made according to the parallelism criterion; those that satisfy the criterion... j Place in set K After the traversal is complete, the set... K The feature point pairs corresponding to all the serial numbers in the feature point sets of the real-time and baseline images are the fine matching results.

2. The fast and precise matching method for multimodal image point features based on assisted navigation information according to claim 1, characterized in that, Based on the feature point sets corresponding to the real-time image and the baseline image, the mixed distance in the row and column directions of the two is calculated and expressed as: in, , The first i The feature points are mixed distances in the row and column directions. C This represents the number of columns in the real-time chart. , These represent feature points in the feature point sets of the real-time graph and the baseline graph, respectively. , The row position; , Representing feature points respectively , The column position.

3. The fast and precise matching method for multimodal image point features based on assisted navigation information according to claim 2, characterized in that, The mixing distance and mixing direction of each feature point pair are calculated using the row and column mixing distance, and are expressed as follows: 。 4. The fast and precise matching method for multimodal image point features based on assisted navigation information according to claim 1, characterized in that, The accuracy of the fine matching results can be adjusted by adjusting the preset threshold.

5. A terminal device, comprising a processor, a memory, and a computer program stored in the memory; characterized in that, When the processor executes the computer program, it implements the fast and precise matching method for multimodal image point features based on assisted navigation information as described in any one of claims 1-4.

6. A computer-readable storage medium storing a computer program; characterized in that, When the computer program is executed by the processor, it implements the fast and precise matching method for multimodal image point features based on assisted navigation information as described in any one of claims 1-4.

Images