Multi-modal image point feature rapid and precise matching method based on auxiliary navigation information

Abstract

The invention discloses a multi-modal image point feature rapid fine matching method based on auxiliary navigation information, and the method comprises the steps: obtaining a rough matching result of feature points on a real-time image and a reference image, including feature point sets corresponding to the real-time image and the reference image, calculating the mixing distance of the real-time image and the reference image in the row and column directions, and determining the mixing distance and the mixing direction of each feature point pair; constructing a parallelism criterion in combination with a preset threshold; initializing a feature screening array, and traversing different feature point pairs; in the traversing process, the parallelism criterion is used for screening, and the number of the feature point pairs meeting the parallelism criterion is recorded at the corresponding positions of the feature screening array; determining a serial number at the maximum value in the feature screening array, re-traversing other feature point pairs according to the serial number, and storing the serial numbers of all the feature point pairs meeting the parallelism criterion in a set; and after traversal is completed, feature point pairs corresponding to all the serial numbers in the set in the feature point set of the real-time image and the reference image are fine matching results.

Description

Technical Field

[0001] The present invention relates to the technical field of scene matching, and in particular to a method for fast and precise matching of multimodal image point features based on auxiliary navigation information. Background Art

[0002] With the rapid development of fields such as visual scene matching navigation and multimodal remote sensing image fusion, research on multimodal image matching is gaining momentum both domestically and internationally. For example, the "infrared real-time image + visible light reference image" approach in scene matching navigation and the "SAR remote sensing image + visible light reference image" approach in remote sensing image fusion are examples. Currently, the mainstream solution to these engineering problems is point features + region descriptors. This approach completes the matching task through three sequential steps: extraction of point features and region descriptors, coarse matching, and fine matching. However, due to differences in image sensor information sources, imaging weather conditions, and other factors, the similarity of point feature descriptors often struggles to reach the same level as that of homologous images. This leads to a high proportion of mismatches in the coarse matching point pairs, significantly increasing the time required for the fine matching process.

[0003] However, for engineering tasks such as scene matching navigation, the time consumption of scene matching directly determines the robustness of the navigation task. If the update frequency is too low, the navigation system will diverge seriously. Summary of the Invention

[0004] The purpose of the present invention is to provide a multimodal image point feature fast and precise matching method based on auxiliary navigation information for point feature matching of heterogeneous images, so as to ensure the real-time performance of the matching.

[0005] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0006] A fast and precise matching method for multimodal image point features based on auxiliary navigation information includes:

[0007] Obtaining a rough matching result of feature points on the real-time image and the reference image, including a set of feature points corresponding to the real-time image and the reference image;

[0008] Based on the feature point sets corresponding to the real-time image and the reference image, the mixed distance between the two in row and column directions is calculated;

[0009] The mixed distance and mixed direction of each feature point pair are calculated using the mixed distance in row and column directions;

[0010] By mixing distance and mixed direction and combining them with preset thresholds, a parallelism criterion is constructed;

[0011] Initialize the feature screening array. The length of the feature screening array is the same as the number of feature point pairs.

[0012] Traverse between different feature point pairs; use the parallelism criterion to screen during the traversal process, and record the number of feature point pairs that meet the parallelism criterion in the corresponding position of the feature screening array;

[0013] Determine the serial number of the maximum value in the feature screening array, and re-traverse other feature point pairs with this serial number, and store the serial numbers of all feature point pairs that meet the parallelism criterion in a set; after the traversal is completed, the feature point pairs corresponding to all serial numbers in the set in the feature point sets of the real-time image and the reference image are the precise matching results.

[0014] Furthermore, based on the feature point sets corresponding to the real-time image and the reference image, the mixed distance between the two in row and column directions is calculated, which is expressed as:

[0015]

[0016] Among them, lx[i] and ly[i] are the mixed distances of the i-th feature point in the row and column directions respectively, C is the number of columns of the real-time image; ptRt[i].x and ptBenc[i].x represent the row positions of the feature points ptRt[i] and ptBenc[i] in the feature point sets of the real-time image and the reference image respectively; ptRt[i].y and ptBenc[i].y represent the column positions of the feature points ptRt[i] and ptBenc[i] respectively.

[0017] Furthermore, the mixed distance in row and column directions is used to calculate the mixed distance and mixed direction of each feature point pair, which can be expressed as:

[0018]

[0019] Among them, L[i] and A[i] represent the mixing distance and mixing direction of feature point pair i respectively.

[0020] Furthermore, by mixing distance and mixed direction and combining with preset thresholds, a parallelism criterion is constructed, including:

[0021] There are two parallelism criteria. The first one is the distance criterion:

[0022] |L[i]-L[j]|<t1

[0023] The second is the angle proximity criterion:

[0024] |A[i]-A[j]|<t2

[0025] Among them, t1 and t2 are preset thresholds, L[j] and A[j] are the mixing distance and mixing direction of feature point pair i.

[0026] Furthermore, the accuracy of the precise matching result is adjusted by adjusting the preset threshold.

[0027] Furthermore, different feature point pairs are traversed. During the traversal process, the parallelism criterion is used for screening, and the number of feature point pairs that meet the parallelism criterion is recorded in the corresponding position of the feature screening array, including:

[0028] Execute a double loop, the outer loop traverses i∈[1,N], and the inner loop traverses j∈[1,N]. After the end, the position of each element of the feature screening array Π is assigned; if and only if the distance similarity criterion and the angle similarity criterion are met at the same time, the value of the i-th element Π(i) in the array Π is increased by 1; N is the number of feature point pairs.

[0029] Furthermore, the sequence number of the maximum value in the feature screening array is determined, and the sequence number is used to traverse other feature point pairs again, and the sequence numbers of all feature point pairs that meet the parallelism criterion are stored in a set, including:

[0030] Perform maximum value determination to obtain the serial number of the feature point pair that makes the array π at the maximum value, and assign the result to the variable I; let i = I, and re-traverse j∈[1,N], determine according to the parallelism criterion, and put the j that meets the criterion into the set K.

[0031] Furthermore, if there are multiple maximum values, the smallest serial number among the multiple maximum values ​​is assigned to 1.

[0032] A terminal device includes a processor, a memory, and a computer program stored in the memory; when the processor executes the computer program, the method for fast and precise matching of multimodal image point features based on auxiliary navigation information is implemented.

[0033] A computer-readable storage medium stores a computer program; when the computer program is executed by a processor, the method for fast and precise matching of multimodal image point features based on auxiliary navigation information is implemented.

[0034] Compared with the prior art, the present invention has the following technical features:

[0035] 1. The present invention provides a novel multimodal image point feature fast and precise matching method, which can achieve fast matching of point features. Compared with the classic RANSAC and FSC precise matching methods, the advantage is that it is very fast.

[0036] 2. The present invention has the function of adjusting the precision of precise matching. The precision of precise matching results can be adjusted by two thresholds of the parallelism criterion, which is equivalent to the threshold design of the number of iterations in precise matching methods such as RANSAC and FSC.

[0037] 3. The present invention has been verified through a large number of experiments. For the problem of matching 250*250 pixel infrared and 650*650 pixel visible light, on the Nvidia-NX computing platform, the average precise matching time of the present invention is 3ms, and the matching accuracy is 1.1 pixels; while the matching time and accuracy of RANSAC are 97ms and 1.7 pixels respectively, and the matching time and accuracy of FSC are 114ms and 1.4 pixels respectively, showing the advantages of the present invention in speed and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is the overall flow chart of the present invention;

[0039] Figure 2 a set of examples input for the present invention;

[0040] Figure 3 The precise matching results of the present invention for the example;

[0041] Figure 4 The results of the FSC precise matching method are used for the sample;

[0042] Figure 5 is the result of using the RANSAC precise matching method for the example;

[0043] Figure 6 Input Sample 2 for the present invention;

[0044] Figure 7 This is the exact matching result for sample 2. DETAILED DESCRIPTION

[0045] In multimodal image matching tasks, due to differences in image sensor information sources, the similarity of point feature descriptors is usually difficult to reach the level of homologous images, resulting in a relatively high proportion of incorrectly matched point pairs in the image point pairs input in the precise matching step. Traditional point feature precise matching methods all rely on cyclic iterations to screen out correct point pairs, such as RANSAC (Random Sampling Consensus Algorithm) and FAC (Fast Sampling Consensus Algorithm). A high proportion of incorrectly matched point pairs will result in a large number of cyclic iterations, which will consume a lot of time in the precise matching process. To address this problem, the present invention designs a multimodal image point feature fast precise matching method based on auxiliary navigation information. The auxiliary navigation information is used to correct the real-time image so that it has the same rotation and scale as the reference image. Subsequently, the corrected real-time image is used to perform a rough matching of feature points with the reference image to obtain a corresponding feature point set. The following precise matching process is then performed:

[0046] Algorithm input: the feature point set ptRt on the real-time image and the feature point set ptBenc on the reference image. The corresponding relationship between the two is expressed as follows:

[0047] (ptRt[i](x,y),ptBenc[i](x,y)),i∈[1,N] (1)

[0048] Among them, ptRt[i] is the i-th feature point in the feature point set ptRt, ptBenc[i] is the i-th feature point in the feature point set ptBenc; ptRt[i](x,y) represents the position of the feature point ptRt[i] on the real-time image, ptBenc[i](x,y) represents the position of the feature point ptBenc[i] on the reference image, (x,y) are the row position and column position respectively, and N is the number of feature point pairs.

[0049] Based on the above input feature point sets ptRt and ptBenc, first calculate the mixed distance between the two in row and column directions:

[0050]

[0051] Among them, lx[i] and ly[i] are the mixed distances of the i-th feature point in the row and column directions respectively, C is the number of columns in the real-time image; ptRt[i].x and ptBenc[i].x represent the row positions of the feature points ptRt[i] and ptBenc[i] respectively; ptRt[i].y and ptBenc[i].y represent the column positions of the feature points ptRt[i] and ptBenc[i] respectively.

[0052] Among them, formula (2) can also be replaced by:

[0053]

[0054] Where D represents the number of columns in the benchmark graph.

[0055] Calculate the mixing distance L[i] and mixing direction A[i] of the i-th feature point pair based on the row and column direction mixing distances lx[i] and ly[i]:

[0056]

[0057] Initialize the feature screening array π; the array π is a one-dimensional array, the length of the array π is the same as the number of feature point pairs, and is used to store the number of feature point pairs that meet the parallelism criterion:

[0058] Π(i)=0,i∈[1,N] (5)

[0059] Among them, Π(i) represents the i-th element in the array Π; N represents the number of feature point pairs in the feature point sets ptRt and ptBenc.

[0060] For the i-th feature point pair, traverse the 1st to N-th feature point pairs, and the traversal sequence is recorded as j; the judgment is made based on the parallelism criterion; the parallelism criterion includes two, the first one is the close distance criterion:

[0061] |L[i]-L[j]|<t1 (6) The second is the angle similarity criterion:

[0062] |A[i]-A[j]|<t2 (7)

[0063] Among them, t1 and t2 are preset thresholds, which are set to 4 and 0.3 respectively in this solution; the accuracy of the precise matching result can be adjusted by adjusting the preset thresholds.

[0064] If and only if both criteria are met, the value of the i-th element Π(i) in the array Π is increased by 1:

[0065] Π(i)=Π(i)+1 (8)

[0066] As described above, a double loop is executed, the outer loop traverses i∈[1,N], and the inner loop traverses j∈[1,N]. After the end, the position of each element of the feature screening array π is assigned.

[0067] Then the maximum value is determined to obtain the sequence number of the feature point pair that makes the array π the maximum value, and the result is assigned to the variable I:

[0068]

[0069] Here, argmax is a function for finding the location of the maximum value.

[0070] If there are multiple maximum values, the smallest serial number among the multiple maximum values ​​is assigned to I.

[0071] Let i = I, and re-traverse j∈[1,N], and make a judgment based on the parallelism criteria of equations (6) and (7). The j that meets the criteria is placed in the set K. After the traversal is completed, the feature point pairs corresponding to all the sequence numbers in the final set K in the feature point sets ptRt and ptBenc are the exact matching results, which are expressed as follows:

[0072] (ptRt_r[i](x,y),ptBenc_r[i](x,y)),i∈K (10)

[0073] Among them, ptRt_r[i](x,y) is the position of the feature point after coarse matching and screening of the feature point set ptRt on the real-time image, and ptBenc_r[i](x,y) is the position of the feature point after coarse matching and screening of the feature point set ptBenc on the reference image. The two correspond one to one, that is, they match each other. Finally, the homography matrix H is calculated based on the feature point pairs to complete fast matching.

[0074] Example:

[0075] In this embodiment, Figure 6 The example given is used for illustration. The row and column sizes of the real-time graph and the benchmark graph are 200*200 and 500*500 respectively. Figure 6 The numbers 1 to 8 in represent 1 to 8 pairs of matching points, that is, N = 8, and the coordinates (x, y) of the matching point pairs are:

[0076] (50,180)—(60,450); (70,180)—(80,450); (50,35)—(75,103); (70,35)—(212,220); (14,77)—(156,362); (170,150)—(312,335); (170,4)—(312,189); (20,152)—(398,76).

[0077] Among them, the left side is the coordinates of the point on the real-time map, and the right side is the coordinates of the point on the reference map.

[0078] Using the method of the present invention, the feature screening array Π is [2,2,1,4,4,4,4,1], and the result of I is 4; the K set is [4,5,6,7], that is, the 4th, 5th, 6th, and 7th matching point pairs are the feature point pairs obtained by the precise matching result of this scheme. The final precise matching result is as follows: Figure 7 Finally, the homography matrix H is calculated based on the feature point pairs, and the precise matching result is obtained. In addition, the accuracy of the precise matching result can be adjusted by adjusting the preset thresholds t1 and t2 in formulas (6) and (7).

[0079] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A multimodal image point feature fast and precise matching method based on auxiliary navigation information, characterized by: include: Obtaining a rough matching result of feature points on the real-time image and the reference image, including a set of feature points corresponding to the real-time image and the reference image; Based on the feature point sets corresponding to the real-time image and the reference image, the mixed distance between the two in row and column directions is calculated; The mixed distance and mixed direction of each feature point pair are calculated using the mixed distance in row and column directions; By mixing distance and mixed direction and combining them with preset thresholds, a parallelism criterion is constructed; Initialize the feature screening array. The length of the feature screening array is the same as the number of feature point pairs. Traverse between different feature point pairs; use the parallelism criterion to screen during the traversal process, and record the number of feature point pairs that meet the parallelism criterion in the corresponding position of the feature screening array; Determine the serial number of the maximum value in the feature screening array, and re-traverse other feature point pairs with this serial number, and store the serial numbers of all feature point pairs that meet the parallelism criterion in a set; after the traversal is completed, the feature point pairs corresponding to all serial numbers in the set in the feature point sets of the real-time image and the reference image are the precise matching results.

2. The multimodal image point feature fast and precise matching method based on auxiliary navigation information according to claim 1, characterized in that: Based on the feature point sets corresponding to the real-time image and the reference image, the mixed distance between the two in row and column directions is calculated, which is expressed as: Among them, lx[i] and ly[i] are the mixed distances of the i-th feature point in the row and column directions respectively, C is the number of columns of the real-time image; ptRt[i].x and ptBenc[i].x represent the row positions of the feature points ptRt[i] and ptBenc[i] in the feature point sets of the real-time image and the reference image respectively; ptRt[i].y and ptBenc[i].y represent the column positions of the feature points ptRt[i] and ptBenc[i] respectively.

3. The multimodal image point feature fast and precise matching method based on auxiliary navigation information according to claim 1, characterized in that: The mixed distance and mixed direction of each feature point pair are calculated using the row and column direction mixed distance, which can be expressed as: Among them, L[i] and A[i] represent the mixing distance and mixing direction of feature point pair i respectively.

4. The method for fast and precise matching of multimodal image point features based on auxiliary navigation information according to claim 1, characterized in that: By mixing distance and direction, combined with preset thresholds, a parallelism criterion is constructed, including: There are two parallelism criteria. The first one is the distance criterion: |L[i]-L[j]|<t1 The second is the angle proximity criterion: |A[i]-A[j]|<t2 Among them, t1 and t2 are preset thresholds, L[j] and A[j] are the mixing distance and mixing direction of feature point pair i.

5. The method for fast and precise matching of multimodal image point features based on auxiliary navigation information according to claim 1, characterized in that: Adjust the precision of the precise matching result by adjusting the preset threshold.

6. The method for fast and precise matching of multimodal image point features based on auxiliary navigation information according to claim 1, characterized in that: Traverse between different feature point pairs; During the traversal process, the parallelism criterion is used for screening, and the number of feature point pairs that meet the parallelism criterion is recorded in the corresponding position of the feature screening array, including: Execute a double loop, the outer loop traverses i∈[1,N], and the inner loop traverses j∈[1,N]. After the end, the position of each element of the feature screening array Π is assigned; if and only if the distance similarity criterion and the angle similarity criterion are met at the same time, the value of the i-th element Π(i) in the array Π is increased by 1; N is the number of feature point pairs.

7. The method for fast and precise matching of multimodal image point features based on auxiliary navigation information according to claim 1, characterized in that: Determine the sequence number of the maximum value in the feature screening array, and use this sequence number to traverse other feature point pairs again, and store the sequence numbers of all feature point pairs that meet the parallelism criteria in a set, including: Perform maximum value determination to obtain the serial number of the feature point pair that makes the array π at the maximum value, and assign the result to the variable I; let i = I, and re-traverse j∈[1,N], determine according to the parallelism criterion, and put the j that meets the criterion into the set K.

8. The multimodal image point feature fast and precise matching method based on auxiliary navigation information according to claim 7, characterized in that: If there are multiple maximum values, the smallest serial number among the multiple maximum values ​​is assigned to I.

9. A terminal device comprising a processor, a memory, and a computer program stored in the memory; characterized in that: When the processor executes the computer program, it implements the multimodal image point feature fast and precise matching method based on auxiliary navigation information according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program; wherein: When the computer program is executed by a processor, the method for fast and precise matching of multimodal image point features based on auxiliary navigation information according to any one of claims 1 to 8 is implemented.

Images