Feature point matching method, device, electronic device and storage medium
By constructing affine-independent descriptors and filtering mismatched point pairs using point pair transformation errors, the problem of mismatched point pairs in image registration is solved, and the accuracy of image registration is improved.
Patent Information
- Application Number
- CN202210178765.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-25
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-02-25
AI Technical Summary
In the prior art, when registering images, there are many mismatched pairs in the registration point pair, resulting in poor image registration effect, especially in images with large affine transformation, monotonic features and low overlapping areas, it is difficult to effectively filter out mismatches. Points right.
By obtaining the nearest points of feature points in the image, constructing affine-independent descriptors, filtering out candidate mismatch point pairs, and using point pair transformation errors and mean values to determine the true mismatch point pairs, gradually deleting mismatch point pairs to improve registration Accuracy.
It effectively reduces mismatch point pairs, improves the accuracy and effect of image registration, especially in image registration with large affine transformation and low overlap areas, which significantly improves the matching accuracy.
Smart Images

Figure CN114723791B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a feature point matching method, device, electronic equipment and storage medium. Background Art
[0002] In the image registration process, it is usually necessary to match the feature points extracted from the two images to obtain the registration point pairs used to calculate the transformation relationship between the images, and remove the mismatched point pairs in the registration point pairs.
[0003] When existing feature point matching methods are applied to images with large affine transformations, monotonous features, and low overlapping areas, it is difficult to screen out all mismatched point pairs, especially those located at the edge of the image or with only small errors. The registration point pairs contain more mismatched point pairs, resulting in poor image registration effect. Summary of the invention
[0004] The present invention provides a feature point matching method, device, electronic device and storage medium, which are used to solve the technical problem in the prior art that when images are registered, there are many mismatched point pairs contained in the registration point pairs, resulting in poor image registration effect.
[0005] The present invention provides a feature point matching method, comprising:
[0006] Obtain a current set of matching point pairs between the reference image and the image to be registered during the current feature point matching process;
[0007] Determine a candidate mismatched point pair in the current feature point matching process based on an affine-independent descriptor of each matching point pair in the current matching point pair set; the affine-independent descriptor is determined based on a feature point in the matching point pair and a neighboring point of the feature point;
[0008] Determine the point pair transformation error of the candidate mismatching point pair based on the candidate mismatching point pair, determine the point pair transformation error mean between the reference image and the image to be registered based on the current matching point pair set, determine whether the candidate mismatching point pair belongs to a real mismatching point pair based on the point pair transformation error and / or the point pair transformation error mean, and use the current matching point pair set after deleting the real mismatching point pair as the next matching point pair set;
[0009] If none of the candidate mismatched point pairs belong to real mismatched point pairs, all matching point pairs in the current matching point pair set are determined as registration feature point pairs between the reference image and the image to be registered.
[0010] According to the feature point matching method provided by the present invention, determining whether the candidate mismatched point pair belongs to a real mismatched point pair based on the point pair transformation error and / or the point pair transformation error mean value includes:
[0011] If the mean change of the point pair transformation error between the reference image and the image to be registered after deleting any candidate mismatched point pair is greater than a first preset threshold, then determining any candidate mismatched point pair as a true mismatched point pair;
[0012] The change in the mean value of the point pair transformation error is determined based on the mean value of the point pair transformation error and the mean value of the point pair transformation error between the reference image and the image to be registered after deleting any candidate mismatched point pair.
[0013] According to the feature point matching method provided by the present invention, determining whether the candidate mismatched point pair belongs to a real mismatched point pair based on the point pair transformation error and / or the point pair transformation error mean value includes:
[0014] When the maximum value of the point pair transformation errors of the candidate mismatched point pairs is greater than the mean value of the point pair transformation errors corresponding to the current matching point pair set and the mean value of the point pair transformation errors corresponding to the current matching point pair set is greater than a first preset threshold, for any candidate mismatched point pair, if the change in the mean value of the point pair transformation errors between the reference image and the image to be registered after deleting the any candidate mismatched point pair is less than the first preset threshold, and the point pair transformation error of the any candidate mismatched point pair is greater than the mean value of the point pair transformation errors corresponding to the current matching point pair set, then the any candidate mismatched point pair is determined as a true mismatched point pair.
[0015] According to the feature point matching method provided by the present invention, determining whether the candidate mismatched point pair belongs to a real mismatched point pair based on the point pair transformation error and / or the point pair transformation error mean value includes:
[0016] If the maximum value of the point pair transformation errors of the candidate mismatched point pairs is less than or equal to the mean value of the point pair transformation errors corresponding to the current matching point pair set, and the affine-independent descriptor of the candidate mismatched point pairs is a preset value, then the candidate mismatched point pairs whose point pair transformation errors are greater than a second preset threshold are determined as true mismatched point pairs.
[0017] According to the feature point matching method provided by the present invention, the affine-independent descriptor is determined based on the following steps:
[0018] Performing feature point detection on the reference image and the image to be registered to obtain a plurality of matching point pairs; the matching point pairs include a first feature point in the reference image and a second feature point in the image to be registered corresponding to the first feature point;
[0019] Taking any first feature point in the reference image as the current first feature point, determining the first neighboring point of the current first feature point in the reference image based on the distance between the remaining first feature points in the reference image and the current first feature point, and taking the first neighboring point and the second feature point corresponding to the first neighboring point in the image to be registered as the first neighboring point set of the current first feature point;
[0020] Based on the current second feature point corresponding to the current first feature point and the distance between the current second feature point and the remaining second feature points in the image to be registered, determine the second neighboring point of the current second feature point in the image to be registered, and use the second neighboring point and the first feature point corresponding to the second neighboring point in the reference image as the second neighboring point set of the current second feature point;
[0021] Based on the first neighbor point set of the current first feature point and the second neighbor point set of the current second feature point, an affine-independent descriptor of the current matching point pair is determined.
[0022] According to the feature point matching method provided by the present invention, the determining of the affine-independent descriptor of the current matching point pair based on the first neighbor point set of the current first feature point and the second neighbor point set of the current second feature point includes:
[0023] Determine a first affine triangle and a second affine triangle of the current first feature point based on the current first feature point and the first neighbor point set, and determine an area ratio of the affine triangle of the current first feature point based on the first affine triangle and the second affine triangle;
[0024] Determine a first affine triangle and a second affine triangle of the current second feature point based on the current second feature point and the second neighbor point set, and determine an area ratio of the affine triangle of the current second feature point based on the first affine triangle and the second affine triangle;
[0025] An affine-independent descriptor of the current matching point pair is determined based on an area ratio of the affine triangle of the current first feature point and an area ratio of the affine triangle of the current second feature point.
[0026] According to the feature point matching method provided by the present invention, determining the first affine triangle and the second affine triangle of the current first feature point based on the current first feature point and the first neighbor point set includes:
[0027] Based on the distance between each first neighbor point in the first neighbor point set and the current first feature point, the first neighbor points are arranged in ascending order to obtain a first neighbor point sequence, and the first neighbor point of the current sequence number and the first neighbor point of the next sequence number are selected from the first neighbor point sequence in turn to form a first affine triangle with the current first feature point;
[0028] Based on the distance between each second neighbor point in the first neighbor point set and the second feature point corresponding to the current first feature point, the second neighbor points are arranged in ascending order to obtain a second neighbor point sequence, and the second neighbor point with the current sequence number and the second neighbor point with the next sequence number are selected from the second neighbor point sequence in turn, and the second feature point corresponding to the current first feature point forms a second affine triangle. The present invention provides a feature point matching device, comprising:
[0029] An acquisition unit, used to acquire a current set of matching point pairs between the reference image and the image to be registered during the current feature point matching process;
[0030] A screening unit, configured to determine a candidate mismatched point pair in the current feature point matching process based on an affine-independent descriptor of each matching point pair in the current matching point pair set; the affine-independent descriptor is determined based on a feature point in the matching point pair and a neighboring point of the feature point;
[0031] a judgment unit, configured to determine a point pair transformation error of the candidate mismatching point pair based on the candidate mismatching point pair, determine a mean value of the point pair transformation error between the reference image and the image to be registered based on the current matching point pair set, determine whether the candidate mismatching point pair belongs to a true mismatching point pair based on the point pair transformation error and / or the mean value of the point pair transformation error, and use the current matching point pair set after deleting the true mismatching point pair as the next matching point pair set;
[0032] A determination unit is used to determine all matching point pairs in the current matching point pair set as registration feature point pairs between the reference image and the image to be registered if none of the candidate mismatching point pairs belong to real mismatching point pairs.
[0033] The present invention provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the feature point matching method is implemented when the processor executes the program.
[0034] The present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the feature point matching method is implemented.
[0035] The feature point matching method, device, electronic device and storage medium provided by the present invention determine the affine-independent descriptor of each matching point pair through the feature points and the neighboring points of the feature points in each matching point pair between the reference image and the image to be registered, and then determine the candidate mismatched point pair in the current matching point pair set, and then determine whether the candidate mismatched point pair belongs to the real mismatched point pair according to the point pair transformation error of the candidate mismatched point pair and / or the mean value of the point pair transformation error between the reference image and the image to be registered, and after deleting the real mismatched point pair, obtain the registered feature point pair between the reference image and the image to be registered. Since the local adjacency relationship between the feature points is used when determining the candidate mismatched point pair, and the global information of the feature point transformation in the image is used when determining the real mismatched point pair, the mismatched point pairs in the registration process can be effectively deleted, thereby improving the accuracy of image registration. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0037] Figure 1 It is one of the schematic diagrams of the real mismatched point pairs provided by the present invention;
[0038] Figure 2 It is a flow chart of the feature point matching method provided by the present invention;
[0039] Figure 3 It is a flow chart of the method for determining candidate mismatched point pairs provided by the present invention;
[0040] Figure 4 This is the second schematic diagram of the real mismatched point pair provided by the present invention;
[0041] Figure 5 It is a structural schematic diagram of a feature point matching device provided by the present invention;
[0042] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0044] It should be noted that the terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0045] In order to solve the technical problem in the prior art that, during image registration, there are many mismatched point pairs included in the registration point pairs, resulting in poor image registration effect, the inventive concept of the present invention is as follows: an affine-independent descriptor is constructed using the local adjacency relationship of feature points, candidate mismatched point pairs are extracted using the affine-independent descriptor, and then the real mismatched point pairs in the candidate mismatched point pairs are screened out in combination with global information, and then the real mismatched point pairs are eliminated for image registration.
[0046] The reference image and the image to be registered in the embodiment of the present invention are images that need to be registered. For example, the reference image and the image to be registered may be remote sensing images of the earth taken by a spacecraft.
[0047] Feature points include the center point, edge point, intersection point of lines, etc. of objects in the image. They can be obtained using the SURF (Speeded Up Robust Features) algorithm, SIFT (Scale Invariant Feature Transform) algorithm, etc. For a feature point in the reference image, there is a corresponding feature point in the image to be registered. This group of feature points constitutes a matching point pair. There may be multiple matching point pairs in the reference image and the image to be registered. According to the positions of these matching point pairs, the image to be registered can be mapped to the reference image, thereby completing image registration.
[0048] The affine transformation in image registration is the transformation that transforms the feature points in the drawing to be registered to the positions of the corresponding feature points in the reference drawing. Affine independence refers to the characteristic that the spatial position of a feature point does not change relative to other feature points under affine transformation. Affine-independent descriptors are used to describe the affine-independent characteristics of feature points in matching point pairs, and affine-independent descriptors are used to measure the degree of affine-independent characteristics.
[0049] In the embodiment of the present invention, the affine-independent descriptor is determined based on the following steps:
[0050] Performing feature point detection on the reference image and the image to be registered to obtain a plurality of matching point pairs; the matching point pairs include a first feature point in the reference image and a second feature point in the image to be registered corresponding to the first feature point;
[0051] Taking any first feature point in the reference image as the current first feature point, determining the first neighboring point of the current first feature point in the reference image based on the distance between the remaining first feature points in the reference image and the current first feature point, and taking the first neighboring point and the second feature point corresponding to the first neighboring point in the image to be registered as the first neighboring point set of the current first feature point;
[0052] Based on the current second feature point corresponding to the current first feature point and the distance between the current second feature point and the remaining second feature points in the image to be registered, determine the second neighboring point of the current second feature point in the image to be registered, and use the second neighboring point and the first feature point corresponding to the second neighboring point in the reference image as the second neighboring point set of the current second feature point;
[0053] Based on a first neighbor point set of a current first feature point and a second neighbor point set of a current second feature point, an affine-independent descriptor of a current matching point pair is determined.
[0054] Specifically, the SURF algorithm, SIFT algorithm, etc. can be used to detect feature points of the reference image and the image to be registered to obtain multiple matching point pairs. Among them, any matching point pair C includes a first feature point P in the reference image and a second feature point P' in the image to be registered corresponding to the first feature point.
[0055] Take any first feature point P in the reference image i (i∈[1,n], i is the feature point number, n is the number of feature points) is the current first feature point, and the remaining first feature points in the reference image are calculated with respect to the current first feature point P. i The distance between them is calculated, and the remaining first feature points are arranged in order of distance from small to large, and K nearest neighbor points are selected as the first nearest neighbor points of the current first feature point in the reference image {P i1 , P i2 , ..., P ij , ..., P iK}, j∈[1,K]. In the image to be registered, determine the second feature point corresponding to each first neighbor point one by one. The first neighbor point and the second feature point corresponding to the first neighbor point in the image to be registered are taken as the first neighbor point set V of the current first feature point i .
[0056] The first nearest neighbor point set V i It can be expressed as {C i1 , C i2 , ..., C ij , ..., C iK}.
[0057] According to the current first feature point P i The corresponding current second feature point P′i , calculate the remaining second feature points in the image to be registered and the current second feature point P′ i The distance between them is calculated, and the remaining second feature points are arranged in order of distance from small to large, and K′ neighboring points are selected as the second neighboring points {P′ i1 , P′ i2 , ..., P′ ij′ , ..., P′ iK′}, j′∈[1,K′]. In the reference image, determine the first feature point corresponding to each second neighbor point one by one. The second neighbor point and the first feature point corresponding to the second neighbor point in the reference image are taken as the current second feature point P′ i The second nearest neighbor point set V i′ .
[0058] The second nearest neighbor point set V i′ It can be expressed as {C i1′ , C i2′ , ..., C ij′ , ..., C iK′}.
[0059] Then according to the current first feature point P i The first nearest neighbor point set V i , and the current second feature point P′ i The second nearest neighbor point set V i′ , determine the current matching point pair C i (P i ,P′ i ) is an affine-independent descriptor of .
[0060] Based on any of the above embodiments, based on the first neighbor point set of the current first feature point and the second neighbor point set of the current second feature point, determining the affine-independent descriptor of the current matching point pair includes:
[0061] Based on the current first feature point and the first neighbor point set, determine a first affine triangle and a second affine triangle of the current first feature point, and determine an area ratio of the affine triangle of the current first feature point based on the first affine triangle and the second affine triangle;
[0062] Based on the current second feature point and the second neighbor point set, determine a first affine triangle and a second affine triangle of the current second feature point, and determine an area ratio of the affine triangle of the current second feature point based on the first affine triangle and the second affine triangle;
[0063] An affine-independent descriptor of the current matching point pair is determined based on the area ratio of the affine triangle of the current first feature point and the area ratio of the affine triangle of the current second feature point.
[0064] Specifically, point P i The coordinates can be expressed as (x i ,y i ). Then we can determine a triangle based on three non-collinear points P1(x1,y1), P2(x2,y2) and P3(x3,y3), and get the representation of the triangle area (Triangle-AreaRepresentation, TAR), which is expressed as:
[0065] TAR(P1,P2,P3)=abs(x1y2+x2y3+x3y1-x3y2-x1y3-x2y1) / 2
[0066] Assume that the image transformation relationship between the reference image and the image to be registered can be expressed as:
[0067]
[0068] In the above formula, (x, y) is the coordinate of point P in the reference image, (x′, y′) is the coordinate of point P′ in the reference image, (P, P′) is the matching point pair C, and a1, a2, a3, a4, b1 and b2 are calculation coefficients.
[0069] Using the above relationship, we can get:
[0070] TAR(P′1,P′2,P′3)=(a1a4-a2a3)TAR(P1,P2,P3)
[0071] Among them, (P1, P′1) is the matching point pair C1 in the reference image and the image to be registered, (P2, P′2) is the matching point pair C2 in the reference image and the image to be registered, and (P3, P′3) is the matching point pair C3 in the reference image and the image to be registered.
[0072] Using TAR_ratio to represent the area ratio, we can get:
[0073]
[0074] According to the above formula, for any three correct matching point pairs (C1, C2, C3), TAR_ratio(C1, C2, C3) is the same constant.
[0075] According to the above formula, we can also know that when selecting any three correct matching point pairs, we need to avoid the collinear situation. At this time, the area of the triangle formed is zero, TAR(P′1,P′2,P′3)=0, and the threshold β can be set to limit TAR(P′1,P′2,P′3). That is, the selection of matching point pairs should satisfy abs(TAR(P′1,P′2,P′3))>β. Considering that the accuracy of the TAR_ratio value will decrease when TAR is small, the value of β can be set, for example, β=0.2.
[0076] Based on the above analysis, for the current matching point pair C i (P i ,P′ i ), according to the current first feature point P i The first nearest neighbor point set V i , and the current second feature point P′ i The second nearest neighbor point set V i′ , the current matching point pair C can be constructed i (P i ,P′ i )’s affine-independent descriptor, the specific steps are as follows:
[0077] According to the current matching point C i The current first feature point P in i , and the first nearest neighbor point set V i The first nearest point P of the two ij and P ik , j and k are the neighbor point numbers, the first affine triangle of the current first feature point can be determined, and the area TAR(P i ,P ij ,P ik ). According to the current first feature point P i The corresponding second feature point P′ i , the first nearest neighbor point P ij The corresponding second feature point A, the first neighbor point P ik The corresponding second feature point B obtains the area of the second affine triangle TAR(P′ i ,A,B). Then we can solve the area ratio of the affine triangle of the current first feature point TAR_ratio(C i ,C ij ,C ik ):
[0078]
[0079] Similarly, based on the current second feature point, and the first neighbor point and the second neighbor point in the second neighbor point set, the area ratio TAR_ratio(C i ,C ij′ ,C ik′ ), j′ and k′ are the sequence numbers of the neighboring points.
[0080] Then we can get:
[0081] TAR_ratio KNN (i) = TAR_ratio(C i ,C ij ,C ik ) / TAR_ratio(C i ,C ij′ ,C ik′ )
[0082] When the matching point pair is a correct matching point pair, TAR_ratio KNN (i)∈[1,ε], otherwise TAR_ratio KNN (i)>ε. TAR_ratio can be determined based on the positioning error of the feature points and the error of the image transformation model. KNN (i) is the value range. ε is a constant greater than 1. When the neighboring points of the matching point pair are the correct matching point pair, if TAR_ratio KNN (i)>ε, then the current matching point pair C i (P i ,P′ i ) is the real mismatched point pair.
[0083] Figure 1 is one of the schematic diagrams of the real mismatched point pairs provided by the present invention, such as Figure 1 As shown, for the matching point pair C i (P i ,P′ i ), for the first feature point P i , its first nearest neighbor point set V i ={C i1 (P i1 ,P′ i1 ), C i2 (P i2 ,P′ i2 )}, for the second feature point P′ i , its second nearest neighbor point set V i′ ={C i1′ (P i1′ ,P′ i1′ ), C i2′ (P i2′ ,P′i2′ )}, according to the above method, we can get the matching point pair C i TAR_ratio KNN (i). Due to TAR_ratio KNN (i) is greater than the constant ε, so the matching point pair C i is a real mismatched point pair, whose physical meaning is the feature point P′ i Under affine transformation, transform the feature point P from the image to be registered to the reference image i When the position of a feature point is changed, its spatial position changes relative to other feature points.
[0084] If TAR_ratio KNN (i) is less than 1. For ease of comparison, we can take TAR_ratio KNN The reciprocal of (i).
[0085] To improve accuracy, we can take TAR_ratio KNN (i) and use it as the affine-independent descriptor TAR_ratio KNN (i) to determine whether the current matching point is a candidate mismatch point pair. The mean here draws on the idea of the K-nearest-Neighbor (KNN) algorithm, and K′=K can be expressed as:
[0086]
[0087]
[0088] In the formula, j is the first neighbor point set V i The sequence number of the nearest neighbor point in the middle, j′ is the second nearest neighbor point set V i′ The sequence number of the neighboring points in , K is the number of neighboring points.
[0089] Based on any of the above embodiments, based on the current first feature point and the first neighbor point set, determining a first affine triangle and a second affine triangle of the current first feature point includes:
[0090] Based on the distance between each first neighbor point in the first neighbor point set and the current first feature point, the first neighbor points are arranged in ascending order to obtain a first neighbor point sequence, and the first neighbor point of the current sequence number and the first neighbor point of the next sequence number are selected from the first neighbor point sequence in turn to form a first affine triangle with the current first feature point;
[0091] Based on the distance between each second neighbor point in the first neighbor point set and the second feature point corresponding to the current first feature point, the second neighbor points are arranged in ascending order to obtain a second neighbor point sequence, and the second neighbor point of the current sequence number and the second neighbor point of the next sequence number are selected from the second neighbor point sequence in turn, and the second feature point corresponding to the current first feature point forms a second affine triangle.
[0092] Specifically, for the current first feature point P i For the current second feature point P′, there are K first neighbor points in the first neighbor point set; i For example, the number of its second nearest neighbor points in the second nearest neighbor point set is K′=K. For the convenience of comparison, when determining the affine triangle, the affine triangles can be constructed respectively according to a certain point selection order.
[0093] According to the distance between each first neighbor point in the first neighbor point set and the current first feature point, the first neighbor points are arranged in ascending order to obtain the first neighbor point sequence {P i1 , P i2 , ..., P ij , ..., P iK}, j∈[1,K], K is the number of neighboring points. Then select P ij and P i(j+1) , and the current first feature point P i Construct the first affine triangle.
[0094] According to the distance between each second neighbor point in the first neighbor point set and the second feature point corresponding to the current first feature point, the second neighbor points are arranged in ascending order to obtain the second neighbor point sequence {P′ i1 , P′ i2 , ..., P′ ij , ..., P′ iK}, select P′ from the second neighboring point sequence in turn ij and P′ i(j+1) , and the current first feature point P i The corresponding second feature point P′ i Construct the second affine triangle.
[0095] Similarly, the above method of constructing an affine triangle by selecting points according to distance is also applicable to constructing an affine triangle of the current second feature point.
[0096] The above point selection process can also be implemented in a matrix manner. For example, in the reference image, the feature point set P = {P i ; i=1,2,…,n}, and the distance between any two points, and respectively according to the distance to point P iThe distances are sorted from small to large, and the sequence numbers of the points are stored in the n×n matrix D p In, then D p (i,j) represents point P i The same method can be applied to the images to be registered.
[0097] Based on the above embodiments, Figure 2 is a flow chart of the feature point matching method provided by the present invention, such as Figure 2 As shown, the method includes step 210 , step 220 , step 230 and step 240 .
[0098] Step 210: obtaining a current set of matching point pairs between the reference image and the image to be registered in the current feature point matching process.
[0099] Specifically, there may be mismatched point pairs among these matching point pairs. Mismatched point pairs, also known as outlier point pairs, are point pairs that do not have a corresponding relationship between the reference image and the image to be registered, and cannot be used to represent the similarity of structural features in the two images. If there are mismatched point pairs among the matching point pairs, the image registration effect will be reduced. The more mismatched point pairs there are, the worse the image registration effect will be, or even the image registration cannot be completed.
[0100] Therefore, before image registration, the obtained matching point pairs can be screened to remove the mismatched point pairs and retain the correct matching point pairs. The feature point matching method provided in the embodiment of the present invention can screen and remove the mismatched point pairs in the matching point pair set by iterating multiple rounds of feature point matching process.
[0101] In particular, in the first round of feature point matching, the current matching point pair set is composed of all matching point pairs detected by the feature point detection algorithm.
[0102] Step 220, based on the affine-independent descriptor of each matching point pair in the current matching point pair set, determine the candidate mismatched point pairs in the current feature point matching process; the affine-independent descriptor is determined based on the feature point and the neighboring points of the feature point in the matching point pair.
[0103] Specifically, the neighboring points are also feature points and can be selected based on the distance between the feature points. For example, in the reference image, for the feature point P i , i∈[1,n], i is the feature point number, n is the number of feature points. The feature point P whose distance is less than the preset distance value can be i+1 and feature point P i+2 as its nearest neighbor.
[0104] The affine-independent descriptor can be determined by matching the feature point and the neighboring points of the feature point in the point pair, and the construction method thereof has been described in the above-mentioned embodiment.
[0105] The candidate mismatched point pairs in the current matching point pair set can be screened out by using the affine-independent descriptor.
[0106] Figure 3 FIG. 1 is a flow chart of the method for determining candidate mismatched point pairs provided by the present invention. Figure 3 As shown, the method may include:
[0107] Step 310: perform feature point detection on the reference image and the image to be registered to obtain a plurality of matching point pairs.
[0108] Step 320: respectively determine a first neighbor point set of the first feature point in each matching point pair and a second neighbor point set of the second feature point in each matching point pair;
[0109] Step 330: construct affine triangles according to the first neighbor point set of the first feature point and the second neighbor point set of the second feature point, respectively, and construct an affine-independent descriptor for each matching point pair according to the ratio of the areas of the affine triangles;
[0110] Step 340, traverse all matching points, compare the value of the affine-independent descriptor of each matching point pair with the preset value, and if the affine-independent descriptor is greater than the preset value, then the corresponding matching point pair is used as a candidate mismatching point pair. For example, in the above example, if the affine-independent descriptor is close to 1, it means that the first triangle and the second triangle are highly similar, and it can be considered that the matching point pair (P i , P′ i ) is the correct matching point pair; if the affine-independent descriptor is far from 1, it means that the first triangle and the second triangle are less similar, and it can be considered that the matching point pair (P i , P′ i ) are candidate mismatched point pairs.
[0111] Since the features between the reference drawing and the image to be registered are similar, the overlapping area is small, or there is a large affine deformation, the candidate mismatched point pairs may be the correct matching point pairs. Therefore, it is necessary to further screen them based on the global information of all feature points in the image.
[0112] Step 230, determining the point pair transformation error of the candidate mismatching point pair based on the candidate mismatching point pair, determining the mean value of the point pair transformation error between the reference image and the image to be registered based on the current matching point pair set, and determining whether the candidate mismatching point pair belongs to a real mismatching point pair based on the point pair transformation error and / or the mean value of the point pair transformation error, and using the current matching point pair set after deleting the real mismatching point pair as the next matching point pair set.
[0113] Specifically, the point pair transformation error refers to the error between the transformed position of a feature point in the image to be registered after being transformed into the reference image and the actual position of the feature point corresponding to the feature point in the reference image.
[0114] The point pair transformation error mean refers to the error mean after the feature points in the image to be registered are transformed to the reference image, which can be determined based on the global transformation mean of all feature points and the number of feature points.
[0115] Whether the candidate mismatched point pair belongs to a real mismatched point pair may be determined according to the point pair transformation error and / or the point pair transformation error mean.
[0116] For example, the mean value of the point pair transformation error between the reference image and the image to be registered can be determined based on the current set of matching point pairs. The point pair transformation error of each candidate mismatching point pair is determined based on each candidate mismatching point pair. If the point pair transformation error of any candidate mismatching point pair is greater than a preset point pair transformation error threshold, or the difference between the point pair transformation error and the mean value of the point pair transformation error is greater than a preset difference, it indicates that the candidate mismatching point pair is a true mismatching point pair, otherwise it is a correct matching point pair.
[0117] For another example, the mean value of the first point pair transformation error between the reference image and the image to be registered can be calculated by the current matching point pair set. After deleting the current candidate mismatched point pair, the mean value of the second point pair transformation error between the reference image and the image to be registered can be calculated again. Based on the difference between the first point pair transformation error mean and the second point pair transformation error mean, the influence of the current candidate mismatched point pair on the point pair transformation error mean can be judged. The larger the difference, the greater the influence, and the greater the possibility that the current candidate mismatched point pair belongs to the real mismatched point pair.
[0118] In the current matching point pair set, the real mismatched point pairs are deleted to obtain the next matching point pair set. The next feature point matching process is performed according to the next matching point pair set, and the existing real mismatched point pairs are gradually deleted through multiple rounds of feature point matching processes.
[0119] Step 240: If none of the candidate mismatched point pairs belong to real mismatched point pairs, all matching point pairs in the current matching point pair set are determined as registration feature point pairs between the reference image and the image to be registered.
[0120] Specifically, after several rounds of feature point matching, if none of the candidate mismatched point pairs belong to the real mismatched point pairs, then all the remaining matching point pairs can be considered to be correct matching point pairs, and the feature point matching process can be terminated. All matching point pairs in the current matching point pair set are determined as the registration feature point pairs between the reference image and the image to be registered, and the next step of image registration is performed.
[0121] The feature point matching method provided by the embodiment of the present invention determines the affine-independent descriptor of each matching point pair through the feature points and the neighboring points of the feature points in each matching point pair between the reference image and the image to be registered, and then determines the candidate mismatched point pair in the current matching point pair set, and then determines whether the candidate mismatched point pair belongs to the real mismatched point pair according to the point pair transformation error of the candidate mismatched point pair and / or the mean value of the point pair transformation error between the reference image and the image to be registered, and after deleting the real mismatched point pair, the registration feature point pair between the reference image and the image to be registered is obtained. Since the local adjacency relationship between the feature points is used when determining the candidate mismatched point pair, and the global information of the feature point transformation in the image is used when determining the real mismatched point pair, the mismatched point pairs in the registration process can be effectively deleted, thereby improving the accuracy of image registration.
[0122] Based on the above embodiment, step 230 includes:
[0123] If the change in the mean value of the point pair transformation error between the reference image and the image to be registered after deleting any candidate mismatched point pair is greater than a first preset threshold, then any candidate mismatched point pair is determined as a true mismatched point pair;
[0124] The change in the mean value of the point pair transformation error is determined based on the mean value of the point pair transformation error and the mean value of the point pair transformation error between the reference image and the image to be registered after deleting any candidate mismatched point pair.
[0125] Specifically, the set of initial matching point pairs can be represented by C. The mean value of the point pair transformation error can be represented by E. The matching relationship between the reference image and the image to be registered can be represented by V. Then the best matching relationship is It can be expressed as:
[0126]
[0127] The i-th matching point pair C i The matching relationship V i It can be defined as:
[0128]
[0129] Then the matching point pair C i The point pair transformation error d(C i ,θ) can be expressed as:
[0130] d(C i ,θ)=‖T(θ,P′ i )-p i ‖
[0131] In the above formula, θ is the image transformation model between the reference image and the image to be registered obtained by solving the current matching point pair set, T(θ,P′ i ) is the feature point P′ in the image to be registered i The transformed position after transformation to the reference image, T represents the transformation operation. Obviously, the point transformation error d(C i ,θ) is larger, the matching point pair C i The more likely it is a true mismatch point pair.
[0132] The image transformation model is used to affine transform the feature points in the image to be registered into the reference image, thereby realizing image registration, which can be calculated based on the position coordinates of multiple matching point pairs.
[0133] The global transformation error E(C, V) of the current matching point can be expressed as:
[0134]
[0135] In the current feature point matching process, the point pair transformation error mean E bf It can be expressed as:
[0136]
[0137] Delete any candidate mismatched point pair C i After that, the mean transformation error E between the reference image and the image to be registered is af (i) can be expressed as:
[0138]
[0139] Delete any candidate mismatched point pair C i After that, the mean change of the point pair transformation error E c (i) can be expressed as:
[0140] E c (i) = abs(E bf -E af (i))
[0141] Where j is the serial number of the matching point pair in the current matching point pair set, rn is the number of matching point pairs in the current matching point pair set, θ is the image transformation model obtained by solving the current matching point pair set, and θ′ is the image transformation model obtained by deleting any candidate mismatched point pair C in the current matching point pair set. i The image transformation model is obtained by solving the set of matching point pairs, and abs is the absolute value operation.
[0142] If any candidate mismatched point pair C is deleted iThe mean change of the point pair transformation error between the reference image and the image to be registered is greater than the first preset threshold σ, that is, E c (i)>σ, then any candidate mismatch point pair C i The first preset threshold value can be set as needed.
[0143] Based on any of the above embodiments, step 230 includes:
[0144] When the maximum value of the point pair transformation errors of the candidate mismatched point pairs is greater than the mean value of the point pair transformation errors corresponding to the current matching point pair set and the mean value of the point pair transformation errors corresponding to the current matching point pair set is greater than a first preset threshold, for any candidate mismatched point pair, after deleting any candidate mismatched point pair, the change in the mean value of the point pair transformation errors between the reference image and the image to be registered is less than the first preset threshold, and the point pair transformation error of any candidate mismatched point pair is greater than the mean value of the point pair transformation errors corresponding to the current matching point pair set, then any candidate mismatched point pair is determined to be a true mismatched point pair.
[0145] Specifically, in each round of feature point matching, each candidate mismatched point pair can be arranged in descending order according to the size of the point pair transformation error. Then the maximum value of the point pair transformation error of the candidate mismatched point pair can be expressed as d(C1,θ).
[0146] When the maximum value of the point pair transformation errors of the candidate mismatched point pairs is greater than the mean value of the point pair transformation errors corresponding to the current matching point pair set, and the mean value of the point pair transformation errors corresponding to the current matching point pair set is greater than the first preset threshold, that is, d(C1,θ)>E bf And E bf >σ, for any candidate mismatch point pair C i , after deleting the candidate mismatched point pair, the change in the mean of the point pair transformation error between the reference image and the image to be registered is less than the first preset threshold and the point pair transformation error of the candidate mismatched point pair is greater than the mean of the point pair transformation error corresponding to the current matching point pair set, that is, E c (i) = E bf -E af (i)<σ and d(C i ,θ)>E bf , then the candidate mismatch point pair is determined as the real mismatch point pair.
[0147] In the above calculation process, there may be N candidate mismatched point pairs with large point pair transformation errors, that is, there are N candidate mismatched point pairs that satisfy E bf -E af (N)<σ and E bf -E af(N+1)≥σ. If each candidate mismatch point pair satisfies d(C i ,θ)>E bf , then these N candidate mismatched point pairs are all true mismatched point pairs.
[0148] The above screening method may be executed N times, and each time E is calculated af (i) When subtracting the transformation errors of the point pairs that have been determined to be true mismatched point pairs, it should be considered.
[0149] Based on any of the above embodiments, step 230 includes:
[0150] If the maximum value of the point pair transformation errors of the candidate mismatched point pairs is less than or equal to the mean value of the point pair transformation errors corresponding to the current matching point pair set, and the affine-independent descriptor of the candidate mismatched point pairs is a preset value, then the candidate mismatched point pairs whose point pair transformation errors are greater than the second preset threshold are determined as true mismatched point pairs.
[0151] Specifically, when the affine-independent descriptor is calculated using the triangle area, the preset value is 1.
[0152] If the maximum value of the point pair transformation errors of the candidate mismatched point pairs is less than or equal to the mean value of the point pair transformation errors corresponding to the current matching point pair set, that is, d(C1,θ)≤E bf , and the candidate mismatch point pair C i The affine-independent descriptor of is the preset value, namely TAR_ratio KNN =1.
[0153] The adjacency relationship of the matching point pair set is stable and not easily affected by affine transformation. However, in some extreme cases, when the feature points in a pair of true mismatched point pairs are located on the same edge of the feature point set in the image, the true mismatched point pair may have the same affine-independent descriptor as other correct matching point pairs.
[0154] Figure 4 This is the second schematic diagram of the real mismatched point pair provided by the present invention. Figure 4 As shown, for the real mismatch point pair C i (P i ,P′ i ), for the first feature point P i , the order of its adjacent points is P i1 , P i2 , P i3 and P i4 , for the second feature point P′ i , the order of its adjacent points is P′ i1 , P′ i2 , P′ i3 and P′ i4. (P ij ,P′ ij ) is a correct matching point pair, j = 1, 2, 3, 4. The adjacency order of the feature points in this matching point pair is the same. i TAR_ratio KNN (i)=1.
[0155] However, this point pair is a real mismatched point pair and cannot be screened only by the size of the affine-independent descriptor. In this case, it can be determined by comparing the transformation error of the point pair.
[0156] If the candidate mismatched point pair is the correct matching point pair, the point pair transformation error should be relatively small; if the candidate mismatched point pair is the real mismatched point pair, the point pair transformation error may be relatively large.
[0157] At this time, the second preset threshold σ can be set s The second preset threshold can be calculated based on the mean transformation error E of the point pair corresponding to the current matching point pair set. bf Determine, expressed as:
[0158] σ s =αE bf
[0159] Among them, α is the proportionality coefficient.
[0160] For TAR_ratio KNN = 1 candidate mismatch point pair C i , if d(C i ,θ)>σ s , then the candidate mismatch point pair C i It can be determined as a real mismatched point pair.
[0161] Based on any of the above embodiments, the feature point matching method can be expressed in pseudo code as follows:
[0162] While(1){
[0163] Calculate the image transformation model θ and E according to the current set of matching point pairs bf ;
[0164] Calculate the affine-independent descriptor for each matching point pair and select the candidate mismatched point pair C i ;
[0165] The candidate mismatched point pairs are divided into two groups according to d(C i ,θ) are arranged by size;
[0166] For each candidate mismatch point pair C i :
[0167] {Use to delete the candidate mismatch point pair C i The image transformation model θ′ is obtained by solving the current matching point pair set; calculate E c (i) = abs(E bf -E af (i)); if E c (i)>σ, then C i If it is a real mismatched point pair, delete it from the current matching point pair set; if not, exit the loop;}
[0168] If there are candidate mismatched point pairs but the real mismatched point pairs are not screened out:
[0169] {If d(C1,θ)>E bf And E bf >σ:
[0170] {Let N = 1; for each candidate mismatch point pair: {calculate the division of {C i ; i=1,2,…,N} E of the inner point pairs of the outer set af (N); if E bf -E af (N)<σ and d(C i ,θ)>E bf , then C i If it is a real mismatched point pair, delete it from the current matching point pair set; if not, exit the loop;}}
[0171] If d(C1,θ)≤E bf
[0172] {For all candidate mismatched point pairs whose affine-independent descriptor is 1: {If d(C i ,θ)>σ s , then C i It is a real mismatched point pair and is deleted from the current matching point pair set;}}}
[0173] If no real mismatched point pair is found, then exit the loop;}
[0174] The feature point matching device provided by the present invention is described below. The feature point matching device described below and the feature point matching method described above can be referred to each other.
[0175] Based on any of the above embodiments, Figure 5 is a schematic diagram of the structure of the feature point matching device provided by the present invention, such as Figure 5 As shown, the device comprises:
[0176] An acquisition unit 510 is used to acquire a current set of matching point pairs between the reference image and the image to be registered in the current feature point matching process;
[0177] A screening unit 520 is used to determine a candidate mismatched point pair in the current feature point matching process based on an affine-independent descriptor of each matching point pair in the current matching point pair set; the affine-independent descriptor is determined based on the feature point and the neighboring points of the feature point in the matching point pair;
[0178] A judgment unit 550 is used to determine the point pair transformation error of the candidate mismatching point pair based on the candidate mismatching point pair, determine the point pair transformation error mean between the reference image and the image to be registered based on the current matching point pair set, and determine whether the candidate mismatching point pair belongs to a real mismatching point pair based on the point pair transformation error and / or the point pair transformation error mean, and use the current matching point pair set after deleting the real mismatching point pair as the next matching point pair set;
[0179] The determination unit 540 is configured to determine all matching point pairs in the current matching point pair set as registration feature point pairs between the reference image and the image to be registered if none of the candidate mismatching point pairs are real mismatching point pairs.
[0180] The feature point matching device provided by the embodiment of the present invention determines the affine-independent descriptor of each matching point pair through the feature points and the neighboring points of the feature points in each matching point pair between the reference image and the image to be registered, and then determines the candidate mismatched point pair in the current matching point pair set, and then determines whether the candidate mismatched point pair belongs to the real mismatched point pair according to the point pair transformation error of the candidate mismatched point pair and / or the mean value of the point pair transformation error between the reference image and the image to be registered, and after deleting the real mismatched point pair, the registration feature point pair between the reference image and the image to be registered is obtained. Since the local adjacency relationship between the feature points is used when determining the candidate mismatched point pair, and the global information of the feature point transformation in the image is used when determining the real mismatched point pair, the mismatched point pairs in the registration process can be effectively deleted, thereby improving the accuracy of image registration.
[0181] Based on any of the above embodiments, the judging unit is specifically configured to:
[0182] If the change in the mean value of the point pair transformation error between the reference image and the image to be registered after deleting any candidate mismatched point pair is greater than a first preset threshold, then any candidate mismatched point pair is determined as a true mismatched point pair;
[0183] The change in the mean value of the point pair transformation error is determined based on the mean value of the point pair transformation error and the mean value of the point pair transformation error between the reference image and the image to be registered after deleting any candidate mismatched point pair.
[0184] Based on any of the above embodiments, the judging unit is specifically configured to:
[0185] When the maximum value of the point pair transformation errors of the candidate mismatched point pairs is greater than the mean value of the point pair transformation errors corresponding to the current matching point pair set and the mean value of the point pair transformation errors corresponding to the current matching point pair set is greater than a first preset threshold, for any candidate mismatched point pair, after deleting any candidate mismatched point pair, the change in the mean value of the point pair transformation errors between the reference image and the image to be registered is less than the first preset threshold, and the point pair transformation error of any candidate mismatched point pair is greater than the mean value of the point pair transformation errors corresponding to the current matching point pair set, then any candidate mismatched point pair is determined to be a true mismatched point pair.
[0186] Based on any of the above embodiments, the judging unit is specifically configured to:
[0187] If the maximum value of the point pair transformation errors of the candidate mismatched point pairs is less than or equal to the mean value of the point pair transformation errors corresponding to the current matching point pair set, and the affine-independent descriptor of the candidate mismatched point pairs is a preset value, then the candidate mismatched point pairs whose point pair transformation errors are greater than the second preset threshold are determined as true mismatched point pairs.
[0188] Based on any of the above embodiments, the device further includes a construction unit, and the construction unit includes:
[0189] A detection subunit, used to perform feature point detection on the reference image and the image to be registered, and obtain a plurality of matching point pairs; the matching point pairs include a first feature point in the reference image, and a second feature point in the image to be registered corresponding to the first feature point;
[0190] A first neighbor point determination subunit is used to take any first feature point in the reference image as the current first feature point, determine the first neighbor point of the current first feature point in the reference image based on the distance between the remaining first feature points in the reference image and the current first feature point, and take the first neighbor point and the second feature point corresponding to the first neighbor point in the image to be registered as the first neighbor point set of the current first feature point;
[0191] A second neighbor point determination subunit is used to determine the second neighbor point of the current second feature point in the image to be registered based on the current second feature point corresponding to the current first feature point and the distance between the remaining second feature points in the image to be registered and the current second feature point, and use the second neighbor point and the first feature point corresponding to the second neighbor point in the reference image as the second neighbor point set of the current second feature point;
[0192] The construction subunit is used to determine the affine-independent descriptor of the current matching point pair based on the first neighbor point set of the current first feature point and the second neighbor point set of the current second feature point.
[0193] Based on any of the above embodiments, the construction subunit is specifically used for:
[0194] A first construction module is used to determine a first affine triangle and a second affine triangle of the current first feature point based on the current first feature point and the first neighbor point set, and determine an area ratio of the affine triangle of the current first feature point based on the first affine triangle and the second affine triangle;
[0195] A second construction module is used to determine a first affine triangle and a second affine triangle of the current second feature point based on the current second feature point and the second neighbor point set, and determine an area ratio of the affine triangle of the current second feature point based on the first affine triangle and the second affine triangle;
[0196] The third construction module is used to determine the affine-independent descriptor of the current matching point pair based on the area ratio of the affine triangle of the current first feature point and the area ratio of the affine triangle of the current second feature point.
[0197] Based on any of the above embodiments, the first construction module is specifically used for:
[0198] Based on the distance between each first neighbor point in the first neighbor point set and the current first feature point, the first neighbor points are arranged in ascending order to obtain a first neighbor point sequence, and the first neighbor point of the current sequence number and the first neighbor point of the next sequence number are selected from the first neighbor point sequence in turn to form a first affine triangle with the current first feature point;
[0199] Based on the distance between each second neighbor point in the first neighbor point set and the second feature point corresponding to the current first feature point, the second neighbor points are arranged in ascending order to obtain a second neighbor point sequence, and the second neighbor point of the current sequence number and the second neighbor point of the next sequence number are selected from the second neighbor point sequence in turn, and the second feature point corresponding to the current first feature point forms a second affine triangle.
[0200] Based on any of the above embodiments, Figure 6 A schematic diagram of the structure of an electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor (Processor) 610, a communication interface (Communications Interface) 620, a memory (Memory) 630 and a communication bus (Communications Bus) 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic command in the memory 630 to execute the following method:
[0201] Obtain a current set of matching point pairs between a reference image and an image to be registered during the current feature point matching process; determine candidate mismatched point pairs in the current feature point matching process based on an affine-independent descriptor of each matching point pair in the current set of matching point pairs; the affine-independent descriptor is determined based on the feature points and neighboring points of the feature points in the matching point pairs; determine point pair transformation errors of candidate mismatched point pairs based on the candidate mismatched point pairs, determine a mean value of point pair transformation errors between the reference image and the image to be registered based on the current set of matching point pairs, and determine whether the candidate mismatched point pairs belong to true mismatched point pairs based on the point pair transformation errors and / or the mean value of point pair transformation errors, and use the current set of matching point pairs after deleting the true mismatched point pairs as the next set of matching point pairs; if none of the candidate mismatched point pairs belong to true mismatched point pairs, determine all matching point pairs in the current set of matching point pairs as registration feature point pairs between the reference image and the image to be registered.
[0202] In addition, the logic commands in the above-mentioned memory 630 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several commands to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.
[0203] The processor in the electronic device provided in the embodiment of the present invention can call the logic instructions in the memory to implement the above method. Its specific implementation method is consistent with the implementation method of the aforementioned method and can achieve the same beneficial effects, which will not be repeated here.
[0204] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including:
[0205] Obtain a current set of matching point pairs between a reference image and an image to be registered during the current feature point matching process; determine candidate mismatched point pairs in the current feature point matching process based on an affine-independent descriptor of each matching point pair in the current set of matching point pairs; the affine-independent descriptor is determined based on the feature points and neighboring points of the feature points in the matching point pairs; determine point pair transformation errors of candidate mismatched point pairs based on the candidate mismatched point pairs, determine a mean value of point pair transformation errors between the reference image and the image to be registered based on the current set of matching point pairs, and determine whether the candidate mismatched point pairs belong to true mismatched point pairs based on the point pair transformation errors and / or the mean value of point pair transformation errors, and use the current set of matching point pairs after deleting the true mismatched point pairs as the next set of matching point pairs; if none of the candidate mismatched point pairs belong to true mismatched point pairs, determine all matching point pairs in the current set of matching point pairs as registration feature point pairs between the reference image and the image to be registered.
[0206] When the computer program stored on the non-transitory computer-readable storage medium provided by the embodiment of the present invention is executed, the above method is implemented. Its specific implementation method is consistent with the implementation method of the aforementioned method and can achieve the same beneficial effects, which will not be repeated here.
[0207] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0208] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of commands to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0209] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A feature point matching method, characterized in that: include: Obtain a current set of matching point pairs between the reference image and the image to be registered during the current feature point matching process; Determine a candidate mismatched point pair in the current feature point matching process based on an affine-independent descriptor of each matching point pair in the current matching point pair set; the affine-independent descriptor is determined based on a feature point in the matching point pair and a neighboring point of the feature point; Determine the point pair transformation error of the candidate mismatching point pair based on the candidate mismatching point pair, determine the point pair transformation error mean between the reference image and the image to be registered based on the current matching point pair set, determine whether the candidate mismatching point pair belongs to a real mismatching point pair based on the point pair transformation error and / or the point pair transformation error mean, and use the current matching point pair set after deleting the real mismatching point pair as the next matching point pair set; If none of the candidate mismatched point pairs belong to real mismatched point pairs, all matching point pairs in the current matching point pair set are determined as registration feature point pairs between the reference image and the image to be registered; The affine-independent descriptor is determined based on the following steps: Performing feature point detection on the reference image and the image to be registered to obtain a plurality of matching point pairs; the matching point pairs include a first feature point in the reference image and a second feature point in the image to be registered corresponding to the first feature point; Taking any first feature point in the reference image as the current first feature point, determining the first neighboring point of the current first feature point in the reference image based on the distance between the remaining first feature points in the reference image and the current first feature point, and taking the first neighboring point and the second feature point corresponding to the first neighboring point in the image to be registered as the first neighboring point set of the current first feature point; Based on the current second feature point corresponding to the current first feature point and the distance between the current second feature point and the remaining second feature points in the image to be registered, determine the second neighboring point of the current second feature point in the image to be registered, and use the second neighboring point and the first feature point corresponding to the second neighboring point in the reference image as the second neighboring point set of the current second feature point; Determine an affine-independent descriptor of the current matching point pair based on a first neighbor point set of the current first feature point and a second neighbor point set of the current second feature point; The determining of the affine-independent descriptor of the current matching point pair based on the first neighbor point set of the current first feature point and the second neighbor point set of the current second feature point includes: Determine a first affine triangle and a second affine triangle of the current first feature point based on the current first feature point and the first neighbor point set, and determine an area ratio of the affine triangle of the current first feature point based on the first affine triangle and the second affine triangle; Determine a first affine triangle and a second affine triangle of the current second feature point based on the current second feature point and the second neighbor point set, and determine an area ratio of the affine triangle of the current second feature point based on the first affine triangle and the second affine triangle; Determining an affine-independent descriptor of the current matching point pair based on an area ratio of an affine triangle of the current first feature point and an area ratio of an affine triangle of the current second feature point; The determining, based on the current first feature point and the first neighbor point set, a first affine triangle and a second affine triangle of the current first feature point comprises: Based on the distance between each first neighbor point in the first neighbor point set and the current first feature point, the first neighbor points are arranged in ascending order to obtain a first neighbor point sequence, and the first neighbor point of the current sequence number and the first neighbor point of the next sequence number are selected from the first neighbor point sequence in turn to form a first affine triangle with the current first feature point; Based on the distance between each second neighbor point in the first neighbor point set and the second feature point corresponding to the current first feature point, the second neighbor points are arranged in ascending order to obtain a second neighbor point sequence, and the second neighbor point of the current sequence number and the second neighbor point of the next sequence number are selected from the second neighbor point sequence in turn, and the second feature point corresponding to the current first feature point forms a second affine triangle.
2. The feature point matching method according to claim 1, characterized in that: The determining whether the candidate mismatched point pair belongs to a real mismatched point pair based on the point pair transformation error and / or the point pair transformation error mean value includes: If the mean change of the point pair transformation error between the reference image and the image to be registered after deleting any candidate mismatched point pair is greater than a first preset threshold, then determining any candidate mismatched point pair as a true mismatched point pair; The change in the mean value of the point pair transformation error is determined based on the mean value of the point pair transformation error and the mean value of the point pair transformation error between the reference image and the image to be registered after deleting any candidate mismatched point pair.
3. The feature point matching method according to claim 1, characterized in that: The determining whether the candidate mismatched point pair belongs to a real mismatched point pair based on the point pair transformation error and / or the point pair transformation error mean value includes: When the maximum value of the point pair transformation errors of the candidate mismatched point pairs is greater than the mean value of the point pair transformation errors corresponding to the current matching point pair set and the mean value of the point pair transformation errors corresponding to the current matching point pair set is greater than a first preset threshold, for any candidate mismatched point pair, if the change in the mean value of the point pair transformation errors between the reference image and the image to be registered after deleting the any candidate mismatched point pair is less than the first preset threshold, and the point pair transformation error of the any candidate mismatched point pair is greater than the mean value of the point pair transformation errors corresponding to the current matching point pair set, then the any candidate mismatched point pair is determined as a true mismatched point pair.
4. The feature point matching method according to claim 1, characterized in that: The determining whether the candidate mismatched point pair belongs to a real mismatched point pair based on the point pair transformation error and / or the point pair transformation error mean value includes: If the maximum value of the point pair transformation errors of the candidate mismatched point pairs is less than or equal to the mean value of the point pair transformation errors corresponding to the current matching point pair set, and the affine-independent descriptor of the candidate mismatched point pairs is a preset value, then the candidate mismatched point pairs whose point pair transformation errors are greater than a second preset threshold are determined as true mismatched point pairs.
5. A feature point matching device, characterized in that: A method for implementing the feature point matching method according to any one of claims 1 to 4, comprising: An acquisition unit, used to acquire a current set of matching point pairs between the reference image and the image to be registered during the current feature point matching process; A screening unit, configured to determine a candidate mismatched point pair in the current feature point matching process based on an affine-independent descriptor of each matching point pair in the current matching point pair set; the affine-independent descriptor is determined based on a feature point in the matching point pair and a neighboring point of the feature point; a judgment unit, configured to determine a point pair transformation error of the candidate mismatching point pair based on the candidate mismatching point pair, determine a mean value of the point pair transformation error between the reference image and the image to be registered based on the current matching point pair set, determine whether the candidate mismatching point pair belongs to a true mismatching point pair based on the point pair transformation error and / or the mean value of the point pair transformation error, and use the current matching point pair set after deleting the true mismatching point pair as the next matching point pair set; A determination unit is used to determine all matching point pairs in the current matching point pair set as registration feature point pairs between the reference image and the image to be registered if none of the candidate mismatching point pairs belong to real mismatching point pairs.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the feature point matching method according to any one of claims 1 to 4 is implemented.
7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the feature point matching method according to any one of claims 1 to 4 is implemented.
