Feature point mismatch processing method and device, electronic equipment and readable medium

Abstract

The application relates to a feature point mismatch processing method and device, electronic equipment and a readable medium. The method comprises the following steps: matching initial feature points of two pictures to obtain a matching result; the matching result comprises a plurality of matched first feature point pairs; the moving vectors of the first feature point pairs in the two pictures are calculated; the target feature point pair is determined according to the moving vectors of the first feature point pairs; and the target feature point pair is removed from the matching result. The scheme provided by the application can reduce the probability of feature point mismatch and improve the matching accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a feature point mismatch processing method and device, electronic equipment and a readable medium. BACKGROUND

[0002] Feature point matching of an image plays an important basic role in complex computer vision tasks such as SLAM (Simultaneous localization and mapping) and SFM (structure from motion).

[0003] However, feature point mismatch phenomena are widespread and inevitable, and the related art cannot effectively eliminate the mismatched feature points, thereby reducing the accuracy of the calculation result. SUMMARY

[0004] To solve or partially solve the problems in the related art, the present application provides a feature point mismatch processing method and device, electronic equipment and a readable medium, which can reduce the probability of feature point mismatch and improve the matching accuracy.

[0005] The first aspect of the present application provides a feature point mismatch processing method, which comprises:

[0006] Matching initial feature points of two pictures to obtain a matching result; the matching result comprises a plurality of matched first feature point pairs;

[0007] Calculating the movement vectors of each first feature point pair in the two pictures;

[0008] Determining a target feature point pair that is mismatched according to the movement vectors of each first feature point pair;

[0009] Eliminating the target feature point pair from the matching result.

[0010] In an embodiment, the movement vector comprises a movement length and a movement direction; and determining a target feature point pair that is mismatched according to the movement vectors of each first feature point pair comprises:

[0011] Comparing the movement length of each first feature point pair with a preset length, respectively;

[0012] Based on the comparison result, determining a first feature point pair whose movement length is greater than or equal to the preset length as a second feature point pair;

[0013] Determining a target feature point pair that is mismatched according to the movement direction of each second feature point pair.

[0014] In an embodiment, after comparing the moving length of each first feature point pair with the preset length, the method further comprises:

[0015] determining the first feature point pair with the moving length less than the preset length as a third feature point pair based on the comparison result;

[0016] judging whether each third feature point pair has three-dimensional coordinates;

[0017] if not, eliminating the third feature point pair without three-dimensional coordinates from the matching result.

[0018] In an embodiment, the determining of the target feature point pair with mis-matching according to the moving direction of each second feature point pair comprises:

[0019] determining the adjacent area of each second feature point pair;

[0020] for each second feature point pair, determining other second feature point pairs in the adjacent area as corresponding adjacent second feature point pairs; wherein the corresponding adjacent second feature point pairs of each second feature point pair comprise a plurality of second feature point pairs;

[0021] calculating the included angle between the moving direction of each second feature point pair and the moving direction of each corresponding adjacent second feature point pair to obtain a plurality of included angle results of each second feature point pair;

[0022] determining the target feature point pair with mis-matching according to the plurality of included angle results of each second feature point pair.

[0023] In an embodiment, the determining of the target feature point pair with mis-matching according to the plurality of included angle results of each second feature point pair comprises:

[0024] for each second feature point pair, determining the target proportion of the included angle less than a preset included angle threshold based on the plurality of included angle results;

[0025] determining the target feature point pair with mis-matching according to the target proportion of each second feature point pair.

[0026] In an embodiment, the determining of the target feature point pair with mis-matching according to the target proportion of each second feature point pair comprises:

[0027] determining the second feature point pair with the target proportion less than a preset proportion threshold as the target feature point pair with mis-matching.

[0028] In an embodiment, the two pictures are two pictures with a preset number of frames in between; and the calculating of the moving vector of each first feature point pair in the two pictures comprises:

[0029] obtaining pixel coordinates of each first feature point pair in two pictures with a preset number of frames in the interval;

[0030] calculating a movement vector of each first feature point pair by using the pixel coordinates in the two pictures with the preset number of frames in the interval.

[0031] The second aspect of the present application provides a feature point mis-matching processing device, and the device comprises:

[0032] a matching module, configured to match initial feature points of two pictures to obtain a matching result; the matching result comprises a plurality of matched first feature point pairs;

[0033] a movement vector calculation module, configured to calculate a movement vector of each first feature point pair in the two pictures;

[0034] a target feature point pair determination module, configured to determine a mis-matched target feature point pair according to the movement vector of each first feature point pair;

[0035] a first elimination module, configured to eliminate the target feature point pair from the matching result.

[0036] The third aspect of the present application provides an electronic device, comprising:

[0037] a processor; and

[0038] a memory having executable codes stored thereon, wherein the executable codes, when executed by the processor, cause the processor to perform the method as described above.

[0039] The fourth aspect of the present application provides a computer readable storage medium having executable codes stored thereon, wherein the executable codes, when executed by a processor of an electronic device, cause the processor to perform the method as described above.

[0040] The technical solution provided by the present application can include the following beneficial effects:

[0041] The solution provided by the present application matches initial feature points of two pictures to obtain a matching result; the matching result comprises a plurality of matched first feature point pairs; a movement vector of each first feature point pair in the two pictures is calculated; a mis-matched target feature point pair is determined according to the movement vector of each first feature point pair; and the target feature point pair is eliminated from the matching result. The present application determines whether mis-matching occurs based on the movement vector, and then eliminates the mis-matched target feature point pair, thereby reducing the probability of feature point mis-matching and improving matching accuracy.

[0042] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application, as claimed. BRIEF DESCRIPTION OF DRAWINGS

[0043] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout and in which:

[0044] Figure 1 is a flowchart of a feature point mis-matching processing method according to an embodiment of the present application;

[0045] Figure 2 is another flowchart of a feature point mis-matching processing method according to an embodiment of the present application;

[0046] Figure 3A is a schematic diagram of a previous time (or a next time) of two adjacent pictures;

[0047] Figure 3B is a schematic diagram of a previous time (or a next time) of two adjacent pictures;

[0048] Figure 4 is a flowchart of a feature point mis-matching processing method according to an embodiment of the present application;

[0049] Figure 5 is a schematic diagram of a feature point mis-matching processing apparatus according to an embodiment of the present application;

[0050] Figure 6 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] Embodiments of the present application will be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the application are shown. This application may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0052] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0053] It should be understood that, although the terms "first", "second", "third", etc. can be used in this application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information without departing from the scope of the present application. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0054] In the related art, the feature points of mis-matching cannot be effectively eliminated, resulting in reducing the accuracy of the calculation result.

[0055] To solve the above problems, the embodiment of the present application provides a feature point mis-matching processing method, which judges whether mis-matching occurs based on a movement vector, and then eliminates the target feature point pair of mis-matching, so as to reduce the probability of feature point mis-matching and improve the matching accuracy.

[0056] The technical solutions of the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0057] Figure 1 is a flowchart of the feature point mis-matching processing method shown in the embodiments of the present application.

[0058] Referring to Figure 1 , the feature point mis-matching processing method of the present application comprises:

[0059] S110, matching the initial feature points of two pictures to obtain a matching result; the matching result comprises a plurality of first feature point pairs of matching success.

[0060] In the embodiments of the present application, it can be applied to an image processing system, and the image processing system can acquire two pictures to be matched. Each picture contains a plurality of initial feature points, and the initial feature points can be pixel points on the picture.

[0061] The image processing system can obtain a matching result by matching the plurality of initial feature points corresponding to the two pictures. The matching result can comprise a plurality of first feature point pairs, and each first feature point pair can be a pair of initial feature points of matching success.

[0062] S111, calculating the movement vector of each first feature point pair in the two pictures.

[0063] The image processing system can calculate the movement vector of each first feature point pair in the two pictures, and the movement vector can represent the movement direction and movement length of the pixel position of the first feature point pair on the two pictures.

[0064] S112, determine the target feature point pair of mis-matching according to the movement vector of each first feature point pair.

[0065] After obtaining the movement vector of each first feature point pair, the image processing system can determine whether mis-matching occurs for each first feature point pair according to the movement vector of each first feature point pair, so as to determine the target feature point pair of mis-matching.

[0066] S113, eliminate the target feature point pair from the matching result.

[0067] After screening out the target feature point pair of mis-matching, the image processing system can eliminate the target feature point pair from the matching result, so as to reduce the probability of mis-matching of feature points and effectively improve the matching accuracy.

[0068] It should be noted that SLAM is mainly used to solve the positioning and map construction problem of a robot moving in an unknown environment. The goal of SFM is to automatically recover camera motion and scene structure using two scenes or multiple scenes. The embodiment of the present application can improve the mapping and positioning accuracy in SLAM and improve the target object recovery accuracy in SFM by improving the feature point matching accuracy.

[0069] As can be seen from the example, the scheme provided by the present application matches the initial feature points of two pictures to obtain a matching result. The matching result includes a plurality of first feature point pairs of successful matching. The movement vectors of the first feature point pairs are calculated. The target feature point pair of mis-matching is determined according to the movement vector of each first feature point pair. The target feature point pair is eliminated from the matching result. The present application determines whether mis-matching occurs based on the movement vector, and then eliminates the target feature point pair of mis-matching, so as to reduce the probability of mis-matching of feature points and improve the matching accuracy.

[0070] Figure 2 is another flowchart of the feature point mis-matching processing method shown in the embodiment of the present application.

[0071] Referring to Figure 2 , the feature point mis-matching processing method of the present application comprises:

[0072] S210, match the initial feature points of two pictures to obtain a matching result. The matching result includes a plurality of first feature point pairs of successful matching.

[0073] The embodiment of the present application can match the initial feature points of two pictures by an image processing system, so as to obtain a matching result. The initial feature points can be pixel points on a picture. The matching result can include a plurality of first feature point pairs. Each first feature point pair can be a pair of initial feature points of successful matching.

[0074] In one implementation, matching initial feature points of two images to obtain a matching result may include:

[0075] Extract multiple initial feature points from two images respectively; match the multiple initial feature points corresponding to the two images; determine the initial feature points that are successfully matched in pairs in the two images; determine the initial feature points that are successfully matched in pairs as the first feature point pairs, so as to obtain a matching result containing multiple first feature point pairs.

[0076] The two images can be two images captured at adjacent times, or two frames extracted from the captured video at a preset frame interval, where the preset frame interval is a natural number. Figures 3A-3B This is a schematic diagram of the previous (or next) moment in two adjacent frames, where, Figure 3A It could be an image captured a moment ago. Figure 3B It could be an image captured at a later time. The image processing system can... Figure 3A Multiple initial feature points can be extracted from them, and they can be obtained from... Figure 3B Multiple initial feature points are extracted. For example, in... Figure 3A In the diagram, dark points a, b, and c, and light point d can all be initial feature points; Figure 3B In the diagram, dark points e, f, and g, and light points h can all be initial feature points.

[0077] An image processing system can match multiple initial feature points corresponding to two images, and then determine the initial feature points that are successfully matched in pairs in the two images. The initial feature points that are successfully matched in pairs can be determined as the first feature point pairs, thus finally obtaining multiple first feature point pairs. The image processing system can generate matching results based on multiple first feature point pairs.

[0078] In one example, x1 initial feature points are extracted from the first image, and x2 initial feature points are extracted from the second image. The x1 and x2 initial feature points are then matched. If the p-th initial feature point in the first image successfully matches the q-th initial feature point in the second image, it means that the p-th and q-th initial feature points are common to both images, and therefore they are considered a first feature point pair. If the p-th initial feature point in the first image fails to match the q-th initial feature point in the second image, it means that the p-th and q-th initial feature points are not common to both images, and therefore they are not considered a first feature point pair.

[0079] like Figure 3A and Figure 3B As shown, Figure 3Aa point of the a point of the Figure 3B e point is two two matching successful initial feature points, so a point and e point can be determined as the first feature point pair; and, Figure 3A b point of the b point of the Figure 3B f point is two two matching successful initial feature points, so b point and f point can be determined as the first feature point pair; and, Figure 3A c point of the c point of the Figure 3B g point is two two matching successful initial feature points, so c point and g point can be determined as the first feature point pair; and Figure 3B d point of the d point of the Figure 3B h point is two two matching unsuccessful initial feature points, so d point and h point can be determined as not the first feature point pair.

[0080] The first feature point pair is a pair of two two matching successful initial feature points, in order to facilitate the distinction, the initial feature points constituting the first feature point pair can be determined as matching points. Wherein, the matching point can be composed of a center and a line segment, the center represents the position of the matching point on the current frame picture, and the other end of the line segment represents the position of the matching point on the last frame picture. For example, for the e point of the e point of the Figure 3A , the center of the e point represents the position of the e point on the Figure 3B , and the other end of the line segment of the e point represents the position of the e point on the Figure 3B .

[0081] S211, calculate the movement vector of each first feature point pair in two pictures; the movement vector includes the movement length and the movement direction.

[0082] The image processing system can calculate the movement vector of each first feature point pair in two pictures, the movement vector can represent the movement direction and the movement length of the pixel position of the first feature point pair on the two pictures, therefore, the movement vector includes the movement length and the movement direction.

[0083] In an embodiment, the two pictures are two pictures with a preset number of frames apart; calculating the movement vector of each first feature point pair in the two pictures can include:

[0084] Obtaining the pixel coordinates of each first feature point pair in two pictures with a preset number of frames apart; using the pixel coordinates in the two pictures with a preset number of frames apart, calculating the movement vector of each first feature point pair.

[0085] In specific implementation, the image processing system can obtain the pixel coordinates of each first feature point pair in two pictures with a preset number of frames apart, and then can use the pixel coordinates of each first feature point pair in two pictures with a preset number of frames apart to calculate the movement vector of each first feature point pair in two pictures with a preset number of frames apart. Wherein, the pixel coordinates can be two-dimensional coordinates P(u,v).

[0086] In an example, the first picture and the second picture are two pictures with a preset frame number in between. Wherein, for a certain first feature point pair i, assuming that the first feature point pair i is successfully matched by an initial feature point a on the first picture and an initial feature point e on the second picture, the initial feature point a and the initial feature point e are matching points constituting the first feature point pair i. If the pixel coordinate of the matching point a on the first picture is P i,1 (u i,1 ,v i,1 ), and the pixel coordinate of the matching point e on the second picture is P i,2 (u i,2 ,v i,2 ), then the pixel coordinates of the first feature point pair i in the two pictures with the preset frame number in between are P i,1 and P i,2 , respectively. The image processing system can perform a difference operation on the two pixel coordinates P i,1 and P i,2 , and the difference value is the movement vector Vec i of the first feature point pair i in the two pictures with the preset frame number in between, i.e. Vec i = P i,2 - P i,1 .

[0087] S212, compare the movement lengths of each first feature point pair with a preset length, respectively.

[0088] After obtaining the movement vector Vec i of each first feature point pair, the movement vector Vec i includes a movement length |Vec i | and a movement direction Vec i , and the image processing system can first compare the movement length |Vec i | of each first feature point pair with a preset length ε, respectively.

[0089] It should be noted that the parameter ε can be set according to actual conditions, for example, the parameter ε can be set as 2, and the embodiments of the present application do not limit this.

[0090] S213, based on the comparison result, determine the first feature point pair with a movement length greater than or equal to the preset length as a second feature point pair.

[0091] If the movement length of a certain first feature point pair is greater than or equal to the preset length, i.e. |Vec i | ≥ ε, then the first feature point pair can be determined as a second feature point pair, so as to include the second feature point pair in the consideration range of the subsequent algorithm of the present application, i.e. for |Vec ithe second feature point pair |Vec

[0092] In an embodiment, after comparing the movement length of each first feature point pair with the preset length respectively, the method can further comprise:

[0093] determining the first feature point pair with the movement length less than the preset length as a third feature point pair based on the comparison result; judging whether each third feature point pair has a three-dimensional coordinate; and if not, eliminating the third feature point pair without the three-dimensional coordinate from the matching result.

[0094] If the movement length of a certain first feature point pair is less than the preset length, i.e. i |<ε, it means that the movement distance of the first feature point pair is too short, and it is very likely that the relative movement of the two images is small, which will affect the judgment of the subsequent image processing system on other second feature point pairs. Therefore, the embodiment of the present application can determine the first feature point pair as a third feature point pair, and not consider the third feature point pair in the subsequent algorithm of the present application, i.e. for the third feature point pair |Vec i |<ε, the subsequent step of determining the mis-matched target feature point pair according to the movement direction is not performed.

[0095] In the field of visual SLAM and SFM, each initial feature point has a pixel coordinate when initially matched, i.e. each initial feature point has a two-dimensional coordinate P(u, v), but does not necessarily have a three-dimensional coordinate (x, y, z), while the three-dimensional coordinate (x, y, z) plays an important role in the field of visual SLAM and SFM. Therefore, for the third feature point pair |Vec i |<ε, the embodiment of the present application can judge whether each third feature point pair has a three-dimensional coordinate (x, y, z).

[0096] If a certain third feature point pair has a three-dimensional coordinate (x, y, z), the image processing system can determine the third feature point pair with the three-dimensional coordinate (x, y, z) as a correctly matched effective feature point pair, and retain the effective feature point pair in the matching result.

[0097] If a certain third feature point pair does not have a three-dimensional coordinate (x, y, z), since the third feature point pair does not have a substantial role in the field of visual SLAM and SFM, the third feature point pair can be directly eliminated from the matching result.

[0098] In addition, if it is not in the field of visual SLAM and SFM, but only ordinary image feature point matching, the embodiment of the present application can directly eliminate all third feature point pairs |Vec i |<ε from the matching result.

[0099] S214, Based on the movement direction of each second feature point pair, determine the target feature point pair that was mismatched.

[0100] Regarding |Vec i For second feature point pairs with |≥ε, the image processing system can determine whether each second feature point pair has been mismatched based on the direction of movement of each second feature point pair, thereby identifying the mismatched second feature point pairs as target feature point pairs.

[0101] In one embodiment, determining the mismatched target feature point pair based on the movement direction of each second feature point pair may include:

[0102] Determine the neighborhood of each pair of second feature points; for each pair of second feature points, determine the other pairs of second feature points in the neighborhood as the corresponding adjacent pairs of second feature points; wherein, each pair of second feature points has multiple corresponding adjacent pairs of second feature points; calculate the angle between the movement direction of each pair of second feature points and the movement direction of the corresponding adjacent pairs of second feature points to obtain multiple angle results for each pair of second feature points; based on the multiple angle results for each pair of second feature points, determine the target feature point pairs that are mismatched.

[0103] In the specific implementation, the pixel coordinates P of the matching point on the second image can be used. i,2 (u i,2 v i,2 ), as the origin coordinates of the coordinate axis, and then a range of horizontal coordinates can be set to (u i,2 -m,u i,2 +m), the range of the ordinate is (v i,2 -m,v i,2 The rectangular region of +m is used as the neighboring region of the second feature point pair formed by the matching point.

[0104] like Figure 3B As shown, the pixel coordinates P of point e are... i,2 (u i,2 v i,2 Using the origin coordinates of the coordinate axes as the origin, establish an abscissa range of (u) i,2 -m,u i,2 +m), the range of the ordinate is (v i,2 -m,v i,2 The rectangular region Y is the neighborhood of the second feature point pair formed by point e.

[0105] It should be noted that the parameter m can be set according to conditions such as image size, motion characteristics, and data characteristics. For example, the parameter m can be set to 30. This application embodiment does not limit this.

[0106] In a neighboring region, multiple pairs of second feature points move in similar directions. For example... Figure 4 As shown, region Y is the neighborhood of point e. Points e, f, and g within neighborhood Y move in similar directions. Therefore, embodiments of this application can determine whether a mismatch has occurred based on whether the moving direction of the second feature point pair is similar to the moving direction of its corresponding adjacent second feature point pair.

[0107] Based on this, embodiments of this application can first determine the corresponding adjacent pairs of second feature points for each pair of second feature points. In specific implementations, the image processing system, for each pair of second feature points, can determine other pairs of second feature points in its neighborhood as its corresponding adjacent pairs of second feature points. Here, each pair of second feature points may include multiple corresponding adjacent pairs of second feature points. For example, in... Figure 5 In region Y, the corresponding adjacent second feature point pair of point e includes points f and g.

[0108] After obtaining the corresponding adjacent pairs of second feature points for each pair, the image processing system can calculate the movement direction Vec for each pair of second feature points. i The direction of movement Vec for each adjacent pair of second feature points k The angle between them is used to obtain multiple angle results for each pair of second feature points.

[0109] In one example, suppose that the corresponding adjacent pairs of second feature points i include n, where n is a positive integer greater than 1. The image processing system can determine the movement direction Vec of the n adjacent pairs of second feature points. k Form the set of movement directions {Vec k Then the set of movement directions {Vec} can be calculated. k Each movement direction Vec in} k The movement directions Vec of the second feature point pair i respectively i The angle between k , where angle k = <Vec k Vec i >, thus obtaining the n included angles of the second feature point pair i.

[0110] Specifically, each included angle can be calculated using the cosine formula of the included angle. k ,For example:

[0111]

[0112] After obtaining the plurality of angle results for each second feature point pair, for each second feature point pair, it can be determined whether there is a movement abnormality based on the plurality of angle results, and then the second feature point pair with the movement abnormality can be determined as the target feature point pair of mis-matching.

[0113] In an embodiment, determining the target feature point pair of mis-matching according to the plurality of angle results of each second feature point pair can include:

[0114] For each second feature point pair, a target proportion of angles less than a preset angle threshold is determined based on the plurality of angle results, and the target feature point pair of mis-matching is determined according to the target proportion of each second feature point pair.

[0115] In an example, for the second feature point pair i, a number x of angles angle k less than a preset angle threshold a, i.e., x is the number satisfying angle k < a, is determined based on the n angle results. The image processing system can calculate the ratio of x to n, which is the target proportion P of the second feature point pair i, i.e., P = x / n, so as to determine whether the second feature point pair i is the target feature point pair of mis-matching based on the target proportion P.

[0116] According to the above example, the target feature point pair of mis-matching is determined from the plurality of second feature point pairs.

[0117] It should be noted that the parameter a can be set according to conditions such as picture size, motion characteristics, data characteristics, etc., for example, the parameter a can be set as a = π / 4, and the embodiments of the present application do not limit this.

[0118] In an embodiment, determining the target feature point pair of mis-matching according to the target proportion of each second feature point pair can include:

[0119] The second feature point pair with a target proportion less than a preset proportion threshold is determined as the target feature point pair of mis-matching.

[0120] In an example, for the second feature point pair i, the target proportion P of the second feature point pair i can be compared with a preset proportion threshold β. If the target proportion P is greater than or equal to the preset proportion threshold β, i.e., x / n ≥ β, it can be determined that the second feature point pair i is not the target feature point pair of mis-matching; if the target proportion P is less than the preset proportion threshold β, i.e., x / n < β, it can be determined that the second feature point pair i is the target feature point pair of mis-matching.

[0121] According to the above example, the second feature point pair with a target proportion less than a preset proportion threshold is determined as the target feature point pair of mis-matching from the plurality of second feature point pairs.

[0122] In addition, the image processing system can determine the second feature point pair with a target ratio greater than or equal to a preset ratio threshold as a correct matching effective feature point pair, and reserve the effective feature point pair in the matching result.

[0123] It should be noted that the parameter β can be set according to the picture size, motion characteristics, data characteristics, and the like, for example, the parameter β can be set as 0.6, and the embodiments of the present application do not limit this.

[0124] S215, eliminating the target feature point pair from the matching result.

[0125] After screening the mis-matched target feature point pair, the image processing system can eliminate the target feature point pair from the matching result, so as to reduce the probability of feature point mis-matching and effectively improve the matching accuracy. The embodiments of the present application can improve the mapping and positioning accuracy in SLAM and improve the target object restoration accuracy in SFM by improving the feature point matching accuracy.

[0126] As can be seen from the example, the scheme provided by the present application matches the initial feature points of two pictures to obtain a matching result; the matching result includes a plurality of matching successful first feature point pairs; the moving vector of each first feature point pair in the two pictures is calculated; the moving vector includes a moving length and a moving direction; the moving length of each first feature point pair is compared with a preset length respectively; based on the comparison result, the first feature point pair with a moving length greater than or equal to the preset length is determined as a second feature point pair; and the mis-matched target feature point pair is determined according to the moving direction of each second feature point pair. The present application judges whether mis-matching occurs based on the moving length and the moving direction, and then eliminates the mis-matched target feature point pair, so as to reduce the probability of feature point mis-matching and improve the matching accuracy.

[0127] Figure 5 is a feature point mis-matching processing flowchart shown by the embodiments of the present application, in order to enable those skilled in the art to better understand the embodiments of the present application, the embodiments of the present application are described below through the following examples.

[0128] S401, matching the initial feature points of two pictures with a preset frame interval to obtain a matching result; wherein the matching result includes a plurality of matching successful first feature point pairs, such as a first feature point pair i, which is described below.

[0129] S402, calculating the moving vector Vec i of the first feature point pair i in the two pictures with a preset frame interval; wherein the moving vector Vec i includes a moving length and a moving direction.

[0130] S403, judging whether the moving length |Vec i of the first feature point pair i is less than a preset length ε; if yes, jumping to step S410; if no, jumping to step S404.

[0131] S404, if |Vec i |≥ε, determining the first feature point pair i as the second feature point pair i; determining the adjacent area of the second feature point pair i; assuming that the adjacent area of the second feature point pair i contains other n second feature point pairs. i |≥ε, determining the first feature point pair i as the second feature point pair i; determining the adjacent area of the second feature point pair i; assuming that the adjacent area of the second feature point pair i contains other n second feature point pairs.

[0132] S405, determining the n second feature point pairs in the adjacent area of the second feature point pair i as the corresponding adjacent second feature point pairs of the second feature point pair i.

[0133] S406, calculating the moving direction Vec i of the second feature point pair i. k k S407, calculating the moving direction Vec k of each second feature point pair.

[0134] S407, calculating the number x of the second feature point pairs satisfying angle k <α; wherein, α is a preset angle threshold.

[0135] S408, judging whether the target ratio P of the second feature point pair i is less than a preset ratio threshold β; if yes, jumping to step S409a; if no, jumping to step S409b; wherein, the target ratio P is the ratio of the n angle results satisfying angle k <α, i.e. P=x / n.

[0136] S409a, if x / n<β, determining that the second feature point pair i is a mis-matched target feature point pair, thus eliminating the second feature point pair i from the matching result, and ending the flow.

[0137] S409b, if x / n≥β, determining that the second feature point pair i is a correctly matched effective feature point pair, thus keeping the second feature point pair i in the matching result, and ending the flow.

[0138] S410, if |Vec i |<ε, determining the first feature point pair i as the third feature point pair i. i |<ε.

[0139] S411, judging whether the third feature point pair i has three-dimensional coordinates (x, y, z); if yes, jumping to step S412a; if no, jumping to step S412b.

[0140] S412a, if the third feature point pair i has three-dimensional coordinates (x, y, z), it is determined that the third feature point pair i is a correct matching valid feature point pair, and therefore the third feature point pair i is retained in the matching result, and the flow ends.

[0141] S412b, if the third feature point pair i does not have three-dimensional coordinates (x, y, z), the third feature point pair i without three-dimensional coordinates (x, y, z) is directly excluded from the matching result.

[0142] Corresponding to the foregoing application function implementation method embodiments, the present application also provides a feature point mis-matching processing device, an electronic device, a computer readable storage medium, and corresponding embodiments.

[0143] Figure 6 is a structural schematic diagram of the feature point mis-matching processing device shown in the embodiments of the present application.

[0144] Referring to Figure 6 , the present application provides a feature point mis-matching processing device, which can include:

[0145] The matching module 501 is configured to match the initial feature points of two pictures to obtain a matching result, wherein the matching result includes a plurality of matching successful first feature point pairs.

[0146] The movement vector calculation module 502 is configured to calculate the movement vectors of each first feature point pair in the two pictures.

[0147] The target feature point pair determination module 503 is configured to determine the mis-matching target feature point pair according to the movement vectors of each first feature point pair.

[0148] The first exclusion module 504 is configured to exclude the target feature point pair from the matching result.

[0149] In an embodiment, the movement vector includes a movement length and a movement direction; the target feature point pair determination module 503 can include:

[0150] The comparison submodule is configured to compare the movement length of each first feature point pair with a preset length, respectively.

[0151] The second feature point pair determination submodule is configured to determine, based on the comparison result, the first feature point pair with a movement length greater than or equal to the preset length as a second feature point pair.

[0152] The target feature point pair determination submodule is configured to determine the mis-matching target feature point pair according to the movement direction of each second feature point pair.

[0153] In an embodiment, after comparing the moving lengths of each first feature point pair with the preset length respectively, the device can further include:

[0154] a third feature point pair determining module configured to determine, based on the comparison result, the first feature point pair with the moving length less than the preset length as a third feature point pair;

[0155] a judging module configured to judge whether each third feature point pair has a three-dimensional coordinate;

[0156] a second eliminating module configured to eliminate the third feature point pair without the three-dimensional coordinate from the matching result if the third feature point pair does not have the three-dimensional coordinate.

[0157] In an embodiment, the target feature point pair determining sub-module can include:

[0158] a neighboring area determining unit configured to determine a neighboring area of each second feature point pair;

[0159] a neighboring second feature point pair determining unit configured to determine, for each second feature point pair, other second feature point pairs in the neighboring area thereof as corresponding neighboring second feature point pairs; wherein the corresponding neighboring second feature point pairs of each second feature point pair include a plurality of second feature point pairs;

[0160] an included angle calculating unit configured to calculate an included angle between a moving direction of each second feature point pair and a moving direction of each corresponding neighboring second feature point pair to obtain a plurality of included angle results of each second feature point pair;

[0161] a target feature point pair determining unit configured to determine, according to the plurality of included angle results of each second feature point pair, a target feature point pair of mis-matching.

[0162] In an embodiment, the target feature point pair determining unit can include:

[0163] a target proportion determining sub-unit configured to determine, for each second feature point pair, a target proportion of an included angle less than a preset included angle threshold based on the plurality of included angle results;

[0164] a target feature point pair determining sub-unit configured to determine, according to the target proportion of each second feature point pair, a target feature point pair of mis-matching.

[0165] In an embodiment, the target feature point pair determining sub-unit can be specifically configured to:

[0166] determine the second feature point pair with the target proportion less than a preset proportion threshold as the target feature point pair of mis-matching.

[0167] In an embodiment, the two pictures are two pictures with a preset number of frames in between; the moving vector calculating module 502 can include:

[0168] The pixel coordinate obtaining submodule is configured to obtain pixel coordinates of each first feature point pair in two pictures with a preset frame number interval.

[0169] The movement vector calculating submodule is configured to calculate a movement vector of each first feature point pair by using the pixel coordinates in the two pictures with the preset frame number interval.

[0170] In an embodiment, the matching module 501 can include:

[0171] The initial feature point extracting submodule is configured to extract a plurality of initial feature points from the two pictures respectively.

[0172] The matching submodule is configured to match the plurality of initial feature points corresponding to the two pictures.

[0173] The matching success determining submodule is configured to determine the initial feature points that are successfully matched in pairs in the two pictures.

[0174] The first feature point pair determining submodule is configured to determine the initial feature points that are successfully matched in pairs as first feature point pairs, so as to obtain a matching result including a plurality of first feature point pairs.

[0175] As can be seen from the example, the scheme provided in the present application matches the initial feature points of the two pictures to obtain a matching result; the matching result includes a plurality of first feature point pairs that are successfully matched; the movement vectors of each first feature point pair in the two pictures are calculated; the target feature point pairs that are mis-matched are determined according to the movement vectors of each first feature point pair; and the target feature point pairs are excluded from the matching result. The present application judges whether mis-matching occurs based on the movement vectors, and then excludes the target feature point pairs that are mis-matched, thereby reducing the probability of feature point mis-matching and improving the matching accuracy.

[0176] As to the apparatus in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments of the method, and will not be described in detail here.

[0177] ​ FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application.

[0178] Referring to FIG. 6, ​ The electronic device 600 includes a memory 610 and a processor 620.

[0179] The processor 620 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can be any conventional processor.

[0180] The memory 610 can include various types of storage units, such as a system memory, a read-only memory (ROM), and a permanent storage device. Among them, the ROM can store static data or instructions required by the processor 620 or other modules of the computer. The permanent storage device can be a read-and-write storage device. The permanent storage device can be a non-volatile storage device that does not lose stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a mass storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device can be a removable storage device (such as a floppy disk, an optical drive). The system memory can be a read-and-write storage device or a volatile read-and-write storage device, such as a dynamic random access memory. The system memory can store some or all of the instructions and data required by the processor during runtime. In addition, the memory 610 can include a combination of any computer-readable storage media, including various types of semiconductor memory chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic disks and / or optical disks. In some embodiments, the memory 610 can include a read-and / or write removable storage device, such as a compact disc (CD), a read-only digital versatile disc (such as DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a min SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include a carrier wave and a transient electronic signal transmitted through a wireless or wired transmission.

[0181] The memory 610 stores executable code, which, when processed by the processor 620, can cause the processor 620 to perform part or all of the above-mentioned methods.

[0182] Furthermore, the method according to the present application can also be implemented as a computer program or a computer program product, which comprises computer program code instructions for executing some or all of the steps of the above-mentioned method according to the present application.

[0183] Alternatively, the present application can also be implemented as a computer readable storage medium (or a non-transitory machine readable storage medium or a machine readable storage medium) having stored thereon executable codes (or computer programs or computer instruction codes) which, when executed by a processor of an electronic device (or a server, etc.), cause the processor to perform some or all of the steps of the above-mentioned method according to the present application.

[0184] The above has described the embodiments of the present application, the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, practical application, or improvement to the technology in the market, or to enable other ordinary skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for handling feature point mismatches, characterized in that, The method includes: The initial feature points of two images are matched to obtain a matching result; the matching result includes multiple successfully matched pairs of first feature points. Calculate the movement vector of each first feature point pair in the two images; Based on the movement vectors of each of the first feature point pairs, determine the mismatched target feature point pairs; Remove the target feature point pair from the matching result; The movement vector includes a movement length and a movement direction; determining the mismatched target feature point pair based on the movement vectors of each first feature point pair includes: The movement length of each of the first feature point pairs is compared with a preset length; Based on the comparison results, the first feature point pair whose moving length is greater than or equal to the preset length is determined as the second feature point pair; Determine the neighborhood of each pair of second feature points; For each pair of second feature points, other pairs of second feature points within the adjacent region are determined as corresponding adjacent pairs of second feature points; wherein, each pair of second feature points includes multiple corresponding adjacent pairs of second feature points. Calculate the angle between the moving direction of each second feature point pair and the moving direction of each corresponding adjacent second feature point pair to obtain multiple angle results for each second feature point pair. Based on the multiple angle results of each second feature point pair, the mismatched target feature point pair is determined.

2. The method according to claim 1, characterized in that, After comparing the movement length of each of the first feature point pairs with a preset length, the method further includes: Based on the comparison results, the first feature point pair whose moving length is less than the preset length is determined as the third feature point pair; Determine whether each pair of third feature points has three-dimensional coordinates; If a third feature point pair lacks three-dimensional coordinates, it is removed from the matching results.

3. The method according to claim 1, characterized in that, The step of determining the mismatched target feature point pair based on the multiple included angle results of each of the second feature point pairs includes: For each pair of second feature points, a target proportion with an angle less than a preset angle threshold is determined based on the multiple angle results; Based on the target proportion of each second feature point pair, the mismatched target feature point pairs are determined.

4. The method according to claim 3, characterized in that, The step of determining the mismatched target feature point pairs based on the target proportion of each of the second feature point pairs includes: The second feature point pair whose target ratio is less than a preset ratio threshold is determined as a mismatched target feature point pair.

5. The method according to claim 1, characterized in that, The two images are two images spaced apart by a preset number of frames; the calculation of the movement vector of each first feature point pair in the two images includes: Obtain the pixel coordinates of each first feature point pair in two images with a preset frame interval; The movement vector of each first feature point pair is calculated using the pixel coordinates in two images with a preset frame interval.

6. A processing device for feature point mismatch, characterized in that, The device includes: The matching module is used to match the initial feature points of two images to obtain the matching results; the matching results include multiple successfully matched pairs of first feature points. The motion vector calculation module is used to calculate the motion vector of each first feature point pair in the two images; The target feature point pair determination module is used to determine the mismatched target feature point pairs based on the movement vectors of each of the first feature point pairs. The first elimination module is used to eliminate the target feature point pairs from the matching results; The movement vector includes a movement length and a movement direction; the target feature point pair determination module includes: The comparison submodule is used to compare the movement length of each first feature point pair with a preset length. The second feature point pair determination submodule is used to determine the first feature point pair whose moving length is greater than or equal to the preset length as the second feature point pair based on the comparison result; The target feature point pair determination submodule is used to determine the neighborhood of each second feature point pair; for each second feature point pair, other second feature point pairs in the neighborhood are determined as the corresponding adjacent second feature point pairs; wherein, each second feature point pair has multiple corresponding adjacent second feature point pairs; the angle between the movement direction of each second feature point pair and the movement direction of the corresponding adjacent second feature point pairs is calculated to obtain multiple angle results for each second feature point pair; based on the multiple angle results for each second feature point pair, the mismatched target feature point pairs are determined.

7. An electronic device, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-5.

8. A computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-5.

Images