Feature Point Matching Method, Device, Medium and Program Product of Image

Abstract

An embodiment of the present application provides a method, device, medium and program product for image feature point matching, the method comprising: sampling multiple target matching point pairs from multiple initial matching point pairs; the multiple initial matching point pairs are obtained by performing initial matching on feature points on a first image and feature points on a second image; based on first position distribution information of first feature points on the first image in the multiple target matching point pairs, and second position distribution information of second feature points on the second image in the multiple target matching point pairs, determining whether the multiple target matching point pairs are matching point pairs that meet target conditions; if so, screening the multiple initial matching points based on a basic matrix constructed by the multiple target matching point pairs to obtain matching feature point pairs of the first image and the second image; wherein a pair of matching feature point pairs corresponds to the same target point in space.

Description

[0001] Technology Neighborhood

[0002] The present application relates to the field of image processing technology, and in particular to a method, device, medium and program product for matching feature points of an image. Background Art

[0003] The extraction and matching of stereoscopic image feature points is an important task in computer vision. The extraction and matching of stereoscopic image feature points mainly include: extracting feature points of two images at different shooting angles, and matching the feature points extracted from the two images.

[0004] Among them, the matching methods used when matching the feature points extracted from the two images include brute force matching, cross matching, K-nearest neighbor matching, etc. The above matching methods are only rough matching. Due to the complexity and changeability of actual scenes, the feature points extracted from the images have problems such as scale, rotation and similarity. After rough matching, there are still many incorrect matching point pairs.

[0005] In practice, the main goal of feature point matching is to try to identify feature points in both images that belong to the same target point in space as a correct matching point pair. For example, if a target point A in space has corresponding feature points in both image 1 and image 2, the matching goal is to find the feature points corresponding to target point A in both image 1 and image 2, and then match these two feature points. Therefore, it is necessary to eliminate incorrect matching point pairs in the coarse matching process to retain only correct matching point pairs as much as possible.

[0006] In the related art, the main method for screening out the correct matching point pairs includes the RANSAC (Random Sample Consensus) method based on the basic matrix. This method screens matching point pairs based on the idea of epipolar constraints. The main process is: first sample multiple matching point pairs, then use the sampled target matching point pairs to calculate a basic matrix, and then use the basic matrix to screen out point pairs that meet the epipolar constraints from all matching point pairs. The matching point pairs that meet the epipolar constraints are screened out as the correct matching point pairs.

[0007] However, the above method is not accurate enough for screening the initial matching point pairs, and there will still be a large number of incorrect matching point pairs, resulting in inaccurate feature point matching results. When the proportion of incorrect matching point pairs is high, the time consumed by the matching process will be greatly extended, resulting in long-term occupation of the computing resources of the terminal device. Summary of the Invention

[0008] In view of the above problems, the embodiments of the present application are proposed to overcome the above problems or at least partially solve the above problems.

[0009] In order to solve the above problems, the present application provides a method for matching feature points of an image, the method comprising:

[0010] Sampling a plurality of target matching point pairs from a plurality of initial matching point pairs; the plurality of initial matching point pairs are obtained by performing initial matching on feature points on the first image and feature points on the second image;

[0011] Determining whether the plurality of target matching point pairs are matching point pairs that meet a target condition based on first position distribution information of first feature points on the first image among the plurality of target matching point pairs and second position distribution information of second feature points on the second image among the plurality of target matching point pairs;

[0012] If so, multiple initial matching points are screened based on the basic matrix constructed by multiple target matching point pairs to obtain matching feature point pairs of the first image and the second image; wherein a pair of matching feature point pairs corresponds to the same target point in space.

[0013] An embodiment of the present application further discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the image feature point matching method as described in the first aspect.

[0014] An embodiment of the present application further discloses a computer-readable storage medium, which stores a computer program that enables a processor to execute the image feature point matching method as described in the first aspect of the present application.

[0015] An embodiment of the present application further discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the image feature point matching method described in the first aspect.

[0016] In practice, when multiple matching point pairs are correct matching point pairs, the position distribution of each feature point in the multiple matching point pairs on their respective images is roughly the same. For example, the difference between the two images of the same object is not large. Therefore, the image feature point matching method of the embodiment of the present application can be used to verify the multiple matching point pairs based on the difference between the first position distribution information and the second position distribution information. Since the first position distribution information represents the relative position relationship of multiple first feature points on the first image, and the second position distribution information represents the relative position relationship of multiple second feature points on the second image, it is possible to accurately determine whether the multiple target matching point pairs are matching point pairs that meet the target conditions. In this way, it can be ensured that the multiple sampled matching point pairs are as correct matching point pairs as possible, that is, the probability that the sampled target matching point pairs are correct matching point pairs is increased, thereby producing the following advantages:

[0017] On the one hand, when verification passes, a fundamental matrix can be constructed based on target matching point pairs with a higher probability of being correct. When the probability that the target matching point pairs used to calculate the fundamental matrix are correct matching point pairs is higher, the accuracy of the calculated fundamental matrix is also higher, thereby improving the accuracy of the fundamental matrix calculation. Furthermore, since correct matching point pairs are screened based on the fundamental matrix, the accuracy of the epipolar lines calculated using the fundamental matrix is higher when the accuracy of the fundamental matrix is improved. As a result, the probability that the matching point pairs that are screened and meet the epipolar constraints are correct is higher, thereby significantly reducing the number of incorrect matching point pairs after screening and improving the accuracy of the feature point matching results.

[0018] On the other hand, since the basic matrix is constructed only when multiple target matching point pairs are verified (the target matching point pairs are matching point pairs that meet the target conditions), it is possible to avoid using the basic matrix constructed by matching point pairs with a higher error rate to screen feature points, thereby ensuring that each matching feature point pair screened out is a point pair with a higher accuracy rate. In this way, the feature point matching results of the first image and the second image can be obtained as soon as possible, and the time consumed by the entire matching process will be greatly shortened, thereby avoiding long-term occupation of the computing resources of the terminal device. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 Schematic diagram of binocular epipolar constraint in an embodiment of the present application;

[0021] Figure 2 This is a flowchart of the steps of a method for matching feature points of an image in an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of obtaining position distribution information through geometric structure in the implementation of this application;

[0023] Figure 4 Schematic diagram of the topological structure obtained in two ways in the embodiment of the present application;

[0024] Figure 5 This is a flowchart of how to determine distribution distance difference information and distribution direction difference information in an embodiment of the present application;

[0025] Figure 6 This is a structural block diagram of a feature point matching device for an image in an embodiment of the present application;

[0026] Figure 7 This is a structural block diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0027] To make the above-mentioned purposes, features, and advantages of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of this application.

[0028] In recent years, significant progress has been made in AI-based research on computer vision, deep learning, machine learning, image processing, and image recognition. Artificial Intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems for simulating and extending human intelligence. AI is a comprehensive discipline encompassing numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, a key branch of AI, specifically enables machines to understand the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous location and mapping (SLAM), computational photography, and robotic navigation and position determination. With the research and advancement of artificial intelligence technology, this technology has been applied in many fields, such as urban management, traffic management, building management, park management, facial access, facial attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone imaging, cloud services, smart homes, wearable devices, unmanned driving, autonomous driving, smart medical care, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile Internet, live streaming, beauty, makeup, medical beauty, smart temperature measurement and other fields.

[0029] The research goal of computer vision is to enable computers to perceive three-dimensional environments through two-dimensional images, including the shape, position, posture, and movement of objects in the three-dimensional environment, and to describe, store, identify, and understand them. Therefore, extracting and matching feature points from stereoscopic images has become a key task in computer vision.

[0030] In related technologies, after feature point extraction and matching, a feature point purification algorithm is introduced to eliminate incorrect matching point pairs and retain correct matching point pairs. The main methods are as follows:

[0031] (1) The RANSAC method based on the homography matrix H is a projection relationship matrix between two planes and is not suitable for most scenarios.

[0032] (2) The RANSAC method based on the basic matrix F. This method uses the idea of epipolar constraint to screen out correctly matched matching point pairs. As described in the background technology section above, since this method is based on the idea of epipolar constraint, it can only constrain the epipolar line direction. There will still be a large number of incorrect matching point pairs. When the proportion of incorrect matching point pairs is high (>50%), the time to screen out the correct matching point pairs will increase by 5 times or 10 times.

[0033] (3) Based on the improvement of method (2), the main issues of data sampling, degradation and threshold setting of the RANSAC method are considered. However, there is also the problem that only the direction of the extreme lines can be constrained, resulting in a large number of incorrect matching point pairs. When the proportion of incorrect matching point pairs is high (>50%), the time will increase dramatically.

[0034] Therefore, in the related technology, the correct matching point pairs are screened out based on the idea of epipolar constraints, but there are still problems such as a large number of incorrect matching point pairs and excessive calculation time. Excessive calculation time will cause more computer computing resources to be occupied, which is not conducive to the performance maintenance of computer hardware and the full utilization of computing resources.

[0035] To facilitate understanding of this application, the RANSAC method based on epipolar constraints is first introduced.

[0036] Reference Figure 1 As shown in Figure 1, a schematic diagram of binocular epipolar constraints is shown. Figure 1 As shown, P is a point in the real space, O1 and O2 can be understood as the points where the camera lens is located, I1 is the imaging plane corresponding to O1, I2 is the imaging plane corresponding to O2, the imaging point of P on the imaging plane I1 is P1, and the imaging point on the imaging plane I2 is P2, the intersection of the line between O1 and O2 and the two imaging planes is e1 and e2, which are also called poles, the straight line between e1 and P1 is the pole line L1, and the straight line between e2 and P2 is the pole line L2.

[0037] Among them, the significance of the polar line is that when point P moves along the straight line direction of P-P1, its imaging point on the imaging plane I2 must move on the polar line L2; when point P moves along the straight line direction of P-P2, its imaging point on the imaging plane I1 must move on the polar line L1.

[0038] That is to say, if points P1 and P2 are matched, then P2 will be located on the epipolar line projected from P1 onto the imaging plane I2. Therefore, the idea of epipolar constraint is adopted in the RANSAC method. Specifically, the epipolar line of P1 on the imaging plane I2 is first calculated, and then the distance from P2 to the epipolar line is calculated. If the distance is very small, then there is a high probability that P1 and P2 are the imaging points of point P, and the two are the correct matching point pair.

[0039] Specifically, the epipolar line of point P1 on imaging plane I2 can be calculated using the formula L2 = F * P1, where F is the fundamental matrix. Therefore, the accuracy of the fundamental matrix calculation must be guaranteed. Generally speaking, to calculate the fundamental matrix, multiple matching point pairs are first sampled from the multiple initial matching point pairs obtained by coarse matching. The fundamental matrix is then calculated based on these multiple sampled matching point pairs. The calculation method of the fundamental matrix can refer to related technologies, for example, a normalized 8-point estimation algorithm can be used.

[0040] Based on the above principles, we can know that:

[0041] First, if the basic matrix is closer to the correct matrix in the real situation, the closer the epipolar line of point P1 on the imaging plane I2 is to the epipolar line in the real situation (reflecting the camera pose in the real situation), the higher the accuracy of determining whether P2 is the point matching P1;

[0042] Second, the RANSAC method based on the basic matrix relies on the epipolar line to constrain the matching point pairs. Although this method can obtain a straight line (epiped) of P1 on the imaging plane I2, it cannot know the specific position of P1 on the imaging plane I2. It relies on calculating the distance from P2 to the epipolar line, which is still a point-to-line constraint. It can only be constrained in the direction of the epipolar line. Therefore, there will be a problem of incorrect matching point pairs after screening.

[0043] In view of this, the present application proposes a feature point matching scheme based on the RANSAC method based on the idea of epipolar constraint. The core improvement idea is: for the sampled target matching point pairs, according to the difference between the position distribution information of the feature points in these target matching point pairs on the two images, the sampled target matching point pairs are verified for validity, so as to increase the probability that the sampled target matching point pairs are correct matching point pairs, thereby improving the accuracy of the calculated basic matrix, and then calculating the basic matrix based on the sampled target matching point pairs after the verification, and then using the basic matrix to filter out the matching point pairs that meet the epipolar constraint from all matching point pairs, thereby achieving the purpose of proposing the incorrect matching feature point pairs in the matching points and obtaining matching feature point pairs with high accuracy.

[0044] On the one hand, each match is performed based on a valid fundamental matrix, avoiding the need to screen initial matching point pairs based on a less accurate fundamental matrix. This ensures that each screening of correct matching feature point pairs is highly accurate and effective, allowing for the quickest possible screening of matching feature point pairs that meet expectations, shortening matching time. On the other hand, the improved accuracy of the fundamental matrix calculation results in a higher accuracy rate for epipolar lines calculated from the fundamental matrix. Consequently, the probability that the selected matching point pairs that meet the epipolar constraints are correct is higher, further enhancing matching accuracy.

[0045] Based on the above description, a method for matching feature points of an image in this application is introduced. Figure 2 As shown, a flowchart of the steps of a feature point matching method for an image in an embodiment of the present application is shown, wherein the feature point matching method for an image in the present application can be applied to electronic devices, including but not limited to: servers, smart phones, tablets, e-book readers, MP3 (Moving Picture Experts GrouAudio Layer III, Moving Picture Experts Group Audio Layer 3) players, MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) players, laptop computers, car computers, PDAs, desktop computers, set-top boxes, smart TVs, wearable devices, etc.

[0046] like Figure 2 As shown, the following steps may be specifically included:

[0047] Step S201: Sampling multiple target matching point pairs from multiple initial matching point pairs.

[0048] Among them, the multiple initial matching point pairs are obtained by initially matching the feature points on the first image and the feature points on the second image; each matching point pair in the multiple target matching point pairs includes a first feature point located on the first image and a second feature point located on the second image.

[0049] Step S202: Based on the first position distribution information of the first feature points on the first image among the multiple target matching point pairs, and the second position distribution information of the second feature points on the second image among the multiple target matching point pairs, determine whether the multiple target matching point pairs are matching point pairs that meet the target conditions.

[0050] Step S203: If yes, then based on the basic matrix constructed by the multiple target matching points, the multiple initial matching points are screened to obtain matching feature point pairs of the first image and the second image.

[0051] Among them, a pair of matching feature points corresponds to the same target point in the space.

[0052] In this embodiment, the first image and the second image can be two images with a common field of view taken from different perspectives, such as two images taken from a binocular perspective, the first image can be an image from a left perspective, and the second image can be an image from a right perspective; or, the first image can be an image from a right perspective, and the second image can be an image from a left perspective.

[0053] The multiple initial matching point pairs are obtained by performing an initial matching of feature points on the first image and feature points on the second image. This initial matching can be referred to as a coarse matching. Specifically, when performing the initial matching, feature extraction can be performed on the first image and the second image, respectively, to obtain multiple feature points on the first image and multiple feature points on the second image. After the feature points are extracted, they can be described, specifically using descriptors, such as using a histogram of pixel gradients near the feature points to describe the feature points. The feature point descriptors are mainly used to match feature points between the two images, that is, feature points with similar descriptors on the two images are treated as a pair of matching feature point pairs, so as to roughly perform an initial matching of the imaging points of the same spatial point in the two images.

[0054] Among them, when performing the initial matching of the extracted feature points, any existing matching method can be used, such as brute force matching, cross matching, and K-means matching method. This application does not limit the method of initial matching. Among them, the brute force matching method is to arbitrarily select a feature point on the first image, and then calculate the matching degree between the descriptors with all the feature points on the second image, sort the matching degrees, and finally select the feature point with the highest matching degree as a pair of matching points. Cross matching is a brute force matching after a brute force matching, and if the matched point is still the point matched for the first time, it is considered to be a correct match. For example, using the brute force matching method, the feature point matched by feature point A is feature point B; conversely, using feature point B for brute force matching, if the matched point is still feature point A, it is considered to be a correct match.

[0055] After obtaining multiple initial matching point pairs, since there are erroneous matching point pairs among the initial matching point pairs, it is necessary to further screen the multiple initial matching point pairs to eliminate the erroneous matching point pairs and extract the correct matching point pairs that belong to the correct match. This application adopts the RANSAC method based on the basic matrix for screening, so it is necessary to first sample multiple target matching point pairs for calculating the basic matrix.

[0056] Among them, a random sampling function can be used to sample a target number of matching point pairs from multiple initial matching point pairs. The sampled matching point pairs are called target matching point pairs. The target number can be determined by the number of point pairs required to calculate the basic matrix. For example, if the 8-point method is used to calculate the basic matrix, 8 matching point pairs need to be sampled; if the 7-point method is used to calculate the basic matrix, 7 matching point pairs need to be sampled.

[0057] Among them, since the initial matching point pair is obtained by initially matching the feature points on the first image and the feature points on the second image, an initial matching point pair includes a feature point on the first image and a feature point on the second image, and each sampled matching point pair includes the first feature point on the first image and the second feature point on the second image.

[0058] It should be noted that the first feature point in this application refers to a feature point located on the first image among the sampled target matching point pairs, and the second feature point refers to a feature point located on the second image among the sampled target matching point pairs.

[0059] Among them, the first position distribution information of the first feature point on the first image and the second position distribution information of the second feature point on the second image in the multiple target matching point pairs can be obtained. Then, the validity of the multiple target matching point pairs can be verified based on the first position distribution information and the second position distribution information. The validity verification refers to determining whether the multiple target matching point pairs are matching point pairs that meet the target conditions. The target conditions can be: the number of erroneous matching point pairs in the sampled multiple target matching point pairs is less than the set number threshold, or the difference between the first position distribution information and the second position distribution information is less than the target difference. The target difference can be set in advance. When the target difference is used as the target condition, the verification of the target matching point pairs can be quantified, thereby improving the verification efficiency of the target matching point pairs.

[0060] In practice, since the first and second images are images with a common field of view captured from different perspectives, for example, the first and second images are obtained by capturing the same scene from different perspectives, and the same object in the scene has similar shapes in the first and second images, the relative positional relationships of the object's various parts in the first and second images are similar. Therefore, based on the difference between the first and second position distribution information, the validity of multiple target matching point pairs can be verified, ensuring that the number of incorrect matching point pairs among the sampled multiple matching point pairs is small.

[0061] The position distribution information may reflect the position distribution of multiple feature points sampled from each image, such as the spacing between feature points and the relative positional orientation between feature points. Specifically, the first position distribution information may include the spacing between first feature points on the first image and / or the relative positional orientation between feature points; the relative positional orientation is the azimuth angle of one feature point relative to another feature point. Of course, the second position distribution information may also include the spacing between second feature points on the second image and / or the relative positional orientation between feature points.

[0062] The difference between the first position distribution information and the second position distribution information may be determined by calculating the similarity between the first position distribution information and the second position distribution information, or calculating the difference between the first position distribution information and the second position distribution information.

[0063] If the difference between the two is small, for example, the similarity is higher than the similarity threshold, or the difference is less than the threshold difference, then it can be considered that the sampled target matching point pair is likely to be a correct match with a high probability, meets the target condition, and can pass the validity verification; if the difference between the first position distribution information and the second position distribution information is large, for example, the similarity is lower than the similarity threshold, or the difference is greater than the threshold difference, then it can be considered that the sampled target matching point pair is likely to be a correct match with a low probability, does not meet the target condition, and can fail the validity verification.

[0064] If the multiple target matching point pairs do not meet the target condition, the process may return to step S201 to resample from the multiple initial matching point pairs to obtain multiple new target matching point pairs.

[0065] In this embodiment, after the sampled target matching point pairs are verified, a basic matrix can be constructed based on multiple target matching point pairs. In practice, the method of constructing the basic matrix can refer to relevant technologies. For example, if 8 matching point pairs are sampled, the basic matrix is calculated using the 8-point method; if 7 matching point pairs are sampled, the basic matrix is calculated using the 7-point method.

[0066] After obtaining the fundamental matrix, the fundamental matrix can be used to eliminate incorrect matching point pairs from all initial matching point pairs to obtain correctly matched matching point pairs, which are the matching feature point pairs. The specific process can be: for each initial matching point pair, the fundamental matrix is used to calculate the epipolar line. Then, the distance from the feature point located in the second image of each initial matching point to the epipolar line after being mapped to the first image is calculated. If the distance is less than or equal to the target distance, for example, less than 3 pixels, then the initial matching point pair is a correct matching point pair. If the distance is greater than the target distance, then the initial matching point pair is an incorrect matching point pair and needs to be eliminated. In this way, incorrect matching point pairs can be continuously eliminated to screen out the correct matching feature point pairs.

[0067] By adopting the technical solution of the embodiment of the present application, the validity of the sampled target matching point pairs is verified based on the difference between the first position distribution information and the second position distribution information, thereby improving the probability that the matching point pairs that pass the verification are correct matches, thereby constructing a basic matrix with higher accuracy. When the accuracy of the basic matrix is improved, the accuracy of the extreme lines calculated by the basic matrix is higher, and thus, the probability that the matching point pairs screened out that meet the extreme constraints are correct matching point pairs is higher, thereby greatly reducing the number of existing erroneous matching point pairs and improving the accuracy of the feature point matching results.

[0068] In addition, since the basic matrix is constructed only when it is determined that multiple target matching point pairs meet the target conditions, if not, new multiple target matching point pairs are resampled. Therefore, each screening can be performed based on a basic matrix with higher accuracy, so that the effectiveness and accuracy of screening out the correct matching point pairs can be guaranteed, and the correct matching point pairs that meet the expected expectations can be screened out as quickly as possible. As a result, the time consumed by the entire matching process will be greatly shortened, avoiding long-term occupation of the computing resources of the terminal device.

[0069] The core improvement idea of the present application has been clearly explained through the above embodiments. Next, the optional implementation methods for realizing the above core improvement idea will be introduced. As described above, after obtaining multiple initial matching point pairs, the feature point matching process as a whole includes: a verification stage of verifying the sampled matching point pairs (determining whether multiple target matching point pairs are matching point pairs that meet the target conditions), and a stage of eliminating erroneous point pairs by constructing a basic matrix based on the sampled target matching point pairs and using the basic matrix to screen out correct matching point pairs that meet the epipolar constraints.

[0070] Below, the optional implementation methods of this application are introduced in stages.

[0071] 1.1, the verification phase of verifying the sampled target matching point pairs.

[0072] Since this application needs to verify the sampled multiple target matching point pairs based on the difference between the first position distribution information composed of multiple first feature points on the first image and the first position distribution information composed of multiple second feature points on the second image, it is necessary to first obtain the first position distribution information and the second position distribution information.

[0073] 1.1.1, the process of obtaining location distribution information.

[0074] In one optional approach, position distribution information can be obtained based on the geometric structure formed by multiple feature points. This geometric structure is a point-line connection structure formed by the lines connecting the feature points, which can fully reflect the relative positional relationships of the feature points. In this approach, the first position distribution information is obtained based on the geometric structure of multiple first feature points in the first image, and the second position distribution information is obtained based on the geometric structure of multiple second feature points in the second image.

[0075] When this optional method is adopted, it is necessary to ensure consistency between the connection method of the multiple first feature points and the connection method of the multiple second feature points, so as to compare the difference between the first position distribution information and the second position distribution information at the same scale.

[0076] Reference Figure 3 As shown in FIG, a schematic diagram of obtaining position distribution information through geometric structure is shown. Figure 3 As shown, 8 matching point pairs are sampled, and the corresponding 8 feature points can be connected in the same connection method on the first image taken at O1 and the second image taken at O2, to obtain a geometric structure consisting of 8 first feature points on the first image and a geometric structure consisting of 8 second feature points on the second image; the geometric structure can fully describe the shape features composed of multiple feature points, so that the position distribution information obtained may include: the area size of the geometric structure, the spacing between points, and the angle between points.

[0077] This implementation allows for flexible configuration of the lines connecting feature points to match the image being matched. For example, if the image is of an object with a clearly defined shape, the image will contain objects with distinct shape features. For example, if an image is of a table and chair, the image will contain objects with distinct shape features. As a result, the geometric structure between feature points can more clearly reflect the shape features of the object. The difference between the first and second position distribution information determined thereby can better reflect the shape differences between the same batch of matching point pairs in the two images, improving the accuracy of validating the effectiveness of multiple target matching point pairs in this scenario.

[0078] Alternatively, position distribution information can be obtained from a topological structure formed by multiple feature points. This topological structure accurately describes the relative positional relationships between feature points. Specifically, the topological structure focuses on describing the relationships between feature points, including the distance relationship and the direction relationship between feature points. This is applicable to both images with unclear and obvious shape features.

[0079] In this manner, the process of obtaining the first location distribution information and the second location distribution information can be described as follows:

[0080] First, a first topological structure consisting of multiple first feature points on the first image and a second topological structure consisting of multiple second feature points on the second image are obtained; then, first position distribution information is determined based on the first topological structure, and second position distribution information is determined based on the second topological structure.

[0081] In a specific implementation, for multiple first feature points, a first topological structure composed of multiple first feature points can be obtained according to a specified relationship description rule. For multiple second feature points, a second topological structure composed of multiple second feature points can also be obtained according to the above relationship description rule. The relationship description rule can be: the relationship between a feature point and all other feature points, or it can be a single direction relationship between multiple feature points. This application does not limit the relationship description rule, and when determining the topological structure, it can be flexibly carried out according to actual circumstances.

[0082] For example, refer to Figure 4 As shown in the figure, the topological structures obtained in the two cases are shown. When the relationship description rule is the relationship between a feature point and all other feature points, one of the feature points can be used as the central feature point, and the central feature point can be connected with all other feature points to obtain the following: Figure 4 The topology shown in 4-1; Figure 4-1 The feature points p14 and p24 are the central feature points.

[0083] When the relationship description rule is a single direction relationship between multiple feature points, we can start from the left side of the image and sequentially connect multiple feature points from the left to the right to obtain the following: Figure 4 The topological structure shown in 4-2. Of course, in practice, it is also possible to start from the top of the image and connect multiple feature points from the top to the bottom in sequence.

[0084] In this optional manner, the position distribution information obtained based on the first topological structure may include: the distance between two adjacent feature points and the angle between three adjacent connected feature points. Figure 4 It can also be seen that when the topological structure is used to describe the positional relationship between feature points, the position distribution information can more finely reflect the positional distribution relationship of multiple first feature points on the first image and the positional distribution relationship of multiple second feature points on the second image, and can reduce the difficulty of calculating the differences between the position distribution information.

[0085] 1.1.2, Matching point pair verification based on location distribution information.

[0086] After obtaining the first position distribution information and the second position distribution information, it may be determined whether multiple matching point pairs in the multiple target matching point pairs are valid matching point pairs based on distribution difference information between the first position distribution information and the second position distribution information.

[0087] In an optional embodiment, since the position distribution information can reflect the relative position relationship between feature points, where the relative position relationship can include the size of the spacing distance between feature points and the orientation of the feature points, the first position distribution information can include the spacing distance between the first feature points and the relative position direction between the first feature points, and the second position distribution information can include the spacing distance between the second feature points and the relative position direction between the second feature points. Therefore, the validity of the matching point pair can be verified based on the difference between the spacing distance between the feature points and the relative position direction between the feature points.

[0088] In a specific implementation, distribution difference information corresponding to multiple target matching point pairs can be determined based on the first position distribution information and the second position distribution information. Furthermore, based on the feature point distribution difference information, it can be determined whether the multiple target matching point pairs satisfy the target condition. Verifying whether the target matching point pairs satisfy the target condition can also be understood as verifying the validity of the target matching point pairs.

[0089] The distribution difference information may include distribution distance difference information and / or distribution direction difference information.

[0090] Among them, the distribution distance difference information is represented by the difference between the first distance between the two first feature points in any two target matching point pairs and the second distance between the two second feature points, and the distribution direction difference information is represented by the difference between the first angle between the three first feature points in any three target matching point pairs and the second angle formed by the three second feature points.

[0091] Specifically, the distribution distance difference information represents the average difference in spacing distances between all matching point pairs in the multiple target matching point pairs. In one implementation, it can be obtained based on the average of the difference between the spacing distance (first distance) between two adjacent connected first feature points on the first image and the spacing distance (second distance) between two adjacent connected second feature points on the second image. The distribution direction difference information represents the comprehensive difference in orientation between all matching point pairs in the multiple target matching point pairs. In one implementation, it can be obtained based on the average of the difference between the first angle of every three first feature points on the first image and the second angle of every three second feature points on the second image.

[0092] In practice, the validity of multiple target matching point pairs can be verified through distribution distance difference information, or the validity of multiple target matching point pairs can be verified based on distribution direction difference information, or the validity of the set can be verified based on both distribution distance difference information and distribution direction difference information.

[0093] When implementing it, refer to Figure 5 As shown, Figure 5 A flowchart showing how to determine the distribution distance difference information and the distribution direction difference information is shown. Figure 5 The steps of determining the distribution distance difference information and the distribution direction difference information are shown respectively, as follows:

[0094] Step S501 : for every two adjacent first feature points, determine a distance ratio between a first distance between two second feature points and a second distance between two corresponding second feature points, and determine the distribution distance difference information based on each distance ratio.

[0095] Step S502 : for every three adjacent first feature points, determine the angle difference between the first angle between the three first feature points and the second angle between the corresponding three second feature points, and determine the distribution direction difference information based on the obtained angle differences.

[0096] In this embodiment, since the first position distribution information may include the spacing distance between the first feature points and the relative position directions between the first feature points, the second position distribution information may include the spacing distance between the second feature points and the relative position directions between the second feature points.

[0097] Therefore, the first distance between two adjacent first feature points and the first angle between three adjacent first feature points can be determined from the first position distribution information; and the second distance between two adjacent second feature points and the second angle between three adjacent second feature points can be determined from the second position distribution information.

[0098] Then, according to the pairing relationship between the first feature point and the second feature point, the ratio between the first distance and the corresponding second distance is taken as the distance ratio. Since the multiple target matching point pairs include multiple matching point pairs, according to the distance ratio determination process, two matching point pairs correspond to one distance ratio. Therefore, the multiple target matching point pairs have multiple distance ratios.

[0099] Among them, the first distance can reflect the length of each line segment when multiple first feature points are connected on the first image, and the second distance can reflect the length of each line segment when multiple second feature points are connected on the second image; in this way, the distance ratio can reflect the difference between the side length on the first image and the corresponding side length on the second image.

[0100] Similarly, according to the pairing relationship between the first feature point and the second feature point, the ratio between the first angle and the corresponding second angle is used as the angle difference. Accordingly, a pair of multiple target matching points has multiple angle differences.

[0101] For example, Figure 4 As shown in Figure 4-2, 8 matching point pairs are sampled. The 8 points included in the left image (the first image taken at O1) are p11-p18, and the 8 points included in the right image (the second image taken at O2) are p21-p28; accordingly, p11 and p21 are a pair of matching point pairs, p12 and p22 are a pair of matching point pairs, and so on.

[0102] First, we can obtain the distance ratio corresponding to each pair of matching points. Figure 4 As shown, the ratio of the spacing between p11 and p12 to the spacing between p21 and p22 is d1, which is the ratio of the distances between the matching point pair of p11 and p21 and the matching point pair of p12 and p22. From this, we can obtain the distance ratios of two matching point pairs in the other six matching point pairs. For example, the distance ratio between the matching point pair of p13 and p23 and the matching point pair of p15 and p25 is d2, the distance ratio between the matching point pair of p15 and p25 and the matching point pair of p14 and p24 is d3, and the distance ratio between the matching point pair of p14 and p24 and the matching point pair of p16 and p26 is d4. Thus, d1-d7 reflect the length differences between each line segment in the topological structure of the first image and the corresponding line segment in the topological structure of the second image.

[0103] Among them, for the determination of the angle ratio, such as Figure 4 As shown in Figure 4-2, the angle can be represented by the angle formed by connecting three feature points. For example, on the first image, p11, p12, and p13 form angle θ1, and on the second image, p21-p23 form angle θ2. The difference between θ1 and θ2 is the angle difference. Similarly, six angle differences can be obtained. Since the three feature points form a triangle, the angle can be any angle in the triangle.

[0104] In this embodiment, the difference between each two distance ratios can be obtained, thereby obtaining the distribution distance difference information between each two distance ratios. For example, the difference between d1 and d2 is used as the distribution distance difference information, and the difference between d1 and d3 is used as the distribution distance difference information. In this way, multiple distribution distance difference information is obtained, which can reflect the difference in spacing distance between multiple first feature points and multiple second feature points as a whole.

[0105] The difference between two corresponding angles in the first and second images can be obtained to obtain distribution direction difference information. For example, the difference between θ1 and θ2 can be used as distribution direction difference information. In this way, multiple distribution direction difference information can be obtained, which can comprehensively reflect the differences in the positions of the multiple first feature points and the multiple second feature points.

[0106] In this way, when verifying the validity of multiple target matching point pairs based on the distribution distance difference information, the validity verification can be determined to have passed if the distribution distance difference information is all less than the target difference, where the target difference can be set to a small value such as 0.1. Of course, the average value of each distribution distance difference information can also be obtained and compared with the target difference. If the average value is less than the target difference, the validity verification has passed.

[0107] When verifying the validity of multiple target matching point pairs based on the distribution direction difference information, the validity verification can be determined to be passed when the distribution direction difference information is all less than the target angle, wherein the target angle can be set to 2 O Of course, the average value of the difference information in each distribution direction can also be obtained, and the average value can be compared with the target angle. If the average value is smaller than the target angle, the validity verification is passed.

[0108] When verifying the validity of multiple target matching point pairs based on distribution direction difference information and distribution distance difference information, it can be determined that the validity verification has passed if the distribution distance difference information is all smaller than the target difference and the distribution direction difference information is all smaller than the target angle.

[0109] For example, Figure 4 As shown, when the distribution distance difference information is smaller than the target difference, and the distribution direction difference information is smaller than the target angle, the topological structure composed of multiple first feature points on the first image and the topological structure composed of multiple second feature points on the second image have a high similarity. In practice, the probability that the sampled target matching point pairs are correctly matched matching point pairs is high.

[0110] According to the implementation method of the embodiment of the present application, since the distribution distance difference information is determined based on the ratio between the spacing between the first feature points and the spacing between the corresponding second feature points, the comparison scale of the interval distance can be placed on the scale of the distance ratio. Compared with the difference in distance, it can better characterize the similarity between the position distributions and avoid misjudgment caused by the size change of the object in the two images. For example, the size of object A in the first image is larger than the size of object A in the second image, that is, object A is reduced in size in the second image. Using the distance ratio can avoid misjudgment caused by the different sizes of the two images, thereby improving the effectiveness of determining the difference between the first position distribution information and the second position distribution information.

[0111] Of course, in some embodiments, the first feature points and the second feature points in the target matching point pairs need not be collinear. Therefore, another validity verification may be performed on the target matching point pairs to ensure that the first feature points and the second feature points in the target matching point pairs are not collinear.

[0112] In a specific implementation, a check can be made to see whether a plurality of first feature points on the first image are collinear, and a check can be made to see whether a plurality of second feature points on the second image are collinear. If the check result indicates that the plurality of first feature points are not collinear, and the plurality of second feature points are not collinear, a step of screening the plurality of initial matching points based on a fundamental matrix constructed using the plurality of target matching point pairs is performed.

[0113] When determining whether multiple first feature points on a first image are collinear, two first feature points can be randomly selected and a line can be fitted. The equation of the line is y = k*x + b. The distances from the remaining first feature points to the line are then determined. If any first feature points are closer than the target distance, such as less than 2 pixels, then the three first feature points are considered collinear and the verification is rejected. Similarly, the collinearity of multiple second feature points on a second image can be determined.

[0114] It should be noted that when multiple first feature points are not collinear, it means that the multiple first feature points are not located on the same straight line, so they are dispersed on the first image. Therefore, the feature points based on the dispersed distribution can fully reflect the relative position relationship between the feature points, enhance the complexity of the position distribution, and thus highlight the difference between the first position distribution information and the second position distribution information, helping to more effectively verify multiple target matching point pairs.

[0115] Accordingly, when the verification is passed and the plurality of first feature points and the plurality of second feature points are not collinear, the plurality of initial matching point pairs may be subsequently screened based on the basic matrix.

[0116] In practice, the non-collinearity verification may be performed before the step of determining whether a plurality of target matching point pairs meet a target condition based on the first position distribution information and the second position distribution information.

[0117] 2.1, the stage of eliminating erroneous point pairs using the basic matrix.

[0118] When screening feature point pairs based on the basic matrix, in an optional example, multiple correct matching point pairs that meet the epipolar constraints can be directly screened out from multiple initial matching point pairs, and these multiple correct matching point pairs can be used as a screening result; and continue to repeat the above steps S201-S203 multiple times, and based on the number of correct matching point pairs screened out by each execution of the above steps S201-S203, the correctness of the basic matrix is verified, and finally the basic matrix with the largest number of correct matching point pairs is screened out, and the correct matching point pairs screened out based on the basic matrix are used as the final feature point matching results.

[0119] In another optional example, multiple candidate matching point pairs that meet the epipolar constraints can be first screened out from multiple initial matching point pairs, and then the multiple candidate matching point pairs can be screened. In specific implementation, constraints on the multiple candidate matching point pairs in the non-episode direction can be added to screen out correctly matched matching point pairs, wherein the constraints in the non-episode direction can be achieved by comparing the difference between the position distribution information of the feature points in the candidate matching point pairs located on the first image and the position distribution information of the feature points located on the second image.

[0120] In a specific implementation, epipolar lines may be constructed based on the basic matrix, and based on the epipolar lines, multiple candidate matching point pairs that meet the epipolar constraints may be screened out from multiple initial matching point pairs.

[0121] Next, based on the difference between the position distribution information of the feature points located on the first image and the position distribution information of the feature points located on the second image in the multiple candidate matching point pairs, matching point pairs that are correctly matched are obtained from the multiple candidate matching point pairs.

[0122] Among them, the polar lines are used to map the feature points on the first image to the second image, or to map the feature points on the second image to the first image; the process of determining the difference between the position distribution information of the feature points on the first image in the candidate matching point pair and the position distribution information of the feature points located on the second image can refer to the above-mentioned process of determining the difference between the first position distribution information and the second position distribution information, and will not be repeated here.

[0123] Specifically, from multiple candidate matching point pairs, matching point pairs whose distribution distance difference information is less than the target difference and whose distribution direction difference information is less than the target angle can be selected as the correct matching point pairs. The determination of the distribution distance difference information and the distribution direction difference information can be referred to the relevant process described above and will not be repeated here.

[0124] Of course, in a specific implementable method, since the sampled multiple target matching point pairs are verified, the probability that multiple matching point pairs among the multiple target matching point pairs are correct matching point pairs is high. These correct matching point pairs can reflect the imaging differences of the space in the first image and the second image. Therefore, based on the difference between the first position distribution information and the second position distribution information, the differences between the multiple candidate matching point pairs in the first image and the second image can be verified, thereby screening out the correctly matched matching point pairs.

[0125] In a specific implementation, the imaging difference between the first image and the second image can be represented by a distance ratio, which represents the ratio of the distance between the two first feature points in any two target matching point pairs to the distance between the two second feature points. Similarly, the first feature point is located on the first image and the second feature point is located on the second image.

[0126] Among them, the first distance ratio corresponding to multiple target matching point pairs can be determined based on the first position distribution information and the second position distribution information; then the first distance ratio can represent the first distance ratio of the distance between the two first feature points and the distance between the two second feature points in the two target matching point pairs.

[0127] Accordingly, when obtaining the correct matching point pairs that belong to the correct match from the candidate matching point pairs, the second distance ratio corresponding to the two candidate matching points can be determined first. The second distance ratio can be determined based on the distance between the feature points located in the first image and the distance between the feature points located in the second image in the two candidate matching point pairs.

[0128] Then, based on the first distance ratio and the second distance ratio, matching feature point pairs of the first image and the second image are screened out from the plurality of candidate matching point pairs.

[0129] In a specific implementation, the first distance ratio corresponding to multiple target matching point pairs can be the average of the ratios of the distances between any two first feature points to the distances between any two second feature points. As shown in the above example, it can be the average of the distance ratios. For example, if multiple distance ratios d1-d7 are obtained, the average of the multiple distance ratios is the first distance ratio.

[0130] In this way, a second distance ratio corresponding to each pair of candidate matching points can be determined. This second distance ratio is the ratio of the distance between the feature points in the first image to the distance between the feature points in the second image. That is, because each candidate matching point pair includes one feature point in the first image and one feature point in the second image, the two candidate matching point pairs have two feature points in the first image and two feature points in the second image. Thus, the distance between the two feature points in the first image and the distance between the two feature points in the second image can be obtained, and the second distance ratio can be obtained by comparing these two distances.

[0131] When the difference between the second distance ratio and the first distance ratio is greater than the target difference, it can be considered that the imaging difference between the two candidate matching point pairs in the first image and the second image does not conform to the imaging difference in reality, and the matching point pair is an incorrect matching point pair. When the difference between the second distance ratio and the first distance ratio is less than or equal to the target difference, it can be considered that the imaging difference between the two candidate matching point pairs in the first image and the second image conforms to the imaging difference in reality, and the matching point pair is a correct matching point pair.

[0132] Thus, the second distance ratio corresponding to each two candidate matching point pairs can be determined, so that the correct matching point pairs that are correctly matched can be screened out based on the difference between the first distance ratio and the second distance ratio corresponding to each two candidate matching point pairs.

[0133] When two candidate matching point pairs are determined to be correct, they can be screened out. If they are not correct, each of the two candidate matching point pairs can be combined with other candidate matching point pairs to determine whether the second distance ratio between each candidate matching point pair and the other candidate matching point pairs is less than or equal to the first distance ratio. This allows incorrect matching point pairs to be eliminated while retaining correct matching point pairs.

[0134] When this implementation is used, incorrect matching point pairs are eliminated again by utilizing the first distance ratio corresponding to multiple target matching point pairs and the second distance ratio corresponding to each pair of candidate matching point pairs. Both the first distance ratio and the second distance ratio represent the differences in the positional distribution of multiple feature points on the two images. Thus, while constraining the matching point pairs in the epipolar direction, the matching point pairs are further constrained in the non-episodic direction, thereby significantly reducing the output of incorrect matching point pairs and further improving the accuracy of feature point matching. Furthermore, since only the second distance ratio of each candidate matching point pair needs to be calculated for comparison with the first distance ratio, there is no need to calculate the angle between the lines connecting the feature points. This reduces the computational effort required to re-screen the candidate matching point pairs, thereby increasing the speed of re-screening.

[0135] 3.1, the screening process of the initial matching point pairs.

[0136] As mentioned above, the RANSAC method based on the fundamental matrix is a continuous iterative process. Each iteration samples multiple target matching point pairs and verifies the validity of the sampled target matching point pairs. After verification, the fundamental matrix is calculated based on the sampled target matching point pairs, and the initial matching point pairs are then filtered using the fundamental matrix. In this way, each iteration will obtain a feature point matching result.

[0137] Accordingly, in one embodiment, the entire process is described as follows:

[0138] First, when multiple sampling operations are performed on multiple initial matching point pairs, multiple target matching point pairs corresponding to each sampling operation are obtained; for the matching feature point pairs of the first image and the second image obtained based on the multiple target matching points obtained in each sampling operation, the number of the matching feature point pairs is determined; and based on the number of matching feature point pairs, the results of matching the feature points on the first image and the feature points on the second image are determined from the matching feature point pairs corresponding to the multiple sampling operations.

[0139] In this embodiment, each operation is performed to obtain multiple target matching point pairs, and then based on the basic matrix constructed based on the multiple target matching points, a screening result for screening multiple initial matching point pairs can be obtained. One screening result corresponds to multiple matching feature point pairs that have been screened out, and it is also known how many matching feature point pairs have been screened out by the screening result.

[0140] In practice, the screening result with the largest number of matching feature point pairs may be used as the result of matching the feature points on the first image with the feature points on the second image.

[0141] Assuming that the multiple matching feature point pairs obtained as a result of a screening are represented by a candidate point pair set, in this embodiment, based on the number of matching feature point pairs included in the candidate point pair set, a target point pair set with the largest number of matching feature point pairs can be screened out from the multiple candidate point pair sets, and the target point pair set can be used as the final matching result.

[0142] The number of times of multiple sampling operations can be determined by using relevant technology. For example, if the number of matching point pairs to be sampled is n, w n is the probability that all n points are correct matching points (also known as inliers); 1-w n is the probability that at least one feature point among n points is a wrong matching point, (1-w n ) k=w represents the probability that the algorithm will never select all n points as inliers. Therefore, the probability that at least one of the selected points is all inliers is 1-(1-w n ) k , the probability of the internal point can be specified as q, and the number of iterations m can be obtained according to the following formula (1).

[0143]

[0144] Generally, an iteration end condition is set during the iteration process. The iteration end condition can be the proportion of incorrect matching points eliminated, etc. In this way, during the iteration process, if the iteration has not been completed m times, the iteration end condition can be met. If the iteration end condition is not met, the iteration will continue until the iteration has reached m times and then end.

[0145] Due to the adoption of the technical solution of the embodiment of the present application, the validity of the sampled multiple target matching point pairs is verified in advance, so that the sampled target matching point pairs are correct matching point pairs, thereby improving the calculation accuracy of the basic matrix. In this way, each matching process is effective and accurate, and thus, the iteration end condition can be reached more quickly. In this way, the iteration can be ended in advance before the number of iterations m is reached, thereby avoiding a long time when the proportion of incorrect matching point pairs is high.

[0146] Next, combine Figure 4 The schematic diagram shown in Figure 4-2 illustrates the image feature point matching method of the present application through a specific example:

[0147] 1. Select the first image and the second image with a common field of view, extract the feature points of each image, and complete the initial matching of the feature points through brute force matching. Assume that 100 matching point pairs are obtained.

[0148] 2. Data sampling: randomly sample 8 target matching point pairs from 100 matching point pairs. Correspondingly, 8 feature points will be sampled on the first image and 8 corresponding feature points will be sampled on the second image.

[0149] 3. Data verification: In order to ensure the validity of the basic matrix calculation, it is necessary to verify the selected 8 target matching point pairs. The verification process is as follows:

[0150] 3-1. Verification of collinearity.

[0151] After randomly selecting 2 matching point pairs from the 8 target matching points, such as matching point pair A (including feature point A1 on the first image and feature point A2 on the second image) and matching point pair B (including feature point B1 on the first image and feature point B2 on the second image), a straight line is fitted on the first image based on feature point A1 and feature point B1, and it is verified whether the other 6 feature points sampled on the first image are also on the straight line; similarly, a straight line is fitted on the second image based on feature point A2 and feature point B2, and it is verified whether the other 6 feature points sampled on the second image are also on the straight line.

[0152] When it is verified that any three feature points are not collinear, the verification is passed and the process goes to step 3-2. When it is verified that three feature points are collinear, the process returns to step 2 and resamples 8 target matching point pairs.

[0153] 3-2. Topology constraints.

[0154] like Figure 4 As shown in Figure 4-2, if the eight target matching point pairs randomly selected from the first and second images are accurately matched, then their topological structures are consistent. For example, in the triangle formed by point pairs p11-p21, p12-p22, and p13-p23, the ratios of the lengths of the three sides are similar.

[0155] The verification process is as follows: calculate the distance of each line segment on the first image and the distance of each line segment on the second image, then determine the ratio of the distances between the corresponding line segments on the two images, i.e., the distance ratio; and obtain d1-d7;

[0156] Then, the angle between two adjacent line segments on each image is calculated. For example, the angle θ is calculated according to formula (2) on the first image: 1i , calculate the angle θ on the second image according to formula (3) 2i ;

[0157]

[0158]

[0159] Calculate the difference between each two distance ratios, which is the distribution distance difference information. Calculate the difference between the two corresponding angles on the first image and the second image, which is the distribution direction difference information. If the distribution distance difference information is less than 0.1 and the distribution direction difference information is less than 2 O , the verification is successful and we go to step 4. If not, the verification fails and we go back to step 2 to resample 8 target matching point pairs.

[0160] When the verification is passed, a first distance ratio is calculated, and the first distance ratio is, for example, an average of d1, d2, d3, d4, d5, d6, and d7.

[0161] 4. F matrix calculation: The 8-point method is used to calculate the F matrix.

[0162] 5. Model (F matrix) verification.

[0163] 5-1. Based on the F matrix and the epipolar equation, the epipolar line can be constructed and used to map the feature points on the first image to the second image. When eliminating incorrect matching point pairs, the epipolar lines of all 100 feature points in the first image on the second image can be obtained based on the basic matrix, and the feature points p on the second image that match the first image can be determined. 2n The distance d to the epipolar line n , where the calculation of the distance can refer to the relevant technology and will not be repeated here; when d n When the distance is less than or equal to the specified distance, it is considered that p 2n is a correctly matched point pair. In this way, multiple candidate matching point pairs that meet the epipolar constraints are obtained. Suppose there are 50 candidate matching point pairs.

[0164] 5-2. Based on the first distance ratio calculated in step 3-2, calculate the second distance ratio for every two candidate matching point pairs that satisfy the 5-1 epipolar constraint, and screen out matching point pairs whose absolute value of the difference between the first distance ratio and the second distance ratio is less than a specified value (which can be 0.1). The screened matching point pairs are matching feature point pairs. Assume that 20 matching feature point pairs are screened out.

[0165] 6. Repeat steps 2 to 5 above to select the F matrix with the largest number of matching feature point pairs, and output the F matrix and the matching feature point pairs obtained under the F matrix. Assume that there are 5 F matrices, and the number of matching feature point pairs obtained by screening 5 times is 20, 25, 26, 18, and 22 respectively. Then output the F matrix corresponding to 25 and output these 25 matching feature point pairs as the final feature point matching result.

[0166] The technical solution of the embodiment of the present application has the following advantages:

[0167] (1) Reduced calculation time

[0168] Since the validity of multiple target matching point pairs is verified, it is ensured that the sampled multiple target matching point pairs are as correct matching point pairs as possible, thereby improving the accuracy of the basic matrix calculation. The matching point pairs that meet the epipolar constraints are the correct matching point pairs in the actual situation, so that the correct feature point matching results can be obtained as soon as possible, thereby avoiding excessive iteration and calculation, and solving the problem of high time consumption.

[0169] (2) The accuracy of feature point matching results is improved in two aspects.

[0170] On the one hand, the accuracy of the sampled multiple target matching point pairs is improved, thereby improving the accuracy of the basic matrix calculation and providing accurate epipolar constraints for the subsequent screening process, so that erroneous matching point pairs can be accurately deleted.

[0171] On the other hand, in the process of eliminating incorrect matching point pairs, for candidate matching point pairs that meet the epipolar constraint, the first distance ratio of the verified target matching point pair is used as a constraint, and the candidate matching point pairs are constrained in the non-episode direction for re-screening, which greatly reduces the output of incorrect matching point pairs.

[0172] Based on the same inventive concept, the present application also provides an image feature point matching device, referring to Figure 6 As shown in FIG, a structural block diagram of an image feature point matching device is disclosed, such as Figure 6 As shown, the device may include the following modules:

[0173] A sampling module 601 is configured to sample a plurality of target matching point pairs from a plurality of initial matching point pairs; the plurality of initial matching point pairs are obtained by performing initial matching on feature points on a first image and feature points on a second image;

[0174] A verification module 602 is configured to determine whether the plurality of target matching point pairs are matching point pairs that meet a target condition based on first position distribution information of feature points on the first image and second position distribution information of feature points on the second image among the plurality of target matching point pairs;

[0175] The screening module 603 is used to screen multiple initial matching point pairs based on the basic matrix constructed by multiple target matching point pairs to obtain matching feature point pairs of the first image and the second image; wherein, a pair of matching feature point pairs corresponds to the same target point in space.

[0176] Optionally, the first position distribution information and the second position distribution information are obtained according to the following steps:

[0177] Acquire a first topological structure formed by a plurality of the first feature points on the first image, and a second topological structure formed by a plurality of the second feature points on the second image;

[0178] The first position distribution information is determined based on the first topological structure, and the second position distribution information is determined based on the second topological structure.

[0179] Optionally, the verification module 602 includes:

[0180] A difference information acquisition unit, configured to determine distribution difference information corresponding to a plurality of target matching point pairs;

[0181] a verification unit, configured to determine whether the plurality of target matching point pairs are matching point pairs that meet a target condition based on the distribution difference information; wherein the distribution difference information includes distribution distance difference information and / or distribution direction difference information;

[0182] Among them, the distribution distance difference information is represented by the difference between the first distance between the two first feature points in any two of the target matching point pairs and the second distance between the two second feature points; the distribution direction difference information is represented by the difference between the first angle between the three first feature points in any three of the target matching point pairs and the second angle formed by the three second feature points.

[0183] Optionally, the difference information acquiring unit is configured to determine, for each two adjacent first feature points, a distance ratio between a first distance between two of the first feature points and a second distance between two corresponding second feature points;

[0184] The distribution distance difference information is determined based on each of the distance ratios.

[0185] Optionally, the difference information acquisition unit is used to determine the angle difference between the angle between the three first feature points and the angle between the corresponding three second feature points for every three adjacent first feature points, and determine the distribution direction difference information based on the obtained angle differences.

[0186] Obtaining a distance ratio between two adjacent first feature points and a distance between two corresponding second feature points, and determining the distribution distance difference information based on each of the obtained distance ratios;

[0187] The first verification unit is specifically configured to determine the distribution direction difference information based on an angle difference between three adjacent first feature points and an angle between three corresponding second feature points, and based on each of the obtained angle differences.

[0188] Optionally, the device further comprises:

[0189] a collinearity verification module, configured to verify whether a plurality of the first feature points on the first image are collinear, and to verify whether a plurality of the second feature points on the second image are collinear;

[0190] The screening module 603 is used to perform the step of screening the multiple initial matching points based on the basic matrix constructed based on the multiple target matching point pairs if the verification result indicates that the multiple first feature points and the multiple second feature points are not collinear.

[0191] Optionally, the screening module 603 includes:

[0192] an extreme construction unit, configured to construct epipolar lines based on the fundamental matrix, wherein the epipolar lines are used to map feature points on the first image to the second image, or to map feature points on the second image to the first image;

[0193] a first constraint unit, configured to select, based on the epipolar lines, a plurality of candidate matching point pairs that meet epipolar constraints from the plurality of initial matching point pairs;

[0194] The second constraint unit is used to screen the multiple candidate matching point pairs based on the difference between the position distribution information of the feature points located on the first image and the position distribution information of the feature points located on the second image in the multiple candidate matching point pairs, so as to obtain matching feature point pairs of the first image and the second image.

[0195] Optionally, the device further comprises:

[0196] a first distance ratio determining module, configured to determine, based on the first position distribution information and the second position distribution information, a first distance ratio corresponding to a plurality of target matching point pairs, wherein the first distance ratio represents a distance ratio between two first feature points and a distance between two second feature points in any two target matching point pairs;

[0197] The second constraint unit includes:

[0198] a first distance ratio determining subunit, configured to determine a second distance ratio corresponding to each pair of candidate matching points based on a distance between feature points located in the first image and a distance between feature points located in the second image in each pair of candidate matching points;

[0199] The screening subunit is configured to obtain a correct matching point pair that is a correct match from the plurality of candidate matching point pairs based on the first distance ratio and the second distance ratio.

[0200] Optionally, the device further comprises:

[0201] a resampling module, configured to perform multiple sampling operations on the multiple initial matching point pairs and verify the validity of the resampled multiple target matching point pairs;

[0202] An acquisition module is used to perform multiple sampling on multiple initial matching point pairs to obtain multiple target matching point pairs corresponding to each sampling operation;

[0203] a quantity determination module, configured to determine the number of matching feature point pairs of the first image and the second image obtained based on the plurality of target matching points obtained in each sampling operation;

[0204] The result acquisition module is used to determine the matching results of the feature points on the first image and the feature points on the second image from the matching feature point pairs corresponding to multiple sampling operations based on the number of matching feature point pairs.

[0205] It should be noted that the device embodiment is similar to the method embodiment, so the description is relatively simple, and the relevant parts can be referred to the method embodiment.

[0206] The present application also discloses an electronic device, Figure 7 As shown, a structural schematic diagram of an electronic device according to an embodiment of the present application is shown, including a memory 701, a processor 702, and a computer program stored in the memory 701 and executable on the processor 702, wherein the processor 702 implements the feature point matching method of the image as described when executing.

[0207] like Figure 7 As shown, in one embodiment, the electronic device can completely include an input device 703 and an output device 707, wherein, when executing the image feature point matching method of the embodiment of the present application, the input device 703 can obtain the first image and the second image, the image feature point matching method can be executed by the processor 702, and the output device 707 can output the corrected image.

[0208] Of course, in one embodiment, memory 701 may include volatile memory and non-volatile memory, wherein volatile memory can be understood as random access memory used to store and save data. Non-volatile memory refers to computer memory whose stored data does not disappear when the current is turned off. Of course, the computer program of the image feature point matching method of the present application can be stored in volatile memory or non-volatile memory, or in either volatile memory or non-volatile memory.

[0209] An embodiment of the present application further discloses a computer-readable storage medium, which stores a computer program that enables a processor to execute the image feature point matching method described in the present application.

[0210] An embodiment of the present application further discloses a computer program product, including a computer program / instruction, which implements the image feature point matching method when executed by a processor.

[0211] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0212] It will be understood by those skilled in the art that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0213] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0214] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0215] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0216] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0217] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0218] The above is a detailed introduction to the image feature point matching method, device, medium and program product provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for matching feature points of an image, characterized in that: The method comprises: Sampling a plurality of target matching point pairs from a plurality of initial matching point pairs; the plurality of initial matching point pairs are obtained by performing initial matching on feature points on the first image and feature points on the second image; Based on first position distribution information of first feature points on the first image among the plurality of target matching point pairs, and second position distribution information of second feature points on the second image among the plurality of target matching point pairs, determine whether the plurality of target matching point pairs are matching point pairs that meet a target condition; the target condition is: the number of erroneous matching point pairs among the sampled plurality of target matching point pairs is less than a set number threshold, or the difference between the first position distribution information and the second position distribution information is less than a target difference; the difference between the first position distribution information and the second position distribution information is determined based on the following method: calculating the similarity between the first position distribution information and the second position distribution information, or calculating the difference between the first position distribution information and the second position distribution information; If so, based on the basic matrix constructed by the plurality of target matching point pairs, the plurality of initial matching points are screened to obtain matching feature point pairs of the first image and the second image; wherein a pair of the matching feature point pairs corresponds to the same target point in space; The first location distribution information and the second location distribution information are obtained according to the following steps: Acquire a first topological structure formed by a plurality of the first feature points on the first image, and a second topological structure formed by a plurality of the second feature points on the second image; The first position distribution information is determined based on the first topological structure, and the second position distribution information is determined based on the second topological structure.

2. The method according to any one of claim 1, characterized in that The determining whether the plurality of target matching point pairs are matching point pairs that meet a target condition based on first position distribution information of first feature points on the first image among the plurality of target matching point pairs and second position distribution information of second feature points on the second image among the plurality of target matching point pairs comprises: Determining distribution difference information corresponding to a plurality of target matching point pairs based on the first position distribution information and the second position distribution information; Based on the distribution difference information, determining whether the plurality of target matching point pairs are matching point pairs that meet the target condition; wherein the distribution difference information includes distribution distance difference information and / or distribution direction difference information; Among them, the distribution distance difference information is represented by the difference between the first distance between two of the first feature points in any two of the target matching point pairs and the second distance between two of the second feature points, and the distribution direction difference information is represented by the difference between the first angle between three of the first feature points in any three of the target matching point pairs and the second angle formed by three of the second feature points.

3. The method according to claim 2, characterized in that The distribution difference information includes distribution distance difference information; and determining the distribution difference information corresponding to the plurality of target matching point pairs based on the first position distribution information and the second position distribution information includes: For every two adjacent first feature points, determining a distance ratio between a first distance between the two first feature points and a second distance between corresponding two second feature points; The distribution distance difference information is determined based on each of the distance ratios.

4. The method according to claim 2, characterized in that The distribution difference information includes distribution direction difference information; and determining the distribution difference information corresponding to the plurality of target matching point pairs based on the first position distribution information and the second position distribution information includes: For every three adjacent first feature points, determining an angle difference between a first angle between the three first feature points and a second angle between the corresponding three second feature points; Based on each of the angle differences, the distribution direction difference information is determined.

5. The method according to any one of claims 1 to 4, characterized in that Before screening a plurality of initial matching points based on a basic matrix constructed by the plurality of target matching point pairs to obtain matching feature point pairs of the first image and the second image, the method further includes: Verifying whether a plurality of first feature points on the first image are collinear, and verifying whether a plurality of second feature points on the second image are collinear; If the verification result indicates that the first feature points are not collinear and the second feature points are not collinear, the step of screening the initial matching points based on the basic matrix constructed based on the target matching point pairs is performed.

6. The method according to any one of claims 1 to 3, characterized in that The step of screening a plurality of initial matching points based on a basic matrix constructed based on the plurality of target matching point pairs to obtain matching feature point pairs of the first image and the second image includes: constructing epipolar lines based on the fundamental matrix, where the epipolar lines are used to map feature points on the first image to the second image, or to map feature points on the second image to the first image; Based on the epipolar lines, a plurality of candidate matching point pairs that meet epipolar constraints are selected from the plurality of initial matching point pairs; Based on the difference between the position distribution information of the feature points located on the first image in the multiple candidate matching point pairs and the position distribution information of the feature points located on the second image, the multiple candidate matching point pairs are screened to obtain matching feature point pairs of the first image and the second image.

7. The method according to claim 6, characterized in that When it is determined that the plurality of target matching point pairs meet the target condition, the method further includes: Determining, based on the first position distribution information and the second position distribution information, a first distance ratio corresponding to a plurality of target matching point pairs, where the first distance ratio represents a distance ratio between two first feature points and a distance between two second feature points in any two target matching point pairs; The method of screening the plurality of candidate matching point pairs based on a difference between position distribution information of feature points located on the first image and position distribution information of feature points located on the second image to obtain matching feature point pairs of the first image and the second image includes: Determining a second distance ratio corresponding to each pair of candidate matching points based on a distance between two feature points located on the first image and a distance between two feature points located on the second image in each pair of candidate matching points; Based on the first distance ratio and the second distance ratio, a plurality of candidate matching point pairs are screened to obtain matching feature point pairs of the first image and the second image.

8. The method according to any one of claim 1, characterized in that The step of sampling a plurality of target matching point pairs from a plurality of initial matching point pairs includes: Performing multiple sampling operations on the multiple initial matching point pairs to obtain multiple target matching point pairs corresponding to each sampling operation; The method further comprises: determining the number of matching feature point pairs of the first image and the second image obtained based on the plurality of target matching points obtained in each sampling operation; Based on the number of matching feature point pairs, the matching results of the feature points on the first image and the feature points on the second image are determined from the matching feature point pairs respectively corresponding to the multiple sampling operations.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the method, the method for matching feature points of an image according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that The computer program stored therein enables the processor to execute the image feature point matching method according to any one of claims 1 to 8.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for matching feature points of an image according to any one of claims 1 to 8 is implemented.

Images