Feature point matching method and device, storage medium and electronic device

Abstract

The present disclosure relates to the technical field of image processing, in particular to a feature point matching method and device, a computer readable storage medium and an electronic device. The method comprises: obtaining a forward optical flow and a reverse optical flow between a to-be-registered image and a reference image; determining an optical flow error of a candidate pixel point in the to-be-registered image or the reference image according to the forward optical flow and the reverse optical flow; selecting a target pixel point from the candidate pixel point according to the optical flow error of the candidate pixel point; performing repeated texture detection on the target pixel point to obtain a detection result, and determining a matching feature point pair between the to-be-registered image and the reference image according to the detection result. The precision of the matching feature point pair obtained by the technical scheme of the embodiment of the present disclosure is high.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to a feature point matching method and apparatus, a computer-readable storage medium, and an electronic device. Background Technology

[0002] In computer vision applications, it is often necessary to match feature points from different images.

[0003] However, the feature point matching methods in related technologies have poor matching accuracy, resulting in low matching degree of the obtained feature point pairs.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a feature point matching method, a feature point matching method apparatus, a computer-readable medium, and an electronic device, thereby improving the matching accuracy of the feature point matching method to at least a certain extent, and thus enhancing the matching degree of the obtained matching feature point pairs.

[0006] According to a first aspect of this disclosure, a feature point matching method is provided, comprising: acquiring forward optical flow and reverse optical flow between an image to be registered and a reference image; determining optical flow error of candidate pixels in the image to be registered or the reference image based on the forward optical flow and reverse optical flow; selecting a target pixel from the candidate pixels based on the optical flow error of the candidate pixels; performing duplicate texture detection on the target pixel to obtain a detection result, and determining a matching feature point pair between the image to be registered and the reference image based on the detection result.

[0007] According to a second aspect of this disclosure, a feature point matching apparatus is provided, comprising: an optical flow acquisition module for acquiring forward optical flow and reverse optical flow between an image to be registered and a reference image; an error determination module for determining the optical flow error of candidate pixels in the image to be registered or the reference image based on the forward and reverse optical flow; a pixel selection module for selecting a target pixel from the candidate pixels based on the optical flow error of the candidate pixels; and an image matching module for performing duplicate texture detection on the target pixel to obtain a detection result, and determining a matching feature point pair between the image to be registered and the reference image based on the detection result.

[0008] According to a third aspect of this disclosure, a computer-readable medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method described above.

[0009] According to a fourth aspect of this disclosure, an electronic device is provided, characterized in that it includes: one or more processors; and a memory for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method described above.

[0010] One embodiment of this disclosure provides a feature point matching method that acquires the forward and reverse optical flow between an image to be registered and a reference image; determines the optical flow error of candidate pixels in the image to be registered or the reference image based on the forward and reverse optical flow; selects target pixels from the candidate pixels based on the optical flow error; performs duplicate texture detection on the target pixels to obtain detection results, and determines matching feature point pairs between the image to be registered and the reference image based on the detection results. Compared with the prior art, on the one hand, by determining the optical flow error through the forward and reverse optical flow between the image to be registered and the reference image, and acquiring the target pixels based on the optical flow error, inaccurate pixels are eliminated, improving the accuracy of feature point matching; on the other hand, by determining the matching feature point pairs between the image to be registered and the reference image based on the duplicate texture detection results, the influence of duplicate texture on feature point matching is reduced, further improving the accuracy of feature point matching and the matching degree of the obtained matching feature point pairs.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0013] Figure 1 A schematic diagram of an exemplary system architecture to which embodiments of the present disclosure may be applied is shown;

[0014] Figure 2 A flowchart illustrating a feature point matching method in an exemplary embodiment of this disclosure is shown schematically.

[0015] Figure 3 This schematic diagram illustrates a forward optical flow image in an exemplary embodiment of the present disclosure;

[0016] Figure 4 This schematic diagram illustrates a reverse optical flow image in an exemplary embodiment of the present disclosure;

[0017] Figure 5 This schematically illustrates a flowchart for determining optical flow error in an exemplary embodiment of the present disclosure;

[0018] Figure 6 This schematically illustrates another flowchart for determining optical flow error in an exemplary embodiment of the present disclosure;

[0019] Figure 7 This schematically illustrates a flowchart of determining a detection result in an exemplary embodiment of the present disclosure;

[0020] Figure 8 The illustration schematically shows a first image block and a second image block in an exemplary embodiment of the present disclosure;

[0021] Figure 9 A flowchart illustrating a repetitive texture detection method in an exemplary embodiment of this disclosure is shown schematically.

[0022] Figure 10 A flowchart illustrating another feature point matching method in an exemplary embodiment of this disclosure is shown schematically.

[0023] Figure 11 The diagram illustrates the calculation principle of epipolar error in an exemplary embodiment of the present disclosure.

[0024] Figure 12 A flowchart illustrating another feature matching method in an exemplary embodiment of this disclosure is shown schematically.

[0025] Figure 13 This schematic diagram illustrates the composition of a feature point matching device according to an exemplary embodiment of the present disclosure;

[0026] Figure 14 This schematic diagram illustrates the composition of another feature point matching device in an exemplary embodiment of the present disclosure;

[0027] Figure 15 A schematic diagram of an electronic device to which embodiments of the present disclosure may be applied is shown. Detailed Implementation

[0028] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0029] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0030] Commonly used feature point matching schemes in related technologies include descriptor-based feature matching, optical flow-based feature point matching, block matching-based feature point matching, and deep learning-based feature point matching. Feature point matching schemes are prone to errors, potentially leading to mismatches. Therefore, a feature point matching scheme should include steps such as feature point matching and feature point selection, both of which directly affect the quality of the feature point matching.

[0031] Commonly used mismatch point removal algorithms include: the RANSAC point removal algorithm based on homography matrix, the RANSAC point selection scheme based on fundamental matrix, and the feature point selection scheme based on GMS (Grid-based Motion Statistics). Among these, the homography matrix-based selection method is only applicable to scenes where the image content is planar; the fundamental matrix-based selection method is based on epipolar constraints, and when the image texture is repetitive along the epipolar direction, mismatch points in that direction are difficult to remove; the GMS-based selection scheme determines whether a point to be selected is a correctly matched feature point based on the probability of a correct match point in the region it is in, which can easily lead to the retention or deletion of entire areas of mismatch points. These mismatch point selection schemes cannot guarantee that mismatch points in regions with repetitive textures will be effectively removed.

[0032] Based on the above-mentioned shortcomings, this disclosure proposes a feature point matching method. Figure 1 A schematic diagram of a system architecture capable of implementing the feature point matching method of this disclosure is shown. This system architecture 100 may include a terminal 110 and a server 120. The terminal 110 may be a terminal device such as a smartphone, tablet, desktop computer, or laptop computer. The server 120 generally refers to a backend system providing image processing, feature point matching, and other related services in this exemplary embodiment, and may be a single server or a cluster of multiple servers. The terminal 110 and the server 120 can be connected via a wired or wireless communication link for data exchange.

[0033] In one implementation, the feature point matching method described above can be performed by the terminal 110. For example, after a user takes a picture using the terminal 110 or selects an image to be registered and a reference image from the photo album of the terminal 110, the terminal 110 performs feature point matching on the image to be registered and the reference image and outputs the matched feature point pair.

[0034] In one implementation, the server 120 can execute the above-described feature point matching method. For example, after a user takes a picture using the terminal 110 or after the user has a picture to be registered and a reference picture in the album of the terminal 110, the terminal 110 uploads the picture to the server 120, and the server 120 performs feature point matching between the picture to be registered and the reference picture, and returns the matched feature point pair to the terminal 110.

[0035] As can be seen from the above, the execution subject of the feature point matching method in this exemplary embodiment can be the aforementioned terminal 110 or server 120, and this disclosure does not limit it in this regard.

[0036] The following is combined Figure 2 The image quality evaluation method in this exemplary embodiment will be described. Figure 2 An exemplary flow of this image quality assessment method is shown, which may include:

[0037] Step S210: Obtain the forward optical flow and reverse optical flow between the image to be registered and the reference image;

[0038] Step S220: Determine the optical flow error of candidate pixels in the image to be registered or the reference image based on the forward optical flow and the reverse optical flow;

[0039] Step S230: Select a target pixel from the candidate pixels based on the optical flow error of the candidate pixels;

[0040] Step S240: Perform repeat texture detection on the target pixels to obtain detection results, and determine the matching feature point pair between the image to be registered and the reference image based on the detection results.

[0041] Based on the above method, on the one hand, the optical flow error is determined by the forward and reverse optical flow between the image to be registered and the reference image, and the target pixel is obtained based on the optical flow error, thus eliminating inaccurate pixels and improving the accuracy of feature point matching. On the other hand, the matching feature point pairs between the image to be registered and the reference image are determined based on the results of repeated texture detection, thereby reducing the influence of repeated textures on feature point matching and further improving the accuracy of feature point matching and the matching degree of the obtained matching feature point pairs.

[0042] The following is about Figure 2 Each step in the process will be explained in detail.

[0043] refer to Figure 2 In step S210, the forward optical flow and reverse optical flow between the image to be registered and the reference image are obtained.

[0044] In one disclosed example implementation, the forward optical flow is the optical flow from the reference image to the image to be registered, and the reverse optical flow is the optical flow from the image to be registered to the reference image.

[0045] In this example embodiment, before obtaining the forward and reverse optical flow between the image to be registered and the reference image, the image to be registered and the reference image can be preprocessed first. The preprocessing can include, but is not limited to, converting the image to be registered and the reference image into grayscale images, adjusting their size, stretching their brightness, etc., and can also be customized according to user needs. In this example embodiment, no specific limitation is made.

[0046] In this example implementation, the DIS (Dense Inverse Search) algorithm can be used to obtain the forward and reverse optical flow between the image to be registered and the reference image.

[0047] DIS (Dense Inverse Search) is an optical flow calculation method based on image pyramids. Its steps include fast multi-scale optical flow inverse search, sparse optical flow densification, and rapid variational refinement of the dense optical flow. Compared to pixel-level optical flow estimation, DIS is a pixel-block-level optical flow inverse search method. A single gradient calculation can be used for multiple inverse searches, reducing computational load. After completing the block-level optical flow calculation, the block optical flow is weighted according to certain rules to calculate the optical flow of each pixel, achieving sparse optical flow densification. Finally, variational optimization is performed on the dense optical flow map based on gradient information and smoothness, making the dense optical flow image more reliable. The DIS optical flow algorithm calculates dense optical flow information between images and can directly select reliable optical flow information for feature matching according to specific rules, obtaining high-quality feature point matching pairs.

[0048] In this example implementation, the process may specifically include constructing an image pyramid between the image to be registered and the reference image, initializing some computational overhead, calculating the block integral map of the image based on the gradient of the current layer image, solving for the sparse image optical flow field through inverse search based on image blocks, denseening the sparse optical flow field to obtain the dense optical flow of the image, and performing variational refinement on the dense optical flow image, transforming the densed optical flow map to the size of the next layer image as the initial optical flow for optical flow search in the next layer image, and transforming the bottom layer optical flow image to the same size as the original image and multiplying it by the corresponding magnification ratio to obtain the optical flow corresponding to the image to be registered and the reference image.

[0049] The aforementioned forward and reverse optical flows can be represented in the form of images, as shown below. Figure 3 and Figure 4 As shown, Figure 3 An image of forward optical flow is shown. Figure 4 The image shows a reverse optical flow diagram, where the optical flow of each pixel in the two-dimensional image is a two-dimensional vector. As shown in the optical flow diagram, region A represents negative optical flow in the x-direction and positive optical flow in the y-direction; region B represents positive optical flow in both the x-direction and y-direction; region C represents positive optical flow in the x-direction and negative optical flow in the y-direction; and region D represents negative optical flow in both the x-direction and y-direction.

[0050] It should be noted that the method of obtaining optical flow using DIS (Dense Inverse Search) can be found in the relevant technical literature, and will not be repeated here.

[0051] In step S220, the optical flow error of candidate pixels in the image to be registered or the reference image is determined based on the forward optical flow and the reverse optical flow.

[0052] In one exemplary embodiment of this disclosure, the optical flow error corresponding to candidate pixels in a reference image can be calculated based on forward and reverse optical flow. In this case, the candidate pixels can include all pixels in the reference image, pixels in each region, or can be customized according to user needs; no specific limitation is made in this exemplary embodiment. The optical flow error can be represented in the form of an optical flow error image, as shown in the reference image. Figure 5 As shown, specifically, determining the optical flow error corresponding to the candidate pixel in the reference image may include steps S510 to S530.

[0053] In step S510, the first matching point of the candidate pixel in the reference image in the image to be registered is determined based on the forward optical flow.

[0054] In this example embodiment, the first matching point in the image to be registered, corresponding one-to-one with all candidate pixels in the reference image, can be directly determined based on the aforementioned forward optical flow. For example, suppose the candidate pixel in the reference image is A. n Then its corresponding first matching point in the image to be registered is B. n A1 n and B n The values ​​of n in the equation correspond one-to-one, that is, A11 corresponds to B1, and A13 corresponds to B3.

[0055] In step S520, the second matching point of the first matching point in the reference image is determined based on the reverse optical flow.

[0056] After obtaining the first matching point, the second matching point corresponding to the first matching point in the reference image can be calculated using the reverse optical flow. For example, the first matching point calculated using the reverse optical flow is B. n The corresponding second matching point A2 in the reference image n A2 n and B n The values ​​of n in the equation correspond one-to-one, that is, A21 corresponds to B1, and A23 corresponds to B3.

[0057] In step S530, the optical flow error is determined based on the candidate pixels in the reference image and the second matching point corresponding to the candidate pixels.

[0058] After determining the second matching point, the optical flow error can be determined by predicting the second matching point based on the candidate pixels in the reference image. Specifically, the second matching point A2 can be calculated. n With candidate pixel A1 n The distance between the two points can be used as the optical flow error. The distance between each candidate pixel and the second matching point can be calculated to obtain multiple optical flow errors. The image composed of multiple optical flow errors can be used as the optical flow error image. Alternatively, the distance between some candidate pixels and the second matching point can be calculated as the optical flow error image. In this example embodiment, no specific limitation is made.

[0059] In another exemplary embodiment of this disclosure, the optical flow error corresponding to candidate pixels in the image to be registered can be calculated based on the forward and reverse optical flow. In this case, the candidate pixels can include all pixels in the image to be registered, or pixels in each region, or can be customized according to user needs; no specific limitation is made in this exemplary embodiment. The optical flow error can be represented in the form of an optical flow error image, referring to... Figure 6 As shown, specifically, determining the optical flow error corresponding to the candidate pixel in the reference image may include steps S610 to S630.

[0060] In step S610, the third matching point of the candidate pixel in the image to be registered in the reference image is determined based on the reverse optical flow.

[0061] In this example embodiment, the third matching point in the reference image corresponding to all candidate pixels in the image to be registered can be directly determined based on the reverse optical flow described above. For example, suppose the candidate pixel in the image to be registered is B1. n Then its corresponding third matching point in the reference image is A. n , where A n and B1 nThe values ​​of n in the equation correspond one-to-one, that is, A1 corresponds to B11, and A3 corresponds to B13.

[0062] In step S620, the third matching point is determined as the fourth matching point in the image to be registered based on the forward optical flow.

[0063] After obtaining the third matching point, the fourth matching point corresponding to the first matching point in the image to be registered can be calculated using the reverse optical flow. For example, the third matching point calculated using the reverse optical flow is A. n The fourth matching point B2 in the image to be registered n Among them, B2 n and A n The values ​​of n in the equation correspond one-to-one, that is, B21 corresponds to A1, and B23 corresponds to A3.

[0064] In step S630, the optical flow error is determined based on the candidate pixels in the image to be registered and the fourth matching point corresponding to the candidate pixels.

[0065] After determining the fourth matching point, the optical flow error can be determined by predicting the fourth matching point based on the candidate pixels in the image to be registered. Specifically, the fourth matching point B2 can be calculated. n With candidate pixel B1 n The distance between them is used as the optical flow error mentioned above. The distance between each candidate pixel and the fourth matching point can be calculated to obtain multiple optical flow errors. The image composed of multiple optical flow errors can be used as the optical flow error image. Alternatively, the distance between some candidate pixels and the fourth matching point can be calculated as the optical flow error image. In this example embodiment, no specific limitation is made.

[0066] In step S230, a target pixel is selected from the candidate pixels based on the optical flow error of the candidate pixels.

[0067] In this example implementation, after determining the optical flow error, a target pixel can be selected from the candidate pixels based on the optical flow error. Specifically, a preset optical flow threshold can be set first, and then multiple target pixels can be selected from multiple candidate pixels based on the preset optical flow threshold.

[0068] In this example implementation, candidate pixels with optical flow errors less than the preset optical flow threshold can be determined as target pixels. Specifically, optical flow errors less than the preset optical flow threshold can be determined as target optical flow, and candidate pixels corresponding to the target optical flow can be determined as target pixels. The value of the preset optical flow threshold can be customized according to user needs, and is not specifically limited in this example implementation.

[0069] The optical flow error calculated by forward and reverse optical flow, and the target pixel determined from the candidate pixels based on the optical flow error and the preset optical flow threshold, can increase the reliability of the target pixel and make the subsequent feature point matching more accurate.

[0070] In step S240, repeat texture detection is performed on the target pixels to obtain detection results, and matching feature point pairs between the image to be registered and the reference image are determined based on the detection results.

[0071] In this example embodiment, after obtaining the target pixel, a repeating texture detection can be performed on the target pixel to obtain a detection result. The detection result may include whether the target pixel is a repeating texture pixel or a non-repeating texture pixel. When performing repeating texture detection on the target pixel, steps S710 to S750 can be executed for the target pixel. Specifically:

[0072] In step S710, a first image block including the target pixel is obtained;

[0073] In one example embodiment of this disclosure, when obtaining a first image block including the target pixel, an image block of a preset size can be determined centered on the target pixel as the first image block. The preset size can be 9*9, 7*7, 11*11, etc., and can also be customized according to user needs. For example, when detecting large repeating textures, a larger preset size can be used. In this example embodiment, no specific limitation is made.

[0074] In step S720, the gradient of the first image patch is determined;

[0075] In this example embodiment, after determining the first image block, we can determine the gradient of the first image block, that is, determine the direction of the edge of the first image block. The gradient can be horizontal, vertical, or any direction tilted at any angle from the horizontal, such as tilted at 45 degrees or 23 degrees from the horizontal. In this example embodiment, no specific limitation is made.

[0076] In step S730, at least one second image block with the same size as the first image block is determined along the direction of the gradient;

[0077] After determining the gradient of the first image block, at least one second image block with the same size as the first image block can be determined along the direction of the gradient. For example, a second image block with the same size as the first image block can be determined along the positive direction of the gradient, and then a second image block with the same size as the first image block can be determined along the negative direction of the gradient. In this example embodiment, when selecting the second image block, the target pixel can be used as the starting point and the step size can be 5 to determine the second image block.

[0078] For details, please refer to Figure 8 As shown, assume that pixel coordinate point p is the target pixel point {P} as For a pixel in the image, first take a 9x9 image patch centered at pixel p, denoted as patch(p). Then, starting from pixel p, take two 9x9 image patches, one before and one after p, with a step size of 5, along the gradient direction of p, denoted as patch(p). l ) and patch(p r The step size can be half the size of the first image block. If the size of the first image block is odd, any integer that is half the size of the nearest one can be used. It can also be customized according to the usage scenario. At the same time, there are multiple second image blocks in one direction. For example, according to the step size of 1, 2 or 3, two or more second image blocks are taken before and after the gradient direction of p.

[0079] In step S740, the similarity between the first image patch and the second image patch is calculated;

[0080] In one exemplary embodiment of this disclosure, after obtaining the first image patch and the second image patch, the similarity between the first image patch and the second image patch is calculated. Specifically, patch(p) and patch(p) can be extracted respectively. l ) and patch(p r The pixel value sequence is arranged from left to right and from top to bottom, and the results are denoted as: L l L0 and L r ; Calculate L0 and L l The similarity is denoted as S. 0l Calculate L0 and L r The similarity is S 0r The similarity can be calculated using the following formula:

[0081]

[0082] Here, L1 and L2 are the pixel value sequences of two image blocks to be compared, that is, the pixel value sequences corresponding to the first image block and the second image block, respectively. and d(L1,L2) is the mean pixel value of the first and second image blocks. d(L1,L2) represents the similarity, with a value ranging from -1 to 1. The closer a value is to 1, the more similar the two image blocks are. If L1 and L2 are equal, meaning the two image blocks are the same, then the numerator equals the denominator, and the correlation calculation result equals 1. Here, k represents the pixel identifier of the first and second image blocks, i.e., the sorting sequence number of the pixels.

[0083] It should be noted that other methods can also be used to determine the similarity between the first and second image patches, such as comparing the grayscale histograms and gradient histograms of the image regions to determine whether the two image regions are similar. Alternatively, the similarity of image regions can be measured by calculating the sum of squared errors (SSD), sum of absolute errors (SAD), mean absolute difference (MAD), mean sum of squared errors (MSD), normalized product correlation (NCC), structural similarity (SSIM), etc., but these methods are not specifically limited in this example implementation.

[0084] In step S750, the detection result is obtained based on the similarity.

[0085] In this example implementation, after calculating the similarity, the detection result can be determined based on the similarity. Specifically, multiple similarities corresponding to the same target pixel are determined, and the detection result of whether the target pixel is a duplicate texture pixel is determined by comparing the similarity threshold and the similarity.

[0086] In this example implementation, the maximum value among multiple similarity scores corresponding to the same target pixel can be determined first. Then, the maximum value is compared with the similarity threshold. If the maximum value is greater than the preset threshold, the target pixel is determined to be a repeating texture pixel. If the maximum value is less than or equal to the similarity threshold, the target pixel is determined to be a non-repeating texture pixel.

[0087] After obtaining the above detection results, if the target pixel is a pixel in the reference image, then the matching feature point corresponding to the target pixel in the reference image that is a non-repeating texture pixel is determined in the image to be registered.

[0088] If the target pixel is a pixel in the image to be registered, then the matching feature point corresponding to the target pixel in the image to be registered, which is a non-repeating texture pixel, is determined in the reference image.

[0089] The above steps can remove duplicate texture pixels, avoiding the generation of incorrect optical flow due to the repetition of brightness features in duplicate texture areas, thus preventing matching errors and improving the accuracy of feature point matching.

[0090] In one exemplary embodiment of this disclosure, the weight information of the target pixel can also be determined based on the aforementioned similarity, and then the matching feature point pair between the image to be registered and the reference image can be determined based on the weight information. Wherein, the greater the phase velocity, the smaller the weight information corresponding to the target pixel.

[0091] In this example implementation, the aforementioned weight information can be used for feature point matching. For example, when using feature points for parameter optimization or model optimization, the target loudness points in the repeating texture region are given a lower weight for optimization, while the target pixel points in the non-repeating texture region are given a higher weight for optimization, thereby reducing the impact of mismatches caused by repeating textures and weak textures to a certain extent.

[0092] The following can be used as a reference. Figure 9 The above-mentioned duplicate texture detection is described in detail as follows: First, step S910 is executed to input the target pixel. Then, step S920 is executed to determine the first image block centered on the target pixel. Next, step S930 is executed to determine the gradient corresponding to the first image block. Then, step S940 is executed to obtain a second image block along the positive and negative directions of the gradient. Then, step S950 is executed to calculate the similarity between the two second image blocks and the first image block. After obtaining the similarity, step S960 is executed to determine the maximum value of the similarity. Then, step S970 is executed to determine whether the maximum value is greater than the similarity threshold. If it is, step S980 is executed to determine the target pixel as a duplicate texture pixel. If not, step S990 is executed to determine the target pixel as a non-duplicate texture pixel.

[0093] In one exemplary embodiment of this disclosure, reference is made to Figure 10 As shown, the feature point matching method of this disclosure may further include steps S250 and S260.

[0094] In step S250, the epipolar error of the feature points in the matched feature point pair is detected.

[0095] In this example implementation, the fundamental matrix between the image to be registered and the reference image can be determined first. Then, the epipolar lines corresponding to the pixels in the reference image in the image to be registered can be determined. Finally, the distance from the pixel in the image to be registered to the epipolar line corresponding to the pixel in the reference image is determined as the epipolar error. Specifically, the reference... Figure 11As shown, the fundamental matrix F of the image to be registered and the reference image is obtained. The epipolar line L corresponding to a feature point p1 in the left image is obtained using the matrix F. Finally, the distance from the feature point p2 corresponding to p1 in the right image to the epipolar line L is calculated and denoted as the epipolar error of this pair of feature points.

[0096] Among them, reference Figure 11 As shown, the epipolar constraint means that for any point in the left image L, its corresponding matching point in the right image R must lie on a certain straight line, which is called the epipolar line. Point P is a point in the world coordinate system, p is a point in the left image, and p′ is a point in the right image. Then Ppp′ forms a plane called the epipolar plane, and the two straight lines l1 and l2 on the left and right sides are called the epipolar lines. l With O r These are the optical centers of the left and right cameras. If a point P is known in the left image, we want to find the epipolar line containing the corresponding matching point p′ in the right image. If the aforementioned matching feature point is a mismatch, then the relationship described above does not hold. The epipolar line found through the epipolar constraint will have a large distance from the matching point, meaning the epipolar error is large. Therefore, points with large epipolar errors can be directly identified as mismatches and removed based on the magnitude of the epipolar error.

[0097] In step S260, the matching feature point pair is updated according to the epipolar error.

[0098] In this example implementation, after obtaining the epipolar error, the matching feature point pair can be updated using the epipolar error. Specifically, the labeling difference of the epipolar error corresponding to the target loudness can be determined first, and then an error threshold can be determined based on the labeling difference. Finally, the feature matching point pair can be updated based on the epipolar error and the error threshold.

[0099] For example, after calculating the polar error σ, points where the polar error is greater than 3σ from the mean can be removed according to the 3σ rule.

[0100] It should be noted that steps S250 and S260 above are for deleting mismatched points, which can further improve the accuracy of feature point matching and make the matching degree of the obtained feature point pairs higher.

[0101] In one example implementation, the method for deleting mismatched points is not limited to the above-mentioned epipolar error; a conventional RANSAC sieve point scheme can also be used. It is preferable to customize it according to user needs, and no specific limitation is made in this example implementation.

[0102] The following reference Figure 12As shown, the publicly disclosed feature point matching method is described in detail. First, step S1210 is performed for preprocessing, that is, preprocessing the reference image and the image to be registered. Then, step S1220 is performed to obtain the forward and backward optical flows, that is, to obtain the forward and backward optical flows between the reference image and the image to be registered through the DIS module. Then, step S1230 is performed to obtain the target pixel points. Then, step S1240 is performed to repeat texture detection. Then, step S1250 is performed to obtain the matching feature point pairs based on the detection results. Then, step S1260 is performed to delete mismatched points. Finally, step S1270 is performed to obtain the updated matching feature point pairs.

[0103] In summary, in this exemplary embodiment, on the one hand, optical flow error is determined by the forward and reverse optical flow between the image to be registered and the reference image, and target pixels are obtained based on the optical flow error, eliminating inaccurate pixels and improving the accuracy of feature point matching. On the other hand, matching feature point pairs between the image to be registered and the reference image are determined based on the duplicate texture detection results, deleting target pixels that are detected as duplicate texture pixels, reducing the influence of duplicate textures on feature point matching, further improving the accuracy of feature point matching and the matching degree of the obtained matching feature point pairs. Furthermore, after obtaining the matching feature point pairs, the epipolar error between each matching feature point pair is calculated, and whether it is a mismatch is determined based on the epipolar error, deleting mismatched matching feature points, further improving the accuracy of feature point matching and the matching degree of the obtained matching feature point pairs.

[0104] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0105] Further reference Figure 13 As shown, this example embodiment also provides a feature point matching device 1300, including an optical flow acquisition module 1310, an error determination module 1320, a pixel selection module 1330, and an image matching module 1340. Wherein:

[0106] The optical flow acquisition module 1310 can be used to acquire the forward optical flow and the reverse optical flow between the image to be registered and the reference image.

[0107] The error determination module 1320 can be used to determine the optical flow error of candidate pixels in the image to be registered or the reference image based on the forward optical flow and the reverse optical flow.

[0108] In one example implementation, the forward optical flow is the optical flow from a reference image to an image to be registered, and the reverse optical flow is the optical flow from the image to be registered to a reference image. The error determination module 1320 is configured to determine a first matching point of candidate pixels in the reference image in the image to be registered based on the forward optical flow; determine a second matching point of the first matching point in the reference image based on the reverse optical flow; and determine the optical flow error based on the candidate pixels in the reference image and the second matching point corresponding to the candidate pixels.

[0109] In one example implementation, the error determination module 1320 may also be configured to determine a third matching point of a candidate pixel in the image to be registered in the reference image based on the reverse optical flow; determine a fourth matching point of the third matching point in the image to be registered based on the forward optical flow; and determine the optical flow error based on the candidate pixel in the image to be registered and the fourth matching point corresponding to the candidate pixel.

[0110] The pixel selection module 1330 can be used to select a target pixel from the candidate pixels based on the optical flow error of the candidate pixels.

[0111] In one example implementation, the pixel selection module 1330 may be configured to acquire a preset optical flow threshold; determine a target optical flow in the optical flow error according to the preset optical flow threshold; and select candidate pixels in the reference image that correspond to the target optical flow as the target pixels.

[0112] The image matching module 1340 can be used to perform repeating texture detection on the target pixels to obtain detection results, and determine the matching feature point pairs between the image to be registered and the reference image based on the detection results.

[0113] In one example embodiment of this disclosure, the image matching module 1340 may be configured to perform the following operations on the target pixel: acquire a first image block including the target pixel; determine the gradient of the first image block; determine at least one second image block of the same size as the first image block along the direction of the gradient; calculate the similarity between the first image block and the second image block; and obtain the detection result based on the similarity.

[0114] In this example embodiment, when at least one second image block with the same size as the first image block is determined along the direction of the gradient, the image matching module 1340 can determine one second image block with the same size as the first image block along the positive direction of the gradient, and determine another second image block with the same size as the first image block along the negative direction of the gradient; and calculate the similarity between the first image block and the second image block.

[0115] In this example embodiment, the image matching module 1340 can be configured to obtain the maximum value of the similarity between the first image patch and the second image patch; and obtain the detection result based on the maximum value and the similarity threshold.

[0116] In one example embodiment of this disclosure, the image matching module 1340 can be configured to obtain a detection result of whether the target pixel is a repeating texture pixel by comparing the similarity with a similarity threshold.

[0117] In one example embodiment of this disclosure, the image matching module 1340 may be configured to determine weight information of target pixels based on the similarity; and to determine matching feature point pairs between the image to be registered and the reference image based on the weight information.

[0118] In one exemplary embodiment of this disclosure, the image matching module 1340 may be configured to determine the matching feature point corresponding to the target pixel point in the reference image that is detected as a non-repeating texture pixel point in the reference image; or to determine the matching feature point corresponding to the target pixel point in the image to be registered that is detected as a non-repeating texture pixel point in the reference image.

[0119] In one exemplary embodiment of this disclosure, reference is made to Figure 14 As shown, the feature point matching device may further include an error detection module 1350 and a matching update module 1360, wherein,

[0120] The error detection module 1350 can be used to detect the epipolar error of the feature points in the matched feature point pair.

[0121] In one example implementation, the error detection module 1350 may be configured to determine the fundamental matrix between the image to be registered and the reference image; determine the epipolar line corresponding to a pixel in the reference image in the image to be registered; and determine the distance from the pixel in the image to be registered to the epipolar line as the epipolar line error.

[0122] The matching update module 1360 can be used to update the matching feature point pair according to the epipolar error.

[0123] In one example embodiment of this disclosure, the matching update module 1360 may be configured to determine the standard deviation of the epipolar error corresponding to all target pixels; determine an error threshold based on the standard deviation; and update the matching feature points based on the epipolar error and the error threshold.

[0124] The specific details of each module in the above-mentioned device have been described in detail in the method section of the implementation. For any undisclosed details, please refer to the implementation content of the method section, and therefore will not be repeated here.

[0125] Exemplary embodiments of this disclosure also provide an electronic device for performing the above-described feature point matching method. This electronic device may be the terminal 110 or the server 130 described above. Generally, the electronic device may include a processor and a memory, the memory for storing executable instructions of the processor, and the processor configured to perform the above-described image feature point matching method by executing the executable instructions.

[0126] The following is based on Figure 15 Taking the mobile terminal 1500 as an example, the construction of this electronic device will be described by way of example. Those skilled in the art will understand that, apart from components specifically designed for mobile purposes, Figure 15 The structure can also be applied to fixed types of equipment.

[0127] like Figure 15 As shown, the mobile terminal 1500 may specifically include: a processor 1501, a memory 1502, a bus 1503, a mobile communication module 1504, an antenna 1, a wireless communication module 205, an antenna 2, a display screen 1506, a camera module 1507, an audio module 1508, a power module 1509, and a sensor module 1510.

[0128] Processor 1501 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, an encoder, a decoder, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). The feature point matching method in this exemplary embodiment can be executed by an AP, GPU, or DSP. When the method involves neural network-related processing, it can be executed by an NPU.

[0129] An encoder can encode (i.e., compress) images or videos. For example, it can encode a target image into a specific format to reduce data size for easier storage or transmission. A decoder can decode (i.e., decompress) the encoded data of an image or video to restore the image or video data. For example, it can read the encoded data of a target image, decode it, and restore the target image data, then perform feature point matching and related processing on the data. The mobile terminal 1500 can support one or more encoders and decoders. In this way, the mobile terminal 1500 can process images or videos in various encoding formats, such as: JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap) image formats, MPEG (Moving Picture Experts Group) 1, MPEG2, H.263, H.264, HEVC (High Efficiency Video Coding) video formats.

[0130] The processor 1501 can be connected to the memory 1502 or other components via the bus 1503.

[0131] The memory 1502 can be used to store computer executable program code, which includes instructions. The processor 1501 executes various functional applications and data processing of the mobile terminal 1500 by running the instructions stored in the memory 1502. The memory 1502 can also store application data, such as images, videos, and other files.

[0132] The communication function of mobile terminal 1500 can be implemented through mobile communication module 1504, antenna 1, wireless communication module 1505, antenna 2, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 1504 can provide 2G, 3G, 4G, and 5G mobile communication solutions for mobile terminal 1500. Wireless communication module 1505 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for mobile terminal 200.

[0133] The display screen 1506 is used to implement display functions, such as displaying the user interface, images, and videos. The camera module 1507 is used to implement shooting functions, such as capturing images and videos. The audio module 1508 is used to implement audio functions, such as playing audio and capturing voice. The power module 1509 is used to implement power management functions, such as charging the battery, powering the device, and monitoring battery status. The sensor module 1510 may include a depth sensor 15101, a pressure sensor 15102, a gyroscope sensor 15103, a barometric pressure sensor 15104, etc., to implement corresponding sensing and detection functions.

[0134] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0135] Exemplary embodiments of this disclosure also provide a computer-readable storage medium having a program product stored thereon capable of implementing the methods described above in this specification. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0136] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0137] In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0138] Furthermore, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0139] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0140] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A feature point matching method, characterized in that, include: Obtain the forward optical flow and the reverse optical flow between the image to be registered and the reference image; wherein, the forward optical flow is the optical flow from the reference image to the image to be registered, and the reverse optical flow is the optical flow from the image to the reference image; Determining the optical flow error of candidate pixels in the image to be registered or the reference image based on the forward optical flow and the reverse optical flow includes: determining a first matching point of the candidate pixels in the reference image in the image to be registered based on the forward optical flow; determining a second matching point of the first matching point in the reference image based on the reverse optical flow; and determining the optical flow error based on the candidate pixels in the reference image and the corresponding second matching points; or... The candidate pixel in the image to be registered is determined as a third matching point in the reference image based on the reverse optical flow; the third matching point is determined as a fourth matching point in the image to be registered based on the forward optical flow; and the optical flow error is determined based on the candidate pixel in the image to be registered and the fourth matching point corresponding to the candidate pixel. Based on the optical flow error of the candidate pixels, a target pixel is selected from the candidate pixels. Repeated texture detection is performed on the target pixels to obtain detection results, and matching feature point pairs between the image to be registered and the reference image are determined based on the detection results; The step of performing duplicate texture detection on the target pixel to obtain the detection result includes: The following operation is performed on the target pixel: Obtain a first image block including the target pixels; Determine the gradient of the first image block; A second image block with the same size as the first image block is determined along the positive direction of the gradient, and another second image block with the same size as the first image block is determined along the negative direction of the gradient. Calculate the similarity between the first image patch and the second image patch; Obtain the maximum similarity between the first image patch and the second image patch; The detection result is obtained based on the maximum value and the similarity threshold.

2. The method according to claim 1, characterized in that, The step of selecting a target pixel from the candidate pixels based on the optical flow error of the candidate pixels includes: Obtain a preset optical flow threshold; The target optical flow is determined from the optical flow error based on the preset optical flow threshold. The candidate pixels in the reference image that correspond to the target optical flow are taken as the target pixels.

3. The method according to claim 1, characterized in that, The step of obtaining the detection result based on the similarity includes: By comparing the similarity with the similarity threshold, the detection result of whether the target pixel is a repeating texture pixel is obtained.

4. The method according to claim 1, characterized in that, The step of determining the matching feature point pair between the image to be registered and the reference image based on the detection result includes: The weight information of the target pixel is determined based on the similarity. The matching feature point pairs between the image to be registered and the reference image are determined based on the weight information.

5. The method according to claim 1, characterized in that, The detection result includes whether the target pixel is a repeating texture pixel or a non-repeating texture pixel; determining the matching feature point pair between the image to be registered and the reference image based on the detection result includes: The matching feature point in the image to be registered is determined by identifying the target pixel in the reference image that is a non-repeating texture pixel; or The matching feature points in the reference image are determined for the target pixels in the image to be registered that are detected as non-repeating texture pixels.

6. The method according to claim 1, characterized in that, After obtaining the matching feature point pairs, the method further includes: Detect the epipolar error of the feature points in the matched feature point pair; The matching feature point pairs are updated based on the epipolar error.

7. The method according to claim 6, characterized in that, The epipolar error in detecting the feature points in the matched feature points includes: Determine the fundamental matrix between the image to be registered and the reference image; Determine the epipolar lines of the pixels in the reference image in the image to be registered; The distance from the epipolar line to the pixel point in the image to be registered that corresponds to the pixel point in the reference image is determined as the epipolar error.

8. The method according to claim 6, characterized in that, The step of updating the matching feature point pair based on the epipolar error includes: Determine the standard deviation of the epipolar error for all target pixels; The error threshold is determined based on the standard deviation. The matching feature points are updated based on the epipolar error and the error threshold.

9. A feature point matching device, characterized in that, include: An optical flow acquisition module is used to acquire the forward optical flow and the reverse optical flow between the image to be registered and the reference image; wherein, the forward optical flow is the optical flow from the reference image to the image to be registered, and the reverse optical flow is the optical flow from the image to the reference image; An error determination module is used to determine the optical flow error of candidate pixels in the image to be registered or the reference image based on the forward optical flow and the reverse optical flow, including: determining a first matching point of the candidate pixels in the reference image in the image to be registered based on the forward optical flow; determining a second matching point of the first matching point in the reference image based on the reverse optical flow; determining the optical flow error based on the candidate pixels in the reference image and the corresponding second matching point; or, determining a third matching point of the candidate pixels in the image to be registered in the reference image based on the reverse optical flow; determining a fourth matching point of the third matching point in the image to be registered based on the forward optical flow; and determining the optical flow error based on the candidate pixels in the image to be registered and the corresponding fourth matching point. Pixel selection module: used to select a target pixel from the candidate pixels based on the optical flow error of the candidate pixels; An image matching module is used to perform repeat texture detection on the target pixels to obtain detection results, and determine the matching feature point pairs between the image to be registered and the reference image based on the detection results; The step of performing duplicate texture detection on the target pixel to obtain the detection result includes: The following operation is performed on the target pixel: Obtain a first image block including the target pixels; Determine the gradient of the first image block; A second image block with the same size as the first image block is determined along the positive direction of the gradient, and another second image block with the same size as the first image block is determined along the negative direction of the gradient. Calculate the similarity between the first image patch and the second image patch; Obtain the maximum similarity between the first image patch and the second image patch; The detection result is obtained based on the maximum value and the similarity threshold.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the feature point matching method as described in any one of claims 1 to 8.

11. An electronic device, characterized in that, include: One or more processors; as well as A memory for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the feature point matching method as described in any one of claims 1 to 8.

Images