Feature point matching methods, devices, equipment and media for stereo images

Abstract

This invention provides a method, apparatus, device, and medium for feature point matching of stereo images. It involves acquiring a stereo image pair including a first image and a second image; detecting and extracting edge points of the image pair; acquiring grayscale information and gradient information of the edge points; calculating a first evaluation value containing gradient information and a second evaluation value containing grayscale information; and then effectively fusing gradient information and grayscale information based on a weighted sum of the first and second evaluation values. This effectively reduces matching uncertainty and improves accuracy. Finally, the optimal matching point is determined from the candidate matching points. The calculation can be performed based on the edge points and their grayscale and gradient information, resulting in a simple, low-complexity, and low-calculation approach. This achieves both improved edge matching accuracy and faster matching speed.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, device, and medium for feature point matching of stereo images. Background Technology

[0002] Currently, stereo matching is a key technology in binocular stereo vision measurement, solving the problem of matching midpoints in stereo images. Local stereo matching algorithms are characterized by low computational complexity, high computation speed, and good real-time performance. Local stereo matching utilizes a cost function (or similarity metric) to determine disparity by comparing image patches of the same size in the left and right views.

[0003] There are two common methods: one is to use gradient information for 3D matching of edge points, and the other is to use grayscale information for matching. The former selects too many candidate points when edge points are too densely packed. When the gradient direction and magnitude differences between candidate points are small, using only gradient as a feature for point matching has high uncertainty, leading to a significant decrease in matching accuracy. Similarly, the latter is also too simplistic; when the grayscale feature values ​​between candidate points are highly similar, the matching uncertainty is also high. Although both methods are relatively fast and computationally inefficient, their matching accuracy is not high.

[0004] There are also some feature point matching methods with high accuracy, such as SIFT, SURF, and ORB. For example, SIFT and SURF concatenate multiple gradient directions of neighboring points into a complex descriptor to describe similar features, while ORB compares the gray values ​​of the neighbors with the gray value of the center point and sets the gray values ​​of the neighbors to 0 and 1, forming a string of binary code to describe similarity. In order to improve accuracy, the complexity of the algorithm and the computation time are increased.

[0005] Therefore, there is an urgent need for a feature point matching method, device, equipment, and medium for stereo images, as well as an edge stereo matching algorithm that can balance accuracy and real-time performance to improve the above problems. Summary of the Invention

[0006] The purpose of this invention is to provide a method, apparatus, device, and medium for feature point matching of stereoscopic images, so as to improve the accuracy of edge matching and have a better matching speed.

[0007] In a first aspect, the present invention provides a feature point matching method for a stereo image, comprising: acquiring a stereo image pair, the image pair including a first image and a second image; detecting and extracting edge points of the image pair; and acquiring grayscale information and gradient information of the edge points;

[0008] For each edge point in the first image, the following processing is performed sequentially:

[0009] An edge point is selected from the first image as the point to be matched. Candidate matching points are filtered in the second image using the epipolar geometry principle. A first evaluation value is calculated between the gradient information of each candidate matching point and the point to be matched using a similarity evaluation function. A second evaluation value is calculated between the grayscale information of each candidate matching point and the point to be matched using a similarity evaluation function. Based on the weighted value of the first evaluation value and the second evaluation value, the best matching point that matches the point to be matched is determined from the candidate matching points.

[0010] The beneficial effects of the method of the present invention are as follows: by acquiring a stereo image pair containing a first image and a second image; detecting and extracting edge points of the image pair; and acquiring grayscale information and gradient information of the edge points; calculating a first evaluation value containing gradient information and a second evaluation value containing grayscale information respectively; and then, based on the weighted value of the first evaluation value and the second evaluation value, effectively fusing gradient information and grayscale information, which can effectively reduce the uncertainty of matching and thus improve accuracy; finally, determining the best matching point that matches the point to be matched from the candidate matching points. During the calculation, the weighted value of the first evaluation value and the second evaluation value can be calculated based on the edge points and the grayscale information and gradient information of the edge points. The overall steps are relatively simple, with low complexity and small computational load, thereby achieving both improved edge matching accuracy and good matching speed.

[0011] Optionally, the step of detecting and extracting edge points of the image pair and obtaining grayscale and gradient information of the edge points includes: performing edge detection on the image pair using an edge detection algorithm to obtain single-pixel edge features; extracting the single-pixel edge features through thresholding to obtain edge points; and calculating the grayscale value, gradient direction, and gradient magnitude corresponding to the edge points.

[0012] Optionally, after detecting and extracting the edge points of the image pair, the method further includes: performing grayscale detection on the image, and if it is a color image, converting the image into a grayscale image.

[0013] Optionally, it also includes: saving the matched points in the first image as a first point set; and saving all the best matching points in the second image as a second point set; selecting a point from the second point set as a new point to be matched, using the epipolar geometry principle to filter out the corresponding matching points in the first image, and saving all the matching points as a third point set after all the points in the second point set have been matched.

[0014] By comparing the coordinates of each point in the first and third point sets, if the difference between any two points is within a threshold range, then the new unmatched points corresponding to both points within the threshold range are recorded as the final matching points. The advantage of this approach is that it first uses a small number of edge points in the first image as matching objects, thereby further narrowing down the search range for matching points. Then, it uses all the best matching points in the second image as new matching objects, using a mutually corresponding method to filter matching points again, reducing the probability of false matches. This ensures that the overall process is simple, has low complexity, low computational cost, and good matching results for images with obvious and clear edges.

[0015] Optionally, the similarity evaluation function satisfies the following formula:

[0016]

[0017] In the formula, S(x,y) is the weighting value, I1 and I2 are the first and second images in binocular stereo vision, respectively. The neighborhood I2 centered at (u,v) is taken as the template, W is the selected template neighborhood, M is the number of pixels occupied by the template, I1(u,v) is the gray value of the coordinate point (u,v) in the first image, and I2(u+x,v+y) is the gray value of the coordinate point (u+x,v+y) in the second image. Let (u, v) be the gradient magnitude of the coordinate point (u, v) in the first image along the x-axis. Let be the gradient magnitude of the coordinate point (u,v) in the first image along the y-axis. Let (u+x, v+y) be the gradient magnitude of the coordinate point (u+x, v+y) in the second image along the x-axis. Let K be the gradient magnitude of the coordinate point (u+x, v+y) in the second image along the y-axis, and K be the weighting factor.

[0018] Secondly, the present invention also provides a feature point matching device for a stereo image, the device comprising modules / units for performing any of the possible design methods described in the first aspect. These modules / units can be implemented in hardware or by hardware executing corresponding software.

[0019] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a program executable on the processor, and when the program is executed by the processor, the electronic device implements a method for performing any of the possible designs described above.

[0020] Fourthly, the present invention also provides a readable storage medium storing a program, which, when executed, implements a method of any possible design of any of the above aspects. Attached Figure Description

[0021] Figure 1 A flowchart illustrating a feature point matching method for a stereo image provided in an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of image pairs in binocular stereo vision provided in an embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram of the feature point matching system for stereo images provided in an embodiment of the present invention;

[0024] Figure 4 A flowchart for matching edge points in a first image and edge points in a second image, provided as an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of the structure of a feature point matching device for a stereoscopic image provided by the present invention;

[0026] Figure 6 This is a schematic diagram of the structure of an electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art. The terms "comprising" and similar expressions used herein mean that the element or object preceding the word covers the element or object listed following the word and its equivalents, but do not exclude other elements or objects.

[0028] The following explanations of some terms used in the embodiments of the present invention are provided to facilitate understanding by those skilled in the art.

[0029] The epipolar principle, or epipolar constraint principle, states that in two images, the coordinates of a pixel in the first image and its corresponding pixel in the second image lie on a straight line, which is called the epipolar line.

[0030] like Figure 1 As shown, in view of the problems existing in the prior art, the present invention provides a feature point matching method for stereo images, including:

[0031] S101, acquire a stereo image pair, the image pair including a first image and a second image.

[0032] The stereo image pair can be a stereo image pair in binocular stereo vision, for example, such as Figure 2 As shown, it can be Figure 2 The image on the left is used as the first image of the stereo image pair, and the image on the right is used as the second image of the stereo image pair. It should be understood that the stereo image pair can also refer to other types of stereo images, and this embodiment does not limit this.

[0033] S102, detect and extract edge points of the image pair; and obtain grayscale information and gradient information of the edge points.

[0034] In some embodiments, after detecting and extracting the edge points of the image pair, the method further includes: performing grayscale detection on the image; if it is a color image, converting the image into a grayscale image, and then obtaining the grayscale information and gradient information of the edge points; if it is not a color image, directly obtaining the grayscale information and gradient information of the edge points. In this embodiment, the grayscale information of the edge points mainly refers to the grayscale value, and the gradient information of the edge points may include gradient magnitude and gradient direction, etc.

[0035] In some specific embodiments, the extraction of edge points includes a detection process and an extraction process. In the detection process, the Canny operator in the edge detection algorithm can be used to perform edge detection on the image pair to obtain single-pixel edge features. The edge points detected by the Canny operator are not only relatively dense but also highly accurate. In the extraction process, the single-pixel edge features are extracted through threshold processing.

[0036] S103, for each edge point of the first image, the following processing is performed sequentially: an edge point is selected from the first image as the point to be matched; n candidate matching points are selected in the second image using the epipolar geometry principle; a first evaluation value between the gradient information of each candidate matching point and the gradient information of the point to be matched is calculated using a similarity evaluation function; and a second evaluation value between the grayscale information of each candidate matching point and the grayscale information of the point to be matched is calculated using a similarity evaluation function; based on the weighted value of the first evaluation value and the second evaluation value, the best matching point that matches the point to be matched is determined from the n candidate matching points.

[0037] In some embodiments, the similarity evaluation function satisfies the following formula:

[0038]

[0039] In the formula, S(x,y) is the weighting value, I1 and I2 are the first and second images in binocular stereo vision, respectively. The neighborhood I2 centered at (u,v) is taken as the template, W is the selected template neighborhood, M is the number of pixels occupied by the template, I1(u,v) is the gray value of the coordinate point (u,v) in the first image, and I2(u+x,v+y) is the gray value of the coordinate point (u+x,v+y) in the second image. Let (u, v) be the gradient magnitude of the coordinate point (u, v) in the first image along the x-axis. Let be the gradient magnitude of the coordinate point (u,v) in the first image along the y-axis. Let (u+x, v+y) be the gradient magnitude of the coordinate point (u+x, v+y) in the second image along the x-axis. Let K be the gradient magnitude of the coordinate point (u+x, v+y) in the second image along the y-axis, and K be the weighting factor.

[0040] It should be understood that by substituting the gray values ​​of the point to be matched (u,v) in the first image and the candidate matching point (u+x,v+y) in the second image into the above formula, the weighted value can be calculated. By substituting the gray values ​​of each candidate matching point into the above formula in turn, n weighted values ​​can be calculated. The candidate matching point corresponding to the smallest weighted value is selected as the best matching point from the n weighted values. This method can achieve both improved edge matching accuracy and good matching speed.

[0041] The following text combines Figure 2 The above method embodiments are illustrated with examples. Figure 2 The left and right images are processed using an edge detection algorithm to detect edges in the image pair, obtaining the features of single-pixel edge points. Thresholding is then used to extract these single-pixel edge features, resulting in edge points. The grayscale values ​​of these edge points are then calculated, and based on these grayscale values, the gradient direction and magnitude of the edge points in each image are calculated. Next, one edge point is selected from the first image as a matching point. Using epipolar geometry, n candidate matching points are selected in the second image. Substituting the grayscale values ​​of the coordinate points into the above formula yields n weighted values. The minimum value among these n weighted values ​​determines the optimal matching point corresponding to the selected edge point in the first image, forming the optimal matching point pair. This algorithm uses a small number of edge points as matching objects and employs epipolar constraints to further narrow down the search range for matching points. By using a matching method combining grayscale and gradient, edge feature point matching is achieved.

[0042] To systematically illustrate the above method embodiments, the following description combines... Figure 3 Provide a system description. Figure 3In this process, the electronic device first acquires a pair of stereo images and uses the Canny operator to perform edge detection on each image in the pair. Then, it extracts the edge points of each image in the pair based on thresholding. Next, it determines whether the edge-detected stereo image pair is a color image. If it is, it converts the stereo image pair to grayscale to obtain a grayscale image; otherwise, it directly obtains the grayscale image. Based on the grayscale image, the grayscale value is calculated, and the gradient magnitude and direction are calculated using the Sobel operator. This yields the grayscale value, gradient magnitude, and direction of the edge points. Finally, based on the grayscale value, gradient magnitude, and direction of the edge points, these are substituted into a similarity evaluation function to obtain mutually matching edge points.

[0043] In some embodiments, after obtaining mutually matching edge points, the following steps may be further performed, such as... Figure 4 As shown, the method further includes: saving the matched points in the first image as a first point set; saving all the best matching points in the second image as a second point set; selecting a point from the second point set as a new point to be matched, and using the epipolar geometry principle to filter out the corresponding matching points in the first image; after all the points in the second point set have been matched, saving all the matching points as a third point set; comparing the coordinate values ​​of each point in the first point set and the third point set, if the difference between any two points is within a threshold range, then recording the new point to be matched corresponding to both points within the threshold range as the final matching point. By first using the points in the first image as points to be matched to obtain the corresponding best matching points in the second image, and then using all the best matching points in the second image as new matching points, the matching points are filtered again using a mutually corresponding method, thereby verifying the best matching points and improving the accuracy of the matching results.

[0044] It should be understood that regarding the setting of the threshold range, in some specific embodiments, it can be directly determined whether the coordinate values ​​of two points are the same. If they are the same, the mutual correspondence constraint is satisfied, and the new unmatched point corresponding to both points is recorded as the final matching point. In other specific embodiments, a common threshold range can be directly set. For example, if the error range of the coordinate values ​​of two points along any coordinate axis is within the common threshold range, it is considered that the mutual correspondence constraint is satisfied, and the new unmatched point corresponding to both points is recorded as the final matching point. Multiple threshold ranges can also be set according to actual needs. For example, the threshold range for the x-axis coordinate can be set as the first threshold interval, and the threshold range for the y-axis coordinate can be set as the second threshold interval.

[0045] To facilitate understanding, this embodiment further elaborates on the specific implementation method of the above method in conjunction with a specific application scenario, which includes the following steps:

[0046] Step a: Obtain a pair of stereo images, which includes a first image and a second image.

[0047] Step b: Perform edge detection on the first image and the second image respectively to obtain the first image and the second image containing their respective edge points.

[0048] Specifically, this step may include: using the Canny operator to detect the single-pixel edge features of the first image and the second image respectively; then applying thresholding to the single-pixel edge features of the first image and the second image respectively to obtain the edge points A{A1, A2, ..., Ai, ..., AN} of the first image, where set A belongs to the pixels in the first image and set B belongs to the pixels in the second image.

[0049] Step c: Perform grayscale detection on the first and second images containing their respective edge points after edge detection. If it is a color image, convert the image to grayscale first and then obtain the grayscale value. Otherwise, directly obtain the grayscale value of the pixel and calculate the gradient magnitude and gradient direction of the pixel based on the grayscale value.

[0050] Step d: Randomly select a point Ai from set A as the point to be matched. Calculate the epipolar line of point Ai in the second image based on the fundamental matrix, and select several points within a small distance of the epipolar line as a temporary matching point set. Then, select candidate matching points b{b1, b2, b3, ...} from the temporary matching point set whose gradient direction is the same as that of the edge point, and set b belongs to set B. Calculate the weighted value between the edge point Ai and each of the selected candidate matching points b{b1, b2, b3, ...} using the evaluation function in the above formula. Select the candidate matching point bi corresponding to the smallest weighted value as the best matching point that matches the point Ai. Repeat the above steps until all edge points in set A in the first image are matched. Save the matched points in the first image as set M1, and the matched points in the second image as set M2.

[0051] Step e: From the set of matched points M2{M1, M2, ..., Mj, ..., MN} in the second image, select a point Mj as the point to be matched. Using the epipolar geometry principle, filter out the corresponding temporary matching point set in the first image. Then, filter out the candidate matching points c{c1, c2, c3, ..., ci...cn} whose gradient direction is the same as the edge point from the temporary matching point set. Then, use the evaluation function in the above formula to calculate the weighted value between the edge point Mj and the filtered candidate matching points. Take the candidate matching point ci corresponding to the smallest weighted value as the best matching point that matches the point to be matched Mj. Repeat the above steps until the best matching point is found in the first image and saved, denoted as M1. T .

[0052] Step g, then compare M1 and M1 in sequence. TIf the x and y coordinates of two points in the first image are the same, then the mutual correspondence constraint is satisfied. The point in the second image that corresponds to two points with the same coordinates is recorded as the final matching point corresponding to the point in the first image.

[0053] The algorithm described above uses a small number of edge points as matching targets and employs epipolar constraints to further narrow down the search range for matching points. It uses a matching method combining grayscale and gradient to achieve edge feature point matching, and further filters matching points using a corresponding approach to reduce false matches. The overall algorithm is relatively simple, with low complexity and low computational cost, and it shows good matching results for images with clear and distinct edges.

[0054] like Figure 5 As shown, based on the above-mentioned feature point matching method for stereo images, the present invention provides a feature point matching device for stereo images, comprising: an acquisition unit 501 for a stereo image pair, the image pair including a first image and a second image; a detection unit 502 for detecting and extracting edge points of the image pair; the acquisition unit for acquiring grayscale information and gradient information of the edge points; and a processing unit 504 for sequentially performing the following processing on each edge point of the first image: selecting an edge point from the first image as a point to be matched; filtering candidate matching points in the second image using epipolar geometry principles; calculating a first evaluation value between the gradient information of each candidate matching point and the point to be matched using a similarity evaluation function; and calculating a second evaluation value between the grayscale information of each candidate matching point and the point to be matched using a similarity evaluation function; and determining the best matching point that matches the point to be matched from the candidate matching points based on the weighted value of the first evaluation value and the second evaluation value.

[0055] In some embodiments, it also includes:

[0056] The conversion unit 503 is used to perform grayscale detection on the image after the detection unit 502 detects and extracts the edge points of the image pair, and if it is a color image, convert the image into grayscale.

[0057] It should be understood that all relevant content of each step involved in the above method embodiments can be referenced to the functional description of the corresponding functional module, and will not be repeated here.

[0058] In other embodiments of the present invention, an electronic device 600 is disclosed, such as... Figure 6As shown, the device may include: one or more processors 601; memory 602; display 603; one or more application programs (not shown); and one or more computer programs 604. These devices can be connected via one or more communication buses 605. The one or more computer programs 604 are stored in the memory 602 and configured to be executed by the one or more processors 601. The one or more computer programs 604 include instructions that can be used to perform actions such as... Figure 1 , Figure 2 and Figure 3 Each step in the corresponding embodiment.

[0059] While embodiments of the present invention have been described in detail above, it will be apparent to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations fall within the scope and spirit of the invention as set forth in the claims. Furthermore, the invention described herein may have other embodiments and can be implemented or carried out in various ways.

Claims

1. A feature point matching method for a stereo image, characterized in that, include: Acquire a stereo image pair, wherein the image pair includes a first image and a second image; Detect and extract edge points of the image pairs; The method for obtaining the grayscale and gradient information of the edge points includes: performing edge detection on the image pair using an edge detection algorithm to obtain single-pixel edge features; extracting the single-pixel edge features through thresholding to obtain edge points; and calculating the grayscale value, gradient direction, and gradient magnitude corresponding to the edge points. For each edge point in the first image, the following processing is performed sequentially: An edge point is selected from the first image as the point to be matched. Candidate matching points are filtered in the second image using the epipolar geometry principle. A first evaluation value is calculated between the gradient information of each candidate matching point and the point to be matched using a similarity evaluation function. A second evaluation value is calculated between the grayscale information of each candidate matching point and the point to be matched using a similarity evaluation function. Based on the weighted value of the first evaluation value and the second evaluation value, the best matching point that matches the point to be matched is determined from the candidate matching points. Save the matched points in the first image as a first point set; and save all the best matching points in the second image as a second point set; Select a point from the second point set as a new point to be matched, and use the epipolar geometry principle to filter out the corresponding matching points in the first image. After all the points in the second point set have been matched, save all the matching points as the third point set. Compare the coordinate values ​​of each point in the first point set and the third point set. If the difference between any two points is within the threshold range, then record the new unmatched points corresponding to both points within the threshold range as the final matching points. The similarity evaluation function satisfies the following formula: In the formula, For weighted values, , These are the first and second images in binocular stereo vision, respectively. Take the neighborhood of the center As a template, The selected template neighborhood is M, where M is the number of pixels occupied by the template. Coordinates of the points in the first image grayscale value, Coordinates of the points in the second image grayscale value, Coordinates of the points in the first image Gradient magnitude in the x-axis direction, Coordinates of the points in the first image Gradient magnitude in the y-axis direction, Coordinates of the points in the second image Gradient magnitude in the x-axis direction, Coordinates of the points in the second image Gradient magnitude in the y-axis direction, This is the weighting factor.

2. The method according to claim 1, characterized in that, After detecting and extracting the edge points of the image pair, the method further includes: Perform grayscale detection on the image; if it is a color image, convert it to grayscale.

3. A feature point matching device for a stereoscopic image, characterized in that, include: Acquisition unit for stereo image pairs, the image pairs including a first image and a second image; The detection unit is used to detect and extract edge points of the image pair, and to perform edge detection on the image pair using an edge detection algorithm to obtain single-pixel edge features; The acquisition unit is used to acquire the grayscale information and gradient information of the edge points, and extract the single-pixel edge features through threshold processing to obtain the edge points; The processing unit is used to calculate the gray value, gradient direction, and gradient magnitude corresponding to the edge point; it is also used to perform the following processing sequentially for each edge point of the first image: An edge point is selected from the first image as the point to be matched. Candidate matching points are filtered in the second image using the epipolar geometry principle. A first evaluation value is calculated between the gradient information of each candidate matching point and the point to be matched using a similarity evaluation function. A second evaluation value is calculated between the grayscale information of each candidate matching point and the point to be matched using a similarity evaluation function. Based on the weighted value of the first evaluation value and the second evaluation value, the best matching point that matches the point to be matched is determined from the candidate matching points. The processing unit is further configured to save the matched points in the first image as a first point set; and save all the best matching points in the second image as a second point set; Select a point from the second point set as a new point to be matched, and use the epipolar geometry principle to filter out the corresponding matching points in the first image. After all the points in the second point set have been matched, save all the matching points as the third point set. Compare the coordinate values ​​of each point in the first point set and the third point set. If the difference between any two points is within the threshold range, then record the new unmatched points corresponding to both points within the threshold range as the final matching points. The similarity evaluation function satisfies the following formula: In the formula, For weighted values, , These are the first and second images in binocular stereo vision, respectively. Take the neighborhood of the center As a template, The selected template neighborhood is M, where M is the number of pixels occupied by the template. Coordinates of the points in the first image grayscale value, Coordinates of the points in the second image grayscale value, Coordinates of the points in the first image Gradient magnitude in the x-axis direction, Coordinates of the points in the first image Gradient magnitude in the y-axis direction, Coordinates of the points in the second image Gradient magnitude in the x-axis direction, Coordinates of the points in the second image Gradient magnitude in the y-axis direction, This is the weighting factor.

4. The apparatus according to claim 3, characterized in that, Also includes: The conversion unit is used to perform grayscale detection on the image after the detection unit detects and extracts the edge points of the image pair. If it is a color image, the image is converted into a grayscale image.

5. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a program that can run on the processor, and when the program is executed by the processor, causes the electronic device to perform the method of any one of claims 1 to 2.

6. A computer-readable storage medium storing a program therein, characterized in that, When the program is executed, it implements the method of any one of claims 1 to 2.

Images