An image stereo matching optimization method based on searching for points of the same name

Abstract

The present invention discloses an image stereo matching optimization method based on homonymous point search, comprising the following steps: dividing the binocular camera shooting picture into a left eye image and a right eye image, respectively performing epipolar correction on the left eye image and the right eye image; determining the space point to be measured, taking the projection point of the space point on the left eye image as the key point, searching on the left epipolar line of the right eye image according to the key point, and determining the homonymous point corresponding to the key point; performing triangulation based on the coordinates of the key point and the coordinates of the homonymous point, and calculating the world coordinates of the space point. The present invention can improve the accuracy and speed of object size detection.

Description

Technical Field

[0001] The present invention relates to the technical field of binocular cameras, and in particular to an image stereo matching optimization method based on searching for points of the same name. Background Art

[0002] Solving the world coordinates of spatial points is the focus of research in the field of stereo matching. If a spatial point is a point on an object, then solving the world coordinates can measure the distance from the object to the camera; if some spatial points are a series of key points of the object, these key points can reflect the contour information of the object, then solving the world coordinates of these spatial points can approximately solve the size of the object. In binocular stereo vision, accurately calculating the world coordinates of spatial points plays a significant role in object distance measurement, size detection and three-dimensional reconstruction.

[0003] Most existing methods for calculating the world coordinates of spatial points are based on stereo matching algorithms. The depth map of the left image is obtained through the stereo matching algorithm, and then the disparity value of the projection point of the spatial point on the left image is obtained based on the depth map. Finally, the world coordinates of the point are solved based on the camera internal parameters. However, the stereo matching algorithm calculates the depth map for the entire image, which has extremely high time complexity and is not suitable for real-time application scenarios. At the same time, some classic stereo matching algorithms such as SGBM are easily constrained by the parameter of the disparity range value. Setting a relatively large disparity range is inefficient when detecting the world coordinates of points on distant objects, and setting a relatively small disparity range will result in the inability to measure when calculating the world coordinates of points on close objects. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes an image stereo matching optimization method based on same-name point search, which can improve the accuracy and speed of object size detection.

[0005] In order to achieve the above object, the technical solution of the present invention is as follows:

[0006] An image stereo matching optimization method based on same-name point search includes the following steps:

[0007] The images captured by the binocular camera are divided into left-eye image and right-eye image, and epipolar correction is performed on the left-eye image and the right-eye image respectively;

[0008] Determine the spatial point to be measured, take the projection point of the spatial point on the left eye image as the key point, search on the left epipolar line of the right eye image according to the key point, and determine the same-name point corresponding to the key point;

[0009] The world coordinates of the spatial point are calculated by triangulating the coordinates of the key points and the points with the same name.

[0010] Preferably, the step of searching on the left epipolar line of the right eye image according to the key point to determine the same-name point corresponding to the key point specifically comprises the following steps:

[0011] The left and right images are downsampled by 16 times at the same time. Based on the downsampled key points, the Census cost is calculated for all pixels in the search area along the left epipolar line in the downsampled right image, and the point with the smallest Census cost is used as the first candidate same-name point corresponding to the key point;

[0012] Based on the down-sampling gradient encoding method, the search area of ​​the first candidate same-name points at the original resolution is screened to determine the same-name points.

[0013] Preferably, based on the down-sampling gradient encoding method, the search area of ​​the first candidate homonymous points at the original resolution is screened to determine the homonymous points, which specifically includes the following steps:

[0014] Determine the area corresponding to the first candidate same-name point in the right image at the original resolution as the search area; based on the key points in the left image at the original resolution, take the four pixels in the row corresponding to the epipolar line in the search area as the second candidate same-name point;

[0015] A 31*31 area around the key point in the original resolution is cut out, and the key point is located in the center of the cut area, to obtain the key area; a 31*31 area around the second candidate same-name point in the original resolution is cut out, and the four second candidate same-name points are respectively located in the center of the corresponding cut area, to obtain four comparison areas;

[0016] The key area and the four comparison areas are downsampled by 16 times to a size of 7*7, and gradient encoding is performed on the 7*7 area of ​​the key area and the four comparison areas;

[0017] The corresponding gradient coding arrays of the four comparison areas are compared with the corresponding gradient coding array of the key area in turn, and the second candidate homonymous point corresponding to the comparison area with the closest comparison result is taken as the final homonymous point.

[0018] Based on the above technical scheme, the beneficial effects of the present invention are as follows: the present invention is an image stereo matching optimization method based on same-name point search, the binocular camera captures a picture into a left-eye image and a right-eye image, and the left-eye image and the right-eye image are respectively subjected to epipolar line correction; the spatial point to be measured is determined, the projection point of the spatial point on the left-eye image is used as the key point, and the same-name point corresponding to the key point is determined based on the key point. The coordinates of the key point and the coordinates of the same-name point are triangulated to calculate the world coordinates of the spatial point. The present invention further improves the accuracy and speed of the existing world coordinate system in stereo vision. Compared with the use of traditional stereo matching algorithms, the algorithm proposed by the present invention can solve the world coordinates of the spatial point with higher accuracy and faster speed, while avoiding the limitation of the maximum search parallax, and can further improve the efficiency and accuracy of object size detection, and achieve better object detection intelligence and automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a schematic flow chart of an image stereo matching optimization method based on searching for points of the same name in an embodiment;

[0020] Figure 2 is a schematic diagram of binocular camera projection points and same-name points in an image stereo matching optimization method based on same-name point search in an embodiment;

[0021] Figure 3 A schematic diagram of searching for same-name points on the epipolar lines of a downsampled image in an image stereo matching optimization method based on same-name point search in an embodiment

[0022] Figure 4 is a schematic diagram illustrating Census transformation in an image stereo matching optimization method based on same-name point search in an embodiment;

[0023] Figure 5 It is a schematic diagram of pixel information of a 16-fold down-sampled image and an original resolution image in an image stereo matching optimization method based on same-name point search in an embodiment;

[0024] Figure 6 is a schematic diagram of an original resolution image search area in an image stereo matching optimization method based on same-name point search in an embodiment;

[0025] Figure 7 is a schematic diagram of down-sampling gradient encoding in an image stereo matching optimization method based on same-name point search in an embodiment;

[0026] Figure 8 is a schematic diagram of a binocular camera model in an image stereo matching optimization method based on same-name point search in an embodiment;

[0027] Fig. 9 This is a schematic diagram of verifying the matching of homonymous points in the Middlebury dataset in Experiment 1;

[0028] Fig.10 This is a schematic diagram of the real scene verification of the matching of same-name points in Experiment 1;

[0029] Fig.11 This is a schematic diagram of the scene in Experiment 2. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0031] like Figure 1 As shown, this embodiment provides an image stereo matching optimization method based on same-name point search. The method is based on a binocular camera and divides the captured image into a left-eye image and a right-eye image. Figure 2 For a point P in space, first find the projection point p of the point in the left camera image. L ; Then search the same-name point p of the projection point of the left image in the right image by the same-name point search method R ; Finally, the world coordinates of the space point are calculated through triangulation, and then the tasks of measuring distance or solving object size in binocular vision tasks are completed. The specific processing process is as follows:

[0032] 1. Search for points with the same name

[0033] The search for homonymous points is divided into two steps. The first step is to search for the first candidate homonymous points along the epipolar line on the 16-fold downsampled image. Since each pixel after downsampling contains the information of a small area in the original resolution image, the second step is to determine 4 second candidate homonymous points in the search area on the original resolution image, and determine the final homonymous points from the 4 second candidate homonymous points.

[0034] 1.116 times downsampled image epipolar line homonymous point search area

[0035] The binocular camera is calibrated to obtain the camera internal parameters, and the internal parameters are used for epipolar correction. The pixel coordinates (x, y) of the key point in the left-eye image key point regression are intercepted and the pixel grayscale values ​​of the RGB three channels of the key point are recorded. In the right image, the pixel with the same coordinates (x, y) as the key point in the left image is found. Due to the nature of binocular camera shooting, the same-name point cannot appear on the right side of the pixel, so the same-name point of the left image key point is searched on the left epipolar line of the pixel. Traditional stereo matching algorithms mostly use enumeration methods, which calculate all pixel points in the search area of ​​the right-eye image by setting the maximum disparity search value. This method has high algorithm time complexity and is easily affected by image noise. In order to avoid using enumeration methods to search for same-name points, the present invention proposes a search method based on image downsampling to quickly screen the location of the same-name point area and then use gradient coding to finally confirm the location of the same-name point. The search method is as follows Figure 3 As shown in the figure, the left and right images are simultaneously downsampled 16 times, and the horizontal and vertical coordinates of the key point coordinates in the left view are also processed accordingly as the key points of the image after 16 times downsampling, and the same-name points of the key points are searched in the 16 times downsampled image of the right view. This method can avoid the limitation of the maximum disparity value, reduce the search area, and reduce the time complexity of the algorithm. At the same time, downsampling the image can also limit the impact of noise on disparity calculation.

[0036] SGM (Semi-Global Match) is a stereo matching algorithm, and its cost calculation method uses Census transformation. The first step of matching homonymous points draws on its cost calculation idea, and through Census cost calculation, candidate homonymous points are screened out in the 16-fold downsampled epipolar area proposed in the previous step, and then the next step is performed.

[0037] Census cost is based on Census transformation, which is a non-parametric regional algorithm. Its transformation is based on the grayscale of the neighborhood, while also retaining the relative position information of the neighborhood pixels. Census transformation workers select a fixed-size window that should be centered on the pixel, and then compare the grayscale value of the pixel in the window with the grayscale value of the center pixel. When the grayscale of the pixel in the window is greater than the grayscale of the center pixel, the pixel is marked as 1, otherwise it is marked as 0. After marking all the pixels, these values ​​are spliced ​​bit by bit to obtain the transformation code to calculate the Hamming distance. See Figure 4 , Census cost assumes that the same-name points should have the smallest Hamming distance. The Census cost is calculated for all pixels in the image epipolar search area after 16 times downsampling, and the minimum value is taken as the first candidate same-name point. The Census cost calculation formula is shown in formula (1-2-1), where I(p) is the center pixel of the window, I(q) is the pixel of the window other than the center pixel, and σ(I(p), I(q)) is the Census code value of a point in the window.

[0038] C TL (p) is the Census value of point p in the left image, C TR (p,d)) is the Census value of the point where the parallax of the right eye image is d.

[0039]

[0040] h(p,d)=Hamming(C TL (p),C TR (p,d))

[0041] Census transformation is more susceptible to noise, resulting in deviations in Hamming distance calculation. However, since the present invention uses Census transformation on a 16-fold downsampled image, 16-fold downsampling can eliminate part of the noise effect, so that Census transformation can effectively avoid noise influence in the scenario of the present invention.

[0042] 1.2 Original resolution search area for candidate homonymous points

[0043] After 16 times downsampling, the width and height of the image are reduced to 1 / 4 of the original image. Figure 5 As shown in the figure, each pixel of the image after 16 times downsampling contains the pixel information of the 4*4 small area of ​​the original resolution image. The first candidate homonymous point obtained in the first step also represents the information of the corresponding 4*4 area pixel of the original resolution image. The final homonymous point needs to be further searched in this area. However, since the image has been epipolar corrected, Figure 6 As shown in the figure, in this area, only the 4 pixels in the row where the epipolar line is located need to be searched as the second candidate homonymous points. Assuming the coordinates of the candidate homonymous points downsampled by 16 times are (x, y), the search area in the right eye image at the original resolution is [4*x:4*(x+1), y].

[0044] Since both the left and right cameras of the binocular camera introduce random Gaussian noise when shooting, this will have a great impact on the final screening of candidate homonymous points. In order to suppress random noise and obtain more abstract semantic information of the key point neighborhood for candidate homonymous points to compare, the present invention introduces downsampling again here. The 31*31 area around the key point in the original resolution is cut out and the key point is located in the center of the cut area to obtain the key area; the 31*31 area around the second candidate homonymous point in the original resolution is cut out and the four second candidate homonymous points are respectively located in the center of the corresponding cut area to obtain four comparison areas. The cut key area and the four comparison areas are all downsampled 16 times to a size of 7*7, and the 7*7 area is subjected to domain gradient encoding, and then compared with the encoding of the key point. Gradient coding is performed by calculating the grayscale difference with the pixels in the surrounding 8 directions to perform coding assignment. The assignment formula is shown in formula (1-2-2). I(d,p) represents the outer point when the direction is d, and I(d,q) represents the inner point when the direction is d. The outer and inner points are as follows: Figure 7 As shown, in the 7*7 window area, the outermost point is subtracted from the second outermost point along the 8 direction until the center point of the area.

[0045]

[0046] The coding information around the key points of the left image and the coding information around the candidate same-name points of the right image are placed in a one-dimensional array for comparison. The comparison method is to set Error_value and assign an initial value of 0. The key point neighborhood coding information array is compared bit by bit with the same-name point area coding information array. If the corresponding bit values ​​are different, the value of Error_value is +1. Finally, the candidate same-name point with the smallest Error_value value is used as the final matching same-name point.

[0047] 2. Triangulation

[0048] After obtaining the pixel coordinates of the key points through key point regression in the first step and the pixel coordinates of the searched points with the same name in the previous step, triangulation can be performed in combination with the camera intrinsic parameters. The principle of triangulation is to establish an equation to solve the world coordinates through the relationship between the world coordinate system and the pixel coordinate system.

[0049] Formula (1-3-1) expresses the relationship between pixel coordinates and world coordinates, [uv 1] T is the pixel coordinate of the image, where u represents the pixel coordinate along the y direction and v represents the pixel coordinate along the x direction. c is the distance from the object to the camera in the image camera coordinate system. is the intrinsic parameter matrix of the camera, where f x is the ratio of focal length to pixel width in the x direction, f yis the ratio of focal length to pixel height along y direction, u0 and v0 are the offsets of pixel point from the center of image. is the rotation and translation matrix of the camera, where Indicates that the camera coordinate system is Direction offset, R 11 Represents the projection of the x-axis of the camera coordinate system in the x-direction of the world coordinate system, R 22 represents the projection of the y-axis of the camera coordinate system on the y-direction of the world coordinate system, and the same is true for the others. These intrinsic parameter matrices can be obtained by camera calibration, and the final equation that needs to be solved is [x W y W z W 1] T That is the world coordinate of the camera.

[0050]

[0051] The stereo camera model is as follows Figure 8 As shown, O L and O R They are the optical centers of the left and right cameras, which point to a point p in space through their optical centers, leaving a point p on the left imaging plane. L , leaving point p on the right imaging plane R , p L and p R They are a pair of points with the same name. In a binocular camera, the left camera is generally used as the reference, and the pixel coordinate system of the right camera is calibrated and rotated to align with the camera coordinate system of the left camera. In this case, the left camera does not need to be rotated or translated, and only the right camera needs to be rotated and translated. The point P (x W ,y W ,z w ) can be combined to establish equations (1-3-2) and (1-3-3):

[0052]

[0053]

[0054] Combining equation (1-3-2) and equation (1-3-3) and simplifying them, we get equation (1-3-4):

[0055]

[0056] The M matrix is:

[0057]

[0058] The b matrix is:

[0059]

[0060] Solving equation (1-3-5) by the least squares method yields:

[0061] X=(M T M) -1 M T b (1-3-5)

[0062] The X solved in formula (1-3-3) is the world coordinate of the space point P. In the present invention, the coordinates of the key points can be solved, and the size of the object can be approximately calculated by solving the coordinates of multiple key points and then using the space Pythagorean theorem.

[0063] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0064] Experiment 1

[0065] This experiment will quantitatively illustrate the accuracy of the search for homonymous points in the present invention. Since the present invention searches for homonymous points in the right eye image based on the projection points of the left eye image, in order to illustrate the accuracy of the search for homonymous points, the searched homonymous points will be compared with the true value points in Experiment 1 to calculate their matching accuracy. The true value points are determined using the Harris corner point detection method. Harris corner point detection is performed on both the left eye image and the right eye image, and the corner points that appear in both the left eye image and the right eye image are regarded as a pair of projection points and homonymous points. Fig. 9 As shown, the corner points detected in the left image are used as projection points, and the corner points of the same angle detected in the right image are used as true value points. The pixel coordinates of the projection point and the true value point are extracted, and the distance from the projection point to the true value point is solved by Euclidean distance. The shorter the Euclidean distance, the more accurate the search for homonymous points. Substituting this Euclidean distance into the accuracy formula can quantitatively explain whether the search for homonymous points is accurate. The accuracy formula is shown in formula (2-1-1), p is the pixel coordinate of the search for homonymous points, T is the pixel coordinate of the true value point, and Euclidean is the Euclidean distance for solving p and T.

[0066]

[0067] In the experiment, tests were conducted on multiple pairs of Middlebury binocular images. Multiple corner points were tested on each pair of binocular images. The error distance was the Euclidean distance, and the accuracy was the result of converting the Euclidean distance through formula (2-1-1). The test results are shown in Table 1.

[0068] Table 1 Test results on the Middlebury dataset

[0069]

[0070] This experiment was also tested in a real scene in the laboratory. Fig.10 The test results are shown in Table 2.

[0071] Table 2 Test results in real scenarios

[0072]

[0073]

[0074] Experiment 2

[0075] In order to verify the accuracy of this algorithm, this experiment uses a distance measurement experiment compared with the traditional SGBM algorithm. The experimental environment is as follows: Fig.11 As shown, the binocular camera uses a 12 cm baseline ordinary binocular camera. The binocular camera is fixed, and the light receiving plate can be moved on the experimental table. The light receiving plate is placed at 0.3 meters, 0.5 meters, 1 meter, 1.2 meters and 1.5 meters away from the binocular camera for experiments.

[0076] When verifying the accuracy of the SGBM algorithm, 10 points are randomly selected in the light receiving plate image that appears in the depth map generated by the SGBM algorithm. The algorithm will output the three-dimensional world coordinates of these 10 points, and the average Z coordinate is taken as the distance measurement result of the SGBM algorithm at this distance and recorded. When verifying the accuracy of the algorithm proposed by the present invention, 10 point coordinates are randomly intercepted on the light receiving plate that appears in the left eye image of the binocular camera as key points, and the 10 point coordinates are respectively input into the algorithm to search for the same-name points in the right eye image to obtain the pixel coordinates of the same-name points, and the coordinates of the same-name points in pairs of left and right eyes are sent to the triangulation ranging algorithm to obtain the world coordinates of these 10 points, and the average Z coordinate is taken as the distance measurement result of this algorithm and recorded, as shown in Table 1. Table 3 shows that the measurement accuracy of this algorithm is higher than that of the SGBM algorithm.

[0077] Table 3 Ranging accuracy experimental results

[0078]

[0079] Experiment 3

[0080] This experiment will verify the running time of this algorithm, and also use it to compare with the SGBM algorithm for verification. Since the SGBM algorithm has a maximum parallax range parameter, it usually sets a relatively small parallax range when measuring points on distant objects to ensure the time efficiency of the algorithm; when measuring close objects, it usually sets a relatively large parallax range to ensure that the world coordinates of points on close objects can be successfully measured. As in the previous experiment, the binocular camera is fixed, and the light receiving plate can be moved on the experimental table. The light receiving plate is placed at 0.3 meters, 0.6 meters, 1 meter and 1.5 meters away from the binocular camera for experiments. Different distances between the light receiving plate and the camera are used to simulate different parallax ranges.

[0081] When verifying the efficiency of the SGBM algorithm, the left and right eye images are fed into the SGBM algorithm, and after the depth map is output, the depth value of a certain point on the light receiving plate is read and the world coordinates of the point are calculated in combination with the camera internal parameters, and the time for calculating the world coordinates is recorded. When verifying the algorithm of the present invention, the left and right eye images and the coordinates of a certain point on the light receiving plate are fed into the algorithm of the present invention, and the time for outputting the world coordinates is calculated and recorded, as shown in Table 4.

[0082] It can be seen from Table 4 that the time taken by this algorithm to calculate the world coordinates of a single point is at least 79% faster than that of the SGBM algorithm. At the same time, the time efficiency of the SGBM algorithm is limited by the parallax range. The larger the parallax range, the more time the SGBM algorithm takes. However, the algorithm of the present invention is not limited by the maximum parallax range. Regardless of the distance of the object from the camera, the calculation time it takes is basically the same.

[0083] Table 4 Algorithm running time experimental results

[0084] Serial number Parallax range The distance of the object from the camera SGBM algorithm Algorithm of the present invention Efficiency improvement ratio 1 0-50 1.5 m 0.129s 0.027s 79.1％ 2 0-100 1 meter 0.202s 0.026s 87.1％ 3 0-200 0.6m 0.241s 0.027s 88.8％ 4 0-300 0.3m 0.283s 0.028s 90.1％

[0085] The above is only a preferred implementation of the image stereo matching optimization method based on the search for same-name points disclosed in the present invention, and is not intended to limit the protection scope of the embodiments of this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of this specification shall be included in the protection scope of the embodiments of this specification.

Claims

1. An image stereo matching optimization method based on searching for points of the same name, characterized in that: The steps include: The images captured by the binocular camera are divided into left-eye image and right-eye image, and epipolar correction is performed on the left-eye image and the right-eye image respectively; Determine the spatial point to be measured, take the projection point of the spatial point on the left eye image as the key point, search on the left epipolar line of the right eye image according to the key point, and determine the same-name point corresponding to the key point; Based on the coordinates of the key points and the coordinates of the points with the same name, triangulation is performed to calculate the world coordinates of the spatial points, where: The method of searching on the left polar line of the right eye image according to the key point to determine the same-name point corresponding to the key point specifically includes the following steps: downsampling the left eye image and the right eye image by 16 times at the same time, calculating the Census cost for all pixel points in the search area along the left polar line in the downsampled right eye image based on the key point after downsampling, and taking the point with the smallest Census cost as the first candidate same-name point corresponding to the key point; based on the gradient encoding method of downsampling, screening the search area of ​​the first candidate same-name point at the original resolution to determine the same-name point; Based on the down-sampling gradient coding method, the search area of ​​the first candidate homonymous point at the original resolution is screened to determine the homonymous point, which specifically includes the following steps: determining the area corresponding to the first candidate homonymous point in the right eye image at the original resolution as the search area; based on the key point in the left eye image at the original resolution, taking the four pixels in the row corresponding to the polar line in the search area as the second candidate homonymous point; cutting out a 31*31 area around the key point in the original resolution and the key point is located in the center of the cut-out area to obtain the key area; cutting out a 31*31 area around the second candidate homonymous point in the original resolution and the four second candidate homonymous points are respectively located in the center of the corresponding cut-out area to obtain four comparison areas; down-sampling the key area and the four comparison areas by 16 times to a size of 7*7, and gradient coding the 7*7 areas of the key area and the four comparison areas; sequentially comparing the corresponding gradient coding arrays of the four comparison areas with the corresponding gradient coding array of the key area, and taking the second candidate homonymous point corresponding to the comparison area with the closest comparison result as the final homonymous point.

Images