Associated region feature assisted binocular vision depth estimation method

Abstract

The invention discloses a correlated region feature assisted binocular vision depth estimation method, which comprises the following steps of: 1, matching geometric shapes of object images in left and right views through barycentric coordinates and rotational inertia of the geometric shapes of the object images; 2, through intersection of a key point connecting line on the geometric shape contour of the image and an epipolar line, obtaining a search range of a minimum scale matched with imaging points on the epipolar line of the left and right views; 3, constructing geometric figures of imaging points and key points on the geometric shape contour of the image, calculating the rotational inertia of constructing the geometric figures, matching the imaging points in the left and right views, and calculating the parallax and depth of the matched binocular imaging points; according to the invention, through subdividing the geometric shape area of the image, the imaging point matching range of parallax calculation is reduced, and the binocular matching efficiency is improved; according to the method, the associated region features are used for assisting parallax calculation, and missing of matching feature information of poor regions such as shielding, weak texture and exposure is made up, so that the parallax calculation accuracy is enhanced, and the binocular stereo matching precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a binocular vision depth estimation method assisted by associated region features. Background Art

[0002] Binocular vision measurement involves application fields such as robot navigation, unmanned driving, medical imaging, and virtual reality. The existing two-dimensional image information is difficult to meet the actual needs in the application field, and high-precision three-dimensional depth information of target objects and scenes is urgently needed. Compared with measurement methods such as laser radar and structured light, passive binocular vision measurement technology with low cost, easy deployment, and strong adaptability can better meet the actual application requirements of many target objects and scenes. Stereo matching algorithm is an important technology for obtaining three-dimensional depth information from two-dimensional images, which greatly affects the accuracy of binocular vision three-dimensional depth measurement.

[0003] Traditional stereo matching algorithms use features such as the color of image pixels to measure the correlation of pixels, match imaging points in the image, calculate the disparity values ​​of imaging points, and obtain the depth of binocular imaging points. However, there is a large matching ambiguity in poor areas such as occlusion, weak texture, and exposure, making it difficult to produce high-quality disparity, which affects the accuracy of depth measurement. End-to-end stereo matching methods use deep convolutional neural networks to extract effective feature information such as lighting and color changes in poor image areas to achieve better accuracy. However, convolutional neural networks lose detail information during downsampling and multi-scale fusion, resulting in blurred edge disparity, limited matching accuracy and convergence speed, and other problems. In addition, the training of complex deep convolutional network structures requires a large amount of data sets and high hardware resources, which limits the practical application of binocular vision measurement technology. Summary of the invention

[0004] In view of the above-mentioned problems or shortcomings, the present invention provides a binocular visual depth estimation method assisted by associated area features, in order to reduce the matching ambiguity of poor areas such as weak texture, exposure, occlusion, etc., to improve the quality of the disparity map, thereby enhancing the binocular stereo matching accuracy and improving the accuracy of the depth measurement of the target object.

[0005] In order to achieve the above-mentioned object of the invention, the present invention adopts the following technical solution: The feature of the binocular vision depth estimation method assisted by the associated region feature of the present invention is that it comprises the following steps:

[0006] Step 1: Use a binocular vision camera to obtain the left view and right view ;

[0007] Step 2: Left view and right view Perform epipolar correction and grayscale processing respectively, and obtain the left grayscale image accordingly And right grayscale image ;

[0008] Step 3: Extract the left grayscale image The geometric outlines in And its corresponding geometric area is ,in, express The A geometric shape outline, express The corresponding geometric area;

[0009] Extract the right grayscale image The geometric outlines in And its corresponding geometric area is ,in, express The A geometric shape outline, express The corresponding geometric area, is the number of geometric shape contours, ;

[0010] Step 4: Calculate using formula (1) Around its center of gravity Moment of inertia :

[0011] (1)

[0012] In formula (1), for Middle The coordinates of the pixels, express The total number of pixels in express The centroid coordinates of Indicates The quality of the pixel, that is, The gray value of each pixel;

[0013] Step 5: Calculate using formula (2) Around its center of gravity Moment of inertia :

[0014] (2)

[0015] In formula (2), for Middle The coordinates of the pixels, express The total number of pixels in express The centroid coordinates of To indicate the The quality of the pixel, that is, The gray value of each pixel;

[0016] Step 6: Use formula (3) to obtain the left view and right view The matching value of the corresponding geometric shape area :

[0017] (3)

[0018] In formula (3), is the weight coefficient of the moment of inertia, is the weight coefficient of the center of gravity;

[0019] Step 7: Use formula (4) to calculate Any Points The radius of the osculating circle ;

[0020] (4)

[0021] In formula (4), For the Points The length of the line connecting it to its left adjacent point, For the Points The length of the line connecting it to its right adjacent point, For the Points The length of the line between the left and right adjacent points of For the Points The area of ​​the triangle enclosed by its left and right adjacent points;

[0022] According to formula (4), Any Points The radius of the osculating circle ;

[0023] Step 8: Set the left threshold ,when Time, point for On the key point, we can obtain All key points ,in, is the number of key points;

[0024] Step 9: Set the right threshold ,when Time, point for On the key point, we can obtain All key points ;

[0025] Step 10: Extract using ORB algorithm and The features are matched to obtain key point matching pairs ,in, Represents any key point matching pair;

[0026] Step 11 No. Imaging points , At its extreme The upper position is , thus obtaining the epipolar line Upper position exist The corresponding point in ;

[0027] Step 12: Connect the M key points in pairs and connect the connecting lines with the polar lines The intersection point is recorded as ;in, Indicates intersection points, Indicates the total number of intersections;

[0028] Step 13: If There is an intersection in With imaging point If they overlap, then take The corresponding two end points of the line exist The corresponding key points ,Will The connection and polar line of exist The intersection point in is denoted as , then the intersection That is the Imaging points exist The corresponding imaging point in the image is obtained; and step 18 and step 19 are executed, otherwise, step 14 is executed;

[0029] Step 14: If There exists a point at the imaging point The intersection point on the left is taken as the image point The nearest left intersection The two end points of the corresponding line exist The corresponding key points ,Will The connection and polar line of exist The intersection point in is denoted as , and and The points between Waiting for matching point ,in, Represents polar line superior to Any pixel between For point Solstice The number of pixels;

[0030] like There is no image point in the The intersection point on the left side takes the geometric shape outline With polar line The left point of the intersection is denoted by , thus obtaining middle and The points to be matched between ; Take the distance imaging point Nearest right intersection , The two end points of the line connecting the points are denoted by , and in In, take and Match the corresponding key points, denoted as ;

[0031] Step 15: The points to be matched Respectively and key points Connect, form Right triangles to be matched , and calculate the moment of inertia of each right triangle about its center of gravity ; in, Indicate point , , Surrounded by right triangles to be matched, express moment of inertia about its center of gravity;

[0032] Step 16: Place the imaging point The two endpoints of the corresponding line Forming the base left triangle , and calculate Moment of inertia about its center of gravity ;

[0033] Step 17: Use formula (5) to obtain Middle imaging point exist The corresponding imaging point Location :

[0034] (5)

[0035] Step 18: According to the imaging point In right view The horizontal axis in , use formula (6) to calculate the disparity value ,

[0036] (6)

[0037] Step 19: Based on the parallax value , use formula (7) to calculate the imaging point With imaging point Depth ,

[0038] (7)

[0039] In formula (7), is the focal length of the binocular vision camera, is the optical center distance of the binocular vision camera.

[0040] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the binocular vision depth estimation method, and the processor is configured to execute the program stored in the memory.

[0041] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the binocular vision depth estimation method when the computer program is executed by a processor.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. The present invention matches the geometric shapes of the object images in the left and right views through the centroid coordinates and moment of inertia of the geometric shapes of the object images, and subdivides the image geometric shape area by intersecting the key point connection line on the image geometric shape contour with the polar line, thereby narrowing the search range of the disparity calculation imaging point matching and improving the binocular matching efficiency.

[0044] 2. The present invention constructs geometric figures of imaging points and key points on the contour of the image geometric shape, calculates the moment of inertia of the constructed geometric figures, compensates for the missing matching feature information in the bad area, assists in the parallax calculation of the imaging points in the left and right views, increases the accuracy of the parallax calculation, and improves the accuracy of binocular stereo matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of the process of binocular vision depth estimation method assisted by associated region features;

[0046] Figure 2 It is a schematic diagram of the intersection of the key point connecting line and the polar line and the coincidence of the imaging point in the left grayscale image;

[0047] Figure 3 The schematic diagram is that the intersection point of the key point connection line and the intersecting polar line is located on the left side of the imaging point in the left grayscale image;

[0048] Figure 4 This is a schematic diagram showing that the intersection of the key point connecting line and the polar line is not located to the left of the imaging point in the left grayscale image. DETAILED DESCRIPTION

[0049] In this embodiment, a binocular vision depth estimation method assisted by associated region features utilizes associated region features to assist in calculating imaging point disparity and guide subsequent depth measurement, thereby reducing the imaging point matching range of disparity calculation and improving binocular matching efficiency; compensating for the missing matching feature information of poor areas such as occlusion, weak texture, and exposure, and improving binocular stereo matching accuracy. Specifically, Figure 1 As shown, the method comprises the following steps:

[0050] Step 1: Use a binocular vision camera to obtain the left view and right view ;

[0051] Step 2: Use Fusiello correction method to correct the left view and right view Perform epipolar correction and grayscale processing to obtain the left grayscale image And right grayscale image ;

[0052] Step 3: Extract the left grayscale image The geometric outlines in And its corresponding geometric area is ,in, express The A geometric shape outline, express The corresponding geometric area;

[0053] Extract the right grayscale image The geometric outlines in And its corresponding geometric area is ,in, express The A geometric shape outline, express The corresponding geometric area, is the number of geometric shape contours, .

[0054] Step 4: Calculation No. Step moment , No. Step moment , No. Step moment ,in ,

[0055] (1)

[0056] In formula (1), for Middle The coordinates of the pixels, express The total number of pixels in Indicates The quality of the pixel, that is, The gray value of a pixel.

[0057] Step 5: Calculation The centroid coordinates of ,

[0058] (2)

[0059] Step 6. Calculation Around its center of gravity Moment of inertia ,

[0060] (3)

[0061] Step 7: Calculation No. Step moment , No. Step moment , No. Step moment ,in ,

[0062] (4)

[0063] In formula (4), for Middle The coordinates of the pixels, express The total number of pixels in To indicate the The quality of the pixel, that is, The gray value of a pixel.

[0064] Step 8. Calculation The centroid coordinates of ,

[0065] (5)

[0066] Step 9. Calculation Around its center of gravity Moment of inertia ,

[0067] (6)

[0068] Step 10: Construction and Similarity determination function ,

[0069] (7)

[0070] Formula (7) is is the weight coefficient of the moment of inertia, is the weight coefficient of the center of gravity.

[0071] Step 11: Use formula (8) to obtain the left view The geometric area in and right view Medium geometry area The matching value of :

[0072] (8)

[0073] Step 12: Calculate using formula (9) Any Points The radius of the osculating circle :

[0074] (9)

[0075] In formula (9), For the Points The length of the line connecting it to its left adjacent point, For the Points The length of the line connecting it to its right adjacent point, For the Points The length of the line between the left and right adjacent points of For the Points The area of ​​the triangle enclosed by its left and right adjacent points.

[0076] Step 13: Calculate using formula (9) Any Points The radius of the osculating circle :

[0077] (10)

[0078] In formula (10), For the Points The length of the line connecting it to its left adjacent point, For the Points The length of the line connecting it to its right adjacent point, For the Points The length of the line between the left and right adjacent points of For the Points The area of ​​the triangle enclosed by its left and right adjacent points.

[0079] Step 14: Set the left threshold ,when Time, point for On the key point, we can obtain All key points ,in, is the number of key points;

[0080] Step 15: Set the right threshold ,when Time, point for On the key point, we can obtain All key points ;

[0081] Step 16: Extract using ORB algorithm and The features are matched to obtain key point matching pairs ,in, Represents any key point matching pair;

[0082] Step 17 No. Imaging points , At its extreme The upper position is , thus obtaining the epipolar line Upper position exist The corresponding point in ;

[0083] Step 18: middle Connect the key points in pairs and connect the lines with the polar lines The intersection point is recorded as ;in, Indicates intersection points, Indicates the number of intersections.

[0084] Step 19: If There is an intersection in With imaging point Overlap, such as Figure 2 As shown, in Winning The two endpoints of the line connecting the points ;exist In, with Match the corresponding key points , Connection Central Line The intersection point is recorded as , then the intersection That is the Imaging points exist The corresponding imaging point in the image is obtained; and step 25 and step 26 are executed, otherwise, step 20 is executed;

[0085] Step 20: If There exists a point at the imaging point The intersection point on the left, such as Figure 3 As shown, take the distance imaging point Nearest left intersection , The two end points of the line connecting the points are ;

[0086] exist In, take and Match the corresponding key points , and connect Central Line At point , and and The points between Waiting for matching point ,in, Represents polar line superior to The pixels between For point Solstice The number of pixels;

[0087] like There is no image point in the The intersection point on the left, such as Figure 4 As shown, the geometric shape contour is taken With polar line The left point of the intersection is denoted by , thus obtaining middle and Points to be matched ;

[0088] Step 21: Get the distance imaging point The nearest right intersection point is denoted as , The two end points of the line connecting the points are denoted by ; and In, take and Match the corresponding key points, denoted as ;

[0089] Step 22: The points to be matched Respectively and key points Connect, form Right triangles to be matched , and calculate the moment of inertia of each right triangle about its center of gravity ; in, Indicate point , , A right triangle to be matched is formed. express moment of inertia about its center of gravity;

[0090] Step 23: Place the imaging point The two endpoints of the corresponding line Forming the base left triangle , and calculate Moment of inertia about its center of gravity ;

[0091] Step 24: Use formula (11) to obtain Middle imaging point exist The corresponding imaging point Location :

[0092] (11)

[0093] Step 25: According to the imaging point In right view The horizontal axis in , use formula (12) to calculate the disparity value :

[0094] (12)

[0095] Step 26: According to the parallax value , use formula (13) to calculate the imaging point With imaging point Depth :

[0096] (13)

[0097] In formula (13), is the focal length of the binocular vision camera, is the optical center distance of the binocular vision camera.

[0098] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0099] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

[0100] In summary, the present invention uses the centroid coordinates and moment of inertia of the object image geometry to match the object image geometry in the left and right views, subdivide the image geometry area, narrow the matching search range of the disparity calculation imaging point, and improve the binocular matching efficiency; constructs the geometric figures of the imaging points and the key points on the image geometry contour, uses the associated area features to assist the matching of the imaging points in the left and right views, compensates for the missing feature information of the bad areas such as occlusion, weak texture, exposure, etc., enhances the accuracy of the disparity calculation, and improves the binocular matching accuracy.

Claims

1. A binocular vision depth estimation method assisted by associated region features, characterized in that: The steps include: Step 1: Use a binocular vision camera to obtain the left view and right view ; Step 2: Left view and right view Perform epipolar correction and grayscale processing respectively, and obtain the left grayscale image accordingly And right grayscale image ; Step 3: Extract the left grayscale image Geometric outlines in And its corresponding geometric area is ,in, express The A geometric shape outline, express The corresponding geometric area; Extract the right grayscale image Geometric outlines in And its corresponding geometric area is ,in, express The A geometric shape outline, express The corresponding geometric area, is the number of geometric shape contours, ; Step 4: Calculate using formula (1) Around its center of gravity Moment of inertia : (1) In formula (1), for Middle The coordinates of the pixels, express The total number of pixels in express The centroid coordinates of Indicates The quality of the pixel, that is, The gray value of each pixel; Step 5: Calculate using formula (2) Around its center of gravity Moment of inertia : (2) In formula (2), for Middle The coordinates of the pixels, express The total number of pixels in express The centroid coordinates of To indicate the The quality of the pixel, that is, The gray value of each pixel; Step 6: Use formula (3) to obtain the left view and right view The matching value of the corresponding geometric shape area : (3) In formula (3), is the weight coefficient of the moment of inertia, is the weight coefficient of the center of gravity; Step 7: Use formula (4) to calculate Any Points The radius of the osculating circle ; (4) In formula (4), For the Points The length of the line connecting it to its left adjacent point, For the Points The length of the line connecting its right adjacent point, For the Points The length of the line between the left and right adjacent points of For the Points The area of ​​the triangle enclosed by its left and right adjacent points; According to formula (4), Any Points The radius of the osculating circle ; Step 8: Set the left threshold ,when Time, point for On the key point, we can obtain All key points ,in, is the number of key points; Step 9: Set the right threshold ,when Time, point for On the key point, we can obtain All key points ; Step 10: Extract using ORB algorithm and The features are matched to obtain key point matching pairs ,in, Represents any key point matching pair; Step 11 No. Imaging points , At its extreme The upper position is , thus obtaining the epipolar line Upper position exist The corresponding point in ; Step 12: Connect the M key points in pairs and connect the connecting lines with the polar lines The intersection point is recorded as ;in, Indicates intersection points, Indicates the total number of intersections; Step 13: If There is an intersection in With imaging point If they overlap, then take The corresponding two end points of the line exist The corresponding key points ,Will The connection and polar line of exist The intersection point in is denoted as , then the intersection That is the Imaging points exist The corresponding imaging point in the image is obtained; and step 18 and step 19 are executed, otherwise, step 14 is executed; Step 14: If There exists a point at the imaging point The intersection point on the left is taken as the image point The nearest left intersection The two end points of the corresponding line exist The corresponding key points ,Will The connection and polar line of exist The intersection point in is denoted as , and and The points between Waiting for matching point ,in, Represents polar line superior to Any pixel between For point Solstice The number of pixels; like There is no image point in the The intersection point on the left side takes the geometric shape outline With polar line The left point of the intersection is denoted by , thus obtaining middle and The points to be matched between ; Take the distance imaging point Nearest right intersection , The two end points of the line connecting the points are denoted by , and in In, take and Match the corresponding key points, denoted as ; Step 15: The points to be matched Respectively and key points Connect, form Right triangles to be matched , and calculate the moment of inertia of each right triangle about its center of gravity ; in, Indicate point , , Surrounded by There are right triangles to be matched. express moment of inertia about its center of gravity; Step 16: Place the imaging point The two endpoints of the corresponding line Forming the base left triangle , and calculate Moment of inertia about its center of gravity ; Step 17: Use formula (5) to obtain Middle imaging point exist The corresponding imaging point Location : (5) Step 18: According to the imaging point In right view The horizontal axis in , use formula (6) to calculate the disparity value , (6) Step 19: Based on the parallax value , use formula (7) to calculate the imaging point With imaging point Depth , (7) In formula (7), is the focal length of the binocular vision camera, is the optical center distance of the binocular vision camera.

2. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the binocular vision depth estimation method according to claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the binocular visual depth estimation method according to claim 1 are executed.

Images