Binocular distance measuring method with three layers of filling of parallax map

Abstract

The application discloses a binocular distance measuring method with three-layer filling of a parallax map, and is characterized in that the method comprises the following steps: 1) calibrating a binocular camera; 2) graying left and right pictures; 3) setting a parallax threshold t; 4) filling parallax values of E err ; 5) filling parallax values of B blo ; and 6) obtaining a depth map. The method processes different kinds of problems on the parallax map by using filling techniques respectively, and the processed parallax map retains details, the depth of each pixel point in the background exists, and conforms to a real image, and a target individual in the image is calculated to obtain a safe, effective and practical distance.

Description

TECHNICAL FIELD

[0001] The application relates to the field of computer vision, and particularly relates to a binocular distance measuring method with disparity layer filling. Figure Three TECHNICAL FIELD BACKGROUND

[0002] Computer vision aims to study visual behaviors such as identification, tracking and measurement by replacing human eyes with machines. With the development of machine vision technology, binocular distance measurement is an important algorithm technology. Compared with radar distance measurement, binocular camera distance measurement can obtain more real-time environmental information while being much cheaper.

[0003] In the binocular distance measurement process, noise points are inevitably generated. However, the existing disparity map post-processing method mainly smoothes the image, but the smoothed image is prone to the problem of blurred disparity, which will lead to the same depth of the rugged depth area, thereby seriously affecting the depth measurement. SUMMARY

[0004] The application aims at the shortage of the prior art and provides a binocular distance measuring method with disparity layer filling. Figure Three The method fills different types of problems in the disparity map by using filling technology respectively, the processed disparity map retains details, the depth of each pixel point in the background exists, and conforms to a real image. The target individual in the image is calculated to obtain a safe, effective and practical distance.

[0005] The technical scheme for realizing the application is as follows:

[0006] A binocular distance measuring method with disparity layer filling, comprising the following steps: Figure Three

[0007] 1) Calibrating the binocular camera: collecting left and right pictures by using the camera, and performing stereoscopic correction and distortion correction on the pictures, then performing instance segmentation on the left picture, and recording the pixel point positions covered by n instances respectively, using T to represent the coordinate set of the n instances:

[0008] T={T 1, T 2, …,T n}(1),

[0009] T i ={(x1,y1),(x2,y2),...,(x num ,y num )}(2),

[0010] Wherein, num is the number of pixel points contained in the i-th instance, and (x, y) is the coordinate of the pixel point contained in the i-th instance. ​

[0011] 2) Grayscale left and right pictures: After grayscale processing, the left and right pictures are globally stereo matched to obtain a disparity map. The disparity values in each instance region are unified, and the second maximum value in the instance region is assigned as the disparity value of the entire instance. The second maximum value is selected to replace all disparity values in the instance region by traversing all disparity values of the instance:

[0012] d sec =sec(δ(T i ))(3),

[0013] δ(T i )={δ(x1,y1),δ(x2,y2),...,δ(x num ,y num )}(4),

[0014] where δ(x,y) is the disparity value of the pixel point, δ(T i ) is the disparity value of all pixel points of the i-th instance, and sec function refers to the second maximum value;

[0015] 3) Set the disparity threshold t: judge the hole and classify it into the mismatched region E err and the occluded region B blo . The disparity is substituted into the depth formula as shown in formula (5):

[0016] depth=f*b / (b-d)(5),

[0017] where f is the focal length, b is the baseline length, and d is the disparity of the instance. The minimum value of depth is substituted into the depth back-propagation disparity formula (6):

[0018] d=b*(depth-f) / depth(6),

[0019] The obtained disparity d is the disparity threshold t. When classifying the hole, the hole region is regarded as a row of horizontal pixels. In each row, the disparity values of the first depth pixel on the left and the first depth pixel on the right are compared. If the difference between them is greater than t, the entire region is regarded as B blo . If the difference between them is less than or equal to t, it is regarded as E err .

[0020] 4) Fill the disparity value of E err : Make a flat rectangle that can accommodate E err at E err . Take the four corner points of the rectangle as known points and fill the disparity value of E err using bilinear interpolation method.

[0021] 5) Fill B blodisparity value: in B blo Fill in the disparity values ​​of the more distant areas on the left and right sides, specifically by: [filling in B]. blo The region is considered as rows of horizontal pixels, as shown in formula (7):

[0022] B blo,yi ={(x1,y i (x2,y) i ) … (x m ,y i )}(7)

[0023] Among them B blo,yi For B blo The set of pixels in the i-th row of the region, where m is B. blo The number of pixels in this row

[0024] Compare δ(x1-1, y) i ) and δ(x m +1, y i Assign the smaller disparity value to the hole pixels in the Blo row, as shown in formula (8):

[0025] δ(B blo,yi )=min( δ(x1-1, y i ), δ(x) m +1, y i ) )(8)

[0026] Where, (x1-1, y i ) is B blo,yi The first pixel on the left has depth, δ(x1-1, y). i (x) represents the disparity value of that pixel; m +1, y i ) is B blo,yi The first pixel on the right has depth, δ(x) m +1, y i ) represents the disparity value of that pixel;

[0027] 6) Obtain the depth map: Substitute the processed disparity map into the depth calculation formula to obtain the depth map. The information in the depth map image includes the distance of each instance and the pixel-level depth at the background. The depth calculation formula is shown in formula (9):

[0028] depth=f*b / (bd)(9)

[0029] Where f is the focal length, b is the baseline length, and d is the disparity value of a pixel.

[0030] The technical scheme firstly distinguishes the target and the background in the picture, takes the second maximum value of the parallax value of the target for the convenience of subsequent target ranging, can also avoid accidental collision and ensure the safety distance, and can also avoid the accidental phenomenon that the maximum value is too extreme and leads to inaccurate ranging, so that the target can measure the safe, effective and practical distance; the hollow area of the parallax map is classified into a connected area and an occluded area, the parallax of the connected area is filled by using a bilinear interpolation method, and the parallax of the occluded area is filled with the parallax of the side with a smaller value, so that the depth of the background pixels is closer to the real world.

[0031] Compared with the prior art, the technical scheme can restore the real world depth in detail on the background environment and can measure the safe, effective and practical distance of the target without reducing the image quality, and greatly improves the efficiency and practicability of binocular ranging. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 A model schematic diagram for binocular camera calibration in the embodiment;

[0033] Figure 2 A schematic diagram of imaging of a point in the physical world in the left and right cameras of the binocular camera in the embodiment;

[0034] Figure 3 A flow block diagram of the working principle of binocular ranging in the embodiment;

[0035] Figure 4 A model schematic diagram of instance segmentation in the embodiment.

[0036] Figure 2 In the formula, b is the distance between the two center points of the left and right cameras, that is, the baseline length; d is the distance between the two imaging points of the left and right cameras after the coordinate point in the physical world is imaged on the mirror surface of the left and right cameras, that is, the parallax; f is the distance from the mirror to the center point, that is, the focal length; and depth is the distance between the coordinate point in the physical world and the center point of the camera, that is, the depth to be calculated. IMPLEMENTATION

[0037] The content of the application will be further described below in combination with the drawings and embodiments, but is not a limitation on the application. EMBODIMENT

[0038] REFERENCE Figure 3 A binocular ranging method with parallax layer filling, comprising the following steps: Figure Three

[0039] 1) Calibrate the binocular camera: collect left and right pictures by using the camera and perform stereoscopic correction and distortion correction on the pictures, then perform instance segmentation on the left picture, record the pixel point positions covered by n instances respectively, and use T to represent the coordinate set of the n instances: ​

[0040] T={T 1, T 2, …,T n}(1),

[0041] T i ={(x1,y1),(x2,y2),...,(x num ,y num )}(2),

[0042] Wherein, num is the number of pixel points contained in the i-th instance, (x, y) is the coordinate of the pixel point contained in the i-th instance;

[0043] 2) Grayscale left and right pictures: the global stereo matching of the grayscale left and right pictures is performed to obtain a disparity map, the disparity values in each instance region are unified, the second maximum value in the instance region is assigned as the disparity value of the entire instance, and the second maximum value is selected to replace all disparity values in the instance region by traversing all disparity values of the instance:

[0044] d sec =sec(δ(T i ))(3),

[0045] δ(T i )={δ(x1,y1),δ(x2,y2),...,δ(x num ,y num )}(4),

[0046] Wherein, δ(x, y) is the disparity value of the pixel point, δ(T i ) is the disparity value of all pixel points of the i-th instance, and sec function refers to the second maximum value;

[0047] 3) Set the disparity threshold t: judge the empty hole, classify it into the mismatched region E err and the occluded region B blo , and the disparity is substituted into the depth formula as shown in formula (5):

[0048] depth=f*b / (b-d)(5),

[0049] Wherein, f is the focal length, b is the baseline length, and d is the disparity of the instance. The 1 / 15th of the minimum value of depth is substituted into the depth back-propagation disparity formula (6):

[0050] d=b*(depth-f) / depth(6),

[0051] The obtained disparity d is the disparity threshold t. During hole classification, the area at the hole is treated as rows of horizontal pixels. Within each row, the disparity values ​​of the first pixel with depth on the left and the first pixel with depth on the right are compared. If the difference between the two is greater than t, the entire area is considered as B. blo If the difference between the two is less than or equal to t, then it is considered E. err ;

[0052] 4) Fill in E err disparity value: in E err The place can just accommodate E err Given a flat rectangle, fill E using bilinear interpolation with its four corner points as known points. err disparity value;

[0053] 5) Fill in B blo disparity value: in B blo Fill in the disparity values ​​of the more distant areas on the left and right sides, specifically by: [filling in B]. blo The region is considered as rows of horizontal pixels, as shown in formula (7):

[0054] B blo,yi ={(x1,y i (x2,y) i ) … (x m ,y i )}(7)

[0055] Among them B blo,yi For B blo The set of pixels in the i-th row of the region, where m is B. blo The number of pixels in this row

[0056] Compare δ(x1-1, y) i ) and δ(x m +1, y i Assign the smaller disparity value to the hole pixels in the Blo row, as shown in formula (8):

[0057] δ(B blo,yi )=min( δ(x1-1, y i ), δ(x) m +1, y i ) )(8)

[0058] Where, (x1-1, y i ) is B blo,yi The first pixel on the left has depth, δ(x1-1, y). i (x) represents the disparity value of that pixel; m +1, y i ) is B blo,yiThe first pixel on the right has a depth pixel, δ(x m +1, y i ) is the disparity value of the pixel;

[0059] 6) Obtain the depth map: substitute the processed disparity map into the depth calculation formula to obtain the depth map. The information in the depth map picture includes the distance of each instance and the depth of the pixel level in the background. The depth calculation formula is shown in formula (9):

[0060] depth=f*b / (b-d) (9),

[0061] wherein f is the focal length, b is the baseline length, and d is the disparity value of the pixel.

[0062] In this example, as shown in Figure 1 , it is extremely critical to collect the original view image with good effect in the binocular distance measurement process. The calibration process of the binocular camera is referred to Figure 1 , and the principle of obtaining the disparity map by binocular distance measurement is referred to Figure 2 . According to the deepened part of the triangle in the figure, the following similar triangle relationship can be obtained:

[0063] (depth-f) / depth=d / b (1),

[0064] It can be derived that:

[0065] depth=f*b / (b-d) (2),

[0066] depth is the distance to be obtained.

[0067] In the process of measuring distance by using the binocular camera in this example, since the camera is a convex mirror, there will be a deviation in the light mapping imaging, and the image will be distorted. Therefore, the binocular camera needs to obtain the internal and external parameters and the distortion coefficient through calibration to correct the collected pictures.

[0068] In order to measure the safe and accurate distance of the target, first, instance segmentation is performed on the Figure 4 left image. The segmentation process is referred to Figure 4 . Different target individuals are distinguished as instances, and the pixel coordinates contained in each instance T are recorded. After obtaining the preliminary disparity map through stereo matching of the left and right images, the second maximum value d i is filled in δ(T sec ) to fill the entire T i ;

[0069] Stereo matching will inevitably lead to the generation of noise points. In order to filter out the influence of noise points on distance measurement, the processing method of this example is to fill the disparity in the hollow area. The hollow area in the original disparity map can be classified into a mismatching area E err and a blocking area Bblo E err The characteristics are small area, high parallax connectivity, and B blo The characteristics are a slightly larger area and disparity values ​​on the left and right sides. The distance obtained by directly calculating the depth of the disparity map for these two types of void areas will have a large error.

[0070] To distinguish between these two different feature regions, this example sets a disparity threshold t to differentiate E. err and B blo The formula for calculating t is shown in formula (3):

[0071] t=b*(depth-f) / depth(3)

[0072] Where b is the baseline length, f is the camera focal length, and depth is 1 / 15 of the depth of the nearest target.

[0073] When classifying holes, the area at the hole is treated as rows of horizontal pixels. Within each row, the disparity values ​​of the first pixel with depth on the left and the first pixel with depth on the right are compared. If the difference between the two is greater than t, the entire area is considered B. blo If the difference between the two is less than or equal to t, then it is considered E. err ;

[0074] To fill E err In this example, bilinear interpolation is used to fill the disparity values ​​of a region. Given its small gaps and real-world connectivity, bilinear interpolation can accurately reproduce the real-world depth, making it a suitable method for filling the E region. err Appropriate methods for depth;

[0075] To fill B blo In this example, region B blo The region is considered as rows of horizontal pixels, as shown in formula (4):

[0076] B blo,yi ={(x1,y i (x2,y) i … (x) m ,y i )}(4)

[0077] Among them B blo,yi For B blo The set of pixels in the i-th row of the region, where m is B. blo The number of pixels in this row

[0078] Compare δ(x1-1, y) i ) and δ(x m +1, y iAssign the smaller disparity value to the hole pixels in the Blo row, as shown in formula (5):

[0079] δ(B blo,yi )=min( δ(x1-1, y i ), δ(x) m +1, y i ) )(5)

[0080] Where, (x1-1, y i ) is B blo,yi The first pixel on the left has depth, δ(x1-1, y). i (x) represents the disparity value of that pixel; m +1, y i ) is B blo,yi The first pixel on the right has depth, δ(x) m +1, y i ) represents the disparity value of that pixel;

[0081] For B blo Its area is slightly larger and it has the characteristics of front and rear occlusion. Since the occluded part is farther away, according to the characteristic that the parallax is smaller at a greater distance, in this example, in B blo Filling with smaller disparity values ​​effectively restores the distance to the occluded area; in this example, the method is to fill with B. blo Appropriate methods for depth;

[0082] Finally, the depth calculation formula is applied to the processed disparity map, as shown in formula (6):

[0083] depth=f*b / (bd)(6)

[0084] Where f is the focal length, b is the baseline length, d is the disparity of the pixel, and the resulting depth is the depth of the pixel. Formula (6) can convert a disparity map into a depth map. Each pixel in the background of the image can reflect the depth of the real world, and each individual in the target area has a practical and unique depth.

Claims

1. A binocular range finding method with disparity map three-layer filling, characterized in that, Comprising the following steps: 1) Binocular camera calibration: left and right pictures are collected by the camera and stereoscopic correction and distortion correction are performed on the pictures, then the left picture is instance segmented, and the pixel point positions covered by n instances are recorded, and T represents the coordinate set of the n instances: T={T 1, T 2, …,T n} (1), T i {(x1,y1),(x2,y2),...,(x num ,y num )}(2), Wherein, num is the number of pixel points contained in the i-th instance, and (x, y) is the coordinate of the pixel point contained in the i-th instance. 2) Grayscale left and right pictures: the grayscale left and right pictures are globally stereomatched to obtain a disparity map, the disparity values in each instance region are unified, the second maximum value in the instance region is assigned as the disparity value of the entire instance, the entire disparity value of the instance is traversed, and the second maximum value is selected to replace all disparity values in the instance region: d sec =sec(δ(T i ))(3), δ(T i ) = { δ(x1,y1), δ(x2,y2),..., δ(x num ,y num )} (4), wherein, δ(x, y) is the disparity value of the pixel point, δ(T i ) is the disparity value of all pixel points of the i-th instance, and the sec function refers to the second maximum value. 3) Set disparity threshold t: judge the hole, and classify as a mismatch region E err and the occlusion region B blo The disparity is substituted into the depth formula as shown in equation (5): depth = f * b / (b - d) (5), Wherein, f is the focal length, b is the baseline length, and d is the disparity of the instance. The 1 / 15th of the minimum depth value is substituted into the depth back-propagation disparity formula (6): d = b * (depth - f) / depth (6), The obtained disparity d is the disparity threshold t. In the hole classification, the region of the hole is regarded as a row of horizontal pixels. In each row, the disparity values of the first depth pixel on the left and the first depth pixel on the right are compared. If the difference between the two is greater than t, the entire region is regarded as B blo , and if the difference between the two is less than or equal to t, the entire region is regarded as E err . 4) fill E err the disparity value of E at the four corners of the rectangle, and fill the disparity value of E by bilinear interpolation. err err the disparity value of E at the four corners of the rectangle, and fill the disparity value of E by bilinear interpolation. err the disparity value of E at the four corners of the rectangle, and fill the disparity value of E by bilinear interpolation.​ 5) fill B blo The disparity value at B blo is filled with the disparity value of the farther one of the upper left and right, specifically: the B blo region is treated as a row of horizontal pixels, as shown in equation (7): B blo,yi {(x1,y i ) (x2,y i ) … (x m ,y i )}(7), where B blo,yi is B blo a set of pixels in the i-th row of the region, m is B blo the number of pixels in the row, Compare δ(x1-1, y) i ) and δ(x m +1, y i Assign the smaller disparity value to the hole pixels in the Blo row, as shown in formula (8): δ(B blo,yi )=min( δ(x1-1, y i ), δ(x m +1, y i ) )(8), Where, (x1-1, y i ) is B blo,yi The first pixel on the left has depth, δ(x1-1, y). i (x) represents the disparity value of that pixel; m +1, y i ) is B blo,yi The first pixel on the right has depth, δ(x) m +1, y i ) represents the disparity value of that pixel; 6) Obtain the depth map: substitute the processed disparity map into the depth calculation formula to obtain the depth map. The information in the depth map picture includes the distance of each instance and the pixel-level depth of the background. The depth calculation formula is shown in formula (9): depth = f * b / (b - d) (9), Wherein, f is the focal length, b is the baseline length, and d is the pixel point disparity value.

Images