Indoor free space estimation and obstacle detection method based on UV disparity map

Abstract

The present invention discloses a method for indoor free space estimation and obstacle detection based on a UV-disparity map. The method comprises: improving the UV-disparity method by using a sliding window method to divide the disparity map into several sub-windows, each of which generates a sub-V-disparity map. Within the window, a piecewise linear fitting method is used to sequentially connect multiple extracted line segments to obtain ground-related lines; a new V-disparity map filtering method is used to filter the sub-V-disparity map using an edge filter, multiplying the filtered image with the original V-disparity map using the weighted coefficient, and then filtering the V-disparity filtered image using a maximum filter and a block filter for road detection; and integrating the V-disparity calculation results into a probability estimation framework for ground estimation. The ground confidence map (RCM) and the obstacle confidence map (OCM) are weighted and accumulated to form the final confidence map model. The free space is then optimized and estimated using a two-dimensional dynamic programming method based on the confidence map.

Description

Technical Field

[0001] The present invention belongs to the fields of image processing and computer vision, and relates to an indoor free space estimation and obstacle detection method based on a UV disparity map. Background Art

[0002] Humans have long hoped to emulate the human visual system to empower machines to automatically identify and detect objects and environments, thereby helping them complete various tasks. Indoor mobile robots are playing an increasingly important role in our lives. Therefore, enabling machines to recognize objects is a significant and challenging international challenge. Image acquisition is essentially a two-dimensional projection of the three-dimensional world. Traditional object recognition is mostly based on monocular color images, which are susceptible to variations in texture and lighting, occlusion, and perspective. In binocular stereo vision, however, a binocular camera simulates the left and right eyes of a person, capturing the same object from two different viewing angles to produce two images. By incorporating depth information through triangulation and specialized algorithms, this method overcomes many of the shortcomings of traditional color images.

[0003] Free space refers to the area of ​​space in a scene that can be safely explored without collision. Generally speaking, free space estimation algorithms first detect the nearest obstacles in the field of view in various directions. The area between the obstacles and the detection system is free space. Therefore, free space estimation and obstacle detection are essentially the same.

[0004] Free space estimation in indoor scenarios helps mobile robots and other devices safely explore their surroundings and avoid collisions and other hazards. Unlike road detection, free space estimation focuses on the road surface rather than the road itself. It determines whether a collision is likely, rather than whether the vehicle is on a legal path. The output is a segmentation of obstacles and the ground, rather than a segmentation of road and non-road. Free space estimation is crucial in most indoor scenarios.

[0005] In free space estimation, obstacle detection, and other problems, commonly used methods are categorized as window methods and region methods. The window method uses a series of rectangular regions as candidate regions for objects. This type of method is widely used in traditional two-dimensional image recognition and can effectively perform recognition tasks, with high detection rates and speeds. However, this method has a low detection rate for occluded objects and grouped objects, making it unsuitable for object recognition in three-dimensional scenes. The region method uses non-overlapping, category-labeled regions as candidate regions for objects. This method is primarily used in the field of semantic segmentation and offers high accuracy, but it cannot distinguish individual instances within a group.

[0006] Free space estimation methods based on stereo vision include the occupancy grid method, plane fitting method, YOZ fitting method, and V-parallax method. The occupancy grid method has been widely used in the field of mobile robot navigation. The occupancy grid method is essentially a gridding of a bird's-eye view of the road scene. The presence of obstacles in three-dimensional space is indicated by whether the corresponding square is occupied. This is generally represented using a two-dimensional matrix or grid, with each cell in the occupancy grid maintaining an occupancy probability, that is, the probability of an obstacle. This probability is generally generated using a statistical histogram method based on a point cloud. The plane fitting method fits the ground surface with a plane or curved surface in the three-dimensional point cloud space. The YOZ fitting method is essentially a side view of the road scene. Three-dimensional points are mapped onto the YOZ plane and line fitting or curve fitting methods are used to detect line segments.

[0007] The above theories are widely used in practice, but these methods all require the calculation of 3D point cloud coordinates, incurring additional computational overhead and placing high demands on camera parameter calibration. Therefore, the classic V-parallax method reduces this reliance on camera parameter calibration. The V-parallax method maps 3D (u, v, d) coordinates to a 2D (v, d) coordinate system, where (u, v) represents image coordinates and d represents disparity. During the V-parallax process, obstacles are mapped as vertical line segments, while the ground is mapped as diagonal lines, or ground-related lines. The V-parallax method has achieved great success in free-space estimation due to its lack of camera extrinsic parameters and its computational speed. The U-parallax method also overcomes the loss of horizontal information during the V-parallax method's mapping process. By mapping 3D (u, v, d) coordinates to a 2D (u, d) coordinate system, obstacles are mapped as horizontal line segments, thereby determining their horizontal width. Combined with the V-parallax method, obstacle height information can be further obtained.

[0008] The V-disparity method transforms the free-space estimation problem into a ground-related line extraction problem—a linear fitting problem. While this method fully exploits the structural information of the ground, it lacks the ability to detect uneven ground surfaces due to its assumption of a flat ground surface. Furthermore, current mainstream methods struggle to simultaneously detect multiple ground surfaces with varying slope parameters in the same image. Summary of the Invention

[0009] In response to the problems of current traditional methods requiring the calculation of 3D point cloud coordinates, high computational overhead, and high requirements for camera parameter calibration; and the problem that the single V-disparity method also loses horizontal information during the mapping process, the present invention proposes a free space estimation and obstacle detection method based on a unified algorithm structure of an improved UV disparity map and confidence map calculation model. This method can be applied to household robots in indoor scenes to estimate free space and detect obstacles after image acquisition to avoid collisions. After acquiring an image using a binocular stereo camera, a disparity map is obtained from the image. After certain processing and corrections are performed on the disparity map, a UV disparity map is generated. Ground estimation is performed based on the improved V-disparity method, and a confidence map based on the V-disparity calculation result is calculated. Dynamic programming is used to perform free space estimation on the confidence map, and the confidence map is output.

[0010] Specifically, for free space estimation on uneven terrain, particularly in scenarios with transverse slopes and multiple terrains, the improved UV disparity method uses a sliding window approach. This method divides the disparity map into several sub-windows, generating a sub-V-disparity map for each sub-window. Within each window, a piecewise linear fitting method is used to sequentially connect the extracted line segments to obtain ground correlation lines. To enhance the stability of ground correlation lines, a new V-disparity map filtering method is employed. The sub-V-disparity maps are filtered with an edge filter, and the filtered image is used as a weighting factor to multiply the original V-disparity map. The filtered image is then filtered using a maximum filter and a block filter to obtain a V-disparity filtered image. This differs from the direct use of the V-disparity method for road detection. The V-disparity calculation results are integrated into a probabilistic estimation framework for ground estimation. The confidence map calculation model includes a ground confidence map (RCM) and an obstacle confidence map (OCM). Both the RCM and OCM models are functions of the difference between the disparity values ​​and the ground prediction values. The weighted accumulation of the RCM and OCM results constitutes the final confidence map model. Free space is optimized using a two-dimensional dynamic programming method based on the final confidence map.

[0011] The technical solution of the present invention is:

[0012] The indoor free space estimation and obstacle detection method based on UV disparity map includes the following steps:

[0013] 1) Get the disparity map.

[0014] 1-1) Use a binocular stereo camera to capture RGB images of the real scene and convert them into grayscale images.

[0015] 1-2) Determine the disparity map based on the parallel binocular stereo vision model.

[0016] 2) Correct the disparity map.

[0017] 2-1) Use bicubic interpolation to correct the disparity map for invalid non-edge points. Set the disparity values ​​of all invalid matching points in the disparity map to -1. Weight the disparity of each pixel's upper, lower, left, and right neighbors and reassign them using an improved bicubic interpolation algorithm.

[0018] 2-2) Correction of invalid edge points. After edge detection, the edge image is processed by morphological closing operation, that is, dilation first and then erosion, filling the invalid area between adjacent edge points and assigning values, highlighting the obstacle entity without changing the obstacle position; for areas with larger invalid edge points, no value is assigned and they are marked. The marked area is judged and the area A is compared. i With area threshold A v If A i v , then area A i Not calculated, belonging to the marginal area; if A i >A v , then judge A i Is it an edge area? Calculate the edge line area and A i If the ratio is greater than the threshold, it belongs to the edge area, and the disparity map is corrected by the closing operation correction method. Otherwise, it does not belong to the edge area and is corrected by improving the bicubic difference.

[0019] 3) Obtain the U disparity map and the V disparity map.

[0020] 3-1) Perform horizontal and vertical projection on the corrected disparity map, and project the plane disparity image into line segments to obtain the UV disparity map.

[0021] 3-2) Generate a U-disparity map. The U-disparity map is formed by accumulating the number of pixels with the same disparity in each column in the U direction. This is done by scanning the disparity map column by column from top to bottom. The horizontal length of the disparity map is used as the abscissa of the U-disparity map, the maximum disparity value is used as the ordinate, and the cumulative sum of pixels with the same disparity value in each column is used as the intensity of the coordinate point in the U-disparity map.

[0022] 3-3) Generate the V disparity map. The disparity map is scanned horizontally from left to right, row by row, by accumulating the number of pixels with the same disparity in each row in the V direction. The vertical length of the disparity map is used as the ordinate of the U disparity map, the maximum disparity value is used as the abscissa, and the cumulative sum of pixels with the same disparity value in each row is used as the intensity of the coordinate point of the V disparity map. This forms the V disparity map.

[0023] 4) Ground estimation.

[0024] ​4-1) Generate a sub-V disparity map. Using the sliding window method, the disparity map is divided into multiple sub-windows. The ground is assumed to be locally flat in each sub-window, and a sub-V disparity map is generated in each sub-window for ground estimation.

[0025] 4-2) Filtering the sub-V disparity map. Due to the unevenness of the real ground, the projection of the ground on the V disparity map exhibits certain distribution characteristics. At the same time, non-ground projections also have a certain impact on the extraction of ground-related lines. To reduce the impact of ground-distributed projections and non-ground projections, the V disparity map needs to be filtered before linear fitting. The present invention adopts a new V disparity filtering method, specifically: first, the Gaussian-smoothed V disparity map is filtered using an edge filter, using the Sobel operator to enhance horizontal and oblique edges and weaken vertical edges. The filtered sub-V disparity map is then converted into a binary image. Unlike traditional filtering methods, this algorithm treats the V disparity map as a weighted coefficient map, multiplying the binary filtered image with the original V disparity map to obtain a new sub-V disparity map. The maximum value points are then selected column by column in the new sub-V disparity map and retained, while other pixels are removed. After the above operation, vertical lines still exist and are not fully removed. These remnants appear as isolated points, and the distance between them is much larger than the distance between points on the ground-related lines. Therefore, a block counting filter is used to remove the residue. If there are no other points in the central pixel block, the central point is filtered out, otherwise it is retained.

[0026] 4-3) Extract ground correlation lines. Use Hough transform to extract ground correlation lines, and use multiple line segments to fit the different slopes of the ground. When the ground is flat, usually only one ground correlation line is detected; when the ground has a continuously changing longitudinal slope, multiple line segments will be detected. These line segments are connected together to form a continuous broken line to represent the ground correlation line. Using the fitted ground correlation line, it is easy to get the predicted disparity value d of any line on the ground * (v).

[0027] 5) Calculate the confidence map.

[0028] 5-1) Calculate the obstacle confidence map (OCM) and the ground confidence map (RCM). A confidence map is a two-dimensional image, similar to an occupancy grid, where each pixel value represents a probability of belonging to an obstacle. In the OCM, the confidence value represents the probability of belonging to an obstacle; in the RCM, the confidence value represents the probability of belonging to the ground.

[0029] OCM is defined as:

[0030] RCM is defined as:

[0031] Among them, d(u,v) represents the disparity value at the point (u,v) on the disparity map, and d *(v) represents the ground disparity predicted in row v. Δd max Indicates the maximum difference in disparity. Construct OCM and RCM window by window, and use d for the overlapping area between sliding windows. * (v) Mean substitution.

[0032] 5-2) Calculate the confidence map. The final confidence map is composed of the weighted accumulation of OCM and RCM, as shown below:

[0033] C(u,v)=C r Road(u,v)+C o Obstacle(u,v)

[0034] Among them, C r is the ground confidence weight, set to a constant value. o is the obstacle confidence weight, which is learned through ROC curve analysis.

[0035] 6) Dynamic programming to estimate free space.

[0036] 6-1) Constructing a dynamic programming model to estimate free space

[0037] The boundary between the ground and the obstacle can be obtained by finding the optimal segmentation path in the confidence map. A smoothness constraint is introduced to reduce the drastic changes between boundaries. The calculation formula includes data terms and smoothness terms, and is solved through dynamic programming as follows:

[0038]

[0039] Among them, v * (u) is the estimated boundary edge between the ground and the bottom edge of the obstacle, u a ,u b are adjacent column pixels, and the data items are the confidence values ​​c(u,v), s(v i ,v i-1 ) is a smoothness constraint to penalize drastic changes between adjacent columns, as follows:

[0040]

[0041] Among them, C s It is a continuity constraint parameter, which needs to be learned through ROC curve analysis and estimation. The threshold T s It is a fixed value, supplemented by continuity constraints, which allows changes between boundaries while limiting the maximum value of the changes between boundaries. The specific solution process is as follows:

[0042] 6-2) Calculate the minimum cumulative cost. Calculate recursively from the rightmost column to the leftmost column of the image as follows:

[0043]

[0044] Among them, (i,v i ) represents the vth row in the bottom i i The pixel node of the row, c(i,v i ) is the node (i,v i ) confidence value, s(v i ,v i+1 ) is from node (i,v i ) to node (i+1,v i+1 ) connection cost, C i (v i ) is from the rightmost node to the node (i,v i ) is the minimum cumulative cost.

[0045] 6-3) Find the optimal path.

[0046]

[0047] Use the above formula to recursively solve the best path. The sequence is the optimal path to the boundary. The area before the boundary is the free space.

[0048] 7) Detect obstacles.

[0049] 7-1) Detect obstacles through U-disparity map.

[0050] Scan the U-disparity map and mark obstacle points. Consider each row and two columns of points as a cell. Scan the U-disparity map row and column by row. If the number of valid (on the ground and no more than two meters high) disparities within a cell is at least 2 or 1, mark the cell as a valid obstacle point. In the row-by-row loop, generate a detection line through the locations of the marked valid obstacle point cells in each row.

[0051] 7-2) Detect obstacle distance.

[0052] Classify the detection lines. Process the generated detection lines. If two detection lines are close, they are considered to be the detection lines corresponding to the same obstacle. Finally, each type of detection line corresponds to one obstacle.

[0053] Mark the obstacle with a box in the disparity map and merge the boxes. Find the adjacent obstacle boxes of the obstacle box and merge them. Loop through each obstacle in turn. If the merged obstacle exceeds the pre-set threshold size, it will not be merged.

[0054] Finally, the physical position and size information of the obstacle are calculated.

[0055] The beneficial effects of the present invention are as follows: a free space estimation algorithm and an obstacle detection method for indoor use are proposed. The free space estimation algorithm applied to indoor scenes uses an improved UV parallax method to generate a confidence map model, and uses the confidence map to perform dynamic programming to estimate the optimal result, which can better determine the free space in the indoor environment. At the same time, the algorithm avoids redundant calculation of 3D point cloud information; in addition, the obstacle detection algorithm has a good detection effect for complex and diverse obstacle information in indoor scenes, which can help indoor robots locate and navigate, safely explore the surrounding environment, and avoid collisions and other dangers. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Flowchart of the present invention.

[0057] Figure 2 Detailed flowchart of the ground estimation steps.

[0058] Figure 3 This is a parallax image collected and acquired by the household robot in the embodiment.

[0059] Figure 4 for Figure 3 The confidence map model is composed of the weighted accumulation of the obstacle confidence map and the ground confidence map.

[0060] Figure 5 for Figure 3 Obstacle detection results of the captured image. DETAILED DESCRIPTION

[0061] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0062] The present invention solves some common problems in road surface detection and obstacle detection based on the free space estimation and obstacle detection method based on the improved UV disparity map. The basic flow chart is as follows Figure 1 As shown, the details are as follows:

[0063] 1) Get the disparity map:

[0064] 1-1) Collect RGB images and grayscale images of real scenes through binocular stereo cameras.

[0065] 1-2) Determine the disparity map based on a parallel binocular stereo vision model. The SGM disparity estimation method combines local and global algorithms and is a semi-global matching algorithm. The algorithm's computational process includes: using image mutual information to calculate the matching cost and employing hierarchical mutual information (HMI) to improve computational speed; aggregating costs using a path-optimal global energy function; calculating disparity using a winner-takes-all algorithm; and finally refining the disparity using left-right consistency checking and sub-pixel interpolation.

[0066] 2) Correct the disparity map

[0067] 2-1) Use bicubic interpolation to correct the disparity map for invalid non-edge points. Set the disparity values ​​of all invalid matching points in the disparity map to -1. Weight the disparity of each pixel's upper, lower, left, and right neighbors and reassign them using an improved bicubic interpolation algorithm.

[0068] This interpolation method uses the disparity values ​​of 16 points above, below, left, and right of the interpolated point as a benchmark to calculate the disparity estimate of the interpolated point. Bicubic interpolation not only considers the influence of the interpolated point's immediate neighbors on its disparity value, but also the influence of the interpolation rate of change between its immediate neighbors on its disparity value, resulting in an image that is closer to reality.

[0069] The interpolation basis functions selected are as follows:

[0070]

[0071] The destination pixel value f(i+u,j+v) can be obtained by the following bicubic difference formula:

[0072] f(i+u,j+v)=[A]*[B]*[C]

[0073] in:

[0074] [A]=[S(u+2)S(u+1)S(u)S(u-1)S(u-2)]

[0075]

[0076] [C]=[S(v+3)S(v+2)S(v+1)S(v)S(v-1)S(v-2)S(v-3)] T

[0077] Using a 5×7 convolution kernel, the disparity value of a certain pixel is more similar to the disparity value in the same row, and may be significantly different from the disparity value in the same column.

[0078] 2-2) To correct invalid edge points, after edge detection, the edge image is processed by morphological closing operation, that is, dilation followed by erosion, to fill the invalid area between adjacent edge points, highlighting the obstacle entity without changing the obstacle position. The definition of dilation is:

[0079] Dst(x,y)=max(src(x+x',y+y'))

[0080] The location of corrosion is:

[0081] Dst(x,y)=min(src(x+x',y+y'))

[0082] For the area with a large area of ​​invalid edge points, no value is assigned and it is marked. The marked area is judged and the area A is compared. i With area threshold A v If A i v , then area A i Not calculated, belonging to the marginal area; if A i >A v , then judge A i Is it an edge area? If it is an edge area, the disparity map is corrected by the closing operation correction method. If it is not an edge area, it is corrected by the bicubic difference.

[0083] 3) Obtain the U disparity map and the V disparity map.

[0084] 3-1) The corrected disparity map is projected horizontally and vertically, converting the planar disparity image into line segments to obtain a UV disparity map. This conversion of traditional plane detection of the target into line segment detection simplifies the calculation method and improves computational speed. The position information of the detected line segments can be used to determine the precise location of the road surface and obstacles.

[0085] 3-2) Generate a U-disparity map. The U-disparity map is formed by accumulating the number of pixels with the same disparity in each column in the U direction. This is done by scanning the disparity map column by column from top to bottom. The horizontal length of the disparity map is used as the abscissa of the U-disparity map, the maximum disparity value is used as the ordinate, and the cumulative sum of pixels with the same disparity value in each column is used as the intensity of the coordinate point in the U-disparity map.

[0086] 3-3) Generate the V disparity map. The disparity map is scanned horizontally from left to right, row by row, by accumulating the number of pixels with the same disparity in each row in the V direction. The vertical length of the disparity map is used as the ordinate of the U disparity map, the maximum disparity value is used as the abscissa, and the cumulative sum of pixels with the same disparity value in each row is used as the intensity of the coordinate point of the V disparity map. This forms the V disparity map.

[0087] 4) Ground estimation (basic process such as Figure 2 shown).

[0088] 4-1) Generate a sub-V disparity map. Using the sliding window method, the disparity map is divided into multiple sub-windows. The ground is assumed to be locally flat in each sub-window, and a sub-V disparity map is generated in each sub-window for ground estimation.

[0089] ​4-2) Filter the sub-V disparity map. First, the Gaussian-smoothed V disparity map is filtered using an edge filter. The Sobel operator is used to enhance horizontal and oblique edges and weaken vertical edges. The filtered sub-V disparity map is then converted into a binary image. Unlike traditional filtering methods, this algorithm treats the V disparity map as a weighted coefficient map. The binary filtered image is multiplied with the original V disparity map to obtain a new sub-V disparity map. The maximum value points are then selected column by column on the sub-V disparity map to be retained, and the other pixel points are removed. After the above operations, the vertical lines still exist and are not fully cleared. These residual parts appear as some isolated points, and the distance between them is much larger than the distance between the points on the ground related lines. Therefore, a block counting filter is used to remove the residue. If there are no other points in the central pixel block, the central point is filtered out, otherwise it is retained.

[0090] 4-3) Extract the ground correlation line. Use the Hough transform to extract the ground correlation line, fitting multiple line segments to the different ground slopes. When the ground is flat, only one ground correlation line is typically detected. When the ground has a continuously changing longitudinal slope, multiple line segments are detected. These line segments are connected together to form a continuous polyline to represent the ground correlation line. Using the fitted ground correlation line, the predicted disparity value d*(v) for any ground line can be easily obtained.

[0091] 5) Calculate the confidence map.

[0092] 5-1) Calculate the obstacle confidence map (OCM) and the ground confidence map (RCM). In the OCM, the confidence value represents the probability of belonging to an obstacle; in the RCM, the confidence value represents the probability of belonging to the ground.

[0093] OCM is defined as:

[0094] RCM is defined as:

[0095] Among them, d(u,v) represents the disparity value at the point (u,v) on the disparity map, and d * (v) represents the ground disparity predicted in row v. Δd max Indicates the maximum difference in disparity. Construct OCM and RCM window by window, and use the mean value to replace the overlapping area between sliding windows.

[0096] 5-2) Calculate the confidence map. The final confidence map is composed of the weighted accumulation of OCM and RCM, as shown below:

[0097] C(u,v)=C r Road(u,v)+C o Obstacle(u,v)

[0098] Among them, C ris the road confidence weight, set to a constant value. o is the obstacle confidence weight, which is learned through ROC curve analysis. The confidence map model composed of the weighted accumulation of the obstacle confidence map (OCM) and the ground confidence map is as follows: Figure 4 As shown, it can be clearly seen that the confidence models corresponding to the obstacles in the figure have distinct levels.

[0099] 6) Dynamic programming to estimate free space.

[0100] 6-1) Constructing a dynamic programming model to estimate free space

[0101] The boundary between the ground and the obstacle can be obtained by finding the optimal segmentation path in the confidence map. A smoothness constraint is introduced to reduce the drastic changes between boundaries. The formula includes data terms and smoothness terms, and is solved through dynamic programming as follows:

[0102]

[0103] Among them, v * (u) is the estimated boundary edge between the ground and the bottom edge of the obstacle, u a ,u b are adjacent column pixels, and the data items are the confidence values ​​c(u,v), s(v i ,v i-1 ) is a smoothness constraint to penalize drastic changes between adjacent columns, as follows:

[0104]

[0105] Among them, C s It is a continuity constraint parameter, which needs to be learned through ROC curve analysis and estimation. The threshold T s It is a fixed value, supplemented by continuity constraints, which allows changes between boundaries while limiting the maximum value of the changes between boundaries.

[0106] 6-2) Calculate the minimum cumulative cost. Calculate recursively from the rightmost column to the leftmost column of the image as follows:

[0107]

[0108] Among them, (i,v i ) represents the vth row in the bottom i i The pixel node of the row, c(i,v i ) is the node (i,v i ) confidence value, s(v i ,v i+1 ) is from node (i,v i ) to node (i+1,v i+1 ) connection cost, C i (vi ) is from the rightmost node to the node (i,v i ) is the minimum cumulative cost.

[0109] 6-3) Find the optimal path.

[0110]

[0111] Use the above formula to recursively solve the best path. The sequence is the optimal path to the boundary. The area before the boundary is the free space.

[0112] 7) Detect obstacles.

[0113] 7-1) Detect obstacles through U-disparity map.

[0114] Scan the U-disparity map and mark obstacle points. Consider each row and two columns of points as a cell. Scan the U-disparity map row and column by row. If the number of valid (on the ground and no more than two meters high) disparities within a cell is at least 2 or 1, mark the cell as a valid obstacle point. In the row-by-row loop, generate a detection line through the locations of the marked valid obstacle point cells in each row.

[0115] 7-2) Detect obstacle distance.

[0116] Classify the detection lines. Process the generated detection lines. If two detection lines are close, they are considered to be the detection lines corresponding to the same obstacle. Finally, each type of detection line corresponds to one obstacle.

[0117] Mark the obstacle with a box in the disparity map and merge the boxes. Find the adjacent obstacle boxes of the obstacle box and merge them. Loop through each obstacle in turn. If the merged obstacle exceeds the pre-set threshold size, it will not be merged.

[0118] Finally, the physical position and size information of the obstacle are calculated. The obstacle detection results are as follows Figure 5 As shown, we can see that all obstacles in the picture (chairs, corners, boxes, flower beds) are detected, and the detection effect is good for multiple categories of indoor obstacles.

Claims

1. Indoor free space estimation and obstacle detection method based on UV disparity map, characterized by: The specific steps are as follows: 1) Obtain a disparity map; 1-1) Use a binocular stereo camera to capture RGB images of real indoor scenes and convert them into grayscale images; 1-2) Determining a disparity map based on a parallel binocular stereo vision model; 2) Correct the disparity map; 2-1) Use bicubic interpolation to correct the disparity map for non-edge invalid points; The disparity values ​​of all invalid matching points in the disparity map are set to -1, and the disparity of the upper, lower, left, and right neighboring points of each pixel in the image are weighted and reassigned using the improved bicubic interpolation algorithm; 2-2) To correct invalid edge points, after edge detection, the edge image is processed by morphological closing operations, that is, dilation followed by erosion, to fill the invalid areas between adjacent edge points, highlighting the obstacle entity without changing the obstacle position; for areas with large invalid edge points, no value is assigned and they are marked; Judge the marked area and compare the area A i With area threshold A v The size of i v , then area A i Not calculated, belonging to the marginal area; if A i >A v , then judge A i Is it an edge area? Calculate the edge line area and A i If the ratio is greater than the threshold, it belongs to the edge area, and the disparity map is corrected by the closing operation correction method. Otherwise, it does not belong to the edge area and is corrected by improving the bicubic difference.​ 3) Obtain U disparity map and V disparity map; 3-1) Perform horizontal and vertical projection on the corrected disparity map, and project the plane disparity image into line segments to obtain a UV disparity map; 3-2) Generating a U-disparity map; accumulating the number of pixels with the same disparity in each column of the disparity map in the U direction, i.e., scanning each column vertically from top to bottom, using the horizontal length of the disparity map as the abscissa of the U-disparity map, the maximum disparity value as the ordinate, and the cumulative sum of pixels with the same disparity value in each column as the intensity of the coordinate point of the U-disparity map, thereby forming a U-disparity map; 3-3) Generating a V disparity map; accumulating the number of pixels with the same disparity in each row of the disparity map in the V direction, i.e., scanning the disparity map row by row from left to right in the horizontal direction, using the vertical length of the disparity map as the ordinate of the U disparity map, the maximum disparity value as the abscissa, and the cumulative sum of pixels with the same disparity value in each row as the intensity of the coordinate point of the V disparity map, thereby forming the V disparity map; 4) Ground estimation; 4-1) Generate a sub-V disparity map; Use a sliding window method to divide the disparity map into multiple sub-windows. Assume that the ground is locally flat in each sub-window, so as to generate a sub-V disparity map in each sub-window for ground estimation; 4-2) Filter the sub-V disparity map; filter the V disparity map and then perform linear fitting, using a new V disparity filtering method, specifically: first, filter the Gaussian smoothed V disparity map using an edge filter, use the Sobel operator to enhance horizontal and oblique edges, and weaken vertical edges; then convert the filtered sub-V disparity map into a binary image, treat the V disparity map as a weighted coefficient map, and multiply the binary filtered image with the original V disparity map to obtain a new sub-V disparity map; then select the maximum value point column by column on the new sub-V disparity map and retain it, and remove other pixel points; after the above operation, the vertical lines still exist and are not fully cleared. These residual parts appear as some isolated points, and the distance between them is much larger than the distance between the points on the ground related lines; therefore, a block counting filter is used to remove the residuals. If there are no other points in the central pixel block, the central point is filtered out, otherwise it is retained; 4-3) Extract ground correlation lines; use Hough transform to extract ground correlation lines, and use multiple line segments to fit different ground slopes; when the ground is flat, only one ground correlation line is detected; when the ground has a continuously changing longitudinal slope, multiple line segments will be detected, and these line segments are connected together to form a continuous broken line to represent the ground correlation line; using the fitted ground correlation line, the predicted disparity value d of any ground line can be obtained * (v); 5) Calculate the confidence map; 5-1) Calculate the obstacle confidence map (OCM) and the ground confidence map (RCM). In the OCM, the confidence value indicates the probability of belonging to an obstacle; in the RCM, the confidence value indicates the probability of belonging to the ground. OCM is defined as: RCM is defined as: Among them, d(u,v) represents the disparity value at the point (u,v) on the disparity map, and d * (v) represents the ground disparity predicted in row v; Δd max Indicates the maximum difference in disparity; construct OCM and RCM window by window, and use d for the overlapping area between sliding windows * (v) mean substitution; 5-2) Calculate the confidence map; the final confidence map is composed of the weighted accumulation of OCM and RCM, as shown below: C(u,v)=C r Road(u,v)+C o Obstacle(u,v) Among them, C r is the ground confidence weight, set to a constant value; C o is the obstacle confidence weight, which is learned through ROC curve analysis; 6) Dynamic programming to estimate free space; 6-1) Constructing a dynamic programming model to estimate free space The boundary between the ground and the obstacle is obtained by finding the optimal segmentation path in the confidence map. A smoothness constraint term is introduced to reduce the drastic changes between boundaries. The calculation formula includes data terms and smoothness terms and is solved through dynamic programming as follows: Among them, v * (u) is the estimated boundary edge between the ground and the bottom edge of the obstacle, u a ,u b are adjacent column pixels, and the data items are the confidence values ​​c(u,v), s(v i ,v i-1 ) is a smoothness constraint to penalize drastic changes between adjacent columns, as follows: Among them, C s It is a continuity constraint parameter, which needs to be learned through ROC curve analysis and estimation; the threshold T s It is a fixed value, supplemented by continuity constraints, which limits the maximum value of the change between boundaries while allowing changes between boundaries; 6-2) Calculate the minimum cumulative cost; recursively calculate from the rightmost column to the leftmost column of the image as follows: Among them, (i,v i ) represents the vth row in the bottom i i The pixel node of the row, c(i,v i ) is the node (i,v i ) confidence value, s(v i ,v i+1 ) is from node (i,v i ) to node (i+1,v i+1 ) connection cost, C i (v i ) is from the rightmost node to the node (i,v i )’s minimum cumulative cost; 6-3) Find the optimal path; The least expensive The sequence is the optimal path to the boundary; the area before the boundary is the free space; 7) Detect obstacles; 7-1) Detect obstacles through U-disparity map; Scan the U-disparity map and mark the obstacle points. Treat the points in each row and two columns as a cell. Scan the U-disparity map row by row and column by column. If the number of valid disparities in a cell is not less than 2 or 1, mark the cell label as a valid obstacle point. In the row-by-row loop, generate a detection line through the position of the marked valid obstacle point cell in each row. 7-2) Detect obstacle distance; Classify the detection lines; process the generated detection lines. If two detection lines are close, they are considered to be the detection lines corresponding to the same obstacle. Finally, each type of detection line corresponds to one obstacle. Mark the obstacles with boxes in the disparity map and merge them into a single box. Find the adjacent obstacle boxes of the obstacle box and merge them into a single box. Loop through each obstacle in turn. If the merged obstacle exceeds the set threshold size, it will not be merged. Finally, calculate the physical position and size information of the obstacle.

2. The method for indoor free space estimation and obstacle detection based on UV disparity map according to claim 1, characterized in that: The improved bicubic interpolation algorithm described in step 2) is specifically as follows: The interpolation method uses the disparity values ​​of 16 points above, below, left and right of the interpolated point as a reference to calculate and obtain the disparity estimate of the interpolated point; The interpolation basis functions selected are as follows: The destination pixel value f(i+u,j+v) is obtained by the following bicubic interpolation formula: f(i+u,j+v)=[A]*[B]*[C] in: [A]=[S(u+2)S(u+1)S(u)S(u-1)S(u-2)] [C]=[S(v+3)S(v+2)S(v+1)S(v)S(v-1)S(v-2)S(v-3)] T 。

Images