Three-dimensional plane rapid detection and segmentation method based on depth map data

Abstract

The invention discloses a three-dimensional plane rapid detection and segmentation method based on depth map data, and relates to the field of robots, and the method comprises the following steps: converting a depth image into a point cloud, carrying out the downsampling, and constructing a pyramid based on the point cloud; through data access and parameter fitting from top to bottom, a self-adaptive plane block segmentation method is utilized to obtain a self-adaptive plane block, and detailed information of the plane block is stored in a state pyramid; by using a block-level region growing method, marking adjacent and coplanar block-level regions as communicated plane regions to realize combination of plane blocks; and calculating the distance from the pixel point to the surface of the edge of each region, carrying out pixel-level label distribution, distributing the pixel to the region with the minimum distance, and outputting a segmentation result. According to the method, the plane area is represented as a plurality of plane blocks as large as possible through self-adaptive blocking and using less plane fitting operation on the large-area plane area, the number of the plane blocks participating in operation is reduced, the calculation cost is reduced, and the method has high real-time performance and a good segmentation effect.

Description

Technical Field

[0001] The present invention relates to the field of robots, and in particular to a three-dimensional plane rapid detection and segmentation method based on depth map data. Background Art

[0002] Depth cameras are widely used in the field of robotics. They can obtain high-frame-rate and high-resolution depth images of the scene, enabling robots to achieve accurate 3D perception. 3D plane segmentation technology is the key for robots to understand and analyze the 3D geometry of the scene in the depth image. It can provide basic information for multiple functions such as visual odometer, scene understanding, sensor calibration, and object pose estimation.

[0003] At present, plane segmentation methods can be mainly divided into four categories: RANSAC-based methods, Hough transform-based methods, clustering-based methods, and deep learning-based methods. Although the commonly used RANSAC-based method is simple and easy to use, it requires repeated random sampling, parameter fitting, and result verification, resulting in high computational costs. The Hough transform-based method performs well in dealing with noise, but it is often difficult to find a good balance between accuracy and computational cost. The clustering-based method has a faster processing speed, but there is still room for improvement in accuracy. Although the deep learning-based method performs well in general image segmentation, object detection and other problems, this type of method has a large demand for hardware computing power and a large amount of training data, and is therefore not suitable for deployment and application on edge devices.

[0004] Various plane segmentation methods have their own advantages and disadvantages. For scenarios requiring real-time performance on edge devices, how to find a balance between accuracy, computational cost, and ease of use remains an unresolved problem.

[0005] Therefore, technicians in this field are committed to developing a three-dimensional plane fast detection and segmentation method based on depth map data. Summary of the invention

[0006] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is that when the existing method is deployed on an edge device, the calculation is redundant and the performance is poor.

[0007] To achieve the above object, the present invention provides a method for rapid detection and segmentation of a three-dimensional plane based on depth map data, the method comprising the following steps:

[0008] S101: converting the acquired depth image into point cloud data, downsampling the point cloud data, and constructing a data pyramid and a state pyramid based on the downsampled point cloud data;

[0009] S103: obtaining an adaptive plane block by using an adaptive plane block segmentation method through top-down data access and parameter fitting, and storing detailed information of the plane block in the state pyramid;

[0010] S105: using a block-level region growing method, marking adjacent and coplanar block-level regions as connected plane regions, thereby merging coplanar plane blocks;

[0011] S107: Calculate the distance from the pixel point to the surface of each region edge, assign pixel-level labels according to the calculation results, assign pixels to the region with the smallest distance, and output the segmentation result.

[0012] Furthermore, the step S101 includes the following sub-steps:

[0013] S1011: Convert the depth image into the point cloud data, wherein the point cloud data and the depth image maintain the same matrix size;

[0014] S1012: downsampling the point cloud data to create the data pyramid;

[0015] S1013: Storing the point cloud data in the data pyramid, wherein the 0th layer of the data pyramid stores the original point cloud data, and the width and height of the coordinate matrix in subsequent layers of the data pyramid are halved layer by layer;

[0016] S1014: Storing state information of the algorithm in the state pyramid, wherein the state pyramid and the data pyramid have the same data structure and indexing method.

[0017] Furthermore, the correspondence between pixels and regions at different levels is established in the data pyramid, and each block is represented by a pyramid index:

[0018] The pixel at position [r, c] in layer l corresponds to the pixel at position [2r, 2c] in layer l-1;

[0019] The pyramid index is represented as follows:

[0020] i p = [l, r, c, s];

[0021] Index i p The corresponding area in the l-1th layer is: [l-1, 2r, 2c, 2s];

[0022] The area includes four sub-blocks, which are:

[0023] [l-1, 2r, 2c, s], [l-1, 2r+s, 2c, s], [l-1, 2r, 2c+s, s] and [l-1, 2r+s, 2c+s, s];

[0024] Among them, l represents the layer number, (r, c) represents the pixel coordinates, s represents the size of the block, and i p Pyramid index.

[0025] Furthermore, in step S103, the depth image is divided into a plurality of square regions, each of which has the same size, and a hierarchical index is used to represent the position and size of a block in the depth image, and the index value corresponds to a relative spatial position;

[0026] For any block with a hierarchical index, the sampling points are directly extracted from the data pyramid using the pyramid index, where

[0027] The hierarchical index is: i h =[i0,i1,...,i n ], (n≤N),

[0028] The pyramid index is: i p =[Nn, r, c, s min ],

[0029] The following conversion formula is used to complete the conversion between the hierarchical index and the pyramid index:

[0030]

[0031]

[0032] Among them, i h is the hierarchical index, i p is the pyramid index, i0 represents the initial partition area where a block is located, and the remaining indices [i1, ..., i n ] represents the relative position in the area, 0, 1, 2, 3 correspond to the upper left, upper right, lower left, and lower right sub-blocks respectively, s init is the initial block size, s min is the minimum block size, j, n is the number of layers, N is the maximum number of layers, N = log2(s init / s min ), (r, c) is the pixel coordinate, ω0 is the width of the initial block, is the floor operator.

[0033] Furthermore, the adaptive plane block segmentation method adopts a recursive method to perform adaptive plane block segmentation, including the following steps:

[0034] S1031: Obtaining point set data D from the data pyramid D using the hierarchical index i i , use queue Q to represent the area to be processed;

[0035] S1032: Check each index in the queue Q in turn to determine whether the plane area corresponding to the index satisfies the plane condition;

[0036] S1033: Recording the block information that meets the plane condition into the state pyramid, and adding the four sub-blocks of the block that does not meet the plane condition into the queue Q;

[0037] S1034: Determine whether the data in the queue Q has been processed. If not, continue processing.

[0038] Furthermore, the adaptive plane block segmentation method performs plane fitting through principal component analysis, and uses the plane normal vector as the eigenvector corresponding to the minimum eigenvalue in the principal component analysis method. The plane condition is that the block on which the plane fitting is performed is a continuous block and the plane normal vector of the block is lower than a preset threshold.

[0039] Furthermore, in step S105, the block-level region growing method uses region growing to merge coplanar planar blocks and adopts a breadth-first search strategy to expand the search area, including the following sub-steps:

[0040] S1051: Select the largest unmarked and unvisited block from the state pyramid S as a seed point, add the seed point to the queue Q, and store the block index in the current plane area into the set R;

[0041] S1052: For each patch q∈Q, calculate a set of patches N adjacent to the patch q, and sort them by area to ensure that region growth gives priority to larger patches;

[0042] S1053: Check each adjacent patch n∈N to determine whether n is coplanar with q. If so, add patch n to R and Q.

[0043] S1054: Obtain the indexes of all blocks in R, and store the plane parameters and labels in the state pyramid S according to the indexes, thereby completing the growth of a region;

[0044] S1055: Repeatedly select new seed points to start the next region growth until there are no new seed points to choose from.

[0045] Furthermore, the block-level region growing method optimizes the traditional neighbor search method. The optimized neighbor search method realizes neighbor search for blocks of variable sizes. The optimized neighbor search method includes the following steps:

[0046] S1: Get a seed block, and calculate the four corner coordinates of the seed block in the Nn layer according to the hierarchical index of the seed block;

[0047] S2: Calculate the coordinates of the four corners in layer 0: p′ i =2 N-n p i ;

[0048] S3: Obtain four edge segments of the seed block: p′1p′2, p′2p′3, p′3p′4, p′1p′4, where the neighbors of the seed block are adjacent to the four edges of the seed block;

[0049] S4: adding row deviations or column deviations to four edges of the seed block to obtain indexes of neighboring blocks;

[0050] S5: Use the index to directly access the neighbor block information in the state pyramid S.

[0051] Furthermore, in step S1, the four-corner coordinates of the seed block in the Nn layer are calculated as follows:

[0052] The upper left corner coordinate p1 of the seed block is calculated by the conversion formula of the hierarchical index and the pyramid index;

[0053] The coordinates of the remaining triangles of the seed block are:

[0054] p2=p1+(0,s min )

[0055] p3=p1+(0,s min )

[0056] p4=p1+(s min ,s min )

[0057] Among them, s min is the minimum block size.

[0058] Furthermore, in step S107, a morphological operation based on a minimum block is used to detect the edge range, and the pixels contained therein are refined to achieve accurate edge detection, including the following steps:

[0059] S1071: converting the preliminary segmentation result into a region mask, where each pixel in the mask corresponds to a minimum block;

[0060] S1072: Use a 3×3 kernel to erode and dilate each mask, discarding the completely eroded area;

[0061] S1073: Calculate the difference between the erosion mask and the dilation mask, and save the difference into the boundary mask;

[0062] S1074: Calculate the distances from all points to the regional plane within the boundary mask, and if the distance of a pixel is less than a predetermined distance, assign the pixel to a corresponding region;

[0063] S1075: When a cell is shared by multiple regions, the distance from the point to the plane is calculated for each region and the pixel is assigned to the region with the smallest distance.

[0064] In a preferred embodiment of the present invention, compared with the prior art, the present invention has the following beneficial effects:

[0065] 1. The present invention uses fewer plane fitting operations on a large plane area through adaptive block division, represents the plane area as multiple plane blocks as large as possible, reduces the number of plane blocks involved in the calculation, significantly reduces the calculation time of the block division step, and reduces the calculation cost;

[0066] 2. The present invention uses a block state storage pyramid to record the block state of the entire image, uses the query method of "virtual minimum block"-"actual neighbor block", and can obtain the state of the neighbor block through one access. The region growing algorithm can be implemented in data of irregular size;

[0067] 3. The present invention can preferentially fit parameters on a large range of blocks through a top-to-bottom block division method, which can reduce noise interference and has significantly better anti-noise performance than other methods.

[0068] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 is a flow chart of a three-dimensional plane rapid detection and segmentation method according to an embodiment of the present invention;

[0070] Figure 2 is a schematic diagram of an algorithm flow of an embodiment of the present invention;

[0071] Figure 3 is a schematic diagram of the correspondence between pixels and regions between two layers in a pyramid according to an embodiment of the present invention;

[0072] Figure 4 is a schematic diagram of a hierarchical index of region i0 according to an embodiment of the present invention;

[0073] Figure 5 It is a schematic diagram of the algorithm principle of the region growing step of an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0075] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.

[0076] like Figure 1 , Figure 2 As shown, an embodiment of the present invention provides a three-dimensional plane rapid detection and segmentation method based on depth map data, comprising the following steps:

[0077] S101: Convert the acquired depth image into point cloud data, downsample the point cloud data, and construct a data pyramid and a state pyramid based on the downsampled point cloud data.

[0078] The incoming depth image data needs to be converted into a point cloud before processing, and the point cloud needs to be downsampled, and a pyramid is built based on the downsampled point cloud.

[0079] The specific steps include the following sub-steps:

[0080] S1011: converting the depth image into point cloud data, wherein the point cloud data and the depth image maintain the same matrix size;

[0081] S1012: downsampling the point cloud data to create a data pyramid;

[0082] S1013: storing the point cloud data in a data pyramid, wherein the original point cloud data is stored in the 0th layer of the data pyramid, and the width and height of the coordinate matrix in subsequent layers of the data pyramid are halved layer by layer;

[0083] S1014: Storing the state information of the algorithm in the state pyramid. The state pyramid has the same data structure and indexing method as the data pyramid.

[0084] The constructed pyramid includes a data pyramid and a state pyramid. The data pyramid stores point cloud data, and the state pyramid is used to store the state information of the algorithm and the detailed state information of the plane block, including plane parameters, pixel area of ​​the plane, number of layers, etc.

[0085] The state pyramid has the same data structure and indexing method as the data pyramid, which facilitates cross-layer information recording.

[0086] In this embodiment, the correspondence between pixels and regions at different layers is established in the data pyramid, and each block is represented by a pyramid index:

[0087] The pixel at position [r, c] in layer l corresponds to the pixel at position [2r, 2c] in layer l-1;

[0088] The pyramid index is represented as follows:

[0089] ip = [l, r, c, s];

[0090] Index i p The corresponding area in the l-1th layer is: [l-1, 2r, 2c, 2s];

[0091] The area contains four sub-blocks, which are:

[0092] [l-1, 2r, 2c, s], [l-1, 2r+s, 2c, s], [l-1, 2r, 2c+s, s] and [l-1, 2r+s, 2c+s, s];

[0093] Among them, l represents the layer number, (r, c) represents the pixel coordinates, s represents the size of the block, and i p Pyramid index.

[0094] S103: Obtain an adaptive plane block by using an adaptive plane block segmentation method through top-down data access and parameter fitting, and store detailed information of the plane block in a state pyramid.

[0095] In this embodiment, the depth image is divided into a plurality of square regions, each of which has the same size, and a hierarchical index is used to represent the position and size of a block in the depth image, where the index value corresponds to the relative spatial position;

[0096] For any block with a hierarchical index, the sampling points are extracted directly from the data pyramid using the pyramid index, where

[0097] The hierarchical index is: i h =[i0,i1,...,i n ], (n≤N),

[0098] The pyramid index is: i p =[Nn, r, c, s min ],

[0099] The following conversion formula is used to complete the conversion between hierarchical index and pyramid index:

[0100]

[0101]

[0102] Among them, i h is the hierarchical index, i p is the pyramid index, i0 represents the initial partition area where a block is located, and the remaining indices [i1, ..., i n ] represents the relative position in the area, 0, 1, 2, 3 correspond to the upper left, upper right, lower left, and lower right sub-blocks respectively, s init is the initial block size, s min is the minimum block size, j, n is the number of layers, N is the maximum number of layers, N = log2(s init / s min ), (r, c) is the pixel coordinate, ω0 is the width of the initial block, is the floor operator.

[0103] The adaptive plane block segmentation method adopts a recursive method to perform adaptive plane block segmentation, performs plane fitting through the principal component analysis method, and uses the plane normal vector as the eigenvector corresponding to the minimum eigenvalue in the principal component analysis method. The plane condition is that the block performing plane fitting is a continuous block and the plane normal vector of the block is lower than the preset threshold.

[0104] The adaptive plane block segmentation method includes the following steps:

[0105] S1031: Use hierarchical index i to obtain point set data D from data pyramid D i , use queue Q to represent the area to be processed;

[0106] S1032: Check each index in the queue Q in turn to determine whether the plane area corresponding to the index satisfies the plane condition;

[0107] S1033: Record the block information that meets the plane condition into the state pyramid, and add the four sub-blocks of the block that does not meet the plane condition into the queue Q;

[0108] S1034: Determine whether the data in queue Q has been processed. If not, continue processing.

[0109] S105: using a block-level region growing method, marking adjacent and coplanar block-level regions as connected plane regions, thereby merging coplanar plane blocks.

[0110] The block-level region growing method uses region growing to merge coplanar planar blocks and adopts a breadth-first search strategy to expand the search area, including the following sub-steps:

[0111] S1051: Select the largest unmarked and unvisited block from the state pyramid S as a seed point, add the seed point to the queue Q, and store the block index in the current plane area into the set R;

[0112] S1052: For each patch q∈Q, calculate the set of patches N adjacent to patch q and sort them by area to ensure that region growth prioritizes larger patches;

[0113] S1053: Check each adjacent patch n∈N to determine whether n is coplanar with q. If so, add patch n to R and Q.

[0114] S1054: Obtain the indexes of all blocks in R, and store the plane parameters and labels in the state pyramid S according to the indexes, thus completing the growth of a region;

[0115] S1055: Repeatedly select new seed points to start the next region growth until there are no new seed points to choose from.

[0116] In this embodiment, since the block size is variable, the traditional neighbor search method cannot be directly applied. In order to solve this problem, this method designs a dedicated neighbor search method to optimize the traditional neighbor search method. The optimized neighbor search method realizes the neighbor search for variable-sized blocks. The optimized neighbor search method includes the following steps:

[0117] S1: Get a seed block and calculate the four corner coordinates of the seed block in the Nn layer according to the hierarchical index of the seed block;

[0118] S2: Calculate the coordinates of the four corners in layer 0: p′ i =2 N-n p i ;

[0119] S3: Get four edge segments of the seed block: p′1p′2, p′2p′3, p′3p′4, p′1p′4. The neighbors of the seed block are adjacent to the four edges of the seed block.

[0120] S4: Add row deviation or column deviation to the four edges of the seed block to obtain the index of the neighbor block;

[0121] S5: Use the index to directly access the neighbor block information in the state pyramid S.

[0122] When specifically calculating the four-corner coordinates of the seed block in the Nn layer, the following calculation method is used:

[0123] The upper left corner coordinate p1 of the seed block is calculated by the conversion formula of the hierarchical index and the pyramid index:

[0124] The following conversion formula is used to complete the conversion between hierarchical index and pyramid index:

[0125]

[0126]

[0127] The coordinates of the remaining triangles of the seed block are:

[0128] p2=p1+(0,s min )

[0129] p3=p1+(0,s min )

[0130] p4=p1+(s min ,s min )

[0131] Among them, i h is the hierarchical index, i p is the pyramid index, i0 represents the initial partition area where a block is located, and the remaining indices [i1, ..., i n ] represents the relative position in the area, 0, 1, 2, 3 correspond to the upper left, upper right, lower left, and lower right sub-blocks respectively, s init is the initial block size, s min is the minimum block size, j, n is the number of layers, N is the maximum number of layers, N = log2(s init / s min ), (r, c) is the pixel coordinate, ω0 is the width of the initial block, is the floor operator.

[0132] S107: Calculate the distance from the pixel point to the surface of each region edge, assign pixel-level labels according to the calculation results, assign pixels to the region with the smallest distance, and output the segmentation result.

[0133] The results of block-by-block segmentation are often rough, and in some cases, accurate edge detection is required. To this end, a morphological operation based on the smallest block is used to detect the edge range and refine the pixels contained therein to achieve accurate edge detection, including the following steps:

[0134] S1071: converting the preliminary segmentation result into a region mask, where each pixel in the mask corresponds to a minimum block;

[0135] S1072: Use a 3×3 kernel to erode and dilate each mask, discarding the completely eroded area;

[0136] S1073: Calculate the difference between the erosion mask and the dilation mask, and save the difference to the boundary mask;

[0137] S1074: Calculate the distances from all points to the region plane within the boundary mask, and if the distance of a pixel is less than a predetermined distance, assign the pixel to the corresponding region;

[0138] S1075: When a cell is shared by multiple regions, the distance from the point to the plane is calculated for each region and the pixel is assigned to the region with the smallest distance.

[0139] Compared with the prior art, the three-dimensional plane rapid detection and segmentation method based on depth map data provided by the embodiment of the present invention has the following advantages:

[0140] 1. When the existing methods are deployed on edge devices, there are problems of computational redundancy and poor time performance. The adaptive depth map blocking algorithm provided by the present invention uses fewer plane fitting operations on large plane areas through adaptive blocking, represents the plane area as multiple plane blocks that are as large as possible, reduces the number of plane blocks involved in the calculation, significantly reduces the number of plane fitting operations, significantly reduces the calculation time of the blocking step, and reduces the calculation cost.

[0141] 2. Aiming at the problem that the existing region growing methods cannot be adapted to data of irregular size, the present invention provides a region growing algorithm under irregular block representation, which uses a block state storage pyramid to record the block state of the entire image, and uses the query method of "virtual minimum block" - "actual neighbor block" to obtain the state of the neighbor block through one access. The region growing algorithm can be implemented in data of irregular size.

[0142] 3. In view of the problem that existing methods are susceptible to noise interference, the present invention adopts a pyramid representation of the original data and a top-down blocking algorithm. The measurement noise in the depth map is usually a fixed probability distribution. Through the top-to-bottom blocking method, the algorithm can preferentially fit parameters on a large range of blocks, which can reduce noise interference. On real data, the noise resistance performance of this method is significantly better than other methods.

[0143] The present invention is described in detail below in conjunction with the preferred embodiments of the present invention.

[0144] The flowchart of this method is as follows Figure 1 and Figure 2 As shown, it includes four main steps:

[0145] Step 1: Convert the input depth image into a point cloud and downsample it to build a data pyramid;

[0146] Step 2: Obtain the adaptive plane block through top-down data access and parameter fitting, and the detailed information of the block is stored in the state pyramid;

[0147] Step 3: Region growing marks adjacent and coplanar blocks as connected planar regions;

[0148] Step 4: Assign pixel-level labels by calculating the point-to-surface distance of pixels at the edge of each region.

[0149] Each step in this preferred embodiment is described in detail below in conjunction with the accompanying drawings.

[0150] Step 1: Build the Pyramid

[0151] First, the single-channel input depth image of size w·h is converted into a point cloud.

[0152] The point cloud and the image have the same matrix size, that is, three w·h matrices are used to represent the x, y, and z coordinates. In this representation, the image and point cloud storage locations correspond one-to-one, which can achieve efficient block access.

[0153] Subsequently, the point cloud is downsampled to create a data pyramid, where the original point cloud is stored in layer 0, and the width and height of the x, y, and z matrices of subsequent layers are halved layer by layer. This pyramid facilitates the establishment of corresponding relationships between pixels and regions in different layers. Figure 3 As shown, the pixel at position [r, c] in the lth layer corresponds to the pixel at position [2r, 2c] in the l-1th layer. Each block can be represented by a pyramid index i p =[l, r, c, s], where l represents the layer number, (r, c) represents the upper left corner pixel coordinates, and s represents the size of the block. p The corresponding area in the l-1th layer can be represented by [l-1, 2r, 2c, 2s], which contains four sub-blocks, namely, [l-1, 2r, 2c, s], [l-1, 2r+s, 2c, s], [l-1, 2r, 2c+s, s] and [l-1, 2r+s, 2c+s, s].

[0154] The state pyramid is used to store the state information of the algorithm. It has the same data structure and indexing method as the data pyramid, which facilitates cross-layer information recording.

[0155] Step 2: Adaptive Planar Block Segmentation

[0156] This method uses the hierarchical index i h =[i0,i1,...,i n ](n≤N) represents the position and size of the block in the image, the index value corresponds to the relative spatial position, and N is the maximum number of layers. Specifically, first divide the w·h matrix into w0·h0 square regions, each of which is s in size. init =w / w0=h / h0. i0 represents the initial partition area where a block is located, and the remaining indexes [i1, i2, ..., i n] represents the relative position in the region (0, 1, 2, 3 correspond to the upper left, upper right, lower left, and lower right sub-blocks, respectively. Figure 4 N is represented by s init and minimum block size s min Determine, the calculation method is N = log2 (s init / s min ). For example, given a 640×480 input depth image, the initial block is 4×3, and the initial block size is s init =160. When setting s min =5, N = 5. In order to reduce redundancy, this method uses a fixed number of s in plane fitting. min 2 points.

[0157] Specifically, for any level index i h =[i0,i1,...,i n ], the sampling points are directly taken from the data pyramid using the pyramid index i p =[Nn, r, c, s min ] Extraction, the conversion formula of the two indexes is as follows:

[0158]

[0159] Among them, i h is the hierarchical index, i p is the pyramid index, i0 represents the initial partition area where a block is located, and the remaining indices [i1, ..., i n ] represents the relative position in the area, 0, 1, 2, 3 correspond to the upper left, upper right, lower left, and lower right sub-blocks respectively, s init is the initial block size, s min is the minimum block size, j, n is the number of layers, N is the maximum number of layers, N = log2(s init / s min ), (r, c) is the pixel coordinate, ω0 is the width of the initial block, is the floor operator.

[0160] In this embodiment, a recursive method is used to perform adaptive plane block segmentation. The input data is the aforementioned data pyramid D, D i Represents the point set data obtained from the pyramid using the hierarchical index i. A queue Q is used to represent the area to be processed. The algorithm checks each index in the queue in turn to determine whether the plane area corresponding to the index meets the plane condition. If the condition is met, the block information is recorded in the state pyramid; if not, the four sub-blocks of this block are added to the queue for further processing.

[0161]

[0162] As shown in Algorithm 1 above, the adaptive plane block segmentation algorithm starts with a hierarchical index of length 1 and first performs a data continuity check. For continuous blocks, plane fitting is performed through principal component analysis (PCA), where the mean square error (MSE) is the minimum eigenvalue and the plane normal vector is the eigenvector corresponding to the minimum eigenvalue. For MSE above a threshold ∈ th The algorithm adds its four sub-blocks to the processing queue Q and recursively performs the same calculation on the sub-blocks until the maximum number of pyramid levels is reached. When a planar block is found, the relevant information of the block (such as plane parameters, number of levels, etc.) will be stored in the state pyramid for use in subsequent steps.

[0163] Step 3: Block-level region growing

[0164] Region growing is used to merge coplanar planar blocks. The process starts with seed selection. The method starts from the largest unmarked and unvisited block and expands the search area using a breadth-first search strategy.

[0165]

[0166] As shown in Algorithm 2, the input data of the block-level region growing algorithm is the state pyramid S. First, the largest unmarked and unvisited block is selected as the seed point and added to the processing queue Q. The set R is used to store the block index in the current plane region. For each patch q∈Q, the set of patches N adjacent to it is calculated and sorted by area to ensure that the region growth gives priority to larger blocks. Next, check each adjacent patch n∈N, and if n is coplanar with q, add it to R and Q. This process continues until Q is empty. At this time, the obtained R is all the block indexes of a region. According to these indexes, the plane parameters and labels are stored in the state pyramid S, and the growth of a region is completed. Repeatedly select new seed points and start the next region growth until there are no new seed points to select.

[0167] like Figure 5 As shown in Figure 1, an example of block-level region growing is shown. In this method, since the block size is variable, the traditional neighbor search method cannot be directly applied. To solve this problem, this method designs a dedicated neighbor search method. Given a block with a hierarchical index of i h =[i0,i1,…,i n ], first calculate the coordinates of the four corners in the Nn layer where it is located, where the coordinate p1 of the upper left corner is obtained by the index conversion formula described in step 2, and the coordinates of the remaining triangles are: p2 = p1 + (0, s min ), p3=p1+(0,s min ), p4=p1+(smin ,s min ). Then calculate the coordinates of the four corners in layer 0, which can be obtained by p i ′=2 N-n p i Calculation. The four edge segments of the seed block can be represented by p1′p2′, p2′p3′, p3′p4′, p1′p4′, and the neighbors of the seed block must be adjacent to these four edges.

[0168] Figure 5 The middle figures (a), (b), and (c) show the region growing process from a large seed block to a minimum block (from A to B). After obtaining the line segment expressions of the four edges, adding row bias or column bias b to the four edges can obtain the neighbor block index. For the top and left edges, b = -s min , for the right and bottom edges, b = 0. Using these indices, the neighbor block information can be directly accessed in the state pyramid S.

[0169] Figure 5 (d), (e) and (f) show the region growing process from a small seed block to a large block (from C to D). First, the minimum block index (i.e., block G) is obtained, which is called the virtual minimum block, but these blocks cannot be guaranteed to be consistent with the actual block segmentation state (i.e., block D). Then, the segmentation state stored in G is checked to obtain the source layer l of the actual block, and the actual neighbor index is the first l-bit index of the virtual minimum block.

[0170] Step 4: Pixel-by-Pixel Boundary Refinement

[0171] The results of block-by-block segmentation are often rough, and in some cases, accurate edge detection is required. To this end, this embodiment uses morphological operations based on minimum blocks to detect edge ranges and refine the pixels contained therein. The segmentation result is first converted into a region mask M, where each pixel in the mask corresponds to a minimum block. Next, a 3×3 kernel is used to erode and dilate each mask M, and the completely eroded area will be discarded. Boundary mask M B is defined as the difference between the erosion and dilation masks. B The distance from all points to the plane of the region is calculated. If the distance of the pixel is less than the plane MSE multiplied by the constant k, it is assigned to the region. For example, k=9 is used in this embodiment. When a cell is shared by multiple regions, the distance from the point to the plane is calculated for each region, and the pixel is assigned to the region with the smallest distance.

[0172] The three-dimensional plane rapid detection and segmentation method based on depth map data provided by the present invention not only has high real-time performance, but also has good segmentation effect. When the input image is 640*480, the processing frame rate of this method on a consumer-grade CPU can reach 700fps, which is more than 1.6 times higher than the existing best method; the segmentation performance comparison test on the data set shows that the accuracy of this method can reach 0.58 and the recall rate can reach 0.61, which is more than 2 times higher than the existing method. Therefore, this method can be integrated into the end-side processing chip of the depth camera, so that the camera can output plane information (such as the pixel area of ​​the plane, plane parameters and other data) while outputting the depth map. Subsequent applications can be secondary developed based on the plane information output by the camera, which can not only reduce the development workload and difficulty of subsequent applications, but also improve the overall real-time performance of the system, providing a more convenient and efficient perception basis for various robots and intelligent devices.

[0173] In addition, in the field of mobile robot applications, this technical solution can be used as a functional module of the perception algorithm. By providing accurate plane detection results, the robot can quickly parse the geometric structure of the surrounding environment. For example, in indoor scenes, the robot can accurately identify basic plane elements such as the ground, walls, and desktops, and this information can be directly used in subsequent autonomous navigation and path planning modules. In industrial robot applications, accurate plane information enables the robot to accurately locate the grasping plane of the workpiece, thereby achieving reliable grasping operations. In the field of warehousing and logistics, robots can complete complex operation tasks such as sorting and palletizing more efficiently based on the detected plane features of the goods. With the rapid development of the robot and intelligent manufacturing industries, the demand for environmental perception capabilities will continue to grow. As a basic perception algorithm, this method is conducive to improving system performance and reducing costs.

[0174] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A three-dimensional plane rapid detection and segmentation method based on depth map data, characterized in that: The method comprises the following steps: S101: converting the acquired depth image into point cloud data, downsampling the point cloud data, and constructing a data pyramid and a state pyramid based on the downsampled point cloud data; S103: obtaining an adaptive plane block by using an adaptive plane block segmentation method through top-down data access and parameter fitting, and storing detailed information of the plane block in the state pyramid; S105: using a block-level region growing method, marking adjacent and coplanar block-level regions as connected plane regions, thereby merging coplanar plane blocks; S107: Calculate the distance from the pixel point to the surface of each region edge, assign pixel-level labels according to the calculation results, assign pixels to the region with the smallest distance, and output the segmentation result.

2. The method according to claim 1, characterized in that The step S101 includes the following sub-steps: S1011: Convert the depth image into the point cloud data, wherein the point cloud data and the depth image maintain the same matrix size; S1012: downsampling the point cloud data to create the data pyramid; S1013: Storing the point cloud data in the data pyramid, wherein the 0th layer of the data pyramid stores the original point cloud data, and the width and height of the coordinate matrix in subsequent layers of the data pyramid are halved layer by layer; S1014: Storing state information of the algorithm in the state pyramid, wherein the state pyramid and the data pyramid have the same data structure and indexing method.

3. The method according to claim 2, characterized in that The data pyramid establishes the correspondence between pixels and regions at different layers, and each block is represented by a pyramid index: The pixel at position [r,c] in layer l corresponds to the pixel at position [2r,2c] in layer l-1; The pyramid index is represented as follows: i p =[l,r,c,s]; Index i p The corresponding area in the l-1th layer is: [l-1,2r,2c,2s]; The area includes four sub-blocks, which are: [l-1,2r,2c,s],[l-1,2r+s,2c,s],[l-1,2r,2c+s,s] and [l-1,2r+s,2c+s,s]; Among them, l represents the layer number, (r,c) represents the pixel coordinates, s represents the size of the block, and i p The pyramid index.

4. The method according to claim 3, characterized in that In the step S103, the depth image is divided into a plurality of square regions, each of which has the same size, and a hierarchical index is used to represent the position and size of a block in the depth image, where the index value corresponds to a relative spatial position; For any block with a hierarchical index, the sampling points are directly extracted from the data pyramid using the pyramid index, where The hierarchical index is: i h =[i0,i1,…,i n ],(n≤n), The pyramid index is: i p =[Nn,r,c,s min ], The following conversion formula is used to complete the conversion between the hierarchical index and the pyramid index: Among them, i h is the hierarchical index, i p is the pyramid index, i0 represents the initial partition area where a block is located, and the remaining indices [i1,…,i n ] represents the relative position in the area, 0, 1, 2, 3 correspond to the upper left, upper right, lower left, and lower right sub-blocks respectively, s init is the initial block size, s min is the minimum block size, j, n is the number of layers, N is the maximum number of layers, N = log2 (s init / s min ), (r, c) is the pixel coordinate, ω0 is the width of the initial block, is the floor operator.

5. The method according to claim 4, characterized in that The adaptive plane block segmentation method adopts a recursive method to perform adaptive plane block segmentation, and includes the following steps: S1031: Obtaining point set data D from the data pyramid D using the hierarchical index i i , use queue Q to represent the area to be processed; S1032: Check each index in the queue Q in turn to determine whether the plane area corresponding to the index satisfies the plane condition; S1033: Recording the block information that meets the plane condition into the state pyramid, and adding the four sub-blocks of the block that does not meet the plane condition into the queue Q; S1034: Determine whether the data in the queue Q has been processed. If not, continue processing.

6. The method according to claim 5, characterized in that The adaptive plane block segmentation method performs plane fitting through the principal component analysis method, and uses the plane normal vector as the eigenvector corresponding to the minimum eigenvalue in the principal component analysis method. The plane condition is that the block on which the plane fitting is performed is a continuous block and the plane normal vector of the block is lower than a preset threshold.

7. The method according to claim 6, characterized in that In step S105, the block-level region growing method uses region growing to merge coplanar planar blocks and adopts a breadth-first search strategy to expand the search area, including the following sub-steps: S1051: Select the largest unmarked and unvisited block from the state pyramid S as a seed point, add the seed point to the queue Q, and store the block index in the current plane area into the set R; S1052: For each patch q∈Q, calculate a set of patches N adjacent to the patch q, and sort them by area to ensure that region growth gives priority to larger patches; S1053: Check each adjacent patch n∈N to determine whether N is coplanar with q. If so, add patch n to R and Q. S1054: Obtain the indexes of all blocks in R, and store the plane parameters and labels in the state pyramid S according to the indexes, thereby completing the growth of a region; S1055: Repeatedly select new seed points to start the next region growth until there are no new seed points to choose from.

8. The method according to claim 7, characterized in that The block-level region growing method optimizes the traditional neighbor search method. The optimized neighbor search method realizes neighbor search for blocks of variable sizes. The optimized neighbor search method includes the following steps: S1: Get a seed block, and calculate the four corner coordinates of the seed block in the Nn layer according to the hierarchical index of the seed block; S2: Calculate the coordinates of the four corners in layer 0: p′ i =2 N-n p i ; S3: Obtain four edge segments of the seed block: p′1p′2, p′2p′3, p′3p′4, p′1p′4, where the neighbors of the seed block are adjacent to the four edges of the seed block; S4: adding row deviations or column deviations to four edges of the seed block to obtain indexes of neighboring blocks; S5: Use the index to directly access the neighbor block information in the state pyramid S.

9. The method according to claim 8, characterized in that In step S1, the four corner coordinates of the seed block in the Nn layer are calculated as follows: The upper left corner coordinate p1 of the seed block is calculated by the conversion formula of the hierarchical index and the pyramid index; The coordinates of the remaining triangles of the seed block are: p2=p1+(0,s min ) p3=p1+(0,s min ) p4=p1+(s min ,s min ) Among them, s min is the minimum block size.

10. The method according to claim 9, characterized in that In step S107, a morphological operation based on a minimum block is used to detect the edge range, and the pixels contained therein are refined to achieve accurate edge detection, including the following steps: S1071: converting the preliminary segmentation result into a region mask, where each pixel in the mask corresponds to a minimum block; S1072: Use a 3×3 kernel to erode and dilate each mask, discarding the completely eroded area; S1073: Calculate the difference between the erosion mask and the dilation mask, and save the difference into the boundary mask; S1074: Calculate the distances from all points to the regional plane within the boundary mask, and if the distance of a pixel is less than a predetermined distance, assign the pixel to a corresponding region; S1075: When a cell is shared by multiple regions, the distance from the point to the plane is calculated for each region and the pixel is assigned to the region with the smallest distance.

Images