A three-dimensional plane fast detection and segmentation method based on depth map data

Abstract

The application discloses a kind of three-dimensional plane fast detection segmentation methods based on depth map data, it is related to robot field, including the following steps: depth image is converted into point cloud and is reduced sampling, pyramid is constructed based on point cloud;Through top-down data access and parameter fitting, adaptive plane block segmentation method is used to obtain adaptive plane block, and detailed information of plane block is stored in state pyramid;Using block level region growing method, adjacent and coplanar block level region is marked as connected plane region, and the merging of plane block is realized;The pixel distance of the edge of each region is calculated to the surface, pixel level label assignment is carried out, pixel is assigned to the region with minimum distance, and the segmentation result is output.The application uses less plane fitting operation on large area plane region by adaptive blocking, represents plane region as multiple as possible plane blocks, reduces the number of plane blocks participating in operation, reduces the cost of calculation, has high real-time and good segmentation effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robots, in particular to a three-dimensional plane rapid detection and segmentation method based on depth map data. BACKGROUND

[0002] Depth cameras have been widely used in the field of robots, which can obtain high frame rate and high resolution depth images of the scene, so that the robot can realize accurate 3D perception function. The 3D plane segmentation technology is the key to the robot to understand and analyze the 3D geometry of the scene in the depth image, which can provide basic information for visual odometry, scene understanding, sensor calibration and object pose estimation and other functions.

[0003] At present, the plane segmentation method can be mainly divided into four categories, RANSAC-based method, Hough transform-based method, clustering-based method and deep learning-based method. The commonly used RANSAC-based method is simple and easy to use, but it needs to repeat random sampling, parameter fitting and result verification, resulting in high computational cost. The Hough transform-based method performs well in dealing with noise, but it is often difficult to find a good balance between accuracy and computational cost. The clustering-based method has fast processing speed, but there is still room for improvement in accuracy. Although the deep learning-based method performs well in general image segmentation and object detection, such method has great demand for hardware computing power and large amount of training data, and thus is not suitable for deployment and application on edge devices.

[0004] Each type of plane segmentation method has its own advantages and disadvantages. For the scene requiring real-time performance on edge devices, how to balance the accuracy, computational cost and ease of use is still a problem to be solved.

[0005] Therefore, the skilled in the art is committed to developing a three-dimensional plane rapid detection and segmentation method based on depth map data. SUMMARY

[0006] In view of the above defects of the prior art, the technical problem to be solved by the present application is that the existing method has computational redundancy and poor performance when deployed on edge devices.

[0007] To achieve the above purpose, the present application provides a three-dimensional plane rapid detection and segmentation method based on depth map data, which comprises the following steps:

[0008] S101: converting the collected depth image into point cloud data, and performing down-sampling on the point cloud data, and constructing a data pyramid and a state pyramid based on the down-sampled point cloud data;

[0009] S103: Obtain adaptive planar blocks by adaptive planar block segmentation method through top-down data access and parameter fitting, and store detailed information of the planar blocks in the state pyramid;

[0010] S105: Mark adjacent and coplanar block-level regions as connected planar regions by using block-level region growing method, and realize merging of coplanar planar blocks;

[0011] S107: Calculate pixel distance of each region edge to the plane, and perform pixel-level label assignment according to the calculation result, assign pixels to the region with the smallest distance, and output the segmentation result.

[0012] Further, the step S101 comprises the following sub-steps:

[0013] S1011: Convert the depth image into the point cloud data, which maintains the same matrix size as the depth image;

[0014] S1012: Perform down-sampling processing on the point cloud data to create the data pyramid;

[0015] S1013: Store the point cloud data in the data pyramid, the 0th layer of the data pyramid stores the original point cloud data, and the width and height of the coordinate matrix in the subsequent layers of the data pyramid are halved layer by layer;

[0016] S1014: Store the state information of the algorithm in the state pyramid, and the state pyramid has the same data structure and indexing method as the data pyramid.

[0017] Further, the data pyramid establishes a corresponding relationship between pixels and regions in different layers, and each block is represented by a pyramid index:

[0018] The pixel at position [r, c] in the lth layer corresponds to the pixel at position [2r, 2c] in the (l-1)th layer;

[0019] The pyramid index is represented as follows:

[0020] i p =[l, r, c, s];

[0021] Index i p The corresponding region in the (l-1)th layer is: [l-1, 2r, 2c, 2s];

[0022] The region contains four sub-blocks, which are:

[0023] [l-1, 2r, 2c, s], [l-1, 2r+s, 2c, s], [l-1, 2r, 2c+s, s] and [l-1, 2r+s, 2c+s, s];

[0024] Where l represents the layer number, (r, c) represents the pixel coordinates, s represents the block size, and i p For the pyramid index.

[0025] Further, in step S103, the depth image is divided into multiple square regions, each region being the same size, and a hierarchical index is used to represent the position and size of the blocks in the depth image, with the index value corresponding to the relative spatial position;

[0026] For any block with a hierarchical index, sampling points are extracted directly from the data pyramid using the pyramid index, where,

[0027] The hierarchical index is: i h = [i0, i1, ..., i n ],(n≤N),

[0028] The pyramid index is: i p =[Nn, r, c, s] min ],

[0029] The conversion between the hierarchical index and the pyramid index is performed using the following conversion formula:

[0030]

[0031]

[0032] Among them, i h For hierarchical indexing, i p For pyramid indexes, i0 represents the initial partition region where a block is located, and the remaining indices [i1, ..., i...] n The symbol ] represents the relative position within the area, with 0, 1, 2, and 3 corresponding to the top-left, top-right, bottom-left, and bottom-right sub-blocks, respectively. init s is the initial block size. min Let j and n be the minimum block size, j and n be the layer numbers, and N be the maximum layer number. N = log2(s) init / s min (r, c) are pixel coordinates, and ω0 is the width of the initial block. This is the floor operator.

[0033] Furthermore, the adaptive planar block segmentation method employs a recursive approach for adaptive planar block segmentation, including the following steps:

[0034] S1031: obtaining point set data D from the data pyramid D using a hierarchical index i i using a queue Q to represent the regions to be processed;

[0035] S1032: checking each index in the queue Q in turn to determine whether the planar region corresponding to the index satisfies a planar condition;

[0036] S1033: recording the block information that satisfies the planar condition to the state pyramid S and adding the four sub-blocks of the block that does not satisfy the planar condition to the queue Q;

[0037] S1034: determining whether the data in the queue Q has been processed, and if not, continuing to process.

[0038] Further, the adaptive planar block segmentation method performs planar fitting by principal component analysis, taking the planar normal vector as the eigenvector corresponding to the smallest eigenvalue in the principal component analysis, and the planar condition is that the block performing planar fitting is a continuous block and the planar normal vector of the block is lower than a preset threshold.

[0039] Further, in the step S105, the block-level region growing method uses region growing to merge coplanar planar blocks and adopts a breadth-first search strategy to expand the search region, including the following sub-steps:

[0040] S1051: selecting the largest unmarked and unvisited block from the state pyramid S as a seed point and adding the seed point to the queue Q and storing the block index in the current planar region to the set R;

[0041] S1052: for each patch q e Q, calculating a set N of patches adjacent to the patch q and sorting by area to ensure that region growing gives priority to larger blocks;

[0042] S1053: checking each adjacent patch n e N to determine whether n and q are coplanar, and if so, adding the patch n to R and Q;

[0043] S1054: obtaining the indices of all blocks in R and storing the planar parameters and labels in the state pyramid S according to the indices, completing the growth of one region;

[0044] S1055: repeating the selection of a new seed point to start the next region growing until there is no new seed point to select.

[0045] Further, the block-level region growing method optimizes the traditional neighbor search method, and the optimized neighbor search method realizes neighbor search for variable-size blocks, and the optimized neighbor search method includes the following steps:

[0046] S1: obtaining a seed block, and calculating the coordinates of the four corners in the N-n layer where the seed block is located according to the hierarchical index of the seed block;

[0047] S2: calculating the coordinates of the four corners in the 0th layer: p' i = 2 N-n p i ;

[0048] S3: obtaining four edge segments of the seed block: p'1p'2, p'2p'3, p'3p'4, and p'1p'4, the neighbors of the seed block being adjacent to the four edges of the seed block;

[0049] S4: adding row or column offsets to the four edges of the seed block to obtain the indexes of the neighbor blocks;

[0050] S5: directly accessing the neighbor block information in the state pyramid S using the indexes.

[0051] Further, in the step S1, the coordinates of the four corners in the N-n layer where the seed block is located are calculated in the following manner:

[0052] The coordinates of the upper left corner p1 of the seed block are calculated by a conversion formula of the hierarchical index and the pyramid index;

[0053] The coordinates of the remaining three corners of the seed block are:

[0054] p2 = p1 + (0, s min )

[0055] p3 = p1 + (0, s min )

[0056] p4 = p1 + (s min , s min )

[0057] where s min is the minimum block size.

[0058] Further, in the step S107, a morphological operation based on the minimum block is used to detect the edge range and refine the pixels contained therein, so as to achieve accurate edge detection, including the following steps:

[0059] S1071: converting the preliminary segmentation result into a region mask, each pixel in the mask corresponding to a minimum block;

[0060] S1072: performing erosion and expansion on each mask using a 3x3 kernel, and discarding the completely eroded regions;

[0061] S1073: Calculate the difference between the erosion mask and the dilation mask, and save the difference to the boundary mask;

[0062] S1074: Calculate the distance of all points in the boundary mask to the region plane, if the distance of a pixel is less than a predetermined distance, assign the pixel to the corresponding region;

[0063] S1075: When a cell is shared by multiple regions, calculate the distance of the point to the plane for each region, and assign the pixel to the region with the smallest distance.

[0064] In the preferred embodiment of the present application, compared with the prior art, the present application has the following beneficial effects:

[0065] 1. The present application uses fewer plane fitting operations on large area plane regions by adaptive blocking, represents the plane region as a plurality of plane blocks as large as possible, reduces the number of plane blocks participating in the operation, and significantly reduces the calculation time of the blocking step, thereby reducing the calculation cost.

[0066] 2. The present application records the blocking state of the entire image by using a blocking state storage pyramid, uses a "virtual minimum block" - "actual neighbor block" query method, and obtains the state of the neighbor block by one access, so that the region growing algorithm can be implemented in irregular size data.

[0067] 3. The present application can preferentially fit parameters on a large range of blocks by a top-down blocking method, can reduce the interference of noise, and has significantly better noise resistance than other methods.

[0068] The concept, specific structure and technical effects of the present application will be further described below with reference to the accompanying drawings, so as to fully understand the purpose, features and effects of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 is a flow chart of the three-dimensional plane rapid detection and segmentation method of the embodiment of the present application;

[0070] Figure 2 is a schematic diagram of the algorithm flow of the embodiment of the present application;

[0071] Figure 3 is a schematic diagram of the correspondence relationship between pixels and regions in two layers in the pyramid of the embodiment of the present application;

[0072] Figure 4 is a schematic diagram of the hierarchical index of region i0 of the embodiment of the present application;

[0073] Figure 5 is a schematic diagram of the region growing step algorithm principle of the embodiment of the present application. DETAILED DESCRIPTION

[0074] The technical content of the present application will be described in more detail below with reference to the accompanying drawings of the specification, so that the preferred embodiments of the present application can be more clearly understood and be convenient for understanding. The present application can be embodied in many different forms and the protection scope of the present application is not limited to the embodiments mentioned herein.

[0075] In the drawings, the same components have the same reference numerals, and components having similar structures or functions are denoted by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present application is not limited to the size and thickness of each component. In order to make the drawing clearer, the thickness of some components is appropriately exaggerated in some places in the drawing.

[0076] As shown in Figure 1 , Figure 2 , the three-dimensional plane fast detection and segmentation method based on depth map data provided by the embodiment of the present application comprises the following steps:

[0077] S101: converting the collected depth image into point cloud data, and performing down-sampling on the point cloud data, and constructing a data pyramid and a state pyramid based on the down-sampled point cloud data.

[0078] For the incoming depth image data, it needs to be converted into point cloud before processing, and the point cloud is down-sampled, and the pyramid is constructed based on the down-sampled point cloud.

[0079] The specific steps include the following sub-steps:

[0080] S1011: converting the depth image into point cloud data, the point cloud data and the depth image maintaining the same matrix size;

[0081] S1012: performing down-sampling processing on the point cloud data to create a data pyramid;

[0082] S1013: storing the point cloud data in the data pyramid, the 0th layer of the data pyramid storing the original point cloud data, and the width and height of the coordinate matrix in the subsequent layers of the data pyramid being halved layer by layer;

[0083] S1014: storing the state information of the algorithm in the state pyramid, the state pyramid having the same data structure and indexing method as the data pyramid.

[0084] In the constructed pyramid, including the data pyramid and the state pyramid, the data pyramid saves the point cloud data, and the state pyramid is used to store the state information of the algorithm and the detailed state information of the plane block, including, for example, the plane parameters, the pixel area of the plane, the layer number, etc.

[0085] The state pyramid and the data pyramid have the same data structure and indexing method, which facilitates cross-layer recording information.

[0086] In this embodiment, a correspondence between pixels and regions is established between different layers in the data pyramid, and each block is represented by a pyramid index:

[0087] The pixel at position [r, c] in layer l corresponds to the pixel at position [2r, 2c] in layer (l-1).

[0088] The pyramid index is represented as follows:

[0089] ip = [l, r, c, s];

[0090] Index i p The corresponding region in layer l-1 is: [l-1, 2r, 2c, 2s];

[0091] The region contains four sub-blocks, which are as follows:

[0092] [l-1, 2r, 2c, s], [l-1, 2r+s, 2c, s], [l-1, 2r, 2c+s, s] and [l-1, 2r+s, 2c+s, s];

[0093] Where l represents the layer number, (r, c) represents the pixel coordinates, s represents the block size, and i p For the pyramid index.

[0094] S103: Through top-down data access and parameter fitting, adaptive planar blocks are obtained using the adaptive planar block segmentation method, and the detailed information of the planar blocks is stored in the state pyramid.

[0095] In this embodiment, the depth image is divided into multiple square regions, each of the same size. A hierarchical index is used to represent the position and size of the blocks in the depth image, and the index value corresponds to the relative spatial position.

[0096] For any block with a hierarchical index, sampling points are extracted directly from the data pyramid using the pyramid index, where...

[0097] The hierarchical index is: i h = [i0, i1, ..., i n ],(n≤N),

[0098] The pyramid index is: i p =[Nn, r, c, s] min ],

[0099] The following conversion formula is used to convert between hierarchical indexes and pyramid indexes:

[0100]

[0101]

[0102] Among them, i h For hierarchical indexing, i p For pyramid indexes, i0 represents the initial partition region where a block is located, and the remaining indices [i1, ..., i...] n The symbol ] represents the relative position within the area, with 0, 1, 2, and 3 corresponding to the top-left, top-right, bottom-left, and bottom-right sub-blocks, respectively. init s is the initial block size. min Let j and n be the minimum block size, j and n be the layer numbers, and N be the maximum layer number. N = log2(s) init / s min (r, c) are pixel coordinates, and ω0 is the width of the initial block. This is the floor operator.

[0103] The adaptive planar block segmentation method uses a recursive approach to perform adaptive planar block segmentation. It performs planar fitting through principal component analysis and uses the planar normal vector as the eigenvector corresponding to the minimum eigenvalue in the principal component analysis. The planar condition is that the block to be fitted is a continuous block and the planar normal vector of the block is lower than a preset threshold.

[0104] The adaptive planar block segmentation method includes the following steps:

[0105] S1031: Use hierarchical index i to obtain the point set data D from the data pyramid D. i Queue Q is used to represent the region to be processed;

[0106] S1032: Check each index in queue Q in turn to determine whether the planar region corresponding to the index satisfies the planar condition;

[0107] S1033: Record the information of blocks that meet the planar conditions into the state pyramid, and add the four sub-blocks of blocks that do not meet the planar conditions to queue Q;

[0108] S1034: Determine whether the data in queue Q has been processed. If not, continue processing.

[0109] S105: Using the block-level region growth method, adjacent and coplanar block-level regions are marked as connected plane regions, thereby merging coplanar plane blocks.

[0110] The block-level region growing method uses region growing to merge coplanar planar blocks and employs a breadth-first search strategy to expand the search area, including the following sub-steps:

[0111] S1051: Select the largest unlabeled and unvisited patch from the state pyramid S as a seed point, and add the seed point to the queue Q, store the patch index in the current planar region to the set R;

[0112] S1052: For each patch q∈Q, calculate the patch set N adjacent to the patch q, and sort by area to ensure that region growing gives priority to larger patches;

[0113] S1053: Check each adjacent patch n∈N to determine whether n and q are coplanar, if coplanar, add the patch n to R and Q;

[0114] S1054: Get the index of all patches in R, and store the planar parameters and labels in the state pyramid S according to the index, complete a region growth;

[0115] S1055: Repeat the selection of new seed points to start the next region growth until there are no new seed points to select.

[0116] In this embodiment, since the block size is variable, the traditional neighbor search method cannot be directly applied. In order to solve this problem, a special neighbor search method is designed to optimize the traditional neighbor search method. The optimized neighbor search method realizes the neighbor search of the variable size block, and the optimized neighbor search method includes the following steps:

[0117] S1: Get a seed block, calculate the four corner coordinates in the N-n layer where the seed block is located according to the hierarchical index of the seed block;

[0118] S2: Calculate the coordinates corresponding to the four corners in the 0th layer: p′ i =2 N-n p i ;

[0119] S3: Get the four edge segments of the seed block: p′1p′2, p′2p′3, p′3p′4, p′1p′4, the neighbors of the seed block are adjacent to the four edges of the seed block;

[0120] S4: Add row offset or column offset to the four edges of the seed block to obtain the index of the neighbor block;

[0121] S5: Use the index to directly access the neighbor block information in the state pyramid S.

[0122] In the specific calculation of the four corner coordinates in the N-n layer where the seed block is located, the following calculation method is adopted:

[0123] The left upper corner coordinate p1 of the seed block is calculated by the conversion formula of the hierarchical index and the pyramid index:

[0124] The conversion formula is as follows:

[0125]

[0126]

[0127] The coordinates of the remaining three triangles of the seed block are as follows:

[0128] p2=p1+(0,s min )

[0129] p3=p1+(0,s min )

[0130] p4=p1+(s min ,s min )

[0131] Where i h is the hierarchical index, i p is the pyramid index, i0 represents the initial division region where a block is located, the remaining indexes [i1,..., i n ] represent the relative position in the region, 0, 1, 2, and 3 correspond to the upper left, upper right, lower left, and lower right sub-blocks, respectively, s init is the initial block size, s min is the minimum block size, j and n are the layer number, and N is the maximum number of layers, N = log2(s init / s min ), (r, c) is the pixel coordinate, ω0 is the width of the initial block, and is the floor operator.

[0132] S107: Calculate the pixel distance from the edge of each region to the surface, and assign the pixel-level label according to the calculation result. The pixel is assigned to the region with the smallest distance, and the segmentation result is output.

[0133] The result of block-by-block segmentation is often rough, and in some cases, accurate edge detection is required. Therefore, morphological operations based on the minimum block are used to detect the edge range and refine the pixels contained therein, to achieve accurate edge detection, including the following steps:

[0134] S1071: Convert the preliminary segmentation result to a region mask, where each pixel in the mask corresponds to a minimum block;

[0135] S1072: Use a 3x3 kernel to erode and dilate each mask, and discard the completely eroded regions;

[0136] S1073: Calculate the difference between the eroded mask and the expanded mask, and save the difference to the boundary mask;

[0137] S1074: Calculate the distance of all points to the region plane within the boundary mask, and assign the pixel to the corresponding region if the distance of the pixel is less than a predetermined distance;

[0138] S1075: When a cell is shared by multiple regions, the distance of the point to the plane will be calculated for each region, and the pixel will be assigned to the region with the smallest distance.

[0139] Compared with the prior art, the three-dimensional plane fast detection and segmentation method based on depth map data provided by the embodiment of the application has the following advantages:

[0140] 1. For the problem of calculation redundancy and poor time performance when the existing method is deployed on edge devices, the adaptive depth map blocking algorithm provided by the application uses fewer plane fitting operations on large area plane regions by adaptive blocking, represents the plane region as multiple plane blocks as large as possible, reduces the number of plane blocks participating in operation, and significantly reduces the number of plane fitting operations and the time spent in the blocking step, thereby reducing the calculation cost.

[0141] 2. For the problem that the existing region growing method cannot adapt to irregular size data, the region growing algorithm under irregular block representation provided by the application uses a blocking state storage pyramid to record the blocking state of the entire image, uses a "virtual minimum block"-"actual neighbor block" query method, and obtains the state of the neighbor block through one access. The region growing algorithm can be implemented in irregular size data.

[0142] 3. For the problem that the existing method is susceptible to noise interference, the application adopts a pyramid representation of the original data and a top-down blocking algorithm. For the measurement noise in the depth map, which is usually a fixed probability distribution, the algorithm can preferentially fit parameters on a large range of blocks through the top-down blocking method, which can reduce the interference of noise. In real data, the anti-noise performance of the method is significantly better than that of other methods.

[0143] The application will be described in detail below in conjunction with the preferred embodiments of the application.

[0144] The flowchart of the method is shown in Figure 1 and Figure 2 , which includes four main steps:

[0145] Step 1: Convert the input depth image into a point cloud and perform downsampling to construct a data pyramid;

[0146] Step 2: Obtain adaptive plane blocks through top-down data access and parameter fitting, and store the detailed information of the blocks in a state pyramid;

[0147] Step 3: Region growing marks adjacent and coplanar blocks as connected planar regions;

[0148] Step 4: Pixel-level label assignment is performed by calculating the point-to-plane distance of each region edge pixel.

[0149] Each step in the preferred embodiment is described in detail below in conjunction with the accompanying drawings.

[0150] Step 1: Building the pyramid

[0151] First, the input depth image of size w·h in a single channel is converted into a point cloud.

[0152] The point cloud and the image maintain the same matrix size, i.e., the x, y, and z coordinates are represented by three w·h matrices, respectively. In this representation, the image and the point cloud are one-to-one in storage location, enabling efficient block access.

[0153] Subsequently, the point cloud is down-sampled to create a data pyramid, where the original point cloud is stored in layer 0, and the x, y, and z matrix width and height are halved in each subsequent layer. This pyramid facilitates the establishment of pixel and region correspondence between different layers. As shown in Figure 3 , the pixel at position [r, c] in layer l corresponds to the pixel at position [2r, 2c] in layer l-1. Each block can be represented by the pyramid index i p =[l, r, c, s], where l represents the layer number, (r, c) represents the top-left pixel coordinates, and s represents the block size. p The corresponding region in layer l-1 can be represented by [l-1, 2r, 2c, 2s], which contains four sub-blocks: [l-1, 2r, 2c, s], [l-1, 2r+s, 2c, s], [l-1, 2r, 2c+s, s], and [l-1, 2r+s, 2c+s, s].

[0154] The state pyramid is used to store the state information of the algorithm, with the same data structure and indexing method as the data pyramid, facilitating cross-layer information recording.

[0155] Step 2: Adaptive planar block segmentation

[0156] This method uses hierarchical index i h =[i0, i1,..., i n ](n≤N) to represent the position and size of blocks in the image, with index values corresponding to relative spatial positions, and N being the maximum number of layers. Specifically, the w·h matrix is first divided into w0·h0 square regions, each with a size of s init =w / w0=h / h0. i0 represents the initial division region of a block, and the remaining indexes [i1, i2,..., in] represent the subsequent division regions. nrepresents the relative position within the region (0, 1, 2, 3 correspond to the top-left, top-right, bottom-left and bottom-right sub-blocks, respectively, as shown in Figure 4 N is determined by s init and the minimum block size s min , and is calculated as N = log2(s init / s min ). For example, given an input depth image of 640x480, the initial block size is s init = 160. When s min = 5, N = 5. To reduce redundancy, the method uses a fixed number of s min 2 points in the plane fitting.

[0157] Specifically, for any block with hierarchical index i h = [i0, i1,..., i n ], the sampling points are directly extracted from the data pyramid using pyramid index i p = [N-n, r, c, s min ], and the conversion formula between the two indices is as follows:

[0158]

[0159] where i h is the hierarchical index, i p is the pyramid index, i0 represents the initial division region where a block is located, and the remaining indices [i1,..., i n ] represent the relative position within the region, 0, 1, 2, 3 correspond to the top-left, top-right, bottom-left and bottom-right sub-blocks, respectively, s init is the initial block size, s min is the minimum block size, j and n are the layer numbers, N is the maximum number of layers, N = log2(s init / s min ), (r, c) are the pixel coordinates, ω0 is the width of the initial block, and is the floor operator.

[0160] In this embodiment, the adaptive plane block segmentation is performed in a recursive manner. The input data is the aforementioned data pyramid D, D i represents the point set data obtained from the pyramid using the hierarchical index i. A queue Q is used to represent the regions to be processed. The algorithm sequentially checks each index in the queue to determine whether the plane region corresponding to the index satisfies the plane condition. If the condition is satisfied, the block information is recorded in the state pyramid; if not, the four sub-blocks of the block are added to the queue for further processing.

[0161]

[0162] As shown in Algorithm 1, the adaptive planar block segmentation algorithm starts from the level index of length 1, and first performs data continuity check. For the continuous block, planar fitting is performed by principal component analysis (PCA), where the mean square error (MSE) is the smallest eigenvalue, and the planar normal vector is the eigenvector corresponding to the smallest eigenvalue. For the block with MSE higher than a threshold value th , the algorithm adds its four sub-blocks to the queue Q to be processed, and performs the same calculation on the sub-blocks recursively until the maximum pyramid level is reached. When a planar block is found, the relevant information of the block (such as planar parameters, the level it belongs to, etc.) will be stored in the state pyramid for subsequent steps to use.

[0163] Step 3: Block-level region growing

[0164] Region growing is used to merge coplanar planar blocks. The process starts from seed selection, and this method starts from the largest unmarked and unvisited block, and uses a breadth-first search strategy to expand the search area.

[0165]

[0166] As shown in Algorithm 2, the input data of the block-level region growing algorithm is the state pyramid S. First, the largest unmarked and unvisited block is selected as the seed point, and it is added to the processing queue Q. Set R is used to store the block indices in the current planar region. For each patch q ∈ Q, calculate the set of neighboring patches N adjacent to it, and sort them by area to ensure that region growing gives priority to larger blocks. Then, check each neighboring patch n ∈ N, if n is coplanar with q, add it to R and Q. This process continues until Q is empty, at which time the R obtained is all the block indices of a region, and the planar parameters, labels are stored in the state pyramid S according to these indices, i.e. the growth of a region is completed. Repeat the selection of new seed points to start the next region growth until there are no new seed points to choose.

[0167] As shown in Algorithm 2, the input data of the block-level region growing algorithm is the state pyramid S. First, the largest unmarked and unvisited block is selected as the seed point, and it is added to the processing queue Q. Set R is used to store the block indices in the current planar region. For each patch q ∈ Q, calculate the set of neighboring patches N adjacent to it, and sort them by area to ensure that region growing gives priority to larger blocks. Then, check each neighboring patch n ∈ N, if n is coplanar with q, add it to R and Q. This process continues until Q is empty, at which time the R obtained is all the block indices of a region, and the planar parameters, labels are stored in the state pyramid S according to these indices, i.e. the growth of a region is completed. Repeat the selection of new seed points to start the next region growth until there are no new seed points to choose. Figure 5 As shown in Algorithm 2, the input data of the block-level region growing algorithm is the state pyramid S. First, the largest unmarked and unvisited block is selected as the seed point, and it is added to the processing queue Q. Set R is used to store the block indices in the current planar region. For each patch q ∈ Q, calculate the set of neighboring patches N adjacent to it, and sort them by area to ensure that region growing gives priority to larger blocks. Then, check each neighboring patch n ∈ N, if n is coplanar with q, add it to R and Q. This process continues until Q is empty, at which time the R obtained is all the block indices of a region, and the planar parameters, labels are stored in the state pyramid S according to these indices, i.e. the growth of a region is completed. Repeat the selection of new seed points to start the next region growth until there are no new seed points to choose. h As shown in Algorithm 2, the input data of the block-level region growing algorithm is the state pyramid S. First, the largest unmarked and unvisited block is selected as the seed point, and it is added to the processing queue Q. Set R is used to store the block indices in the current planar region. For each patch q ∈ Q, calculate the set of neighboring patches N adjacent to it, and sort them by area to ensure that region growing gives priority to larger blocks. Then, check each neighboring patch n ∈ N, if n is coplanar with q, add it to R and Q. This process continues until Q is empty, at which time the R obtained is all the block indices of a region, and the planar parameters, labels are stored in the state pyramid S according to these indices, i.e. the growth of a region is completed. Repeat the selection of new seed points to start the next region growth until there are no new seed points to choose. n As shown in Algorithm 2, the input data of the block-level region growing algorithm is the state pyramid S. First, the largest unmarked and unvisited block is selected as the seed point, and it is added to the processing queue Q. Set R is used to store the block indices in the current planar region. For each patch q ∈ Q, calculate the set of neighboring patches N adjacent to it, and sort them by area to ensure that region growing gives priority to larger blocks. Then, check each neighboring patch n ∈ N, if n is coplanar with q, add it to R and Q. This process continues until Q is empty, at which time the R obtained is all the block indices of a region, and the planar parameters, labels are stored in the state pyramid S according to these indices, i.e. the growth of a region is completed. Repeat the selection of new seed points to start the next region growth until there are no new seed points to choose. min As shown in Algorithm 2, the input data of the block-level region growing algorithm is the state pyramid S. First, the largest unmarked and unvisited block is selected as the seed point, and it is added to the processing queue Q. Set R is used to store the block indices in the current planar region. For each patch q ∈ Q, calculate the set of neighboring patches N adjacent to it, and sort them by area to ensure that region growing gives priority to larger blocks. Then, check each neighboring patch n ∈ N, if n is coplanar with q, add it to R and Q. This process continues until Q is empty, at which time the R obtained is all the block indices of a region, and the planar parameters, labels are stored in the state pyramid S according to these indices, i.e. the growth of a region is completed. Repeat the selection of new seed points to start the next region growth until there are no new seed points to choose. min ​min , s min ). Then the coordinates of the four corners in the 0-th layer are calculated, which can be done by p i ' = 2 N-n p i . The four edge segments of the seed block can be represented by p1'p2', p2'p3', p3'p4', p1'p4', and the neighbors of the seed block must be adjacent to these four edges.

[0168] Figure 5 Figures (a), (b) and (c) in the middle show the region growing process from a large seed block to the smallest block (from A to B). After obtaining the edge segment expressions, the row or column offset b is added to the four edges to obtain the neighbor block index. For the top and left edges, b = -s min , and for the right and bottom edges, b = 0. Using these indices, the neighbor block information can be directly accessed in the state pyramid S.

[0169] Figure 5 Figures (d), (e) and (f) in the middle show the region growing process from a small seed block to a large block (from C to D). First, the smallest block index (i.e., block G) is obtained, which is called the virtual smallest block, but these blocks are not guaranteed to be consistent with the actual block partition state (i.e., block D). Then, the partition state stored in G is checked to obtain the source layer l of the actual partition, and the actual neighbor index, which is the first l-bit index of the virtual smallest block.

[0170] Step 4: Pixel-wise boundary refinement

[0171] The result of the block-wise partition is often rough, and in some cases, accurate edge detection is needed. For this purpose, the present embodiment uses a morphological operation based on the smallest block to detect the edge range and refine the pixels contained therein. The partition result is first converted into a region mask M, in which each pixel corresponds to a smallest block. Then, a 3x3 kernel is used to erode and dilate each mask M, and the completely eroded regions will be discarded. The boundary mask M B is defined as the difference between the eroded and dilated masks. Then, the distance of all points to the region plane is calculated within the mask M B . If the distance of this pixel is less than the plane MSE multiplied by a constant k, it is assigned to the region. For example, k = 9 is used in the present embodiment. When a cell is shared by multiple regions, the distance of the point to the plane will be calculated for each region, and the pixel will be assigned to the region with the smallest distance.

[0172] The three-dimensional plane fast detection and segmentation method based on depth map data provided by the application not only has high real-time performance, but also has good segmentation effect. When the input image is 640*480, the processing frame rate of the method on a consumer-grade CPU can reach 700fps, which is 1.6 times higher than that of the existing best method; in the comparison test of segmentation performance on the data set, the accuracy of the method can reach 0.58, and the recall rate can reach 0.61, which is 2 times higher than that of the existing method. Therefore, the method can be integrated into the end-side processing chip of the depth camera, so that the camera can output plane information (such as pixel area of the plane, plane parameters and other data) while outputting the depth map. The subsequent application can be based on the plane information output by the camera for secondary development, which can not only reduce the development workload and difficulty of the subsequent application, but also improve the overall real-time performance of the system, and provides a more convenient and efficient perception basis for various robots and intelligent devices.

[0173] In addition, in the field of mobile robot applications, the technical solution can be used as a functional module of a perception algorithm. By providing accurate plane detection results, the robot can quickly analyze the geometric structure of the surrounding environment. For example, in an indoor scene, the robot can accurately identify basic plane elements such as the ground, walls and tabletops, and these information can be directly used in subsequent autonomous navigation and path planning modules. In industrial robot applications, accurate plane information enables the robot to accurately position the grasping plane of the workpiece, thereby realizing reliable grasping operation. In the field of warehouse logistics, the robot can complete complex operations such as sorting and stacking more efficiently according to the detected plane features of the goods. With the rapid development of robots and intelligent manufacturing industry, the demand for environmental perception capability will continue to grow. As a basic perception algorithm, the method is conducive to improving system performance and reducing cost.

[0174] The above detailed the preferred embodiments of the application. It should be understood that those skilled in the art can make many modifications and changes without creative labor based on the concept of the application. Therefore, any technical solution obtained by logical analysis, reasoning or limited experiment based on the existing technology according to the concept of the application shall be within the protection scope determined by the claims.

Claims

1. A method for fast detection and segmentation of three-dimensional planes based on depth map data, characterized in that, The method comprises the following steps: S101: converting the collected depth image into point cloud data, and down-sampling the point cloud data, and constructing a data pyramid and a state pyramid based on the down-sampled point cloud data; S103: obtaining adaptive plane blocks by using a top-down data access and parameter fitting, and storing detailed information of the plane blocks in the state pyramid; S105: marking adjacent and coplanar block-level regions as connected plane regions by using a block-level region growing method, and merging the coplanar plane blocks; S107: calculating a pixel distance of each region edge to a plane, and assigning a pixel to a region with the smallest distance according to a calculation result, and outputting a segmentation result; Wherein, The step S101 comprises the following sub-steps: S1011: converting the depth image into the point cloud data, which maintains the same matrix size as the depth image; S1012: down-sampling the point cloud data to create the data pyramid; S1013: storing the point cloud data in the data pyramid, wherein the 0th layer of the data pyramid stores the original point cloud data, and the width and height of the coordinate matrix in the subsequent layers of the data pyramid are halved layer by layer; S1014: storing state information of the algorithm in the state pyramid, wherein the state pyramid has the same data structure and indexing method as the data pyramid; The data pyramid establishes a corresponding relationship between pixels and regions at different layers, and each block is represented by a pyramid index: No. Mid-layer position The pixels and the first Mid-layer position The corresponding pixels; The pyramid index is represented in the following manner: ； Index In the first The layer corresponding to the region is: ; The region includes four sub-blocks, and the sub-blocks are: ； wherein, denotes the number of layers, denotes the pixel coordinates, denotes the size of the block, is the pyramid index; In the step S105, the block-level region growing method uses region growing to merge coplanar plane blocks, and uses a breadth-first search strategy to expand the search region, comprising the following sub-steps: S1051: From the state pyramid select the largest untagged and unvisited block as a seed point and add the seed point to the queue store the block index in the current plane region to the set store the block index in the current plane region to the set S1052: For each patch , compute the set of patches adjacent to the patch and sort by area to ensure region growing prefers larger patches; S1053: Check each adjacent patch ,judge and Are they coplanar? If so, then patch them. Add to and middle; S1054: obtaining the indices of all the blocks and storing the plane parameters and labels into the state pyramid according to the indices in which the growing of one region is completed; S1055: repeatedly selecting a new seed point to start the next region growing until there is no new seed point available.

2. The method of claim 1, wherein, In the step S103, the depth image is divided into multiple square regions, each of which has the same size, the position and size of the blocks in the depth image are represented by hierarchical indexes, and the index value corresponds to the relative spatial position; For any block with a hierarchical index, the sampling point is directly extracted from the data pyramid using the pyramid index, wherein, The hierarchical index is: , The pyramid index is: , The following conversion formula is used to complete the conversion between the hierarchical index and the pyramid index: wherein, is a hierarchical index, is a pyramid index, denotes the initial partition region where a block is located, the rest of the indices denotes the relative position within the region, 0, 1, 2, 3 correspond to the top-left, top-right, bottom-left and bottom-right sub-blocks, respectively, is the initial block size, is the minimum block size, is the number of levels, is the maximum number of levels, , is the pixel coordinate, is the width of the initial block, is the floor operator.

3. The method of claim 2, wherein, The adaptive plane block segmentation method uses a recursive method to perform adaptive plane block segmentation, comprising the following steps: S1031: using a hierarchical index from the data pyramid acquiring point set data , using a queue representing a region to be processed; S1032: Check the queues sequentially. For each index in the table, determine whether the planar region corresponding to the index satisfies the planar condition; S1033: record the block information satisfying the planar condition to the state pyramid, add four sub-block indexes of the block not satisfying the planar condition to the queue In some embodiments; S1034: judging whether the data of the queue is processed or not, if not, continue to process.

4. The method of claim 3, wherein, The adaptive plane block segmentation method performs plane fitting by principal component analysis, and uses the plane normal vector as the eigenvector corresponding to the minimum eigenvalue in the principal component analysis, and the plane condition is that the block performing plane fitting is a continuous block and the plane normal vector of the block is lower than a preset threshold.

5. The method of claim 4, wherein, The block-level region growing method optimizes the traditional neighbor search method, and the optimized neighbor search method realizes neighbor search on variable-size blocks, and the optimized neighbor search method comprises the following steps: S1: obtaining a seed block, calculating the four-corner coordinates of the layer in which the seed block is located according to the hierarchical index of the seed block; S1: obtaining a seed block, calculating the four-corner coordinates of the layer in which the seed block is located according to the hierarchical index of the seed block; S2: calculate the coordinates corresponding to the four corners in the 0th layer: ; S3: obtaining four edge segments of the seed block, neighbors of the seed block being adjacent to the four edges of the seed block. , the neighbors of the seed block being adjacent to the four edges of the seed block. S4: add row or column bias to the four edges of the seed block to obtain the index of the neighbor block; S5: Use the index to directly access neighbor block information in the state pyramid S5: Use the index to directly access neighbor block information in the state pyramid 6. The method of claim 5, wherein, In the step S1, the seed block is located in the The four-corner coordinates in the layer are calculated as follows: upper left corner coordinates of the seed block calculated from a conversion formula of the hierarchical index and the pyramid index The coordinates of the remaining three triangles of the seed block are: wherein is the minimum block size.

7. The method of claim 6, wherein, In the step S107, a minimum block based morphological operation is used to detect the edge range and refine the pixels contained therein, to achieve accurate edge detection, including the following steps: S1071: convert the preliminary segmentation result into a region mask, each pixel in the mask corresponding to a minimum block; S1072: use a 3x3 kernel to erode and dilate each mask, discard completely eroded regions; S1073: calculate the difference between the eroded mask and the expanded mask, and save the difference to the boundary mask; S1074: calculate the distance of all points to the region plane in the boundary mask, if the distance of a pixel is less than a predetermined distance, assign the pixel to the corresponding region; S1075: when a cell is shared by multiple regions, calculate the distance of the point to the plane for each region, and assign the pixel to the region with the smallest distance.

Images