Scene accessory component extraction method and device based on point cloud data

Abstract

The disclosed embodiments disclose a method and device for extracting scene accessory components based on point cloud data, which comprises acquiring point cloud data collected in a specified scene; performing plane segmentation processing on the point cloud data, and obtaining a point cloud cluster belonging to a wall plane in the specified scene based on the result of the plane segmentation processing; projecting the point cloud cluster of the wall plane onto a corresponding wall plane to obtain an occupancy grid map; detecting occlusion areas in the occupancy grid map, and performing filling processing on the occlusion areas; wherein the occlusion areas are areas of the wall in the specified scene that are occluded by objects; extracting a bounding box of a hole area in the filled occupancy grid map, and determining the position information of the accessory components of the wall in the specified scene based on the bounding box of the hole area. Thus, the position information of the accessory components can be robustly and accurately extracted from the point cloud data, thereby improving data processing efficiency.

Description

Technical Field

[0001] The present disclosure relates to three-dimensional reconstruction technology and point cloud processing technology, and in particular to a method and device for extracting scene accessory components based on point cloud data. Background Art

[0002] With the continuous development of LiDAR detection technology, it is now possible to quickly obtain the depth information of the target being measured, and then generate point cloud data of the target being measured. Point cloud data carries high-precision, high-resolution and high-dimensional geometric information, which can intuitively represent information such as the shape, surface and texture of objects in space, and can be used for applications such as three-dimensional modeling and scene reconstruction. In an application example, after obtaining point cloud data of an indoor scene through laser scanning, in addition to being able to directly browse the colored point cloud, it is often necessary to further perform scene analysis and other applications, such as extracting the position of the wall, extracting the position of wall accessories such as doors and windows, etc. The current method for calculating the position and size of the accessories in the point cloud data is to manually measure them in the point cloud editing software. This method is cumbersome to operate and inefficient, and the accuracy of the detection results is difficult to guarantee. Summary of the Invention

[0003] In view of the above problems in the prior art, embodiments of the present disclosure provide a method and apparatus for extracting scene attachment components based on point cloud data.

[0004] A first aspect of the embodiments of the present disclosure provides a method for extracting scene attachment components based on point cloud data, comprising:

[0005] Get the point cloud data collected in the specified scene;

[0006] Performing plane segmentation processing on the point cloud data, and obtaining point cloud clusters belonging to the wall plane in the specified scene based on the results of the plane segmentation processing;

[0007] Project the point cloud cluster of the wall plane onto the corresponding wall plane to obtain the occupancy grid map;

[0008] Detect the occluded areas in the occupancy grid and fill them; the occluded areas are the areas where the walls in the specified scene are blocked by objects;

[0009] The bounding box of the hole area is extracted from the filled occupancy grid map, and the position information of the attached components of the wall in the specified scene is determined based on the bounding box of the hole area.

[0010] As a possible implementation of the first aspect, detecting an occluded area in an occupancy grid map and filling the occluded area further includes:

[0011] Perform a closing operation on the occupancy grid map;

[0012] Fill the occluded area in the occupancy grid map after the closing operation.

[0013] As a possible implementation of the first aspect, detecting an occluded area in an occupancy grid map and filling the occluded area includes:

[0014] For unoccupied pixels in the occupancy grid, obtain the corresponding 3D points in the point cloud data.

[0015] Calculate whether there is point cloud occlusion in the direction of a ray passing through the 3D point; where the ray passing through the 3D point is a ray that has the 3D point as its endpoint and extends into the room along the normal direction of the wall plane;

[0016] If there is a point cloud occlusion in the ray direction, the pixel point in the occupancy grid map is filled as occupied.

[0017] As a possible implementation of the first aspect, the method further includes:

[0018] Performing plane segmentation processing on the point cloud data, and obtaining point cloud clusters belonging to the top plane and the bottom plane in the specified scene based on the results of the plane segmentation processing;

[0019] Fit the point cloud clusters of the top plane and the bottom plane to obtain the position information of the top plane and the bottom plane;

[0020] Based on the position information of the accessory component, the position information of the top plane, and the position information of the bottom plane, the relative position information of the accessory component and the top plane, and the relative position information of the accessory component and the bottom plane are determined.

[0021] As a possible implementation of the first aspect, the method further includes:

[0022] Extracting edge point cloud data whose distance from an edge line of the attached component is less than a first threshold value according to position information of the attached component of the wall in the specified scene;

[0023] Perform plane extraction on the edge point cloud data to obtain the position information of the edge plane. The edge plane is a plane parallel to the edge line of the attached component and perpendicular to the wall plane.

[0024] According to the position information of the edge plane, the position information of the auxiliary component is adjusted.

[0025] As a possible implementation of the first aspect, the method further includes:

[0026] Among multiple wall planes in a specified scene, a wall plane that is parallel to the wall plane and whose distance satisfies a preset condition is determined as the opposite wall plane of the wall;

[0027] According to the position information of the auxiliary component, detecting whether there is an auxiliary component on the opposite side at the corresponding position on the opposite wall plane;

[0028] According to the detection results of the auxiliary components on the opposite side, the position information of the auxiliary components is verified.

[0029] As a possible implementation of the first aspect, verifying the position information of the accessory component according to the detection result of the opposite-side accessory component includes:

[0030] If it is detected that no opposite-side auxiliary component exists at the corresponding position on the opposite-side wall plane, it is determined that the position information of the auxiliary component is misjudged information.

[0031] As a possible implementation of the first aspect, verifying the position information of the accessory component according to the detection result of the opposite-side accessory component includes:

[0032] If it is detected that an opposite-side auxiliary component exists at a corresponding position on the opposite-side wall plane, the position information of the auxiliary component is adjusted based on the position information of the opposite-side auxiliary component.

[0033] As a possible implementation of the first aspect, adjusting the position information of the accessory component based on the position information of the opposite-side accessory component includes:

[0034] The intersection or union of the area where the attached component and the opposite-side attached component are located is adjusted to the area where the attached component is located.

[0035] A second aspect of the embodiments of the present disclosure provides a scene accessory component extraction device based on point cloud data, comprising:

[0036] An acquisition unit, used to acquire point cloud data collected in a specified scene;

[0037] A segmentation unit is used to perform plane segmentation processing on the point cloud data, and obtain a point cloud cluster belonging to the wall plane in the specified scene based on the result of the plane segmentation processing;

[0038] A projection unit, configured to project the point cloud cluster of the wall plane onto the corresponding wall plane to obtain an occupancy grid map;

[0039] A filling unit is used to detect the occluded area in the occupancy grid map and fill the occluded area; wherein the occluded area is the area of ​​the wall in the specified scene that is blocked by the object;

[0040] The extraction unit is used to extract the bounding box of the hole area in the occupancy grid map after the filling process, and determine the position information of the auxiliary components of the wall in the specified scene based on the bounding box of the hole area.

[0041] As a possible implementation of the second aspect, the filling unit is further configured to:

[0042] Perform a closing operation on the occupancy grid map;

[0043] Fill the occluded area in the occupancy grid map after the closing operation.

[0044] As a possible implementation of the second aspect, the filling unit is configured to:

[0045] For unoccupied pixels in the occupancy grid, obtain the corresponding 3D points in the point cloud data.

[0046] Calculate whether there is point cloud occlusion in the direction of a ray passing through the 3D point; where the ray passing through the 3D point is a ray that has the 3D point as its endpoint and extends into the room along the normal direction of the wall plane;

[0047] If there is a point cloud occlusion in the ray direction, the pixel point in the occupancy grid map is filled as occupied.

[0048] As a possible implementation of the second aspect, the apparatus further includes a processing unit configured to:

[0049] Performing plane segmentation processing on the point cloud data, and obtaining point cloud clusters belonging to the top plane and the bottom plane in the specified scene based on the results of the plane segmentation processing;

[0050] Fit the point cloud clusters of the top plane and the bottom plane to obtain the position information of the top plane and the bottom plane;

[0051] Based on the position information of the accessory component, the position information of the top plane, and the position information of the bottom plane, the relative position information of the accessory component and the top plane, and the relative position information of the accessory component and the bottom plane are determined.

[0052] As a possible implementation of the second aspect, the apparatus further includes an adjustment unit configured to:

[0053] Extracting edge point cloud data whose distance from an edge line of the attached component is less than a first threshold value according to position information of the attached component of the wall in the specified scene;

[0054] Perform plane extraction on the edge point cloud data to obtain the position information of the edge plane. The edge plane is a plane parallel to the edge line of the attached component and perpendicular to the wall plane.

[0055] According to the position information of the edge plane, the position information of the auxiliary component is adjusted.

[0056] As a possible implementation of the second aspect, the apparatus further includes a testing unit, which includes:

[0057] A matching subunit is configured to determine, among multiple wall planes in a specified scene, a wall plane that is parallel to the wall plane and whose distance satisfies a preset condition as the opposite wall plane of the wall;

[0058] The detection subunit is used to detect whether the opposite side accessory component exists at a corresponding position on the opposite side wall plane according to the position information of the accessory component;

[0059] The verification subunit is used to verify the position information of the auxiliary component according to the detection result of the auxiliary component on the opposite side.

[0060] As a possible implementation of the second aspect, the check subunit is configured to:

[0061] If it is detected that no opposite-side auxiliary component exists at the corresponding position on the opposite-side wall plane, it is determined that the position information of the auxiliary component is misjudged information.

[0062] As a possible implementation of the second aspect, the check subunit is configured to:

[0063] If it is detected that an opposite-side auxiliary component exists at a corresponding position on the opposite-side wall plane, the position information of the auxiliary component is adjusted based on the position information of the opposite-side auxiliary component.

[0064] As a possible implementation of the second aspect, the check subunit is configured to:

[0065] If it is detected that an opposite side auxiliary component exists at a corresponding position on the opposite side wall plane, the intersection or union of the auxiliary component and the area where the opposite side auxiliary component is located is adjusted to the area where the auxiliary component is located.

[0066] A third aspect of the present disclosure provides an electronic device, including:

[0067] a memory for storing a computer program product;

[0068] The processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, implements any one of the methods of the first aspect.

[0069] A fourth aspect of an embodiment of the present disclosure provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the method of any one of the above-mentioned first aspects is implemented.

[0070] Based on the embodiments of the present disclosure, by projecting the point cloud data in a specified scene into an occupancy grid map and by detecting the occluded areas blocked by objects in the scene in the occupancy grid map, interference factors in the extracted information can be effectively filtered out, and the position information of the attached components can be robustly and accurately extracted from the point cloud data, thereby improving data processing efficiency.

[0071] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0073] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0074] Figure 1 This is a flowchart of an embodiment of the method for extracting scene attachment components disclosed herein;

[0075] Figure 2 A schematic diagram of point cloud data projection according to an embodiment of the method for extracting attached components from a scene disclosed herein;

[0076] Figure 3 A schematic diagram of the locations of attached components in one embodiment of the method for extracting attached components from a scene disclosed herein;

[0077] Figure 4 A schematic diagram of point cloud data projection according to an embodiment of the method for extracting attached components from a scene disclosed herein;

[0078] Figure 5 A schematic diagram of point cloud data projection according to an embodiment of the method for extracting attached components from a scene disclosed herein;

[0079] Figure 6 This is a flowchart of an embodiment of the method for extracting scene attachment components disclosed herein;

[0080] Figure 7 This is a structural diagram of an embodiment of the device for extracting auxiliary components from a scene disclosed herein;

[0081] Figure 8 This is a structural diagram of an embodiment of the device for extracting auxiliary components from a scene disclosed herein;

[0082] Figure 9 This is a structural diagram of an embodiment of the device for extracting auxiliary components from a scene disclosed herein;

[0083] Figure 10 A block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0084] The following first introduces the existing methods, and then introduces the technical solution of this application in detail.

[0085] In the related art, scene analysis is further performed on the point cloud data of the indoor scene obtained by laser scanning, such as extracting the position of the wall surface, extracting the position of the wall accessories such as doors and windows, etc., and manually measuring them in the point cloud editing software. This method has high labor costs, is cumbersome to operate, and is relatively inefficient, and the accuracy of the detection results is difficult to guarantee. Alternatively, there is also a method in the related art to extract information of the accessories based on the image recognition algorithm, but this method requires additional camera configuration, such as the configuration of a camera and other equipment, which adds additional costs. In summary, the existing method of extracting scene accessories is costly, inefficient, and difficult to guarantee accuracy.

[0086] Based on the technical problems existing in the above-mentioned prior art, an embodiment of the present application provides a method for extracting scene accessory components based on point cloud data. By projecting the point cloud data in a specified scene into an occupancy grid map, and by detecting the occluded areas blocked by objects in the scene in the occupancy grid map, the interference factors in the extracted information can be effectively filtered out, and the position information of the accessory components can be robustly and accurately extracted from the point cloud data, thereby improving data processing efficiency without increasing additional labor costs and equipment costs, thereby solving the technical problems mentioned in the prior art of high cost, low efficiency, and difficulty in ensuring accuracy.

[0087] Figure 1 This is a flow chart of an embodiment of the method for extracting scene attachment components based on point cloud data disclosed in the present invention. Figure 1 Specifically, it may include:

[0088] Step S110: Acquire point cloud data collected within a specified scene. In one example, the specified scene may include the indoor area of ​​a specified house. Based on the house's floor plan, the point cloud acquisition device can be controlled to sequentially move within each room to collect point cloud data. The point cloud acquisition device can be a mobile laser scanning device, such as a 3D laser scanner or lidar.

[0089] In step S120 , plane segmentation processing is performed on the point cloud data, and a point cloud cluster belonging to the wall plane in the specified scene is obtained based on the result of the plane segmentation processing.

[0090] First, a point cloud plane segmentation algorithm is used to segment the point cloud data collected in a specified scene. Points in the point cloud data are assigned to clusters belonging to the same plane, resulting in point cloud clusters representing wall planes in the specified scene. The plane normal, length, and height information obtained from the plane segmentation process are then used to identify multiple wall planes in the specified scene. For example, a plane with a horizontal normal and a height and length exceeding a certain threshold is considered a wall plane. This method is used to filter all wall planes in the specified scene.

[0091] An exemplary point cloud plane segmentation algorithm includes the following steps:

[0092] 1) Preprocessing of point cloud data: including data filtering, downsampling and other operations.

[0093] 2) Plane Detection: Detect planes in point cloud data using appropriate algorithms, such as those based on Random Sample Consensus (RANSAC) or Region Growing. These algorithms find specific plane models by fitting or clustering samples or point sets in the data.

[0094] 3) Plane Parameter Estimation: After finding the plane, estimate its parameters. Usually, the least squares method or the method of minimizing the sum of squared residuals is used to fit the representation and description of the plane for further use and analysis.

[0095] 4) Plane Segmentation: Based on the plane model and corresponding parameters, the point cloud data is segmented according to the plane, and the points are assigned to clusters belonging to the same plane. This can be achieved by calculating the distance between the point and the plane or the angle between the normal vector.

[0096] In step S130, the point cloud clusters of the wall plane are projected onto the corresponding wall plane to obtain an occupancy grid map. For the wall plane fitted in the specified scene, the point cloud clusters belonging to the wall plane are projected onto the corresponding wall plane. The spatial data is converted from a three-dimensional coordinate system to a two-dimensional coordinate system through projection technology, and an occupancy grid map is generated in this process. The projected occupancy grid map refers to an occupancy grid map, which divides the space into multiple small grids (grids), each grid representing a specific spatial area, and can mark whether these areas are occupied.

[0097] Specifically, the projected point cloud data can be drawn onto the image at a certain resolution to obtain an occupancy grid map. The resolution can be set according to the density of the point cloud. A grid in the occupancy grid map represents a pixel. Resolution refers to the size of a pixel. For example, the density of the point cloud is one point per 1 cm. When the point cloud is discretized onto the image, the resolution can be set to a pixel size of 1 cm. All points in the point cloud are projected onto the plane. If a point is projected onto a grid on the plane, the pixel corresponding to the grid is set to white, indicating that the grid is occupied. After projecting the point cloud cluster on the wall plane, the occupied pixels indicate that the position belongs to the wall surface.

[0098] The occupancy grid obtained after projection is as follows Figure 2As shown in the figure, the black hole area in the occupancy grid may be caused by the attachment of the wall components such as windows, doors, and doorways. Since there is no wall in this area, the black hole area appears. Figure 2 The black hole area with the label number 1 is caused by a window on the wall. In addition, some black areas may be caused by curtains, radiators, etc. blocking the wall, resulting in the failure to collect point cloud data on the wall in that area, thus resulting in a black hole area. Figure 2 The black hole area numbered 2 is caused by the obstruction of the curtain, and the black hole area numbered 3 is caused by the obstruction of the radiator.

[0099] Step S140 , detecting the occlusion area in the occupancy grid image and performing filling processing on the occlusion area; wherein the occlusion area is an area of ​​the wall in the specified scene that is occluded by an object.

[0100] Due to the presence of obstructions such as curtains, radiators, pendants, and decorative paintings, the wall point cloud may sometimes be incomplete. Black holes will appear in the obstructed areas in the occupancy grid map. In this case, the occupancy grid map needs to be further processed. First, the point cloud data of the space in front of the wall can be detected to determine whether the wall is obstructed by objects. If there are obstructions on the wall, the obstructed area will be additionally processed, and the area obstructed by the object will also be filled with white. Figure 2 In the example shown, if the black hole region labeled 1 is detected as free of obstructions, and the black hole regions labeled 2 and 3 are detected as having obstructions, the corresponding regions 2 and 3 are filled with white. After the filling process, only the black region labeled 1 remains in the occupied raster image. This region is not caused by an obstruction, but by an attached wall component such as a window.

[0101] Step S150 : extracting a bounding box of the hole area in the filled occupancy grid map, and determining position information of an attached component of the wall in the specified scene based on the bounding box of the hole area.

[0102] Among them, the bounding box is an algorithm for finding the optimal bounding space of a discrete point set. The basic idea is to use a slightly larger geometric body with simpler characteristics (called a bounding box) to approximately replace complex geometric objects. Figure 2 For example, the rectangular bounding box of the hole area labeled 1 can be extracted from the filled occupancy grid map. The window location information can be determined based on the bounding box of the hole area. Specifically, the area within the rectangular bounding box is used as the window location. The horizontal extent of the rectangular bounding box corresponds to the width of the window, and the vertical extent corresponds to the height of the window.

[0103] like Figure 3 As shown, the white rectangular border represents the border of the extracted rectangular bounding box, represents the border line of the extracted window, and the entire black area represents the entire wall area where the window is located.

[0104] In addition to windows, the above method can also be used to extract location information for other ancillary components such as doors and doorways. When collecting point cloud data, the door of the room can be left open. This will create a clear black hole in the occupancy grid when processing the point cloud data using the above method, making the extraction result more accurate.

[0105] In summary, based on the embodiments of the present disclosure, by projecting the point cloud data in a specified scene into an occupancy grid map, and by detecting the occluded areas blocked by objects in the scene in the occupancy grid map, the interference factors in the extracted information can be effectively filtered out, and the position information of the attached components can be robustly and accurately extracted from the point cloud data, thereby improving data processing efficiency.

[0106] In one embodiment, detecting the occluded area in the occupancy grid map and filling the occluded area further includes:

[0107] Perform a closing operation on the occupancy grid map;

[0108] Fill the occluded area in the occupancy grid map after the closing operation.

[0109] See also Figure 2 In the process of projecting the point cloud clusters on the wall plane to obtain the occupancy grid map, the projected point cloud data is drawn onto the image at a certain resolution. If the density of the point cloud is low, some locations on the wall may not be scanned. Figure 2 Due to the point cloud density, the white areas corresponding to the wall in the occupancy grid map are sparsely distributed, with one occupied point per pixel. The wall locations on the occupancy grid map are not completely continuous. To address this, image morphological operations are required to cluster the sparse points into a single white area.

[0110] For example, you can perform a closing operation on an image to fill small holes. Closing is an operation in mathematical morphology, defined as dilation followed by erosion. Specifically, closing is achieved by first dilating the image and then performing an erosion operation. Closing can eliminate small holes in the image, fill small breaks, and connect narrow gaps. In addition, closing can exclude small black holes (black areas) and usually connect narrow gaps to fill holes that are smaller than the structural element. The result of the closing operation is as follows: Figure 4 Compare Figure 2 and Figure 4 The result of the closing operation is Figure 2All the small black dots in the white area are filled with white. The black hole areas left after filling are those areas where there is no point cloud projection in a large range, that is, Figure 2 and Figure 4 The areas are numbered 1, 2, and 3.

[0111] After the occupancy grid map is closed, the occlusion area in the occupancy grid map can be further detected and filled. Figure 4 On the basis of , the two areas corresponding to the labels 2 and 3 are also filled with white. After filling, we get the following Figure 5 As shown in the image, it can be seen that the black hole area left after filling is the area where the small holes and the blocked area are filled, leaving only the auxiliary components of the wall labeled 1, that is, the area where the window is located. Figure 5 In the filled occupancy grid map shown in the figure, the bounding box of the hole area can be further extracted. The bounding box after extraction is as follows: Figure 3 As shown in FIG, the position information of the accessory component can be determined based on the bounding box. It can be seen that the above filling process effectively filters out the interference factors in the extracted information, based on which the position information of the accessory component can be accurately extracted from the point cloud data.

[0112] like Figure 6 As shown, in one embodiment, Figure 1 In step S140, the occlusion area in the occupied grid image is detected and filled, including:

[0113] Step S210 : For unoccupied pixel points in the occupancy grid map, obtain the three-dimensional points corresponding to the pixel points in the point cloud data.

[0114] For unoccupied pixels in the occupancy grid, the wall plane information can be used to restore them to their original 3D space, obtaining the corresponding 3D point in the point cloud data. Specifically, based on the 3D plane equation corresponding to the wall plane, the 3D coordinates of the pixel in the occupancy grid are obtained, providing positioning information in 3D space. This converts the 2D pixel in the occupancy grid into a 3D point in 3D space.

[0115] Step S220 , calculating whether there is point cloud occlusion in the direction of the ray passing through the 3D point; wherein the ray passing through the 3D point is a ray that has the 3D point as an endpoint and extends into the room along the normal direction of the wall plane.

[0116] Specifically, a ray is emitted from a three-dimensional point into the room along the normal direction of the wall plane, and it is calculated whether there is occlusion of the original point cloud in the direction of the ray.

[0117] Step S230: If there is a point cloud occlusion in the ray direction, the pixel point in the occupancy grid map is filled as an occupied state.

[0118] See also Figure 4 In the two areas corresponding to the numbers 2 and 3, it is detected that there is a point cloud occlusion in the direction of the ray emitted from the pixel point as the endpoint. Then the pixels in these two areas are filled as occupied in the occupancy grid map, that is, the two areas corresponding to the numbers 2 and 3 are also filled with white. After filling, the following is obtained Figure 5 The image shown. Figure 5 In the filled occupancy grid map shown in the figure, the bounding box of the hole area can be further extracted. Figure 5 The boundary line of the black and white areas in the image is drawn, and the bounding box after extraction is as follows Figure 3 As shown in the figure, the position information of the auxiliary component can be determined based on the bounding box. It can be seen that by filling the occluded area, the interference factors in the extracted information are effectively filtered out, so that only the auxiliary component is left in the filled image. Based on this, the position information of the auxiliary component can be accurately extracted from the point cloud data.

[0119] In one embodiment, the above method further comprises:

[0120] Performing plane segmentation processing on the point cloud data, and obtaining point cloud clusters belonging to the top plane and the bottom plane in the specified scene based on the results of the plane segmentation processing;

[0121] Fit the point cloud clusters of the top plane and the bottom plane to obtain the position information of the top plane and the bottom plane;

[0122] Based on the position information of the accessory component, the position information of the top plane, and the position information of the bottom plane, the relative position information of the accessory component and the top plane, and the relative position information of the accessory component and the bottom plane are determined.

[0123] Among them, the top plane in the specified scene includes the plane where the ceiling of the room is located, and the bottom plane includes the plane where the floor of the room is located. First, the point cloud data collected in the specified scene is subjected to plane segmentation processing using the point cloud plane segmentation algorithm, and the points in the point cloud data are assigned to clusters belonging to the same plane, thereby obtaining point cloud clusters belonging to the top plane and the bottom plane respectively. Then, the normal, length, and height information of the plane obtained by the plane segmentation process are used to obtain the top plane and the bottom plane in the specified scene. For example, a plane whose normal is in the vertical direction and whose width and length exceed a certain threshold can be determined as the top plane or the bottom plane. Based on the above method, all the top planes or bottom planes in the specified scene are filtered out, and the height range of the room can also be determined based on the position information of the top plane or the bottom plane.

[0124] Furthermore, the positional information of accessory components in the occupancy grid can be restored to three-dimensional space to obtain the 3D positional information of the accessory components. In one example, the height of the lower edge of a window from the ground and the height of the upper edge of the window from the ceiling can be further combined with the height information of the ceiling and the floor to calculate the height of the lower edge of the window from the floor and the upper edge of the window from the ceiling of the room. This relative positional information of the accessory components relative to the top plane and the bottom plane can provide on-site detection data for applications such as wall coloring, maintenance, and accessory component installation.

[0125] In one example, the distance between the accessory component and the upper and lower edges of the image can also be measured in the occupancy grid map of the two-dimensional projection, thereby obtaining the relative position information between the accessory component and the top plane, and the accessory component and the bottom plane. However, the upper edge of the two-dimensional projection image of the wall is not necessarily the location of the ceiling. For example, if there is a wall cabinet above the window, when extracting the point cloud data of the wall plane, only the area below the wall cabinet can be extracted, and the location of the ceiling cannot be extracted. Therefore, calculating the distance from the window to the ceiling in the two-dimensional projection image of the wall point cloud may be inaccurate, and it is necessary to combine the global ceiling information of the entire room to obtain the accurate values ​​of the relative position information between the accessory component and the top plane, and the accessory component and the bottom plane.

[0126] In one embodiment, the above method further comprises:

[0127] Extracting edge point cloud data whose distance from an edge line of the attached component is less than a first threshold value according to position information of the attached component of the wall in the specified scene;

[0128] Perform plane extraction on the edge point cloud data to obtain the position information of the edge plane. The edge plane is a plane parallel to the edge line of the attached component and perpendicular to the wall plane.

[0129] According to the position information of the edge plane, the position information of the auxiliary component is adjusted.

[0130] In step S150, a set of accessory components in the specified scene is obtained. The rectangular bounding boxes corresponding to the accessory components can be used to describe their positions and sizes. The vertices of the rectangular bounding box determine the position of the accessory component, while the length and width of the rectangular bounding box determine the size of the accessory component. Based on this, the sizes of the accessory components can be further optimized and adjusted for greater accuracy. Taking a window as an example, the optimization method is as follows:

[0131] A) Most windows are embedded in a wall. The edges of the rectangle containing the window frame often have a plane perpendicular to the wall, such as the window sill, side edge, top edge, or upper edge of the window. First, based on the edges of the rectangular bounding box obtained in step S150, the point cloud data surrounding a certain range is segmented. Specifically, a first threshold can be set to extract edge point cloud data whose distance from the edge of the window bounding box is less than the first threshold.

[0132] B) Extract the plane of the point cloud data extracted in step A to obtain a plane parallel to the edge line of the rectangular bounding box and perpendicular to the wall plane, i.e., the edge plane, that is, the plane where the window sill, window side, top edge, upper edge, etc. are located.

[0133] C) Adjusting the corresponding window position information based on the edge plane position information to obtain a more accurate window size. The rectangular bounding box obtained in step S150 is obtained by projecting continuous three-dimensional space information onto a two-dimensional image. Since two-dimensional images are discrete, for example, there is one pixel per centimeter, its accuracy can reach 1 centimeter. To further improve accuracy, the original point cloud data can be used to extract the edge plane where the window border is located. The position information of the edge plane is then adjusted to the position information of the corresponding window border to optimize the window position and size, further improving the accuracy of the position information.

[0134] Taking the doorway as an example, the edge of the rectangle containing the doorway's border also contains a plane perpendicular to the wall, namely the edge plane, such as the ground, the side edge, top edge, and upper edge of the doorway. The same method can be used to optimize the doorway's position information, referring to the window example above, so I will not repeat it here.

[0135] In the above implementation, the edge planes of the accessory components are extracted from the original point cloud data to adjust and optimize the extraction results, thereby further improving the accuracy of the position information of the accessory components.

[0136] In one embodiment, the above method further comprises:

[0137] Among multiple wall planes in a specified scene, a wall plane that is parallel to the wall plane and whose distance satisfies a preset condition is determined as the opposite wall plane of the wall;

[0138] According to the position information of the auxiliary component, detecting whether there is an auxiliary component on the opposite side at the corresponding position on the opposite wall plane;

[0139] According to the detection results of the auxiliary components on the opposite side, the position information of the auxiliary components is verified.

[0140] Because laser point cloud scanning provides three-dimensional surface information, for an interior wall, there are two matching wall surfaces on either side of the wall, each of which is considered an opposite wall surface. In this case, the above steps will detect two matching accessory components at corresponding locations on the two wall surfaces, corresponding to the same accessory component on the wall. Taking windows as an example, the set of accessory components obtained through the above processing steps will include two windows located at the same location on the same wall surface. Based on this situation, further filtering of accessory components on interior walls can be performed.

[0141] Specifically, multiple wall planes in a specified scene are matched with each other, and point cloud planes belonging to both sides of the same wall are found to form a pair. For example, two planes that are parallel to each other and whose distance is approximately equal to the wall thickness are found and determined to be the point cloud planes on both sides of the same wall. The two wall planes that are successfully matched with each other are the opposite wall surfaces. For a certain accessory component, based on the position information of the accessory component, a detection is performed on the corresponding position of the opposite wall plane to determine whether the opposite accessory component exists. Based on the detection result of the opposite accessory component, the position information of the accessory component is verified. If the opposite accessory component is detected at the corresponding position of the opposite wall plane, it is considered that the detection result of the accessory component is accurate.

[0142] The disclosed embodiment further verifies the detection results of the accessory components by matching two corresponding accessory components on the point cloud planes on both sides of the same wall, thereby improving the accuracy of the accessory component extraction results.

[0143] In one embodiment, the position information of the accessory component is verified based on the detection result of the opposite accessory component, including:

[0144] If it is detected that no opposite-side auxiliary component exists at the corresponding position on the opposite-side wall plane, it is determined that the position information of the auxiliary component is misjudged information.

[0145] These misjudgments may be caused by incomplete or missing scans due to holes or mirrored surfaces in the wall. Furthermore, during laser scanning, the scanning angle and orientation of the point cloud acquisition device constantly change, potentially leaving areas of the wall undetected by the laser points. In this case, using the aforementioned accessory component extraction method, a window-like hole might be detected in the point cloud data on the wall surface, and this hole could be misjudged as a window.

[0146] For the interior walls of a house, scanning the entire house can detect both sides of the same wall. If two matching accessory components are detected at the same position on both sides of the surface, the detection result of the accessory component is considered to be accurate. If only one accessory component is detected on both sides of the same wall, and no matching accessory component is detected at the corresponding position on the opposite window plane, it is considered that the accessory component does not really exist and is just a misjudgment result caused by a hole in the wall.

[0147] The disclosed embodiment matches two corresponding accessory components on the point cloud planes on both sides of the same wall, and further verifies the detection results of the accessory components, which can effectively reduce the misjudgment rate of the accessory component extraction results and improve the accuracy of the extraction results.

[0148] In one embodiment, the position information of the accessory component is verified based on the detection result of the opposite accessory component, including:

[0149] If it is detected that an opposite-side auxiliary component exists at a corresponding position on the opposite-side wall plane, the position information of the auxiliary component is adjusted based on the position information of the opposite-side auxiliary component.

[0150] If two matching accessory components are detected at the same location on either side of the wall, the detection result for that accessory component is considered accurate. Based on this, the final detection result for the position information can be adjusted based on the position information of that accessory component and that of the accessory component on the opposite side. For example, the average position information of that accessory component and that of the accessory component on the opposite side can be calculated and used as the final detection result for the accessory component's position information. This average position information is likely to be closer to the true value, and this adjustment can further reduce detection errors and improve accuracy.

[0151] In one embodiment, adjusting the position information of the accessory component based on the position information of the opposite-side accessory component includes:

[0152] The intersection or union of the area where the attached component and the opposite-side attached component are located is adjusted to the area where the attached component is located.

[0153] If two matching accessory components are detected at the same position on the surfaces of both sides of the wall, the intersection or union of the areas where the two accessory components are located can be adjusted to the area where the corresponding accessory component is located. Specifically, the detection results of the position information of the two matching accessory components may not be completely equal, and there may be a relatively small difference between the two. In one example, in order to purchase paint for painting the wall, it is necessary to calculate the area of ​​the wall. In order to avoid insufficient amount of purchased paint, it is necessary to reduce the extracted window area as much as possible to expand the wall area so that the amount of purchased paint is sufficient. In this case, the intersection of the areas where the two matching accessory components are located can be adjusted to the area where the corresponding accessory component is located, so as to achieve the purpose of making the extracted window area as small as possible and making the amount of purchased paint sufficient, thereby meeting the needs of actual applications.

[0154] The disclosed embodiment further adjusts the detection results of the accessory components by matching two corresponding accessory components on the point cloud planes on both sides of the same wall. This not only improves the accuracy of the accessory component extraction results, but also makes the accessory component extraction results more meet the specific application requirements of actual usage scenarios, and has wide practicality in various specific application scenarios or environments.

[0155] like Figure 7 As shown, the present disclosure also provides an embodiment of a corresponding device for extracting scene attachment components based on point cloud data. For the beneficial effects or technical problems solved by this device, please refer to the description of the methods corresponding to each device or the description in the Summary of the Invention, and will not be repeated here.

[0156] In an embodiment of the scene attachment component extraction device based on point cloud data, the device includes:

[0157] An acquisition unit 100 is used to acquire point cloud data collected in a specified scene;

[0158] The segmentation unit 200 is used to perform plane segmentation processing on the point cloud data, and obtain a point cloud cluster belonging to the wall plane in the specified scene based on the result of the plane segmentation processing;

[0159] The projection unit 300 is used to project the point cloud cluster of the wall plane onto the corresponding wall plane to obtain an occupancy grid map;

[0160] The filling unit 400 is used to detect the occlusion area in the occupancy grid map and fill the occlusion area; wherein the occlusion area is the area of ​​the wall in the specified scene that is blocked by the object;

[0161] The extraction unit 500 is configured to extract a bounding box of the hole area in the filled occupancy grid map, and determine position information of an attached component of a wall in a specified scene based on the bounding box of the hole area.

[0162] In one embodiment, the filling unit 400 is further configured to:

[0163] Perform a closing operation on the occupancy grid map;

[0164] Fill the occluded area in the occupancy grid map after the closing operation.

[0165] In one embodiment, the filling unit 400 is used to:

[0166] For unoccupied pixels in the occupancy grid, obtain the corresponding 3D points in the point cloud data.

[0167] Calculate whether there is point cloud occlusion in the direction of a ray passing through the 3D point; where the ray passing through the 3D point is a ray that has the 3D point as its endpoint and extends into the room along the normal direction of the wall plane;

[0168] If there is a point cloud occlusion in the ray direction, the pixel point in the occupancy grid map is filled as occupied.

[0169] like Figure 8 As shown, in one embodiment, the apparatus further includes a processing unit 600, configured to:

[0170] Performing plane segmentation processing on the point cloud data, and obtaining point cloud clusters belonging to the top plane and the bottom plane in the specified scene based on the results of the plane segmentation processing;

[0171] Fit the point cloud clusters of the top plane and the bottom plane to obtain the position information of the top plane and the bottom plane;

[0172] Based on the position information of the accessory component, the position information of the top plane, and the position information of the bottom plane, the relative position information of the accessory component and the top plane, and the relative position information of the accessory component and the bottom plane are determined.

[0173] In one embodiment, the apparatus further includes an adjustment unit 700 configured to:

[0174] Extracting edge point cloud data whose distance from an edge line of the attached component is less than a first threshold value according to position information of the attached component of the wall in the specified scene;

[0175] Perform plane extraction on the edge point cloud data to obtain the position information of the edge plane. The edge plane is a plane parallel to the edge line of the attached component and perpendicular to the wall plane.

[0176] According to the position information of the edge plane, the position information of the auxiliary component is adjusted.

[0177] like Figure 8 and Figure 9As shown, in one embodiment, the above device further includes a testing unit 800, and the testing unit 800 includes:

[0178] The matching subunit 810 is configured to determine, among multiple wall planes in a specified scene, a wall plane that is parallel to the wall plane and whose distance satisfies a preset condition as the opposite wall plane of the wall;

[0179] The detection subunit 820 is configured to detect whether an opposite-side accessory component exists at a corresponding position on the opposite-side wall plane according to the position information of the accessory component;

[0180] The verification subunit 830 is used to verify the position information of the auxiliary component according to the detection result of the auxiliary component on the opposite side.

[0181] In one embodiment, the syndrome unit 830 is configured to:

[0182] If it is detected that no opposite-side auxiliary component exists at the corresponding position on the opposite-side wall plane, it is determined that the position information of the auxiliary component is misjudged information.

[0183] In one embodiment, the syndrome unit 830 is configured to:

[0184] If it is detected that an opposite-side auxiliary component exists at a corresponding position on the opposite-side wall plane, the position information of the auxiliary component is adjusted based on the position information of the opposite-side auxiliary component.

[0185] In one embodiment, the syndrome unit 830 is configured to:

[0186] If it is detected that an opposite side auxiliary component exists at a corresponding position on the opposite side wall plane, the intersection or union of the auxiliary component and the area where the opposite side auxiliary component is located is adjusted to the area where the auxiliary component is located.

[0187] The scene accessory component extraction device based on point cloud data in the embodiment of the present disclosure corresponds to the above-mentioned scene accessory component extraction method embodiments based on point cloud data in the present disclosure in terms of specific implementation and beneficial technical effects. The relevant contents can be referenced to each other and will not be repeated here.

[0188] Below, reference Figure 10 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be either or both of the first device and the second device, or a standalone device independent of them, and the standalone device may communicate with the first device and the second device to receive collected input signals from them.

[0189] Figure 10 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0190] like Figure 10As shown, the electronic device includes one or more processors and memory.

[0191] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0192] The memory may store one or more computer program products. The memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program products may be stored on a computer-readable storage medium, and the processor may execute the computer program products to implement the scene-attached component extraction method based on point cloud data and / or other desired functions of the various embodiments of the present disclosure described above.

[0193] In one example, the electronic device may further include an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0194] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0195] The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0196] Of course, to simplify, Figure 10 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application scenarios.

[0197] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the scene attachment component extraction method based on point cloud data according to various embodiments of the present disclosure described in the above part of this specification.

[0198] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0199] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps of the scene attachment component extraction method based on point cloud data according to various embodiments of the present disclosure described in the above part of this specification.

[0200] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0201] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0202] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments are sufficient. For system embodiments, since they largely correspond to method embodiments, their description is relatively simple. For relevant parts, references to the description of the method embodiments are sufficient.

[0203] The block diagrams of the devices, devices, equipment, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0204] The methods and apparatus of the present disclosure may be implemented in many ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Therefore, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0205] It should also be noted that in the apparatus, device, and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0206] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0207] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for extracting scene accessory components based on point cloud data, characterized in that: include: Get the point cloud data collected in the specified scene; performing plane segmentation processing on the point cloud data, and obtaining a point cloud cluster belonging to the wall plane in the specified scene based on a result of the plane segmentation processing; Projecting the point cloud cluster of the wall plane onto the corresponding wall plane to obtain an occupancy grid map; Detecting an occlusion area in the occupancy grid map and performing a filling process on the occlusion area; wherein the occlusion area is an area where a wall in the specified scene is occluded by an object; A bounding box of the hole area is extracted from the filled occupancy grid map, and position information of an attached component of the wall in the specified scene is determined based on the bounding box of the hole area.

2. The method according to claim 1, characterized in that The detecting of the occluded area in the occupancy grid map and filling the occluded area further includes: performing a closing operation on the occupancy grid map; Fill the occluded area in the occupancy grid map after the closing operation.

3. The method according to claim 1, characterized in that The detecting of the occluded area in the occupancy grid map and filling the occluded area includes: For unoccupied pixel points in the occupancy grid map, obtaining a three-dimensional point corresponding to the pixel point in the point cloud data; Calculating whether there is point cloud occlusion in the direction of a ray passing through the three-dimensional point; wherein the ray passing through the three-dimensional point is a ray that has the three-dimensional point as an endpoint and extends into the room along the normal direction of the wall plane; If there is a point cloud occlusion in the ray direction, the pixel point in the occupancy grid map is filled as an occupied state.

4. The method according to claim 1, wherein The method further comprises: performing plane segmentation processing on the point cloud data, and obtaining point cloud clusters respectively belonging to a top plane and a bottom plane in the specified scene based on a result of the plane segmentation processing; Fitting the point cloud clusters of the top plane and the bottom plane to obtain position information of the top plane and the bottom plane; Based on the position information of the accessory member, the position information of the top plane, and the position information of the bottom plane, relative position information of the accessory member and the top plane, and relative position information of the accessory member and the bottom plane are determined.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Extracting edge point cloud data whose distance from an edge line of the attachment component is less than a first threshold value based on position information of the attachment component of the wall in the specified scene; Performing plane extraction on the edge point cloud data to obtain position information of an edge plane, where the edge plane is a plane parallel to the edge line of the attachment component and perpendicular to the wall plane; The position information of the auxiliary component is adjusted according to the position information of the edge plane.

6. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Among the multiple wall planes in the specified scene, a wall plane that is parallel to the wall plane and whose distance satisfies a preset condition is determined as the opposite wall plane of the wall; Detecting whether an opposite-side auxiliary component exists at a corresponding position on the opposite-side wall plane according to the position information of the auxiliary component; The position information of the accessory component is verified according to the detection result of the accessory component on the opposite side.

7. The method according to claim 6, characterized in that The verifying of the position information of the accessory component according to the detection result of the opposite-side accessory component includes: If it is detected that no opposite-side auxiliary component exists at the corresponding position of the opposite-side wall plane, it is determined that the position information of the auxiliary component is misjudged information.

8. The method according to claim 6, characterized in that The verifying of the position information of the accessory component according to the detection result of the opposite-side accessory component includes: If it is detected that an opposite-side auxiliary component exists at a corresponding position on the opposite-side wall plane, the position information of the auxiliary component is adjusted based on the position information of the opposite-side auxiliary component.

9. The method according to claim 8, characterized in that The adjusting the position information of the accessory component based on the position information of the opposite-side accessory component includes: The intersection or union of the areas where the accessory component and the opposite-side accessory component are located is adjusted to the area where the accessory component is located.

10. A scene accessory component extraction device based on point cloud data, characterized in that: include: An acquisition unit, used to acquire point cloud data collected in a specified scene; a segmentation unit, configured to perform plane segmentation processing on the point cloud data, and obtain a point cloud cluster belonging to the wall plane in the specified scene based on a result of the plane segmentation processing; a projection unit, configured to project the point cloud cluster of the wall plane onto a corresponding wall plane to obtain an occupancy grid map; A filling unit, configured to detect an occluded area in the occupancy grid map and perform filling processing on the occluded area; wherein the occluded area is an area where a wall in the specified scene is occluded by an object; The extraction unit is configured to extract a bounding box of a hole area in the occupancy grid map after the filling process, and determine position information of an attached component of the wall in the specified scene based on the bounding box of the hole area.

Images