A room structure analysis method and device

Abstract

The present invention discloses a room structure analysis method and device, the method comprising: step S1, obtaining three-dimensional object point cloud data and three-dimensional air point cloud data; step S2, identifying the three-dimensional object point cloud to obtain furniture point cloud, and projecting the three-dimensional object point cloud data through two-dimensional objects to obtain a rotation correction matrix, and performing rotation correction on the three-dimensional object point cloud, the three-dimensional air point cloud and the furniture point cloud according to the rotation correction matrix; step S3, combining the corrected three-dimensional object point cloud with furniture information belonging to each point to generate an interference-resistant two-dimensional top-view object projection and a normal vector projection, and respectively generating a two-dimensional room mask, two-dimensional corner points belonging to each room, and the correlation between the corner points; step S4, generating the local boundary of the room and the global boundary of the floor according to the corner points, and optimizing based on the corner points and parallel distances of the boundaries; step S5, integrating furniture position and channel position information to generate a two-dimensional room structure.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and machine learning, and in particular to a room structure analysis method and device. Background Art

[0002] With the popularization and extensive development of computer technology, people can obtain housing information more conveniently and quickly. Currently, on the real estate transaction platform, each house will be attached with a corresponding floor plan, which describes the overall properties of the house. From the floor plan, you can intuitively see the layout, planning, orientation and size of the entire house, which is an indispensable source of information for customers to understand the house.

[0003] In order to provide customers with authentic property information, most real estate trading platforms, in addition to showing customers floor plans, will also display detailed information corresponding to the floor plans, such as the overall detailed information of the floor plan, the rooms the floor plan consists of, and the specific information of each room.

[0004] However, currently, house floor plans and their detailed information are basically recorded manually offline, which is not only time-consuming and labor-intensive, but also results in differences due to different people's different measurement standards. In addition, there are a large number of houses, and it is almost impossible to manually record every floor plan offline.

[0005] The prior art closest to the present invention is the technical proposal published in the ICCV (IEEE International Conference on Computer Vision) hosted by IEEE in 2019: "Floor-SP: Inverse CAD for Floorplans by Sequential Room-wise Shortest Path", i.e., Floor-SP: Inverse CAD for Floorplans by Sequential Room-wise Shortest Path, such as Figure 1 As shown in the figure, the technical solution first performs room segmentation on the two-dimensional projection map of the three-dimensional scanning of the room, thereby simplifying the floor plan of the entire floor into the reconstruction of multiple individual room polygons, and taking the mask of each room together with the overall projection as the input for subsequent processing. For each room, it first identifies the relationship between the corner points of the room through a neural network, and then constructs an energy function related to points and lines, hoping to minimize the value of the energy function so that the house structure diagram can have as few corner points and edges as possible, and these corner points and edges can be shared by more rooms. Finally, the results of different rooms are merged in a loop to output the final house structure diagram.

[0006] However, this technical solution has some shortcomings: for example, when the house data is not a rough house, but has interference items such as furniture and residents, the simple two-dimensional projection data will often become very "dirty", which will greatly affect the accuracy of subsequent predictions; the optimization based on the energy function is accompanied by extremely strong priors, and the structure is often too simple. It is very likely to connect two parallel but close walls into one wall just to save a line; and if a heuristic algorithm is used, the resources and time consumed by the calculation will be very large.

[0007] In general, the existing Floor-SP technical solution has some technical problems that need to be solved:

[0008] 1. The projection processing of data is too simple: the two-dimensional image as input data is easily affected by other objects such as furniture, residents, and temporary objects. In addition to the wall projection, there are many other points and lines in the projection image, which greatly affects the accuracy and reliability of subsequent predictions;

[0009] 2. Lack of necessary preprocessing steps before data projection: The 3D models obtained by current mainstream modeling methods often follow their own coordinate system, rather than the world coordinate system of the scanned object in the real world, and it is even more impossible to ensure that the main wall direction in the input model is parallel to the x-axis and y-axis. Therefore, if the main direction of the input model cannot be automatically calibrated, the results of multiple predictions may be different, thus affecting its reliability, and it will also contradict the prior assumption that the main directions are mostly horizontal and vertical during the optimization process;

[0010] 3. Energy function optimization based on strong priors: When constructing an energy function, it is usually based on one's own prior understanding of the object, such as as many corners and lines as possible should be shared. However, such priors not only make the final output result too simple sometimes, but also make it difficult to predict some situations that were not anticipated in advance. In addition, the heuristic optimization method used will consume a lot of computing resources and time when the situation is more complicated.

[0011] 4. The results lack information about objects in the room: In many applications based on room structure, such as house viewing or interior decoration, users are often more concerned about the furniture inside the room in addition to the basic room type. However, the existing technology does not involve this at all.

[0012] 5. Lack of channel information in the room: The connectivity between rooms and the location of doors are also very important parts of the room. However, the existing methods do not generate any channel structure. Even if the corner points of the channel are obtained during the detection, they are "optimized" by the energy function because they are on the same straight line as the corner points of the entire wall. Summary of the invention

[0013] In order to overcome the deficiencies of the above-mentioned prior art, the purpose of the present invention is to provide a room structure analysis method and device, so as to achieve the purpose of room structure analysis based on computer vision and scene object recognition technology.

[0014] To achieve the above object, the present invention proposes a room structure analysis method, comprising the following steps:

[0015] Step S1, using an RGBD camera and an odometer camera to scan the room that needs structural analysis to obtain RGBD camera data and odometer camera data, and converting them into three-dimensional object point cloud data and three-dimensional air point cloud data;

[0016] Step S2, obtaining furniture point cloud data by recognizing the three-dimensional object point cloud, and obtaining a rotation correction matrix by projecting the three-dimensional object point cloud data through a two-dimensional object, and performing rotation correction on the three-dimensional object point cloud, the three-dimensional air point cloud, and the furniture point cloud according to the obtained rotation correction matrix;

[0017] Step S3, combining the corrected 3D object point cloud with the furniture information of each point to generate an anti-interference 2D top-view object projection and normal vector projection, and respectively generating a 2D room mask, a 2D corner point of each room, and the correlation between the corner points through three groups of neural networks;

[0018] Step S4, generating a local boundary of the room and a global boundary of the floor according to the corner points, and optimizing based on the corner points and parallel distances of the boundaries;

[0019] Step S5, integrating the furniture position information and the passage position information to generate the final two-dimensional room structure.

[0020] Preferably, step S2 further comprises:

[0021] Step S200, using sparse convolution to process the three-dimensional object point cloud data, based on the U-Net network, the processed data is first deepened after each downsampling convolution, and then symmetrically upsampled and the number of channels is gradually reduced, and finally after the full connection layer and Softmax, the category with the maximum probability is taken as the furniture category classification result of the point, thereby obtaining furniture point cloud data;

[0022] Step S201 , obtaining a rotation correction matrix through two-dimensional object projection of the original three-dimensional object point cloud data, and performing rotation correction on the three-dimensional object point cloud, three-dimensional air point cloud and furniture point cloud data according to the rotation correction matrix.

[0023] Preferably, in step S201, based on the original three-dimensional object point cloud data, the ground plane is extracted as the xy plane, and then the z-axis is rotated to a direction perpendicular to the ground plane, and the z-axis coordinate of the lowest point is made to be 0, so as to obtain a first transformation matrix; then the three-dimensional object point cloud data is projected onto a two-dimensional plane to obtain a projection image, and the line segments are detected in the projection image using the Hough transform to determine the main direction of the line segments, and the plane of the projection image is rotated around the z-axis according to the angle of the main direction of the line segments, and the point with the minimum xy coordinate value is made the origin of the coordinates to obtain a second transformation matrix, thereby obtaining a final rotation transformation matrix based on the first transformation matrix and the second transformation matrix.

[0024] Preferably, in step S201, during rotation correction, the number and length of line segments are counted at different angles, the direction of the line segment with the longest total length is regarded as the main direction of the line segment, and the rotation angle is limited to within 45 degrees, and the main direction of the line segment is rotated to the main direction, or a direction perpendicular to the main direction, so as to achieve the purpose of fine-tuning.

[0025] Preferably, step S3 further comprises:

[0026] Step S300, extracting points of interest based on the corrected three-dimensional object point cloud and the furniture point cloud, thereby outputting noise-resistant two-dimensional top-view object projection and two-dimensional top-view normal vector projection;

[0027] Step S301, using the classic Mask-RCNN network to calculate the feature map of the input noise-resistant two-dimensional top-view object projection image, and combining the region generation network to generate candidate bounding boxes for each room, while performing semantic segmentation on the pixels in each bounding box to extract its room mask;

[0028] Step S302, according to the two-dimensional room mask extracted in step S301 and the noise-resistant two-dimensional top-view object projection map and the noise-resistant two-dimensional top-view normal vector projection map obtained in step S300, a convolutional layer based on dilated convolutions is used to form an encoder to extract features from the image, and then the feature map is passed through a decoder based on a ResNet network structure and input into two parallel branches of the convolutional layer, so as to obtain the two-dimensional coordinates of the corner points and the probability confidence values ​​of several directions in which the corner points extend out of the wall line segments;

[0029] Step S303, combining the corner point information obtained in step S302 with the room mask, the two-dimensional top-view object projection, and the two-dimensional top-view normal vector projection, combining the local room information with the overall floor information in the neural network, and thereby predicting the probability value of each corner point for each room in turn.

[0030] Preferably, after step S3, the method further includes:

[0031] Step S304, sort all rooms according to the number of pixel values ​​occupied by the masks, traverse and calculate the overlap rate between the room masks, and for rooms whose overlap rate exceeds a preset threshold, merge the masks and corner point information of the small rooms into the large room, and delete duplicate points.

[0032] Preferably, step S4 further comprises:

[0033] Step S400, for the corner points associated and merged in each room, optimize and generate the local boundary belonging to the room based on the slope;

[0034] Step S401 , merging local boundaries to generate a global boundary, closing and merging boundaries based on corner points between boundary lines and parallel boundary spacing.

[0035] Preferably, in step S400, all corner points shared by multiple rooms are locked unchanged, and for local corner points of a single room, the slope values ​​of two consecutive edges and their corresponding angles are calculated, and if they are less than a preset threshold, they are merged into the same edge.

[0036] Preferably, step S5 further comprises:

[0037] Step S500, fusing the optimized global boundary with the channel position information in the air point cloud;

[0038] Step S501, generating a final two-dimensional room structure based on the corrected three-dimensional furniture classification point cloud data and the boundary line segment information of the passage and the door integrated in step S500.

[0039] In order to achieve the above object, the present invention also provides a room structure analysis device, comprising:

[0040] A point cloud data generating unit, used for scanning a room requiring structural analysis using an RGBD camera and an odometer camera to obtain RGBD camera data and odometer camera data, and converting the data into three-dimensional object point cloud data and three-dimensional air point cloud data;

[0041] A rotation correction unit is used to obtain furniture point cloud data by recognizing the three-dimensional object point cloud, and to obtain a rotation correction matrix by projecting the three-dimensional object point cloud through a two-dimensional object, and to perform rotation correction on the three-dimensional object point cloud, the three-dimensional air point cloud and the furniture point cloud according to the rotation correction matrix;

[0042] The room mask and corner point extraction association unit is used to combine the corrected 3D object point cloud with the furniture information of each point to generate an anti-interference 2D top-view object projection and normal vector projection, and generate 2D room masks, 2D corner points of each room, and the association between corner points through three sets of neural networks;

[0043] Boundary generation unit, used to generate the local boundary of the room and the global boundary of the floor based on the corner points, and optimize based on factors such as the corner points and parallel distance of the boundary;

[0044] The room structure generation unit is used to integrate the furniture position information and the channel position information to generate the final two-dimensional room structure.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. The present invention combines a semantic segmentation network based on Submanifold Sparse Convolutional Networks. In order to prevent recognition errors such as objects of a certain type mixed with recognition points of other types, the floor and wall parts are removed from the recognition results to obtain mutually unconnected furniture clusters. The number of three-dimensional points belonging to different furniture in each cluster is counted, and the category with the largest voting value is taken as the furniture classification of the cluster, and its two-dimensional minimum bounding box information is obtained. In the final room type detection result, the type and position of the furniture are reflected in the form of a minimum bounding box, and it can also be replaced by any furniture model.

[0047] 2. The present invention combines the information during three-dimensional data collection. In addition to the object points obtained by scanning the target, the space between the data collector and the data point is defined as an air point. Combined with the wall segment position in the room structure recognition result, the proportion of air points near the divergent positions along the way exceeds the continuous length and continuous width of the threshold, and the channel position is dug and marked in the wall result segment.

[0048] 3. The present invention uses a more direct optimization based on geometric features for local and global diagonal points and edges, thereby greatly improving the calculation speed while retaining more detailed features compared to the optimization based on energy functions. The computational cost of this optimization method will only increase linearly with the complexity brought by factors such as the number of rooms and room shapes, thereby maintaining a relatively stable high efficiency.

[0049] 4. The present invention does not project the entire 3D floor point cloud data into 2D as input, but combines the results of the semantic segmentation network to project only the identified wall part into a 2D plane as input, and sets a threshold to filter out possible noise and prediction errors. In this way, even if there are objects in the room, only wall-related information will be included in the input projection data, which can keep it simple and correct.

[0050] 5. In the preprocessing stage, the present invention combines Hough transform to detect two-dimensional straight lines. By counting the length of the wall in each direction in the two-dimensional projection and obtaining the main direction of the three-dimensional data, the main direction is rotated to keep the direction of the main wall of the input data parallel to the coordinate axis. This can not only eliminate the influence of angle offset in the data modeling process, so that inputs at different angles have the same output results, but also make the horizontal and vertical prior assumptions of wall direction correction more reliable, making the results more robust. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flow chart of the room structure analysis in the prior art;

[0052] Figure 2 A flow chart of the steps of a room structure analysis method of the present invention;

[0053] Figure 3 It is a flow chart of a room structure analysis method in a specific embodiment of the present invention;

[0054] Figure 4 It is a network flow chart of three-dimensional point cloud furniture recognition in a specific embodiment of the present invention;

[0055] Figure 5 A flowchart of generating a rotation correction matrix in a specific embodiment of the present invention;

[0056] Figure 6 This is a flow chart of room mask extraction in an embodiment of the present invention;

[0057] Figure 7 This is a flowchart of corner point extraction for a single room in a specific embodiment of the present invention;

[0058] Figure 8 It is a flowchart of associating multiple room corner points in a specific embodiment of the present invention;

[0059] Fig. 9 It is a schematic diagram of local boundary optimization in a specific embodiment of the present invention;

[0060] Fig.10 It is a schematic diagram of global boundary optimization in a specific embodiment of the present invention;

[0061] Fig.11 This is a system architecture diagram of a room structure analysis device of the present invention. DETAILED DESCRIPTION

[0062] The following describes the implementation of the present invention through specific examples and in conjunction with the accompanying drawings. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific examples, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0063] Figure 2 This is a flow chart of the steps of a room structure analysis method of the present invention. Figure 3 FIG. 1 is a flow chart of a room structure analysis method in a specific embodiment of the present invention. Figure 2 and Figure 3 As shown, a room structure analysis method of the present invention comprises the following steps:

[0064] Step S1, using an RGBD camera and an odometer camera to scan the room for structural analysis to obtain RGBD camera data and odometer camera data, and converting them into three-dimensional object point cloud data and three-dimensional air point cloud data.

[0065] In a specific embodiment of the present invention, the RGBD camera data and the odometer camera data are converted into three-dimensional object point cloud data by the existing Slam algorithm, that is, the RGBD camera data (providing one frame of point cloud) and the odometer camera data (providing camera position and orientation) are fused by the Slam algorithm to obtain multi-frame point clouds (i.e., three-dimensional object point cloud). The three-dimensional air point cloud data, i.e., the space between the observed position and the obstacle, is calculated by the Octomap algorithm, that is, the three-dimensional air point cloud is the space between the camera position and the object in front of the camera, which is evenly divided into cubes of equal size by Octomap, each cube corresponds to an air point, and the fusion of multiple frames is the air point cloud. Since the Slam algorithm and the Octomap algorithm are commonly used methods in the prior art to calculate the three-dimensional object point cloud data and the three-dimensional air point cloud data, they will not be described in detail here.

[0066] Step S2, the furniture point cloud information is obtained by recognizing the three-dimensional object point cloud, and the rotation correction matrix is ​​obtained by projecting the three-dimensional object point cloud data through the two-dimensional object, so as to perform rotation correction on the three-dimensional object point cloud, the three-dimensional air point cloud and the furniture point cloud.

[0067] Specifically, step S2 further includes:

[0068] Step S200, using sparse convolution to process the three-dimensional object point cloud data, based on the U-Net network, the processed data is increased by 32 in channel depth after each downsampling convolution, and finally after the fully connected layer and Softmax, the category with the maximum probability is taken as the furniture category classification result of the point, thereby obtaining the furniture point cloud data.

[0069] Figure 4 3D point cloud furniture recognition network flow chart in a specific embodiment of the present invention. Figure 4As shown, the input of the network is the generated 3D object point cloud data, and the output is a furniture point cloud corresponding to the input, but its RGB data is replaced by the predicted furniture category. Specifically, the network first performs a scale transformation on the input 3D object point cloud data. In a specific embodiment of the present invention, in view of the sparse characteristics of point cloud data in space, the present invention uses a novel sparse convolution to process point cloud data. It only performs convolution operations at spatial points where data exists. This not only improves the computational efficiency, but also ensures that the spatial information of the data will not "expand" and become blurred after several convolution operations, that is, Figure 4 The conv in are all sparse convolutions to replace ordinary convolutions, such as Figure 4 As shown, the present invention is based on the U-Net network, and its backbone network (backbone) adopts the ResNet network structure. First, the channel depth is increased by 32 after each downsampling convolution, and then the upsampling is performed symmetrically, and the number of channels is gradually reduced by 32. Finally, after the full connection layer and Softmax, the category with the maximum probability is taken as the classification result of the furniture category at that point.

[0070] A simple explanation of the relationship between the 3D object point cloud, 3D air point cloud and furniture point cloud is that the 3D object point cloud is the data of all scanned objects, the 3D air point cloud is the part of the entire space not occupied by the object point cloud, and the furniture point cloud is a part of the 3D object point cloud.

[0071] Step S201 , obtaining a rotation correction matrix through two-dimensional object projection of the original three-dimensional object point cloud data, and performing rotation correction on the three-dimensional object point cloud, the three-dimensional air point cloud and the furniture point cloud according to the rotation correction matrix.

[0072] In order to facilitate the subsequent correction of the wall direction and make the structure output result more stable, the present invention needs to rotate the architectural structure of the above point cloud so that the main direction of the wall is parallel to the horizontal or vertical direction.

[0073] Specifically, Figure 5As shown, in step S201, according to the original object point cloud file, that is, the three-dimensional object point cloud data obtained in step S1, the ground plane is extracted as the xy plane through ransac (random sampling consensus algorithm), and then the z axis is rotated to a direction perpendicular to the ground plane, and the z-axis coordinate of the lowest point is made to be 0, so as to obtain the transformation matrix T1; then the three-dimensional object point cloud data is projected onto the two-dimensional plane to obtain a projection image, and the line segments are detected by Hough transform in the projection image to determine the main direction of the line segments (the direction of the line segment with the longest total length is regarded as the main direction of the line segment), and the plane of the projection image is rotated around the z axis according to the angle of the main direction of the line segment, that is, it is rotated around the z axis until the line segment corresponding to the wall is parallel to the main direction of the coordinate axis (that is, the x-axis direction or the y-axis direction), and the minimum point of the xy coordinate is made as the coordinate origin, so as to obtain the transformation matrix T2, and then according to the transformation matrices T1 and T2, the final rotation transformation matrix T is obtained, that is, T=T2*T1.

[0074] Specifically, in this embodiment, it is assumed that the xy plane is parallel to the horizontal plane in the point cloud of the three-dimensional object, and the z axis is perpendicular to the horizontal plane, and then the xy plane is rotated around the z axis. First, for the z axis, the plane is extracted by ransac, because it will extract the plane with the largest number of points, so the ground plane will be obtained, because the points of the entire ground in a floor must be more than that of a certain wall, and the normal vector direction of the ground plane is taken as the z axis, and the first rotation matrix can be obtained.

[0075] For the rotation of the xy plane, the three-dimensional object point cloud data is projected onto a two-dimensional plane to obtain a projection image, so that the single-channel grayscale value of a certain pixel represents the number of points at different heights under the plane coordinates, and the projection image is subjected to Hough transform to detect line segments. Due to objects such as floors and furniture, points other than the wall surface will be filtered out according to the results of furniture recognition during projection. In addition, due to the existence of factors such as scanning errors and noise, there will be many scattered short lines in the results of the Hough transform (if the shortest length of the line segment is set to filter out these lines, many short corner structures will be lost). Therefore, in order to resist interference, the present invention counts the number and length of line segments at different angles, and regards the direction of the line segment with the longest total length as the main direction of the line segment. In order to implement the concept of retaining the original data information as much as possible, the rotation angle is limited to within 45 degrees, and the main axis of the data (i.e., the main direction of the line segment) is rotated to the main direction of the coordinate axis, i.e., the x-axis direction or the y-axis direction, so as to achieve the purpose of fine-tuning, thereby obtaining a second transformation matrix, and multiplying it by the first on the left to obtain a complete rotation correction matrix.

[0076] Finally, the rotation correction matrix is ​​used to perform rotation correction on the three-dimensional object point cloud, three-dimensional air point cloud and furniture point cloud respectively.

[0077] Step S3, combining the corrected three-dimensional object point cloud with the furniture information of each point to generate an interference-resistant two-dimensional top-view object projection and a normal vector projection, and respectively generating a two-dimensional room mask, two-dimensional corner points belonging to each room, and the correlation between the corner points through three groups of neural networks. In the present invention, the three groups of neural networks are a room mask extraction network, a single-room corner point extraction network, and a multi-room corner point association network.

[0078] Specifically, step S3 further includes:

[0079] Step S300, based on the corrected 3D object point cloud and extracting points of interest through the furniture point cloud, output noise-resistant 2D top-view object projection and 2D top-view normal vector projection, and the 2D projection images used in subsequent predictions are these two projections.

[0080] Since the subsequent network detects independent room information and corresponding corner point information based on the two-dimensional contour, the three-dimensional point cloud data is projected onto a two-dimensional plane, so that the single-channel grayscale value of a pixel represents the number of points at different heights under the plane coordinates, and a two-dimensional top-view image after projection is obtained (after furniture recognition filtering). However, due to the existence of objects such as floors and furniture, as well as factors such as scanning errors and noise, the present invention sets a threshold value (the default value is 50) for the two-dimensional top-view image after projection as follows, and increases the grayscale value of wall points while filtering non-wall points, thereby obtaining a noise-resistant two-dimensional top-view object projection:

[0081]

[0082] At the same time, the average normal vector of each plane point in the vertical direction is calculated during the projection process, and the output is another three-channel projection image, that is, the output of the two-dimensional top-down normal vector projection. The normal vector can greatly assist the detection of corner points, because the normal vector of the wall should be parallel to the ground. In the present invention, the corner point refers to the corner where two walls meet. The normal vectors of these two walls are parallel to the ground but have different angles on the xy plane. The intersection of these two groups of normal vectors is the corner.

[0083] In step S301, the classic Mask-RCNN network is used to calculate the feature map of the input noise-resistant two-dimensional top-view object projection image. The region generation network (RPN) is combined to generate candidate bounding boxes for each room, and the pixels in each bounding box are semantically segmented to extract its room mask.

[0084] like Figure 6As shown, the extraction of rooms belongs to an instance segmentation task. In the present invention, the Mask-RCNN network, which is a classic in the field of this task, is used to extract the room. The Mask-RCNN network first calculates the feature map of the input noise-resistant two-dimensional top-view object projection map (corrected) through a series of feature pyramid networks (FPN) with ResNet101 as the backbone, and then combines the region generation network (RPN) to obtain a feature map (FeatureMap) with ROI (region of interest). While generating the candidate bounding boxes (Box Regression) and classification results (classification, but the score is obtained here) of each room, the pixels in each bounding box are semantically segmented to extract its room mask (MaskBranch).

[0085] This step is performed before corner point calculation in order to simplify the structural detection of the entire room to the detection of a single room. Considering that the structure of most rooms is a cube or a similar collection, the detection can achieve higher accuracy at a lower difficulty.

[0086] In step S302, according to the two-dimensional room mask extracted in step S301 and the noise-resistant two-dimensional top-view object projection image and the noise-resistant two-dimensional top-view normal vector projection image obtained in step S300, an encoder composed of a convolutional layer based on dilated convolution extracts features from the image, and then passes the feature map through a decoder based on a ResNet network structure and inputs the two parallel branches of the convolutional layer to obtain the two-dimensional coordinates of all corner points and the probability confidence values ​​of several directions in which the corner points extend out of the wall line segments.

[0087] In a specific embodiment of the present invention, Figure 7 As shown, its input includes a two-dimensional room mask, a noise-resistant two-dimensional top-down object projection map (corrected), and a noise-resistant two-dimensional top-down normal vector projection map (corrected). The output corner point information includes the two-dimensional coordinates of the corner point and the probability confidence values ​​of 36 directions (divided into 10 degrees) in which the corner point extends out of the wall segment.

[0088] Specifically, the room mask calculated in step S301 is expanded so that it can completely cover the part of the room on the two-dimensional top-view projection and the two-dimensional normal vector projection. The features of the picture are extracted by forming an encoder (all conv-s1 and con-s2 parts in the figure) through a convolutional layer based on dilated convolution, and then the feature map passes through a decoder module based on ResNet (the residual part in the figure) and is input into two parallel branches of the convolutional layer. Dilated convolution is used here to obtain a larger receptive field with a smaller number of parameters, while different strides can compress the image. The first branch predicts a set of confidence maps, each of which represents a different corner point of the room; the second branch predicts the degree of correlation between the corner points, that is, which two corner points may be connected to form a wall. Both sets of branches control the output results to a probability value within 0-1 through the Sigmoid layer.

[0089] Step S303, combining the corner point information obtained in step S302 with the room mask, the two-dimensional top-view object projection, and the two-dimensional top-view normal vector projection, combining the local room information with the overall floor information in the neural network, and thereby predicting the probability value of each corner point for each room in turn.

[0090] In a specific embodiment of the present invention, Figure 8 As shown in the figure, the main body of the network feature extraction and reasoning part (i.e., conv-s1 and conv-s2 in the figure) is also a series of dilated convolutions, which are then calculated through the fully connected layer and the sigmoid layer to obtain the probability value of the corner point belonging to the room. Corner points above a certain threshold (the default value is 0.5) are considered to belong to the room. If the probability value of a corner point for multiple rooms is higher than the threshold, it is considered to be shared.

[0091] Step S304, sort all rooms according to the number of pixel values ​​occupied by the masks, traverse and calculate the overlap rate between the room masks, and for rooms with a high overlap rate (the default is 30%), merge the masks and corner point information of the small rooms into the large room, and delete duplicate points.

[0092] There may not be a complete conclusion about the independence of some rooms. For example, a semi-open kitchen is connected to the living room but is blocked. For rooms like this, the calculation of the masks may overlap. In the present invention, such cases are merged into the same room. Therefore, all rooms are sorted according to the number of pixel values ​​occupied by the masks, and the overlap rate between the room masks is traversed and calculated. For rooms with a high overlap rate (the default is 30%), the masks and corner point information of the small rooms are merged into the large room, and the duplicate points are deleted.

[0093] Step S4, generating the local boundary of the room and the global boundary of the floor according to the corner points, and optimizing based on factors such as the corner points and parallel distance of the boundaries.

[0094] Specifically, step S4 further includes:

[0095] Step S400: For the corner points associated and merged in each room, a local boundary belonging to the room is optimized and generated based on the slope.

[0096] Since the structure of a single room is usually simple and the corner points are formed around the boundary, in the present invention, the mask boundary of the single room is calculated by the eight-domain boundary tracking algorithm, and its corner points are matched with the nearest points on the boundary. The order of connecting the corner points is obtained by traversing the boundary, and the corner points are connected in sequence to obtain the boundary.

[0097] The prediction results of the neural network usually have certain deviations due to the prediction accuracy or the distribution of the data. In addition, in order not to miss the prediction results of corner points, the threshold of the confidence probability will be set relatively low, but this will result in some redundant corner points. Therefore, it is necessary to optimize the prediction results of the corner points of each room. Fig. 9 As shown, all corner points shared by multiple rooms are locked unchanged to prevent the global structure from being affected. For the local corner points of a single room, the slope values ​​of two consecutive edges and their corresponding angles are calculated (the denominator plus the minimum value prevents it from being 0):

[0098]

[0099] If it is less than a certain threshold (the default is 30 degrees), it will be merged into the same edge. In addition, since the point cloud data has been normalized before, there is a priori assumption that most walls are parallel or perpendicular to the main direction. If the angle between the line segment and the horizontal or vertical direction is less than a certain threshold (the default is 18 degrees), it will be corrected to the horizontal or vertical edge.

[0100] Step S401 , merging local boundaries to generate a global boundary, closing and merging boundaries based on corner points between boundary lines and parallel boundary spacing.

[0101] like Fig.10 As shown in the figure, all local edges are merged into a list and duplicate edges are deleted. In order to efficiently calculate the intersection relationship, all global edges are classified into horizontal edges, vertical edges and oblique edges. For example, horizontal edges only intersect with vertical edges and oblique edges. The endpoints near the corner points that are not shared with other lines are pulled to the corner points to achieve the closure of the wall structure. At the same time, since most room structures have multiple walls on the same horizontal plane, parallel edges and overlapping edges are traversed, and edges with a spacing less than a certain threshold (the default is 3 pixels) are merged to ensure the simplicity of the structure.

[0102] Step S5, integrating the furniture position information and the passage (door) position information to generate the final two-dimensional room structure.

[0103] Specifically, step S5 further includes:

[0104] Step S500 , fusing the optimized global boundary with the channel position information in the air point cloud.

[0105] The 3D air point cloud data is processed, and the positions that are all air within a certain height (the default is 0.5 meters to 2 meters) are defined as passable areas. Comparing the generated wall structure data with the passable area data, if for a section of wall, the proportion of air points within a certain range in a continuous length direction (the length is not less than the threshold, the default is 0.5 meters) and a continuous width direction (the width is not less than the threshold, the default is 0.5 meters) exceeds a certain threshold (the default is 0.5), then this section is considered to be a channel structure in the wall.

[0106] Step S501, generating a final two-dimensional room structure based on the corrected three-dimensional furniture classification point cloud data and the boundary line segment information of the passage and the door integrated in step S500.

[0107] In a specific embodiment of the present invention, the floor and wall parts are removed from the result of the furniture recognition network to obtain unconnected furniture clusters, and the number of three-dimensional points belonging to different furniture in each cluster is counted. The category with the largest voting value is taken as the furniture classification of the cluster, and its two-dimensional minimum bounding box information is obtained.

[0108] Fig.11 This is a system architecture diagram of a room structure analysis device of the present invention. Fig.11 As shown, the present invention provides a room structure analysis device, comprising:

[0109] The point cloud data generating unit 1 is used to scan the room to be structurally analyzed using an RGBD camera and an odometer camera to obtain RGBD camera data and odometer camera data, and convert them into three-dimensional object point cloud data and three-dimensional air point cloud data.

[0110] In a specific embodiment of the present invention, the RGBD camera data and the odometer camera data are converted into three-dimensional object point cloud data by the existing Slam algorithm, that is, the RGBD camera data (providing one frame of point cloud) and the odometer camera data (providing camera position and orientation) are fused by the Slam algorithm to obtain multi-frame point clouds (i.e., three-dimensional object point cloud). The three-dimensional air point cloud data, i.e., the space between the observed position and the obstacle, is calculated by the Octomap algorithm, that is, the three-dimensional air point cloud is the space between the camera position and the object in front of the camera, which is evenly divided into cubes of equal size by Octomap, each cube corresponds to an air point, and the fusion of multiple frames is the air point cloud. Since the Slam algorithm and the Octomap algorithm are commonly used methods in the prior art to calculate the three-dimensional object point cloud data and the three-dimensional air point cloud data, they will not be described in detail here.

[0111] The rotation correction unit 2 is used to obtain furniture point cloud information by recognizing the three-dimensional object point cloud, and to obtain a rotation correction matrix for the three-dimensional object point cloud data through a two-dimensional object projection, and to perform rotation correction on the three-dimensional object point cloud, three-dimensional air point cloud and furniture point cloud according to the obtained rotation correction matrix.

[0112] Specifically, the rotation correction unit 2 further includes:

[0113] The furniture point cloud information recognition module 201 is used to process the three-dimensional object point cloud data using sparse convolution. The processed data is based on the U-Net network, and the channel depth is increased by 32 after each downsampling convolution. Finally, after the fully connected layer and Softmax, the category with the maximum probability is taken as the furniture category classification result of the point, thereby obtaining the furniture point cloud data.

[0114] The projection and rotation correction module 202 is used to obtain a rotation correction matrix for the original three-dimensional object point cloud data via a two-dimensional object projection, and to perform rotation correction on the three-dimensional object point cloud, the three-dimensional air point cloud and the furniture point cloud according to the rotation correction matrix.

[0115] In order to facilitate the subsequent correction of the wall direction and make the structure output result more stable, the present invention needs to rotate the architectural structure of the point cloud so that the main direction of the wall is parallel to the horizontal or vertical direction.

[0116] In the present invention, according to the three-dimensional object point cloud data, the projection and rotation correction module 202 extracts the ground plane as the xy plane through ransac (random sampling consensus algorithm), and then rotates the z axis to a direction perpendicular to the ground plane, and makes the z-axis coordinate of the lowest point 0, to obtain the transformation matrix T1; then the three-dimensional object point cloud data is projected onto a two-dimensional plane to obtain a projection image, and the Hough transform is used to detect the line segment in the projection image to determine the main direction of the line segment (the direction of the line segment with the longest total length is regarded as the main direction of the line segment), and the plane of the projection image is rotated around the z axis according to the angle of the main direction of the line segment, that is, it is rotated around the z axis until the line segment corresponding to the wall is parallel to the main direction of the coordinate axis (that is, the x-axis direction or the y-axis direction), and the xy coordinate minimum point is made the coordinate origin, to obtain the transformation matrix T2, and then according to the transformation matrices T1 and T2, the final rotation transformation matrix T is obtained, that is, T=T2*T1.

[0117] Finally, the rotation correction matrix is ​​used to perform rotation correction on the three-dimensional object point cloud, three-dimensional air point cloud and furniture point cloud respectively.

[0118] The room mask and corner point extraction association unit 3 is used to combine the corrected three-dimensional object point cloud and the furniture information of each point to generate an interference-resistant two-dimensional top-view object projection and a normal vector projection, and generate a two-dimensional room mask, two-dimensional corner points belonging to each room, and the association between corner points through three sets of neural networks. The three sets of neural networks are the room mask extraction network, the single room corner point extraction network, and the multi-room corner point association network.

[0119] Specifically, the room mask and corner point extraction association unit 3 further includes:

[0120] The anti-noise processing module 301 is used to extract points of interest based on the corrected three-dimensional object point cloud and the furniture point cloud, and output a noise-resistant two-dimensional top-view object projection and a two-dimensional top-view normal vector projection. The two-dimensional projection images used in subsequent predictions are these two projections.

[0121] Since the subsequent network detects independent room information and corresponding corner point information based on the two-dimensional contour, the three-dimensional point cloud data is projected onto a two-dimensional plane, so that the single-channel grayscale value of a pixel represents the number of points at different heights under the plane coordinates, and a two-dimensional top-view image after projection is obtained (after furniture recognition filtering). However, due to the existence of objects such as floors and furniture, as well as factors such as scanning errors and noise, the present invention sets a threshold value (the default value is 50) for the two-dimensional top-view image after projection as follows, and increases the grayscale value of wall points while filtering non-wall points, thereby obtaining a noise-resistant two-dimensional top-view object projection:

[0122]

[0123] At the same time, the average normal vector of each plane point in the vertical direction is calculated during the projection process, and the output is another three-channel projection image, that is, the output is a two-dimensional top-down normal vector projection. The normal vector can greatly assist the detection of corner points, because the normal vector of the wall should be parallel to the ground. In the present invention, the corner point refers to the corner where two walls meet. The normal vectors of these two walls are parallel to the ground but have different angles on the xy plane. The intersection of these two groups of normal vectors is the corner.

[0124] The room mask extraction module 302 is used to calculate the feature map of the input noise-resistant two-dimensional top-view object projection map using the classic Mask-RCNN network, and combine the region generation network (RPN) to generate candidate bounding boxes for each room, and perform semantic segmentation on the pixels in each bounding box to extract its room mask.

[0125] This step is performed before corner point calculation in order to simplify the structural detection of the entire room to the detection of a single room. Considering that the structure of most rooms is a cube or a similar collection, the detection can achieve higher accuracy at a lower difficulty.

[0126] The single room corner point extraction module 303 is used to extract features from the picture based on the two-dimensional room mask extracted by the room mask extraction module 302 and the noise-resistant two-dimensional top-view object projection image and the noise-resistant two-dimensional top-view normal vector projection image obtained by the noise-resistant processing module 301. The encoder composed of convolutional layers based on hole convolution then passes the feature map through a decoder based on the ResNet network structure and inputs the feature map into two parallel branches of the convolutional layer, thereby obtaining the two-dimensional coordinates of the corner points and the probability confidence values ​​of several directions in which the corner points extend out of the wall line segments.

[0127] Specifically, the room mask calculated by the room mask extraction module 302 is expanded so that it can completely cover the part of the room on the two-dimensional top-view projection and the two-dimensional normal vector projection. The encoder is composed of a convolutional layer based on dilated convolution to extract features from the picture, and then the feature map passes through a decoder module based on ResNet and is input into two parallel branches of the convolutional layer. Dilated convolution is used here to obtain a larger receptive field with a smaller number of parameters, while different strides can compress the image. The first branch predicts a set of confidence maps, each of which represents a different corner point of the room; the second branch predicts the degree of correlation between the corner points, that is, which two corner points may be connected to form a wall. Both sets of branches control the output results to probability values ​​within 0-1 through the Sigmoid layer.

[0128] The multi-room corner point association module 304 is used to combine the corner point information obtained by the single-room corner point extraction module 303 with the room mask, the two-dimensional overhead view object projection, and the two-dimensional overhead view normal vector projection, and combine the local room information with the overall floor information in the neural network, so as to predict the probability value of each corner point for each room in turn.

[0129] The room information merging module 305 is used to sort all rooms according to the number of pixel values ​​occupied by the masks, traverse and calculate the overlap rate between the room masks, and for rooms with a high overlap rate (the default is 30%), merge the masks and corner point information of the small rooms into the large room, and delete duplicate points.

[0130] There may not be a complete conclusion about the independence of some rooms. For example, a semi-open kitchen is connected to the living room but is blocked. For rooms like this, the calculation of the masks may overlap. In the present invention, such cases are merged into the same room. Therefore, all rooms are sorted according to the number of pixel values ​​occupied by the masks, and the overlap rate between the room masks is traversed and calculated. For rooms with a high overlap rate (the default is 30%), the masks and corner point information of the small rooms are merged into the large room, and the duplicate points are deleted.

[0131] The boundary generation unit 4 is used to generate the local boundary of the room and the global boundary of the floor according to the corner points, and optimize based on factors such as the corner points and parallel distances of the boundaries.

[0132] Specifically, the boundary generation unit 4 further includes:

[0133] The local boundary generation module 401 is used to optimize and generate the local boundary of each room based on the slope for the corner points associated and merged in each room.

[0134] Since the structure of a single room is usually simple and the corner points are formed around the boundary, in the present invention, the mask boundary of the single room is calculated by the eight-domain boundary tracking algorithm, and its corner points are matched with the nearest points on the boundary. The order of connecting the corner points is obtained by traversing the boundary, and the corner points are connected in sequence to obtain the boundary.

[0135] The prediction results of the neural network usually have certain deviations due to the accuracy of the prediction or the distribution of the data. In addition, in order not to miss the prediction results of corner points, the threshold of the confidence probability will be set relatively low, but this will result in some redundant corner points. Therefore, it is necessary to optimize the prediction results of the corner points of each room. Lock all corner points shared by multiple rooms to prevent the global structure from being affected. For the local corner points of a single room, calculate the slope value of two consecutive edges and their corresponding angle (the denominator plus the minimum value prevents it from being 0):

[0136]

[0137] If it is less than a certain threshold (the default is 30 degrees), it will be merged into the same edge. In addition, since the point cloud data has been normalized before, there is a priori assumption that most walls are parallel or perpendicular to the main direction. If the angle between the line segment and the horizontal or vertical direction is less than a certain threshold (the default is 18 degrees), it will be corrected to the horizontal or vertical edge.

[0138] The global boundary generation module 402 is used to merge local boundaries to generate a global boundary, and to close and merge boundaries based on corner points between boundary lines and parallel boundary spacing.

[0139] The room structure generating unit 5 is used to integrate the furniture position information and the passage (door) position information to generate the final two-dimensional room structure.

[0140] Specifically, the room structure generating unit 5 further includes:

[0141] The channel and gate extraction module 501 is used to fuse the optimized global boundary with the channel position information in the air point cloud.

[0142] The 3D air point cloud data is processed, and the positions that are all air within a certain height (the default is 0.5 meters to 2 meters) are defined as passable areas. Comparing the generated wall structure data with the passable area data, if for a section of wall, the proportion of air points within a certain range in a continuous length direction (the length is not less than the threshold, the default is 0.5 meters) and a continuous width direction (the width is not less than the threshold, the default is 0.5 meters) exceeds a certain threshold (the default is 0.5), then this section is considered to be a channel structure in the wall.

[0143] The final room structure generation module 502 is used to generate the final two-dimensional room structure according to the corrected three-dimensional furniture classification point cloud data and the boundary line segment information of the passage and door integrated into the passage and door extraction module 501.

[0144] In a specific embodiment of the present invention, the floor and wall parts are removed from the result of the furniture recognition network to obtain unconnected furniture clusters, and the number of three-dimensional points belonging to different furniture in each cluster is counted. The category with the largest voting value is taken as the furniture classification of the cluster, and its two-dimensional minimum bounding box information is obtained.

[0145] Compared with the prior art, the present invention has the following advantages:

[0146] 1. The existing technology only cares about the wall part in room type detection, and does not process the internal situation which can also express the room type status. The present invention combines the semantic segmentation network based on point cloud to identify 20 kinds of common furniture such as tables, chairs, beds, bookshelves, sofas, etc., and reflects them in the final room type detection results.

[0147] 2. The prior art only cares about the simplest possible wall segment structure, and discards all channel structures that will increase the number of points and lines. The present invention combines the information during three-dimensional data collection, defines the space between the data collector and the data point as an air point, and analyzes the spatial occupancy and connectivity of the air points near the wall position in the room structure recognition results, and digs and marks the channel position in the wall result segment.

[0148] 3. The prior art uses a strong prior-based energy function to optimize the corner points and line segments of each room. In order to make the result as concise as possible, a large number of detailed features will be lost, and the computational cost of the heuristic optimization method used will increase sharply when the complexity of the situation increases. The present invention optimizes the corner points and edges based on geometric features locally and globally. Compared with the optimization based on the energy function, it can maintain more detailed features and has a significant speed improvement. When the number of rooms, room shapes and other factors bring about an increase in complexity, the increase in resource and time consumption of this optimization method is only linear.

[0149] 4. The prior art lacks processing for the process of projecting 3D data to 2D. When the original 3D data contains interference items such as furniture, residents, and temporarily placed objects, direct projection will cause confusion in the input data. The present invention combines a semantic segmentation network based on point cloud, projects only the identified wall part to a 2D plane as input, and sets a threshold to filter out possible noise and prediction errors. This allows the input data to remain concise and correct when there are objects in the room.

[0150] 5. The prior art does not correct the main direction of the original three-dimensional data, but adopts the prior assumption of horizontal and vertical in the final optimization process, which may cause different results and errors for inputs at different angles. The present invention combines the detection of two-dimensional straight lines to keep the direction of the main wall of the input data parallel to the coordinate axis. This can not only eliminate the impact of angle offset in the data modeling process, making the output result more robust, but also make the prior assumption of horizontal and vertical for wall direction correction more reliable.

[0151] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Any person skilled in the art may modify and alter the above embodiments without violating the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be as set forth in the claims.

Claims

1. A room structure analysis method comprising the following steps: Step S1, using an RGBD camera and an odometer camera to scan the room that needs structural analysis to obtain RGBD camera data and odometer camera data, and converting them into three-dimensional object point cloud data and three-dimensional air point cloud data; Step S2, obtaining furniture point cloud data by recognizing the three-dimensional object point cloud, and obtaining a rotation correction matrix for the three-dimensional object point cloud data through two-dimensional object projection, and performing rotation correction on the three-dimensional object point cloud, the three-dimensional air point cloud and the furniture point cloud according to the obtained rotation correction matrix; Step S3, combining the corrected 3D object point cloud and the furniture information of each point to generate an interference-resistant 2D top-view object projection and normal vector projection, and respectively generating a 2D room mask, a 2D corner point of each room, and the correlation between the corner points through three sets of neural networks; Step S3 further includes: Step S300, extracting points of interest based on the corrected three-dimensional object point cloud and the furniture point cloud, thereby outputting noise-resistant two-dimensional top-view object projection and two-dimensional top-view normal vector projection; Step S301, using the classic Mask-RCNN network to calculate the feature map of the input noise-resistant two-dimensional top-view object projection image, and combining the region generation network to generate candidate bounding boxes for each room, while performing semantic segmentation on the pixels in each bounding box to extract its room mask; Step S302, according to the two-dimensional room mask extracted in step S301 and the noise-resistant two-dimensional top-view object projection map and the noise-resistant two-dimensional top-view normal vector projection map obtained in step S300, a convolutional layer based on dilated convolutions is used to form an encoder to extract features from the image, and then the feature map is passed through a decoder based on a ResNet network structure and input into two parallel branches of the convolutional layer, so as to obtain the two-dimensional coordinates of the corner points and the probability confidence values ​​of several directions in which the corner points extend out of the wall line segments; Step S303, combining the corner point information obtained in step S302 with the room mask, the two-dimensional top-view object projection, and the two-dimensional top-view normal vector projection, combining the local room information with the overall floor information in the neural network, so as to predict the belonging probability value of each corner point for each room in turn; Step S304, sort all rooms according to the number of pixel values ​​occupied by the masks, traverse and calculate the overlap rate between the room masks, and for rooms whose overlap rate exceeds a preset threshold, merge the masks and corner point information of the small rooms into the large room, and delete duplicate points; Step S4, generating a local boundary of the room and a global boundary of the floor according to the corner points, and optimizing based on the corner points and parallel distances of the boundaries; Step S4 further includes: Step S400, for the corner points associated and merged in each room, optimize and generate the local boundary of the room based on the slope; in step S400, lock all the corner points shared by multiple rooms unchanged, and for the local corner points of a single room, calculate the slope value of two consecutive edges and their corresponding angles, and if the slope value is less than a preset threshold, merge them into the same edge; Step S401, merging local boundaries to generate a global boundary, closing and merging boundaries based on corner points between boundary lines and parallel boundary spacing; Step S5, integrating the furniture position information and the passage position information to generate the final two-dimensional room structure.

2. A room structure analysis method as claimed in claim 1, characterized in that: Step S2 further comprises: Step S200, using sparse convolution to process the three-dimensional object point cloud data, based on the U-Net network, the processed data is first deepened after each downsampling convolution, and then symmetrically upsampled and the number of channels is gradually reduced, and finally after the full connection layer and Softmax, the category with the maximum probability is taken as the furniture category classification result of the point, thereby obtaining furniture point cloud data; Step S201 , obtaining a rotation correction matrix through two-dimensional object projection of the original three-dimensional object point cloud data, and performing rotation correction on the three-dimensional object point cloud, three-dimensional air point cloud and furniture point cloud data according to the rotation correction matrix.

3. A room structure analysis method as claimed in claim 2, characterized in that: In step S201, based on the original three-dimensional object point cloud data, the ground plane is extracted as the xy plane, and then the z-axis is rotated to a direction perpendicular to the ground plane, and the z-axis coordinate of the lowest point is made to be 0, so as to obtain a first transformation matrix; then the three-dimensional object point cloud data is projected onto a two-dimensional plane to obtain a projection image, and the line segments are detected in the projection image using the Hough transform to determine the main direction of the line segments, and the plane of the projection image is rotated around the z-axis according to the angle of the main direction of the line segments, and the point with the minimum xy coordinate value is made the origin of the coordinates to obtain a second transformation matrix, thereby obtaining a final rotation transformation matrix based on the first transformation matrix and the second transformation matrix.

4. A room structure analysis method as claimed in claim 2, characterized in that: In step S201, during rotation correction, the number and length of line segments are counted at different angles, the direction of the line segment with the longest total length is regarded as the main direction of the line segment, and the rotation angle is limited to within 45 degrees. The main direction of the line segment is rotated to the main direction, or a direction perpendicular to the main direction, so as to achieve the purpose of fine-tuning.

5. A room structure analysis method as claimed in claim 4, characterized in that , step S5 further comprises: Step S500, fusing the optimized global boundary with the channel position information in the air point cloud; Step S501, generating a final two-dimensional room structure based on the corrected three-dimensional furniture classification point cloud data and the boundary line segment information of the passage and the door integrated in step S500.

6. A room structure analysis device, comprising: A point cloud data generating unit, used for scanning a room requiring structural analysis using an RGBD camera and an odometer camera to obtain RGBD camera data and odometer camera data, and converting the data into three-dimensional object point cloud data and three-dimensional air point cloud data; A rotation correction unit is used to obtain furniture point cloud data by recognizing the three-dimensional object point cloud, and to obtain a rotation correction matrix by projecting the three-dimensional object point cloud through a two-dimensional object, and to perform rotation correction on the three-dimensional object point cloud, the three-dimensional air point cloud and the furniture point cloud according to the rotation correction matrix; The room mask and corner point extraction association unit is used to combine the corrected 3D object point cloud with the furniture information of each point to generate an interference-resistant 2D top-view object projection and normal vector projection, and generate a 2D room mask, 2D corner points belonging to each room, and the association between corner points through three sets of neural networks; extract points of interest based on the corrected 3D object point cloud and the furniture point cloud, and output a noise-resistant 2D top-view object projection and a 2D top-view normal vector projection; use the classic Mask-RCNN network to calculate the feature map of the input noise-resistant 2D top-view object projection map, and combine the region generation network to generate candidate bounding boxes for each room, and perform semantic segmentation on the pixels in each bounding box to extract its room mask; based on the extracted 2D room mask and the obtained noise-resistant 2D top-view object projection map, The noise-resistant two-dimensional top-view normal vector projection map, the convolution layer based on the hole convolution constitutes an encoder to extract features from the picture, and then the feature map passes through a decoder based on the ResNet network structure and is input into two parallel branches of the convolution layer, so as to obtain the two-dimensional coordinates of the corner points and the probability confidence values ​​of several directions in which the corner points extend out of the wall line segment; the obtained corner point information is combined with the room mask, the two-dimensional top-view object projection, and the two-dimensional top-view normal vector projection, and the local room information and the overall floor information are combined in the neural network, so as to predict the probability value of each corner point for each room in turn; all rooms are sorted according to the number of pixel values ​​occupied by the mask, and the overlap rate between the room masks is traversed and calculated. For rooms whose overlap rate exceeds the preset threshold, the mask and corner point information of the small room are merged into the large room, and duplicate points are deleted at the same time; The boundary generation unit is used to generate the local boundary of the room and the global boundary of the floor based on the corner points, and optimize based on the corner points and parallel distance of the boundary; for the corner points after being associated and merged in each room, the local boundary belonging to the room is optimized and generated based on the slope; all corner points shared by multiple rooms are locked unchanged, and for the local corner points of a single room, the slope values ​​of two consecutive edges and their corresponding angles are calculated, and if they are less than a preset threshold, they are merged into the same edge; merge the local boundaries to generate the global boundary, and close and merge the boundaries based on the corner points between the boundary lines and the parallel boundary spacing respectively; The room structure generation unit is used to integrate the furniture position information and the channel position information to generate the final two-dimensional room structure.

Images