A method for reconstructing indoor floor plans based on graph-based 3D scanning

Abstract

The present invention discloses a method for reconstructing a three-dimensional scanned indoor floor plan based on a graph, which is characterized by comprising the following steps: (1) obtaining an open-source data set; (2) constructing a primitive estimation network SeqPNet to obtain a wall point heat map; the primitive estimation network SeqPNet includes: a wall heat map regression layer, a room heat map regression layer, and a point heat map regression layer; successively generating three types of heat maps, namely, wall B, room R, and point P masks to obtain a wall point heat map; (3) performing triangulation and triangle pruning on the wall point heat map to obtain an initial graph; (4) solving an optimal subgraph according to the initial graph to obtain a final reconstructed structure; (5) evaluating the reconstruction result; the present invention effectively captures details and maintains the overall integrity of the floor plan.

Description

Technical Field

[0001] The present invention relates to the technical field of indoor planar structure reconstruction, and particularly relates to a method for reconstructing an indoor floor plan based on 3D scanning of a graph. Background Art

[0002] Currently, there are two methods for planar reconstruction: The first uses a bottom-up primitive detection method, directly extracts corner positions from the input, and obtains a set of line segments as the planar structure representation by combining and predicting edge relationships. This method usually uses an end-to-end network for inference and has a relatively fast inference speed. However, the result is highly affected by the accuracy of the original detection in the first stage, and the lack of global structure information leads to low result integrity (there are unclosed rooms). The second method usually uses a top-down heuristic room optimization method, extracts room masks based on image segmentation and heuristically optimizes the contours of the rooms, and finally combines them into the overall structure. However, this heuristic method has a too high computational time cost, and the result has a certain gap from the CAD-style planar representation. Summary of the Invention

[0003] Object of the Invention: The object of the present invention is to provide a method for reconstructing an indoor floor plan based on 3D scanning of a graph, to solve the problem that false negatives or false positives of the results in the bottom-up method for low-level primitives (such as corner points and connecting edges) will directly lead to false negatives or false positives of the corresponding high-level primitives (rooms); to solve the problem of long calculation time in the method based on subgraph optimization; and to solve the problem of poor structural consistency and compactness of the reconstruction results of most methods.

[0004] Technical Solution: A method for reconstructing an indoor floor plan based on 3D scanning of a graph according to the present invention includes the following steps:

[0005] (1) Obtain an open-source dataset;

[0006] (2) Construct a primitive estimation network SeqPNet to obtain a wall point heat map; the primitive estimation network SeqPNet includes: a wall heat map regression layer, a room heat map regression layer, and a point heat map regression layer; successively generate three types of heat maps, namely wall B, room R, and point P masks to obtain the wall point heat map;

[0007] (3) Perform triangular pruning on the wall point heat map to obtain an initial graph;

[0008] (4) Solve the optimal subgraph according to the initial graph to obtain the final reconstruction structure;

[0009] (5) Evaluate the reconstruction result.

[0010] Further, in step (2), the primitive estimation network SeqPNet is constructed to obtain the initial graph as follows: Using the 2D density / normal image as the input, three types of heatmaps, namely wall heatmap B, room heatmap R, and point heatmap P masks, are sequentially generated through the wall heatmap regression layer, room heatmap regression layer, and point heatmap regression layer; each mask guides the generation of the subsequent mask.

[0011] Further, in step (2), the wall heatmap regression layer is as follows: For the given 2D height / density map F Use a four-layer CNN for preliminary feature extraction to obtain , where C is the number of channels; After being processed by the hourglass module HG, ConvBlocks transformation block, and 2 conv3x3 layers, the intermediate feature is obtained; at the same time, After being processed by 2 conv3x3 layers, the output feature is obtained; , and After feature fusion through the hyperperceptive cross-attention HPCA, is obtained, that is, the wall mask B is generated.

[0012] Further, the feature fusion by the hyperperceptive cross-attention HPCA is as follows: HPCA includes a convolutional branch for extracting local information and a sparse attention branch for extracting global information; among them, in the HPCA sparse attention weighting branch, first, , , are respectively used as the query Value, key Key, and value Query for weighted calculation, and then through the multi-layer perceptron MLP, the sequences Q, K, and V are respectively converted into values q, keys k, and queries v of size ; token selection is performed on k and v; the output of HPCA is used as the input of the room heatmap regression layer; where, R represents the matrix; N represents the row, C represents the column, and m represents the sequence index number.

[0013] Further, in step (2), the room heatmap regression layer is as follows: First, After being processed by the hourglass module HG, ConvBlocks transformation block, and 2 conv3x3 layers, the intermediate feature is obtained; at the same time, After being processed by 2 conv3x3 layers, the output feature is obtained; , and Feature fusion is performed through Hyper-Sensing Cross Attention (HPCA) to obtain That is, a room mask R is generated; is used as the input of the point heatmap regression layer.

[0014] Furthermore, in step (2), the point heatmap regression layer is specifically as follows: After passing through the Hourglass Module (HG), ConvBlocks transformation block, two conv3x3 layers, and upsampling, a wall point heatmap is obtained.

[0015] Furthermore, the loss of the primitive estimation network SeqPNet consists of three parts, namely the wall heatmap loss , the room heatmap loss and the point heatmap loss , and the formula is as follows:

[0016] ;

[0017] Among them, 、 、 represents the weight; L represents the overall loss of the network;

[0018] ;

[0019] ;

[0020] ;

[0021] Among them, y is the ground truth of the wall heatmap, is the predicted wall heatmap, , , , are hyperparameters, which are set to 14, 1, 2.1, and 0.5 respectively.

[0022] ;

[0023] Among them, is the ground truth of the room heatmap, is the predicted room heatmap, n represents the total number of samples, and i represents the index number;

[0024] ;

[0025] Among them, is the ground truth of the point heatmap, is the predicted point heatmap, n represents the total number of samples, and p represents the index number.

[0026] Further, step (3) is specifically as follows: For the triangle set T generated by the wall point set V, delete the longest edges of the triangles in T to obtain the edge connection set E, and obtain the graph G. The formula is as follows: ; ; ;

[0027] where, is the longest edge in t. Delete the points in G with degrees less than 2, and the finally generated graph is used as the initial graph; is the edge in triangle t.

[0028] Further, step (4) is specifically as follows: Solve the optimal subgraph based on the structural weight, and optimize the graph using the greedy algorithm; among them, the structural weight includes the confidence term and the length term.

[0029] Further, step (5) is specifically as follows: Use the new metrics and to evaluate the reconstruction result; specifically as follows:

[0030] That is, the structural consistency metric. The formula is as follows:

[0031] ;

[0032] where, represents the number of rooms overlapping with other rooms, represents the number of hanging points and cutting points in the structure, and respectively represent the number of correctly reconstructed rooms and the number of corner points, is the scaling factor;

[0033] MAnE That is, the metric metric, which measures the overall deformation of the structure; the formula is as follows:

[0034] ;

[0035] where, represents the i-th angular deviation with all angular deviations within 5 degrees, is the number of false negative angles, is the number of correctly reconstructed angles; for the angles that are not correctly reconstructed, an angular deviation is assigned;

[0036] That is, the compactness metric, which measures the compactness of the reconstructed structure. The formula is as follows:

[0037] ;

[0038] where, Indicates the number of rooms correctly reconstructed, Indicates the total number of corner points in the reconstruction result.

[0039] Beneficial effects: Compared with the prior art, the present invention has the following remarkable advantages: Different from the existing pixel-level optimization methods, the graph representation of the present invention can greatly reduce the search space by establishing a new primitive (i.e., wall points); A guided primitive estimation network is proposed to generate points that can capture more comprehensive wall structure information, which has higher fault tolerance compared with traditional primitive corner points; A new structural weight is designed, which takes into account both the confidence of the real wall and the influence of the wall length. By introducing it into the subgraph optimization based on the greedy algorithm, the reconstruction result has less structural deformation and contains more structural details compared with other methods; For the graph structure generated by the triangulation method that does not change the vertex set, a triangular pruning processing algorithm is designed. This algorithm is based on the fact that there are few acute angles in indoor architectural design, and preliminarily reduces the search space of the subgraph optimization based on the greedy algorithm; Different from the metrics that individually measure the quality of single primitive reconstruction (recall / precision of rooms / edges / corner points), three metrics are introduced to evaluate the overall structural quality for more comprehensive and thorough comparison; Experiments show that the present invention effectively captures details and maintains the overall integrity of the floor plan. Description of the Drawings

[0040] Figure 1 is the flow schematic diagram of the present invention;

[0041] Figure 2 is the schematic diagram of the network structure prototype of the present invention;

[0042] Figure 3 Represents the process of triangular pruning of the present invention;

[0043] Figure 4 Represents the process of solving the optimal subgraph of the present invention. Detailed Embodiment

[0044] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0045] As Figure 1 shown, the embodiment of the present invention provides a method for reconstructing a 3D scanned indoor floor plan based on a graph, including the following steps:

[0046] (1) Obtain open-source datasets: Structured3 dataset; Lianjia-s dataset.

[0047] (2) As Figure 2As shown, the primitive estimation network SeqPNet is constructed to obtain the wall point heat map; the primitive estimation network SeqPNet includes: a wall heat map regression layer, a room heat map regression layer, and a point heat map regression layer; three types of heat maps, namely wall B, room R, and point P masks, are generated in sequence to obtain the wall point heat map; the construction of the primitive estimation network SeqPNet to obtain the initial map is as follows: taking the 2D density / normal image as the input, sequentially passing through the wall heat map regression layer, the room heat map regression layer, and the point heat map regression layer to generate three types of heat maps in sequence: wall B, room R, and point P masks; each mask guides the generation of the subsequent mask. Among them, the wall heat map regression layer is as follows: for the given 2D height / density map F using a four-layer CNN for preliminary feature extraction to obtain , where C is the number of channels; after being processed by the hourglass module HG, the ConvBlocks transformation block, and 2 conv3x3 layers, the intermediate feature is obtained; at the same time, after being processed by 2 conv3x3 layers, the output feature is obtained; , and are subjected to feature fusion through the hyperperceptive cross-attention HPCA to obtain that is, the wall mask B is generated.

[0048] The feature fusion by the hyperperceptive cross-attention HPCA is as follows: HPCA includes a convolutional branch for extracting local information and a through-sparse attention branch for extracting global information; among them, in the HPCA sparse attention weighting branch, first , , are used as the query Value, key Key, and value Query for weighted calculation respectively, and then through the multi-layer perceptron MLP, the sequences Q, K, and V are respectively converted into values q, key k, and query v with a size of ; token selection is performed on k and v; the output of HPCA is used as the input of the room heat map regression layer; where, R represents the matrix; N represents the row, C represents the column, and m represents the sequence index number. The calculation process is as follows:

[0049] First, calculate the contribution score of the token according to the rows and columns of q and k ( and ). The calculation process is:

[0050] , (1)

[0051] , (2)

[0052] Sparse attention reduces the computational cost by retaining the rows and columns with high contributions and pruning the others. In equations (1) - (2), , is the i-th column vector in q. is the feature vector, 、 are the slices based on the first and second dimensions, representing the vector of the r-th row and j-th column, and the vector of the j-th row and c-th column respectively. Then, select rows and columns according to the following conditions:

[0053]

[0054]

[0055]

[0056] Among them, . Based on and select tokens (reshaped k) and (reshaped v) for the sparse attention weighted calculation in HPCA: and , for the sparse attention weighted calculation in HPCA:

[0057]

[0058] In equation (6), is the number of attention heads, set to 8. Use HPCA for cross-information fusion to make the network have more accurate reconstruction ability.

[0059] The room heatmap regression layer is as follows: First, is processed through the hourglass module HG, the ConvBlocks transformation block, and 2 conv3x3 layers to obtain the intermediate feature ; at the same time, is processed through 2 conv3x3 layers to obtain the output feature ; 、 and are subjected to feature fusion through the hyper-perceptual cross-attention HPCA to obtain that is, generate the room mask R; is used as the input of the point heatmap regression layer.

[0060] The point heatmap regression layer is as follows: Take After passing through the hourglass module HG, the ConvBlocks transformation block, two conv3x3 layers, and upsampling processing, a wall point heat map is obtained.

[0061] The loss of the primitive estimation network SeqPNet consists of three parts: the wall heat map loss , the room heat map loss and the point heat map loss , and the formula is as follows:

[0062] ;

[0063] Among them, 、 、 represents the weight; L represents the overall loss of the network;

[0064] ;

[0065] ;

[0066] ;

[0067] Among them, y is the ground truth of the wall heat map, is the predicted wall heat map, , , , are hyperparameters, which are set to 14, 1, 2.1, and 0.5 respectively.

[0068] ;

[0069] Among them, is the ground truth of the room heat map, is the predicted room heat map, n represents the total number of samples, and i represents the index number;

[0070] ;

[0071] Among them, is the ground truth of the point heat map, is the predicted point heat map, n represents the total number of samples, and p represents the index number.

[0072] (3) Perform triangular pruning on the wall point heat map to obtain an initial map; as Figure 3 shown, specifically as follows: For the triangular set T generated by the wall point set V, delete the longest side of the triangles in T to obtain the edge connection set E, and obtain the graph G. The formula is as follows:

[0073] The formula is as follows: ; ; ;

[0074] Among them, is the longest side in t. Delete the points in G with degrees less than 2, and the finally generated graph is used as the initial graph; is the edge in the triangle t.

[0075] (4) Solve the optimal subgraph based on the initial graph to obtain the final reconstructed structure; as Figure 4 shown, specifically as follows: Solve the optimal subgraph based on the structure weight, and use the greedy algorithm to optimize the graph; among them, the structure weight includes a confidence term and a length term. The specific process is as follows:

[0076] Confidence term ( ): The weight of this term reflects the probability that the connected edge is a real wall, which is jointly calculated by the room (R) and wall (B) masks predicted by the primitive estimation network. The higher the heat value of the connected edge corresponding in the room heat map , the lower the probability that the connected edge is a wall, while the higher the heat value corresponding in the wall mask , the higher the probability that it is a wall. is used to measure the comprehensive probability of authenticity. The confidence term is calculated by the following formula, where and are the two endpoints of e:

[0077] ;

[0078] ;

[0079] ;

[0080] Length term ( ): It is used to refine the confidence of short edges in the structure weight. Due to the slight misalignment between the wall points and the edge heat map, short edges are more sensitive to this misalignment, resulting in lower confidence and may be discarded. Therefore, is used to refine the weight of short edges in. Specifically, when processing edges with low confidence, set a threshold . For edges where is less than zero, if the length is less than , then set to 1 to ensure that the weight of the final edge is greater than 0. For edges with higher confidence, no adjustment is made, and the result is . The reason is that edges with higher confidence values do not need to be adjusted. The formula is as follows:

[0081] ;

[0082] The overall formula for the structural weight is as follows:

[0083] ;

[0084] To ensure the integrity of the wall connection edges, it should be ensured that the degree of each node in the figure is greater than or equal to 2, that is, there are no cut points and cut edges in the optimal graph. Therefore, the connection edges with a node degree less than 2 are set to -100.

[0085] Optimal subgraph solution: Based on the proposed structural weights, a greedy algorithm is used to optimize the graph, improving the accuracy and efficiency of structural reconstruction. Based on the definition of structural weights, the goal of this step is to find a subgraph with the largest sum of weights. The greedy strategy can ensure that the maximum value is achieved at each step. It further ensures that the sum of all structural weights in the final subgraph is the largest, indicating a locally optimal state. It should be noted that all negative edges are traversed in sequence, and all deletable edges are deleted step by step according to the optimization strategy. For the negative edges retained in the final subgraph, any attempt to delete them will inevitably lead to a decrease in the sum of structural weights. Although it cannot be guaranteed that the obtained subgraph is globally optimal, using the locally optimal strategy can significantly improve the computational efficiency, and the use of structural weights ensures the reliability of the algorithm. In addition, to improve the computational efficiency of the greedy algorithm, it is carried out on an increasing sequence queue Q, which only contains edges with structural weights less than 0. On the queue Q, the edges in the queue are traversed according to the following steps:

[0086] (a) Calculate the sum of the structural weights of all edges in the current graph, denoted as W.

[0087] (b) After removing the edge with the smallest structural weight in the queue, calculate the sum of the structural weights of all edges in the current graph, denoted as .

[0088] (c) Compare the values of W and . If the structural weight increases (W > ), remove the edge with the smallest weight. Otherwise, retain the connection edge. Pop the first element from the queue.

[0089] (d) Repeat steps (a)-(c) until the queue is empty.

[0090] To make the final reconstruction result more compact, a set of connected edges composed of degree-2 nodes is selected for Douglas-Peucker curve simplification, and then the degree-2 nodes forming an angle greater than 160 degrees are deleted to obtain the final result.

[0091] (5) Evaluate the reconstruction result. Specifically as follows: Use the new metrics and to evaluate the reconstruction result; specifically as follows:

[0092] That is, the structural consistency index, and the formula is as follows:

[0093] ;

[0094] Wherein, represents the number of rooms overlapping with other rooms, represents the number of suspension points and cutting points in the structure, and respectively represent the number of correctly reconstructed rooms and the number of corner points, is the scaling factor;

[0095] MAnE That is, the metric index, which measures the overall deformation of the structure; the formula is as follows:

[0096] ;

[0097] Wherein, represents the i-th angular deviation with all angular deviations within 5 degrees, is the number of false negative angles, is the number of correctly reconstructed angles; for angles that are not correctly reconstructed, an angular deviation is assigned ;

[0098] That is, the compactness index, which measures the compactness of the reconstructed structure, and the formula is as follows:

[0099] ;

[0100] Wherein, represents the number of correctly reconstructed rooms, represents the total number of corner points in the reconstruction result.

Claims

1. A method for reconstructing indoor plan views from three-dimensional scanning based on a graph, characterized in that: The following steps are involved: (1) Obtain open source datasets; (2) Construct a primitive estimation network SeqPNet to obtain a wall point heat map; the primitive estimation network SeqPNet includes: a wall heat map regression layer, a room heat map regression layer and a point heat map regression layer; generate three types of heat map masks of wall B, room R and point P in sequence to obtain a wall point heat map; the wall heat map regression layer is as follows: for a given 2D height / density map F Use four-layer CNN to extract features , where C is the number of channels; After the hourglass module HG, ConvBlocks conversion block and 2 conv3x3 layers, the intermediate features are obtained ; At the same time After 2 conv3x3 layers, the output features are obtained ;Will , and After super-perceptual cross attention HPCA, feature fusion is performed to obtain That is, a wall mask B is generated; (3) Triangulate and prune the wall point heat map to obtain the initial map; (4) Solve the optimal subgraph based on the initial graph to obtain the final reconstructed structure; (5) Evaluate the reconstruction results.

2. The method for reconstructing indoor plan views based on three-dimensional scanning according to claim 1, characterized in that: In step (2), the primitive estimation network SeqPNet is constructed to obtain the initial image as follows: taking the 2D density / normal image as input, three types of heat maps are sequentially generated through the wall heat map regression layer, the room heat map regression layer and the point heat map regression layer: wall B, room R and point P masks; each mask guides the generation of subsequent masks.

3. The method for reconstructing indoor plan views based on three-dimensional scanning according to claim 1, characterized in that: The feature fusion of super-perceptual cross-attention HPCA is as follows: HPCA includes a convolution branch for extracting local information and a sparse attention branch for extracting global information; among them, in the HPCA sparse attention weighted branch, first , , The sequences Q, K and V are converted into integers of size 1 through the multilayer perceptron MLP. The value q, key k and query v of ; token selection is performed on k and v; the output of HPCA is used as the input of the room heat map regression layer; where ; R represents the matrix; N represents the row, C represents the column; m represents the sequence index number.

4. The method for reconstructing indoor plan views based on three-dimensional scanning of a graph according to claim 2, characterized in that: In step (2), the room heat map regression layer is as follows: First, After the hourglass module HG, ConvBlocks conversion block and 2 conv3x3 layers, the intermediate features are obtained ; At the same time After 2 conv3x3 layers, the output features are obtained ;Will , and After super-perceptual cross attention HPCA, feature fusion is performed to obtain That is, generate the room mask R; As input to the point heatmap regression layer.

5. The method for reconstructing indoor plan views based on three-dimensional scanning of a graph according to claim 2, characterized in that: In step (2), the point heat map regression layer is as follows: After the hourglass module HG, ConvBlocks conversion block, 2 conv3x3 layers and upsampling processing, the wall point heat map is obtained.

6. The method for reconstructing indoor plan views based on three-dimensional scanning according to claim 2, characterized in that: The primitive estimation network SeqPNet loss consists of three parts: wall heat map loss , room heat map loss and point heatmap loss , the formula is as follows: ； in, 、 、 represents weight; L represents the overall loss of the network; ； ； ； Among them, y is the true value of the wall heat map, is the predicted wall heat map, , , , is a hyperparameter; ； in, is the true value of the room heat map, To predict the room heat map, n represents the total number of samples and i represents the index number; ； in, is the true value of the point heat map, is the predicted point heat map, n represents the total number of samples, and p represents the index number.

7. The method for reconstructing indoor plan views based on three-dimensional scanning according to claim 1, characterized in that: Step (3) is as follows: For the triangle set T generated by the wall point set V, delete the longest side of the triangle in T to obtain the edge set E, and obtain the graph G, the formula is as follows: ; ; ; in, is the longest edge in t, delete the points with degree less than 2 in G, and the resulting graph is used as the initial graph; is a side of triangle t.

8. The method for reconstructing indoor plan views based on three-dimensional scanning of a graph according to claim 1, characterized in that: Step (4) is as follows: solve the optimal subgraph based on the structural weight and optimize the graph using a greedy algorithm; wherein the structural weight includes a confidence term and a length term.

9. The method for reconstructing indoor plan views based on three-dimensional scanning of a graph according to claim 1, characterized in that: Step (5) is as follows: Using the new indicator and Evaluate the reconstruction results; details are as follows: That is, the structural consistency index, the formula is as follows: ； in, Indicates the number of rooms that overlap with other rooms. Indicates the number of hanging points and cutting points in the structure, and Represent the number of correctly reconstructed rooms and corner points, is the scaling factor; MAnE That is, a metric that measures the overall deformation of the structure; the formula is as follows: ； in, represents the i-th angle deviation of all angle deviations within 5 degrees, is the number of false negative angles, is the number of correctly reconstructed angles; for angles that are not correctly reconstructed, an angle deviation is assigned ; That is, the compactness index, which measures the compactness of the reconstructed structure, and the formula is as follows: ； in, represents the number of rooms that were correctly reconstructed, Indicates the total number of corner points in the reconstruction result.

Images