Graph-based three-dimensional scanning indoor plane graph reconstruction method

Abstract

The invention discloses a graph-based three-dimensional scanning indoor plane graph reconstruction method, which is characterized by comprising the following steps of: (1) acquiring an open source data set; (2) constructing a primitive estimation network SeqPNet to obtain a wall point heat map; the primitive estimation network SeqPNet comprises a wall heat map regression layer, a room heat map regression layer and a point heat map regression layer; sequentially generating three types of heat map wall B, room R and point P masks to obtain a wall point heat map; (3) triangularizing and triangularly pruning the wall point heat map to obtain an initial map; (4) solving an optimal sub-graph according to the initial graph to obtain a final reconstruction structure; (5) evaluating a reconstruction result; according to the method, details are effectively captured, and the overall integrity of the planar graph is kept.

Description

Technical Field

[0001] The present invention relates to the technical field of indoor plane structure reconstruction, and in particular to a method for reconstructing an indoor plane map based on a three-dimensional scan. Background Art

[0002] There are currently two methods for plane reconstruction: the first uses a bottom-up primitive detection method to directly extract corner positions from the input, and obtains a set of line segments as a plane structure representation by combining and predicting edge relationships. This method usually uses an end-to-end network for reasoning, which has a faster reasoning speed. However, the results are highly affected by the original detection accuracy in the first stage, and the lack of global structural information leads to low integrity of the results (there are unclosed rooms). The second method is usually a top-down heuristic room optimization method, which extracts room masks based on image segmentation and heuristically optimizes the outline of the room, and finally combines them into an overall structure. However, this heuristic method has a high computational time cost, and the results are somewhat different from the CAD-style plane representation. Summary of the invention

[0003] Purpose of the invention: The purpose of the present invention is to provide a graph-based method for reconstructing indoor plan views from three-dimensional scans, to solve the problem in bottom-up methods that false negatives or false positives for low-level primitives (such as corner points and edges) will directly lead to false negatives or false positives for corresponding high-level primitives (rooms); to solve the problem of long calculation time in subgraph optimization methods; and to solve the problem of poor structural consistency and compactness of reconstruction results of most methods.

[0004] Technical solution: The method for reconstructing an indoor plan view based on a three-dimensional scan of a graph according to the present invention comprises the following steps: (1) Obtain open source datasets; (2) Construct a primitive estimation network SeqPNet to obtain a wall point heat map; the primitive estimation network SeqPNet includes: a wall heat map regression layer, a room heat map regression layer, and a point heat map regression layer; generate three types of heat map masks of wall B, room R, and point P in sequence to obtain a wall point heat map; (3) Perform triangle pruning on the wall point heat map to obtain the initial map; (4) Solve the optimal subgraph based on the initial graph to obtain the final reconstructed structure; (5) Evaluate the reconstruction results.

[0005] Furthermore, in step (2), the primitive estimation network SeqPNet is constructed to obtain the initial image as follows: taking the 2D density / normal image as input, three types of heat maps are sequentially generated through the wall heat map regression layer, the room heat map regression layer and the point heat map regression layer: wall B, room R and point P masks; each mask guides the generation of subsequent masks.

[0006] Furthermore, in step (2), the wall heat map regression layer is as follows: For a given 2D height / density map F Use four-layer CNN to extract features , where C is the number of channels; After the hourglass module HG, ConvBlocks conversion block and 2 conv3x3 layers, the intermediate features are obtained ; At the same time After 2 conv3x3 layers, the output features are obtained ;Will , and After super-perceptual cross attention HPCA, feature fusion is performed to obtain That is, the wall mask B is generated.

[0007] Furthermore, the feature fusion of super-perceptual cross-attention HPCA is as follows: HPCA includes a convolution branch for extracting local information and a sparse attention branch for extracting global information; among them, in the HPCA sparse attention weighted branch, first , , The sequences Q, K and V are converted into integers of size 1 through the multilayer perceptron MLP. The value q, key k and query v of are selected; tokens are selected for k and v; the output of HPCA is used as the input of the room heat map regression layer; where R represents a matrix; N represents a row, C represents a column, and m represents a sequence index number.

[0008] Furthermore, in step (2), the room heat map regression layer is as follows: First, After the hourglass module HG, ConvBlocks conversion block and 2 conv3x3 layers, the intermediate features are obtained ; At the same time After 2 conv3x3 layers, the output features are obtained ;Will , and After super-perceptual cross attention HPCA, feature fusion is performed to obtain That is, generate the room mask R; As input to the point heatmap regression layer.

[0009] Furthermore, in step (2), the point heat map regression layer is as follows: After the hourglass module HG, ConvBlocks conversion block, 2 conv3x3 layers and upsampling processing, the wall point heat map is obtained.

[0010] Furthermore, the primitive estimation network SeqPNet loss consists of three parts: wall heat map loss , room heat map loss and point heatmap loss , the formula is as follows: ; in, 、 、 represents weight; L represents the overall loss of the network; ; ; ; Among them, y is the true value of the wall heat map, is the predicted wall heat map, , , , are hyperparameters and are set to 14, 1, 2.1, and 0.5 respectively.

[0011] ; in, is the true value of the room heat map, To predict the room heat map, n represents the total number of samples and i represents the index number; ; in, is the true value of the point heat map, is the predicted point heat map, n represents the total number of samples, and p represents the index number.

[0012] Furthermore, step (3) is as follows: for the triangle set T generated by the wall point set V, delete the longest side of the triangle in T to obtain the edge set E, and obtain the graph G, the formula is as follows: ; ; ; in, is the longest edge in t, delete the points with degree less than 2 in G, and the resulting graph is used as the initial graph; is a side of triangle t.

[0013] Furthermore, step (4) is specifically as follows: solving the optimal subgraph based on the structural weight, and optimizing the graph using a greedy algorithm; wherein the structural weight includes a confidence term and a length term.

[0014] Furthermore, step (5) is as follows: Using the new indicator and Evaluate the reconstruction results; details are as follows: That is, the structural consistency index, the formula is as follows: ; in, Indicates the number of rooms that overlap with other rooms. Indicates the number of hanging points and cutting points in the structure, and Represent the number of correctly reconstructed rooms and corner points, is the scaling factor; MAnE That is, a metric that measures the overall deformation of the structure; the formula is as follows: ; in, represents the i-th angle deviation of all angle deviations within 5 degrees, is the number of false negative angles, is the number of correctly reconstructed angles; for angles that are not correctly reconstructed, an angle deviation is assigned ; That is, the compactness index, which measures the compactness of the reconstructed structure, and the formula is as follows: ; in, represents the number of rooms that were correctly reconstructed, Indicates the total number of corner points in the reconstruction result.

[0015] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: Unlike the existing pixel-level optimization method, the graph representation of the present invention can greatly reduce the search space by establishing a new primitive (i.e., wall point); a guided primitive estimation network is proposed to generate points that can capture more comprehensive wall structure information, which has higher fault tolerance than traditional primitive corner points; a new structural weight is designed, which takes into account both the confidence of the real wall and the influence of the wall length. By introducing it into the subgraph optimization based on the greedy algorithm, the reconstruction result has smaller structural deformation and contains more structural details compared with other methods; for the graph structure generated by the triangulation method without changing the vertex set, a triangle pruning processing algorithm is designed. This algorithm is based on the fact that there are few sharp angles in indoor architectural design, and preliminarily reduces the search space of the subgraph optimization based on the greedy algorithm; different from the indicators that measure the quality of a single primitive reconstruction (recall rate / precision rate of room / edge / corner point), three indicators are introduced to evaluate the overall structural quality for a more comprehensive and comprehensive comparison; experiments show that the present invention effectively captures details and maintains the overall integrity of the floor plan. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 It is a schematic diagram of the network structure prototype of the present invention; Figure 3 The process of triangle pruning of the present invention is shown; Figure 4 It represents the process of solving the optimal subgraph of the present invention. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0018] like Figure 1 As shown, an embodiment of the present invention provides a method for reconstructing an indoor plan view based on a three-dimensional scan, comprising the following steps: (1) Obtain open source datasets: Structured3 dataset; Lianjia-s dataset.

[0019] (2) If Figure 2As shown, the primitive estimation network SeqPNet is constructed to obtain the wall point heat map; the primitive estimation network SeqPNet includes: wall heat map regression layer, room heat map regression layer and point heat map regression layer; three types of heat map wall B, room R and point P masks are generated in sequence to obtain the wall point heat map; the primitive estimation network SeqPNet is constructed to obtain the initial map as follows: taking the 2D density / normal image as input, the sequence generates three types of heat maps in sequence through the wall heat map regression layer, room heat map regression layer and point heat map regression layer: wall B, room R and point P mask; each mask guides the generation of subsequent masks. Among them, the wall heat map regression layer is as follows: for a given 2D height / density map F Use four-layer CNN to extract features , where C is the number of channels; After the hourglass module HG, ConvBlocks conversion block and 2 conv3x3 layers, the intermediate features are obtained ; At the same time After 2 conv3x3 layers, the output features are obtained ;Will , and After super-perceptual cross attention HPCA, feature fusion is performed to obtain That is, the wall mask B is generated.

[0020] The feature fusion of super-perceptual cross-attention HPCA is as follows: HPCA includes a convolution branch for extracting local information and a sparse attention branch for extracting global information; among them, in the HPCA sparse attention weighted branch, first , , The sequences Q, K and V are converted into integers of size 1 through the multilayer perceptron MLP. The value q, key k and query v are selected for k and v. The output of HPCA is used as the input of the room heat map regression layer. R represents a matrix, N represents a row, C represents a column, and m represents a sequence index. The calculation process is as follows: First, calculate the contribution score of the token based on the rows and columns of q and k ( and ). The calculation process is: , (1) , (2) Sparse attention reduces computational cost by retaining rows and columns with high contributions and pruning other rows and columns. In formula (1)-formula (2), , is the i-th column vector in q. is the eigenvector, , is a slice based on the first and second dimensions, representing the vector of the rth row and jth column, and the vector of the jth row and cth column, respectively. Then, select according to the following conditions Rows and columns:

[0021]

[0022]

[0023] in, .based on and from (reshape k) and (Reshape v) Select token and , used for sparse attention weighted computation in HPCA:

[0024] In formula (6), is the number of attention heads, which is set to 8. HPCA is used for cross-information fusion to enable the network to have more accurate reconstruction capabilities.

[0025] The room heat map regression layer is as follows: First, After the hourglass module HG, ConvBlocks conversion block and 2 conv3x3 layers, the intermediate features are obtained ; At the same time After 2 conv3x3 layers, the output features are obtained ;Will , and After super-perceptual cross attention HPCA, feature fusion is performed to obtain That is, generate the room mask R; As input to the point heatmap regression layer.

[0026] The point heat map regression layer is as follows: After the hourglass module HG, ConvBlocks conversion block, 2 conv3x3 layers and upsampling processing, the wall point heat map is obtained.

[0027] The primitive estimation network SeqPNet loss consists of three parts: wall heat map loss , room heat map loss and point heatmap loss , the formula is as follows: ; in, 、 、 represents weight; L represents the overall loss of the network; ; ; ; Among them, y is the true value of the wall heat map, is the predicted wall heat map, , , , are hyperparameters and are set to 14, 1, 2.1, and 0.5 respectively.

[0028] ; in, is the true value of the room heat map, To predict the room heat map, n represents the total number of samples and i represents the index number; ; in, is the true value of the point heat map, is the predicted point heat map, n represents the total number of samples, and p represents the index number.

[0029] (3) Perform triangle pruning on the wall point heat map to obtain the initial map; Figure 3 As shown, the details are as follows: For the triangle set T generated by the wall point set V, delete the longest side of the triangle in T to obtain the edge set E, and get the graph G. The formula is as follows: The formula is as follows: ; ; ; in, is the longest edge in t, delete the points with degree less than 2 in G, and the resulting graph is used as the initial graph; is a side of triangle t.

[0030] (4) Solve the optimal subgraph based on the initial graph to obtain the final reconstructed structure; Figure 4As shown in the figure, the specific process is as follows: solve the optimal subgraph based on the structural weight, and use the greedy algorithm to optimize the graph; where the structural weight includes the confidence term and the length term. The specific process is as follows: Confidence term ( ): This weight reflects the probability that the edge is a real wall, which is calculated jointly by the room (R) and wall (B) masks predicted by the primitive estimation network. The edge corresponds to the thermal value in the room heat map The higher it is, the lower the probability that the edge is a wall, and the corresponding heat value in the wall mask is The higher it is, the higher the probability that it is a wall. Used for measurement The overall probability of authenticity, the confidence term The calculation formula is as follows, where and are the two endpoints of e: ; ; ; Length term ( ): used to refine the confidence of short edges in the structure weight. Since there is a slight misalignment between the wall points and the edge heatmap, short edges are more sensitive to this misalignment, resulting in lower confidence and may be discarded. Therefore, use To refine Specifically, when processing edges with lower confidence, a threshold is set ,for An edge less than zero has a length less than , then Set to 1 to ensure that the final edge weight is greater than 0. For edges with higher confidence, no adjustment is made, and the result is The reason is that edges with higher confidence values ​​do not need to be adjusted. The formula is as follows: ; The overall formula for the structural weight is as follows: ; In order to ensure the integrity of the wall edges, the degree of each node in the graph should be greater than or equal to 2, that is, there are no cut points and cut edges in the optimal graph. Therefore, for edges with node degrees less than 2, it is set to -100.

[0031] Optimal subgraph solution: Based on the proposed structural weights, a greedy algorithm is used to optimize the graph, which improves the accuracy and efficiency of structural reconstruction. Based on the definition of structural weights, the goal of this step is to find a subgraph with the largest sum of weights, and the greedy strategy can ensure that the maximum value is reached at each step. It further ensures that the sum of all structural weights in the final subgraph is the largest, indicating a local optimal state. It is worth noting that all negative edges are traversed in turn, and all edges that can be deleted are deleted step by step according to the optimization strategy. As for the negative edges retained in the final subgraph, any attempt to delete them will inevitably lead to a decrease in the sum of structural weights. Although it cannot be guaranteed that the obtained subgraph is globally optimal, the use of a local optimal strategy can significantly improve the computational efficiency, and the use of structural weights ensures the reliability of the algorithm. In addition, in order to improve the computational efficiency of the greedy algorithm, it is performed on an increasing sequence queue Q, which only contains edges with structural weights less than 0. On the queue Q, the edges in the queue are traversed according to the following steps: (a) Calculate the sum of the structural weights of all edges in the current graph, denoted as W.

[0032] (b) After removing the edge with the smallest structural weight in the queue, calculate the sum of the structural weights of all edges in the current graph, denoted as .

[0033] (c) Compare W and If the weight of the structure increases (W> ), remove the edge with the smallest weight. Otherwise, keep the edge. Pop the first element from the queue.

[0034] (d) Repeat steps (a)-(c) until the queue is empty.

[0035] In order to make the final reconstruction result more compact, a set of connected edges consisting of degree 2 nodes are selected for Douglas-Peucker curve simplification, and then the degree 2 nodes that form an angle greater than 160 degrees are deleted to obtain the final result.

[0036] (5) Evaluate the reconstruction results. The details are as follows: Use the new indicators and Evaluate the reconstruction results; details are as follows: That is, the structural consistency index, the formula is as follows: ; in, Indicates the number of rooms that overlap with other rooms. Indicates the number of hanging points and cutting points in the structure, and Represent the number of correctly reconstructed rooms and corner points, is the scaling factor; MAnE That is, a metric that measures the overall deformation of the structure; the formula is as follows: ; in, represents the i-th angle deviation of all angle deviations within 5 degrees, is the number of false negative angles, is the number of correctly reconstructed angles; for angles that are not correctly reconstructed, an angle deviation is assigned ; That is, the compactness index, which measures the compactness of the reconstructed structure, and the formula is as follows: ; in, represents the number of rooms that were correctly reconstructed, Indicates the total number of corner points in the reconstruction result.

Claims

1. A method for reconstructing indoor plan views from three-dimensional scanning based on a graph, characterized in that: The following steps are involved: (1) Obtain open source datasets; (2) Construct a primitive estimation network SeqPNet to obtain a wall point heat map; the primitive estimation network SeqPNet includes: a wall heat map regression layer, a room heat map regression layer, and a point heat map regression layer; generate three types of heat map masks of wall B, room R, and point P in sequence to obtain a wall point heat map; (3) Triangulate and prune the wall point heat map to obtain the initial map; (4) Solve the optimal subgraph based on the initial graph to obtain the final reconstructed structure; (5) Evaluate the reconstruction results.

2. The method for reconstructing indoor plan views based on three-dimensional scanning according to claim 1, characterized in that: In step (2), the primitive estimation network SeqPNet is constructed to obtain the initial image as follows: taking the 2D density / normal image as input, three types of heat maps are sequentially generated through the wall heat map regression layer, the room heat map regression layer and the point heat map regression layer: wall B, room R and point P masks; each mask guides the generation of subsequent masks.

3. The method for reconstructing indoor plan views based on three-dimensional scanning according to claim 2, characterized in that: In step (2), the wall heat map regression layer is as follows: For a given 2D height / density map F Use four-layer CNN to extract features , where C is the number of channels; After the hourglass module HG, ConvBlocks conversion block and 2 conv3x3 layers, the intermediate features are obtained ; At the same time After 2 conv3x3 layers, the output features are obtained ;Will , and After super-perceptual cross attention HPCA, feature fusion is performed to obtain That is, the wall mask B is generated.

4. The method for reconstructing indoor plan views based on three-dimensional scanning of a graph according to claim 3, characterized in that: The feature fusion of super-perceptual cross-attention HPCA is as follows: HPCA includes a convolution branch for extracting local information and a sparse attention branch for extracting global information; among them, in the HPCA sparse attention weighted branch, first , , The sequences Q, K and V are converted into integers of size 1 through the multilayer perceptron MLP. The value q, key k and query v are selected for k and v. The output of HPCA is used as the input of the room heat map regression layer. R represents a matrix, N represents a row, C represents a column, and m represents a sequence index number.

5. The method for reconstructing indoor plan views based on three-dimensional scanning of a graph according to claim 2, characterized in that: In step (2), the room heat map regression layer is as follows: First, After the hourglass module HG, ConvBlocks conversion block and 2 conv3x3 layers, the intermediate features are obtained ; At the same time After 2 conv3x3 layers, the output features are obtained ;Will , and After super-perceptual cross attention HPCA, feature fusion is performed to obtain That is, generate the room mask R; As input to the point heatmap regression layer.

6. The method for reconstructing indoor plan views based on three-dimensional scanning according to claim 2, characterized in that: In step (2), the point heat map regression layer is as follows: After the hourglass module HG, ConvBlocks conversion block, 2 conv3x3 layers and upsampling processing, the wall point heat map is obtained.

7. The method for reconstructing indoor plan views based on three-dimensional scanning of a graph according to claim 2, characterized in that: The primitive estimation network SeqPNet loss consists of three parts: wall heat map loss , room heat map loss and point heatmap loss , the formula is as follows: ； in, 、 、 represents weight; L represents the overall loss of the network; ； ； ； Among them, y is the true value of the wall heat map, is the predicted wall heat map, , , , is a hyperparameter; ； in, is the true value of the room heat map, To predict the room heat map, n represents the total number of samples and i represents the index number; ； in, is the true value of the point heat map, is the predicted point heat map, n represents the total number of samples, and p represents the index number.

8. The method for reconstructing indoor plan views based on three-dimensional scanning of a graph according to claim 1, characterized in that: Step (3) is as follows: For the triangle set T generated by the wall point set V, delete the longest side of the triangle in T to obtain the edge set E, and obtain the graph G, the formula is as follows: ; ; ; in, is the longest edge in t, delete the points with degree less than 2 in G, and the resulting graph is used as the initial graph; is a side of triangle t.

9. The method for reconstructing indoor plan views based on three-dimensional scanning of a graph according to claim 1, characterized in that: Step (4) is as follows: solve the optimal subgraph based on the structural weight and optimize the graph using a greedy algorithm; wherein the structural weight includes a confidence term and a length term.

10. The method for reconstructing indoor plan view based on three-dimensional scanning of a graph according to claim 1, characterized in that: Step (5) is as follows: Using the new indicator and Evaluate the reconstruction results; details are as follows: That is, the structural consistency index, the formula is as follows: ； in, Indicates the number of rooms that overlap with other rooms. Indicates the number of hanging points and cutting points in the structure, and Represent the number of correctly reconstructed rooms and corner points, is the scaling factor; MAnE That is, a metric that measures the overall deformation of the structure; the formula is as follows: ； in, represents the i-th angle deviation of all angle deviations within 5 degrees, is the number of false negative angles, is the number of correctly reconstructed angles; for angles that are not correctly reconstructed, an angle deviation is assigned ; That is, the compactness index, which measures the compactness of the reconstructed structure, and the formula is as follows: ； in, represents the number of rooms that were correctly reconstructed, Indicates the total number of corner points in the reconstruction result.

Images