A method and system for deep visual perception of architectural space resolution

Abstract

The application belongs to the field of indoor space understanding and reconstruction, and discloses a deep visual perception method and system for building space analysis, which comprises the following steps: point cloud completion preprocessing and transfer learning pretraining; construction of a point cloud understanding and segmentation network model enhanced by a double-level attention mechanism; stage-by-stage training and index evaluation of the space analysis segmentation network model; visual interpretability analysis and parameterized modeling of the network model. The application can provide reliable space semantic analysis based on point cloud data scanned by a laser radar. The effect of the network model is verified through index evaluation and visual interpretability analysis. Further, a parameterized space model is established in a BIM software based on the analyzed space point cloud data, and practical effects are obtained. The developed method helps to propose a new reliable computer vision space analysis framework and an automatic process that helps to automatically analyze and generate a parameterized space model, so as to promote the process of engineering management automation and intelligentization.

Description

Technical Field

[0001] This invention belongs to the field of indoor space understanding and reconstruction, and in particular relates to a depth visual perception method and system for architectural space analysis. Background Technology

[0002] Indoor scene understanding is crucial for intelligent devices such as robots used in digital construction. Human spatial perception is intuitive and simple; people develop spatial awareness through vision and touch—that is, the location, boundaries, and composition of a space. However, for construction robots, spatial perception and understanding not only rely on raw data captured by sensors and cameras but also require processing and interpreting this data through advanced computer vision and machine learning techniques. With the development of information technology, the spatial perception and understanding capabilities of construction robots have rapidly improved, bringing innovation to the construction field and opening up new possibilities for the future development of robotics. Through more accurate spatial understanding, construction robots can more effectively assist humans in handling complex tasks, while improving work efficiency and safety in the construction industry.

[0003] Currently developed methods for indoor spatial understanding can be mainly divided into two categories: image-based methods and point cloud methods. One-stop methods that combine image understanding and modeling of semantic and spatial information are time-efficient, but accuracy needs improvement. Segmenting point clouds to understand semantic information and then performing spatial understanding based on that semantic information requires overcoming the challenges of point cloud acquisition, but it can yield more accurate spatial models and better understanding.

[0004] 3D reconstruction of a three-dimensional space using images is a complex process involving the analysis of a series of 2D images to reconstruct the 3D structure of an object or scene. This technique typically uses photogrammetry or structured light scanning to identify common feature points across multiple photographs for spatial scene reconstruction, particularly involving sophisticated computer vision algorithms for feature detection, matching, and understanding multi-view geometry. Depth information is crucial for spatial reconstruction and is usually obtained using binocular image sensors or laser sensors. While image-based methods offer rich texture and color details, image-based reconstruction faces numerous challenges in complex indoor environments where factors such as variable lighting, reflective surfaces, and occlusion can reduce reconstruction accuracy.

[0005] Segmenting and reconstructing space using point clouds from laser scanners or structured light sensors is crucial for creating detailed 3D spatial models. This process involves extracting semantic and spatial information and building the spatial model. Advanced algorithms, particularly semantic segmentation algorithms, can automatically extract this semantic and spatial data from point clouds. LiDAR technology is used to collect raw data and then segment and classify points related to surfaces or structures, including noise reduction, plane and edge detection, and geometry-based feature classification. The subsequent reconstruction stage connects these points to form a coherent spatial model, which is widely used in virtual walkthroughs, redevelopment planning, or Building Information Modeling (BIM).

[0006] In summary, indoor 3D reconstruction and spatial understanding are achieved through image and point cloud data, transforming real-world space into digital information that machines can understand and process. Segmentation modeling methods make this data more ordered and structured, which is beneficial for subsequent applications such as BIM and environmental perception in autonomous vehicles. With advancements in computing technology and machine learning algorithms, the efficiency and accuracy of these reconstruction techniques will continue to improve, making machine understanding of space closer to human intuitive perception.

[0007] Therefore, how to use intelligent algorithms to understand the point cloud information of completed building interiors and establish a parametric spatial model has become a crucial issue. This invention aims to explore an alternative approach, proposing a novel attention-enhanced deep learning method to achieve the understanding and reconstruction of completed building interior spaces. Summary of the Invention

[0008] To address the problems existing in the prior art, this invention provides a depth visual perception method and system for architectural space analysis.

[0009] This invention is implemented as follows: a depth visual perception method and system for architectural space analysis, the method comprising:

[0010] S1: Point cloud completion preprocessing and transfer learning pretraining;

[0011] S2: Construction of a point cloud understanding and segmentation network model enhanced by a two-level attention mechanism;

[0012] S3: Phased training and performance evaluation of spatial parsing segmentation network models;

[0013] S4: Visual interpretability analysis and parametric spatial modeling of network models.

[0014] Furthermore, S1 includes:

[0015] (1) Point cloud completion preprocessing; The point cloud scanning method based on the station is simple to operate, but the disadvantage is that it cannot restore objects that are far away from the station and are occluded. If we want to scan objects that cannot be scanned by a single station, we need to scan from another station, which requires a lot of station information and consumes more time and resources. Considering that the objects contained in the completed building site, such as beams, floor slabs, columns, ground, walls and other objects have a simple structure, we can use the symmetry of the components to complete the missing point cloud without losing key features. That is, we can complete the missing point cloud by copying part of the point cloud of the component itself to improve the learnability of the data.

[0016] (2) Transfer learning pre-training; appropriate transfer learning methods can effectively improve the performance and generalization ability of the network model; after removing some categories from the public indoor space point cloud dataset S3DIS, a sub-dataset is generated, and the network model is pre-trained on this dataset and then trained and validated on the completed building point cloud dataset collected on site.

[0017] Furthermore, S2 includes the construction of a two-layer attention-enhanced spatial parsing network model:

[0018] The input data first undergoes farthest point sampling (FPS) and KNN clustering. The FPS algorithm randomly selects a point from the input point cloud, then iterates to select the point furthest from the selected point, considering all features of the point cloud within limited computational resources. After sampling, the KNN algorithm clusters points near the sampled point to preserve local features. Two-dimensional convolution is then used to model the pattern of the feature data to obtain the ability to extract point cloud features. During feature extraction, the Efficient Channel Attention (ECA) mechanism helps to efficiently assign attention weights to convolutional networks in different channels, making the effective channels play a more significant role. The Self-Attention mechanism generates globally refined attention weights based on the input features, enabling the network to better understand the features given the input.

[0019] The self-attention mechanism (SA) processes input features to generate refined attention features. The input feature map F is fed into three independent 1×1 convolutional layers to reshape it into three one-dimensional vectors Fq, Fk, and Fv. Fq and Fk are then multiplied to obtain an n×n covariance matrix, i.e., the attention map. The attention map is then multiplied by Fv to obtain the output feature map, and the residuals of the input F are concatenated to generate a refined self-attention feature Fr. Here, the self-attention feature Fr is weighted with F, i.e., out = α * Fr + F. Here, α is a learnable parameter, automatically adjusted by the optimizer to control the intensity of the attention mechanism. After the refined features are generated by the self-attention mechanism, deconvolution layer by layer upsampling increases the size of the feature map while reducing the number of channels, gradually restoring it to the size of the input feature map to enable segmentation.

[0020] The shallow and deep features are fused, and the fused features are segmented by a one-dimensional convolutional layer. Finally, the input point cloud group is segmented into 6 categories: ceiling, floor, beam, column, fence and wall. These six categories represent the semantic information of the point cloud, that is, the actual meaning of spatial resolution, and are also the basic elements that constitute architectural space.

[0021] Furthermore, S3 specifically includes:

[0022] (1) Evaluation of network model training performance; To evaluate the training performance, a bidirectional symmetric loss function is proposed, which is a combination of the forward module Focal loss and the reverse module Revert CE loss with certain weights. The forward module uses the network output value as the predicted value and the label as the true value, and assigns different penalty weights to point clouds of different segmentation difficulty levels. The reverse module uses the network output value as the true value and the label as the predicted value for calculation. This loss function can improve the robustness of the network model to noisy data. The standard cross-entropy loss function is more sensitive to noisy data. Due to the noise or incorrect labels in the dataset, the standard cross-entropy loss function will cause the network model to learn from these incorrect labels, resulting in a significant drop in performance. In the point cloud dataset, considering the subtle noise in the point clouds of mutually occluding objects, these points are key features for semantic segmentation.

[0023] CE and RCE losses can be calculated according to the following formulas (1) and (2), SCE loss L s It can be calculated according to formula (3):

[0024]

[0025]

[0026] L s=α·CE + β·RCE (3)

[0027] Where M is the number of categories, y o，c To observe the true label of o, p o，c α is the predicted probability that observation o is classified as c, and β are the weights of CE and RCE respectively;

[0028] Furthermore, reducing the focus on easily classified samples is beneficial to the classification performance of male samples; the focus loss introduces a modulation factor on the basis of the CE loss to reduce the weight of easily classified samples; the focus loss is defined by formula (4):

[0029]

[0030] Where (1-p) o，c ) γ γ is the modulation factor, and γ is the focus parameter. Focus loss reduces the weight of easily classified samples by using this modulation factor.

[0031] As a combination, Equation (5) gives a newly defined loss;

[0032] Loss=θ·L f +(1-θ)·L r (5)

[0033] Where L f Indicates focal loss; L r This represents the RCE loss, where θ and 1-θ are the weights, specified as hyperparameters.

[0034] Furthermore, step (2) specifically includes:

[0035] (2) Multi-factor testing and evaluation of network models; In order to evaluate the effectiveness of the validation and testing phases, the mean intersection-union ratio (mIoU), as well as the P-value, F1 score, R-value, and AUC-ROC curve index are introduced to comprehensively evaluate the performance of the network model on the test set; The following four general evaluation indicators, precision, recall, F1 score, and IoU, can be expressed by formulas (6-9):

[0036]

[0037]

[0038]

[0039]

[0040] This refers to the class index; TP, FP, and FN represent true positives, false positives, and false negatives, respectively; N is the number of points; all classes use overall precision (OA), weighted average (WA), and macro average (MA) precision, recall, F1 score, and mean IoU (MIoU); their calculation results are given by the formula:

[0041]

[0042]

[0043]

[0044]

[0045] The ROC curve reflects the continuous relationship between sensitivity and specificity. The horizontal axis of the ROC curve represents the False Positive Rate (FPR), which is the proportion of samples that are actually negative but are incorrectly predicted as positive. The vertical axis represents the True Positive Rate (TPR), which is the proportion of samples that are actually positive but are correctly predicted as positive. The principle behind plotting the ROC curve is as follows: A set of thresholds is specified, and the predicted probabilities output by the network model are divided into binary classification predictions according to these thresholds. The FPR and TPR for each threshold are calculated, and points corresponding to each threshold are determined on a coordinate system and connected sequentially. Finally, the area under the curve (AUC) is calculated. The AUC value ranges from 0 to 1; the closer the AUC value is to 1, the better the network model's performance.

[0046] Furthermore, S4 specifically includes:

[0047] (1) Visual interpretability analysis of the network model:

[0048] To gain a more intuitive understanding of the machine's state during feature learning and the basis for feature extraction, a visual interpretability analysis was performed on the proposed network. Grad-CAM is a popular interpretability analysis method, and its calculation formula is as follows:

[0049]

[0050]

[0051]

[0052] Here, Grad refers to the calculation of the category activation value Y for the target category c. c Relative to feature map A kThe gradient of each feature map channel is calculated, and then global average pooling is performed on the gradient of each channel to obtain the importance weight of that channel. The final calculation The heatmap also needs to be upsampled to the size of the input tensor to highlight the areas most important for the network model to make specific class decisions;

[0053] (2) Parametric reconstruction of space:

[0054] In Revit, the FARO As build for Revit plugin helps build a spatial model after importing a categorized point cloud. To build the model, first create a component template library and develop materials for various structural components (such as beams, columns, slabs, etc.) appearing in the scene. Next, randomly select categorized points from the point cloud for fitting. Finally, combine each component according to boundary conditions to form a complete spatial model.

[0055] Building a spatial model on segmented point clouds, rather than on the original point cloud data, can yield faster and clearer boundaries. This avoids the influence of other types of components when fitting a certain type of component based on the point cloud. In the interior scenes of completed buildings, the combination of various components often follows certain patterns, which reduces the difficulty of building a spatial model.

[0056] Another object of the present invention is to provide a depth visual perception system for architectural space analysis that implements the aforementioned depth visual perception method for architectural space analysis, the system comprising:

[0057] The preprocessing training module is used for point cloud completion preprocessing and transfer learning pretraining.

[0058] The network model building module, connected to the preprocessing training module, is used to build a point cloud understanding and segmentation network model enhanced by a two-level attention mechanism.

[0059] The training and evaluation module, connected to the network model building module, is used for deep learning evaluation and interpretability and parameterized space model reconstruction.

[0060] The modeling module, connected to the training and evaluation modules, is used for visual interpretability analysis and parametric spatial modeling of network models.

[0061] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps of the depth visual perception method for architectural space analysis.

[0062] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0063] First, this invention uses a dataset of point cloud data of completed buildings collected on-site to verify the effectiveness of the proposed method. The point cloud data of completed buildings collected by a laser scanner is parsed and segmented, and parametric modeling is performed on the segmentation results. The results show that: (1) In cross-validation, the average mIoU achieved by this method is 0.970, the variance is 4.05E-05, and the maximum value is 0.983, which shows excellent overall performance. Moreover, the network model of this method achieves an mIoU of 0.9713 on a representative dataset, which is better than the 0.9445 achieved by the current advanced point cloud semantic segmentation network PointNet++; (2) Visual interpretability analysis shows that the feature extraction layer correctly distinguishes most categories, showing good overall confidence; (4) The time for building a spatial model after parsing is reduced by about 37% compared with manual modeling without parsing, and the occurrence of errors is effectively reduced, indicating that segmentation and modeling play different but complementary roles in the process of spatial understanding. This invention provides a new deep machine vision method for accurate spatial parsing and rapid reconstruction.

[0064] Secondly, this invention develops a novel machine vision method for architectural spatial analysis and reconstruction. It proposes a point cloud analysis network enhanced with a two-level attention mechanism, capable of providing reliable spatial semantic analysis based on point cloud data scanned by LiDAR. The effectiveness of the network model is verified through index evaluation and visual interpretability analysis. Furthermore, a parametric spatial model is established in BIM software using the analyzed spatial point cloud data, achieving practical results. The developed method contributes to proposing a new and reliable computer vision spatial analysis framework and an automated process for automatic analysis and parametric spatial model generation, thereby promoting the automation and intelligentization of engineering management.

[0065] Third, as supplementary evidence of the inventive step of the claims of this invention, it is also reflected in the following important aspects:

[0066] (1) The expected benefits and commercial value of this transformation are significant. The proposed technical solution can effectively improve the understanding and modeling accuracy of indoor scenes. For unmanned autonomous systems with high market demand, such as construction robots and autonomous driving construction machinery, this invention can effectively meet the intelligent needs of these devices, thus having significant economic benefits and market application prospects.

[0067] (2) This invention solves a long-standing technical problem that the industry urgently needs to solve but has not been able to overcome, namely, how to accurately and efficiently achieve object segmentation and scene understanding from point cloud data in complex indoor environments.

[0068] (3) Overcoming technical bias, this invention not only combines a two-level attention mechanism and transfer learning methods, but also incorporates visual interpretability analysis technology. This innovative technical solution not only enhances the model's segmentation efficiency and accuracy, but also improves the model's interpretability, enabling it to better understand and analyze indoor scenes. Through this method, this invention effectively solves multiple challenges in point cloud data processing under complex indoor environments, including problems such as overlapping, uneven distribution, and imbalance of point cloud data.

[0069] Fourth, the depth visual perception method for architectural space analysis provided by this invention solves several technical problems of existing technologies and brings about significant technological progress. The following is a detailed analysis of these aspects:

[0070] First, this method addresses the issue of incomplete point cloud data. In architectural spatial analysis, point cloud data is often incomplete due to various reasons (such as sensor limitations and occlusion), which severely affects the accuracy of subsequent spatial analysis and visual perception. By utilizing the symmetry of components for point cloud completion preprocessing, these gaps can be effectively filled, improving data quality and providing a more accurate foundation for subsequent depth visual perception.

[0071] Secondly, this method utilizes transfer learning techniques to improve the performance and generalization ability of the network model. Transfer learning is an effective machine learning method that can transfer knowledge learned from one task to another related task, thereby accelerating the learning process for new tasks and improving model performance. In this method, by pre-training the network model on a public indoor spatial point cloud dataset and then using field-collected data for training and validation, existing knowledge can be effectively utilized to improve the model's performance on new data.

[0072] Furthermore, this method enhances the understanding and segmentation capabilities of point cloud data by constructing a point cloud understanding and segmentation network model with a two-level attention mechanism. Attention mechanisms help the model focus more on important information when processing data, thereby improving model performance. The introduction of a two-level attention mechanism allows the model to focus on different information at different levels, further improving the accuracy of point cloud understanding and segmentation.

[0073] Finally, this method also emphasizes the visual interpretability analysis of the network model and parametric spatial modeling. This not only improves the interpretability of the model, allowing people to better understand the model's decision-making process, but also presents the analytical results in a more intuitive and understandable way through parametric spatial modeling, providing strong support for the design and optimization of architectural space.

[0074] In summary, this deep visual perception method for architectural space analysis has achieved significant technological advancements by addressing the problem of incomplete point cloud data, improving model performance through transfer learning, constructing a network model enhanced by a two-level attention mechanism, and emphasizing the visual interpretability and parametric spatial modeling of the model. It provides a more accurate and efficient method for the analysis and design of architectural spaces. Attached Figure Description

[0075] Figure 1 This is an overall framework diagram of spatial analysis and reconstruction provided in the embodiments of the present invention;

[0076] Figure 2 This is a structural diagram of the two-level attention-enhanced spatial parsing network provided in an embodiment of the present invention;

[0077] Figure 3 This is a structural diagram of the efficiency channel attention module provided in an embodiment of the present invention;

[0078] Figure 4 This is the structure of the self-attention module provided in the embodiments of the present invention;

[0079] Figure 5 This is the main flowchart of parametric modeling provided in the embodiments of the present invention;

[0080] Figure 6 These are the index curves (a) for the pre-training stage and (b) for the single-region transfer learning stage provided in this embodiment of the invention, where the loss value is a relative value.

[0081] Figure 7 This refers to the AUC-ROC curve provided in the embodiments of the present invention;

[0082] Figure 8 The visual interpretability analysis provided in this embodiment of the invention includes ceiling (a), floor (b), beam (c), column (d), railing (e), and wall (f);

[0083] Figure 9 This is the parametric modeling of region 11 provided in this embodiment of the invention;

[0084] Figure 10 These are errors that are prone to occur in the traditional modeling methods provided in the embodiments of the present invention;

[0085] Figure 11 This is a structural diagram of the depth visual perception system for architectural space analysis provided in an embodiment of the present invention. Detailed Implementation

[0086] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0087] Reference Figure 1 As shown, the following is a detailed explanation of the working principle of the architectural space resolution depth visual perception method mentioned:

[0088] S1: Point cloud completion preprocessing and transfer learning pretraining

[0089] #(1) Point cloud completion preprocessing

[0090] Working principle: In the real world, due to occlusion, scanning angle, and other factors, the collected architectural space point cloud data is incomplete. The point cloud completion preprocessing stage uses the symmetry properties of components to identify missing parts and completes the entire point cloud by copying existing symmetrical parts. This results in a complete point cloud model, providing a more accurate data foundation for subsequent processing.

[0091] #(2) Transfer learning pre-training

[0092] How it works: Transfer learning is a common strategy in deep learning that allows a model trained on one task to transfer the learned features and knowledge to another related task. In this stage, pre-training is first performed on the publicly available indoor spatial point cloud dataset S3DIS. The S3DIS dataset is a large-scale point cloud dataset containing multiple categories; by pre-training on this dataset, the model can learn the basic features and patterns of the space. Then, further training and validation are performed on actual field-collected point cloud datasets to adapt to specific application environments and requirements.

[0093] S2: Construction of a Point Cloud Understanding and Segmentation Network Model Enhanced by a Two-Level Attention Mechanism

[0094] How it works: This step constructs a network model that incorporates a two-level attention mechanism to improve the accuracy and efficiency of point cloud understanding. The attention mechanism allows the network model to automatically focus on the most important parts when processing data, which is especially important in point cloud processing because point cloud data is typically large and contains a lot of redundant information. The two-level attention mechanism consists of local attention and global attention. Local attention focuses on specific regions within the point cloud, while global attention focuses on the entire point cloud space.

[0095] S3: Phased Training and Evaluation of Spatial Analytic Segmentation Network Models

[0096] How it works: In this step, the network model is trained in stages, gradually improving its performance on spatial parsing tasks. Typically, this involves multiple training epochs, each optimizing the model's parameters and evaluating its performance using a range of metrics (such as precision, recall, and IoU). Through this iterative process, the model's segmentation accuracy can be continuously improved.

[0097] S4: Visual interpretability analysis and parametric space modeling of network models

[0098] How it works: Finally, a visual interpretability analysis of the model is performed to understand its decision-making process. This is typically achieved through techniques such as feature mapping and activation maps. Visual interpretability helps researchers and users understand how the model works and how to improve it. Simultaneously, parametric spatial modeling is performed, which means converting the model's output into a parametric form that can be used for practical architectural design and analysis, such as a CAD model or BIM (Building Information Modeling).

[0099] This method improves the ability to automatically analyze building spaces by combining advanced deep learning and point cloud processing technologies, thereby supporting applications such as smart city planning, automated building monitoring, and augmented reality.

[0100] This embodiment provides a novel method for architectural space analysis and parametric spatial model establishment, including the following steps:

[0101] Step 1: Point cloud data completion preprocessing and network model pretraining. Specifically, this includes:

[0102] Step 1.1 Point Cloud Data Completion and Preprocessing: In this example, the Trimble X7 stationary laser point cloud scanner collects point cloud data from the 5th and 6th floors of a completed building in Wuhan, Hubei Province. This example uses the collected data as a case study. The building point cloud consists of two layers. After cleaning the original point cloud data and removing irrelevant points, the initial dataset contains approximately 16.53 million points. The point cloud is manually labeled into six categories: ceiling, floor, beams, columns, railings, and walls. To divide the dataset into training and testing sets, the 5th layer is relatively evenly divided into regions 1-8, and the 6th layer into regions 9-16. The number of points in each category for each layer is shown in Table 1.

[0103] Table 1. Number of points of each category in the point cloud of layer 5 and layer 6.

[0104]

[0105]

[0106] Based on a labeled point cloud dataset, point clouds of occluded columns, beams, ceilings, and walls are completed. For column point clouds, the component point cloud is divided into 20 parts along the direction to be completed. The program automatically finds the descending edge of the point cloud count and starts completing the subsequent parts from the part with the highest count, using the scanned complete column point cloud to complete the point cloud of the part occluded by the fence to the boundary with the floor, and distinguishes the blurred parts of the fence and column. All types of point clouds are processed according to boundaries and symmetry. Finally, all point clouds are downsampled at a minimum distance of 0.01m to obtain the dataset used in the experiment.

[0107] Step 1.2 Transfer Learning Data Preparation. The publicly available large-scale indoor space point cloud dataset S3DIS has 13 categories. To better adapt to the data pattern in this example, irrelevant categories of points in the S3DIS dataset were removed, retaining only five categories: ceiling, floor, beam, column, and wall. Compared to the point cloud dataset collected by LiDAR, the category of guardrail is missing. A subset of this S3DIS dataset, downsampled at a minimum distance of 0.01m, was used as the data source for the pre-training stage.

[0108] Step 2: Construction of a point cloud understanding and segmentation network model enhanced by a two-layer attention mechanism, specifically including:

[0109] Step 2.1: Two-level attention-enhanced spatial parsing network;

[0110] The proposed two-level attention-enhanced spatial parsing network for parsing space and generating semantic labels consists of an encoder, a decoder, and a label predictor, as shown in the following structure. Figure 2 As shown, the input data is passed through an encoder consisting of four Efficient Channel Attention (ECA) enhanced feature extraction modules, and then through a decoder consisting of one Self Attention (SA) feature selection module and three upsampling and propagation modules. The encoder and decoder are connected to each other at their respective levels. Finally, the output of the decoder is used by a segmentation category predictor to determine the semantic category.

[0111] In the encoder, the feature extraction module first performs farthest point sampling (FPS) on the input data according to a certain number of sampling points. It randomly selects a point from the input point cloud, then selects the point furthest from the selected point, iterating continuously to consider all features of the point cloud within limited computational resources. Subsequently, each sampling point is clustered using KNN with a certain number of points to enrich the local features around the key sampling points. The sampled and clustered point set then undergoes feature extraction through a set of two-dimensional convolutional layers. During this process, the size of the feature set continuously decreases while the number of channels continuously increases. To enhance the weight of key features and reduce the weight of irrelevant features, an Efficient Channel Attention (ECA) mechanism is introduced in this module, with the structure shown below. Figure 3 As shown, it helps to efficiently assign attention weights to convolutional networks with different channels, making the effective channels play a more significant role.

[0112] In the decoder, the data first passes through a self-attention mechanism module, the structure of which is as follows: Figure 4 As shown, this method enables the input features to be processed to generate refined attention features. The input feature map F is fed into three independent convolutional layers with a kernel size of 1×1 to reshape it into three one-dimensional vectors Fq, Fk, and Fv. Then, Fq and Fk are multiplied to obtain a covariance matrix of size n×n, i.e., the attention map. Next, the attention map is multiplied by Fv to obtain the output feature map, and the residuals of the input F are concatenated to generate a refined self-attention feature Fr. Here, the self-attention feature Fr is weighted with F, i.e., out = α * Fr + F. Here, α is a learnable parameter, and the optimizer automatically adjusts the strength of the attention mechanism. Afterward, the data is passed through three sets of upsampling modules to restore the features to their original size.

[0113] Both the encoder and decoder have a four-layer structure. To achieve better network performance, a hierarchical skip connection technique is introduced, where the top-down encoder layer is interconnected with the corresponding bottom-up decoder layer. This means that the output of each encoder layer not only goes to the next encoder layer but is also merged with the underlying data of its corresponding decoder layer before entering that decoder layer. This feature fusion process helps generate concrete features from abstract features.

[0114] Finally, shallow and deep features are fused, and the fused features are segmented by a one-dimensional convolutional layer. The input point cloud is then divided into six categories: ceiling, floor, beam, column, fence, and wall. These six categories represent the semantic information of the point cloud, i.e., the practical meaning of spatial resolution, and are also the basic elements constituting architectural space.

[0115] Step 2.2: Consider the loss function for imbalanced classes in a bidirectional symmetric manner;

[0116] This invention combines two loss functions: Symmetric Cross Entropy (SCE) loss and focus loss. The standard cross-entropy loss function is sensitive to noisy data. Due to noise or incorrect labels in the dataset, standard cross-entropy can cause the network model to learn from these incorrect labels, resulting in a significant performance drop. In point cloud datasets, considering the subtle noise in point clouds with occluding objects, these points are crucial features for semantic segmentation. Therefore, Symmetric Cross Entropy (SCE) loss is introduced. The SCE loss is the sum of the standard cross-entropy loss and the inverse cross-entropy loss. The inverse cross-entropy loss is calculated by swapping the predicted value of the standard cross-entropy loss with the true value. It encourages the predicted probability distribution to be close to a uniform distribution in the face of noise in the true labels, making the network model more robust to label noise. Combining the forward and inverse cross-entropy loss functions reduces the dependence on noisy labels, helping the network model generalize better to unseen data, especially when this new data contains noise patterns different from the training data. Therefore, it is very useful for training on large-scale datasets. The focus loss is used to distinguish indistinguishable categories and reduce the impact of imbalanced data within each category. These two loss functions are combined in the consistency cost weight γ to train a better network model.

[0117] CE and RCE losses can be calculated according to the following formulas (1) and (2), SCE loss L s It can be calculated according to formula (3):

[0118]

[0119]

[0120] L s =α·CE + β·RCE (3)

[0121] Where M is the number of categories, y o，c To observe the true label of o, p o，c α is the predicted probability that observation o is classified as c, and β are the weights of CE and RCE respectively;

[0122] Furthermore, reducing the focus on easily classified samples is beneficial to the classification performance of difficult-to-classify samples; the focus loss introduces a modulation factor on the basis of the CE loss to reduce the weight of easily classified samples; the focus loss is defined by formula (4):

[0123]

[0124] Where (1-p) o，c ) γγ is the modulation factor, and γ is the focus parameter. Focus loss reduces the weight of easily classified samples by using this modulation factor.

[0125] As a combination, Equation (5) gives a newly defined loss;

[0126] Loss=θ·L f +(1-θ)·L r (5)

[0127] Where L f Indicates focal loss; L r This represents the RCE loss, where θ and 1-θ are the weights, specified as hyperparameters.

[0128] Furthermore, step (3) specifically includes:

[0129] The following four general evaluation metrics are introduced: precision, recall, F1 score, IoU, and ROC, which can be expressed by formulas (6-9):

[0130]

[0131]

[0132]

[0133]

[0134] This refers to the class index; TP, FP, and FN represent true positives, false positives, and false negatives, respectively; N is the number of points; all classes use overall precision (OA), weighted average (WA), and macro average (MA) precision, recall, F1 score, and mean IoU (MIoU); their calculation results are given by the formula:

[0135]

[0136]

[0137]

[0138]

[0139] In addition, the ROC (Receiver Operating Characteristic) curve and AUC (Area Under the Curve) are also important metrics for evaluating classification network models. The ROC curve reflects the continuous relationship between sensitivity and specificity. The horizontal axis of the ROC curve represents the False Positive Rate (FPR), which is the proportion of samples that are actually negative but are incorrectly predicted as positive. The vertical axis represents the True Positive Rate (TPR), which is the proportion of samples that are actually positive but are correctly predicted as positive. The principle behind plotting the ROC curve is to specify a set of thresholds, divide the predicted probabilities output by the network model into binary classification predictions according to the thresholds, calculate the FPR and TPR for each threshold division, then determine the points corresponding to each threshold on the coordinate system and connect them sequentially. Finally, calculate the area under the curve with respect to the horizontal axis, i.e., the AUC value. The AUC value ranges from 0 to 1; the closer the AUC value is to 1, the better the network model's performance. Even if the network model predicts labels instead of probabilities, an ROC curve can still be plotted and the AUC value calculated. In summary, ROC curves provide an intuitive and comprehensive way to evaluate and compare classifier performance, visualize network models, and determine the optimal classification threshold.

[0140] Step 3: Phased training and performance evaluation of the spatial parsing segmentation network model;

[0141] The computational experiments used the PyTorch framework developed by Facebook to build the network model in Python. The model was implemented on a hardware environment with an i7 12700 CPU, an RTX 2080Ti GPU with 22GB of RAM, and 32GB of RAM, and on the Ubuntu 22.04 operating system. The optimal hyperparameters for the training phase were determined through grid search, as shown in Table 2.

[0142] Table 2. Hyperparameter Table

[0143]

[0144]

[0145] The values ​​of these fundamental hyperparameters remain constant throughout all subsequent training phases. The training process is divided into three phases: pre-training, transfer learning cross-validation, and single-region transfer learning.

[0146] Pre-training phase. Pre-training is performed on the prepared S3DIS dataset. The weights saved from this training phase will serve as the initial weights for all subsequent training phases. During training on the S3DIS training and test sets, the relative loss, training set accuracy, and test set mIoU of the proposed network are as follows: Figure 6 As shown in (a). At the end of training, the weights of the network model that performed best in this stage are selected as the initial weights for subsequent training stages.

[0147] The cross-validation phase of transfer learning. In the cross-validation phase, to evaluate the overall performance of the network model, the proposed network was cross-validated 16 times on an indoor as-built building scene dataset. Each cross-validation used a single region as the test set and other regions as the training set, using controlled variables to eliminate interference from other factors. In this phase, each training iteration used pre-trained weights from the S3DIS dataset as initial weights. Finally, a representative region was selected whose performance metric (mIoU) should be closest to the average mIoU of the cross-validation. Data from this region can better reflect the average performance of the entire dataset, thus providing standardization and capability in subsequent experiments. The experimental results of this phase are shown in Table 3.

[0148] Table 3. Experimental results of cross-validation

[0149]

[0150]

[0151] The results show that the network model performs well on each test set, with an overall mean cross-section ratio (MCR) of 0.970, indicating high accuracy in identifying and segmenting point cloud data in different regions. The variance is 4.05244E-05, and the low variance suggests that the network model's performance is relatively stable across different test sets without significant fluctuations. Region 11 has an mIoU of 0.9713, which is closest to the average among all regions. Therefore, Region 11 will be used as the representative dataset for the completed building site collection in the subsequent experimental phase.

[0152] Single-region transfer learning stage. In this stage, the point cloud data of region 11 provided by the cross-validation results is used as the test set, and the point cloud data of 15 other scenes are used as the training set. The relative loss value, training set accuracy, and test set mIoU of the proposed network during this training stage are as follows: Figure 6As shown in (b). On this platform, the training time for each round in this stage is approximately 1 minute. Using the optimal weights from transfer learning as initial weights effectively reduces the training time for subsequent stages. Subsequently, this region is used as a test set for ablation experiments, interpretability analysis, and modeling understanding. The results calculated according to the evaluation index formula are shown in Table 4, and the AUC-ROC curve is shown in... Figure 7 As shown in Table 5, the mIoU comparison with other advanced methods is shown in Table 5.

[0153] Table 4. Test results for Region 11

[0154]

[0155] Table 5. Comparison with other advanced methods

[0156]

[0157]

[0158] The results show that the proposed method achieves an mIoU value of 0.9713 for region 11, which is higher than the mIoU value of 0.9445 achieved by PointNet++. This higher accuracy demonstrates that the machine has achieved effective semantic understanding of the space and possesses high practical value. In the visualization of the segmentation results, a small portion of the guardrail in contact with the ground is incorrectly predicted as ground. This is due to noise during the dataset calibration process, and some points are incorrectly labeled at the interface between the two types of point clouds, but this does not affect the overall recognition effect. The AUC-ROC curve image shows that the curve has a high envelope area with the x-axis, indicating that the proposed method performs well in the point cloud segmentation of standard floors of completed buildings and has great application potential. The evaluation index table shows that most point clouds were successfully segmented, indicating that the proposed method can effectively handle and learn point cloud data features. Test results of different networks under the same dataset distribution are presented. By comparing the results given in Table 5, it can be observed that the proposed network shows better performance compared to state-of-the-art point cloud networks such as dgcnn, pointnet, pointnet++, and ASPCNET.

[0159] Step 4: Visual interpretability analysis and parametric modeling of the network model; specifically including:

[0160] Step 4.1: Visual interpretability analysis of the network model

[0161] To gain a more intuitive understanding of the machine's state during feature learning and the basis for feature extraction, a visually interpretable analysis was performed on the proposed network. Grad-CAM (Gradient-weighted Class Activation Mapping) is a popular interpretability analysis method used to highlight the regions in a convolutional neural network that contribute most to a specific class decision. The basic idea of ​​Grad-CAM is to use the gradient of the target class with respect to the output of the last convolutional layer of the network; these gradients are considered an indicator of the importance of the class decision. The calculation formula for Grad-CAM is as follows:

[0162]

[0163]

[0164]

[0165] Here, Grad refers to the calculation of the category activation value Y for the target category c. c Relative to feature map A k The gradient of each feature map channel is calculated, and then global average pooling is performed on the gradient of each channel to obtain the importance weight of that channel. The final calculation The heatmap also needs to be upsampled to the size of the input tensor to highlight the areas most important for the network model to make specific class decisions.

[0166] The data from the last feature extraction layer of the proposed network is extracted, and a heatmap is generated according to the above formula, as shown below. Figure 8 As shown. Among these six categories, the ceiling, floor, and beam categories are relatively well-interpreted, such as... Figure 8 As shown in (a), (b), and (c), the orange-red areas indicate that the Grad-CAM heat value of the point cloud segmentation network is high in this category, indicating that the network model can accurately learn and classify based on these salient features. In the column, protection, and wall categories, as... Figure 8 As shown in (d), (e), and (f), although the highlighted red areas are not widely distributed, the final segmentation results are acceptable. This is because these categories are more difficult to subdivide. Point clouds require more attention mechanisms to aid in differentiation. For the more difficult-to-segment categories of beams and columns, the point cloud segmentation network demonstrates interpretable features of interest in the image. Figure 8 In (d), the vertically continuous point clouds in this column are highlighted in red. This means the network focuses on the vertically continuous features of these point clouds and uses this to determine the category of these points. However, it is possible to... Figure 8As observed in (c), (d), and (e), point clouds that do not belong to this category are also areas of greater interest to neural networks. This is because features are extracted in intermediate layers, and operations are performed based on these features in subsequent layers of the neural network.

[0167] Step 4.2, Parametric Reconstruction of Space

[0168] In Revit, the FARO As build for Revit plugin helps complete the process of building a spatial model after importing categorized point clouds, such as... Figure 5 As shown. To build a spatial model, first create a component template library to develop materials for various structural components (such as beams, columns, slabs, etc.) appearing in the scene. Next, randomly select points categorized into classes on the point cloud for fitting. Finally, combine each component according to boundary conditions to form a complete spatial model.

[0169] Building spatial models on segmented point clouds, rather than raw point cloud data, yields faster and clearer boundaries. This avoids the influence of other types of components when fitting a particular type of component based on the point cloud. In the interior scenes of completed buildings, the combination of various components often follows certain patterns, which reduces the difficulty of building spatial models.

[0170] The modeling process for the point cloud segmentation results in Region 11 took approximately 5 minutes to generate a detailed parametric spatial model, while traditional methods took approximately 8 minutes to model the region. Automatic semantic segmentation of the point cloud endows it with labeled information, dividing and clustering it into different spatial elements, which is crucial for spatial understanding. Errors can occur in the manual reconstruction of the ring beam, such as... Figure 10 As shown, the beams were incorrectly modeled at some ceiling points. Based on the segmentation results of Region 11, the point clouds were fitted to the parametric spatial model according to the component categories using the FARO As Built for Autodesk Revit plugin. Finally, the point clouds were merged according to their positions to obtain the parametric spatial model of Region 11. The visualization of this model is shown below. Figure 9 As shown in Table 6, the dimensions of each structural component are as follows.

[0171] Table 6. Component dimensions and modeling time in parametric modeling

[0172]

[0173] In the modeling results for a single region, the dimensions of the floor slab protrusion are 2000*70mm. Since only data from this region is considered as the modeling source, the thickness information of the walls and floor slabs cannot be obtained. Therefore, the thickness of the walls and cladding is uniformly set to 200mm, and the thickness of the floor slab is set to 300mm. In the spatial understanding process, segmentation provides abstract semantic understanding, while modeling provides concrete physical location understanding. Spatial understanding is achieved through classification and then clustering. When these two techniques are combined, they complement each other, providing a comprehensive and integrated spatial understanding. This integrated understanding enables computer systems to better support complex design and planning tasks, improve decision-making quality, and promote automation and intelligence. In summary, segmentation and modeling play different but complementary roles in the spatial understanding process. Segmentation enables computers to identify and interpret various elements and their functions in space by providing semantic understanding. On the other hand, modeling complements this understanding by providing a concrete and accurate representation of the physical space, including size, shape, and relative position.

[0174] like Figure 11 As shown, this embodiment of the invention provides a depth visual perception system for architectural space analysis that implements the aforementioned depth visual perception method for architectural space analysis. The system includes:

[0175] The preprocessing training module is used for point cloud completion preprocessing and transfer learning pretraining.

[0176] The network model building module, connected to the preprocessing training module, is used to build a point cloud understanding and segmentation network model enhanced by a two-level attention mechanism.

[0177] The training and evaluation module, connected to the network model building module, is used for deep learning evaluation and interpretability and parameterized space model reconstruction.

[0178] The modeling module, connected to the training and evaluation modules, is used for visual interpretability analysis and parametric spatial modeling of network models.

[0179] This invention provides a computer device, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the depth visual perception method for architectural space analysis.

[0180] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0181] The spatial parsing network enhanced by a two-level attention mechanism provided in this invention achieves an mIoU value of 0.9713 in field experiments, higher than the mIoU value of 0.9445 achieved by PointNet++. This higher accuracy demonstrates that the machine has achieved effective semantic understanding of space and possesses high practical value. The AUC-ROC curve shows a high envelope area with the x-axis, indicating that this method performs well in point cloud segmentation of standard floors in completed buildings, showing great application potential. Table 4 shows that most point clouds were successfully segmented, indicating that the method can effectively process and learn point cloud data features. Test results of different networks under the same dataset distribution are presented. By comparing the results given in Table 5, it can be observed that the proposed network model shows better performance compared to advanced point cloud networks such as dgcnn, pointnet, pointnet++, and ASPCNET.

[0182] The modeling process for point cloud semantic segmentation results took approximately 5 minutes to generate a detailed parametric spatial model, while traditional methods took approximately 8 minutes to model the region. Automatic semantic segmentation of point clouds endows them with labeled information, dividing and clustering them into different spatial elements, which is crucial for spatial understanding. Errors can occur in the manual reconstruction of ring beams, such as... Figure 10 As shown, the beam was incorrectly modeled at some ceiling points. This method can significantly improve modeling speed and avoid errors.

[0183] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A depth visual perception method for architectural space analysis, characterized in that, The method includes: S1: Point cloud completion preprocessing and transfer learning pretraining; S2: Construction of a point cloud understanding and segmentation network model enhanced by a two-level attention mechanism; S3: Phased training and performance evaluation of point cloud understanding and segmentation network models; S4: Visual interpretability analysis and parametric spatial modeling of network models; S1 includes: (1) Point cloud completion preprocessing; using the symmetry of the components to complete the missing point cloud, that is, by copying part of the point cloud of the component itself to complete the missing part of the point cloud; (2) Transfer learning pre-training; After removing some categories from the publicly available indoor space point cloud dataset S3DIS, a sub-dataset is generated. The network model is pre-trained on this dataset and then trained and validated on the point cloud dataset of completed buildings collected on site. S2 includes: (1) A two-level attention-enhanced spatial parsing network; (2) A two-way symmetric loss function considering the imbalanced categories; (3) Multi-factor evaluation indicators and their determination; Step S2-(1) specifically includes: The input data first undergoes farthest point sampling (FPS) and KNN clustering. The FPS algorithm randomly selects a point from the input point cloud, then iterates to select the point furthest from the selected point, considering all features of the point cloud within limited computational resources. After sampling, the KNN algorithm clusters points near the sampled point to preserve local features. Two-dimensional convolution is then used to model the pattern of the feature data to extract point cloud features. During feature extraction, an efficiency channel attention mechanism (ECA) assigns attention weights to convolutional networks in different channels; a self-attention mechanism generates globally refined attention weights based on the input features. The self-attention mechanism (SA) processes input features to generate refined attention features. The input feature map F is fed into three independent 1×1 convolutional layers to reshape it into three one-dimensional vectors Fq, Fk, and Fv. Fq and Fk are then multiplied to obtain an n×n covariance matrix, i.e., the attention map. The attention map is then multiplied by Fv to obtain the output feature map, and the residuals of the input F are concatenated to generate a refined self-attention feature Fr. The self-attention feature Fr is weighted with F, i.e., out = α * Fr + F. Here, α is a learnable parameter, automatically adjusted by the optimizer to control the intensity of the attention mechanism. After the refined features are generated by the self-attention mechanism, deconvolution layer by layer upsampling increases the size of the feature map while reducing the number of channels, gradually restoring it to the size of the input feature map to enable segmentation. The shallow and deep features are fused, and the fused features are segmented by a one-dimensional convolutional layer. Finally, the input point cloud group will be segmented into 6 categories: ceiling, floor, beam, column, fence and wall.

2. The depth visual perception method for architectural space analysis as described in claim 1, characterized in that, Step S2-(2) specifically includes: It combines two loss functions: Symmetric Cross Entropy (SCE) loss and focus loss. Cross-entropy (CE) loss and reverse cross-entropy (RCE) loss can be calculated according to the following formulas (1) and (2), where SCE loss L s It can be calculated according to formula (3): ； ； ； in, It is the number of categories. For observation The true label, It is observation Classified as The predicted probability, and What are the respective weights of CE and RCE? The focus loss is defined by formula (4): ； in It is the modulation factor. It is the focus parameter, and the focus loss reduces the weight of easily classified samples through this modulation factor; As a combination, formula (5) gives a newly defined loss; ； in Indicates focal loss; Indicates RCE loss, and The weights are specified as hyperparameters. Furthermore, step S2-(3) specifically includes: The following four general evaluation metrics are introduced: precision (P), recall (R), f1 score (F1), and IoU, which can be expressed by formulas (6-9): ； ； ； ； TP, FP, and FN represent true positive, false positive, and false negative, respectively; n is the number of points.

3. The depth visual perception method for architectural space analysis as described in claim 1, characterized in that, S3 specifically includes: (1) Evaluation of training effect of network model; A bidirectional symmetric loss function is used, which is a combination of the focus loss of the forward module and the cross-entropy loss of the reverse module according to a certain weight; The forward module uses the network output value as the predicted value and the label as the real value, and assigns different penalty weights to point clouds of different categories with different segmentation difficulty; The reverse module uses the network output value as the real value and the label as the predicted value for calculation. The mean intersection-union ratio mIoU, as well as P value, F1 value, R value, and AUC-ROC curve index are introduced to evaluate the performance of the network model on the test set; (2) Interpretability analysis of the network model; using the Grad-CAM algorithm, the output of the network model is backpropagated to calculate the heat map of the feature extraction layer, highlighting the region where the convolutional neural network contributes the most to the decision of a specific category, so as to verify that the corresponding region is correctly identified and understood by the neural network. (3) Parametric BIM spatial model reconstruction: In Autodesk, after importing the parsed point cloud, the FARO As-built for Revit plugin is used to complete the process of building the spatial model. In this process, the component template library is first established for various structural components that appear in the scene. Secondly, points are randomly selected on the point cloud that is divided into categories for fitting. Finally, the components are combined according to the boundary conditions to form a complete parametric spatial model.

4. The depth visual perception method for architectural space analysis as described in claim 1, characterized in that, S4 specifically includes: (1) Visual interpretability analysis of the network model: The calculation formula for Grad-CAM is as follows: ； ； ； Grad refers to the target category Calculate category activation value Relative to feature map The gradient of each feature map channel is calculated, and then global average pooling is performed on the gradient of each channel to obtain the importance weight of that channel. The final calculation yielded The heatmap also needs to be upsampled to the size of the input tensor to highlight the areas most important for the network model to make specific class decisions; (2) Parametric reconstruction of space: In Revit, the FARO As build for Revit plugin builds a spatial model after importing a categorized point cloud. To build this model, first create a component template library and develop materials for the various structural components that appear in the scene. Next, randomly select points from the point cloud that are categorized and fit them. Finally, combine each component according to the boundary conditions to form a complete spatial model. A spatial model is built on the segmented point cloud rather than the original point cloud data.

5. A depth visual perception system for architectural space analysis implementing the depth visual perception method for architectural space analysis as described in any one of claims 1-4, characterized in that, The system includes: The preprocessing training module is used for point cloud completion preprocessing and transfer learning pretraining. The network model building module, connected to the preprocessing training module, is used to build a point cloud understanding and segmentation network model enhanced by a two-level attention mechanism. The training and evaluation module, connected to the network model building module, is used for deep learning evaluation, interpretability, and parametric space model reconstruction. The modeling module, connected to the training and evaluation modules, is used for visual interpretability analysis and parametric spatial modeling of network models.

6. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the depth visual perception method for architectural space analysis as described in any one of claims 1-4.

Images