A building indoor scene deep learning reconstruction method and system based on point cloud 2D-3D fusion

Abstract

The application discloses a kind of based on point cloud 2D-3D fusion's building indoor scene deep learning reconstruction method and system, the present application generates two-dimensional image directly from point cloud, and two-dimensional image and three-dimensional point cloud data are combined, more accurate point cloud semantic classification is realized using depth learning model.First, a plurality of angle two-dimensional images are generated using virtual camera, and the images are semantically segmented using a YOLO model.Next, the classification results of the YOLO model are combined with the point-by-point classification results of the point cloud using a PointNet series model, and the final semantic labels of each point are updated based on confidence.The present application can effectively process complex or sparse point cloud data, effectively distinguish between similar shape point cloud components, and improve the reconstruction accuracy and semantic segmentation of point cloud data.Meanwhile, it can be applied in building information modeling (BIM), providing support for building operation, maintenance and intelligent construction management.

Description

Technical Field

[0001] This invention belongs to the field of civil engineering and artificial intelligence interaction technology, specifically relating to a deep learning reconstruction method and system for building interior scenes based on point cloud 2D-3D fusion. Background Technology

[0002] Point cloud data, as crucial information for architectural scene modeling, is commonly used in tasks such as scene reconstruction and object recognition. While traditional point cloud semantic segmentation methods have achieved some success, the sparse nature and complex local geometry of point cloud data mean that existing deep learning algorithms still suffer from low segmentation accuracy when dealing with categories with few instances or similar shapes. In recent years, deep learning-based 2D and 3D fusion algorithms have made some progress in improving the accuracy of deep learning models, but the challenge of pre-registering image and point cloud data remains. How to better fuse 2D image information to improve segmentation accuracy remains a pressing research problem in this field. Summary of the Invention

[0003] This invention addresses the problems existing in point cloud reconstruction of architectural interior scenes in existing technologies by providing a deep learning-based method and system for reconstructing architectural interior scenes based on 2D-3D point cloud fusion. The aim is to improve the reconstruction accuracy of point clouds in architectural interior scenes by combining point cloud semantic segmentation and image semantic segmentation techniques. Unlike traditional methods, this invention does not require a 2D image registered with the point cloud; instead, it directly generates a corresponding 2D image from the point cloud and simultaneously performs semantic segmentation and recognition on both the point cloud and the image. Finally, the segmentation results are fused, which improves the accuracy of point cloud semantic segmentation.

[0004] To address the above technical problems, this invention provides the following technical solution: a deep learning reconstruction method for architectural interior scenes based on point cloud 2D-3D fusion, comprising the following steps:

[0005] S1. Use deep learning algorithms to perform semantic segmentation on the original 3D point cloud data of the indoor scene to obtain 3D semantic segmentation results;

[0006] S2. Generate several two-dimensional images from the original three-dimensional point cloud data using a virtual camera. Then, use the YOLO model to perform target detection, semantic segmentation, and generate a mask for the two-dimensional graphics. Finally, obtain the two-dimensional semantic segmentation result of voxel reconstruction through pose adjustment.

[0007] S3. Based on the two-dimensional semantic segmentation result obtained in step S2, and combined with the three-dimensional semantic segmentation result in step S1, the semantic label is updated, and the category with the highest comprehensive confidence is selected as the final semantic label to obtain the final semantic segmentation result.

[0008] S4. Based on the final semantic segmentation results, construct a Building Information Model (BIM) to reconstruct the building's interior scene.

[0009] Then, the final scene reconstruction result.

[0010] Furthermore, step S2, as described above, generates several two-dimensional images from the original three-dimensional point cloud data using a virtual camera, including the following sub-steps:

[0011] S2.1 Set up multiple virtual camera positions in the 3D point cloud data scene and capture images from different angles;

[0012] S2.2 Record the depth information and camera intrinsic and extrinsic parameters of each image. The depth information is used to calculate the 3D point cloud position corresponding to each image pixel.

[0013] S2.3 Map the pixels of each image to the 3D point cloud to generate the correspondence between the image and the point cloud.

[0014] Furthermore, the aforementioned step S2.2 includes the following sub-steps:

[0015] S2.2-1. When generating a 2D image, obtain the intrinsic and extrinsic parameter matrices of the virtual camera, establish the correspondence between point cloud data and the image, and map the 3D points in the camera coordinate system to the 2D image coordinate system using the virtual camera's intrinsic parameter matrix. The camera's intrinsic parameter matrix is ​​as follows:

[0016]

[0017] Among them, f x and f y c represents the focal length of the camera in the x and y directions, respectively. x and c y These are the coordinates of the principal point;

[0018] S2.2-2. Use the camera's extrinsic parameter matrix E to transform 3D points from the world coordinate system to the camera coordinate system. The extrinsic parameter matrix consists of a rotation matrix and a translation vector:

[0019]

[0020] Where R is the rotation matrix and t is the translation vector.

[0021] Furthermore, the aforementioned step S2.3 includes the following sub-steps:

[0022] S2.3-1. Using the intrinsic parameter matrix K and the extrinsic parameter matrix E, along with the captured image and depth information, calculate the position of each pixel in three-dimensional space. For any pixel (u,v) in the image and its corresponding depth value D(u,v), calculate the three-dimensional point (X) in the camera coordinate system using the following formula.cam ,Y cam Z cam ),

[0023]

[0024] Z cam =D(u, v)

[0025] 3D point P in camera coordinate system cam Represented as:

[0026]

[0027] S2.3-2 transforms 3D points in the camera coordinate system into 3D points in the world coordinate system, as shown in the following equation, after generating the point cloud,

[0028] A voxel mesh was constructed to aggregate points from different viewpoints;

[0029] P world =E -1 ·P cam

[0030] Among them, P world It is a three-dimensional point in the world coordinate system;

[0031] S2.3-3. Based on the 3D point cloud data generated from the image, the voxel mesh method is used to aggregate multiple 3D points from different viewpoints into the corresponding voxels. The voxel category is determined according to the category of the most points in that voxel.

[0032] Furthermore, the aforementioned step S3 includes the following sub-steps:

[0033] S3.1 For each point λ in the point cloud, the point-by-point classification probability obtained through the PointNet model is expressed as:

[0034] P PointNet family (λ)=[p1(λ),p2(λ),…,p n (λ)]

[0035] Where, p i (λ) is the predicted probability that point λ belongs to the i-th class, and the sum of all probabilities is 1, that is:

[0036] S3.2 When the YOLO prediction results are projected onto the 3D point cloud, a confidence score is provided for each predicted object category. For a point λ in the 3D space corresponding to the YOLO 2D detection result, the confidence score P based on YOLO is... YOLO (λ) is represented by the following formula:

[0037] P YOLO (λ)=[q1(λ), ​​q2(λ),…,q n (λ)]

[0038] Where, q i (λ) is the confidence score of point λ belonging to class i based on YOLO prediction;

[0039] S3.3 For each point λ in the point cloud, its final class confidence score is obtained by adding the confidence scores of the PointNet series models and YOLO for each class, as shown in the following formula:

[0040] P combined (λ)=P PointNet family (λ)+P YOLO (λ)

[0041] S3.4 Determine the label of point λ, first at P PointNet family (λ) and P YOLO Find the category with the highest confidence in (λ):

[0042] In P PointNet family The category with the highest confidence in (λ) is:

[0043] C PointNet family (λ)=argmax(P PointNet family (λ))

[0044] In P combined The category with the highest confidence in (λ) is:

[0045] C combined (λ)=argmax(P combined (λ))

[0046] S3.5 The decision to update the semantic label of point λ is based on the comparison of the confidence scores of the two categories in step S3.4.

[0047] The specific update rules are as follows:

[0048] P combined (λ,C combined (λ))and P PointNet family (λ,C PointNet family (λ))

[0049] If the confidence score of the merged label is greater than the confidence score of the original PointNet series labels, then update the label to C.combined (λ);

[0050] Otherwise, retain the original label C. PointNet family (λ),

[0051] The formula for updating tags is:

[0052]

[0053] Another aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the methods described in the present invention.

[0054] The present invention also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor of the steps of any of the methods described in the present invention.

[0055] Compared to existing technologies, the beneficial technical effects of the present invention using the above technical solution are as follows: By generating images directly from point clouds using 2D images without pre-registration, and combining 2D and 3D data, the semantic segmentation accuracy of point clouds is improved. This method effectively processes complex and sparse point cloud data, and by combining the classification results of YOLO and PointNet series models and improving classification accuracy through confidence updates, it achieves more accurate reconstruction of building interior scenes, supporting intelligent construction and maintenance of Building Information Modeling (BIM). Attached Figure Description

[0056] Figure 1 This is a flowchart of the method of the present invention.

[0057] Figure 2 This is an example diagram of how the present invention assigns semantic labels to point clouds using majority voting; in the diagram, (a) is a classification diagram of consistent voxel p, and (b) is an example diagram of majority voting for voxel Q.

[0058] Figure 3 These are visualizations of point cloud scenes in Autodesk Recap; (a) is the visualization of scene 1, and (b) is the visualization of scene 2.

[0059] Figure 4 This is an example diagram of the process of updating semantic information using the Point-YOLO method; in the diagram, (a) is a schematic diagram before the semantic information is updated, (b) is a schematic diagram after the semantic information is updated, (c) is a semantic update process diagram of point 1, and (d) is a semantic update process diagram of point 2.

[0060] Figure 5The diagram shows the confusion matrix comparison results for scenario 1; in the diagram, (a) is a schematic diagram of the PointNet++ confusion matrix results, and (b) is a schematic diagram of the PointNet++-YOLO confusion matrix results.

[0061] Figure 6 The diagram shows the comparison results of the confusion matrix in scenario 2; in the diagram, (a) is a schematic diagram of the PointNet++ confusion matrix result, and (b) is a schematic diagram of the PointNet++-YOLO confusion matrix result. Detailed Implementation

[0062] To better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.

[0063] In this invention, various aspects of the invention are described with reference to the accompanying drawings, in which numerous illustrative embodiments are shown. Embodiments of the invention are not limited to those depicted in the drawings. It should be understood that the invention is implemented through any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed herein are not limited to any particular implementation. Furthermore, some aspects of the invention disclosed may be used alone or in any suitable combination with other aspects of the invention disclosed.

[0064] refer to Figure 1 This invention provides a deep learning-based method for reconstructing architectural interior scenes using 2D-3D point cloud fusion. First, a point-based deep learning algorithm, such as PointNet or PointNet++, is used to perform semantic segmentation on the point cloud. Second, a simulated camera method is used to generate virtual images from the point cloud scene, and these images are then semantically segmented using deep learning algorithms such as YOLO. Finally, based on the 2D segmentation results, a 3D voxel representation with detected object labels is reconstructed. This 3D voxel representation is used to optimize and update the semantic segmentation of the point cloud based on the image segmentation results.

[0065] This implementation uses two different scenarios for experimental verification: Scenario 1 (architectural interior scene) and Scenario 2 (UGD dataset for underground parking garage). The architectural interior dataset contains six regions (Area 1 to Area 6) and defines 13 semantic categories: ceiling, floor, wall, beam, column, window, door, table, chair, sofa, bookshelf, panel, and clutter. The UGD dataset was scanned in the underground parking garage of the Humanities Building at Southeast University using a Leica RTC360 scanner. It includes XYZ coordinates, RGB color values, and density information, but does not include registered image data. The UGD dataset is divided into five regions (Area 1 to Area 5) and includes 10 component categories: duct, cable tray, ceiling, fire extinguisher, floor, light fixture, column, pipe, wall, and door. Because this dataset combines large building components (such as ceilings and floors) with smaller but complex electromechanical system (MEP) components, its scene segmentation task is quite challenging.

[0066] The datasets for Scene 1 and Scene 2 are divided into training and test sets, respectively. In Scene 1, point cloud data from Areas 1, 2, 3, 4, and 6 are used as the training set, and 42 office scenes from Area 5 are used as the test set. For Scene 2, Areas 1, 2, and 5 are used as the training set, and Areas 3 and 4 are used as the test set. The scene legends in Autodesk Recap are shown as follows. Figure 3 The point cloud scene is visualized in Autodesk Recap; (a) is the visualization of scene 1, and (b) is the visualization of scene 2.

[0067] The specific steps of this invention are as follows:

[0068] S1. Semantic segmentation of raw 3D point cloud data for indoor scenes is performed using deep learning algorithms to obtain 3D semantic segmentation results. This invention utilizes deep learning algorithms based on PointNet and PointNet++ to perform semantic segmentation of point cloud data. PointNet processes each point through shared multilayer perceptrons (MLPs), extracts features, and aggregates features using symmetric functions (such as max pooling) to obtain global features, making it suitable for processing unordered point cloud data. However, PointNet suffers from insufficient capture of local geometric structures. To address this, PointNet++ introduces a hierarchical learning mechanism, selecting center points and forming local regions through farthest point sampling and radius search, thereby capturing multi-scale local structural information.

[0069] S2. Several 2D images are generated from the original 3D point cloud data using a virtual camera. Then, the YOLO model is used to perform object detection, semantic segmentation, and generate a mask for the 2D graphics. Finally, the 2D semantic segmentation result of voxel reconstruction is obtained through pose adjustment. This invention utilizes YOLOv8 for image semantic segmentation. YOLOv8 is a fundamental image object detection and segmentation algorithm that combines high speed and accuracy. This method is based on a convolutional neural network (CNN) and can directly predict bounding boxes and class probabilities. It divides the image into multiple grids, with each grid predicting an object and its confidence score.

[0070] Step S2 generates several two-dimensional images from the original 3D point cloud data using a virtual camera, including the following sub-steps:

[0071] S2.1 Set up multiple virtual camera positions in the 3D point cloud data scene and capture images from different angles;

[0072] S2.2 Record the depth information and camera intrinsic and extrinsic parameters of each image. The depth information is used to calculate the 3D point cloud position corresponding to each image pixel. Specifically, step S2.2 includes the following sub-steps:

[0073] S2.2-1. When generating a 2D image, obtain the intrinsic and extrinsic parameter matrices of the virtual camera, establish the correspondence between point cloud data and the image, and map the 3D points in the camera coordinate system to the 2D image coordinate system using the virtual camera's intrinsic parameter matrix. The camera's intrinsic parameter matrix is ​​as follows:

[0074]

[0075] Among them, f x and f y c represents the focal length of the camera in the x and y directions, respectively. x and c y These are the coordinates of the principal point;

[0076] S2.2-2. Use the camera's extrinsic parameter matrix E to transform 3D points from the world coordinate system to the camera coordinate system. The extrinsic parameter matrix consists of a rotation matrix and a translation vector:

[0077]

[0078] Where R is the rotation matrix and t is the translation vector.

[0079] S2.3 Map the pixels of each image to the 3D point cloud to generate the correspondence between the image and the point cloud. Specifically, step S2.3 includes the following sub-steps:

[0080] S2.3-1. Using the intrinsic parameter matrix K and the extrinsic parameter matrix E, along with the captured image and depth information, calculate the position of each pixel in three-dimensional space. For any pixel (u,v) in the image and its corresponding depth value D(u,v), calculate the three-dimensional point (X) in the camera coordinate system using the following formula. cam ,Y cam Z cam )

[0081]

[0082] Z cam =D(u, v)

[0083] 3D point P in camera coordinate system cam Represented as:

[0084]

[0085] S2.3-2 transforms 3D points in the camera coordinate system into 3D points in the world coordinate system, as shown in the following formula. After generating the point cloud, a voxel mesh is constructed to aggregate points from different viewpoints.

[0086] P world =E -1 ·P cam

[0087] Among them, P world It is a three-dimensional point in the world coordinate system;

[0088] S2.3-3. Based on the 3D point cloud data generated from the image, a voxel mesh method is used to aggregate multiple 3D points from different viewpoints into corresponding voxels. The voxel category is determined based on the category with the most points within that voxel. Voxel meshes help resolve conflicts when multiple points from different viewpoints correspond to the same voxel. In this case, a majority voting mechanism is used to assign voxel labels to ensure that the category with the most points within the voxel is selected. For example... Figure 2 As shown, Figure 2This is an example diagram illustrating the use of majority voting to assign semantic labels to point clouds in this invention; in the diagram, (a) is a schematic diagram of consistent classification for voxel p, and (b) is an example diagram of majority voting for voxel Q. As shown, four pixels from different images (AD) correspond to voxels P or Q, respectively. For voxel P, all four pixels from different images are classified as category 1. Therefore, the category of voxel P is updated to category 1. On the other hand, for voxel Q, two pixels are classified as category 2, and the other two pixels are classified as category 1 and category 3, respectively. In this case, the category of voxel Q is updated to the majority category, i.e., category 2. S3. Based on the two-dimensional semantic segmentation result obtained in step S2, and combined with the three-dimensional semantic segmentation result in step S1, the semantic label is updated, and the category with the highest comprehensive confidence is selected as the final semantic label for the point, thus obtaining the final semantic segmentation result. This invention uses point-based deep learning algorithms (such as the PointNet series) to perform semantic classification on each point, obtaining the classification probability of each point; it uses the YOLO model to detect objects in the image and generate a confidence score for each object, projecting these confidence scores onto a 3D point cloud; by combining the confidence scores of the YOLO model with the classification probabilities of the PointNet series model, it calculates the final classification confidence score for each point in the point cloud; based on the final confidence score, it selects the category with the highest classification probability as the final label for that point.

[0089] This embodiment provides a method for directly generating images from 3D point clouds, avoiding the need for external 2D-3D data registration and the requirement for corresponding 2D images of point clouds lacking registration. The method uses a virtual camera to capture multiple images from a point cloud scene and records the corresponding depth and position information. Specifically, the entire scene is visualized using Open3D, and images are generated at different positions and angles using a virtual camera. During each image generation process, depth data is acquired, and the camera's intrinsic and extrinsic parameters are recorded to establish the correspondence between the point cloud and the image. The specific steps include the following:

[0090] S3.1 For each point λ in the point cloud, the point-by-point classification probability obtained through the PointNet model is expressed as:

[0091] P PointNet family (λ)=[p1(λ), ​​p2(λ),…,p n (λ)]

[0092] Where, p i (λ) is the predicted probability that point λ belongs to the i-th class, and the sum of all probabilities is 1, that is:

[0093] S3.2 When the YOLO prediction results are projected onto the 3D point cloud, a confidence score is provided for each predicted object category. For a point λ in the 3D space corresponding to the YOLO 2D detection result, the confidence score P based on YOLO is... YOLO (λ) is represented by the following formula:

[0094] P YOLO (λ)=[q1(λ), ​​q2(λ),…,q n (λ)]

[0095] Where, q i (λ) is the confidence score of point λ belonging to class i based on YOLO prediction;

[0096] S3.3 For each point λ in the point cloud, its final class confidence score is obtained by adding the confidence scores of the PointNet series models and YOLO for each class, as shown in the following formula:

[0097] P combined (λ)=P PointNet family (λ)+P YOLO (λ)

[0098] S3.4 Determine the label of point λ, first at P PointNet family (λ) and P YOLO Find the category with the highest confidence in (λ):

[0099] In P PointNet family The category with the highest confidence in (λ) is:

[0100] C PointNet family (λ)=argmax(P PointNet family (λ))

[0101] In P combined The category with the highest confidence in (λ) is:

[0102] C combined (λ)=argmax(P combined (λ))

[0103] S3.5 The decision to update the semantic label of point λ is based on the comparison of the confidence scores of the two categories in step S3.4.

[0104] The specific update rules are as follows:

[0105] P combined (λ,C combined (λ))and PPointNet family (λ,C PointNet family (λ))

[0106] If the confidence score of the merged label is greater than the confidence score of the original PointNet series labels, then update the label to C. combined (λ);

[0107] Otherwise, retain the original label C. PointNet family (λ),

[0108] The formula for updating tags is:

[0109]

[0110] S4. Based on the final semantic segmentation results, construct a Building Information Model (BIM) to reconstruct the building's interior scene.

[0111] The method of this invention is applied to point cloud reconstruction of architectural interior scenes, which can significantly improve the accuracy of point cloud semantic segmentation, especially in the processing of point cloud data with similar or sparse shapes, providing higher accuracy and efficiency than existing methods.

[0112] This embodiment first selects PointNet and PointNet++ as benchmark algorithms for experimental verification. Subsequently, PointNext and...

[0113] PointVector was used as a comparison algorithm to further evaluate the performance of the proposed algorithm. In the initial experiments, deep learning algorithms (such as...) were used.

[0114] PointNet and PointNet++ were used for image annotation and training, followed by Point-YOLO algorithm testing. Specifically, PointNet-YOLO combines the YOLO and PointNet algorithms, while PointNet++-YOLO combines the YOLO and PointNet++ algorithms. During image annotation and training, 415 images were randomly selected from the training set of Scene 1, with 1 to 3 images taken from each room. These images were annotated, covering eight categories: beams, columns, windows, doors, tables, chairs, sofas, and panels. Because these categories are relatively few in the S3DIS dataset, they are more likely to be misclassified as more common categories such as floors, walls, and ceilings. In Scene 2, 90 images were randomly selected from the training set of each area, covering seven categories: ducts, cable trays, fire extinguishers, light fixtures, pillars, pipes, and doors. Similar to Scene 1, these components are also relatively few in Scene 2, making misclassification more likely. After image annotation, the YOLOv8 model was trained. Of the labeled images, 90% of the data was used for training, and the remaining 10% served as a validation set. The initial training model used a pre-trained model based on the COCO dataset.

[0115] In the image semantic segmentation process, for Scene 1, each room in the test set was filmed using a virtual camera. The virtual camera was positioned in the center of the room, capturing an image every 45 degrees to achieve a 360-degree panoramic view. Then, the camera moved 0.5 meters in each of the four directions (front, back, left, and right) and captured 32 more images. This resulted in a total of 40 images for each room, effectively covering the entire scene. For Scene 2, due to the larger area, the central point method was insufficient to cover all regions; therefore, Open3D was used for manual filming, capturing a total of 106 images across the two test areas.

[0116] Image training lasted 200 epochs, while point cloud training lasted 100 epochs. All experiments were performed on a single NVIDIA L40 GPU, with training parameters using the model's default settings.

[0117] This invention employs evaluation metrics such as confusion matrix (CM), overall accuracy (OA), precision, recall, F1 score, intersection-over-union ratio (IoU), mean intersection-over-union ratio (mIoU), and mean class accuracy (mAcc) to comprehensively assess the performance of deep learning algorithms. The confusion matrix is ​​a fundamental tool for evaluating classification model performance. By comparing model predictions with true labels, it is broken down into four key components: true positives, false positives, true negatives, and false negatives. mAcc calculates the accuracy for each class and averages it, ensuring that less common classes are not masked by more common classes, thus comprehensively reflecting model performance. Overall accuracy (OA) represents the proportion of correctly classified samples out of the total number of samples. Precision and recall reflect the proportion of correctly predicted positives out of the total predicted positives, and the proportion of correctly predicted positives out of the actual positives, respectively. The F1 score comprehensively considers precision and recall, balancing their weights. Intersection over Union (IoU) measures the degree of overlap between the predicted segment and the ground truth label, while the average IoU (mIoU) provides the average IoU for all classes as an evaluation metric for overall segmentation performance.

[0118] Figure 4 An example illustrating the semantic label update process is shown. In the figure, (a) is a schematic diagram before the semantic information is updated, (b) is a schematic diagram after the semantic information is updated, (c) is a semantic update process diagram for point 1, and (d) is a semantic update process diagram for point 2. For point 1 from the door, when using PointNet for semantic segmentation, this point might be misclassified as floor because the semantic score of board material (0.303) is the highest among all categories. When using the PointNet-YOLO method, YOLO will classify this point as door and give it a confidence score of 0.65. Therefore, adding the PointNet semantic score to the YOLO confidence score, the total score for door is 0.284 + 0.65 = 0.934. Thus, this point will be correctly classified as door.

[0119] For point 2 from the wall, PointNet correctly classifies it as "wall" with a semantic score of 0.641, the highest among all points. However, YOLO classifies it as "chair" with a confidence score of 0.51. In this case, the total score for "chair" is 0.056 + 0.51 = 0.566, still lower than "wall" (0.641). Therefore, the final classification of this point is still "wall".

[0120] Figure 5 and Figure 6 The confusion matrix comparison results of PointNet++ and PointNet++-YOLO in scenario 1 and scenario 2 are shown respectively. Figure 5Figure (a) is a schematic diagram of the PointNet++ confusion matrix result, and (b) is a schematic diagram of the PointNet++-YOLO confusion matrix result. Figure 6 In the diagram, (a) shows the PointNet++ confusion matrix result, and (b) shows the PointNet++-YOLO confusion matrix result. The results show that, regardless of whether it is scenario 1 or scenario 2, the accuracy of various components of PointNet++-YOLO is higher than that of PointNet++.

[0121] Another aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the methods described in the present invention.

[0122] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the methods described in the present invention.

[0123] While the present invention has been described above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A deep learning reconstruction method for architectural interior scenes based on point cloud 2D-3D fusion, characterized in that, Includes the following steps: S1. Use deep learning algorithms to perform semantic segmentation on the original 3D point cloud data of the indoor scene to obtain 3D semantic segmentation results; S2. Generate several two-dimensional images from the original three-dimensional point cloud data using a virtual camera. Then, use the YOLO model to perform target detection, semantic segmentation, and generate a mask for the two-dimensional graphics. Finally, obtain the two-dimensional semantic segmentation result of voxel reconstruction through pose adjustment. S3. Based on the two-dimensional semantic segmentation result obtained in step S2, and combined with the three-dimensional semantic segmentation result in step S1, the semantic label is updated, and the category with the highest comprehensive confidence is selected as the final semantic label to obtain the final semantic segmentation result. Specifically, it includes the following sub-steps: S3.1 For each point λ in the point cloud, the point-by-point classification probability obtained through the PointNet model is expressed as: P PointNetfamily (λ) = [p1(λ), p2(λ),..., p n (λ)] Where, p i (λ) is the predicted probability that point λ belongs to the i-th class, and the sum of all probabilities is 1, that is: S3.2 When the YOLO prediction results are projected onto the 3D point cloud, a confidence score is provided for each predicted object category. For a point λ in the 3D space corresponding to the YOLO 2D detection result, the confidence score P based on YOLO is... YOLO (λ) is represented by the following formula: P YOLO (λ) = [q1(λ), q2(λ),..., q n (λ)] where q i (λ) is a confidence score based on YOLO prediction that the point λ belongs to the i-th class; S3.3 For each point λ in the point cloud, its final class confidence score is obtained by adding the confidence scores of the PointNet series models and YOLO for each class, as shown in the following formula: P combined (λ) = P PointNetfamily (λ) + P YOLO (λ) S3.4, determine the label of point λ, first in P PointNetfamily (λ) and P YOLO (λ) find the class with the highest confidence: In P PointNetfamily The class with the highest confidence in (λ) is: C PointNetfamily (λ) = argmax(P PointNetfamily (λ)) In P combined The class with the highest confidence in (λ) is: C combined (λ) = argmax(P combined (λ)) S3.5 The decision to update the semantic label of point λ is based on the comparison of the confidence scores of the two categories in step S3.

4. The specific update rules are as follows: P combined (λ, C combined (λ))and P PointNet family (λ, C PointNet family (λ))if the fused label confidence is greater than the confidence of the original PointNet series label, update the label C combined (λ). Otherwise, keep original label C PointNet family (λ), The formula for updating tags is: S4. Based on the final semantic segmentation results, construct a Building Information Model (BIM) to reconstruct the building's interior scene.

2. The deep learning reconstruction method for architectural interior scenes based on point cloud 2D-3D fusion as described in claim 1, characterized in that, Step S2 generates several two-dimensional images from the original 3D point cloud data using a virtual camera, including the following sub-steps: S2.1 Set up multiple virtual camera positions in the 3D point cloud data scene and capture images from different angles; S2.2 Record the depth information and camera intrinsic and extrinsic parameters of each image. The depth information is used to calculate the 3D point cloud position corresponding to each image pixel. S2.3 Map the pixels of each image to the 3D point cloud to generate the correspondence between the image and the point cloud.

3. The deep learning reconstruction method for architectural interior scenes based on point cloud 2D-3D fusion as described in claim 1, characterized in that, Step S2.2 includes the following sub-steps: S2.2-1. When generating a 2D image, obtain the intrinsic and extrinsic parameter matrices of the virtual camera, establish the correspondence between point cloud data and the image, and map the 3D points in the camera coordinate system to the 2D image coordinate system using the virtual camera's intrinsic parameter matrix. The camera's intrinsic parameter matrix is ​​as follows: where f x and f y represent the focal length of the camera in the x and y direction, respectively, c x and c y are the coordinates of the principal point; S2.2-2, convert the three-dimensional point from the world coordinate system to the camera coordinate system using the extrinsic matrix E of the camera, which consists of a rotation matrix and a translation vector: Where R is the rotation matrix and t is the translation vector.

4. The deep learning reconstruction method for architectural interior scenes based on point cloud 2D-3D fusion as described in claim 1, characterized in that, Step S2.3 includes the following sub-steps: S2.3-1, by using the intrinsic matrix K and extrinsic matrix E, and the captured image and depth information, calculate the position of each pixel in three-dimensional space, for any pixel (u, v) in the image and its corresponding depth value D(u, v), calculate the three-dimensional point (X cam , Y cam , Z cam ) in the camera coordinate system by the following formula X = f * u * D(u, v) / (Ku1 * Ku2 - K21 * Ku2) * K21 Y = f * v * D(u, v) / (Ku1 * Ku2 - K21 * Ku2) * Ku1 Z = D(u, v) S2.3-2, by using the intrinsic matrix K and extrinsic matrix E, and the captured image and depth information, calculate the position of each pixel in three-dimensional space, for any pixel ( Z cam = D(u, v) Three-dimensional point P in the camera coordinate system cam is expressed as: S2.3-2 transforms 3D points in the camera coordinate system into 3D points in the world coordinate system, as shown in the following equation, after generating the point cloud, A voxel mesh was constructed to aggregate points from different viewpoints; P world =E -1 ·P cam Among them, P world It is a three-dimensional point in the world coordinate system; S2.3-3. Based on the 3D point cloud data generated from the image, the voxel mesh method is used to aggregate multiple 3D points from different viewpoints into the corresponding voxels. The voxel category is determined according to the category of the most points in that voxel.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1-4.

6. A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor of the steps of the method according to any one of claims 1-4.

Images