A method and device for reconstructing a three-dimensional model of a house based on missing point cloud data, and a medium

Abstract

The application belongs to the field of real scene three-dimensional reconstruction, and provides a house three-dimensional model reconstruction method and equipment based on missing point cloud data, and a medium. The method fuses an image generated point cloud method, a neural network method, and a skeleton line extraction method, and provides a new solution idea for processing of missing point cloud data; video or panoramic image data is used to generate point cloud data based on an SFM principle; a region growing algorithm is used to sample and segment the point cloud; a neural network based on PointNet is constructed, and a cross entropy loss function is used to realize point cloud missingness judgment; a high confidence point cloud mapping completion method depending on a sampling point as a Truth Point is used to realize a full-process real scene three-dimensional reconstruction, and images are fused into a three-dimensional scene according to strict geometric relations. The application has wide use value in automatic driving, indoor navigation, augmented reality (AR), robots and other related visual task scenes based on the field of real scene three-dimensional reconstruction.

Description

Technical Field

[0001] This invention belongs to the field of real-scene 3D reconstruction and proposes an improved method for processing missing point cloud data. Real-scene 3D reconstruction involves establishing a 3D model of a building by constructing and fusing panoramic images with a 3D scene model, and then integrating the images into the 3D scene according to strict geometric relationships. The point cloud data used in this process may be missing, and the missing data needs to be located and supplemented before it can be used. Background Technology

[0002] The technology of reconstructing 3D building models using point cloud data can perform real-time measurable 3D modeling of buildings based on the obtained 3D images of the buildings. This allows users to view the entire building from different angles and in different spaces, and obtain accurate data about the building. It has high practical and commercial value.

[0003] When processing point cloud data, we need to be aware that point cloud data has the following characteristics:

[0004] 1. Point clouds are disordered. Due to the influence of acquisition equipment and coordinate system, the arrangement order of 3D points can vary greatly when the same object is scanned using different equipment or positions. Such data is difficult to process directly using an end-to-end model.

[0005] 2. Point clouds exhibit sparsity. In robotics and autonomous driving scenarios, the sampling point coverage of LiDAR is highly sparse relative to the scale of the scene. In the KITTI dataset, if the original LiDAR point cloud is projected onto the corresponding color image, only about 3% of the pixels have corresponding LiDAR points. This extremely strong sparsity makes high-level semantic perception based on point clouds particularly difficult.

[0006] 3. Point clouds have limited information content. The data structure of a point cloud is a set of points composed of coordinates of points in three-dimensional space. In essence, it is a low-resolution resampling of the geometry of the three-dimensional world, and therefore can only provide partial geometric information.

[0007] Therefore, before building a 3D model using point cloud data, if the point cloud data is incomplete, we need to locate, determine, and complete the missing data. For completing missing point cloud data, we propose the following three methods: a domain-based approach; a database-based simulation repair approach; and a neural network approach. Summary of the Invention

[0008] This project provides a method for reconstructing 3D house models based on missing point cloud data, which solves the problem of missing data caused by point cloud data acquisition when constructing 3D house models, and enables rapid construction and complete 3D visualization of real-world house models.

[0009] To achieve the above objectives, the present invention relates to a method and technical process for reconstructing a 3D model of a house based on missing point cloud data, which includes the following steps:

[0010] A method for reconstructing a 3D house model based on missing point cloud data, characterized by the following steps:

[0011] After acquiring 360-degree panoramic data of the house and preprocessing it, a sequence of multi-view images is obtained.

[0012] A three-dimensional sparse point cloud is generated by matching and reconstructing a sequence of multi-view images;

[0013] The 3D sparse point cloud is divided into partitions, and the missing point cloud of each partition is judged based on the encoding and decoding neural network. After the missing point cloud of each partition is filled in according to the judgment result, the 3D model is reconstructed.

[0014] In the above method,

[0015] Acquire 360-degree panoramic data of the house and create a 360-degree sequence of panoramic images;

[0016] Multi-view projection correction is performed on the sequence of 360-degree panoramic images to obtain the corrected sequence of multi-view images;

[0017] Dense matching and SFM reconstruction are performed on the corrected sequence of multi-view images to generate a sparse 3D point cloud. PMVS is used to generate a dense 3D point cloud and smooth it. Then, a region growing algorithm is used to uniformly sample and segment it to obtain a partitioned dense 3D point cloud.

[0018] In the above method,

[0019] Construct an encoder-decoder neural network, where the encoder is used to extract feature vectors from the point cloud and the decoder is used to calculate the missing value results;

[0020] Using a trained encoder-decoder neural network, the partitioned point cloud is used as input parameters to calculate the point cloud missingness result;

[0021] Using the ShapeNet dataset, we trained and tested a deep learning completion algorithm, and then segmented the planar point cloud obtained from the panoramic image.

[0022] Based on the sampling points of the input image and the surface of the 3D model mesh as Truth Points, the images of the corresponding viewpoints in the ShapeNet dataset are copied and aligned to construct a mapping between each input image and the corresponding image in the dataset.

[0023] Set a maximum threshold, select images from the dataset that are below the threshold and have the best performance, and construct a planar point cloud based on the dataset images to fill in the missing planes in the point cloud data.

[0024] Using a trained neural network, the partitioned point cloud is used as input parameters to obtain the point cloud missingness judgment;

[0025] Based on the sampling points of the input image and the surface of the 3D model mesh as feature matching points, the images of the corresponding viewpoints in the ShapeNet dataset are copied and aligned to construct a mapping between each input image and the corresponding image in the dataset.

[0026] Set a maximum threshold, select images from the dataset that fall below the threshold and produce the best results, and construct a planar point cloud based on the dataset images to fill in missing planes in the point cloud data.

[0027] In the above method,

[0028] The coordinates of the camera center are determined by the bundle method, and the camera center is projected onto a horizontal plane. Labels are then placed at the projection points.

[0029] Using the center point of the photograph and the corresponding label position as a reference, the panoramic view, floor plan, and 3D model of the house are unified into the same coordinate system.

[0030] In the method described above, the 360-degree panoramic data of the building is obtained through user uploads or mobile phone photography, and the panoramic data includes panoramic images and video streams. If it is a video stream, frames are extracted to form image data.

[0031] In the above method, when correcting the multi-view projection of 360-degree panoramic images, an equirectangular projection spherical panoramic model is used to transform the original input 360-degree panoramic image into a three-dimensional panoramic spherical space, as shown in the following formula:

[0032]

[0033]

[0034] In the formula: x pano Let y be the coordinates of the image point in the original panoramic image. pano Here, w represents the coordinates of the corresponding point in the 3D panoramic sphere, h represents the original panoramic image length, θ represents the original panoramic image width, and Φ represents the horizontal and vertical rotation angles of the sphere's spatial azimuth and the image capture azimuth.

[0035] In the above method, SFM reconstruction to generate a 3D sparse point cloud includes:

[0036] The SIFT feature operator is used to extract feature points from the multi-view images of the house sequence. Specifically, several horizontal and vertical images are selected to form the multi-view images of the house sequence. A Gaussian feature pyramid is constructed using the multi-view images of the house sequence, and finally the SIFT feature operator is used to extract feature points.

[0037] The image matching method of support line projection and affine invariance constraint is used to match multi-view images of buildings. Specifically, the image matching method of support line projection and affine invariance constraint is used to obtain the image matching point set, restore the pose of the panoramic image camera, and reconstruct a three-dimensional sparse point cloud through multi-view images.

[0038] In the above method, the point cloud is uniformly sampled and segmented using a region growing algorithm, and then the point cloud blocks are obtained after surface fitting.

[0039] In the method described above, the encoder-decoder neural network is based on the BuildNet neural network, specifically constructed as follows:

[0040] Encoder: Used to extract point cloud feature vectors. The input is a block sequence of point clouds. Each unit of the encoder uses a PointNet network to perform initial processing on the input point cloud and outputs local point cloud feature vectors.

[0041] Decoder: Used to determine whether a point cloud has missing features. It takes the point cloud feature vector as input, performs average pooling and sigmoid activation on the decoded 1024*27 feature matrix, outputs the missing feature value of the point cloud block, and updates the weights using backpropagation with the loss function. The formula is as follows:

[0042]

[0043] Loss function: The cross-entropy formula is used to compare the predicted results with the actual results to calculate the loss value. The gradient descent rule is then used to backpropagate the network, adjust the parameter values, and obtain the optimal result.

[0044]

[0045] In the above method, feature point matching is performed during the mapping construction process.

[0046] A matching algorithm is used to obtain images of the corresponding viewpoints using the dataset. The results are then copied and aligned. Specifically, based on the input image and sampling points on the surface of the 3D model mesh, feature point matching is used to copy and align the images of the corresponding viewpoints in the ShapeNet dataset.

[0047] A one-to-one mapping is performed between the point cloud and the images in the dataset to obtain the optimal mapped image. The point cloud is then reconstructed based on the mapped image. Specifically, the shape and skeleton of the point cloud plane are mapped one-to-one with the images in the dataset. The structure with the highest mapping matching degree is selected from the data that meets the mapping accuracy threshold to complete the point cloud.

[0048] In the method described above, when using the partitioned point cloud as input parameters, the segmented point cloud blocks are input as a sequence, and the correlation between adjacent point clouds is calculated. Specifically, the encoder needs to ensure the continuity of the point cloud blocks when extracting features.

[0049] The input point cloud is sampled uniformly, and the point cloud is meshed and divided into 3*3*3 structures.

[0050] The block point cloud is input into the encoder structure in sequence, so that the previous unit input point cloud block and the next unit input point cloud block have overall connectivity.

[0051] In the method described above, each unit of the encoder uses a PointNet network to perform initial processing on the input point cloud to extract local feature vectors. The decoder constructs a two-layer network to perform operations on the decoded 1024*27 feature matrix.

[0052] In the method described above, the input sampling points are used as feature matching points, and the images of the corresponding viewpoints in the ShapeNet dataset are copied and aligned. The optimal matching result is selected for the data that meets the threshold and used to complete the missing point cloud.

[0053] The above method is used to verify the completion effect. A mathematical model is constructed through the connectivity relationship of the point cloud, and its texture framework structure is extracted. This model is then compared with the skeleton structure of the previously input image to verify the rationality of the completion.

[0054] The method described above involves constructing the final 3D model of the room and unifying the panoramic view, floor plan, and 3D model of the house into the same coordinate system. Specifically, this includes:

[0055] The elevation information of the floor and ceiling of each room is obtained. The number of points in the elevation histogram is presented as two peaks, which represent the elevation of the floor and ceiling of the room, respectively.

[0056] The Delaunay triangulation method is used to triangulate the ceiling, walls and floor of each room into polygons, construct the final 3D model of the room, and finally output the constructed 3D model of the room in the form of a vector mesh.

[0057] The coordinates of the photographic center are determined using the bundle method, and the panoramic view, floor plan, and 3D model of the building are unified to the same coordinates using the photographic center point and the corresponding label position as references.

[0058] The above method is used to verify the completion effect, specifically as follows:

[0059] A mathematical model is constructed based on the connectivity relationships of point clouds, and its texture framework structure is extracted.

[0060] The skeleton structure extracted from the input image is compared and verified with the frame structure of the completed point cloud;

[0061] The model similarity index is evaluated. Local matching accuracy is calculated under the conditions of identical azimuth angles and scale normalization. A completion of erroneous point cloud structures accounting for less than 5% of the overall structure is considered to meet the accuracy requirements.

[0062] An apparatus employing the method, characterized in that it includes:

[0063] Module 1: Acquire 360-degree panoramic data of the building and preprocess it to obtain a sequence of multi-view images;

[0064] The second module: After matching and reconstructing the sequence of multi-view images, a three-dimensional sparse point cloud is generated;

[0065] The third module involves partitioning the 3D sparse point cloud, determining the point cloud missingness of each partition based on an encoding-decoding neural network, and then reconstructing the 3D model after filling in the missing point cloud of each partition according to the determination results.

[0066] A storage medium storing an executable program that, when executed, can perform any of the methods described above.

[0067] The present invention has the following positive effects:

[0068] 1) This invention provides a more mobile, faster, and more flexible method for processing missing point clouds. This method performs well in the process of reconstructing 3D models of houses from panoramic images, and has the characteristics of being clear, accurate, and free from noise interference.

[0069] 2) The output of point cloud completion retains the complete structure of the partial observation of the input point cloud, and can infer the complete point cloud shape of the object from the partial observation of the point cloud.

[0070] 3) This invention adopts a convenient process in which the public independently collects and uploads data, and the data is processed by the terminal and then returned as a 3D model. It is simple to operate and easy to promote.

[0071] This invention focuses on the field of real-scene 3D reconstruction and proposes an improved method for processing missing point cloud data. It has wide applications in vision-related tasks such as autonomous driving, 3D reconstruction, indoor navigation, augmented reality (AR), and robotics. 3D structures reconstructed from 2D images often suffer from sparse, biased, and incomplete point cloud data due to resolution limitations and occlusion by irrelevant objects. By using computer technology to determine and complete the point cloud, a reasonable and complete point cloud can be obtained, facilitating subsequent downstream tasks such as classification and segmentation. Attached Figure Description

[0072] Figure 1 This is a flowchart illustrating the reconstruction of a 3D house model based on missing point clouds, as described in this invention.

[0073] Figure 2 This is a schematic diagram of point cloud segmentation according to the present invention.

[0074] Figure 3 This is a flowchart illustrating the feature vector extraction process based on the PointNet network in this invention.

[0075] Figure 4 The flowchart illustrates the process of constructing the BuildNet network to determine the missingness of point clouds in this invention.

[0076] Figure 5 The Sigmoid function used in constructing the neural network in this invention.

[0077] Figure 6a This is the process of adding a model;

[0078] Figure 6b This refers to the process of adding panoramic photos;

[0079] Figure 7 It requires panoramic images;

[0080] Figure 8 This refers to the image upload process;

[0081] Figure 9a 9b and 9c are panoramic images of room A;

[0082] Figure 10a 10b, 10c, and 10d are panoramic images of room B;

[0083] Figure 11 It is the process of generating a floor plan;

[0084] Figure 12 This is the result of generating the plan view;

[0085] Figure 13a 13b is a floor plan of rooms A and B;

[0086] Figure 14 It is the process of generating the house layout;

[0087] Figure 15 This is the result of generating the house layout;

[0088] Figure 16a Images 16b, 16c, and 16d are multi-angle observation images of the building. Detailed Implementation

[0089] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0090] Example:

[0091] This invention discloses a method for processing missing point cloud data in reconstructing 3D building models using point clouds, comprising the following steps:

[0092] Step 1: Users independently upload 360-degree panoramic data of the house, which can be panoramic images or videos of the house, and upload them to the client for processing. The panoramic data provided by the user needs to be taken according to the actual situation of the building, so as to lay a good foundation for the subsequent construction of the 3D model of the house and provide a guarantee.

[0093] Step 2: Generate a 3D point cloud of the house based on the 360-degree panoramic data of the house provided by the user.

[0094] 2.1 First, the 360-degree panoramic video stream is frame-by-frame extracted to create a sequence of 360-degree panoramic images. Then, the 360-degree panoramic images are subjected to multi-view projection correction to obtain a sequence of multi-view images. Specifically, an equirectangular projection spherical panoramic model is used to transform the original input 360-degree panoramic images into a three-dimensional panoramic spherical space. The main relationships are shown below:

[0095]

[0096] In the formula, x pano Let y be the coordinates of the image point in the original panoramic image. pano Let w be the coordinates of the corresponding point in the 3D panoramic sphere, w be the length of the original panoramic image, and h be the width of the original panoramic image.

[0097] 2.2 Next, the panoramic sphere is enclosed by a square with the same side length as its diameter, thus mapping the points on the sphere's surface onto the corresponding cube surfaces, resulting in 6 multi-view images. The cube is then rotated 45°, and the points on the sphere's surface are again mapped onto the corresponding cube surfaces, yielding another 6 multi-view images.

[0098] 2.3 Finally, at least 6 horizontal and 2 vertical images were selected from the 12 multi-view images to form a house sequence multi-view image. House sequence multi-view image matching and SFM reconstruction were then performed. This included extracting feature points from the house sequence multi-view images using the SIFT feature operator; matching the house multi-view images using support line projection and affine invariance constraints to obtain a set of matching points for the house sequence images; and performing aerial triangulation and sparse point cloud generation on the house sequence image matching point set using SFM to recover the camera pose of each frame of the panoramic image and generate a 3D sparse point cloud of the house. First, based on the 3D sparse point cloud of the house, a dense 3D point cloud of the house was generated using PMVS; second, based on the dense 3D point cloud of the house, an irregular triangular mesh of the house was generated; finally, the house mesh patches were traversed, normal vectors were calculated, and the most suitable texture blocks were searched in the corresponding multi-view images and mapped onto the mesh patches to obtain a 3D model of the house with realistic texture.

[0099] Step 3: Construct a deep learning network to determine the missing data in the point cloud.

[0100] 3.1 First, the point cloud is smoothed by surface fitting to reduce or fill the model and reduce its deformation.

[0101] 3.2 The input point cloud is uniformly sampled and the point cloud is gridded into a 3*3*3 block structure.

[0102] 3.3 Construct an encoder and input it into the processing unit in the order of point cloud blocks. Each unit uses a PointNet network to perform initial processing on the input point cloud to extract local feature vectors of the point cloud.

[0103] 3.4 Construct a decoder by concatenating the vectors obtained from each point cloud patch, constructing a two-layer network, and performing operations on the decoded 1024*27 feature matrix. After passing through average pooling and the Sigmoid activation function, calculate whether there are missing points in the point cloud patch.

[0104]

[0105] 3.5 Calculate the loss function, compare the predicted results with the actual results using the cross-entropy formula, calculate the loss value, and use the gradient descent rule to backpropagate the network, adjust the parameter values, and obtain the optimal result.

[0106]

[0107] Step 4: First, using the ShapeNet dataset, the shape and skeleton of the point cloud plane are predicted based on the input image and sampling points on the surface of the 3D model mesh. A matching algorithm is used to obtain images of the corresponding viewpoints, which are then copied and aligned. These images are mapped one-to-one with the images in the dataset to obtain the optimal mapped image. The point cloud is then reconstructed based on the mapped image. Specifically, an encoder with the help of an attention mechanism is used to focus on specific trunk, branch structures, and details in the image to obtain image features perceived by the attention mechanism. The encoder maps the image into space and extracts features. Then, a decoder built from deconvs is used to predict the point cloud from the image features, finally obtaining a new point cloud plane. The skeleton structure of this point cloud plane is then solved and compared with the skeleton structure of the input image to ensure the correctness of the result, thus obtaining the completed point cloud plane.

[0108] Step 5: Integrate the 360° panoramic view, floor plan, and 3D model of the house into a unified 3D system for the house's real-world environment. Using the vector room floor plan data generated in Step 4, the elevation histogram method is used to obtain the floor and ceiling elevation information of each room from the point cloud. The elevation histogram shows two peaks in the point cloud, representing the floor and ceiling elevations of the rooms, respectively. The Delaunay triangulation method is used to polygonize the ceiling, walls, and floor of each room, constructing the final 3D room model. The constructed 3D room model is output as a vector mesh. The coordinates of the camera center are solved using the bundle method, and the camera center is projected onto a horizontal plane, with labels set at the projection points. Using the camera center point and the corresponding label positions as references, the panoramic view, floor plan, and 3D model of the house are unified to the same coordinate system.

[0109] The following is a specific case.

[0110] 1. Add panoramic photos of the house to build the house model, such as... Figure 7 As shown, an example of a panoramic photo is as follows: Figure 8 As shown, room 1 is as follows Figures 9a to 9c As shown. Room 2 is as follows. Figures 10a to 10d As shown.

[0111] During this process, the system will convert the uploaded panoramic photos of the house into relevant point cloud data based on the image multiview projection model, SIFT feature extraction, and aerial triangulation method.

[0112] 2. Extract the house floor plan using the point cloud data from the previous step, such as... Figure 11 As shown, the generated plan view is as follows Figure 12 , Figures 13a to 13b As shown.

[0113] During this process:

[0114] 1) It can use point cloud data to match the original image, construct the structural map of the house and map it onto the plane;

[0115] 2) By using the previously trained missing point cloud neural network model (Build-Net), determine whether the point cloud data is complete and whether there are any missing data.

[0116] 3. Reconstruct a 3D model of a house from point clouds, such as... Figure 14 As shown in the image, the final model construction result is displayed as follows. Figure 15 As shown. It also enables multi-angle observation of the house and simple data measurement, such as... Figures 16a to 16d As shown.

[0117] During this process:

[0118] For missing data, the missing house structure is predicted using the house dataset, and the missing 3D point cloud model with the highest similarity is used to complete the missing data.

[0119] After obtaining complete data, the Delaunay triangulation method and room elevation data are used to reconstruct the 3D model of the house. Finally, the data is output in the form of a mesh, realizing a unified display of the house's panoramic view, floor plan, and 3D model.

[0120] The present invention also provides an apparatus, comprising:

[0121] Module 1: Acquire 360-degree panoramic data of the building and preprocess it to obtain a sequence of multi-view images;

[0122] The second module: After matching and reconstructing the sequence of multi-view images, a three-dimensional sparse point cloud is generated;

[0123] The third module involves partitioning the 3D sparse point cloud, determining the point cloud missingness of each partition based on an encoding-decoding neural network, and then reconstructing the 3D model after filling in the missing point cloud of each partition according to the determination results.

[0124] The present invention also provides a storage medium storing an executable program that, when executed, can run the entire aforementioned method steps.

[0125] It should be understood that any parts not described in detail in this specification belong to the prior art.

[0126] It should be understood that the specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for reconstructing a 3D house model based on missing point cloud data, characterized in that, Includes the following steps: After acquiring 360-degree panoramic data of the house and preprocessing it, a sequence of multi-view images is obtained. A three-dimensional sparse point cloud is generated by matching and reconstructing a sequence of multi-view images; The 3D sparse point cloud is partitioned, and the missing point cloud of each partition is determined based on an encoder-decoder neural network. After filling in the missing point cloud of each partition according to the determination result, the 3D model is reconstructed. Specifically, this includes: Construct an encoder-decoder neural network, where the encoder is used to extract feature vectors from the point cloud and the decoder is used to calculate the missing value results; Using a trained encoder-decoder neural network, the partitioned point cloud is used as input parameters to calculate the point cloud missingness result; Using the ShapeNet dataset, we trained and tested a deep learning completion algorithm, and then segmented the planar point cloud obtained from the panoramic image. Based on the sampling points of the input image and the surface of the 3D model mesh as Truth Points, the images of the corresponding viewpoints in the ShapeNet dataset are copied and aligned to construct a mapping between each input image and the corresponding image in the dataset. Set a maximum threshold, select images from the dataset that are below the threshold and have the best performance, and construct a planar point cloud based on the dataset images to fill in the missing planes in the point cloud data. Using a trained neural network, the partitioned point cloud is used as input parameters to obtain the point cloud missingness judgment; The method further includes: when using the partitioned point cloud as input parameters, taking the segmented point cloud blocks as a sequence as input, and calculating the correlation between adjacent point clouds: The input point cloud is uniformly sampled, and the point cloud is meshed and divided into blocks. The structure; The block point cloud is input into the encoder structure in sequence so that the previous unit input point cloud block and the next unit input point cloud block have overall connectivity. Reconstructing point clouds from mapped images involves mapping the shape and skeleton of the point cloud plane one-to-one with the images in the dataset, and using data that meets the mapping accuracy threshold to select the structure with the highest mapping matching degree to complete the point cloud. The encoder-decoder neural network is based on the BuildNet neural network, specifically: Construct an encoder: used to extract point cloud feature vectors. The input is a block sequence of point clouds. Each unit of the encoder uses a PointNet network to perform initial processing on the input point cloud and outputs local point cloud feature vectors. Construct a decoder: used to determine whether a point cloud has missing parameters. Input the point cloud feature vector, and process the decoded data. The feature matrix is ​​subjected to average pooling and sigmoid activation operations to output the missing value of the point cloud blocks, and the weights are updated by backpropagation using the loss function; the formula is as follows: Constructing the loss function: The cross-entropy formula is used to compare the predicted results with the actual results, calculate the loss value, and use the gradient descent rule to backpropagate the network, adjust the parameter values, and obtain the optimal result. Loss function: 。 2. The method as described in claim 1, characterized in that: The coordinates of the camera center are determined by the bundle method, and the camera center is projected onto a horizontal plane. Labels are then placed at the projection points. Using the center point of the photograph and the corresponding label position as a reference, the panoramic view, floor plan, and 3D model of the house are unified into the same coordinate system.

3. The method as described in claim 1, characterized in that: The 360-degree panoramic data of the house is obtained through user uploads or mobile phone photography, and the panoramic data of the house includes panoramic images and video streams; if it is a video stream, it is extracted frame by frame to form image data.

4. The method as described in claim 1, characterized in that: When performing multi-view projection correction on 360-degree panoramic images, an equirectangular projection spherical panoramic model is used to transform the original input 360-degree panoramic image into a three-dimensional panoramic spherical space. The formula is as follows: ； In the formula: These are the coordinates of the image points in the original panoramic image. These are the coordinates of the corresponding point in the 3D panoramic sphere. h is the length of the original panoramic image, and h is the width of the original panoramic image. and It represents the horizontal and vertical rotation angles of the sphere's spatial azimuth and the image capture azimuth.

5. The method as described in claim 1, characterized in that: SFM reconstruction generates a 3D sparse point cloud, including: The SIFT feature operator is used to extract feature points from the multi-view images of the house sequence. Specifically, several horizontal and vertical images are selected to form the multi-view images of the house sequence. A Gaussian feature pyramid is constructed using the multi-view images of the house sequence, and finally the SIFT feature operator is used to extract feature points. The image matching method of support line projection and affine invariance constraint is used to match multi-view images of buildings. Specifically, the image matching method of support line projection and affine invariance constraint is used to obtain the image matching point set, restore the pose of the panoramic image camera, and reconstruct a three-dimensional sparse point cloud through multi-view images.

6. The method as described in claim 1, characterized in that: The point cloud blocks are obtained by uniformly sampling and segmenting the point cloud using a region growing algorithm and then fitting the surface.

7. The method according to claim 4, characterized in that: Each unit of the encoder uses a PointNet network to initially process the input point cloud, extracting local feature vectors from the point cloud; the decoder constructs a two-layer network to process the decoded point cloud. The characteristic matrix is ​​used for calculation.

8. The method as described in claim 5, characterized in that: The input sampling points are used as feature matching points. The images are copied and aligned with the corresponding viewpoints in the ShapeNet dataset. The optimal matching result is selected from the data that meets the threshold and used to complete the missing point cloud.

9. The method as described in claim 5, characterized in that: To verify the completion effect, a mathematical model was constructed based on the connectivity of the point cloud, and its texture framework structure was extracted. This model was then compared with the skeleton structure of the previously input image to verify the rationality of the completion.

10. The method as described in claim 5, characterized in that: The final 3D model of the room is constructed, and the panoramic view, floor plan, and 3D model of the house are unified to the same coordinate system. Specifically, this includes: The elevation information of the floor and ceiling of each room is obtained. The number of points in the elevation histogram is presented as two peaks, which represent the elevation of the floor and ceiling of the room, respectively. The Delaunay triangulation method is used to triangulate the ceiling, walls and floor of each room into polygons, construct the final 3D model of the room, and finally output the constructed 3D model of the room in the form of a vector mesh. The coordinates of the photographic center are determined using the bundle method, and the panoramic view, floor plan, and 3D model of the house are unified to the same coordinates by using the photographic center point and the corresponding label position as references.

11. The method as described in claim 1, characterized in that: To verify the completion effect, specifically: A mathematical model is constructed based on the connectivity relationships of point clouds, and its texture framework structure is extracted. The skeleton structure extracted from the input image is compared and verified with the frame structure of the completed point cloud; The model similarity index is evaluated. Under the condition of the same azimuth angle and scale normalization, the local matching accuracy is calculated. If the proportion of incorrectly completed point cloud structures is less than 5% of the total, the completed 3D structure is considered to meet the accuracy requirements.

12. A device for reconstructing a 3D model of a house based on missing point cloud data, employing the method described in claim 1, characterized in that: include Module 1: Acquire 360-degree panoramic data of the building and preprocess it to obtain a sequence of multi-view images; The second module: After matching and reconstructing the sequence of multi-view images, a three-dimensional sparse point cloud is generated; The third module involves partitioning the 3D sparse point cloud, determining the point cloud missingness of each partition based on an encoding-decoding neural network, and then reconstructing the 3D model after filling in the missing point cloud of each partition according to the determination results.

13. A storage medium, characterized in that: The device stores an executable program that, when executed by the device, can perform the method described in any one of claims 1 to 11.

Images