Building three-dimensional reconstruction and detail completion system and method based on three-dimensional Gaussian sputtering

Abstract

The invention provides a building three-dimensional reconstruction and detail completion system and method based on three-dimensional Gaussian sputtering, and the system comprises an unmanned aerial vehicle image collection module which is used for shooting a glass curtain wall facade from multiple angles, and obtaining a multi-view high-resolution image; the three-dimensional Gaussian sputtering modeling module is used for generating a sparse three-dimensional point cloud from the multi-view image and generating a three-dimensional model based on a Gaussian sputtering technology; the image completion and detail enhancement module is used for completing details of a transparent region and a reflection region which are missing in the three-dimensional model; and the three-dimensional model optimization and density control module is used for optimizing the density distribution of the three-dimensional model and ensuring the consistency and accuracy of the model under all visual angles. According to the method, the advanced image completion technology and the three-dimensional Gaussian sputtering technology are combined, the bottleneck of the traditional three-dimensional modeling technology in processing high-reflectivity and transparent materials is solved, high-precision three-dimensional reconstruction and detail completion of the glass curtain wall are realized, and the method has important practical application value.

Description

Technical Field

[0001] The present invention relates to the technical field of building three-dimensional reconstruction, and in particular to a system and method for building three-dimensional reconstruction and detail completion based on three-dimensional Gaussian sputtering. Background Art

[0002] 3D modeling of glass curtain wall buildings plays a huge role in many usage scenarios, including industrial buildings and gaming and entertainment industries. It is especially important to obtain an accurate and realistic 3D model of the building when drones are inspecting curtain wall defects. Traditional 3D reconstruction technology is mostly applied to opaque objects, while glass curtain walls have complex optical properties, such as high transparency and reflectivity, which poses a great challenge to 3D modeling.

[0003] Deficiencies of existing technology: 1. Automated 3D modeling is incomplete: Existing automated modeling technologies often lose surface details when dealing with highly reflective and transparent materials and are unable to accurately capture the full picture of the glass curtain wall.

[0004] These techniques may have sparse point clouds or missing details in certain areas, resulting in gaps or distortions in the model.

[0005] 2. Difficulty in completing details: Even if the general shape of the glass curtain wall is captured through multi-view modeling, the interference of transparent areas and reflected light often makes the details unclear.

[0006] The existing technology lacks an effective automatic completion mechanism and is unable to generate realistic details in these areas, especially in the completion of details in transparent and reflective areas.

[0007] 3. Problems in handling highly reflective and transparent materials: Traditional 3D reconstruction technology is easily affected by ambient light when dealing with highly reflective and transparent materials, resulting in inaccurate model geometry and surface texture.

[0008] Due to the particularity of glass materials, image data of transparent areas and strongly reflective areas are often difficult to obtain through direct shooting, resulting in missing or incomplete details of these areas in the 3D model.

[0009] 4. Limitations of multi-view modeling: The single-view modeling method has difficulty capturing all the details of the glass curtain wall, especially under complex lighting conditions.

[0010] Although multi-view modeling can improve this problem, there are still cases of missing details and model distortion, especially in edges and complex reflection areas.

[0011] Therefore, the prior art has deficiencies and needs further improvement. Summary of the invention

[0012] In view of the problems existing in the prior art, the present invention provides a system and method for three-dimensional reconstruction and detail completion of buildings based on three-dimensional Gaussian sputtering.

[0013] To achieve the above object, the specific scheme of the present invention is as follows: The present invention provides a system for three-dimensional reconstruction and detail completion of buildings based on three-dimensional Gaussian sputtering, the system comprising: The drone image acquisition module is used to shoot the glass curtain wall facade from multiple angles and obtain multi-view high-resolution images; 3D Gaussian sputtering modeling module, used to generate sparse 3D point clouds from multi-view images and generate 3D models based on Gaussian sputtering technology; Image completion and detail enhancement module, used to complete the details of missing transparent and reflective areas in 3D models; The 3D model optimization and density control module is used to optimize the density distribution of the 3D model to ensure the consistency and accuracy of the model at all viewing angles.

[0014] Furthermore, the drone image acquisition module can fly around the building to ensure complete image coverage from multiple angles, and the flight route and shooting angle of the drone can be flexibly adjusted according to the characteristics of the building.

[0015] Furthermore, the sparse 3D point cloud generation process in the 3D Gaussian sputtering modeling module restores the 3D structure of the object through camera pose and image matching points, and the generation process uses the camera projection matrix, the camera intrinsic parameter matrix, the camera rotation matrix and the displacement vector.

[0016] Furthermore, the image completion and detail enhancement module adopts a generative adversarial network (GAN) to generate more realistic image content through adversarial training of the generator and the discriminator, which is particularly suitable for detail completion in transparent and reflective areas.

[0017] Furthermore, the generator in the image completion and detail enhancement module adopts a fully convolutional network (FCN) structure to complete the missing image area; the discriminator is used to distinguish the completed image from the real image to ensure that the generated image has a high sense of reality.

[0018] The present invention also provides a method for three-dimensional reconstruction and detail completion of a building based on three-dimensional Gaussian sputtering. Based on the above system, the method comprises the following steps: S1, uses a drone to fly along a predetermined route to collect multi-view high-resolution images of the glass curtain wall; S2, generates sparse 3D point clouds from multi-view images using structured motion (SfM) and generates 3D models based on Gaussian sputtering technology; S3, uses generative adversarial network (GAN) technology to complete the missing details of transparent and reflective areas in the 3D model; S4, optimizes and density controls the 3D model to ensure the consistency and accuracy of the model in all viewing angles; S5, displaying the optimized three-dimensional model.

[0019] Furthermore, in step S1, the flight path and shooting angle of the drone are flexibly adjusted according to the characteristics of the building to ensure complete image coverage from multiple angles.

[0020] Furthermore, in step S2, the sparse 3D point cloud generation process restores the 3D structure of the object through camera pose and image matching points, using the camera projection matrix, the camera intrinsic parameter matrix, the camera rotation matrix and the displacement vector.

[0021] Furthermore, in step S3, the generator and discriminator in the generative adversarial network (GAN) are used for adversarial training to generate more realistic image content, which is particularly suitable for detail completion of transparent and reflective areas; The generator uses a fully convolutional network (FCN) structure to complete the missing image areas; the discriminator is used to distinguish the completed image from the real image to ensure that the generated image has a high sense of reality.

[0022] Furthermore, in step S4, the three-dimensional model is optimized and density controlled, specifically including adaptive density adjustment and multi-view fusion, to ensure the consistency and accuracy of the model under all viewing angles.

[0023] The technical solution of the present invention has the following beneficial effects: 1. High-precision 3D reconstruction: Through multi-angle shooting of drones and structured motion (SfM) technology, a sparse three-dimensional point cloud is generated, which can accurately capture the basic geometric structure of the glass curtain wall.

[0024] Based on Gaussian sputtering technology, each point cloud point is represented as a three-dimensional Gaussian body, and a high-precision three-dimensional model is constructed by optimizing the position, shape and color parameters of these Gaussian bodies.

[0025] Especially when dealing with highly reflective and transparent materials, it can effectively avoid the problems of sparse point clouds and missing details, ensuring the accuracy of the model from all viewing angles.

[0026] 2. Detail completion and enhancement: The Generative Adversarial Network (GAN) technology is used to generate more realistic image content through adversarial training of the generator and the discriminator, which is particularly suitable for detail completion in transparent and reflective areas.

[0027] The generator adopts a fully convolutional network (FCN) structure, which can complete the missing image areas and ensure that the generated images have a high sense of reality.

[0028] The dilated convolution technique can utilize a wider range of contextual information and more effectively capture the overall structure and edge details of the glass curtain wall.

[0029] 3. Adaptive density adjustment: The distribution and density of the 3D Gaussian volume are optimized through an adaptive density adjustment algorithm, ensuring that the density of the Gaussian volume is increased in complex areas (such as transparent and reflective parts) to retain more details.

[0030] Reduce density in simple areas to improve modeling efficiency and ensure the consistency and accuracy of the model from all viewing angles.

[0031] 4. Multi-perspective fusion: By integrating the 3D models generated from different perspectives, we can avoid the loss of details or distortion that may be caused by single-perspective modeling, and ensure that the curtain wall can be accurately presented from all angles.

[0032] The weighted fusion results of each perspective model are used to ensure the consistency and accuracy of the model, especially in complex lighting conditions.

[0033] 5. Broad application prospects: The generated 3D model not only accurately reconstructs the geometric structure of the curtain wall, but also shows the detailed surface texture in transparent and reflective areas.

[0034] This 3D model can be used in building maintenance and defect identification systems to help users fully understand the appearance and condition of the curtain wall and detect potential problems in a timely manner.

[0035] It is particularly suitable for the modeling and maintenance of large-scale glass curtain walls, improving the efficiency and accuracy of building exterior wall inspection and maintenance.

[0036] 6.Flexibility and adaptability: The flight route and shooting angle of the drone can be flexibly adjusted according to the characteristics of the building, ensuring comprehensive coverage and high-resolution image acquisition.

[0037] Different deep neural network models can be selected according to the characteristics of the glass curtain walls of different buildings to meet different appearance detail completion requirements.

[0038] The model's optimization strategy can adjust the parameters of density control to accommodate different types of glass surfaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a schematic diagram of the drone of the present invention flying around a building to collect images; Figure 2 is a schematic diagram of three-dimensional Gaussian sputtering modeling of the present invention; Figure 3 This is an image completion and detail enhancement diagram of the present invention. Figure 1 ; Figure 4 This is an image completion and detail enhancement diagram of the present invention. Figure 2 ; Figure 5 This is an image completion and detail enhancement diagram of the present invention. Figure 3 ; Figure 6 The results of the present invention are shown and the feedback is shown. Figure 1 ; Figure 7 The results of the present invention are shown and the feedback is shown. Figure 2 ; Figure 8 is a system block diagram of the present invention; Fig. 9 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0040] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments; it can be understood that the specific embodiments described herein are only used to explain the present invention rather than to limit the present invention; it should also be noted that, for ease of description, only the parts related to the present invention rather than all are shown in the accompanying drawings.

[0041] Combination Figure 1-Figure 9 As shown, the present invention provides a system for 3D reconstruction and detail completion of buildings based on 3D Gaussian sputtering, the system comprising: The drone image acquisition module is used to shoot the glass curtain wall facade from multiple angles to obtain multi-view high-resolution images; 3D Gaussian sputtering modeling module, used to generate sparse 3D point clouds from multi-view images and generate 3D models based on Gaussian sputtering technology; Image completion and detail enhancement module, used to complete the details of missing transparent and reflective areas in 3D models; The 3D model optimization and density control module is used to optimize the density distribution of the 3D model to ensure the consistency and accuracy of the model at all viewing angles.

[0042] The drone image acquisition module can fly around the building to ensure complete image coverage from multiple angles, and the drone's flight path and shooting angle can be flexibly adjusted according to the characteristics of the building.

[0043] The sparse 3D point cloud generation process in the 3D Gaussian sputtering modeling module restores the 3D structure of the object through camera pose and image matching points. The generation process uses the camera projection matrix, the camera's intrinsic parameter matrix, the camera's rotation matrix and the displacement vector.

[0044] The image completion and detail enhancement module adopts a generative adversarial network (GAN) to generate more realistic image content through adversarial training of a generator and a discriminator, and is particularly suitable for detail completion in transparent and reflective areas.

[0045] The generator in the image completion and detail enhancement module adopts a fully convolutional network (FCN) structure to complete the missing image area; the discriminator is used to distinguish the completed image from the real image to ensure that the generated image has a high sense of reality.

[0046] The present invention also provides a method for three-dimensional reconstruction and detail completion of a building based on three-dimensional Gaussian sputtering. Based on the above system, the method comprises the following steps: S1, uses a drone to fly along a predetermined route to collect multi-view high-resolution images of the glass curtain wall; S2, generates sparse 3D point clouds from multi-view images using structured motion (SfM) and generates 3D models based on Gaussian sputtering technology; S3, uses generative adversarial network (GAN) technology to complete the missing details of transparent and reflective areas in the 3D model; S4, optimizes and density controls the 3D model to ensure the consistency and accuracy of the model in all viewing angles; S5, displaying the optimized three-dimensional model.

[0047] In step S1, the flight path and shooting angle of the drone are flexibly adjusted according to the characteristics of the building to ensure complete image coverage from multiple angles.

[0048] In step S2, the sparse 3D point cloud generation process restores the 3D structure of the object through camera pose and image matching points, using the camera projection matrix, the camera intrinsic parameter matrix, the camera rotation matrix and the displacement vector.

[0049] In step S3, the generator and discriminator in the generative adversarial network (GAN) are used for adversarial training to generate more realistic image content, which is particularly suitable for detail completion in transparent and reflective areas; The generator uses a fully convolutional network (FCN) structure to complete the missing image areas; the discriminator is used to distinguish the completed image from the real image to ensure that the generated image has a high sense of reality.

[0050] In step S4, the three-dimensional model is optimized and density controlled, including adaptive density adjustment and multi-view fusion, to ensure the consistency and accuracy of the model in all viewing angles.

[0051] Working principle: The present invention proposes a system and method for 3D reconstruction and detail completion of buildings based on 3D Gaussian sputtering, which can be divided into the following main steps: 1. UAV image acquisition Get multi-view, high-resolution images of your glass facade, ensuring full coverage from multiple angles.

[0052] Use a drone to fly along a predetermined route and take multi-angle shots around the building.

[0053] The image data collected by the drone will serve as the basis for subsequent three-dimensional reconstruction.

[0054] 2. 3D Gaussian sputtering modeling Generate a 3D point cloud of the glass curtain wall and optimize the model details.

[0055] SfM point cloud generation: Generate sparse 3D point clouds from multi-view images using structured motion (SfM) techniques.

[0056] SfM recovers the 3D structure of an object through camera pose and image matching points. The specific steps include: Camera projection matrix P = intrinsic parameter matrix K × (rotation matrix R + displacement vector t).

[0057] A sparse 3D point cloud is generated through matching points extracted from multiple viewpoint images, providing the basic geometric structure of the glass curtain wall.

[0058] Three-dimensional Gaussian representation: Represent each point cloud point as a three-dimensional Gaussian volume in the form of: The center position μ of the Gaussian body and the covariance matrix Σ describe the shape and size of the Gaussian body.

[0059] These Gaussians have location, shape (represented by the covariance matrix), and color parameters.

[0060] Parameter optimization: Based on the initial 3D model, the Gaussian volumes are optimized by adjusting their size and density.

[0061] Minimize the difference between the generated model and the real image through optimization algorithms such as gradient descent: The loss function L is used to measure the difference between the generated model and the real image.

[0062] The result is a model that accurately represents the building structure from all viewing angles, with particular attention paid to the refinement of curtain wall edges and complex reflective areas.

[0063] 3. Image completion and detail enhancement Fill in missing details in transparent and highly reflective areas.

[0064] Image completion solution: Image completion technology using generative adversarial networks (GANs).

[0065] GAN generates more realistic image content through adversarial training of the generator and the discriminator.

[0066] Generator: The generator adopts a fully convolutional network (FCN) structure to complete the missing image areas.

[0067] The input of the generator is a glass curtain wall image containing missing areas, and the output is a complete image after completion.

[0068] The generator extracts image features of different scales through multiple layers of convolution and gradually fills in the missing areas.

[0069] Dilated Convolution technology is used to extract more extensive contextual information.

[0070] Discriminator: The discriminator is used to distinguish the completed image from the real image, ensuring that the generated image has a high sense of reality.

[0071] The input of the discriminator is the real image and the generated image, and the output is the probability of the image being real or generated.

[0072] Generative Adversarial Training: The generator and discriminator are optimized through adversarial training, where the generator tries to generate image content that is indistinguishable from real images, and the discriminator tries to recognize the difference between real images and generated images.

[0073] The goal of adversarial training is to minimize the loss between the generator and the discriminator.

[0074] The loss function consists of generative adversarial loss and pixel-level reconstruction loss to ensure the authenticity and structural consistency of the generated images.

[0075] 4. 3D model optimization and density control Optimize the distribution and density of 3D Gaussian bodies to improve modeling efficiency.

[0076] Adaptive Density Adjustment: The distribution and density of three-dimensional Gaussian volumes are optimized through an adaptive density adjustment algorithm.

[0077] For complex areas (such as transparent and reflective parts), increase the density of the Gaussian volume to retain more details; reduce the density in simple areas to improve modeling efficiency.

[0078] The optimization strategy determines the density distribution of the Gaussian volume by maximizing the similarity between the reconstructed model and the original image.

[0079] Multi-view fusion: By fusing the 3D models generated from different perspectives, the missing details or distortion that may be caused by single-perspective modeling can be avoided.

[0080] The weighted fusion results of each viewpoint model are used to ensure the consistency and accuracy of the model at all angles.

[0081] 5. Results presentation and feedback Generate a detailed 3D model of the glass curtain wall and apply it to the actual scene.

[0082] 3D model generation: After the joint processing of 3D Gaussian sputtering and image completion, a 3D model of the glass curtain wall containing details is generated.

[0083] The model not only accurately reconstructs the geometric structure of the curtain wall, but also shows detailed surface textures in transparent and reflective areas.

[0084] Feedback and Application: This 3D model can be used in building maintenance and defect identification systems to help users fully understand the appearance and condition of the curtain wall and detect potential problems in a timely manner.

[0085] By combining UAV multi-angle image acquisition, SfM point cloud generation, 3D Gaussian sputtering modeling, image completion and detail enhancement, 3D model optimization and density control, high-precision 3D reconstruction and detail completion of glass curtain walls are achieved. This method performs well in processing highly reflective and transparent materials and has a wide range of practical application value.

[0086] Embodiment 1: The present invention is suitable for scenarios where three-dimensional reconstruction and detail completion of glass curtain walls are required; it is mainly used for the maintenance and inspection of building exterior walls. The glass curtain wall facades are photographed from multiple angles by drones, and image completion and three-dimensional Gaussian sputtering technology are used to generate a three-dimensional model with rich details.

[0087] Technical process: 1. Drone image acquisition: Use drones to fly along a predetermined route to collect multi-view high-resolution images of the glass curtain wall; drones can fly around the building to ensure complete image coverage from multiple angles.

[0088] 2. Three-dimensional Gaussian sputtering modeling: SfM point cloud generation: Generate sparse 3D point clouds from multi-view images using structured motion (SfM) technology. SfM restores the 3D structure of objects through camera poses and image matching points. The specific steps include:

[0089] in, is the camera projection matrix, is the camera’s intrinsic parameter matrix, is the camera's rotation matrix and displacement vector The SfM method generates a sparse 3D point cloud through matching points extracted from multiple viewpoint images. The point cloud provides the basic geometric structure of the glass curtain wall, but due to the high transparency and reflectivity of glass, some areas may have sparse point clouds or missing details.

[0090] Three-dimensional Gaussian representation: Based on Gaussian sputtering technology, each point cloud point is represented as a three-dimensional Gaussian body in the form of:

[0091] in, is the center position of the Gaussian body, is the covariance matrix, which is used to describe the shape and size of Gaussian bodies. These Gaussian bodies have position, shape (represented by the covariance matrix) and color parameters. By optimizing these parameters, the overall structure of the 3D model can be accurately constructed.

[0092] Parameter optimization: Based on the initial 3D model, the Gaussian volumes are optimized, their size and density are adjusted, and the difference with the image is minimized through optimization algorithms such as gradient descent:

[0093] in, is the pixel value of the image at point, is the color value obtained by Gaussian rendering, the loss function It is used to measure the difference between the generated model and the real image. Ultimately, it ensures that the model can accurately present the building structure from all viewing angles, especially the refinement of the curtain wall edges and complex reflection areas.

[0094] 3. Image completion and detail enhancement: Due to the particularity of glass materials, image data of transparent areas and strongly reflective areas are often difficult to obtain through direct shooting, resulting in missing or incomplete details of these areas in the 3D model.

[0095] Image completion solution: In order to complete these missing areas, the image completion technology of Generative Adversarial Network (GAN) is used. GAN generates more realistic image content through adversarial training of generator and discriminator, which is particularly suitable for detail completion of transparent and reflective areas. The network architecture includes: Generator: The generator adopts the Fully Convolutional Network (FCN) structure to complete the missing image area. The input of the generator is the glass curtain wall image containing the missing area. , the output is the completed image. Discriminator: The discriminator is used to distinguish the completed image from the real image, ensuring that the generated image has a high sense of reality. The input of the discriminator is the real image and the generated image, and the output is the probability that the image is real or generated. Convolution operation of the generator: The generator extracts local features in the image through the convolution layer and generates a feature map , the formula is as follows:

[0096] in, is the convolution kernel, represents the convolution operation, is a non-linear activation function (such as ReLU), is the bias term. The generator extracts image features of different scales through multiple layers of convolution and gradually fills in the missing areas. Dilated convolution extracts context information: In order to utilize more extensive context information in the completion process, the generator network uses dilated convolution:

[0097] : Output the value of the sequence at position i; Indicates the sum of k; : Input sequence x at position The value of : The weight of the convolution kernel at position k; in, is the expansion factor, which enables the network to expand the receptive field without increasing parameters, thereby more effectively capturing the overall structure and edge details of the glass curtain wall. Generative adversarial training: The generator and discriminator are optimized through adversarial training. The generator attempts to generate image content that is indistinguishable from real images, while the discriminator attempts to identify the difference between real images and generated images. The goal of adversarial training is to minimize the loss between the generator and the discriminator. Loss function: The loss function of GAN consists of generative adversarial loss and pixel-level reconstruction loss to ensure the authenticity and structural consistency of the generated images:

[0098] Reconstruction losses : The mean square error (MSE) loss is usually used to ensure that the generated image is similar to the real image at the pixel level:

[0099] in, is a real image, is the image generated by the generator. Adversarial loss : The adversarial loss between the generator and the discriminator is used to improve the authenticity of the generated image and make the completed area more realistic. The optimization goal of GAN is the minimum-maximum problem, and the formula is as follows:

[0100] in, represents the probability that the discriminator judges the real image, Represents the probability of the discriminator judging the generated image. During the adversarial training process, the generator is continuously optimized to "fool" the discriminator, making the generated image indistinguishable from the real image in appearance and details. A hyperparameter to balance the reconstruction loss and the adversarial loss, used to adjust the relative weights of the two.

[0101] 4. 3D model optimization and density control: Adaptive density adjustment: To optimize the distribution and density of the 3D Gaussian volume, an adaptive density adjustment algorithm is used. For complex areas in the 3D model (such as transparent and reflective parts), the system increases the density of the Gaussian volume to retain more details. In simple areas, the density is reduced to improve modeling efficiency. The optimization strategy determines the density distribution of the Gaussian volume by maximizing the similarity between the reconstructed model and the original image:

[0102] in, is the number of Gaussians The optimized gradient of is used to adjust the Gaussian volume density in different regions according to the gradient.

[0103] Multi-view fusion: By fusing the 3D models generated from different viewpoints, we can avoid the loss of details or distortion that may be caused by modeling from a single viewpoint, and ensure that the curtain wall can be accurately presented from all viewpoints. The following formula is used to fuse the geometric information from different viewpoints:

[0104] in It is The model generated from the perspective is the importance weight of the perspective, and the final generated model It is the weighted fusion result of each view model.

[0105] 5. Results presentation and feedback: 3D model generation: After the joint processing of 3D Gaussian sputtering and image completion, a detailed 3D model of the glass curtain wall is generated. This model can not only accurately reconstruct the geometric structure of the curtain wall, but also show the detailed surface texture in the transparent and reflective areas.

[0106] Feedback and Application: The 3D model can be used in building maintenance and defect identification systems to help users fully understand the appearance and condition of the curtain wall and detect potential problems in a timely manner.

[0107] By combining 3D Gaussian sputtering technology and image completion technology, the bottleneck encountered by traditional 3D modeling technology in dealing with highly reflective and transparent materials is solved. Through the optimization of 3D Gaussian bodies and adaptive density control, this technology can carefully complete the missing areas in the 3D model while ensuring the consistency of the overall structure. This solution can still accurately capture the texture and details of the glass surface under complex lighting conditions, and is particularly suitable for the modeling and maintenance of large-scale glass curtain walls of buildings.

[0108] 5. Key technical points and innovation Adaptive optimization of the 3D Gaussian sputtering model: By adaptively adjusting the density of the Gaussian body, the model can automatically increase the sampling density and improve the detail retention capability in transparent areas and areas with complex details of the glass curtain wall.

[0109] Context-based image completion technology: Using dilated convolution and global-local consistency networks, it automatically completes details in highly reflective and transparent areas, avoiding the loss of details that cannot be captured by traditional image acquisition methods.

[0110] Multi-view fusion and model consistency verification: Through multi-view fusion technology, ensure that the generated 3D model remains consistent at all angles, avoiding errors caused by a single perspective.

[0111] 6. Technical features Flexibility of drone photography: The flight route and shooting angle of the drone can be flexibly adjusted according to the characteristics of the building to ensure comprehensive coverage and high-resolution image acquisition.

[0112] Alternative image completion network models: Different deep neural network models can be selected according to the glass curtain wall characteristics of different buildings to meet different appearance detail completion requirements.

[0113] Adjustment of model optimization strategy: According to different transparency and reflection characteristics, the optimization strategy of the model can adjust the parameters of density control to adapt to different types of glass surfaces.

[0114] Example 2: Glass curtain wall building maintenance inspection Background: After long-term use, the glass curtain wall of a large commercial complex has many cracks and stains, and needs comprehensive inspection and maintenance. Traditional inspection methods are time-consuming and inefficient, so it is decided to use the method of the present invention to perform three-dimensional reconstruction and detail completion in order to quickly and accurately identify and evaluate curtain wall problems.

[0115] Implementation steps: 1. UAV image acquisition Using a drone equipped with a high-resolution camera, the team flew around the building along a predetermined route to photograph the glass curtain wall facade from multiple angles.

[0116] The drone flight path design ensures shooting from different altitudes and distances to obtain comprehensive image coverage.

[0117] 2. 3D Gaussian sputtering modeling SfM point cloud generation: Sparse 3D point clouds are generated using SfM technology from the collected multi-view images.

[0118] The 3D structure of the object is restored through camera pose and image matching points.

[0119] Three-dimensional Gaussian representation: Represent each point cloud point as a three-dimensional Gaussian volume in the form of: Center position μ Covariance matrix Σ These Gaussians have position, shape and color parameters.

[0120] Parameter optimization: Based on the initial 3D model, the Gaussian volume is optimized and its size and density are adjusted.

[0121] Through optimization algorithms such as gradient descent, the difference between the generated model and the real image is minimized to ensure the accuracy of the model under all viewing angles.

[0122] 3. Image completion and detail enhancement Image completion solution: Image completion technology using generative adversarial networks (GANs).

[0123] The generator uses a fully convolutional network (FCN) structure to complete the missing image areas.

[0124] The discriminator is used to distinguish the completed image from the real image to ensure that the generated image has a high sense of reality.

[0125] Dilated Convolution technology is used to extract more extensive contextual information.

[0126] Generative Adversarial Training: The generator and discriminator are optimized through adversarial training, where the generator tries to generate image content that is indistinguishable from real images.

[0127] The loss function consists of generative adversarial loss and pixel-level reconstruction loss to ensure the authenticity and structural consistency of the generated images.

[0128] 4. 3D model optimization and density control Adaptive Density Adjustment: The distribution and density of three-dimensional Gaussian volumes are optimized through an adaptive density adjustment algorithm.

[0129] For complex areas, such as transparent and reflective parts, increase the density of the Gaussian to preserve more details.

[0130] Multi-view fusion: By fusing the 3D models generated from different perspectives, the missing details or distortion that may be caused by single-perspective modeling can be avoided.

[0131] 5. Results presentation and feedback 3D model generation: After the joint processing of 3D Gaussian sputtering and image completion, a 3D model of the glass curtain wall containing details is generated.

[0132] The model not only accurately reconstructs the geometric structure of the curtain wall, but also shows detailed surface textures in transparent and reflective areas.

[0133] Feedback and Application: The generated 3D model is imported into the building maintenance and defect identification system to help users fully understand the appearance and condition of the curtain wall.

[0134] The system automatically detects and marks cracks and stains, generates detailed maintenance reports, and guides maintenance personnel to carry out targeted repair work.

[0135] Example 3: Glass curtain wall modeling in virtual reality games Background: A virtual reality game development team is making a game with a modern city as the background, and needs to restore the glass curtain wall of a high-rise building in the game. The traditional modeling method cannot meet the high precision and realism required by the game, so it is decided to use the method of the present invention for three-dimensional reconstruction and detail completion.

[0136] Implementation steps 1. UAV image acquisition Using a drone equipped with a high-resolution camera, fly along the building's facade to capture high-definition images of the glass curtain wall from multiple angles.

[0137] The drone flight path design ensures shooting from different heights and angles to obtain comprehensive image coverage.

[0138] 2. 3D Gaussian sputtering modeling SfM point cloud generation: Sparse 3D point clouds are generated using SfM technology from the collected multi-view images.

[0139] The 3D structure of the object is restored through camera pose and image matching points.

[0140] Three-dimensional Gaussian representation: Represent each point cloud point as a three-dimensional Gaussian volume in the form of: Center position μ Covariance matrix Σ ΣThese Gaussians have position, shape and color parameters.

[0141] Parameter optimization: Based on the initial 3D model, the Gaussian volume is optimized and its size and density are adjusted.

[0142] Through optimization algorithms such as gradient descent, the difference between the generated model and the real image is minimized to ensure the accuracy of the model under all viewing angles.

[0143] 3. Image completion and detail enhancement Image completion solution: Image completion technology using generative adversarial networks (GANs).

[0144] The generator uses a fully convolutional network (FCN) structure to complete the missing image areas.

[0145] The discriminator is used to distinguish the completed image from the real image to ensure that the generated image has a high sense of reality.

[0146] Dilated Convolution technology is used to extract more extensive contextual information.

[0147] Generative Adversarial Training: The generator and discriminator are optimized through adversarial training, where the generator tries to generate image content that is indistinguishable from real images.

[0148] The loss function consists of generative adversarial loss and pixel-level reconstruction loss to ensure the authenticity and structural consistency of the generated images.

[0149] 4. 3D model optimization and density control Adaptive Density Adjustment: The distribution and density of three-dimensional Gaussian volumes are optimized through an adaptive density adjustment algorithm.

[0150] For complex areas, such as transparent and reflective parts, increase the density of the Gaussian to preserve more details.

[0151] Multi-view fusion: By fusing the 3D models generated from different perspectives, the missing details or distortion that may be caused by single-perspective modeling can be avoided.

[0152] 5. Results display and feedback 3D model generation: After the joint processing of 3D Gaussian sputtering and image completion, a 3D model of the glass curtain wall containing details is generated.

[0153] The model not only accurately reconstructs the geometric structure of the curtain wall, but also shows detailed surface textures in transparent and reflective areas.

[0154] Feedback and Application: The generated 3D model is imported into the game engine to ensure the realistic reproduction of the glass curtain wall of the high-rise building in the virtual environment.

[0155] Game developers can further enhance the realism and immersion of the game by adjusting the material and lighting effects of the model.

[0156] Game testers test the models to ensure that they perform as expected under different lighting conditions, providing a high-quality user experience.

[0157] Summary: The embodiments demonstrate the practical application of the present invention in different application scenarios. Whether it is building maintenance inspection or modeling in virtual reality games, the present invention can provide high-precision, high-realistic three-dimensional reconstruction and detail completion of glass curtain walls, significantly improving work efficiency and quality.

[0158] The above description is only a preferred embodiment of the present invention, and does not limit the scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the protection scope of the present invention.

Claims

1. A building 3D reconstruction and detail completion system based on 3D Gaussian sputtering, characterized in that: The system includes: The drone image acquisition module is used to shoot the glass curtain wall facade from multiple angles and obtain multi-view high-resolution images; 3D Gaussian sputtering modeling module, used to generate sparse 3D point clouds from multi-view images and generate 3D models based on Gaussian sputtering technology; Image completion and detail enhancement module, used to complete the details of the missing transparent and reflective areas in the 3D model; The 3D model optimization and density control module is used to optimize the density distribution of the 3D model to ensure the consistency and accuracy of the model at all viewing angles.

2. The system according to claim 1, characterized in that The drone image acquisition module can fly around the building to ensure complete image coverage from multiple angles, and the drone's flight path and shooting angle can be flexibly adjusted according to the characteristics of the building.

3. The system according to claim 1, characterized in that The sparse 3D point cloud generation process in the 3D Gaussian sputtering modeling module restores the 3D structure of the object through camera pose and image matching points. The generation process uses the camera projection matrix, the camera's intrinsic parameter matrix, the camera's rotation matrix and the displacement vector.

4. The system according to claim 1, characterized in that The image completion and detail enhancement module adopts a generative adversarial network to generate more realistic image content through adversarial training of the generator and the discriminator, and is particularly suitable for detail completion in transparent and reflective areas.

5. The system according to claim 4, characterized in that The generator in the image completion and detail enhancement module adopts a fully convolutional network structure to complete the missing image area; the discriminator is used to distinguish the completed image from the real image to ensure that the generated image has a high sense of reality.

6. A method for 3D reconstruction and detail completion of buildings based on 3D Gaussian sputtering, based on the system according to any one of claims 1 to 5, characterized in that: The method comprises the following steps: S1, uses a drone to fly along a predetermined route to collect multi-view high-resolution images of the glass curtain wall; S2, generates sparse 3D point clouds from multi-view images using structured motion and generates 3D models based on Gaussian sputtering technology; S3, uses generative adversarial network technology to complete the details of missing transparent and reflective areas in the 3D model; S4, optimizes and density controls the 3D model to ensure the consistency and accuracy of the model in all viewing angles; S5, displaying the optimized three-dimensional model.

7. The method according to claim 6, characterized in that In step S1, the flight path and shooting angle of the drone are flexibly adjusted according to the characteristics of the building to ensure complete image coverage from multiple angles.

8. The method according to claim 6, characterized in that In step S2, the sparse 3D point cloud generation process restores the 3D structure of the object through camera pose and image matching points, using the camera projection matrix, the camera intrinsic parameter matrix, the camera rotation matrix and the displacement vector.

9. The method according to claim 6, characterized in that In step S3, the generator and discriminator in the generative adversarial network are used for adversarial training to generate more realistic image content, which is particularly suitable for detail completion of transparent and reflective areas; The generator uses a fully convolutional network structure to complete the missing image areas; the discriminator is used to distinguish the completed image from the real image to ensure that the generated image has a high sense of reality.

10. The method according to claim 6, characterized in that In step S4, the three-dimensional model is optimized and density controlled, including adaptive density adjustment and multi-view fusion, to ensure the consistency and accuracy of the model in all viewing angles.

Images