A generative model-based building restoration method and system

Abstract

The application belongs to the field of building repair, and particularly discloses a building repair method and system based on a generative model, which comprises the following steps: obtaining a whole three-dimensional model of a damaged building and a damaged part three-dimensional model, the damaged part three-dimensional model having a range greater than that of the building damaged part and completely containing the building damaged part; randomly cutting a certain volume of a truncated model from the whole three-dimensional model multiple times, performing position coding on the voxels in each truncated model, and randomly cutting a certain volume of a secondary truncated model on the truncated model to construct a training set; training a neural process model based on the training set to obtain a generative model; and obtaining a repair model of the building damaged part based on the damaged part three-dimensional model and the generative model to repair the building damaged part. The application solves the problem of relying on original drawings or three-dimensional models during building repair, avoids the need to redesign construction templates, and improves the building repair efficiency.

Description

Technical Field

[0001] This invention belongs to the field of building restoration, and more specifically, relates to a building restoration method and system based on a generative model. Background Technology

[0002] The building repair industry has a wide scope, including structural repair, decorative repair, and leak repair. In addition to its broad scope, the number of buildings requiring repair will continue to increase over time.

[0003] However, current building restoration technology still faces many difficulties and shortcomings, hindering the development of the industry. Current techniques generally rely on comparing existing buildings with complete architectural design drawings or 3D models to determine the location and shape of damaged parts. However, buildings requiring restoration are often planned, designed, and constructed many years ago, especially ancient buildings or decorative components with complex patterns. During restoration, the original drawings or 3D models are often lost, making it difficult to determine a repair plan. Furthermore, even after a new restoration model and plan are determined, custom-made templates are still needed to complete the restoration of the damaged building.

[0004] Therefore, researching how to solve the problems of difficulty in obtaining original drawings of damaged buildings and the complexity of the restoration process is of great significance for promoting the development of the building restoration industry. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a building repair method and system based on generative models, the purpose of which is to achieve rapid repair of damaged buildings in the absence of original drawings.

[0006] To achieve the above objectives, according to one aspect of the present invention, a building repair method based on a generative model is proposed, comprising the following steps:

[0007] Model training phase:

[0008] Obtain the overall 3D model and the 3D model of the damaged part of the building. The 3D model of the damaged part is larger than the damaged part of the building and completely includes the damaged part of the building.

[0009] Multiple truncated models are obtained by randomly selecting a certain volume of the model from the overall 3D model. For each truncated model, the voxels are encoded in position, and a certain volume of the model is randomly selected from the truncated model as a secondary truncated model. The remaining part is the residual truncated model, thereby constructing a training set.

[0010] The neural process model is trained based on the training set, and the trained neural process model is used as a generative model. The input of the neural process model is the position and color information of voxels in the residual truncated model and the position information of voxels in the secondary truncated model. The output is the color information of voxels in the secondary truncated model.

[0011] Model application phase:

[0012] A computational model is pre-built, which is a cuboid that completely contains the three-dimensional model of the damaged part. The voxels in the computational model are then positionally encoded.

[0013] The position and color information of voxels in the 3D model of the damaged part, as well as the position information of voxels outside the 3D model of the damaged part in the calculation model, are input into the generative model to obtain the repair model of the damaged part of the building. The repair of the damaged part of the building is realized based on the repair model.

[0014] As a further preferred embodiment, the neural process model includes a latent space encoder, a deterministic encoder, and a decoder, wherein:

[0015] The latent space encoder includes a self-attention layer that takes the position and color information of voxels in the remaining truncated model as input and outputs a global latent distribution.

[0016] The deterministic encoder includes a self-attention layer and a coordinate attention layer. The self-attention layer takes the position and color information of voxels in the residual truncation model as input and outputs latent variables. The coordinate attention layer takes the latent variables, the position information of voxels in the residual truncation model and the position information of voxels in the quadratic truncation model as input and outputs global latent variables.

[0017] The decoder includes a multilayer perceptron, which takes the position information of voxels, global latent variables, and global latent distribution in the quadratic truncation model as input, and outputs the color information of voxels in the quadratic truncation model.

[0018] As a further preferred embodiment, both the latent space encoder and the deterministic encoder's self-attention layers employ a multi-head attention mechanism.

[0019] As a further optimization, for the neural process model, several input methods are pre-constructed, and different input methods use different numbers of voxel position information and color information as input pairs;

[0020] For the self-attention layer of the latent space encoder, different input methods correspond to different latent distributions. The global latent distribution is obtained by averaging all latent distributions.

[0021] For the self-attention layer of the deterministic encoder, different input methods correspond to different latent variables in the output; and for the coordinate attention layer, the output is the number of intermediate global latent variables corresponding to the latent variables. The global latent variables are obtained by averaging all the intermediate global latent variables.

[0022] As a further preferred embodiment, the volume of the three-dimensional model of the damaged part is 10 to 20 times the volume of the damaged part of the building.

[0023] As a further preferred embodiment, the volume of the remaining truncated model is consistent with the volume of the three-dimensional model of the damaged part; the volume of the secondary truncated model is consistent with the volume of the damaged part of the building.

[0024] As a further preferred method, obtaining the overall 3D model of the damaged building and the 3D model of the damaged parts includes the following steps:

[0025] The drone first flies around the entire building to be repaired for several weeks, and then around the damaged parts of the building for several weeks; during the flight, depth images are collected by the RGB-D camera on the drone.

[0026] Based on the acquired depth images, a 3D model of the overall structure and a 3D model of the damaged parts of the building are constructed using a simultaneous localization and mapping method.

[0027] As a further preferred option, repairing damaged parts of a building based on a repair model includes: obtaining a corresponding physical printed model through 3D printing based on the repair model, and installing the physical printed model onto the damaged parts of the building to complete the repair.

[0028] As a further preferred option, ABS material is used to obtain a corresponding solid printed model through 3D printing; polyurethane building adhesive is applied to the joint between the solid printed model and the damaged part of the building to fix the solid printed model to the damaged part of the building.

[0029] According to another aspect of the present invention, a generative model-based building repair system is provided, comprising a processor for executing the above-described generative model-based building repair method.

[0030] In summary, compared with the prior art, the above-described technical solutions conceived by this invention mainly possess the following technical advantages:

[0031] 1. This invention is based on the overall three-dimensional model of a damaged building, obtains a truncated model and a secondary truncated model, and divides the original three-dimensional spatial information into positional information and color information. A training set is constructed to train the neural process model, thereby realizing the intelligent and automatic generation of the repair model. This reduces the dependence on the original drawings or three-dimensional models when repairing buildings, and can help restorers determine the repair plan for ancient buildings or decorative components with complex patterns on the surface. It has profound engineering practical significance and broad application prospects.

[0032] 2. The neural process model can infer the local information of the secondary truncation model from the overall information of the remaining truncation model, which can perfectly fit the scenario of building restoration. At the same time, the use of attention mechanism and global latent variable design can make the model pay more attention to global information, and the training set is generated by the model of random truncation of the whole building. By combining the use of these three, the global consistency and continuity of the generated damaged part model with the whole can be well guaranteed.

[0033] 3. For neural process models, using input pairs of different scales allows the model to take into account features at different scales in the building, so as to grasp local details and maintain global consistency.

[0034] 4. After the repair plan is determined, the required repair model can be quickly formed by 3D printing, avoiding the need to redesign the construction template during the repair process, improving the efficiency of building repair, and reducing the input of repair materials and repair personnel. Attached Figure Description

[0035] Figure 1 This is a flowchart of a generative model-based building repair method according to an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of the structure of the UAV according to an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the neural process model structure in an embodiment of the present invention.

[0038] In all the accompanying drawings, the same reference numerals are used to denote the same elements or structures, where: 1-UAV, 2-RGB-D camera. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0040] This invention provides a building repair method based on a generative model, such as... Figure 1 As shown, it includes the following steps:

[0041] S1. Obtain the overall 3D model of the damaged building and the 3D model of the damaged parts.

[0042] Specifically, the 3D model of the damaged area is larger than the actual damaged area of ​​the building and completely encompasses it. Preferably, the volume of the 3D model of the damaged area is 10 to 20 times the volume of the actual damaged area, which is estimated by the repair personnel.

[0043] Furthermore, such as Figure 2 As shown, a three-dimensional model is acquired using a drone 1 equipped with an RGB-D camera 2 and a NUC11 microcomputer. The NUC11 microcomputer is used to communicate with a ground base station, make mission planning and decisions, and control the drone body to perform the aforementioned movement operations. The ground base station communicates with the drone's NUC11 microcomputer via a Bluetooth module to transmit data.

[0044] Specifically, the drone is first launched and flies around the entire building for several weeks, then flies to the damaged area and controls the drone to fly around the damaged area. During the flight, the drone continuously collects depth images of the building through an RGB-D camera and sends all the collected depth images to a ground base station, and then uses the Simultaneous Localization and Mapping (SLAM) method to build a 3D model.

[0045] Specifically, the SLAM method uses the same approach to determine the overall 3D model and the 3D model of the damaged area. Here, we take obtaining the overall 3D model as an example, which includes the following steps:

[0046] (1) The UAV flies around the building as a whole. The ground base station obtains the translation and rotation information between two adjacent frames of images by receiving the color and depth information of each two frames of depth images, and uses this to form a visual odometry to estimate the motion of the UAV and obtain the camera pose information.

[0047] (2) The ground base station optimizes the camera pose information by using the extended Kalman filter method to achieve local optimization;

[0048] (3) The ground base station uses the bag-of-words model to identify the features of the depth image and determines whether the UAV has reached the same location by comparing the similarity of the features. If it is determined that the same location has been reached, the pose information is set to be the same to complete the loop closure detection, achieve global optimization, and reduce the cumulative error.

[0049] (4) The ground base station constructs the overall three-dimensional model based on the depth image and camera pose information.

[0050] S2. Based on the overall 3D model, obtain multiple truncated models; for each truncated model, divide it into a secondary truncated model and the corresponding remaining truncated model, and construct a training set.

[0051] Specifically, a certain volume of the model is randomly selected multiple times from the overall 3D model to obtain multiple truncated models; in some embodiments, 50,000 truncated models are obtained.

[0052] Then for each truncation model:

[0053] Using the center of the truncated model as the origin, positional encoding is performed on all voxels contained in the truncated model; specifically, using the center of the smallest cuboid that can completely contain the truncated model as the origin, positional encoding is performed on all voxels contained in the truncated model, and the color information of the remaining positions within the cuboid is 0.

[0054] A certain volume of the model is randomly cut off from the truncated model to obtain a secondary truncated model. The remaining part of the truncated model is the residual truncated model.

[0055] Furthermore, the volume of the remaining cut-off model is consistent with the volume of the three-dimensional model of the damaged part; the volume of the secondary cut-off model is consistent with the volume of the damaged part of the building; that is, the volume of the remaining cut-off model is 10 to 20 times the volume of the secondary cut-off model.

[0056] S3. Train the multi-scale fusion neural process model based on the training set, and use the trained neural process model as a generative model.

[0057] The input to the neural process model is the position and color information of all voxels in the residual truncated model, as well as the position information of all voxels in the corresponding secondary truncated model. The output is the color information of all voxels in the corresponding secondary truncated model.

[0058] Furthermore, such as Figure 3 As shown, the neural process model includes a latent space encoder, a deterministic encoder, and a decoder, wherein:

[0059] The latent space encoder includes a self-attention layer that takes the position and color information of all voxels in the remaining truncated model as input and outputs a global latent distribution.

[0060] The deterministic encoder consists of a self-attention layer and a coordinate attention layer. The self-attention layer takes the position and color information of voxels in the residual truncation model as input and outputs a latent variable. The coordinate attention layer takes the latent variable, the position information of voxels in the residual truncation model and the position information of voxels in the quadratic truncation model as input, calculates coordinate attention, and outputs a global latent variable. Coordinate attention is more suitable for representing spatial features by separating and aggregating channel and spatial information.

[0061] The decoder includes a multilayer perceptron, which takes the position information of voxels, global latent variables, and global latent distribution in the quadratic truncation model as input and the color information of voxels in the quadratic truncation model as output; the position information of voxels refers to the coordinates after position encoding, and the color information of voxels refers to the RGB values.

[0062] Furthermore, several input methods are pre-constructed, with different input methods using input pairs of different scales, i.e., different numbers of voxel position and color information as input pairs. For the self-attention layer of the latent space encoder, different input methods correspond to the same number of latent distributions output. Then, the average of all latent distributions is calculated to obtain the global latent distribution. For the self-attention layer of the deterministic encoder, different input methods correspond to the same number of latent variables output. Then, for the coordinate attention layer, a corresponding number of intermediate global latent variables are output. The average of all intermediate global latent variables is calculated to obtain the global latent variable.

[0063] For example, three input methods can be used: 1) using the position and color information of a single voxel as one input pair, thus forming a series of input pairs; 2) using the average position and average color information of a cube formed by 8 voxels as one input pair, thus forming a series of input pairs; 3) using the average position and average color information of a cube formed by 27 voxels as one input pair, thus forming a series of input pairs. Because of these three input methods, the self-attention layer of the latent space encoder can obtain three latent distributions, which can then be averaged to obtain the global latent distribution. Similarly, the self-attention layer of the deterministic encoder can obtain three latent variables, such as... Figure 3 As shown.

[0064] Furthermore, the self-attention layers of the latent space encoder and the deterministic encoder use multi-head attention, where the number of heads is m; the multilayer perceptron of the decoder has n hidden layers and k hidden units. Larger values ​​of n, m, and k can repair structures and decorate more complex buildings; for example, n=3, m=5, k=8, or n=5, m=8, k=11.

[0065] S4. Input the 3D model of the damaged part into the generative model to obtain the repair model of the damaged part of the building, and realize the repair of the damaged part of the building based on the repair model.

[0066] Specifically, the computational model is set as a minimum cuboid that completely contains the 3D model of the damaged area. All voxels in the computational model are positionally encoded, and the model is divided into two parts: the 3D model of the damaged area and the remaining computational model. The position and color information of the voxels in the 3D model of the damaged area, as well as the position information of the voxels in the remaining computational model, are input into the generative model to obtain the color information of the voxels in the remaining computational model. Voxels with a color information of 0 in the remaining computational model are removed to obtain the repair model.

[0067] Specifically, after receiving the 3D model of the damaged part, the ground base station infers the repair model of the broken part through generative modeling, uses Simplify 3D software to slice the repair model, generates the corresponding G-Code file, and starts 3D printing; after printing, adhesive is applied to the joint between the solid printed model and the building, and the solid printed model is installed to complete the building repair.

[0068] Furthermore, when 3D printing solid models, ABS material is used as the consumable material, and polyurethane building adhesive is used as the binder.

[0069] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A building repair method based on a generative model, characterized in that, Includes the following steps: Model training phase: Obtain the overall 3D model and the 3D model of the damaged part of the building. The 3D model of the damaged part is larger than the damaged part of the building and completely includes the damaged part of the building. Multiple truncated models are obtained by randomly selecting a certain volume of the model from the overall 3D model. For each truncated model, the voxels are encoded in position, and a certain volume of the model is randomly selected from the truncated model as a secondary truncated model. The remaining part is the residual truncated model, thereby constructing a training set. The neural process model is trained based on the training set, and the trained neural process model is used as a generative model. The input of the neural process model is the position and color information of voxels in the residual truncated model and the position information of voxels in the secondary truncated model. The output is the color information of voxels in the secondary truncated model. Model application phase: A computational model is pre-built, which is a cuboid that completely contains the three-dimensional model of the damaged part. The voxels in the computational model are then positionally encoded. The calculation model is divided into two parts: a three-dimensional model of the damaged area and the remaining calculation model. The position and color information of voxels in the 3D model of the damaged part, as well as the position information of voxels in the remaining computational model, are input into the generative model to obtain the color information of voxels in the remaining computational model. The voxels with color information of 0 in the remaining calculation model are removed to obtain the repair model of the damaged parts of the building. The repair of the damaged parts of the building is then carried out based on this repair model. The neural process model includes a latent space encoder, a deterministic encoder, and a decoder, wherein: The latent space encoder includes a self-attention layer that takes the position and color information of voxels in the remaining truncated model as input and outputs a global latent distribution. The deterministic encoder includes a self-attention layer and a coordinate attention layer. The self-attention layer takes the position and color information of voxels in the residual truncation model as input and outputs latent variables. The coordinate attention layer takes the latent variables, the position information of voxels in the residual truncation model and the position information of voxels in the quadratic truncation model as input and outputs global latent variables. The decoder includes a multilayer perceptron, which takes the position information of voxels, global latent variables, and global latent distribution in the quadratic truncation model as input, and outputs the color information of voxels in the quadratic truncation model.

2. The building repair method based on generative models as described in claim 1, characterized in that, Both the latent space encoder and the deterministic encoder use a multi-head attention mechanism in their self-attention layers.

3. The building repair method based on generative models as described in claim 1, characterized in that, For the neural process model, several input methods are pre-constructed, and different input methods use different numbers of voxel position information and color information as input pairs; For the self-attention layer of the latent space encoder, different input methods correspond to different latent distributions. The global latent distribution is obtained by averaging all latent distributions. For the self-attention layer of the deterministic encoder, different input methods correspond to different latent variables in the output; and for the coordinate attention layer, the output is the number of intermediate global latent variables corresponding to the latent variables. The global latent variables are obtained by averaging all the intermediate global latent variables.

4. The building repair method based on generative models as described in claim 1, characterized in that, The volume of the three-dimensional model of the damaged part is 10 to 20 times the volume of the damaged part of the building.

5. The building repair method based on a generative model as described in claim 4, characterized in that, The volume of the remaining truncated model is the same as the volume of the three-dimensional model of the damaged part; the volume of the secondary truncated model is the same as the volume of the damaged part of the building.

6. The building repair method based on generative models as described in claim 1, characterized in that, Obtaining the overall 3D model and the 3D model of the damaged parts of the building includes the following steps: The drone first flies around the entire building to be repaired for several weeks, and then around the damaged parts of the building for several weeks; during the flight, depth images are collected by the RGB-D camera on the drone. Based on the acquired depth images, a 3D model of the overall structure and a 3D model of the damaged parts of the building are constructed using a simultaneous localization and mapping method.

7. The building repair method based on a generative model as described in any one of claims 1-6, characterized in that, Repairing damaged parts of a building based on a repair model includes: obtaining a corresponding physical printed model through 3D printing based on the repair model, and installing the physical printed model onto the damaged part of the building to complete the repair.

8. The building repair method based on a generative model as described in claim 7, characterized in that, Using ABS material, a corresponding solid printed model is obtained through 3D printing; polyurethane building adhesive is applied to the joint between the solid printed model and the damaged part of the building to fix the solid printed model to the damaged part of the building.

9. A building repair system based on a generative model, characterized in that, Includes a processor for performing the generative model-based building repair method as described in any one of claims 1-8.

Images