A method for intelligent repair and beautification of building textures by fusing a deep learning model

Abstract

The application is suitable for the technical field of building model data processing, and provides a building texture intelligent repairing and beautifying method fusing a deep learning model. The method firstly reorganizes the three-dimensional building model texture to generate wall and roof texture pictures to be processed, extracts a to-be-repaired area of the building texture to be repaired by combining low-rank matrix decomposition and a SAM image segmentation algorithm, then determines whether a LAMA image repairing model or a PatchMatch image repairing algorithm is used for texture processing according to the proportion of the to-be-repaired area in the whole texture picture to achieve the maximum repairing effect, finally uses an image diffusion model, a ControlNet control map and a prompt word to perform texture distortion repairing and material beautifying, and outputs the beautified wall and roof texture. The application can simultaneously process problems such as texture occlusion, drawing, noise and window distortion, realizes automatic detection and automatic texture repairing and beautifying of the texture occlusion drawing area, and significantly improves the time consumption of texture editing and processing in the building individual modeling process.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of building model data processing, and particularly relates to a building texture intelligent repairing and beautifying method fusing a deep learning model. BACKGROUND

[0002] With the rapid development of real scene three-dimensional, digital twin city, low-altitude economy and other fields, the single structured modeling technology of three-dimensional building model plays an important role in the rapid construction of urban three-dimensional scenes. The single structured modeling technology of building model generally includes two parts of structured reconstruction of building geometry and building texture mapping and reorganization. The structured reconstruction of building geometry has gradually become mature, and the automatic construction of ordinary regular buildings can be basically realized.

[0003] However, due to the non-structural nature of building texture pictures, after automatic mapping of building texture, there are still problems such as occlusion, pull flower, noise, distortion and the like on the model. In view of these building model texture quality problems, manual editing and repairing with image processing software is often needed, and then the building model is remapped. This scheme needs to spend a lot of manpower to repair and beautify the building texture image, which affects the cycle and cost of the whole building single modeling. According to the statistics of relevant three-dimensional modeling departments, in the process of single modeling of urban building three-dimensional model, the time consumption of building texture mapping and manual editing and repairing accounts for more than 70% of the total time consumption of single modeling.

[0004] The time consumption of 1km 2 The time consumption of single modeling of urban scene building in geometry editing and texture editing is shown in Table 1:

[0005] Table 1 Time consumption statistical analysis of manual building single structured modeling

[0006]

[0007] Through the above statistical analysis, it can be seen that most of the time consumption of manual urban building single modeling is in the process of building texture editing and repairing. The building texture repairing processing mode mainly using manual image processing software editing cannot meet the requirements of urban three-dimensional scene modeling and building model rapid updating.

[0008] Therefore, it is of great significance to realize the automatic intelligent repairing and beautifying technology of occlusion, noise, pull flower, distortion and other distorted textures in the wall and roof textures of building model for urban building three-dimensional model reconstruction, which can greatly improve the efficiency of urban three-dimensional building model reconstruction and three-dimensional scene construction. SUMMARY

[0009] In view of the above problems, the purpose of the present application is to provide a building texture intelligent repair and beautification method fusing a deep learning model, aiming to solve the problem that the repair and beautification of building texture at the present stage mainly rely on manual processing of editing software, realize the repair and beautification of texture by fusing multiple deep learning models, thereby reducing the building monomer modeling cycle and cost, improving the urban building model monomer structured reconstruction effect, increasing the realness of building model wall and roof texture, and reducing the distortion of building model texture caused by original building texture occlusion, quality problems and loss of building geometric reconstruction accuracy.

[0010] The application adopts the following technical solutions:

[0011] The building texture intelligent repair and beautification method fusing a deep learning model comprises the following steps:

[0012] Step S1, according to the texture reorganization of a three-dimensional building monomer model, texture pictures of a wall and a roof to be processed are generated, and an image low-rank matrix segmentation algorithm and a SAM image segmentation algorithm are used to extract a to-be-repaired area of building texture that needs to be repaired;

[0013] Step S2, according to the proportion of the to-be-repaired area in the entire building texture area, a LAMA image repair model or a PatchMach image repair algorithm is used to automatically repair the to-be-repaired area;

[0014] Step S3, a YOLO deep learning model is used to identify billboards and text areas in the repaired wall texture picture, and a text and billboard area sub-picture is cut out and output;

[0015] Step S4, a building texture image-to-image beautification model is established by combining an image diffusion model, a ControlNet control graph and an image-to-image prompt word guide, the repaired texture picture is beautified, and a beautified wall and roof texture picture is output;

[0016] Step S5, the cut-out and output text and billboard area sub-picture is fused into the original position of the wall.

[0017] Further, the specific process of step S1 is as follows:

[0018] S11, load an original three-dimensional building monomer model, read the model geometric information and texture information, sample, reorganize and merge the building texture of the model in the horizontal and vertical directions, and obtain texture pictures of building walls and roofs, respectively;

[0019] S12, an image low-rank matrix segmentation algorithm is used to obtain distorted texture areas in the wall and roof texture pictures, and a first texture repair area is obtained;

[0020] S13, segment the wall and roof texture pictures according to the SAM image segmentation algorithm, extract the wall occlusion and roof covering area using the prompt words, and obtain a second texture repair area;

[0021] S14, merge the first and second texture repair areas to obtain the final repair area of the building wall and roof.

[0022] Further, the specific process of step S12 is as follows:

[0023] (1) The input texture picture is converted into a single-channel matrix D by grayscale processing, and a normalized matrix Dnormal is obtained by normalization processing;

[0024] (2) The normalized matrix Dnormal is decomposed into a low-rank matrix A and a sparse matrix E, wherein the low-rank matrix A represents the main structure information of the building texture, and the sparse matrix E represents the abnormal value of the distorted texture;

[0025] (3) The model is solved by the alternating direction multiplier algorithm to obtain the low-rank matrix A and the sparse matrix E;

[0026] (4) The obtained sparse matrix E is processed to extract the positions and values of the non-zero elements, thereby obtaining the position of the building texture distortion area and generating a first texture repair area in the form of a binary graph.

[0027] Further, the specific process of step S13 is as follows:

[0028] The SAM image segmentation algorithm is used to segment the wall and roof texture pictures, the repair area of the wall occlusion and roof covering is automatically extracted according to the prompt words, then the extracted mask is binarized and connected domain analysis is performed to remove noise and small isolated areas, and the larger area is retained as the second texture repair area of the wall and roof texture.

[0029] Further, the specific process of step S2 is as follows:

[0030] S21, calculate the area ratio of the wall and roof texture repair areas to the corresponding building texture area, respectively, for the repair area with an area ratio less than or equal to 10%, directly repair the repair area based on the LAMA image repair model;

[0031] S22, for the repair area with an area ratio greater than 10% and less than 30%, the current repair area is divided into multiple sub-texture repair areas with an area ratio less than 10% according to the grid, and then the sub-repair area is repaired automatically using the LAMA image repair model;

[0032] S23, for the area ratio greater than or equal to 30% of the to-be-repaired area, the PatchMach image repair algorithm is used to supplement and repair the wall and roof texture.

[0033] Among them, the LAMA image repair model directly repairs the texture picture with a resolution size greater than 256x256, without first downsampling the texture graph.

[0034] Further, the specific process of establishing the building texture graph generation beautification model in step S4 is as follows:

[0035] S41, select Stable Diffusion diffusion model as the base model;

[0036] S42, supplement ControlNet control graph in Stable Diffusion diffusion model

[0037] S43, for the building texture type, style and building element layout requirements of the building texture graph generation, positive and negative prompt words are added to guide the Stable Diffusion diffusion model to generate texture.

[0038] Further, the specific process of step S42 is as follows: the structural information of the building texture is extracted through the preprocessor of Canny edge detection and Depth depth estimation, and the control tensor is fused with the U-Net feature map of the Stable Diffusion diffusion model.

[0039] The beneficial effects of the present application are:

[0040] Firstly, the present application is an intelligent repair scheme of building texture fused with deep learning model, from the identification of building texture to-be-repaired area to the repair of texture image, the whole process is realized automatically, without manual intervention, the building wall and roof texture can be intelligently repaired, which greatly reduces the time-consuming and labor cost of texture editing in the process of building single modeling.

[0041] Secondly, the present application fuses multiple image processing and deep learning algorithms to identify the to-be-repaired area of building wall and texture graph and repair the building texture image, combines the texture image with the pull flower and shielding condition, respectively uses image low-rank matrix decomposition and SAM (Segment Anything Model) image segmentation algorithm to obtain the repair area, and improves the accuracy of building texture to-be-repaired area extraction.

[0042] Thirdly, by in-depth analysis and comparison of the advantages and disadvantages of LAMA (Large Mask Inpainting) image inpainting model and PatchMatch image inpainting algorithm, according to the area proportion of the two types of texture repair area, the corresponding repair algorithm and algorithm subgraph input strategy are adopted for repair, so as to improve the texture repair quality of the model.

[0043] Finally, aiming at the problems of distortion and noise points in building wall texture, the Stable Diffusion diffusion model is used to guide the texture graph to be beautified and repaired, and the problems of text distortion and noise points in the building wall texture are solved. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is the flowchart of the building texture intelligent repair and beautification method provided by the embodiment of the present application, which fuses a deep learning model;

[0045] Figure 2 is the original texture graph of the building wall;

[0046] Figure 3 is the texture graph after occlusion repair of the building wall;

[0047] Figure 4 is the texture graph after beautification and repair of the building wall;

[0048] Figure 5 is the original texture graph of the building roof;

[0049] Figure 6 is the texture graph after covering repair of the building roof;

[0050] Figure 7 is the texture graph after beautification and repair of the building roof. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the present application clearer and more apparent, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0052] During the structural reconstruction of a single building model, the complex three-dimensional spatial relationships between buildings and trees, roof attachments, and debris in urban scenes, as well as the loss of geometric accuracy after structural reconstruction of buildings, lead to distortion problems such as occlusion, blank spaces, streaks, and window distortion in the mapped textures. For example, the wall textures of buildings acquired using oblique photography often have occlusion by trees in low-rise areas, windows on building walls are distorted to a certain extent, and the roof textures of buildings may have snow accumulation in winter. These texture quality issues affect the realism of the building model. As a result, after texture mapping, building textures often have distortion problems, and a large amount of manpower is still required to repair and beautify the occlusion of building wall and roof textures. The method of using image processing software to manually model and repair texture occlusion, streaks, and other problems is costly and time-consuming, which limits the further application of automated modeling technology in the fields of real-life 3D and digital twins.

[0053] To this end, this embodiment provides a method for intelligent restoration and beautification of architectural textures that integrates a deep learning model to solve this problem. In order to illustrate the technical solution described in the present invention, a specific embodiment is provided below for illustration.

[0054] like Figure 1 As shown, the method for intelligent restoration and beautification of architectural textures integrating a deep learning model provided in this embodiment includes the following steps:

[0055] Step S1: Generate texture images of the wall and roof to be processed based on the texture reconstruction of the three-dimensional building monomer model, and use the image low-rank matrix segmentation algorithm and the SAM image segmentation algorithm to extract the area to be repaired where the building texture needs to be repaired.

[0056] This step mainly aims to identify the areas of building texture that need to be repaired. The specific process is as follows:

[0057] S11. Load the original three-dimensional building model, read the model's geometric information and texture information, sample, reorganize, and merge the model's building texture in horizontal and vertical directions, and obtain texture images of the building's walls and roof, respectively.

[0058] By loading the building model and reading relevant information, and then performing the corresponding building texture analysis, we can obtain the texture images of the building walls and roofs. These wall and roof texture images will serve as input data for subsequent intelligent texture restoration and beautification.

[0059] S12. Use an image low-rank matrix segmentation algorithm to obtain distorted texture areas in the wall and roof texture images to obtain a first texture repair area.

[0060] The step adopts an image low-rank matrix segmentation algorithm to obtain noise, carvings, blank areas, etc. in the building wall and roof texture pictures, to obtain a first texture repair area. For the carvings, noise, blank areas, etc. in the building texture, the image matrix can be decomposed into a low-rank matrix representing the main structural features of the building texture and a sparse matrix representing the carvings, noise, etc. This step fully utilizes the global repetition characteristics of the building texture, and through optimization, the carvings, noise, etc. in the building texture are separated out.

[0061] The specific process of step S12 is as follows:

[0062] (1) Texture image preprocessing: The input texture picture is converted into a single-channel matrix D through grayscale processing, and a normalized matrix Dnormal is obtained through normalization processing.

[0063] An input building texture RGB picture I ∈ R m×n×3 is converted into a single-channel matrix D through grayscale processing, and a normalized matrix Dnormal in the range of [0, 1] is obtained.

[0064] (2) Construction of optimization model: The normalized matrix Dnormal is decomposed into a low-rank matrix A and a sparse matrix E, wherein the low-rank matrix A represents the main structural information of the building texture, and the sparse matrix E represents the distorted texture abnormal values.

[0065] Assuming that the building texture image normalized matrix Dnormal is decomposed into a low-rank matrix A and a sparse matrix E, i.e. Dnormal=A+E.

[0066] (3) Solving the optimization problem: The model is solved through an alternating direction multiplier method (ADMM) to obtain the low-rank matrix A and the sparse matrix E.

[0067] The iterative update formula is as follows:

[0068] a. Update the low-rank matrix A, wherein P u represents a soft threshold processing operation, and U (k) is a Lagrange multiplier matrix:

[0069] ;

[0070] b. Update the sparse matrix E, wherein S λ is a soft threshold processing operation, and the core is to shrink the matrix elements to zero, thereby realizing the extraction and retention of the non-zero elements of the sparse matrix:

[0071] ;

[0072] c. Update the Lagrange multiplier matrix U:

[0073] ;

[0074] (4) Post-processing: Process the obtained sparse matrix E, extract the positions and values of the non-zero elements, and obtain the positions of the distorted texture region of the building, and generate the first texture repair area in the form of a binary graph.

[0075] S13, segment the wall and roof texture pictures according to the SAM image segmentation algorithm, and extract the wall occlusion and roof covering area using the prompt words to obtain the second texture repair area.

[0076] Segment the wall and roof texture pictures based on the SAM (Segment Anything Model) segmentation algorithm, such as using tree and snow prompt words to extract the wall tree occlusion and roof snow area to obtain the second texture repair area. SAM is an advanced image segmentation model designed to quickly and accurately segment any target in an image by providing prompt words (such as points, boxes, and text).

[0077] This step uses the SAM image segmentation algorithm to segment the wall and roof texture pictures, automatically masks the repair area of the wall occlusion and roof covering according to the prompt words (such as wall tree and roof snow prompt words), then performs binary processing on the extracted mask, and performs connected component analysis to remove noise and small isolated areas, and retains larger areas as the second texture repair area of the wall and roof texture.

[0078] S14, merge the first and second texture repair areas to obtain the final repair area of the building wall and roof.

[0079] Step S2, according to the proportion of the repair area in the entire building texture area, the LAMA image repair model or PatchMach image repair algorithm is used to automatically repair the repair area.

[0080] LAMA (Large Mask inpainting) is a deep learning model for image inpainting that combines advanced frameworks such as Fast Fourier Convolution (FFC), expands the receptive field, and better captures global information of images. It also performs well on image completion with periodic structures, and can effectively handle various occlusion situations such as building walls and roofs. Although the LAMA model shows good generalization ability when processing pictures with a resolution higher than 256x256, to further ensure the repair effect when processing wall and roof texture pictures with a resolution higher than 256x256, the input of the LAMA image inpainting model is fine-tuned to support direct repair of texture pictures with a resolution of 2048x2048, without the need for downsampling the texture pictures first. At the same time, it avoids the color difference problem caused by splitting a large picture into multiple sub-pictures and then merging them.

[0081] This step selects the LAMA image inpainting model as the deep learning model for texture repair, fine-tunes the input parameters of the LAMA image inpainting model, and supports direct input of images with a size of 2048x2048 to ensure the repair effect of high-resolution building texture pictures, while avoiding color difference caused by sub-picture blocking.

[0082] The specific process of this step is as follows:

[0083] S21, respectively calculate the area proportion of the wall repair area and the roof texture repair area in the corresponding building texture area. For the repair area with an area proportion less than or equal to 10%, directly perform automatic repair on the repair area based on the LAMA image inpainting model.

[0084] Here, the area proportion refers to the proportion of the wall repair area in the entire wall building texture area, and the proportion of the roof texture repair area in the entire roof building texture area. The meaning of area proportion in subsequent steps is the same.

[0085] When the repair area occupies a small proportion of the entire texture area, the LAMA image inpainting model can usually repair the missing area well and ensure the natural and coherent texture repair area.

[0086] S22, for the repair area with an area proportion greater than 10% and less than 30%, the current repair area is divided into multiple sub-texture repair areas with an area proportion less than 10% by grid partitioning, and then the sub-repair area is automatically repaired by the LAMA image inpainting model.

[0087] Although the LAMA image inpainting model uses fast Fourier convolution to increase the receptive field, when the area to be repaired occupies a large proportion of the entire texture area, the limited receptive field will make it difficult for the model to capture the global information of the image, thereby affecting the repair effect of the building texture. Therefore, when the area to be repaired is greater than or equal to 30%, the PatchMach image inpainting algorithm is used to supplement the repair of the wall and roof texture.

[0088] S23, for the area to be repaired with an area ratio greater than or equal to 30%, the PatchMach image inpainting algorithm is used to supplement the repair of the wall and roof texture.

[0089] When the area to be repaired occupies more than 30% of the entire texture area, in order to avoid the problem of poor texture repair effect caused by the large area of the area to be repaired, the PatchMatch image inpainting algorithm is used to supplement the repair of the wall and roof texture, and the repaired building texture image is output.

[0090] Compared with the LAMA image inpainting model, the PatchMatch image inpainting algorithm uses the method of filling the missing area with known area information. Although it is not as good as the LAMA image inpainting model in terms of repair effect of complex images, it can effectively fill the missing area when the occluded repair area is large, and make up for the problem of excessive divergence and uncontrollable quality of the repair area caused by the LAMA image inpainting model when the repair area is too large.

[0091] In the above step S2, different image inpainting algorithms and parameters are selected for repair according to the proportion of different building texture repair areas. The present application analyzes and compares the characteristics and advantages of LAMA image inpainting model and PatchMatch repair algorithm in repairing images with different degrees of occlusion through experiments, and at the same time, in order to solve the problem of unsmooth boundary area of the repaired subgraph caused by splitting the building texture graph into multiple subgraphs, the input of the LAMA network model is fine-tuned to support direct repair of texture graphs with a resolution size of 2048x2048 without the need for downsampling or subgraph splitting operations.

[0092] In view of the diversity of the size of the area to be repaired for different building textures, according to the proportion of the area to be repaired in the entire texture map, a hybrid scheme of directly repairing the area based on the texture repair model, splitting the texture repair area into subgraphs and then repairing the LAMA image, and using the PatchMatch algorithm for repair is adopted. This hybrid scheme can not only repair high-resolution building texture pictures with high quality, but also better adapt to building texture maps with different degrees of occlusion, avoiding the problem of unsatisfactory building texture repair caused by directly using a single-scale LAMA image repair deep learning model when the image repair area is too large.

[0093] Step S3, using a YOLO deep learning model to identify the billboard and text area in the repaired wall texture picture, and cutting out the text and billboard area subgraph.

[0094] The text and billboard area is important information in the building texture map and is an important basis for ensuring the authenticity of the building model. The YOLO (You Only Look Once) model is an advanced deep learning object detection algorithm that can identify text and billboard objects in building textures, and can quickly, efficiently and accurately identify text and billboard areas in building wall texture maps. For example, using the YOLOv8 deep learning model to identify the text area and billboard area on the building wall, and cutting out the text and billboard area subgraph, to prepare for solving the distortion problem of excessive repair of the billboard and text area caused by the large model graph generation and beautification.

[0095] Step S4, combining the image diffusion model, ControlNet control graph and graph generation prompt word guidance to establish a building texture graph generation and beautification model, and beautifying the repaired texture picture to output the beautified wall and roof texture picture.

[0096] In addition to occlusion and texture pull-ups such as trees and snow, building model texture maps often have problems such as distorted building windows and building texture material noise, which cannot be handled by the LAMA image repair algorithm. Based on the StableDiffusion diffusion model (StableDiffusion model) and supplemented by the ControlNet (an extension plug-in of StableDiffusion) control graph and prompt word guidance, a building texture graph generation and beautification model is established to remove noise, repair distortion and other beautification processing of the repaired texture output by the foregoing steps, and output the beautified building wall and roof texture map. The StableDiffusion diffusion model is an open source diffusion model that supports image generation image mode, and its core idea is to first add a large amount of Gaussian noise to the original image to obtain an image almost full of noise, and then learn to remove noise and restore the original image step by step through a U-Net (U-shaped network) neural network.

[0097] Here, the specific process of establishing the building texture image generation beautification model is as follows:

[0098] S41, select Stable Diffusion diffusion model as the basic model.

[0099] Specifically, the Stable Diffusion XL model is selected as the basic model for image inpainting. The Stable Diffusion XL model supports high-resolution image generation of 1024x1024, and has significantly improved details and clarity compared to 1.5 and other models. At the same time, the understanding ability of the prompt word has also been significantly improved.

[0100] S42, supplement ControlNet control graph in the Stable Diffusion diffusion model.

[0101] In order to enable the Stable Diffusion diffusion model to combine the texture structure characteristics of building roofs and walls for beautification and repair, this step supplements the ControlNet control graph to more accurately control the generation of building texture images. Specifically, the structural information of the building texture is extracted through the preprocessor of Canny edge detection and Depth depth estimation, and the control tensor is fused with the U-Net feature map of the Stable Diffusion diffusion model, so that the building texture not only gets the beautification effect, but also guarantees the consistency of the generated texture and the original image geometric features, and meets the reality requirements of building three-dimensional model texture beautification and repair.

[0102] S43, increase positive and negative prompt words for the building texture type, style, and building element layout requirements of the building texture image generation, to guide the Stable Diffusion diffusion model to generate texture.

[0103] The positive prompt words are used to clarify the semantic information, details and quality requirements of building texture generation. The negative prompt words are used to exclude undesirable features and content, and enhance the rationality of the generated results. For example, the positive prompt words for building image beautification in this step are "Input image is facade or roof of a building. Generate a clear, noiseless and high quality image based on input image. Regenerate any blurry or distort area. Do not create new doors, windows or other components that do not exist in input image", and the negative prompt words are "low quality, blurry, noisy, unsharp, distortion". By adding the above prompt words, the Stable Diffusion diffusion model can effectively guide the building texture distortion and noise beautification and repair while ensuring the authenticity as much as possible.

[0104] Step S5, re-fuse the cut-out output text and billboard area subgraph to the original position in the wall surface.

[0105] To avoid over-repair of the Stable Diffusion diffusion model to the building wall surface text and billboard area, this step uses the text and billboard subgraph cut out by step S3 to update the image local area replacement of the beautified building wall texture graph, and uses the Poisson mixed image processing algorithm to fuse the color difference of the joint edge area, so that the text and billboard picture are more natural when fused to the beautified texture. The beautified texture graph retaining the building text and billboard area is output, and the full-automatic repair and beautification of the original building model texture is finally completed. This technical process improves the quality of the building model texture, greatly reduces the time-consuming and cost of manual building model texture repair.

[0106] The building wall and roof texture repair effect using the method of the present application is shown in Figures 2-7 , wherein Figures 2-4 are the original texture graph of the building wall, the repaired texture graph of the building wall after occlusion, and the repaired texture graph of the building wall after beautification, respectively, Figures 5-7 are the original texture graph of the building roof, the repaired texture graph of the building roof after covering, and the repaired texture graph of the building roof after beautification, respectively.

[0107] To sum up, first, the image low-rank matrix segmentation algorithm and the SAM (Segment Anything Model) deep learning image segmentation algorithm are used to automatically detect the occlusion, pull flower and noise area of the wall and roof, and obtain the building texture to be repaired area.

[0108] Second, according to the proportion of the to-be-repaired area in the entire building texture area, the LAMA (Large Mask Inpainting) image repair model and the PatchMatch image repair algorithm are used for automatic texture repair, so as to ensure the repair effect of different occlusion degree texture pictures.

[0109] Third, the Stable Diffusion diffusion model is used to guide the texture after occlusion repair to generate a texture graph, and the ControlNet control graph and prompt word are used to further perform window distortion repair and noise removal, and output the beautified building wall and roof texture graph.

[0110] Finally, in view of the problems of smearing of building wall billboards and text areas and over-repairing caused by the diffusion model, the YOLO (You Only Look Once) model is used to identify the billboards and text areas of the building wall, and then the diffusion model is used for texture repair and then fused to the original position, so that the distortion of the text caused by the diffusion model in the texture beautification is solved.

[0111] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for intelligent repair and beautification of building textures by fusing deep learning models, characterized in that, The method comprises the following steps: Step S1, generating a texture picture of a wall and a roof to be processed according to texture reorganization of a three-dimensional building monomer model, and extracting a to-be-repaired area of the building texture requiring repair by using an image low-rank matrix segmentation algorithm and a SAM image segmentation algorithm; Step S2, automatically repairing the to-be-repaired area by using a LAMA image repair model or a PatchMach image repair algorithm according to the proportion of the to-be-repaired area in the entire building texture area; Step S3, identifying a billboard and a text area in the repaired wall texture picture by using a YOLO deep learning model, and cutting and outputting a text and billboard area sub-picture; Step S4, combining an image diffusion model, a ControlNet control graph, and a graph generation graph prompt word to guide the establishment of a building texture graph generation beautification model, beautifying the repaired texture picture, and outputting a beautified wall and roof texture picture; Step S5, re-fusing the cut and outputted text and billboard area sub-picture to the original position of the wall. 2.The method of claim 1, wherein the method comprises: The specific process of step S1 is as follows: S11, loading an original three-dimensional building monomer model, reading model geometric information and texture information, sampling, reorganizing, and merging the building texture of the model in the horizontal and vertical directions, and respectively obtaining texture pictures of building walls and roofs; S12, obtaining a distorted texture area in the wall and roof texture picture by using an image low-rank matrix segmentation algorithm, and obtaining a first texture repair area; S13, segmenting the wall and roof texture picture according to a SAM image segmentation algorithm, extracting a wall occlusion and a roof covering area by using a prompt word, and obtaining a second texture repair area; S14, merging the first and second texture repair areas to obtain the final to-be-repaired area of the building wall and roof. 3.The method of claim 2, wherein the method comprises: The specific process of step S12 is as follows: (1) performing gray-scale processing on the input texture picture to convert it into a single-channel matrix D, and performing normalization processing to obtain a normalized matrix Dnormal; (2) decomposing the normalized matrix Dnormal into a low-rank matrix A and a sparse matrix E, wherein the low-rank matrix A represents the main structural information of the building texture, and the sparse matrix E represents the abnormal values of the distorted texture; (3) solving the model by using an alternating direction multiplier algorithm to obtain the low-rank matrix A and the sparse matrix E; (4) processing the obtained sparse matrix E to extract the positions and values of the non-zero elements, thereby obtaining the positions of the distorted texture area of the building texture, and generating a first texture repair area in the form of a binary graph. 4.The method of claim 3, wherein the method further comprises: training the fusion deep learning model using a plurality of training images, wherein each of the plurality of training images is associated with a corresponding texture image, and each of the plurality of training images is associated with a corresponding target image. The specific process of step S13 is as follows: The SAM image segmentation algorithm is used to segment the wall and roof texture picture, the mask extraction of the repair area of the wall occlusion and the roof covering is automatically performed according to the prompt word, then the extracted mask is binarized and subjected to connected component analysis to remove noise points and small isolated areas, and the larger areas are retained as the second texture repair area of the wall and roof texture.

5. The method of claim 4, wherein the method of fusing the building texture intelligent repairing and beautifying deep learning model comprises: The specific process of step S2 is as follows: S21, calculate the area ratio of the wall surface finishing repair area and the roof texture repair area to the corresponding building texture area respectively, for the repair area with an area ratio less than or equal to 10%, directly repairing the repair area automatically based on the LAMA image repair model; S22, for the repair area with an area ratio greater than 10% and less than 30%, the current repair area is divided into multiple sub-texture repair areas with an area ratio less than 10% according to the grid, and then the sub-repair area is repaired automatically using the LAMA image repair model; S23, for the repair area with an area ratio greater than or equal to 30%, the PatchMach image repair algorithm is used to repair the wall surface and roof texture; Wherein, the LAMA image repair model directly repairs the texture picture with a resolution size greater than 256x256, without the need to downsample the texture picture first.

6. The method of claim 5, wherein the method of fusing the building texture intelligent repairing and beautifying deep learning model comprises: In the step S4, the specific process of establishing the building texture picture generation beautification model is as follows: S41, select Stable Diffusion diffusion model as the basic model; S42, supplement ControlNet control chart in Stable Diffusion diffusion model; S43, increase positive and negative prompt words for building texture type, style, and building element layout requirements of building texture picture generation to guide the Stable Diffusion diffusion model to generate texture.

7. The method of claim 6, wherein the method of fusing the building texture intelligent inpainting and beautifying deep learning model comprises: The specific process of step S42 is as follows: the structural information of the building texture is extracted through the preprocessor of Canny edge detection and Depth depth estimation, and the control tensor is fused with the U-Net feature map of the Stable Diffusion diffusion model.

Images