Facade texture repairing method and device, electronic equipment and computer storage medium

Abstract

The present application relates to a kind of facade texture repair method, device, electronic equipment and computer storage medium, belong to geographic information surveying and mapping technical field, wherein the facade texture repair method includes: obtaining the multi-view texture image of building facade, and determine the main texture image and the auxiliary texture image in multi-view texture image;With the view angle of main texture image as main view angle, using homography matrix converts auxiliary texture image to main view angle, obtains candidate texture image, and determines the covered area of main texture image and candidate texture image based on the structural similarity of main texture image and candidate texture image;Using the preset texture synthesis cost function to calculate the value of cost when texture synthesis is carried out based on the covered area of main texture image and candidate texture image, determine the texture synthesis mode with minimum cost value, obtain the texture repair image of building facade.The present application can realize the accurate texture repair of building facade.

Description

Technical Field

[0001] This invention relates to the field of geographic information mapping technology, and in particular to a method, apparatus, electronic device, and computer storage medium for facade texture restoration. Background Technology

[0002] In fields such as digital city construction, cultural heritage protection, and Building Information Modeling (BIM) management, high-precision 3D building models are crucial foundational data. Realistic textures, as a key component of 3D models, directly impact their visual appeal and application value. Currently, texture data for 3D building models is primarily acquired through multi-source remote sensing technologies such as UAV oblique photography and terrestrial laser scanning. However, factors such as occlusion from the shooting angle, uneven lighting conditions, and sensor noise often result in issues like missing textures, blurring, misalignment, or color distortion on the model surface, severely limiting the practicality of 3D models.

[0003] Traditional texture restoration techniques primarily rely on manual intervention, using image editing software to manually fill in missing areas. While this method can guarantee a certain level of accuracy, it is inefficient and requires highly skilled operators, making it difficult to meet the automated processing needs of large-scale city-level 3D models. In existing automatic image restoration technologies, whether based on traditional neighborhood matching methods or data-driven deep learning models, the fundamental limitation lies in ignoring the uniqueness of the original content of occluded areas. Traditional methods fill missing areas by searching for local samples, essentially relying on statistical texture inference based on 2D images. Although deep learning models can learn higher-order semantic relationships, the textures they generate are still mean mappings of the training data distribution. Both methods use single-frame images as the basis for restoration, treating occluded areas as unknown random variables, which may lead to restoration results that deviate from the true facade features of buildings.

[0004] This shows that existing texture image restoration techniques cannot accurately restore the true facade features of a building. Summary of the Invention

[0005] In view of this, it is necessary to provide a facade texture restoration method, device, electronic device and computer storage medium to solve the problem that existing texture image restoration technology cannot accurately restore the true facade features of a building.

[0006] To address the aforementioned problems, in a first aspect, the present invention provides a method for repairing facade textures, comprising: Acquire multi-view texture images of the building facade and determine the primary and secondary texture images in the multi-view texture images; Using the perspective of the main texture image as the main perspective, the sub-texture image is transformed to the main perspective using a homography matrix to obtain candidate texture images. The occluded areas of the main texture image and candidate texture images are determined based on the structural similarity between the main texture image and the candidate texture images. The cost of texture synthesis is calculated using a preset texture synthesis cost function to determine the cost of texture synthesis for the occluded area based on the main texture image and candidate texture images. The texture synthesis method with the minimum cost is then determined to obtain the texture restoration image of the building facade.

[0007] In one possible implementation, acquiring multi-view texture images of the building facade and determining the primary and secondary texture images within the multi-view texture images includes: Multi-view images of the building facade are acquired using an image acquisition device, and multi-view texture images are projected based on the camera projection matrix. Determine the line connecting the center point of the image acquisition device to the corresponding image at each viewpoint when acquiring texture images, and the distance between the image acquisition device and the building facade. Calculate the angle between the connecting line and the building facade normal vector, and use the texture image with the smallest angle within the preset distance range as the primary texture image, and the texture images from other perspectives as secondary texture images.

[0008] In one possible implementation, the camera projection matrix is:

[0009] in, K It is a 3×3 intrinsic parameter matrix composed of the intrinsic orientation elements of the texture image. and The principal distance of the interior orientation element. R It is a 3×3 rotation matrix, consisting of the three corner elements of the exterior orientation element in the texture image. t It is a 3×1 offset vector, consisting of three line elements of the exterior orientation element in the texture image. (u, v) This represents the coordinates of a 3D point on a building facade in the pixel coordinate system. This represents the coordinates of a 3D point on the building facade in the camera coordinate system. , and This represents the coordinates of a three-dimensional point on the building facade in the world coordinate system. This indicates the coordinates of a 3D point on the building facade in the camera coordinate system.

[0010] In one possible implementation, the homography matrix is:

[0011] in, For the firsti The homography matrix of a subtexture image converted to the main viewpoint. i Indicates the sequence number of the subtexture image. Indicates the first i The intrinsic parameter matrix corresponding to each subtexture image The intrinsic parameter matrix representing the main texture image. Indicates the first i The rotation matrix that transforms a sub-texture image from its own camera coordinate system to the camera coordinate system of the main texture image. Indicates the first i The translation vector of a sub-texture image from its own camera coordinate system to the camera coordinate system of the main texture image. and Indicates the building facade on the first i The plane equation of the camera coordinate system for each subtexture image. Indicates the number of subtexture images.

[0012] In one possible implementation, determining the occluded regions of the main texture image and candidate texture images based on the structural similarity between the main texture image and the candidate texture image includes: The structural similarity of each pixel in the main texture image and the candidate texture image is calculated using a structural similarity calculation formula, whereby the structural similarity calculation formula is:

[0013]

[0014]

[0015]

[0016] in, Represents the brightness function. This represents a comparison function. This represents the structure function, where x and y represent the pixel values ​​of corresponding pixels in the first and second texture images. This represents the average grayscale value of each pixel in the first texture image. This represents the average grayscale value of each pixel in the second texture image. Let be the covariance of the grayscale values ​​of each pixel in the first texture image and the second texture image. Let be the standard deviation of the grayscale values ​​of each pixel in the first texture image. The standard deviation of the grayscale values ​​of each pixel in the second texture image. , , , , and This is a preset constant; Pixels with structural similarity less than a preset threshold are identified as occluded pixels, and all occluded pixels are identified as occluded regions.

[0017] In one possible implementation, the texture synthesis cost function is:

[0018]

[0019]

[0020] in, E(x) This represents the total energy value. For the energy value of the data item, For smoothing terms and energy terms, x This represents the energy value of the pixel to be synthesized. i , j This indicates the number of the texture image corresponding to the pixel to be synthesized; Indicates the viewpoint priority of the texture image containing the pixel to be synthesized; Indicates the resolution priority of the texture image containing the pixel to be synthesized; This indicates the color priority of the texture image containing the pixel to be synthesized. , p This is a preset constant.

[0021] In one possible implementation, determining the texture synthesis method with the lowest cost to obtain a texture-restored image of the building facade includes: The synthetic texture of the occluded area is determined based on the texture synthesis method with the lowest cost. The synthetic texture of the obscured area and the mask image of the obscured area are used as inputs to a preset image inpainting deep learning model to obtain a textured inpainted image of the building facade.

[0022] Secondly, the present invention also provides a facade texture repair device, comprising: The image acquisition module is used to acquire multi-view texture images of the building facade and determine the main texture image and the sub-texture image in the multi-view texture images; The occlusion region determination module is used to convert the sub-texture image to the main viewpoint using the viewpoint of the main texture image as the main viewpoint, and obtain candidate texture images by using a homography matrix, and determine the occluded regions of the main texture image and the candidate texture images based on the structural similarity between the main texture image and the candidate texture images. The texture restoration module is used to calculate the cost of texture synthesis based on the occluded area of ​​the main texture image and the candidate texture image using a preset texture synthesis cost function, determine the texture synthesis method with the minimum cost, and obtain the texture restoration image of the building facade.

[0023] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the facade texture restoration method described in any of the above embodiments.

[0024] Fourthly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instructions, which, when executed by a processor, can implement the steps in the facade texture restoration method described in any of the above embodiments.

[0025] The beneficial effects of this invention are as follows: The facade texture restoration method provided by this invention acquires multi-view texture images of the building facade and divides these images into primary texture images and secondary texture images to determine the optimal texture image perspective and ensure the accuracy of texture restoration. Using the perspective of the primary texture image as the primary perspective, the secondary texture images are transformed to the primary perspective using a homography matrix to obtain candidate texture images. The occluded areas of the primary and candidate texture images are determined based on the structural similarity between the primary and candidate texture images. By utilizing the structural similarity of the multi-view texture images, the occluded areas of the primary and candidate texture images can be extracted. The occluded areas of the main texture image and candidate texture images are accurately determined. A preset texture synthesis cost function is used to calculate the cost of texture synthesis based on the occluded areas of the main texture image and candidate texture images. The texture synthesis method with the minimum cost is determined to obtain the texture restoration image of the building facade. The texture restoration problem is transformed into the problem of finding the minimum cost, which facilitates the calculation of texture restoration and further improves the accuracy of building facade texture restoration. The invention combines multi-view texture images to restore the texture of the building facade. The invention utilizes the structural similarity of multi-view texture images to effectively restore the true texture of the facade even when the occlusion range of the facade texture is high. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart illustrating a facade texture restoration method provided in an embodiment of the present invention; Figure 2 A flowchart illustrating an implementation method of S101 provided in an embodiment of the present invention; Figure 3 A texture image map provided in an embodiment of the present invention; Figure 4 A flowchart illustrating an implementation method of S103 provided in an embodiment of the present invention; Figure 5 This is a realistic texture synthesis result image provided in an embodiment of the present invention; Figure 6 A comparison diagram showing the results of comparing the actual occlusion area and the actual occlusion extraction area provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of a facade texture repair device provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0028] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0029] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0030] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0031] A specific embodiment of the present invention, such as Figure 1 As shown, a method for restoring facade textures is disclosed, including: S101, acquire multi-view texture images of the building facade, and determine the main texture image and sub-texture image in the multi-view texture images.

[0032] In this embodiment of the invention, a building facade refers to the three-dimensional exterior of a building. Texture images can accurately display the facade appearance of a three-dimensional building in a three-dimensional model. Multi-view texture images of a building facade refer to texture images collected from different perspectives for the same facade of a building. Multi-view texture images can display the appearance of the same facade of a building from different angles. When the facade is obscured, the obscured area of ​​the facade may differ in texture images from different perspectives. For example, in a texture image from one perspective, the lower left corner of the building facade may be obscured by trees, but in a texture image from another perspective, the lower left corner of the building facade may not be obscured. Based on this, texture images from multiple perspectives can be combined to repair the texture of the facade.

[0033] In this embodiment of the invention, for the acquired multi-view texture images, it is necessary to determine the main texture image and the sub-texture image. The main texture image is the texture image with the best viewpoint, which can most accurately and realistically represent the texture of the building facade. The specific method for determining the main texture image and the sub-texture image will be described in detail later in this invention.

[0034] S102, using the perspective of the main texture image as the main perspective, the sub-texture image is transformed to the main perspective using a homography matrix to obtain candidate texture images, and the occluded areas of the main texture image and candidate texture images are determined based on the structural similarity between the main texture image and the candidate texture images.

[0035] In this embodiment of the invention, after determining the main texture image and the sub-texture image, the view of the main texture image is taken as the main view, and all sub-texture images are transformed to the main view to obtain candidate texture images. Specifically, the camera coordinate system of the image acquisition device when acquiring the main texture image can be used as the target to transform the sub-texture images from the camera coordinate system of the sub-texture images to the camera coordinate system of the main texture image. The specific transformation method will be described in detail later in this invention.

[0036] In this embodiment of the invention, after obtaining all candidate texture images, the structural similarity between each pixel in each candidate texture image and the corresponding pixel in the main texture image is calculated, and the occluded regions of the main texture image and candidate texture images are determined based on the structural similarity. The specific calculation method for structural similarity and the method for determining the occluded regions will be described in detail later in this invention.

[0037] S103: The cost of texture synthesis is calculated using a preset texture synthesis cost function to determine the cost of texture synthesis in the occluded area based on the main texture image and candidate texture images. The texture synthesis method with the minimum cost is determined to obtain the texture restoration image of the building facade.

[0038] In this embodiment of the invention, after determining the occluded areas of the main texture image and the candidate texture images, a texture synthesis cost function is constructed, and the texture synthesis cost of the occluded areas of the main texture image and the candidate texture images is calculated based on the texture synthesis cost function. The texture synthesis cost is used to represent the cost of synthesizing the occluded areas using different texture images. For a certain occluded area, it can be synthesized by synthesizing the corresponding areas of the main texture image and the sub-texture image, or it can be synthesized by synthesizing the corresponding areas of two sub-texture images. Therefore, it is necessary to calculate the cost of various synthesis methods to determine the optimal texture synthesis method, and then obtain the texture restoration image of the building facade.

[0039] The facade texture restoration method provided by this invention acquires multi-view texture images of the building facade and divides these images into primary and secondary texture images to determine the optimal texture image perspective and ensure the accuracy of texture restoration. Using the perspective of the primary texture image as the primary viewpoint, the secondary texture image is transformed to the primary viewpoint using a homography matrix to obtain candidate texture images. The occluded areas of the primary and candidate texture images are determined based on their structural similarity. By using the structural similarity of the multi-view texture images, the occluded areas of the primary and candidate texture images can be extracted with relatively high accuracy. The occluded areas of the main texture image and candidate texture images are calculated using a preset texture synthesis cost function. The cost of texture synthesis based on the occluded areas of the main texture image and candidate texture images is then determined to identify the texture synthesis method with the minimum cost, resulting in a texture restoration image of the building facade. This transforms the texture restoration problem into a problem of finding the minimum cost, which facilitates texture restoration calculations and further improves the accuracy of building facade texture restoration. By combining multi-view texture images to restore the texture of the building facade, this invention utilizes the structural similarity of multi-view texture images to effectively restore the true texture of the facade even when the occlusion range of the facade texture is high.

[0040] In some possible embodiments of the present invention, such as Figure 2 As shown, multi-view texture images of the building facade are acquired, and the main texture image and sub-texture image in the multi-view texture image are determined, including: S201 uses an image acquisition device to acquire multi-view images of the building facade, and projects the multi-view images based on the camera projection matrix to obtain multi-view texture images. S202, determine the line connecting the center point of the image acquisition device to the corresponding image when acquiring texture images from various viewpoints, and the distance between the image acquisition device and the building facade. S203, calculate the angle between the connecting line and the building facade normal vector, and use the texture image with the smallest angle within the preset distance range as the main texture image, and the texture images from other perspectives as secondary texture images.

[0041] In this embodiment of the invention, when acquiring multi-view texture images of a building facade, multi-view images of the building facade are acquired using an image acquisition device. This image acquisition device can be a drone equipped with image acquisition equipment, facilitating the acquisition of multi-view images of taller building facades. After acquiring the multi-view images of the building facade, the multi-view images are projected based on a camera projection matrix to obtain a multi-view texture image. The camera projection matrix is ​​as follows:

[0042] in, K It is a 3×3 intrinsic parameter matrix composed of the intrinsic orientation elements of the texture image. and The principal distance of the interior orientation element. R It is a 3×3 rotation matrix, consisting of the three corner elements of the exterior orientation element in the texture image. t It is a 3×1 offset vector, consisting of three line elements of the exterior orientation element in the texture image. (u, v) This represents the coordinates of a 3D point on a building facade in the pixel coordinate system. This represents the coordinates of a 3D point on the building facade in the camera coordinate system. , and This represents the coordinates of a three-dimensional point on the building facade in the world coordinate system. This indicates the coordinates of a 3D point on the building facade in the camera coordinate system.

[0043] In this embodiment of the invention, after obtaining multi-view texture images of the building facade, the center point of the image acquisition device and the corresponding texture image are determined, along with the distance between the image acquisition device and the building facade. Then, the angle between the connecting line and the normal vector of the building facade is calculated. The texture image with the smallest angle within a preset distance range is designated as the primary texture image, and the texture images from the remaining views are designated as secondary texture images. Based on this, the texture image with the best viewing angle can be determined as the primary texture image, ensuring the accuracy of the building facade texture image obtained after texture restoration.

[0044] In some possible embodiments of the present invention, the homography matrix for converting the subtexture image to the main viewpoint is:

[0045] in, For the first i The homography matrix of a subtexture image converted to the main viewpoint. i Indicates the sequence number of the subtexture image. Indicates the first iThe intrinsic parameter matrix corresponding to each subtexture image The intrinsic parameter matrix representing the main texture image. Indicates the first i The rotation matrix that transforms a sub-texture image from its own camera coordinate system to the camera coordinate system of the main texture image. Indicates the first i The translation vector of a sub-texture image from its own camera coordinate system to the camera coordinate system of the main texture image. and Indicates the building facade on the first i The plane equation of the camera coordinate system for each subtexture image. Indicates the number of subtexture images.

[0046] In this embodiment of the invention, due to the different shooting angles of each texture image, the size of the facade texture is not uniform and cannot be directly used for texture synthesis. However, the essence of multi-view texture images is that they are all projections of the same three-dimensional facade. Therefore, the candidate texture image can be converted to the view of the main texture by using the homography matrix as the reference, so as to achieve alignment of the main and candidate texture images and obtain the candidate texture of the best view, laying the foundation for subsequent extraction of the best view occlusion.

[0047] In some possible embodiments of the present invention, determining the occluded regions of the main texture image and the candidate texture image based on the structural similarity between the main texture image and the candidate texture image includes: The structural similarity of each pixel in the main texture image and the candidate texture image is calculated using a structural similarity calculation formula, whereby the structural similarity calculation formula is:

[0048]

[0049]

[0050]

[0051] in, Represents the brightness function. This represents a comparison function. This represents the structure function, where x and y represent the pixel values ​​of corresponding pixels in the first and second texture images. This represents the average grayscale value of each pixel in the first texture image. This represents the average grayscale value of each pixel in the second texture image. Let be the covariance of the grayscale values ​​of each pixel in the first texture image and the second texture image. Let be the standard deviation of the grayscale values ​​of each pixel in the first texture image. The standard deviation of the grayscale values ​​of each pixel in the second texture image. , , , , and This is a preset constant; Pixels with structural similarity less than a preset threshold are identified as occluded pixels, and all occluded pixels are identified as occluded regions.

[0052] In embodiments of the present invention, such as Figure 3 As shown, under the premise that all facade textures maintain the same viewpoint and size, the occluded areas of the main texture and candidate textures are extracted using structural similarity. Structural similarity is usually used to evaluate the similarity between two images. When the structural similarity value is low, it indicates that there is occlusion between any two images. By calculating the structural similarity between the main texture image and multiple candidate texture images, the occluded areas affected by the main texture can be extracted. Similarly, the occluded and unoccluded areas of the candidate textures can also be extracted.

[0053] In some possible embodiments of the present invention, the texture synthesis cost function is:

[0054]

[0055]

[0056] in, E(x) This represents the total energy value. For the energy value of the data item, For smoothing terms and energy terms, x This represents the energy value of the pixel to be synthesized. i , j This indicates the number of the texture image corresponding to the pixel to be synthesized; Indicates the viewpoint priority of the texture image containing the pixel to be synthesized; Indicates the resolution priority of the texture image containing the pixel to be synthesized; This indicates the color priority of the texture image containing the pixel to be synthesized. , p This is a preset constant.

[0057] In this embodiment of the invention, the synthesis cost function is used to calculate the total energy value of texture synthesis. This total energy value includes data term energy value and smoothing term energy value. The data term is mainly determined by the texture image quality. In order to synthesize unobstructed and high-quality facade textures as much as possible, the energy term is constructed by fully considering factors such as viewing angle, resolution, and color. This indicates the viewpoint priority of the texture image where the pixel to be synthesized is located. The closer the viewpoint is to the main texture, the smaller the energy value. This indicates the resolution priority of the texture image containing the pixel to be synthesized; the higher the resolution of the source texture, the lower the energy value. The energy value represents the color priority of the texture image containing the pixel to be synthesized; the smaller the color difference with the main texture, the lower the energy value. The smoothing term evaluates the spatial consistency of the synthesized texture, avoiding excessive fragmentation and obvious texture misalignment. During the solution process, adjacent pixels from the same image are preferentially selected. This invention includes a penalty term in the smoothing term: if adjacent pixel textures come from the same texture image, the penalty value is 0; otherwise, it is p.

[0058] In some possible embodiments of the present invention, such as Figure 4 As shown, the texture synthesis method with the lowest cost is determined to obtain the texture restoration image of the building facade, including: S401, determine the synthesized texture of the occluded area based on the texture synthesis method with the lowest cost; S402, the synthesized texture of the covered area and the mask image of the covered area are used as inputs to the preset image inpainting deep learning model to obtain the texture inpainted image of the building facade.

[0059] In this embodiment of the invention, the synthesized texture restores the occluded area of ​​the main texture image from the best viewpoint to the greatest extent. However, due to the influence of factors such as differences in shooting angles, lighting factors, and systematic errors in different texture images, the synthesized texture has certain unavoidable occlusion, inconsistent tones, and texture misalignment, requiring further texture restoration. The image restoration deep learning model can be a generative adversarial network-based image restoration model (such as the Deepfill model), which uses gated convolution instead of traditional convolution to dynamically learn the features of the missing area, extract features from the known areas of the image, and generate the content of the area to be restored, ensuring that the restored area and the surrounding environment are consistent in semantics and texture. The model is mainly divided into two stages: the coarse restoration stage generates a coarse structure of the area to be restored, and the fine restoration stage optimizes details and improves the texture quality of the restored area. The mask image of the occluded area and the synthesized texture are used as the input data of the model, and the output is the complete restored facade texture.

[0060] In a specific embodiment of this invention, using a real-world 3D model of a city as the data source, an experimental dataset containing different types of occlusion, such as eaves, billboards, auxiliary building structures, and buildings, was constructed. XML files obtained through aerial triangulation were used to parse the interior and exterior orientation elements of the UAV tilted imagery. The coordinates of the four vertices of the building facade were projected onto the UAV tilted imagery to obtain a visible facade texture image dataset. Therefore, each facade dataset contains 3D facade data, a visible facade texture image dataset, and an XML file containing the corresponding interior and exterior orientation elements. Regarding parameter settings, for texture alignment, one texture image with the best viewing angle and five candidate texture images with the closest viewing angles were selected. When using structural similarity for occlusion extraction, , , , The window size is 7; during the texture synthesis process, , In texture restoration, a pre-trained model was used, and targeted texture image training samples were added to optimize it. Occlusion ranges in the original best-view texture were extracted using structural similarity, and real textures were synthesized from the visible regions of multi-view texture images. Finally, deep learning was used for real texture restoration, and the results are as follows: Figure 5 As shown, the realistic texture synthesis result basically eliminates the occlusion of the facade texture. Analysis of the texture occlusion extraction result shows that the area includes not only the area that is actually occluded, but also the area that is not actually occluded. Figure 6 The comparison between the actual occlusion area and the extracted occlusion area is shown. In some cases, the actual texture occlusion range reaches 44.73%, severely impacting the visibility of the 3D model. The excess occlusion range is below 20%, and these pseudo-occluded areas are mainly optimized during texture synthesis to eliminate their impact. While some occlusion still exists in facade textures a and c, the effect is significantly improved after texture restoration.

[0061] The texture synthesis in this embodiment of the invention restores the actual texture of the facade to the greatest extent, which provides richer information for texture restoration. Due to differences in shooting time and errors of multi-view images, the texture synthesis results still have phenomena such as inconsistent color tone and texture misalignment. However, after the uniform lighting and color matching and optimization of texture restoration, the color difference transition is more natural in visual effect, and at the same time, it further eliminates the occlusion of textures to a certain extent.

[0062] To better implement the facade texture restoration method in this embodiment of the invention, based on the facade texture restoration method, correspondingly, as follows: Figure 7 As shown, this embodiment of the invention also provides a facade texture repair device, the facade texture repair device 700 comprising: The image acquisition module 701 is used to acquire multi-view texture images of the building facade and determine the main texture image and sub-texture image in the multi-view texture images. The occlusion region determination module 702 is used to convert the sub-texture image to the main viewpoint using the viewpoint of the main texture image as the main viewpoint, and obtain the candidate texture image by using the homography matrix. Based on the structural similarity between the main texture image and the candidate texture image, the occlusion region of the main texture image and the candidate texture image is determined. The texture restoration module 703 is used to calculate the cost of texture synthesis based on the occluded area of ​​the main texture image and the candidate texture image using a preset texture synthesis cost function, determine the texture synthesis method with the minimum cost, and obtain the texture restoration image of the building facade.

[0063] The facade texture repair device 700 provided in the above embodiments can realize the technical solutions described in the above facade texture repair method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above facade texture repair method embodiments, and will not be repeated here.

[0064] like Figure 8 As shown, the present invention also provides an electronic device 800. The electronic device 800 includes a processor 801, a memory 802, and a display 803. Figure 8 Only some components of the electronic device 800 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0065] In some embodiments, processor 801 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in memory 802 or process data, such as the facade texture restoration method of the present invention.

[0066] In some embodiments, processor 801 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 801 may be local or remote. In some embodiments, processor 801 may be implemented on a cloud platform. In some embodiments, the cloud platform may include private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, internal cloud, multi-cloud, etc., or any combination thereof.

[0067] In some embodiments, memory 802 may be an internal storage unit of electronic device 800, such as a hard disk or memory of electronic device 800. In other embodiments, memory 802 may also be an external storage device of electronic device 800, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 800.

[0068] Furthermore, the memory 802 may include both internal storage units of the electronic device 800 and external storage devices. The memory 802 is used to store application software and various types of data installed on the electronic device 800.

[0069] In some embodiments, display 803 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 803 is used to display information from electronic device 800 and to display a visual user interface. Components 801-803 of electronic device 800 communicate with each other via a system bus.

[0070] In some embodiments, when the processor 801 executes the facade texture repair program in the memory 802, the following steps may be performed: Acquire multi-view texture images of the building facade and determine the primary and secondary texture images in the multi-view texture images; Using the perspective of the main texture image as the main perspective, the sub-texture image is transformed to the main perspective using a homography matrix to obtain candidate texture images. The occluded areas of the main texture image and candidate texture images are determined based on the structural similarity between the main texture image and the candidate texture images. The cost of texture synthesis is calculated using a preset texture synthesis cost function to determine the cost of texture synthesis for the occluded area based on the main texture image and candidate texture images. The texture synthesis method with the minimum cost is then determined to obtain the texture restoration image of the building facade.

[0071] It should be understood that when the processor 801 executes the facade texture repair program in the memory 802, in addition to the functions mentioned above, it can also perform other functions, as can be found in the description of the corresponding method embodiments above.

[0072] Furthermore, this embodiment of the invention does not specifically limit the type of electronic device 800 mentioned. Electronic device 800 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the invention, electronic device 800 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0073] Accordingly, this application also provides a computer-readable storage medium for storing computer-readable programs or instructions. When the programs or instructions are executed by a processor, they can implement the steps or functions of the facade texture restoration methods provided in the above-described method embodiments.

[0074] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0075] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for restoring facade texture, characterized in that, include: Acquire multi-view texture images of the building facade and determine the main texture image and sub-texture image in the multi-view texture images; Using the perspective of the main texture image as the main perspective, the sub-texture image is transformed to the main perspective using a homography matrix to obtain a candidate texture image, and the occluded areas of the main texture image and the candidate texture image are determined based on the structural similarity between the main texture image and the candidate texture image. The cost of texture synthesis based on the occluded area of ​​the main texture image and the candidate texture image is calculated using a preset texture synthesis cost function. The texture synthesis method with the minimum cost value is determined to obtain the texture restoration image of the building facade. The step of determining the occluded regions of the main texture image and the candidate texture image based on the structural similarity between the main texture image and the candidate texture image includes: The structural similarity of each pixel in the main texture image and the candidate texture image is calculated using a structural similarity calculation formula, wherein the structural similarity calculation formula is: in, Represents the brightness function. This represents a comparison function. This represents the structure function, where x and y represent the pixel values ​​of corresponding pixels in the first and second texture images. This represents the average grayscale value of each pixel in the first texture image. This represents the average grayscale value of each pixel in the second texture image. Let be the covariance of the grayscale values ​​of each pixel in the first texture image and the second texture image. Let be the standard deviation of the grayscale values ​​of each pixel in the first texture image. The standard deviation of the grayscale values ​​of each pixel in the second texture image. , , , , and This is a preset constant; Pixels with structural similarity less than a preset threshold are identified as occluded pixels, and all occluded pixels are identified as occluded regions. The texture synthesis cost function is: Where E(x) is the total energy value, For the energy value of the data item, For the smoothing energy term, x represents the energy value of the pixel to be synthesized, and i and j represent the texture image numbers corresponding to the pixel to be synthesized. Indicates the viewpoint priority of the texture image containing the pixel to be synthesized; Indicates the resolution priority of the texture image containing the pixel to be synthesized; This indicates the color priority of the texture image containing the pixel to be synthesized. p are preset constants; The process of acquiring multi-view texture images of the building facade and determining the primary and secondary texture images within the multi-view texture images includes: Multi-view images of the building facade are acquired using an image acquisition device, and multi-view texture images are projected based on a camera projection matrix. Determine the line connecting the center point of the image acquisition device to the corresponding viewpoint texture image when acquiring texture images from each viewpoint, and the distance between the image acquisition device and the building facade; Calculate the angle between the connecting line and the normal vector of the building facade, and take the texture image with the smallest angle within the preset distance range as the main texture image, and the texture images from other perspectives as secondary texture images. The camera projection matrix is: Where K is a 3×3 intrinsic parameter matrix, composed of the intrinsic orientation elements of the texture image. and Let R be the principal distance of the interior orientation element, R be a 3×3 rotation matrix composed of the three corner elements of the exterior orientation element in the texture image, t be a 3×1 offset vector composed of the three line elements of the exterior orientation element in the texture image, and (u, v) represent the coordinates of a 3D point in the building facade in the pixel coordinate system. This represents the coordinates of a 3D point on the building facade in the camera coordinate system. , and This represents the coordinates of a three-dimensional point on the building facade in the world coordinate system. This indicates the coordinates of a 3D point on the building facade in the camera coordinate system.

2. The facade texture restoration method according to claim 1, characterized in that, The homography matrix is: in, Let be the homography matrix of the i-th subtexture image transformed to the main viewpoint, where i represents the index of the subtexture image. This represents the intrinsic parameter matrix corresponding to the i-th subtexture image. The intrinsic parameter matrix representing the main texture image. Let represent the rotation matrix that transforms the i-th sub-texture image from its own camera coordinate system to the camera coordinate system of the main texture image. Let represent the translation vector by which the i-th sub-texture image is transformed from its own camera coordinate system to the camera coordinate system of the main texture image. and Let represent the plane equation of the building facade in the camera coordinate system for the i-th subtexture image. Indicates the number of subtexture images.

3. The facade texture restoration method according to claim 1, characterized in that, The process of determining the texture synthesis method with the lowest cost to obtain the texture-restored image of the building facade includes: The synthesized texture of the occluded area is determined based on the texture synthesis method with the lowest cost. The synthesized texture of the covered area and the mask image of the covered area are used as inputs to a preset image inpainting deep learning model to obtain the texture-inpainted image of the building facade.

4. A facade texture repair device, applicable to the facade texture repair method according to any one of claims 1 to 3, characterized in that, include: The image acquisition module is used to acquire multi-view texture images of the building facade and determine the main texture image and the sub-texture image in the multi-view texture images; The occlusion region determination module is used to convert the sub-texture image to the main viewpoint using the viewpoint of the main texture image as the main viewpoint, and obtain candidate texture images by using a homography matrix, and determine the occluded regions of the main texture image and the candidate texture images based on the structural similarity between the main texture image and the candidate texture images. The texture restoration module is used to calculate the cost of texture synthesis based on the occluded area of ​​the main texture image and the candidate texture image using a preset texture synthesis cost function, determine the texture synthesis method with the minimum cost, and obtain the texture restoration image of the building facade.

5. An electronic device, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the facade texture restoration method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the facade texture restoration method according to any one of claims 1 to 3.

Images