A method, system, device and storage medium for reconstructing building texture
By using differentiable rendering and gradient optimization algorithms in building texture reconstruction, the problems of low efficiency and low quality of building texture reconstruction in the existing technology are solved, and high-quality building texture maps are achieved, which improves visual effects and consistency.
Patent Information
- Application Number
- CN202411290772.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2044-09-14
AI Technical Summary
In the prior art, building texture reconstruction is inefficient and low in quality, especially obtaining high-quality building textures in pictures taken by drones is a challenging problem in three-dimensional reconstruction.
By obtaining the building agent model for plane parameterization, a mapping function between the texture texture pixels and real photo pixels is established, and the weight vector is optimized to generate high-quality texture maps using differentiable rendering and gradient optimization algorithms.
High-quality building texture reconstruction is achieved, the luminosity and content consistency of texture maps is improved, blur and ghosting problems are reduced, and visual effects are improved.
Smart Images
Figure CN118799474B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of building texture reconstruction, and in particular to a building texture reconstruction method, system, device and storage medium. Background Art
[0002] Acquiring high-quality building textures from a large number of drone images is an important last step in the 3D reconstruction task. Usually, a large number of drones are used to shoot the same building from different angles, and then the surface texture of the building in the image is mapped to the 3D model to obtain a 3D building model with texture. Rich texture information can make up for the lack of geometric details and improve the visual effect. Therefore, it is a challenging task to maintain the rich texture information in the original dense reconstruction. How to efficiently and high-quality reconstruct the texture map of the building proxy geometric model from a large number of disordered RGB images is a problem that needs to be solved.
[0003] Early works on image-based texture reconstruction usually select multiple views for each patch and blend them into the texture by using different weighting strategies. However, these methods may produce blurry and ghosting results due to camera calibration errors and inaccurate geometric structures. To reduce the artifacts caused by multi-image blending, later methods select only one view for each patch, and use different data terms and smoothing terms to construct Markov random fields to select the best view for each patch to solve the discrete labeling problem. However, these methods still have the problem of seam misalignment. Block-based optimization methods reduce such artifacts to a certain extent, but it may produce blurry textures. There is also the option of generating architectural proxy texture maps in a plane-based manner. To generate a high-quality texture map for each structural polygon of the input proxy, the following sub-steps need to be performed: image selection, affine transformation, rigid registration, image stitching, and image blending provide a suitable framework for the production of architectural proxy textures with high perspective and photometric quality. However, it contains carefully designed sub-steps and a large number of adjusted parameters.
[0004] In recent years, the research on 3D reconstruction based on deep learning has been hot. The 3D reconstruction method based on Neural Radiance Fields (NeRF) can create continuous volumetric scenes by training a neural network to realize the mapping from coordinate space to color and density, but its training efficiency is low. Therefore, based on the neural radiation field, a 3D reconstruction method based on 3D Gaussian Splatting (3DGS) has emerged. 3DGS is not only an incremental improvement, but also a paradigm shift. 3DGS combines the differentiable rendering pipeline and point-based rendering technology to represent the scene through a learnable 3D Gaussian function. However, they all directly generate scene rendering, which is incompatible with the traditional rendering pipeline, and this representation still has large visual defects for close details. The final quality of texture maps reconstructed by traditional texture reconstruction methods still depends largely on the reconstruction quality and the accuracy of camera calibration and requires multi-step processing; the current deep learning-based 3D reconstruction method is based on the radiation field, which converts the scene expression into a learnable implicit or explicit expression and learns to render the scene through a neural network, so it is generally incompatible with the traditional rendering pipeline.
[0005] Therefore, the prior art needs to be improved. Summary of the invention
[0006] The technical problem to be solved by the present invention is to provide a method, system, device and storage medium for reconstructing building textures in view of the above-mentioned defects of the prior art, aiming to solve the problems of low efficiency and low quality of building texture reconstruction in the prior art.
[0007] The technical solution adopted by the present invention to solve the problem is as follows:
[0008] In a first aspect, an embodiment of the present invention provides a method for reconstructing building textures, wherein the method comprises:
[0009] Acquire a building proxy model, and perform plane parameterization on the building proxy model to obtain a building plane texture atlas; the building plane texture atlas includes a plurality of texture map texels;
[0010] Acquire a real photo pixel set, and establish a mapping function between each texture map texel in the building plane texture atlas and each real photo pixel in the real photo pixel set;
[0011] Obtaining an initial weight vector, and performing differentiable rendering on the texture map texel based on the initial weight vector and the mapping function to obtain a differentiable rendering image;
[0012] Determining weight gradient information based on the differentiable rendering image and real photo pixels, and performing gradient optimization on the initial weight vector using a gradient optimization algorithm based on the weight gradient information to obtain a target weight vector;
[0013] A target texture map is obtained based on the texture map texels and the target weight vector.
[0014] In one implementation, the step of establishing a mapping function between each texture texel in the building plane texture atlas and each real photo pixel in the real photo pixel set includes:
[0015] Get the real camera parameters and virtual camera parameters;
[0016] A mapping function between each texture texel in the building plane texture atlas and each real photo pixel in the real photo pixel set is determined based on the real camera parameters and the virtual camera parameters.
[0017] In one implementation, performing differentiable rendering on the texture map texels based on the initial weight vector and the mapping function to obtain a differentiable rendering image includes:
[0018] Determine, based on the mapping function and the texture map texel, a color value of a real photo pixel corresponding to the texture map texel;
[0019] A differentiable rendering image is determined based on the initial weight vector and the color values of the real photo pixels.
[0020] In one implementation, the step of performing gradient optimization on the initial weight vector using a gradient optimization algorithm includes:
[0021] Get the mask;
[0022] A rendering loss term is determined based on the mask, the real photo pixels, and the differentiable rendering map.
[0023] In one implementation, the step of performing gradient optimization on the initial weight vector using a gradient optimization algorithm includes:
[0024] Obtaining the viewing angle area of the real photo pixel;
[0025] A perspective loss term is determined based on the texture map texels, the mask, and the viewing area of the real photo pixels.
[0026] In one implementation, the step of performing gradient optimization on the initial weight vector using a gradient optimization algorithm includes:
[0027] Acquire a real photo neighborhood pixel, and determine a color value difference based on the real photo pixel and the real photo neighborhood pixel;
[0028] Obtaining a neighborhood weight vector, and determining a weight difference based on the initial weight vector and the neighborhood weight vector;
[0029] A parameter loss term is determined based on the mask, the color value difference, and the weight difference.
[0030] In one implementation, the step of performing gradient optimization on the initial weight vector using a gradient optimization algorithm includes:
[0031] Obtaining hyperparameters, where the hyperparameters include a first hyperparameter, a second hyperparameter, and a third hyperparameter;
[0032] A loss function is determined based on the hyperparameters, the rendering loss term, the perspective loss term, and the parameter loss term.
[0033] In a second aspect, an embodiment of the present invention further provides a building texture reconstruction system, wherein the system comprises:
[0034] A model planarization module is used to obtain a building proxy model and perform plane parameterization on the building proxy model to obtain a building plane texture atlas; the building plane texture atlas includes a plurality of texture map texels;
[0035] A mapping function establishment module, used for acquiring a real photo pixel set, and establishing a mapping function between each texture map texel in the building plane texture atlas and each real photo pixel in the real photo pixel set;
[0036] A differentiable rendering module, used to obtain an initial weight vector, and perform differentiable rendering on the texture map texel based on the initial weight vector and a mapping function to obtain a differentiable rendering image;
[0037] A weight optimization module, used to determine weight gradient information based on the differentiable rendering image and real photo pixels, and to perform gradient optimization on the initial weight vector using a gradient optimization algorithm based on the weight gradient information to obtain a target weight vector;
[0038] A texture reconstruction module is used to obtain a target texture map based on the texture map texel and the target weight vector.
[0039] In one implementation, the mapping function establishment module includes:
[0040] A parameter acquisition unit, used to acquire real camera parameters and virtual camera parameters;
[0041] A function relationship establishing unit is used to determine a mapping function between each texture texel in the building plane texture atlas and each real photo pixel in the real photo pixel set based on the real camera parameters and the virtual camera parameters.
[0042] In one implementation, the differentiable rendering module includes:
[0043] A color value determination unit, configured to determine a color value of a real photo pixel corresponding to the texture map texel based on the mapping function and the texture map texel;
[0044] A differentiable rendering unit is used to determine a differentiable rendering image based on the initial weight vector and the color value of the pixel of the real photo.
[0045] In one implementation, the weight optimization module includes:
[0046] A mask acquisition unit, used for acquiring a mask;
[0047] A rendering loss item acquisition unit is used to determine a rendering loss item based on the mask, real photo pixels and the differentiable rendering image.
[0048] In one implementation, the weight optimization module includes:
[0049] A viewing angle region acquisition unit, used to acquire the viewing angle region of the real photo pixel;
[0050] A perspective loss item determination unit is used to determine the perspective loss item based on the texture map texels, the mask and the viewing angle area of the real photo pixels.
[0051] In one implementation, the weight optimization module includes:
[0052] a color value difference determining unit, configured to obtain a real photo neighborhood pixel, and determine a color value difference based on the real photo pixel and the real photo neighborhood pixel;
[0053] A weight value difference determining unit, configured to obtain a neighborhood weight vector and determine a weight difference based on the initial weight vector and the neighborhood weight vector;
[0054] A parameter loss term determination unit is used to determine a parameter loss term based on the mask, the color value difference and the weight difference.
[0055] In one implementation, the weight optimization module includes:
[0056] A hyperparameter acquisition unit, used to acquire hyperparameters, wherein the hyperparameters include a first hyperparameter, a second hyperparameter, and a third hyperparameter;
[0057] A parameter loss term determination unit is used to determine a loss function based on the hyperparameter, the rendering loss term, the perspective loss term and the parameter loss term.
[0058] In the third aspect, an embodiment of the present invention further provides a terminal device, wherein the terminal device is a commercial display terminal, the terminal device includes a memory, a processor, and a program of a building texture reconstruction method stored in the memory and executable on the processor, and when the processor executes the program of the building texture reconstruction method, the steps of the building texture reconstruction method of any one of the above-mentioned schemes are implemented.
[0059] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein a program of a building texture reconstruction method is stored on the computer-readable storage medium, and when the program of the building texture reconstruction method is executed by a processor, the steps of the building texture reconstruction method described in any one of the above-mentioned schemes are implemented.
[0060] Beneficial effects of the present invention: Compared with the prior art, the present invention provides a method, system, device and storage medium for reconstructing building textures. The present invention first obtains a building proxy model, and plane parameterizes the building proxy model to obtain a building plane texture atlas; the building plane texture atlas contains a number of texture map texels; then obtains a real photo pixel set, and establishes a mapping function between each texture map texel in the building plane texture atlas and each real photo pixel in the real photo pixel set; further obtains an initial weight vector, and performs differentiable rendering on the texture map texel based on the initial weight vector and the mapping function to obtain a differentiable rendering image; then determines weight gradient information based on the differentiable rendering image and the real photo pixel, and performs gradient optimization on the initial weight vector based on the weight gradient information to obtain a target weight vector; finally, obtains a target texture map based on the texture map texel and the target weight vector. The present invention performs building texture reconstruction through differentiable rendering, optimizes and reconstructs considering the mapping relationship between texture map texels and real photo pixels, and obtains high-quality building texture maps. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0062] Figure 1 It is a flowchart of a building texture reconstruction method provided by an embodiment of the present invention.
[0063] Figure 2 It is a mapping relationship diagram between texture map texels of a building proxy model and input real photo pixels in the building texture reconstruction method provided by an embodiment of the present invention.
[0064] Figure 3 It is a schematic diagram of visualizing intermediate texture results of iterative optimization of the building texture reconstruction method provided by an embodiment of the present invention.
[0065] Figure 4 It is a schematic diagram of rendering loss items of the building texture reconstruction method provided by an embodiment of the present invention.
[0066] Figure 5 It is a correspondence diagram of a real photo of a plane, a projected photo, and a mask in the architectural texture reconstruction method provided by an embodiment of the present invention.
[0067] Figure 6 It is an experimental effect diagram of the building texture reconstruction method provided by an embodiment of the present invention.
[0068] Figure 7 It is an overall framework diagram of building texture reconstruction of the building texture reconstruction method provided by an embodiment of the present invention.
[0069] Figure 8 It is a principle block diagram of a building texture reconstruction system provided by an embodiment of the present invention.
[0070] Fig. 9 It is a block diagram of the internal structure principle of the building texture reconstruction device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0071] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0072] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.
[0073] The present embodiment provides a method for reconstructing building textures, and the method can be used to reconstruct high-quality building texture maps. In specific implementation, first, a building proxy model is obtained, and the building proxy model is plane parameterized to obtain a building plane texture atlas; the building plane texture atlas contains a number of texture map texels; then a real photo pixel set is obtained, and a mapping function between each texture map texel in the building plane texture atlas and each real photo pixel in the real photo pixel set is established; further, an initial weight vector is obtained, and based on the initial weight vector and the mapping function, the texture map texel is differentiably rendered to obtain a differentiable rendering map; then, based on the differentiable rendering map and the real photo pixel, weight gradient information is determined, and based on the weight gradient information, the initial weight vector is gradient optimized using a gradient optimization algorithm to obtain a target weight vector; finally, a target texture map is obtained based on the texture map texel and the target weight vector. The present invention uses differentiable rendering to reconstruct building textures, considers the mapping relationship between texture map texels and real photo pixels for optimization and reconstruction, and obtains high-quality building texture maps.
[0074] For example, to reconstruct a building texture, first obtain a building proxy model, and perform plane parameterization on the building proxy model to obtain a building plane texture atlas; the building plane texture atlas contains a number of texture map texels; then obtain a real photo pixel set, and establish a mapping function between each texture map texel in the building plane texture atlas and each real photo pixel in the real photo pixel set; further obtain an initial weight vector, and perform differentiable rendering on the texture map texel based on the initial weight vector and the mapping function to obtain a differentiable rendering image; then determine the weight gradient information based on the differentiable rendering image and the real photo pixel, and use a gradient optimization algorithm to perform gradient optimization on the initial weight vector based on the weight gradient information to obtain a target weight vector; finally, obtain a target texture map based on the texture map texel and the target weight vector. The present invention uses differentiable rendering to reconstruct building textures, considers the mapping relationship between texture map texels and real photo pixels for optimized reconstruction, and obtains high-quality building texture maps.
[0075] Exemplary Methods
[0076] This embodiment provides a method for reconstructing building textures. The method can be applied to a terminal device for reconstructing building textures. Figure 1 As shown, the method includes:
[0077] Step S100, obtaining a building proxy model, and performing plane parameterization on the building proxy model to obtain a building plane texture atlas; the building plane texture atlas includes a plurality of texture map texels.
[0078] In this embodiment, to reconstruct the texture of a three-dimensional building proxy model, first, the three-dimensional building proxy model is obtained, and then the building proxy model is plane parameterized to construct a building plane texture atlas. ,in Represents the i-th building plane texture map. Each plane texture map contains a number of texture map texels. , where uv represents the texture coordinates corresponding to the pixel.
[0079] Step S200, obtaining a real photo pixel set, and establishing a mapping function between each texture texel in the building plane texture atlas and each real photo pixel in the real photo pixel set.
[0080] In this embodiment, it is necessary to obtain a real photo pixel set and construct a functional relationship between it and the texture map texels after planarization in step S100. First, obtain the real photo pixel set , Represents a real photo of the corresponding perspective, which contains a number of real photo pixels , where uv represents the texture coordinates corresponding to the pixel. After the real photo pixel set is obtained, the functional relationship between the real photo pixels and the texture map texels can be established. In a specific implementation, the establishment of a mapping function between each texture map texel in the building plane texture atlas and each real photo pixel in the real photo pixel set includes: obtaining real camera parameters and virtual camera parameters; determining the mapping function between each texture map texel in the building plane texture atlas and each real photo pixel in the real photo pixel set based on the real camera parameters and the virtual camera parameters. Establishing real photo pixels and texture texels The mapping relationship between them requires two sets of camera parameters, one is the real camera parameters used when the drone collects photos of real scenes. The other is a virtual camera placed facing the entire plane for each building plane that needs to generate a texture map. Each texture map texel corresponds to a virtual camera parameter , that is to say, the real camera parameters and virtual camera parameters are obtained. For a given structured building proxy model, a mapping function based on the virtual camera parameters and the respective real camera parameters can be obtained: ,in, is the inverse transformation of the real camera parameters used, such as Figure 2 The figure shows the mapping relationship between the texture map texels of the building proxy model and the pixels of the input real photo.
[0081] Step S300: Obtain an initial weight vector, and perform differentiable rendering on the texture map texel based on the initial weight vector and a mapping function to obtain a differentiable rendering image.
[0082] In this embodiment, once the mapping relationship is constructed, it can be input into the differentiable renderer for differentiable rendering. First, the initial weight vector , is the 3D vector corresponding to the pixel of the first real photo, is the 3D vector corresponding to the pixel of the kth real photo, ,in, represents the weight vector at the i-th iteration. In a specific implementation, the differentiable rendering of the texture map texel based on the initial weight vector and the mapping function to obtain a differentiable rendering image includes: determining the color value of the real photo pixel corresponding to the texture map texel based on the mapping function and the texture map texel; determining the differentiable rendering image based on the initial weight vector and the color value of the real photo pixel. The color value of the real photo pixel is determined based on the mapping function and the texture map texel. The function is obtained, where Texel The pixel corresponding to the kth input photo, is the real camera parameter corresponding to the k-th input photo, is the inverse transformation of the virtual camera parameters corresponding to the plane, is the texel corresponding to the i-th building plane texture map, and then based on the initial weight vector and real photo color values Get the building plane texture map , where the color value on the building plane texture map is given by get, is a 3D vector corresponding to the pixel of the kth real photo, and the range of each weight is [0, 1]. Indicates that the input real photo pixels are open.
[0083] Step S400: determining weight gradient information based on the differentiable rendering image and real photo pixels, and performing gradient optimization on the initial weight vector using a gradient optimization algorithm based on the weight gradient information to obtain a target weight vector.
[0084] In this embodiment, the weight gradient information is determined based on the differentiable rendering image and the real photo pixels, and then the initial weight is gradient optimized using the gradient optimization algorithm. For each texture map, the pixels on each real photo are updated through gradient descent optimization. Weight , and then reassemble to get the corresponding building plane texture map texel Color value , until the texels of the entire texture map are filled, which is the result of an iteration of the texture map. At the kth iteration, the visualization of the texture intermediate process is as follows Figure 3 As shown, a visualization diagram of the intermediate texture result of the Kth iterative optimization.
[0085] In the specific implementation, a gradient descent optimizer, such as SGD and Adam optimizer, is used to obtain the rendering result according to the differentiable renderer and the loss function. The gradient of the input parameters is optimized by gradient descent, that is, the above weight vector set After optimization and multiple iterations, the final building plane texture map is obtained. During the optimization process, a threshold is set according to the zero-value weight vector after optimization of the current viewing angle. If the ratio of the zero-value weight to all weights is greater than 0.4, the current photo is eliminated. Next, the pixels with the largest weight values on each building plane texture map texel are optimized at the same time. k decreases with the number of iterations until it reaches 1. During the iteration process, low-quality photos and pixels are eliminated one by one until the last key pixel is left.
[0086] In one implementation, the use of a gradient optimization algorithm to perform gradient optimization on the initial weight vector includes: obtaining hyperparameters, the hyperparameters including a first hyperparameter, a second hyperparameter, and a third hyperparameter; and determining a loss function based on the hyperparameters, a rendering loss term, a perspective loss term, and a parameter loss term. The texture generation process can be understood as optimizing the weights between all texture map texels and real photo pixels. In the process of minimizing the quality of the reconstructed texture map, the loss function is used to measure the quality of the reconstructed texture map. The loss function in the gradient optimization process includes rendering loss, perspective loss, and parameter loss. In the specific implementation, the hyper parameters are obtained, including , , , are all adjustable hyperparameters, among which is the first hyperparameter, is the second hyperparameter, is the third hyperparameter, according to The loss function is calculated. The initial weights are iteratively optimized according to the loss function to obtain the target weights.
[0087] In one implementation, the step of using a gradient optimization algorithm to perform gradient optimization on the initial weight vector includes: obtaining a mask; and determining a rendering loss term based on the mask, real photo pixels, and a differentiable rendering image. It is defined as the sum of the square differences of the pixel color values on the mask area of k renderings, which is used to measure the visual difference between the rendering and all real photo pixels in the visible part of the rendering result. represents the mask of the k-th real photo pixel, represents the kth real photo pixel, represents the pixel of the rendered photo corresponding to the kth real photo. The rendering loss term is the basic loss function, which is mainly used to minimize the gap between the rendering result and the real photo, such as Figure 4 shown.
[0088] In one implementation, the step of using a gradient optimization algorithm to perform gradient optimization on the initial weight vector includes: determining a perspective loss term based on the perspective area of the texture map texels, the mask, and the real photo pixels. Ensure that the front-facing and viewing-direction-consistent photo pixels are selected to penalize the pixel weights that deviate too much from the view angle and the plane normal. Where k represents the kth real photo, is the weight vector of the i-th building plane texture map corresponding to the k-th real photo, , is the mask area of the projection view, It is real photo pixel The viewing area, Real photo pixels Corresponding to the reverse viewing direction of the photo, is the normal to plane i and real photo pixels The average vector of the average viewing direction of It is real photo pixel direction of sight.
[0089] In one implementation, the use of a gradient optimization algorithm to perform gradient optimization on the initial weight vector includes: obtaining real photo neighborhood pixels, and determining color value differences based on the real photo pixels and the real photo neighborhood pixels; obtaining a neighborhood weight vector, and determining weight differences based on the initial weight vector and the neighborhood weight vector; and determining a parameter loss term based on the mask, color value differences, and weight differences. In a specific implementation, the parameter loss term Ensure that the parameter field remains smooth in areas of consistent photometric quality. Since independently optimizing the parameters of neighboring pixels will ignore the consistency of texture content and introduce noise, ensuring that neighboring texels come from similar photos can maintain the consistency of texture content. Parameter loss term Consider the current texel and a neighborhood The difference in color values of the pixels that make up the corresponding texel And the difference in weight values , to penalize the situation where large color changes occur when adjacent texels belong to different source pixels. It is a building plane texture map The number of texels, It is a building plane texture map The texels on It is a neighborhood Texels within the range, It's a real photo The mask on the upper plane i is projected onto the texture map The mask on is a texel In the mask The binary value on It's a real photo The upper plane i is projected onto the texture map Projected photos on , Texel , Projecting real photos The color value on , Texel , The weight vector of the pixel on the corresponding k-th visible photo, the specific corresponding relationship is as follows Figure 5 shown.
[0090] Step S500: obtaining a target texture map based on the texture map texels and the target weight vector.
[0091] In this embodiment, the target weight can be obtained through gradient optimization in step S400, and then the target texture map is reconstructed according to the target weight and the texture map texels.
[0092] The technical solution in this application is experimented in many architectural scenes of different complexity, and various experimental results show the robustness of the texture reconstruction framework in this application. The framework of this application can complete the task of architectural model texture reconstruction in a more automated and direct way. In general, from the visual results, the texture map finally reconstructed by the framework of this application has the following significant advantages: 1) Texture texels have luminance and content consistency, 2) Texture texels are as consistent as possible in perspective and are directly parallel. Figure 6 As shown, compared with the textures reconstructed by Reality Capture (RC), LTBC, and 3DGS, the solution of the present application is more consistent and parallel in most experimental results. Figure 7 Overall framework diagram for architectural texture reconstruction in this application.
[0093] Beneficial effects of this application:
[0094] First, this application performs plane parameterization on the input building proxy model to construct a texture atlas of the building plane. , establish a mapping relationship between each texture map texel in the texture atlas and the input photo pixel and use it as the optimization parameter of the differentiable renderer. Next, the architectural proxy model with texture map is used to generate a set of rendering results through the differentiable renderer. The gradient information of the parameters to be optimized is calculated based on the rendering results and the input real photo. The input parameters are optimized using the gradient descent optimizer. After multiple optimization iterations, the final texture map is obtained. . The architectural texture reconstruction framework based on differentiable rendering proposed in the present invention considers the mapping relationship between texture map texels and input real photo pixels for optimized reconstruction. Since the calculation of the loss function needs to consider the luminance, viewing angle and content consistency of the texels, the proposed loss function for measuring the quality of the reconstructed texture map is used to optimize the quality of texture map reconstruction, which is different from the processing method of most traditional texture reconstruction algorithms. The method of the present application is to construct a mapping relationship between the texels of the plane texture and the pixels of the input real photo for each plane texture, and consider the viewing angle, luminance, content consistency, and texture integrity to design the loss function related to the mapping relationship. The differentiable renderer obtains relevant information to optimize the quality of texture reconstruction, leading other texture reconstruction methods in efficiency and effect, and has high fidelity.
[0095] This application proposes a texture reconstruction framework based on differentiable rendering, which is used to reconstruct the texture map of the building proxy model and reconstruct a texture atlas with content and perspective consistency for each plane. Given a rough three-dimensional building proxy model, it is decomposed into multiple plane polygons. For each plane polygon, this application constructs the texture map of the plane, and then considers the mapping relationship between the texture map texels and the input real photo pixels. The loss function considers the perspective quality and content consistency, and optimizes the gradient information of the mapping relationship obtained through differentiable rendering, and finally obtains a complete, clear and correct texture map.
[0096] Exemplary Systems
[0097] Based on the above embodiments, the present invention also provides a building texture reconstruction system. Figure 8As shown, the system in this embodiment includes a model planarization module 10 , a mapping function establishment module 20 , a differentiable rendering module 30 , a weight optimization module 40 and a texture reconstruction module 50 . Specifically, the model planarization module 10 is used to obtain a building proxy model and perform plane parameterization on the building proxy model to obtain a building plane texture atlas; the building plane texture atlas contains a plurality of texture map texels; the mapping function establishment module 20 is used to obtain a real photo pixel set and establish a mapping function between each texture map texel in the building plane texture atlas and each real photo pixel in the real photo pixel set; the differentiable rendering module 30 is used to obtain an initial weight vector, and perform differentiable rendering on the texture map texels based on the initial weight vector and the mapping function to obtain a differentiable rendering image; the weight optimization module 40 is used to determine weight gradient information based on the differentiable rendering image and the real photo pixels, and perform gradient optimization on the initial weight vector using a gradient optimization algorithm based on the weight gradient information to obtain a target weight vector; the texture reconstruction module 50 is used to obtain a target texture map based on the texture map texels and the target weight vector.
[0098] In one implementation, the mapping function establishment module includes:
[0099] A parameter acquisition unit, used to acquire real camera parameters and virtual camera parameters;
[0100] A function relationship establishing unit is used to determine a mapping function between each texture texel in the building plane texture atlas and each real photo pixel in the real photo pixel set based on the real camera parameters and the virtual camera parameters.
[0101] In one implementation, the differentiable rendering module includes:
[0102] A color value determination unit, configured to determine a color value of a real photo pixel corresponding to the texture map texel based on the mapping function and the texture map texel;
[0103] A differentiable rendering unit is used to determine a differentiable rendering image based on the initial weight vector and the color value of the pixel of the real photo.
[0104] In one implementation, the weight optimization module includes:
[0105] A mask acquisition unit, used for acquiring a mask;
[0106] A rendering loss item acquisition unit is used to determine a rendering loss item based on the mask, real photo pixels and the differentiable rendering image.
[0107] In one implementation, the weight optimization module includes:
[0108] A viewing angle region acquisition unit, used to acquire the viewing angle region of the real photo pixel;
[0109] A perspective loss item determination unit is used to determine the perspective loss item based on the texture map texels, the mask and the viewing angle area of the real photo pixels.
[0110] In one implementation, the weight optimization module includes:
[0111] a color value difference determining unit, configured to obtain a real photo neighborhood pixel, and determine a color value difference based on the real photo pixel and the real photo neighborhood pixel;
[0112] A weight value difference determining unit, configured to obtain a neighborhood weight vector and determine a weight difference based on the initial weight vector and the neighborhood weight vector;
[0113] A parameter loss term determination unit is used to determine a parameter loss term based on the mask, the color value difference and the weight difference.
[0114] In one implementation, the weight optimization module includes:
[0115] A hyperparameter acquisition unit, used to acquire hyperparameters, wherein the hyperparameters include a first hyperparameter, a second hyperparameter, and a third hyperparameter;
[0116] A parameter loss term determination unit is used to determine a loss function based on the hyperparameter, the rendering loss term, the perspective loss term and the parameter loss term.
[0117] The working principles of each module in the building texture reconstruction system of this embodiment are the same as the principles of each step in the above method embodiment, and will not be repeated here.
[0118] Based on the above embodiments, the present invention further provides a terminal device, the principle block diagram of the terminal device can be as follows: Fig. 9 The terminal device may include one or more processors 100 ( Fig. 9 Only one is shown in the figure), a memory 101 and a computer program 102 stored in the memory 101 and executable on one or more processors 100, for example, a program of a building texture reconstruction method. When one or more processors 100 execute the computer program 102, the various steps in the method embodiment of the building texture reconstruction method can be implemented. Alternatively, when one or more processors 100 execute the computer program 102, the functions of various modules / units in the device embodiment of the building texture reconstruction method can be implemented, which is not limited here.
[0119] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0120] In one embodiment, the memory 101 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 101 may also include both an internal storage unit of the electronic device and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the terminal device. The memory 101 may also be used to temporarily store data that has been output or is to be output.
[0121] Those skilled in the art will understand that Fig. 9 The principle block diagram shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the terminal device to which the solution of the present invention is applied. The specific terminal device includes Fig. 9 More or fewer components may be shown, or certain components may be combined, or may have a different arrangement of components.
[0122] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, operating database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double operational data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0123] In summary, the present invention discloses a method, system, device and storage medium for reconstructing building textures, the method comprising: firstly obtaining a building proxy model, and plane parameterizing the building proxy model to obtain a building plane texture atlas; the building plane texture atlas contains a number of texture map texels; then obtaining a real photo pixel set, and establishing a mapping function between each texture map texel in the building plane texture atlas and each real photo pixel in the real photo pixel set; further obtaining an initial weight vector, and performing differentiable rendering on the texture map texel based on the initial weight vector and the mapping function to obtain a differentiable rendering image; then determining weight gradient information based on the differentiable rendering image and the real photo pixel, and performing gradient optimization on the initial weight vector based on the weight gradient information using a gradient optimization algorithm to obtain a target weight vector; finally obtaining a target texture map based on the texture map texel and the target weight vector. The present invention performs building texture reconstruction through differentiable rendering, optimizes and reconstructs considering the mapping relationship between texture map texels and real photo pixels, and obtains high-quality building texture maps.
[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for architectural texture reconstruction, It is characterized in that The method comprises: Acquire a building proxy model, and perform plane parameterization on the building proxy model to obtain a building plane texture atlas; the building plane texture atlas includes a plurality of texture map texels; Acquire a real photo pixel set, and establish a mapping function between each texture map texel in the building plane texture atlas and each real photo pixel in the real photo pixel set; Obtaining an initial weight vector, and performing differentiable rendering on the texture map texel based on the initial weight vector and the mapping function to obtain a differentiable rendering image; Determining weight gradient information based on the differentiable rendering image and real photo pixels, and performing gradient optimization on the initial weight vector using a gradient optimization algorithm based on the weight gradient information to obtain a target weight vector; A target texture map is obtained based on the texture map texels and the target weight vector.
2. The architectural texture reconstruction method according to claim 1, It is characterized in that The step of establishing a mapping function between each texture texel in the building plane texture atlas and each real photo pixel in the real photo pixel set comprises: Get the real camera parameters and virtual camera parameters; A mapping function between each texture texel in the building plane texture atlas and each real photo pixel in the real photo pixel set is determined based on the real camera parameters and the virtual camera parameters.
3. The architectural texture reconstruction method according to claim 2, It is characterized in that The performing differentiable rendering on the texture map texel based on the initial weight vector and the mapping function to obtain a differentiable rendering image includes: Determine, based on the mapping function and the texture map texel, a color value of a real photo pixel corresponding to the texture map texel; A differentiable rendering image is determined based on the initial weight vector and the color values of the real photo pixels.
4. The architectural texture reconstruction method according to claim 1, It is characterized in that When the gradient optimization algorithm is used to perform gradient optimization on the initial weight vector, the method includes: Get the mask; A rendering loss term is determined based on the mask, the real photo pixels, and the differentiable rendering map.
5. The architectural texture reconstruction method according to claim 4, It is characterized in that When the gradient optimization algorithm is used to perform gradient optimization on the initial weight vector, the method includes: Obtaining the viewing angle area of the real photo pixel; A perspective loss term is determined based on the texture map texels, the mask, and the viewing area of the real photo pixels.
6. The architectural texture reconstruction method according to claim 4, It is characterized in that When the gradient optimization algorithm is used to perform gradient optimization on the initial weight vector, the method includes: Acquire a real photo neighborhood pixel, and determine a color value difference based on the real photo pixel and the real photo neighborhood pixel; Obtaining a neighborhood weight vector, and determining a weight difference based on the initial weight vector and the neighborhood weight vector; A parameter loss term is determined based on the mask, the color value difference, and the weight difference.
7. The architectural texture reconstruction method according to claim 6, It is characterized in that When the gradient optimization algorithm is used to perform gradient optimization on the initial weight vector, the method includes: Obtaining hyperparameters, where the hyperparameters include a first hyperparameter, a second hyperparameter, and a third hyperparameter; A loss function is determined based on the hyperparameters, the rendering loss term, the perspective loss term, and the parameter loss term.
8. A building texture reconstruction system, It is characterized in that The system comprises: A model planarization module is used to obtain a building proxy model and perform plane parameterization on the building proxy model to obtain a building plane texture atlas; the building plane texture atlas includes a plurality of texture map texels; A mapping function establishment module, used for acquiring a real photo pixel set, and establishing a mapping function between each texture map texel in the building plane texture atlas and each real photo pixel in the real photo pixel set; A differentiable rendering module, used to obtain an initial weight vector, and perform differentiable rendering on the texture map texel based on the initial weight vector and a mapping function to obtain a differentiable rendering image; A weight optimization module, used to determine weight gradient information based on the differentiable rendering image and real photo pixels, and to perform gradient optimization on the initial weight vector using a gradient optimization algorithm based on the weight gradient information to obtain a target weight vector; A texture reconstruction module is used to obtain a target texture map based on the texture map texel and the target weight vector.
9. A terminal device, It is characterized in that The terminal device includes a memory, a processor, and a building texture reconstruction method program stored in the memory and executable on the processor. When the processor executes the building texture reconstruction method program, the steps of the building texture reconstruction method as described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a building texture reconstruction method program. When the building texture reconstruction method program is executed by a processor, the steps of the building texture reconstruction method according to any one of claims 1 to 7 are implemented.