Texture reconstruction method and device

Abstract

Embodiments of the invention disclose a texture reconstruction method and apparatus. The method comprises the steps of obtaining to-be-processed data; performing information extraction on the to-be-processed data through a multi-view sparse voxel Transform to obtain sparse voxel features corresponding to the to-be-processed data; utilizing a differentiable unwarping technology to map the sparse voxel features to a UV space, and generating an initial texture map; and optimizing the initial texture map through a pre-trained diffusion model to generate a final texture map.

Description

Technical Field

[0001] The present invention relates to the fields of computer technology and image recognition technology, and in particular to a texture reconstruction method and device. Background Art

[0002] Texture generation technology aims to create high-quality, realistic or stylized textures for three-dimensional objects based on text descriptions or two-dimensional images, which is of great significance in the field of visual recognition. Traditional three-dimensional texture generation technology mainly relies on diffusion models to directly generate UV maps, or iteratively generate them through pre-trained depth-to-image diffusion models. However, these methods usually have problems with insufficient reliability and consistency, resulting in serious mismatch between the generated UV maps and the geometric structure.

[0003] In the prior art, 3D texture generation technology mainly includes: direct diffusion model technology and zero-sample iterative patching technology. Among them, typical representatives of direct diffusion model technology include: TexDreamer, Geometry Aware Texturing. This technology can directly apply the diffusion process in UV space to generate UV maps. It has the advantage of fast speed, but it can only be limited to specific application scenarios and is difficult to generalize to different types of 3D models or different geometric structures. Moreover, since this technology does not fully consider geometric information, the generated UV map may have serious mismatch problems with the actual 3D geometric structure. Zero-sample iterative patching technology is a technology based on the depth-to-image diffusion model. This technology mainly includes: TEXTure, Text2Tex, Intex, Paint3D, which can gradually generate views, cover areas through mesh rotation, and guide patching with alternating rendering. Although no additional training is required, the inference time is long and it is easy to produce artifacts and other distortion phenomena.

[0004] Based on this, this specification provides a texture reconstruction method and device. Summary of the invention

[0005] The embodiments of this specification provide a texture reconstruction method and device for solving the following technical problems: In the prior art, the direct diffusion model technology can only be limited to specific application scenarios and is difficult to generalize to different types of three-dimensional models or different geometric structures. Moreover, since the technology does not fully consider the geometric information, the generated UV map may have a serious mismatch problem with the actual three-dimensional geometric structure. The zero-sample iterative patching technology of the depth-to-image diffusion model has many disadvantages, such as: long inference time and easy to produce artifacts and other distortion phenomena.

[0006] To solve the above technical problems, the embodiments of this specification are implemented as follows:

[0007] The present invention provides a texture reconstruction method, including:

[0008] Get the data to be processed;

[0009] Extracting information from the data to be processed through a multi-view sparse voxel transformer to obtain sparse voxel features corresponding to the data to be processed;

[0010] Using differentiable unwarping technology, the sparse voxel features are mapped to UV space to generate an initial texture map;

[0011] The initial texture map is optimized by a pre-trained diffusion model to generate a final texture map.

[0012] The embodiment of this specification also provides a texture reconstruction device, including:

[0013] Acquisition module, obtains the data to be processed;

[0014] An information extraction module extracts information from the data to be processed through a multi-view sparse voxel Transformer to obtain sparse voxel features corresponding to the data to be processed;

[0015] A mapping module, using a differentiable unwarping technique, maps the sparse voxel features to a UV space to generate an initial texture map;

[0016] The optimization module optimizes the initial texture map through a pre-trained diffusion model to generate a final texture map.

[0017] The texture reconstruction method provided in the embodiments of this specification utilizes efficient three-dimensional feature extraction to capture the geometric details of the object, and utilizes differentiable UV space projection to accurately map the geometric features into the texture space; at the same time, a miniature landscape texture is generated by deeply combining geometric information with a probability diffusion model in texture generation, providing users with high-fidelity miniature landscape texture maps. With the powerful feature extraction capabilities of sparse voxels and Transformer, the key value consistency provided by the diffusion model is utilized to improve the accuracy of texture details, and the texture visual accuracy is high: in the process of iterative optimization of texture mapping using a multi-level progressive diffusion model, geometric consistency constraints ensure the visual consistency of the texture at different viewing angles, thereby ensuring high visual consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 A schematic diagram of the system architecture of a texture reconstruction method provided in an embodiment of this specification;

[0020] Figure 2 A flowchart of a texture reconstruction method provided in an embodiment of this specification;

[0021] Figure 3 A schematic diagram of the structure of a multi-view sparse voxel Transformer provided in an embodiment of this specification;

[0022] Figure 4 A schematic diagram of the structure of the Transformer block provided in the embodiments of this specification;

[0023] Figure 5 A framework diagram of a texture reconstruction method provided in an embodiment of this specification;

[0024] Figure 6 A schematic diagram of a texture reconstruction device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0026] Figure 1 A schematic diagram of the system architecture of a texture reconstruction method provided in an embodiment of this specification. Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0027] The terminal devices 101, 102, 103 interact with the server 105 via the network 104 to receive or send messages, etc. Various client applications may be installed on the terminal devices 101, 102, 103, such as dedicated programs for texture reconstruction methods, etc.

[0028] The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various dedicated or general electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When the terminal devices 101, 102, 103 are software, they may be installed in the electronic devices listed above. They may be implemented as multiple software or software modules (e.g., multiple software or software modules for providing distributed services), or they may be implemented as a single software or software module.

[0029] The server 105 may be a server providing various services, such as a backend server providing services for client applications installed on the terminal devices 101, 102, 103. For example, the server may perform texture reconstruction so as to display the texture reconstruction result on the terminal device server 101, 102, 103.

[0030] The server 105 may be hardware or software. When the server 105 is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server 105 is software, it may be implemented as multiple software or software modules (e.g., multiple software or software modules for providing distributed services), or as a single software or software module.

[0031] Figure 2 The following is a flow chart of a texture reconstruction method provided in an embodiment of the present specification. From a program perspective, the execution subject of the process can be a program installed on an application server or an application terminal. It can be understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 1 As shown, the method includes:

[0032] Step S201: Obtain data to be processed.

[0033] In the embodiment of the present specification, the data to be processed includes: multi-view images to be processed and depth maps corresponding to the multi-view images to be processed. The data to be processed are multi-view images collected from real scenes and depth maps corresponding to the collected multi-view images. Specifically, the real scene can be a real scene existing in the real world. The specific scene of the real scene is not specifically limited. Any real scene that needs to realize a miniature landscape can be used as a collection point for the data to be processed to collect the multi-view images to be processed and the depth maps corresponding to the multi-view images to be processed.

[0034] The methods for obtaining the multi-view images to be processed and the depth maps corresponding to the multi-view images to be processed adopt the existing technologies and will not be described in detail here.

[0035] Multi-view images refer to a group of images of the same scene taken from different angles or positions. The more multi-view images to be processed, the better the final texture reconstruction effect. However, considering the amount of computation, the number of views of the multi-view images to be processed needs to be selected according to the specific application scenario.

[0036] In one embodiment of the present specification, for simple objects or small-scale scenes, 4-6 perspectives are selected for the multi-perspective images to be processed, preferably 6; for complex objects or large scenes, 8-12 perspectives are selected for the multi-perspective images to be processed, preferably 12.

[0037] Step S203: extracting information from the data to be processed through a multi-view sparse voxel transformer to obtain sparse voxel features corresponding to the data to be processed.

[0038] In the embodiment of the present specification, the data to be processed includes: a multi-view image to be processed and a depth map corresponding to the multi-view image to be processed;

[0039] The extracting information from the data to be processed by the multi-view sparse voxel transformer to obtain sparse voxel features corresponding to the data to be processed specifically includes:

[0040] The multi-view image to be processed and the depth map corresponding to the multi-view image to be processed are projected by the multi-view sparse voxel Transformer, and shallow layer information and deep layer information corresponding to the data to be processed are extracted to obtain sparse voxel features corresponding to the data to be processed;

[0041] in,

[0042] The shallow information includes low-level texture information and edge information;

[0043] The deep information includes geometric structure and semantic information.

[0044] In an embodiment of the present specification, the multi-view sparse voxel Transformer is composed of six Transformer blocks, wherein each Transformer block includes: a multi-head self-attention layer, a residual connection and normalization layer, a feedforward neural network layer, and a residual connection and normalization layer.

[0045] In the embodiment of this specification, the structure of the multi-view sparse voxel transformer includes: an input layer, six transformer blocks and an output layer. In order to further understand the multi-view sparse voxel transformer provided in the embodiment of this specification, the structure of the multi-view sparse voxel transformer and the structure of the transformer block will be described below.

[0046] Figure 3 This is a schematic diagram of the structure of the multi-view sparse voxel Transformer provided in the embodiments of this specification. Figure 4 This is a schematic diagram of the structure of the Transformer block provided in the embodiments of this specification. Figure 3 As shown in , the structure of the multi-view sparse voxel Transformer includes an input layer, a first Transformer block, a second Transformer block, a third Transformer block, a fourth Transformer block, a fifth Transformer block, a sixth Transformer block, and an output layer. Figure 4 As shown in the figure, each Transformer block includes: multi-head self-attention layer, feedforward neural network layer, residual connection and normalization layer. Multi-head self-attention can understand the relationship between features, feedforward neural network layer performs the same transformation on the vector at each position, residual connection helps to solve the gradient disappearance problem, and normalization layer ensures the stability of output and accelerates the training process.

[0047] After the data to be processed is input into the input layer, the multi-head self-attention layer of the first Transformer block converts the data to be processed into Q, K, V vectors through linear transformation, and then passes through the feedforward neural network layer, residual connection and normalization layer to obtain the first sparse voxel feature, which is used as the output of the first Transformer block; then, the first sparse voxel feature is used as the input of the second Transformer block, and the multi-head self-attention layer of the second Transformer block converts the first sparse voxel feature into Q, K, V vectors through linear transformation, and then passes through the feedforward neural network layer, residual connection and normalization layer to obtain the second sparse voxel feature, which is used as the output of the second Transformer block; ...; and so on, until the sixth Transformer block obtains the sixth sparse voxel feature, and after the sixth sparse voxel feature is output by the output layer, the shallow information and deep information corresponding to the data to be processed are extracted.

[0048] In the embodiments of the present specification, low-level texture information is surface information that is directly perceived visually. Specifically, low-level texture information includes surface features such as color and texture patterns. Texture patterns include stripes, spots, smoothness and roughness, etc. Edge information is the boundary between different areas in an image. Specifically, edge information includes the contour, edge, angle, etc. of an object.

[0049] Geometric information involves physical properties such as the shape and surface structure of an object. Deep information usually includes the shape of the object (spherical, cubic, etc.), boundaries (clear boundaries between different parts of an object or separation of adjacent objects), and surface structures (patterns, etc.). Semantic information is usually used to characterize the category, function, or contextual meaning of an object, including the category and purpose of the object, and the spatial and functional relationships between objects.

[0050] Step S205: using differentiable unwarping technology, mapping the sparse voxel features to UV space to generate an initial texture map.

[0051] In the above embodiments of this specification, the shallow information and deep information corresponding to the data to be processed are distorted relative to the data to be processed and cannot be used to directly generate texture maps. Based on this, in the embodiments of this specification, the differentiable unwarping technology is first used to generate the initial texture map.

[0052] Differentiable unwarping technology is an image processing method used to correct geometric distortion in images, and the whole process is differentiable. Unwarping is the process of restoring a distorted image to its undistorted state, and the whole unwarping process is differentiable, that is, the model parameters can be optimized by calculating the gradient, so it is called differentiable unwarping technology.

[0053] In the embodiment of this specification, the differentiable unwarping technology is used to map the sparse voxel features to the UV space to generate the initial texture map, which specifically includes:

[0054] Through UV projection, the sparse voxel features are projected into the UV coordinate system to obtain sparse UV features;

[0055] The sparse UV features are input into a pre-trained convolutional neural network model to obtain an initial UV texture map.

[0056] UV projection is to project sparse voxel features into the UV coordinate system. By UV projection, sparse voxel features are projected into the UV coordinate system, so that 3D coordinates can be mapped to the 2D UV coordinate system.

[0057] The UV coordinate system is a two-dimensional texture coordinate system, where U and V represent coordinates in the horizontal and vertical directions respectively, and is usually used to describe texture mapping on the surface of a 3D model.

[0058] In an embodiment of the present specification, the pre-trained neural network model is an encoder-decoder based neural network model, the encoder includes 6 convolutional layers and a pooling layer, and the decoder includes a deconvolution layer and a convolution layer.

[0059] In the embodiments of the present specification, the encoder is used to gradually extract high-level features of the sparse UV features, and the decoder is used to gradually restore the spatial resolution corresponding to the features, thereby achieving restoration of the features and the spatial resolution.

[0060] Specifically, the convolutional layers of the encoder all have 3×3 convolution kernels, the number of convolution kernels of the first convolution layer is 1024, the number of convolution kernels of the second convolution layer is 512, the number of convolution kernels of the third convolution layer is 256, the number of convolution kernels of the fourth convolution layer is 128, the number of convolution kernels of the fifth convolution layer is 64, and the number of convolution kernels of the sixth convolution layer is 32.

[0061] Each layer of the decoder is usually skipped with the features of the corresponding layer of the encoder to fuse shallow and deep features to better preserve detail information. The last layer of the decoder is a convolutional layer that maps the features to the required number of output channels to obtain the initial UV texture map. The output layer usually uses a 1×1 convolution kernel to ensure that the generated UV texture has the correct number of channels and resolution.

[0062] In the embodiment of this specification, the generation expression of the initial texture map is:

[0063]

[0064] in,

[0065] T0 represents the initial texture map;

[0066] Represents the unwarping function;

[0067] Representing the feature extraction of the multi-view sparse voxel transformer;

[0068] α i represents feature weight;

[0069] I i represents the i-th viewing angle image in the multi-view images to be processed;

[0070] D i Represents a depth map corresponding to the i-th viewing angle image in the multi-view images to be processed.

[0071] Step S207: Optimizing the initial texture map by using a pre-trained diffusion model to generate a final texture map.

[0072] In the embodiment of this specification, the optimization of the initial texture map by the pre-trained diffusion model to generate the final texture map specifically includes:

[0073] Performing differentiable rendering on the geometric reconstruction model generated by the initial texture map and the initial texture feature map to generate a multi-view image;

[0074] The multi-view image is input into the pre-trained diffusion model for optimization to generate the final texture map.

[0075] In the embodiments of this specification, the method for generating the geometric reconstruction model adopts the existing technology and will not be described in detail here.

[0076] Differentiable rendering is an end-to-end rendering process that uses gradients for optimization. Through differentiable rendering, multi-view images can be generated.

[0077] In order to ensure that the generated multi-view image can retain the edge information and detail information of the image when generating the texture map, further optimization is required to improve the quality of the texture. In the embodiment of this specification, the optimization is performed through a pre-trained diffusion model.

[0078] In an embodiment of the present specification, the pre-trained diffusion model is a symmetric network structure based on U-net, the encoder of the pre-trained diffusion model includes 12 layers of convolutional layers and pooling layers; the decoder of the pre-trained diffusion model includes a deconvolution layer and a convolution layer.

[0079] The convolution layer in the encoder of the pre-trained diffusion model is a 3×3 convolution kernel, and the encoder of the pre-trained diffusion model is nonlinearly connected through the activation function ReLU. The pooling layer is used to reduce the spatial size of the feature map while increasing the abstract representation ability of the feature. The output of each encoder layer is directly connected to the corresponding layer of the decoder. The decoder part includes deconvolution layers and convolution layers, which are used to gradually restore the spatial resolution of the features. Through the jump connection with the encoder, the decoder can better fuse shallow and deep features to ensure that the details and edge information of the image can be retained when generating texture maps.

[0080] In the embodiment of this specification, the constraints of the pre-trained diffusion model include: key consistency and geometric consistency;

[0081] The expression of the constraint condition of the pre-trained diffusion model is:

[0082]

[0083] in,

[0084] T opt Represents the optimized texture map;

[0085] is a differentiable rendering function;

[0086] G represents the reconstructed geometry;

[0087] I j represents the j-th view image in the multi-view images to be processed;

[0088] λ and μ are regularization parameters;

[0089] K k and V k Key-value consistency

[0090] Represents a geometric consistency constraint.

[0091] In the embodiments of this specification, key value consistency is achieved by matching key point features in images of different view angles, ensuring that the same object has similar feature representations at different view angles. Geometric consistency constrains feature generation by utilizing the three-dimensional geometric relationship of the object, ensuring that the geometric details captured at different view angles can be accurately aligned.

[0092] In the embodiment of the present specification, the acquisition of key value consistency includes: first, extracting key point features from images at different viewing angles in the multi-view image. These features may include but are not limited to color, edge, texture pattern, etc., which can represent important properties of the surface of an object. Subsequently, key point matching is performed. These key point features are found and matched between different viewing angles. This process can be implemented by various algorithms, such as SIFT (scale invariant feature transform), SURF (accelerated robust feature) or methods based on deep learning. The purpose is to find the position where the same object appears in different views and confirm that the features at these positions are similar. Again, consistency constraint calculation is performed. Once the key points are correctly matched, consistency constraints can be applied to ensure that the feature representations of these points at all viewing angles are consistent. This means that if a specific key point appears as a certain texture or color in one viewing angle, then the key point should also maintain the same appearance characteristics as much as possible in other viewing angles. Finally, optimization adjustment is performed, and the generated texture map is gradually optimized using the above consistency constraints during the iteration process of the diffusion model. If the performance of a certain area is inconsistent at different viewing angles, the texture of the area will be adjusted according to the key value consistency principle until the expected consistency level is reached.

[0093] The implementation of key-value consistency helps to solve the geometry mismatch problem encountered when directly using the diffusion model to generate UV texture maps. In this way, it can ensure that the final generated texture not only looks reasonable in a single viewing angle, but also maintains visual consistency and accuracy at different viewing angles of the entire 3D model.

[0094] In the process of generating multi-view images, the relative position relationship between each view and the impact of viewpoint transformation are considered. For example, when moving from one view to another, some areas may become visible or blocked, and these changes need to be correctly reflected in the texture generation process. Use geometric consistency constraints to ensure that features of the same part can be accurately aligned under different views. This includes but is not limited to edge alignment, consistency of surface texture patterns, etc. This alignment can be achieved by calculating the correspondence between features under different views and using this as part of the optimization goal. In the iterative optimization process of the multi-level progressive diffusion model, geometric consistency is involved as an important constraint. Each iteration will make adjustments based on the difference between the current texture map and the geometric model, gradually reducing the deviation between the two until a satisfactory match is achieved.

[0095] In short, geometric consistency is an important means to ensure the authenticity and accuracy of texture mapping. It combines the three-dimensional geometric properties of the object to ensure that the generated texture is both beautiful and consistent with the actual object, and maintains consistency and accuracy in multi-view rendering.

[0096] The final texture map obtained based on the above method, namely the miniature landscape texture, can be output to the user.

[0097] In order to further understand the texture reconstruction method provided by the present embodiment, the texture reconstruction method will be described below in conjunction with the framework of the texture reconstruction method.

[0098] Figure 5 A framework diagram of a texture reconstruction method provided in an embodiment of the present specification. The texture reconstruction method provided in an embodiment of the present specification is implemented by a texture generation module based on a multi-view sparse voxel transformer and a multi-level deep texture optimization module based on geometric embedding learning. Specifically, in the texture generation module of the multi-view sparse voxel transformer, the multi-view image and the discovery map are texture processed by the multi-view sparse voxel transformer to generate sparse voxel features, which are further subjected to UV projection to obtain sparse UV features. The sparse UV features are convolved to obtain an initial UV texture map, i.e., an initial texture map. Subsequently, in the multi-level deep texture optimization module based on geometric embedding learning, the initial UV texture map and the geometric reconstruction model corresponding to the initial UV texture map are subjected to differentiable rendering, and after passing through an encoder and a decoder, and after optimization of constraints, an optimized texture map, i.e., a final texture map, is generated. The final texture map can be used as a miniature landscape texture.

[0099] The texture reconstruction method provided in the embodiments of this specification utilizes efficient three-dimensional feature extraction to capture the geometric details of the object, and utilizes differentiable UV space projection to accurately map the geometric features into the texture space; at the same time, a miniature landscape texture is generated by deeply combining geometric information with a probability diffusion model in texture generation, providing users with high-fidelity miniature landscape texture maps. With the powerful feature extraction capabilities of sparse voxels and Transformer, the key value consistency provided by the diffusion model is utilized to improve the accuracy of texture details, and the texture visual accuracy is high: in the process of iterative optimization of texture mapping using a multi-level progressive diffusion model, geometric consistency constraints ensure the visual consistency of the texture at different viewing angles, thereby ensuring high visual consistency.

[0100] The above content describes in detail a texture reconstruction method. Correspondingly, this specification also provides a texture reconstruction device, such as Figure 6 shown. Figure 6 A schematic diagram of a texture reconstruction device provided in an embodiment of this specification, the texture reconstruction device comprising:

[0101] Acquisition module 601, acquiring data to be processed;

[0102] An information extraction module 603 extracts information from the data to be processed through a multi-view pre-trained diffusion model to obtain sparse voxel features corresponding to the data to be processed;

[0103] A mapping module 605 maps the sparse voxel features to a UV space using a differentiable unwarping technique to generate an initial texture map;

[0104] The optimization module 607 optimizes the initial texture map by using a pre-trained diffusion model to generate a final texture map.

[0105] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0106] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0107] The apparatus, electronic device, non-volatile computer storage medium and method provided in the embodiments of this specification correspond to each other, and therefore, the apparatus, electronic device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, electronic device and non-volatile computer storage medium will not be repeated here.

[0108] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0109] The specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media including storage devices.

[0110] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0111] The above is only an embodiment of this specification and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A texture reconstruction method, characterized in that: The texture reconstruction method comprises: Get the data to be processed; Extracting information from the data to be processed through a multi-view sparse voxel transformer to obtain sparse voxel features corresponding to the data to be processed; Using differentiable unwarping technology, the sparse voxel features are mapped to UV space to generate an initial texture map; The initial texture map is optimized by a pre-trained diffusion model to generate a final texture map.

2. The texture reconstruction method according to claim 1, characterized in that: The multi-view sparse voxel Transformer consists of six Transformer blocks, wherein each Transformer block includes: a multi-head self-attention layer, a residual connection and normalization layer, a feedforward neural network layer, and a residual connection and normalization layer.

3. The texture reconstruction method according to claim 1, characterized in that: The data to be processed includes: multi-view images to be processed and depth maps corresponding to the multi-view images to be processed; The extracting information from the data to be processed by the multi-view sparse voxel transformer to obtain sparse voxel features corresponding to the data to be processed specifically includes: The multi-view image to be processed and the depth map corresponding to the multi-view image to be processed are projected by the multi-view sparse voxel Transformer, and shallow layer information and deep layer information corresponding to the data to be processed are extracted to obtain sparse voxel features corresponding to the data to be processed; in, The shallow information includes low-level texture information and edge information; The deep information includes geometric structure and semantic information.

4. The texture reconstruction method according to claim 1, characterized in that: The differentiable unwarping technology is used to map the sparse voxel features to the UV space to generate an initial texture map, specifically including: Through UV projection, the sparse voxel features are projected into the UV coordinate system to obtain sparse UV features; The sparse UV features are input into a pre-trained convolutional neural network model to obtain an initial UV texture map.

5. The texture reconstruction method according to claim 4, characterized in that: The pre-trained neural network model is an encoder-decoder based neural network model, the encoder includes 6 convolutional layers and a pooling layer, and the decoder includes a deconvolution layer and a convolutional layer.

6. The texture reconstruction method according to claim 4, characterized in that: The generation expression of the initial texture map is: in, T0 represents the initial texture map; Represents the unwarping function; Representing the feature extraction of the multi-view sparse voxel Transformer; α i represents feature weight; I i represents the i-th viewing angle image in the multi-view images to be processed; D i Represents a depth map corresponding to the i-th viewing angle image in the multi-view images to be processed.

7. The texture reconstruction method according to claim 1, characterized in that: The step of optimizing the initial texture map by using a pre-trained diffusion model to generate a final texture map specifically includes: Performing differentiable rendering on the geometric reconstruction model generated by the initial texture map and the initial texture feature map to generate a multi-view image; The multi-view image is input into the pre-trained diffusion model for optimization to generate the final texture map.

8. The texture reconstruction method according to claim 7, characterized in that: The pre-trained diffusion model is a symmetric network structure based on U-net, the encoder of the pre-trained diffusion model includes 12 convolutional layers and pooling layers; the decoder of the pre-trained diffusion model includes a deconvolution layer and a convolution layer.

9. The texture reconstruction method according to claim 7, characterized in that: The constraints of the pre-trained diffusion model include: key value consistency and geometric consistency; The expression of the constraint condition of the pre-trained diffusion model is: in, T opt Represents the optimized texture map; is a differentiable rendering function; G represents the reconstructed geometry; I j represents the j-th view image in the multi-view images to be processed; λ and μ are regularization parameters; K k and V k Key-value consistency Represents a geometric consistency constraint.

10. A texture reconstruction device, characterized in that: The texture reconstruction device comprises: Acquisition module, obtains the data to be processed; An information extraction module extracts information from the data to be processed through a multi-view sparse voxel Transformer to obtain sparse voxel features corresponding to the data to be processed; A mapping module, using a differentiable unwarping technique, maps the sparse voxel features to a UV space to generate an initial texture map; The optimization module optimizes the initial texture map through a pre-trained diffusion model to generate a final texture map.

Images