Texture generation method, device, electronic device, storage medium and computer program product

Abstract

The present invention provides a texture generation method, device, electronic device, storage medium and computer program product. The texture generation method includes: processing a received image to obtain a processing result; projecting image pixels in the processing result back to the surface to generate a partial texture map; extracting a global embedding based on the received image, and combining the extracted global embedding into a global conditional embedding; using the global conditional embedding to adjust the internal features of a U-type network to obtain an adjusted U-type network, wherein each stage of the U-type network includes a hybrid 2D-3D module; inputting the processing result and the partial texture map into the adjusted U-type network for speed and noise prediction, thereby generating a texture. The hybrid 2D-3D module makes the architecture scalable and improves 3D consistency, thereby achieving the purpose of improving scalability, consistency and speed.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a texture generation method, device, electronic equipment, storage medium and computer program product. Background Art

[0002] 3D mesh texture synthesis is a fundamental problem in computer graphics and vision, and is widely used in virtual reality, game design, and animation production. Existing learning-based methods can only generate textures for specific categories due to scalability and data limitations. Existing test-time optimization-based methods use pre-trained 2D diffusion models to generate image priors through score distillation sampling or by synthesizing pseudo multi-views. Although these methods can generate textures for a wide range of objects, they also have disadvantages such as time-consuming optimization for each object, complex parameter adjustment, susceptibility to 2D prior limitations, and poor 3D consistency in texture generation. Summary of the invention

[0003] In view of the problems existing in the above-mentioned technologies, the present invention provides a texture generation method, device, electronic device, storage medium and computer program product.

[0004] In a first aspect, an embodiment of the present disclosure provides a texture generation method, comprising:

[0005] Processing the received image to obtain a processing result;

[0006] Projecting the resulting image pixels back onto the surface to generate a partial texture map;

[0007] extracting a global embedding based on the received image, and combining the extracted global embeddings into a global conditional embedding;

[0008] Using global conditional embedding to adjust the internal features of the U-shaped network, obtaining an adjusted U-shaped network, wherein each stage of the U-shaped network includes a hybrid 2D-3D module;

[0009] The processed results and partial texture maps are input into the conditioned U-network for speed and noise prediction to generate textures.

[0010] Optionally, the processing results include noise texture mapping, position mapping, single image, text prompt and time step.

[0011] Optionally, extracting a global embedding based on the received image, and combining the extracted global embedding into a global conditional embedding, including:

[0012] Extract global embeddings using image and text encoders;

[0013] The global embedding is processed through a multi-layer perceptron and combined into a global conditional embedding based on the multi-layer perceptron processing structure.

[0014] Optionally, the hybrid 2D-3D module includes a UV module and several 3D point cloud blocks.

[0015] Optionally, the UV features are processed by a 2D convolution block of the UV module to extract local features in the UV space.

[0016] Optionally, the hybrid 2D-3D module includes remapping the extracted local features in the UV space back to the 3D space, and reorganizing the UV features into 3D point cloud features, thereby obtaining 3D neighborhood relations and global structural features to improve 3D consistency.

[0017] In a second aspect, the present disclosure also provides a texture generating device, including:

[0018] A processing module, used for processing the received image to obtain a processing result;

[0019] A projection module, used to project the image pixels in the processing result back to the surface to generate a partial texture map;

[0020] an embedding module, for extracting a global embedding based on the received image, and combining the extracted global embedding into a global conditional embedding;

[0021] An adjustment module, used for using global condition embedding to adjust the internal features of the U-shaped network to obtain an adjusted U-shaped network, wherein each stage of the U-shaped network includes a hybrid 2D-3D module;

[0022] The generation module is used to input the processing results and partial texture mapping into the adjusted U-shaped network for speed and noise prediction, thereby generating texture.

[0023] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0024] at least one processor; and,

[0025] a memory communicatively connected to the at least one processor; wherein,

[0026] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any texture generation method described in the first aspect.

[0027] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any texture generation method described in the first aspect.

[0028] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements any of the texture generation methods described in the first aspect.

[0029] The present invention provides a texture generation method, device, electronic device, storage medium and computer program product, wherein the texture generation method makes the architecture scalable and improves 3D consistency by mixing 2D-3D modules, thereby achieving the purpose of improving scalability, consistency and speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0031] Figure 1 A flowchart of a texture generation method provided by an embodiment of the present disclosure;

[0032] Figure 2 A schematic diagram of the structure of a hybrid 2D-3D module provided in an embodiment of the present disclosure;

[0033] Figure 3 A schematic diagram of the structure of a UV module provided in an embodiment of the present disclosure;

[0034] Figure 4 A schematic diagram of the structure of a 3D point cloud block provided in an embodiment of the present disclosure;

[0035] Figure 5 A schematic diagram of the results of the texture generation method provided by the embodiment of the present disclosure and the image generation method using the prior art;

[0036] Figure 6 A functional block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0038] It should be clear that the following embodiments of the present disclosure are described by specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0039] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0040] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0041] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0042] UV is a two-dimensional texture coordinate corresponding to the vertex information of the geometry. UV provides the connection between the surface mesh and how the image texture is applied to the surface, and is used to control the marking points where the pixels on the texture correspond to the vertices on the 3D mesh.

[0043] First, we develop a generative model that can generate high-quality textures for 3D surfaces based on user-defined conditions, such as images or text cues. The modeling process includes the following main steps:

[0044] 1. Data representation: We use UV texture mapping as the mesh texture representation, which is compact and suitable for diffusion training.

[0045] 2. Model construction and learning: This example develops a novel hybrid 2D-3D network structure to effectively handle the unique characteristics of texture mapping. Then, a diffusion model is trained to generate high-resolution texture maps based on single-view images and text descriptions.

[0046] 3. Inference: The trained model can start from a noisy image and generate a high-resolution texture map through iterative denoising.

[0047] For ease of understanding, Figure 1 As shown, this embodiment discloses a texture generation method, including:

[0048] Processing the received image to obtain a processing result;

[0049] The processing results include noise texture mapping, position mapping, single image, text prompt and time step.

[0050] Project the resulting image pixels back onto the surface to generate a partial texture map as an additional input to the network;

[0051] extracting a global embedding based on the received image, and combining the extracted global embeddings into a global conditional embedding;

[0052] Using global conditional embedding to adjust the internal features of the U-shaped network, obtaining an adjusted U-shaped network, wherein each stage of the U-shaped network includes a hybrid 2D-3D module;

[0053] The processed results and partial texture maps are input into the conditioned U-network for speed and noise prediction, thereby generating texture. The network predicts the denoised speed, which is further converted into predictions of noise or original input.

[0054] Optionally, extracting a global embedding based on the received image, and combining the extracted global embedding into a global conditional embedding, including:

[0055] Extract global embeddings using image and text encoders;

[0056] The global embedding is processed through a multi-layer perceptron and combined into a global conditional embedding based on the multi-layer perceptron processing structure.

[0057] Optional, such as Figure 2 , Figure 3 and Figure 4 As shown, the hybrid 2D-3D module includes a UV module and several 3D point cloud blocks.

[0058] Optionally, the UV features are processed by a 2D convolution block of the UV module to extract local features in the UV space.

[0059] Optionally, the hybrid 2D-3D module includes remapping the extracted local features in the UV space back to the 3D space, and reorganizing the UV features into 3D point cloud features, thereby obtaining 3D neighborhood relations and global structural features to improve 3D consistency.

[0060] The hybrid 2D-3D block is designed to efficiently learn features for 2D texture map generation. The block consists of a UV module and several 3D point cloud blocks. First, the input UV features are processed by a 2D convolution block to extract local features in the UV space. 2D convolutions are computationally more efficient than 3D convolutions or KNN search for point clouds and are more suitable for scaling to high resolutions. In addition, 2D convolutions ensure that the aggregation of neighboring features is based on surface neighborhoods rather than volume neighborhoods, thereby more effectively preserving high-resolution information.

[0061] In order to establish 3D connections between islands in UV space, the output UV features are remapped back to 3D space using rasterization technology, and these UV features are reorganized into 3D point cloud features. In 3D space, the main goal is to obtain 3D neighborhood relationships and global structural features to improve 3D consistency, rather than extracting high-resolution detail features. Therefore, relatively sparse features are used and an efficient module is designed to ensure scalability.

[0062] In this embodiment, the hybrid 2D-3D network structure itself can also be optimized, such as introducing more network layers, or changing the network architecture to improve its ability to process UV features and point cloud data.

[0063] In this embodiment, the network training process can also be optimized, such as by improving the training strategy or using a more efficient optimization algorithm, so as to enhance the network's processing capability for complex textures and 3D structures.

[0064] This embodiment may also include the use of more advanced activation functions, regularization techniques, or different loss functions to improve the accuracy and efficiency of the network in generating textures and 3D structures.

[0065] In this embodiment, based on the hybrid 2D-3D network, it can be considered to be combined with other technologies, such as combining it with traditional 3D modeling technology, using the network to generate preliminary texture mapping, and then further refine and optimize it through geometric modeling technology.

[0066] In this embodiment, hybrid 2D-3D networks and other types of machine learning models, such as GANs or VAEs, can also be integrated to take advantage of the advantages of these models in specific areas (such as texture refinement and complex structure processing).

[0067] This embodiment may involve the use of multimodal data (such as images, videos, and sensor data) to assist the network in processing more complex three-dimensional content, further improving the overall performance and applicability of the model.

[0068] This embodiment uses UV texture mapping as a representation for generation, which is not only scalable but also retains high-resolution details. More importantly, it can directly perform supervised learning from real texture mapping without relying entirely on rendering loss, which makes it compatible with the generation model based on the diffusion model and improves the overall generation quality.

[0069] In order to perform effective feature interaction on the mesh surface, this embodiment proposes a scalable 2D-3D hybrid network architecture. The architecture first performs convolution operations in 2D UV space, and then performs sparse convolution and attention layer operations in 3D space. This simple and effective architecture has several key advantages: first, by applying convolution operations in UV space, the network can effectively learn local and high-resolution details; second, by lifting the calculation to 3D space, the network can learn global 3D dependencies and neighborhood relationships, which may be destroyed during UV parameterization, thereby ensuring global 3D coherence. This hybrid design allows the use of sparse features in 3D space instead of dense voxel or point features, making the architecture scalable. By stacking multiple blocks, a large texture diffusion model is trained, which is able to directly synthesize high-resolution textures (e.g., 1024×1024 texture maps) in a feed-forward manner, guided by single-view images and text cues.

[0070] The texture generation method disclosed in this embodiment has the following advantages:

[0071] 1. Direct generation: Supports feed-forward texture generation of objects, without the need for test-time optimization.

[0072] 2. Fast speed: Since there is no need for optimization during testing, the speed is significantly faster than the current method.

[0073] 3. Better effect: The quality of the generated images was evaluated, and the numerical indicators obtained were better and the human eye effect was better, as shown in Table 1 and Figure 5 shown.

[0074] Table 1. Comparison results of the method disclosed in this embodiment with the existing method in terms of FID, KID and time

[0075]

[0076] From Table 1 and Figure 5As shown, it can be seen that the method disclosed in this embodiment achieves better texture quality in multi-view rendering and significantly accelerates the speed. The order of magnitude of KID in Table 1 is 10 -4 .

[0077] This embodiment also discloses a texture generating device, including:

[0078] A processing module, used for processing the received image to obtain a processing result;

[0079] A projection module, used to project the image pixels in the processing result back to the surface to generate a partial texture map;

[0080] an embedding module, for extracting a global embedding based on the received image, and combining the extracted global embedding into a global conditional embedding;

[0081] An adjustment module, used for using global condition embedding to adjust the internal features of the U-shaped network to obtain an adjusted U-shaped network, wherein each stage of the U-shaped network includes a hybrid 2D-3D module;

[0082] The generation module is used to input the processing results and partial texture mapping into the adjusted U-shaped network for speed and noise prediction, thereby generating texture.

[0083] The electronic device disclosed in this embodiment includes a memory and a processor. The memory is used to store non-temporary computer-readable instructions. Specifically, the memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, etc.

[0084] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of the present disclosure, the processor is used to run the computer-readable instructions stored in the memory, so that the electronic device performs all or part of the steps of the texture generation method of each embodiment of the present disclosure.

[0085] Those skilled in the art should be able to understand that in order to solve the technical problem of how to obtain a good user experience, the present embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the protection scope of the present disclosure.

[0086] like Figure 6The present invention provides a schematic diagram of the structure of an electronic device according to an embodiment of the present invention, which is suitable for implementing the electronic device in the embodiment of the present invention. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0087] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the electronic device are also stored. The processing device, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0088] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as magnetic tapes and hard disks; and communication devices. The communication device can allow the electronic device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0089] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, all or part of the steps of the texture generation method of the embodiment of the present disclosure are executed.

[0090] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0091] The computer-readable storage medium disclosed in this embodiment stores non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the texture generation method of each embodiment of the present disclosure are executed.

[0092] The above-mentioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tapes or mobile hard disks), media with built-in rewritable non-volatile memory (e.g., memory cards), and media with built-in ROM (e.g., ROM boxes).

[0093] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0094] The computer program product disclosed in this embodiment includes a computer program / instruction, which, when executed by a processor, implements all or part of the steps of the texture generation method of each embodiment of the present disclosure.

[0095] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0096] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0097] In the present disclosure, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagram of the device, apparatus, equipment, and system involved in the present disclosure is only an illustrative example and is not intended to require or imply that it must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The vocabulary "or" and "and" and "used here refer to vocabulary "and / or", and can be used interchangeably with them, unless the context clearly indicates otherwise. The vocabulary "such as" used here refers to phrases "such as but not limited to", and can be used interchangeably with them.

[0098] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Moreover, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0099] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0100] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0101] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0102] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A texture generation method, characterized in that: include: Processing the received image to obtain a processing result, wherein the processing result includes a noise texture map, a position map, a single image, a text prompt, and a time step; Projecting the resulting image pixels back onto the surface to generate a partial texture map; extracting a global embedding based on the received image, and combining the extracted global embeddings into a global conditional embedding; Using global conditional embedding to adjust the internal features of the U-shaped network, obtaining an adjusted U-shaped network, wherein each stage of the U-shaped network includes a hybrid 2D-3D module; The processed results and partial texture maps are input into the adjusted U-shaped network for speed and noise prediction to generate texture; Extracting a global embedding based on the received image, and combining the extracted global embedding into a global conditional embedding, including: Extract global embeddings using image and text encoders; Processing the global embedding through a multi-layer perceptron, and combining the global conditional embedding based on the multi-layer perceptron processing structure; The hybrid 2D-3D module includes a UV module and several 3D point cloud blocks; The UV features are processed by the 2D convolution block of the UV module to extract local features in the UV space; The hybrid 2D-3D module includes remapping the extracted local features in the UV space back to the 3D space, and reorganizing the UV features into 3D point cloud features, thereby obtaining 3D neighborhood relations and global structural features to improve 3D consistency.

2. A texture generating device, characterized in that: include: A processing module, used to process the received image to obtain a processing result, wherein the processing result includes a noise texture map, a position map, a single image, a text prompt and a time step; A projection module, used to project the image pixels in the processing result back to the surface to generate a partial texture map; an embedding module, for extracting a global embedding based on the received image, and combining the extracted global embedding into a global conditional embedding; An adjustment module, used for using global condition embedding to adjust the internal features of the U-shaped network to obtain an adjusted U-shaped network, wherein each stage of the U-shaped network includes a hybrid 2D-3D module; A generation module, for inputting the processing results and partial texture mapping into the adjusted U-shaped network for speed and noise prediction, thereby generating texture; Extracting a global embedding based on the received image, and combining the extracted global embedding into a global conditional embedding, including: Extract global embeddings using image and text encoders; Processing the global embedding through a multi-layer perceptron, and combining the global conditional embedding based on the multi-layer perceptron processing structure; The hybrid 2D-3D module includes a UV module and several 3D point cloud blocks; The UV features are processed by the 2D convolution block of the UV module to extract local features in the UV space; The hybrid 2D-3D module includes remapping the extracted local features in the UV space back to the 3D space, and reorganizing the UV features into 3D point cloud features, thereby obtaining 3D neighborhood relations and global structural features to improve 3D consistency.

3. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the texture generation method according to claim 1.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the texture generation method according to claim 1.

5. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the texture generation method according to claim 1 is implemented.

Images