Texture image generation method, training method and device

Abstract

The present disclosure provides a texture image generation method, training method, and apparatus, relating to the field of artificial intelligence technology, particularly computer vision, deep learning, large models, and other technical fields, and can be applied to scenarios such as AIGC-based content generation based on artificial intelligence. A specific implementation of the generation method is as follows: extracting text features of text description information for a target object; generating a two-dimensional image based on three-dimensional information about the target object; wherein the two-dimensional image indicates three-dimensional point cloud information and texture distribution information of the target object; processing the two-dimensional image to generate three-dimensional structure fusion features; generating target texture features based on the three-dimensional structure fusion features and text features; and generating a target texture image based on the target texture features, wherein the target texture image is used to generate a virtual image of the target object.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, particularly computer vision, deep learning, and large models, and can be applied to scenarios such as AIGC-based content generation based on artificial intelligence. Specifically, it relates to a texture image generation method, training method, and device. Background Art

[0002] AIGC (AI generated content) can perform tasks such as generating text, images, and virtual images based on artificial intelligence technology, such as digital humans.

[0003] Texture is caused by the diversity of physical properties on an object's surface. Different physical properties represent different grayscale or color information for a specific surface feature. Different physical surfaces produce different texture images. Therefore, texture, as a crucial attribute of an image, plays a crucial role in computer vision and image processing. The geometric consistency between texture images and three-dimensional objects has a significant impact on the generation of virtual images. Summary of the Invention

[0004] The present disclosure provides a texture image generation method, device, electronic device, and storage medium.

[0005] According to one aspect of the present disclosure, a texture image generation method is provided, comprising: extracting text features of text description information for a target object; generating a two-dimensional image based on three-dimensional information for the target object; wherein the two-dimensional image indicates three-dimensional point cloud information and texture distribution information of the target object; processing the two-dimensional image to generate three-dimensional structure fusion features; generating target texture features based on the three-dimensional structure fusion features and text features; and generating a target texture image based on the target texture features, wherein the target texture image is used to generate a virtual image for the target object.

[0006] According to another aspect of the present disclosure, a training method for a texture image generation model is provided, comprising: extracting sample text features of text description information for a sample object; generating a two-dimensional sample image based on three-dimensional information for the sample object; wherein the two-dimensional sample image indicates three-dimensional point cloud information and a texture distribution image of the sample object; processing the two-dimensional sample image to generate a sample three-dimensional structure fusion feature; extracting sample texture features of the texture image for the sample object; adding sample noise to the sample texture features to generate a sample texture feature containing noise; under the guidance of the sample three-dimensional structure fusion feature and the sample text feature, guiding the sample texture feature containing noise to generate a sample texture image and noise information in the sample texture image; generating a loss value based on the objective function and the sample noise and noise information; and adjusting the model parameters of the texture image generation model based on the loss value to obtain a trained texture image generation model.

[0007] According to another aspect of the present disclosure, a texture image generation device is provided, comprising: a feature extraction module, an information processing module, a feature fusion module, a feature generation module, and an image generation module.

[0008] A feature extraction module is configured to extract text features from the text description information for the target object. An information processing module is configured to generate a two-dimensional image based on the three-dimensional information of the target object; the two-dimensional image indicates the three-dimensional point cloud information and texture distribution information of the target object. A feature fusion module is configured to process the two-dimensional image to generate three-dimensional structural fusion features. A feature generation module is configured to generate target texture features based on the three-dimensional structural fusion features and the text features. An image generation module is configured to generate a target texture image based on the target texture features, wherein the target texture image is used to generate a virtual image of the target object.

[0009] According to another aspect of the present disclosure, a training device for a texture image generation model is provided, comprising: a first extraction module, a two-dimensional image generation module, a two-dimensional image processing module, a second extraction module, a noise addition module, a generation module, a loss calculation module and an adjustment module.

[0010] The first extraction module is configured to extract sample text features from the text description information of the sample object. The two-dimensional image generation module is configured to generate a two-dimensional sample image based on the three-dimensional information of the sample object; the two-dimensional sample image indicates the three-dimensional point cloud information and texture distribution image of the sample object. The two-dimensional image processing module is configured to process the two-dimensional sample image to generate a sample three-dimensional structure fusion feature. The second extraction module is configured to extract sample texture features from the texture image of the sample object. The noise addition module is configured to add sample noise to the sample texture features to generate a noisy sample texture feature. The generation module is configured to, under the guidance of the sample three-dimensional structure fusion feature and the sample text feature, guide the noisy sample texture feature to generate a sample texture image and noise information in the sample texture image. The loss calculation module is configured to generate a loss value based on the objective function, the sample noise, and the noise information. The adjustment module is configured to adjust the model parameters of the texture image generation model based on the loss value to obtain a trained texture image generation model.

[0011] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0012] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described above.

[0013] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method described above when executed by a processor.

[0014] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0016] Figure 1 Schematically illustrates an exemplary system architecture to which a method and apparatus for training texture image generation or a texture image generation model according to an embodiment of the present disclosure may be applied;

[0017] Figure 2 The flowchart of the texture image generation method according to the embodiment of the present disclosure is schematically shown;

[0018] Figure 3A Schematically shows a schematic diagram of generating a texture image according to text description and three-dimensional information according to an embodiment of the present disclosure;

[0019] Figure 3B Schematically shows a schematic diagram of generating a texture image according to text description and three-dimensional information according to another embodiment of the present disclosure;

[0020] Figure 4 The flowchart of the training method of the texture image generation model according to the embodiment of the present disclosure is schematically shown;

[0021] Figure 5 Schematically shows a network architecture diagram of a texture image generation model according to an embodiment of the present disclosure;

[0022] Figure 6 A schematic diagram schematically shows the structure of the encoding unit in the texture image generation model according to an embodiment of the present disclosure; and

[0023] Figure 7 Schematically shows a block diagram of a texture image generating device according to an embodiment of the present disclosure;

[0024] Figure 8 A block diagram schematically illustrates a training apparatus for a texture image generation model according to an embodiment of the present disclosure; and

[0025] Figure 9 A block diagram of an electronic device suitable for implementing a method for training a texture image generation or a texture image generation model according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0026] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] Texture generation refers to the process of generating surface texture details of three-dimensional objects through generative models, such as GAN (Generative adversarial nets) or diffusion models.

[0028] In this example, a 2D diffusion model can be used to generate 2D images from multiple perspectives (e.g., 4, 6, 8, or 16 perspectives) based on a 3D mesh of a 3D object. The multi-perspective images are then mapped onto the 3D mesh according to their mapping relationship to generate a texture image.

[0029] However, due to the deviation of camera pose in the process of generating images from different perspectives, for example, there is a deviation between the front view and the left view of the same part of a three-dimensional object, resulting in poor three-dimensional geometric consistency in the generated texture image.

[0030] In related examples, a three-dimensional diffusion model can also be used to generate texture on the surface of the object in three-dimensional space based on the three-dimensional mesh of the three-dimensional object as a model input.

[0031] However, due to the scarcity of 3D mesh samples of 3D objects, the generalization performance of texture generation models directly learned in 3D space is poor.

[0032] In view of this, the embodiments of the present disclosure provide a method for generating a texture image, which utilizes a two-dimensional image containing three-dimensional point cloud information and texture distribution information of a target object, and generates a target texture image under the guidance of three-dimensional structural fusion features that fuse the three-dimensional point cloud and texture distribution, and text features of text description information. Because the three-dimensional structural information of the three-dimensional object is stored in the two-dimensional image, texture generation under three-dimensional position control is achieved, and the three-dimensional geometric consistency of the texture image is improved. At the same time, because the texture image is generated based on the two-dimensional image and text description information, it no longer relies on three-dimensional mesh data, thereby improving the generalization capability of the model applicable to the texture image generation method.

[0033] Figure 1 An exemplary system architecture of a method and apparatus for training texture image generation or a texture image generation model according to an embodiment of the present disclosure is schematically illustrated.

[0034] It should be noted that Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present disclosure may be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments, or scenarios. For example, in another embodiment, an exemplary system architecture to which the method and apparatus for training texture image generation or a texture image generation model may be applied may include a terminal device, but the terminal device may implement the method and apparatus for training texture image generation or a texture image generation model provided by the embodiments of the present disclosure without interacting with a server.

[0035] like Figure 1As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0036] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software (for example only).

[0037] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0038] Server 105 may be a server that provides various services, such as a background management server (for example only) that supports content browsed by users using terminal devices 101, 102, and 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal device.

[0039] It should be noted that the texture image generation or texture image generation model training method provided in the embodiments of the present disclosure can generally be executed by the terminal device 101, 102, or 103. Correspondingly, the texture image generation or texture image generation model training device provided in the embodiments of the present disclosure can also be set in the terminal device 101, 102, or 103.

[0040] Alternatively, the texture image generation or texture image generation model training method provided in the embodiment of the present disclosure may also be generally executed by the server 105. Accordingly, the texture image generation or texture image generation model training device provided in the embodiment of the present disclosure may generally be set in the server 105. The texture image generation or texture image generation model training method provided in the embodiment of the present disclosure may also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the texture image generation or texture image generation model training device provided in the embodiment of the present disclosure may also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0042] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0043] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information. The images used in the examples are all from public image data sources.

[0044] Figure 2 A flowchart of a method for generating a texture image or training a texture image generation model according to an embodiment of the present disclosure is schematically shown.

[0045] like Figure 2 As shown, the method 200 includes operations S210 to S250.

[0046] In operation S210 , text features of text description information for a target object are extracted.

[0047] In operation S220 , a two-dimensional image is generated based on the three-dimensional information on the target object.

[0048] In operation S230 , the two-dimensional image is processed to generate a three-dimensional structure fusion feature.

[0049] In operation S240 , a target texture feature is generated based on the three-dimensional structure fusion feature and the text feature.

[0050] In operation S250 , a target texture image is generated based on the target texture features.

[0051] According to embodiments of the present disclosure, a target object can be any object in a three-dimensional space, such as a car, a person, clothing, an animal, or a plant. The text description information is used to describe the appearance of the target object. For example, if the target object is a two-dimensional character, the text description information may include the character's skin color, appearance, and clothing.

[0052] According to embodiments of the present disclosure, three-dimensional information can be used to describe the three-dimensional geometric shape of a target object. This three-dimensional geometric shape indicates the position of the target object in three-dimensional space. Based on the mapping relationship between three-dimensional space and two-dimensional space, the position of the target object in three-dimensional space can be mapped to the three-dimensional space to obtain a two-dimensional image.

[0053] The 2D image contains the 3D point cloud information and texture distribution information of the target object. The 3D point cloud information can be used to characterize the 3D structure of the target object. The texture distribution information can be used to characterize the texture of the target object at various locations in 3D space. Therefore, 3D structure fusion features can be generated by performing feature fusion on the information in the 2D image.

[0054] Since the three-dimensional structure fusion feature fuses the three-dimensional structure and texture distribution of the target object, when generating a texture image, the three-dimensional geometric consistency of the texture distribution at each position in the target texture image can be further improved.

[0055] For example, three-dimensional point cloud features and texture distribution features can be extracted separately from two-dimensional images, and then the three-dimensional point cloud features and texture distribution features can be fused using any one or several combined feature fusion methods such as feature splicing, feature fusion, corresponding element multiplication, pyramid pooling, deconvolution, multi-scale feature fusion, and weighted summation to generate three-dimensional structure fusion features.

[0056] According to embodiments of the present disclosure, by fusing 3D structural features with textual features, a target texture feature can be generated, guided by the 3D structure and textual description. Then, by processing the target texture feature, a target texture image is generated. The target texture image is used to generate a virtual avatar of the target object.

[0057] For example, a diffusion model can be used to learn the 3D structure and latent structure of each sample object in a sample dataset through diffusion in the latent space under the guidance of the 3D structure and text description. This allows the model to control the generation of a texture image for the target object based on the 3D structure and text description. The target texture image can then be generated by decoding the target texture features.

[0058] For example, a generative adversarial network can be used to generate a target texture image that is sufficient to be identified by a discriminator by inputting three-dimensional structure fusion features and text features into the generative adversarial network.

[0059] According to an embodiment of the present disclosure, a target texture image is generated using a two-dimensional image containing the three-dimensional point cloud information and texture distribution information of the target object, guided by three-dimensional structural fusion features that fuse the three-dimensional point cloud and texture distribution, and text features of the text description information. Because the three-dimensional structural information of the three-dimensional object is stored in the two-dimensional image, texture generation under three-dimensional position control is achieved, improving the three-dimensional geometric consistency of the texture image. At the same time, because the texture image is generated based on the two-dimensional image and text description information, it no longer relies on three-dimensional mesh data, thereby improving the generalization capability of the model applicable to this texture image generation method.

[0060] Reference below Figure 3A and Figure 3B , combined with specific embodiments Figure 2 The method shown is further explained.

[0061] Figure 3A The diagram schematically shows a method of generating a texture image based on text description and three-dimensional information according to an embodiment of the present disclosure.

[0062] like Figure 3A As shown, in embodiment 300A, the target object may be a horse-faced skirt with orchid patterns. The text description information for the target object is "orchid pattern horse-faced skirt" 311. The text description information may be a text prompt directly input by the user or text information converted from the user's voice command.

[0063] First, the three-dimensional information 321 of the orchid pattern horse-face skirt may be processed to obtain a two-dimensional image 322 of the orchid pattern horse-face skirt.

[0064] According to an embodiment of the present disclosure, generating a two-dimensional image based on three-dimensional information of a target object may include the following operations: obtaining texture information of each first position of the target object in the three-dimensional space from the three-dimensional information; and based on a position mapping relationship between the three-dimensional space and the two-dimensional space, mapping the texture information of each first position to each second position in the corresponding two-dimensional space to generate a two-dimensional image.

[0065] For example: according to the three-dimensional geometric shape described by the three-dimensional information 321 and the UV coordinate system, the surface texture of the orchid-patterned horse-face skirt can be unfolded. During the unfolding process, each point of the orchid-patterned horse-face skirt in the three-dimensional space will be mapped to the target position in the two-dimensional image according to the mapping relationship between the three-dimensional space and the two-dimensional space, thereby generating a two-dimensional image 322.

[0066] Generating a 2D image based on 3D information, which indicates the target object's 3D point cloud and texture distribution, provides guidance on the 3D structure for texture generation, improving the 3D geometric consistency of the target texture. Furthermore, generating a 2D image based on 3D information reduces the difficulty of acquiring datasets for model training, compared to directly using 3D mesh data as model input in related examples, thereby improving the model's generalization capabilities.

[0067] Next, text features 312 are extracted from “orchid pattern horse-faced skirt” 311. For example, a text prompt can be constructed based on “orchid pattern horse-faced skirt” 311, and the text prompt is input into the T5 (Transfer Text-to-Text Transformer) model to output text features 312.

[0068] Then, the three-dimensional point cloud features 331 and texture distribution features 332 can be extracted from the two-dimensional image 322 respectively; based on the attention mechanism, the three-dimensional point cloud features 331 and texture distribution features 332 are fused to generate a three-dimensional structure fusion feature 333.

[0069] The attention mechanism can be used to focus on important information with high weights and ignore unimportant information with low weights. It can also exchange important information with other information by sharing it, thereby achieving the transmission of important information. For example, the attention mechanism can be a cross-attention mechanism.

[0070] Therefore, based on the attention mechanism, the three-dimensional point cloud features and texture distribution features are fused, and the obtained three-dimensional structure fusion features indicate the three-dimensional structure corresponding to the texture distribution, further improving the three-dimensional geometric consistency of the texture.

[0071] The embodiments of the present disclosure can be implemented based on a diffusion model. For example, generating a target texture feature based on a three-dimensional structure fusion feature and a text feature may include the following operations: based on an attention mechanism, fusing the initial noise 301 with the text feature 312 to generate a first intermediate feature 341; based on an attention mechanism, fusing the initial noise 301 with the three-dimensional structure fusion feature 333 to generate a second intermediate feature 342; fusing the first intermediate feature 341 with the second intermediate feature 342 to generate a third intermediate feature 343 and generating a target texture feature 344 based on the third intermediate feature 343.

[0072] According to an embodiment of the present disclosure, the initial noise may be random noise conforming to a Gaussian distribution. The first intermediate feature indicates appearance description information for the target object. The second intermediate feature indicates three-dimensional point cloud information and texture distribution information for the target object.

[0073] According to an embodiment of the present disclosure, the attention mechanism may be an interactive attention mechanism.

[0074] The operation of fusing the first intermediate feature with the second intermediate feature includes, but is not limited to, an addition operation, a concatenation operation, and a connection operation.

[0075] For example, the initial noise can be gradually fused with the text features and the three-dimensional structure fusion features according to a predetermined time step, so as to guide the latent variables of the initial noise to generate the target texture features according to the text description and the three-dimensional structure.

[0076] According to an embodiment of the present disclosure, by interactively fusing the initial noise with text features and three-dimensional structure fusion features respectively, the third intermediate feature includes both the appearance description of the target object and the three-dimensional structure and texture distribution, thereby realizing the generation of texture based on three-dimensional position control.

[0077] According to an embodiment of the present disclosure, generating a target texture feature based on the third intermediate feature may include the following operation: gradually adding initial noise to the third intermediate feature at a predetermined time step to generate the target texture feature.

[0078] The predetermined time step can serve as a control condition for the diffusion model. The initial noise added to the text features and 3D structure fusion features can also be added at the predetermined time step. The initial noise added to the third intermediate feature can also be added at the same predetermined time step. This allows the diffusion model to generate target texture features, guided by the text features and 3D structure fusion features.

[0079] Since the target texture features contain noise, during the feature decoding process, noise removal can be performed according to a predetermined time step to efficiently remove the noise and generate the target texture image, further improving the generation efficiency and accuracy of the target texture image.

[0080] As the demand for generation accuracy increases, the complexity of the network architecture of the texture image generation model is also increasing, and the texture image generation path is also growing. The information transmission of each feature in the generation model is also becoming more and more complex.

[0081] Therefore, in order to ensure the effectiveness of information transmission of each feature in the generative model, the probability of information omission of each feature in the information transmission process can be reduced by referencing the residual structure.

[0082] Figure 3B The diagram schematically shows a schematic diagram of generating a texture image according to text description and three-dimensional information according to another embodiment of the present disclosure.

[0083] like Figure 3B As shown, the difference between embodiment 300B and embodiment 300A is that a residual structure is introduced in the generation process of the three-dimensional structure fusion feature and the target texture feature.

[0084] To generate the 3D structure fusion feature, the 3D point cloud feature 331 and the texture distribution feature 332 can be fused to generate an initial fused feature 334. To improve the 3D geometric consistency of the texture, the 3D point cloud feature 331 and the initial fused feature 334 can be further fused to generate a 3D structure fusion feature. Fusion operations include, but are not limited to, feature addition and feature concatenation.

[0085] According to an embodiment of the present disclosure, based on the attention mechanism, the three-dimensional point cloud features and the texture distribution features are fused to generate the three-dimensional structure fusion features, which may include the following operations: based on the attention mechanism, the three-dimensional point cloud features and the texture distribution features are fused to generate the initial fusion features; the three-dimensional point cloud features are fused with the initial fusion features to generate the intermediate fusion features; and the three-dimensional structure fusion features are generated based on the intermediate fusion features.

[0086] For example, the interactive attention mechanism can be used to fuse 3D point cloud features with texture distribution features to generate an initial fused feature. Because the interactive attention mechanism has already filtered out non-essential information based on weight distribution, the intermediate fused feature generated by fusing the 3D point cloud features with the initial fused feature re-references non-essential information. Therefore, to improve model processing efficiency and reduce the processing of redundant features, a multi-layer perceptron (MLP) can be used to process the intermediate fused features to generate a 3D structure fused feature.

[0087] By re-fusing the 3D point cloud features with the initial fused features, the probability of information omission during the information transfer process can be reduced. By filtering the intermediate fused features, the redundant information in the 3D structure fused features is reduced, and the image generation efficiency is improved.

[0088] Regarding the process of generating the target texture feature, according to an embodiment of the present disclosure, the above method also includes the following operations: fusing the noise feature of the initial noise and the third intermediate feature to generate a fourth intermediate feature; and generating the target texture feature based on the fourth intermediate feature.

[0089] For example, noise feature 302 can be extracted from initial noise 301. Then, third intermediate feature 343 and noise feature 302 can be fused based on the self-attention mechanism to generate fourth intermediate feature 345. Fourth intermediate feature 345 can be processed using a multilayer perceptron (MLP) to generate target texture features.

[0090] According to the embodiments of the present disclosure, by fusing the third intermediate feature with the noise feature, the probability of information omission in the fusion process between the text feature and the 3D structure fusion feature can be reduced. By filtering the fourth intermediate feature, redundant information in the noise feature is reduced, thereby improving image generation efficiency.

[0091] Figure 4 The flowchart of the training method of the texture image generation model according to the embodiment of the present disclosure is schematically shown.

[0092] like Figure 4 As shown, the training method 400 includes operations S410 to S480.

[0093] In operation S410 , sample text features of text description information for a sample object are extracted.

[0094] In operation S420, a two-dimensional sample image is generated based on the three-dimensional information on the sample object.

[0095] In operation S430 , the two-dimensional sample image is processed to generate a sample three-dimensional structure fusion feature.

[0096] In operation S440 , sample texture features of the texture image for the sample object are extracted.

[0097] In operation S450, sample noise is added to the sample texture feature to generate a sample texture feature containing noise.

[0098] In operation S460 , under the guidance of the sample three-dimensional structure fusion feature and the sample text feature, the sample texture feature containing noise is guided to generate a sample texture image and noise information in the sample texture image.

[0099] In operation S470, a loss value is generated according to the sample noise and the noise information based on the objective function.

[0100] In operation S480, model parameters of the texture image generation model are adjusted based on the loss value to obtain a trained texture image generation model.

[0101] According to an embodiment of the present disclosure, the two-dimensional sample image indicates three-dimensional point cloud information and a texture distribution image of the sample object.

[0102] According to an embodiment of the present disclosure, the description scope of the text description information and the three-dimensional information of the sample object is the same as the description scope of the text description information and the three-dimensional information of the target object described above, and is not repeated here.

[0103] According to embodiments of the present disclosure, the texture image of the target object can be a noise-free texture map. By continuously adding noise to this texture image, the model, guided by the fusion of sample text features and sample 3D structure features, continuously learns the underlying structure in the texture image in a diffusion manner, enabling the model to generate texture images.

[0104] This training method involves a forward diffusion process and an inverse generation process. The forward diffusion process gradually transforms the texture image of the sample object into pure noise by gradually adding noise. This process is typically a Markov chain, with a small amount of Gaussian noise added at each step, ultimately making the data distribution approach a standard Gaussian distribution. The inverse generation process learns how to gradually remove noise from the noisy sample texture features to restore the original texture image. This process is also a Markov chain, but in the opposite direction, gradually removing noise. The inverse generation process involves complex probability distribution estimation to ensure the generated samples are of high fidelity and diversity. By training the model to learn this inverse process, it is ultimately possible to reconstruct a high-quality texture image of the sample object from pure noise.

[0105] Since the three-dimensional point cloud information and texture distribution information of the sample object are stored in the two-dimensional sample image, there is no need to use scarce three-dimensional grid data to train the model, thereby improving the generalization ability of the model.

[0106] Figure 5 The following schematically shows a network architecture diagram of a texture image generation model according to an embodiment of the present disclosure.

[0107] like Figure 5 As shown, in this embodiment 500 , the forward diffusion process can be implemented by the encoding network 580 , and the reverse generation process can be implemented by the decoding network 590 .

[0108] The input of the encoding network 580 may include sample text features 501, sample time series vectors 502, noisy sample texture features 503, and three-dimensional structure fusion features 507. The extraction or generation process of these features are independent of each other and can be parallel or serial.

[0109] For example, a sample text description may be input into T5 (Transfer Text-to-Text Transformer) 510 , which outputs sample text features 501 .

[0110] For example, a preset time step for adding noise may be input into the MLP 520 , and a sample time series vector 502 may be output.

[0111] According to an embodiment of the present disclosure, adding sample noise to the sample texture feature to generate the sample texture feature containing noise may include the following operations: adding sample noise to the sample texture feature in sequence according to a predetermined time step to generate the sample texture feature containing noise for each moment; wherein the sample noise at each moment is Gaussian distributed.

[0112] For example, a VAE1 (Variational Autoencoder) 530a may be used to add sample noise to the texture image input of the sample object according to a preset time step to perform latent variable encoding, thereby obtaining the noisy sample texture feature 503.

[0113] By gradually adding noise to the texture image of the sample object, it gradually transforms into pure noise. Ultimately, the data distribution is made close to the standard Gaussian distribution. The trained model can ultimately reconstruct high-quality texture images from pure noise.

[0114] According to an embodiment of the present disclosure, generating a two-dimensional sample image based on three-dimensional information of a sample object may include the following operations: obtaining sample texture information of each first position of the sample object in the three-dimensional space from the three-dimensional information of the sample object; and mapping the sample texture information of each first position to each second position in the corresponding two-dimensional space based on a position mapping relationship between the three-dimensional space and the two-dimensional space to generate a two-dimensional image.

[0115] For example, the 3D sample information can be input into the texture mapping module 540 to obtain the 2D sample image 504. The process of generating the 2D sample image is the same as that of the 2D image in the texture image generation method described above, and will not be described in detail here.

[0116] According to the embodiments of the present disclosure, the two-dimensional sample image stores the three-dimensional structure and texture distribution information of the sample object in the form of a two-dimensional image. Therefore, using the two-dimensional sample image as a model input can also provide guidance information of the three-dimensional structure and texture distribution, enabling the model to generate a texture image with three-dimensional geometric consistency.

[0117] According to an embodiment of the present disclosure, processing a two-dimensional sample image to generate a sample three-dimensional structure fusion feature may include the following operations: extracting the sample three-dimensional point cloud feature and the sample texture distribution feature from the two-dimensional sample image respectively; and based on the attention mechanism, fusing the sample three-dimensional point cloud feature and the sample texture distribution feature to generate a sample three-dimensional structure fusion feature.

[0118] For example, a two-dimensional sample image 504 can be input into a VAE2 (Variational Autoencoder) 530b to obtain a sample texture distribution feature 505. Simultaneously, the two-dimensional sample image 504 is input into a point cloud extraction module 550 to obtain a sample point cloud feature 506. The sample texture distribution feature 505 and the sample point cloud feature 506 are then input into an attention module 560 for interactive attention calculation.

[0119] In some embodiments, the features after interactive attention calculation are input into a feedforward neural network to obtain the sample three-dimensional structure fusion features.

[0120] In order to reduce the probability of omission of each feature during the information transmission process, in some embodiments, based on the attention mechanism, the sample three-dimensional point cloud features and the sample texture distribution features can be fused to generate the sample initial fusion features; the sample three-dimensional point cloud features and the sample initial fusion features can be fused to generate the sample intermediate fusion features; and the sample three-dimensional structure fusion features can be generated based on the sample intermediate fusion features.

[0121] like Figure 5 As shown, the sample texture distribution feature 505 is added to the initial fusion feature output by the attention module 560, and then input into the FFN570 (Feed Forward Network) to obtain the sample three-dimensional structure fusion feature 507.

[0122] By adopting the residual grid structure and introducing the previous features in the process of transmitting each feature, the probability of feature omission can be reduced and the efficiency of model training can be improved.

[0123] The coding network 580 may include N cascaded coding units, and the structure of each coding unit may be the same.

[0124] For example, the sample texture feature 501, the sample time-series vector 502, the noisy sample texture feature 503, and the sample 3D structure fusion feature 507 can be input into the first encoding unit 581, which outputs a first latent variable feature. This first latent variable feature and the sample 3D structure fusion feature 507 can then be input into the second encoding unit 582, which outputs a second latent variable feature. Similarly, each encoding unit needs to input the sample 3D structure fusion feature. Under the guidance of the sample 3D structure fusion feature, the latent structure in the texture image of the sample object is learned, guiding the latent variable feature reconstruction to generate the noisy sample texture feature 503.

[0125] Figure 6 The schematic diagram shows the structure of the coding unit in the texture image generation model according to an embodiment of the present disclosure.

[0126] like Figure 6 As shown, the first encoding unit 581 performs an interactive fusion operation between the noisy sample texture feature 503 and the sample text feature 501 and an interactive fusion operation between the noisy sample texture feature 503 and the sample three-dimensional structure fusion feature.

[0127] According to an embodiment of the present disclosure, under the guidance of the sample three-dimensional structure fusion feature and the sample text feature, guiding the sample texture feature containing noise to generate a sample texture image and noise information in the sample texture image can include the following operations: based on the attention mechanism, processing the sample texture feature containing noise to generate the latent feature of the sample texture. Based on the attention mechanism, fusing the latent feature of the sample texture with the sample text feature to generate a first sample intermediate feature. Based on the attention mechanism, fusing the latent feature of the sample texture with the three-dimensional structure fusion feature to generate a second sample intermediate feature. Fusing the first sample intermediate feature and the second sample intermediate feature to generate a third sample intermediate feature; and generating the sample texture image and the noise information in the sample texture image based on the third sample intermediate feature.

[0128] For example, the noisy sample texture features 503 can be input into the first scaling / offset module 5810, which outputs the noisy sample texture features of a predetermined size. The noisy sample texture features of the predetermined size are then input into the multi-head self-attention module 5820 for self-attention interaction, further extracting latent variables that can characterize the underlying structure of the texture image. The latent variables are then input into the first scaling module 5830 for scaling, allowing them to interact with the sample text features 501 and the sample 3D structure fusion features 507, respectively.

[0129] In some embodiments, in order to reduce the risk of information omission during feature transfer, the noisy sample texture feature 503 may be added to the scaled latent variable to obtain a latent feature of the sample texture.

[0130] Next, the sample text features 501 and the latent features of the sample texture can be input into the text interaction attention module 5840, which outputs a first sample intermediate feature. The first sample intermediate feature indicates appearance description information for the sample object. Simultaneously, the sample 3D structure fusion features 507 and the latent features of the sample texture can be input into the structure interaction attention module 5850, which outputs a second sample intermediate feature. The second sample intermediate feature indicates 3D point cloud information and texture distribution information for the sample object.

[0131] In some embodiments, in order to reduce the risk of information omission during feature transmission, the potential features of the sample texture and the third sample intermediate features can be fused to generate a fourth sample intermediate feature; and the target sample texture feature can be generated based on the fourth intermediate feature.

[0132] For example, the first sample intermediate feature, the second sample intermediate feature, and the latent feature of the sample texture may be added to obtain a fourth sample intermediate feature.

[0133] The fourth sample intermediate feature is then input into the second scaling / offset module 5860. The scaling scale and offset ratio configured by the second scaling / offset module 5860 correspond to those of the first scaling / offset module 5810. The scaled and offset intermediate feature is then input into the FFN 5870 and passed through the second scaling module 5890. The scaling scale configured by the second scaling module 5890 corresponds to that of the first scaling module 5830. This results in the noisy sample texture reconstructed feature 510, i.e., the target sample texture feature.

[0134] Guided by the fusion features of sample text features and sample 3D structure, the model learns the latent feature information in the texture image. This latent feature information contains both texture distribution and 3D structure, thus enabling the model to reconstruct latent features that conform to 3D geometric consistency.

[0135] In some embodiments, in order to reduce the risk of information omission during feature transmission, the features processed by the second scaling module 5890 and the fourth sample intermediate features may be added to obtain the noisy sample texture reconstruction feature 510 .

[0136] The operation performed by the second scaling / offset module 5860 is inverse to the operation performed by the first scaling / offset module 5810. The operation performed by the first scaling module 5830 is inverse to the operation performed by the second scaling module 5890. Therefore, the sample timing vector 502 is required to be input to the first scaling module 5830, the second scaling module 5890, the first scaling / offset module 5810, and the second scaling / offset module 5860 in order to convey the timing characteristics of the sample noise addition.

[0137] Finally, the decoding network 590 is used to decode and denoise the noisy sample texture reconstruction features to generate a sample texture image 508 and noise 509.

[0138] According to an embodiment of the present disclosure, the sample noise is a noise time series sequence that conforms to a Gaussian distribution; based on the objective function, a loss value is generated according to the sample noise and noise information, which may include the following operations: extracting the target sample noise at the target moment from the noise time series sequence; extracting the target noise information for the target moment from the noise information in the sample texture image; and based on the objective function, generating a loss value according to the target sample noise and target noise information.

[0139] The texture image generation model is iteratively trained based on noise loss. The goal of training is to enable the model to reconstruct high-quality sample texture features based on pure noise while accurately outputting the added noise.

[0140] Therefore, the loss value can be calculated based on the noise information in the sample noise and sample texture image at the corresponding time. The target time can be any moment in the noise time series that conforms to the Gaussian distribution. When the loss value converges or reaches the maximum number of iterations, it indicates that the training is complete.

[0141] According to the embodiments of the present disclosure, the objective function may be any loss function suitable for training a diffusion model, and the embodiments of the present disclosure do not specifically limit this.

[0142] According to embodiments of the present disclosure, the training method learns the data distribution through a process of forward denoising and backward denoising, and improves the quality of the generated texture by optimizing model parameters. Therefore, optimizing the model parameters based on the difference loss between the noise in the reconstructed image and the initially added noise can improve the training efficiency of the model.

[0143] Figure 7 The block diagram schematically shows a texture image generating apparatus according to an embodiment of the present disclosure.

[0144] like Figure 7 As shown, the generating device 700 includes a feature extraction module 710 , an information processing module 720 , a feature fusion module 730 , a feature generation module 740 and an image generation module 750 .

[0145] The feature extraction module 710 is used to extract text features of the text description information of the target object.

[0146] The mesh processing module 720 is configured to generate a two-dimensional image based on the three-dimensional information of the target object. The two-dimensional image indicates the three-dimensional point cloud information and texture distribution information of the target object.

[0147] The feature fusion module 730 is used to process the two-dimensional image and generate three-dimensional structure fusion features.

[0148] The feature generation module 740 is used to generate target texture features based on the 3D structure fusion features and text features.

[0149] The image generation module 750 is used to generate a target texture image based on target texture features, wherein the target texture image is used to generate a virtual image for the target object.

[0150] According to an embodiment of the present disclosure, the feature fusion module includes a feature extraction submodule and a feature fusion submodule. The feature extraction submodule is used to extract 3D point cloud features and texture distribution features from a 2D image. The feature fusion submodule is used to fuse the 3D point cloud features and texture distribution features based on an attention mechanism to generate 3D structure fusion features.

[0151] According to an embodiment of the present disclosure, the feature fusion submodule includes: a first attention unit, a first fusion unit, and a first processing unit. The first attention unit is configured to fuse 3D point cloud features and texture distribution features based on an attention mechanism to generate initial fused features. The first fusion unit is configured to fuse 3D point cloud features with initial fused features to generate intermediate fused features. The first processing unit is configured to generate 3D structure fusion features based on the intermediate fused features.

[0152] According to an embodiment of the present disclosure, the feature generation module includes: a text feature fusion submodule, a structural feature fusion submodule, an intermediate feature fusion submodule and a generation submodule.

[0153] The text feature fusion submodule is used to fuse the initial noise with the text feature based on the attention mechanism to generate a first intermediate feature; wherein the first intermediate feature indicates the appearance description information for the target object.

[0154] The structural feature fusion submodule is used to fuse the initial noise with the three-dimensional structural fusion feature based on the attention mechanism to generate a second intermediate feature; wherein the second intermediate feature indicates the three-dimensional point cloud information and texture distribution information of the target object.

[0155] The intermediate feature fusion submodule is used to fuse the first intermediate feature and the second intermediate feature to generate a third intermediate feature.

[0156] The generating submodule is used to generate target texture features based on the third intermediate features.

[0157] According to an embodiment of the present disclosure, the generation submodule includes a noise fusion submodule and a texture feature generation submodule. The noise fusion submodule is configured to fuse the noise feature of the initial noise with the third intermediate feature to generate a fourth intermediate feature. The texture feature generation submodule is configured to generate a target texture feature based on the fourth intermediate feature.

[0158] According to an embodiment of the present disclosure, the mesh processing module includes a first acquisition submodule and a second mapping submodule. The first acquisition submodule is configured to acquire texture information of a target object at each first position in three-dimensional space from three-dimensional information. The second mapping submodule is configured to map the texture information at each first position to each corresponding second position in two-dimensional space based on a position mapping relationship between three-dimensional space and two-dimensional space, thereby generating a two-dimensional image.

[0159] Figure 8 A block diagram schematically shows a training device for a texture image generation model according to an embodiment of the present disclosure.

[0160] like Figure 8As shown, the training device 800 may include: a first extraction module 810, a two-dimensional image generation module 820, a two-dimensional image processing module 830, a second extraction module 840, a noise addition module 850, a generation module 860, a loss calculation module 870 and an adjustment module 890.

[0161] A first extraction module 810 is used to extract sample text features of text description information of a sample object;

[0162] A two-dimensional image generation module 820 is configured to generate a two-dimensional sample image based on the three-dimensional information of the sample object; wherein the two-dimensional sample image indicates the three-dimensional point cloud information and texture distribution image of the sample object;

[0163] A two-dimensional image processing module 830 is used to process the two-dimensional sample image to generate a three-dimensional structure fusion feature of the sample;

[0164] A second extraction module 840 is used to extract sample texture features of the texture image of the sample object;

[0165] The noise adding module 850 is used to add sample noise to the sample texture feature to generate the sample texture feature containing noise;

[0166] A generating module 860 is used to guide the sample texture features containing noise to generate a sample texture image and noise information in the sample texture image under the guidance of the sample three-dimensional structure fusion features and the sample text features;

[0167] A loss calculation module 870 is configured to generate a loss value based on the objective function, sample noise, and noise information; and

[0168] The adjustment module 890 is used to adjust the model parameters of the texture image generation model based on the loss value to obtain a trained texture image generation model.

[0169] According to an embodiment of the present disclosure, the noise adding module includes: a noise adding submodule, which is used to add sample noise to the sample texture features in sequence according to a predetermined time step to generate a sample texture feature containing noise for each moment; wherein the sample noise at each moment is Gaussian distributed.

[0170] According to an embodiment of the present disclosure, the generation module includes: a first attention submodule, a second attention submodule, a third attention submodule, a fusion submodule and a processing submodule.

[0171] The first attention submodule is used to process the texture features of samples containing noise based on the attention mechanism and generate potential features of the sample texture.

[0172] The second attention submodule is used to fuse the latent features of the sample texture with the sample text features based on the attention mechanism to generate a first sample intermediate feature; wherein the first sample intermediate feature indicates the appearance description information for the sample object.

[0173] The third attention submodule is used to fuse the latent features of the sample texture with the 3D structure fusion features based on the attention mechanism to generate a second sample intermediate feature. The second intermediate feature indicates the 3D point cloud information and texture distribution information of the sample object.

[0174] The fusion submodule is used to fuse the first sample intermediate feature with the second sample intermediate feature to generate a third sample intermediate feature.

[0175] The processing submodule is used to generate a sample texture image and noise information in the sample texture image based on the third sample intermediate feature.

[0176] According to an embodiment of the present disclosure, the training device further includes a fusion module and a processing module. The fusion module is configured to fuse the latent features of the sample texture with the third sample intermediate features to generate a fourth sample intermediate feature. The processing module is configured to generate a target sample texture feature based on the fourth intermediate feature.

[0177] According to an embodiment of the present disclosure, a two-dimensional image generation module includes: a second acquisition submodule and a second mapping submodule. The second acquisition submodule is configured to acquire sample texture information of each first position of the sample object in three-dimensional space from the three-dimensional information of the sample object. The second mapping submodule is configured to map the sample texture information of each first position to each corresponding second position in two-dimensional space based on a position mapping relationship between three-dimensional space and two-dimensional space, thereby generating a two-dimensional image.

[0178] According to an embodiment of the present disclosure, the 2D image processing module includes a first extraction submodule and a fusion submodule. The first extraction submodule is configured to extract sample 3D point cloud features and sample texture distribution features from the 2D sample image. The fusion submodule is configured to fuse the sample 3D point cloud features and sample texture distribution features based on an attention mechanism to generate sample 3D structural fusion features.

[0179] According to an embodiment of the present disclosure, the fusion submodule includes: a second attention unit, a second fusion unit and a second processing unit.

[0180] The second attention unit is used to fuse the sample 3D point cloud features and the sample texture distribution features based on the attention mechanism to generate the sample initial fusion features.

[0181] The second fusion unit is used to fuse the sample 3D point cloud features with the sample initial fusion features to generate the sample intermediate fusion features.

[0182] The second processing unit is configured to generate a sample three-dimensional structure fusion feature based on the sample intermediate fusion feature.

[0183] According to an embodiment of the present disclosure, the sample noise is a noise time series sequence that conforms to a Gaussian distribution; the loss calculation module includes: a second extraction submodule, a third extraction submodule and a calculation submodule.

[0184] The second extraction submodule is used to extract the target sample noise at the target moment from the noise time series.

[0185] The third extraction submodule is configured to extract target noise information at a target moment from the noise information in the sample texture image.

[0186] The calculation submodule is used to generate a loss value based on the objective function, the target sample noise and the target noise information.

[0187] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0188] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0189] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described above.

[0190] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method described above.

[0191] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0192] like Figure 9As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. Computing unit 901, ROM 902, and RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to bus 904.

[0193] Various components in the device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0194] The computing unit 901 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the method for texture image generation or training a texture image generation model. For example, in some embodiments, the method for texture image generation or training a texture image generation model can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the method for texture image generation or training a texture image generation model described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute a texture image generation or a texture image generation model training method in any other appropriate manner (eg, by means of firmware).

[0195] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0196] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0197] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0198] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0199] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0200] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0201] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0202] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A texture image generation method, comprising: Extract text features of text description information for target objects; generating a two-dimensional image based on the three-dimensional information of the target object; wherein the two-dimensional image indicates the three-dimensional point cloud information and texture distribution information of the target object; Processing the two-dimensional image to generate three-dimensional structure fusion features; Based on an attention mechanism, the initial noise is fused with the text feature to generate a first intermediate feature; wherein the first intermediate feature indicates appearance description information for the target object; Based on an attention mechanism, the initial noise is fused with the three-dimensional structure fusion feature to generate a second intermediate feature; wherein the second intermediate feature indicates three-dimensional point cloud information and texture distribution information of the target object; fusing the first intermediate feature and the second intermediate feature to generate a third intermediate feature; generating a target texture feature based on the third intermediate feature; and A target texture image is generated based on the target texture features, wherein the target texture image is used to generate a virtual image for the target object.

2. The method according to claim 1, wherein The processing of the two-dimensional image to generate three-dimensional structure fusion features includes: Extracting three-dimensional point cloud features and texture distribution features from the two-dimensional image respectively; and Based on the attention mechanism, the three-dimensional point cloud features and the texture distribution features are fused to generate the three-dimensional structure fusion features.

3. The method according to claim 2, wherein: The method of fusing the three-dimensional point cloud features and the texture distribution features based on the attention mechanism to generate the three-dimensional structure fusion features includes: Based on the attention mechanism, the three-dimensional point cloud features and the texture distribution features are fused to generate initial fused features; Fusing the three-dimensional point cloud features with the initial fusion features to generate intermediate fusion features; and The three-dimensional structure fusion feature is generated based on the intermediate fusion feature.

4. The method according to claim 1, wherein Generating a target texture feature based on the third intermediate feature includes: The initial noise is gradually added to the third intermediate feature at a predetermined time step to generate the target texture feature.

5. The method according to claim 1, wherein The method further comprises: fusing the noise feature of the initial noise with the third intermediate feature to generate a fourth intermediate feature; and The target texture feature is generated based on the fourth intermediate feature.

6. The method according to claim 1, wherein Generating a two-dimensional image based on the three-dimensional information of the target object includes: Acquire texture information of each first position of the target object in the three-dimensional space from the three-dimensional information; and Based on the position mapping relationship between the three-dimensional space and the two-dimensional space, the texture information of each first position is mapped to each second position in the corresponding two-dimensional space to generate the two-dimensional image.

7. A training method for a texture image generation model, comprising: Extracting sample text features of text description information for sample objects; generating a two-dimensional sample image based on the three-dimensional information of the sample object; wherein the two-dimensional sample image indicates the three-dimensional point cloud information and texture distribution image of the sample object; Processing the two-dimensional sample image to generate a three-dimensional structure fusion feature of the sample; Extracting sample texture features of the texture image of the sample object; adding sample noise to the sample texture feature to generate a sample texture feature containing noise; Based on the attention mechanism, the texture features of the sample containing noise are processed to generate potential features of the sample texture; Based on an attention mechanism, the latent features of the sample texture are fused with the sample text features to generate a first sample intermediate feature; wherein the first sample intermediate feature indicates appearance description information for the sample object; Based on an attention mechanism, the latent feature of the sample texture is fused with the three-dimensional structure fusion feature to generate a second sample intermediate feature; wherein the second sample intermediate feature indicates three-dimensional point cloud information and texture distribution information of the sample object; Fusing the first sample intermediate feature and the second sample intermediate feature to generate a third sample intermediate feature; generating the sample texture image and noise information in the sample texture image based on the third sample intermediate feature; Based on the objective function, generating a loss value according to the sample noise and the noise information; and Model parameters of the texture image generation model are adjusted based on the loss value to obtain a trained texture image generation model.

8. The method according to claim 7, wherein: The adding sample noise to the sample texture feature to generate the sample texture feature containing noise includes: Sample noise is sequentially added to the sample texture feature according to a predetermined time step to generate a sample texture feature containing noise at each moment; wherein the sample noise at each moment is Gaussian distributed.

9. The method according to claim 8, wherein The method further comprises: fusing the latent feature of the sample texture and the third sample intermediate feature to generate a fourth sample intermediate feature; and A target sample texture feature is generated based on the fourth sample intermediate feature.

10. The method according to claim 7, wherein: Generating a two-dimensional sample image based on the three-dimensional information of the sample object includes: Acquire sample texture information of each first position of the sample object in the three-dimensional space from the three-dimensional information of the sample object; and Based on the position mapping relationship between the three-dimensional space and the two-dimensional space, the sample texture information of each first position is mapped to each second position in the corresponding two-dimensional space to generate the two-dimensional sample image.

11. The method according to claim 7, wherein: The processing of the two-dimensional sample image to generate a three-dimensional structure fusion feature of the sample includes: Extracting sample three-dimensional point cloud features and sample texture distribution features from the two-dimensional sample image respectively; and Based on the attention mechanism, the sample three-dimensional point cloud features and the sample texture distribution features are fused to generate the sample three-dimensional structure fusion features.

12. The method according to claim 11, wherein The method of fusing the sample 3D point cloud features and the sample texture distribution features based on the attention mechanism to generate the sample 3D structure fusion features includes: Based on the attention mechanism, the sample three-dimensional point cloud features and the sample texture distribution features are fused to generate the sample initial fusion features; Fusing the sample 3D point cloud features with the sample initial fusion features to generate sample intermediate fusion features; and The sample three-dimensional structure fusion feature is generated based on the sample intermediate fusion feature.

13. The method according to claim 7, wherein: The sample noise is a noise time series sequence that conforms to the Gaussian distribution; The generating a loss value based on the objective function and according to the sample noise and the noise information includes: extracting target sample noise at a target moment from the noise time series; Extracting target noise information for the target moment from the noise information in the sample texture image; and Based on the objective function, the loss value is generated according to the target sample noise and the target noise information.

14. A texture image generating device, comprising: A feature extraction module is used to extract text features of text description information of the target object; An information processing module, configured to generate a two-dimensional image based on the three-dimensional information of the target object; wherein the two-dimensional image indicates the three-dimensional point cloud information and texture distribution information of the target object; A feature fusion module, used for processing the two-dimensional image to generate three-dimensional structure fusion features; A text feature fusion submodule, configured to fuse the initial noise with the text feature based on an attention mechanism to generate a first intermediate feature; wherein the first intermediate feature indicates appearance description information for the target object; a structural feature fusion submodule, configured to fuse the initial noise with the three-dimensional structural fusion feature based on an attention mechanism to generate a second intermediate feature; wherein the second intermediate feature indicates three-dimensional point cloud information and texture distribution information of the target object; an intermediate feature fusion submodule, configured to fuse the first intermediate feature and the second intermediate feature to generate a third intermediate feature; a generating submodule, configured to generate a target texture feature based on the third intermediate feature; and An image generation module is used to generate a target texture image based on the target texture features, wherein the target texture image is used to generate a virtual image for the target object.

15. The device according to claim 14, wherein The feature fusion module includes: a feature extraction submodule, configured to extract three-dimensional point cloud features and texture distribution features from the two-dimensional image; and The feature fusion submodule is used to fuse the three-dimensional point cloud features and the texture distribution features based on the attention mechanism to generate the three-dimensional structure fusion features.

16. The device according to claim 15, wherein The feature fusion submodule includes: A first attention unit is configured to fuse the three-dimensional point cloud feature and the texture distribution feature based on an attention mechanism to generate an initial fused feature; A first fusion unit is configured to fuse the three-dimensional point cloud feature with the initial fusion feature to generate an intermediate fusion feature; and The first processing unit is configured to generate the three-dimensional structure fusion feature based on the intermediate fusion feature.

17. The device according to claim 14, wherein The generation submodule includes: A noise adding unit is used to gradually add the initial noise to the third intermediate feature according to a predetermined time step to generate the target texture feature.

18. The device according to claim 14, wherein The device further comprises: a noise fusion submodule, configured to fuse the noise feature of the initial noise with the third intermediate feature to generate a fourth intermediate feature; and The texture feature generation submodule is configured to generate the target texture feature based on the fourth intermediate feature.

19. The device according to claim 14, wherein The information processing module includes: a first acquisition submodule, configured to acquire texture information of each first position of the target object in the three-dimensional space from the three-dimensional information; and The second mapping submodule is configured to map the texture information of each first position to each second position in the corresponding two-dimensional space based on a position mapping relationship between the three-dimensional space and the two-dimensional space, so as to generate the two-dimensional image.

20. A training device for a texture image generation model, the training device comprising: A first extraction module is used to extract sample text features of text description information of a sample object; a two-dimensional image generation module, configured to generate a two-dimensional sample image based on the three-dimensional information of the sample object; wherein the two-dimensional sample image indicates the three-dimensional point cloud information and texture distribution image of the sample object; A two-dimensional image processing module, used to process the two-dimensional sample image to generate a three-dimensional structure fusion feature of the sample; A second extraction module is used to extract sample texture features of the texture image of the sample object; A noise adding module, configured to add sample noise to the sample texture feature to generate a sample texture feature containing noise; A first attention submodule is configured to process the texture features of the sample containing noise based on an attention mechanism to generate latent features of the sample texture; A second attention submodule is configured to fuse the latent features of the sample texture with the sample text features based on an attention mechanism to generate a first sample intermediate feature; wherein the first sample intermediate feature indicates appearance description information for the sample object; a third attention submodule, configured to fuse the latent features of the sample texture with the three-dimensional structure fusion features based on an attention mechanism to generate a second sample intermediate feature; wherein the second sample intermediate feature indicates three-dimensional point cloud information and texture distribution information for the sample object; a fusion submodule, configured to fuse the first sample intermediate feature and the second sample intermediate feature to generate a third sample intermediate feature; and a processing submodule, configured to generate the sample texture image and noise information in the sample texture image based on the third sample intermediate feature; a loss calculation module, configured to generate a loss value based on an objective function, the sample noise and the noise information; and An adjustment module is used to adjust the model parameters of the texture image generation model based on the loss value to obtain a trained texture image generation model.

21. The training device according to claim 20, wherein: The noise adding module includes: The noise adding submodule is used to sequentially add sample noise to the sample texture features according to a predetermined time step to generate a sample texture feature containing noise at each moment; wherein the sample noise at each moment is Gaussian distributed.

22. The training device according to claim 20, further comprising: a fusion module, configured to fuse the latent features of the sample texture with the third sample intermediate features to generate a fourth sample intermediate feature; as well as A processing module is used to generate a target sample texture feature based on the fourth sample intermediate feature.

23. The training device according to claim 20, wherein: The two-dimensional image generation module includes: a second acquisition submodule, configured to acquire sample texture information of each first position of the sample object in the three-dimensional space from the three-dimensional information of the sample object; and The second mapping submodule is configured to map the sample texture information at each first position to each second position in the corresponding two-dimensional space based on a position mapping relationship between the three-dimensional space and the two-dimensional space, so as to generate the two-dimensional sample image.

24. The training device according to claim 20, wherein: The two-dimensional image processing module includes: A first extraction submodule is configured to extract sample three-dimensional point cloud features and sample texture distribution features from the two-dimensional sample image; and The fusion submodule is used to fuse the sample three-dimensional point cloud features and the sample texture distribution features based on the attention mechanism to generate the sample three-dimensional structure fusion features.

25. The training device according to claim 24, wherein The fusion submodule includes: A second attention unit is configured to fuse the sample three-dimensional point cloud features and the sample texture distribution features based on an attention mechanism to generate an initial fused feature of the sample; A second fusion unit is configured to fuse the sample three-dimensional point cloud feature with the sample initial fusion feature to generate a sample intermediate fusion feature; and The second processing unit is configured to generate the sample three-dimensional structure fusion feature based on the sample intermediate fusion feature.

26. The training device according to claim 20, wherein: The sample noise is a noise time series sequence that conforms to the Gaussian distribution; The loss calculation module includes: A second extraction submodule is used to extract the target sample noise at the target moment from the noise time series; A third extraction submodule is configured to extract target noise information for the target moment from the noise information in the sample texture image; and A calculation submodule is configured to generate the loss value based on the objective function, the target sample noise, and the target noise information.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.

29. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images