Clothing texture map generation method and device, equipment and storage medium

Abstract

The disclosure provides a clothing texture map generation method and device, equipment and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, deep learning, large model, etc., and can be applied to virtual reality, digital people and other scenes. The specific implementation scheme is: obtaining clothing features based on a clothing reference image; obtaining texture features according to a coordinate mapping image and a texture mask image associated with the clothing reference image; wherein the coordinate mapping image includes coordinates corresponding to the texture map of the clothing model in the clothing model, which is associated with the clothing reference image; the texture mask image includes an image generated based on the transparent channel in the texture map of the clothing model; and generating a clothing texture map based on the clothing features and the texture features.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to computer vision, deep learning, large models and other technical fields, and can be applied to scenarios such as virtual reality and digital humans. Background Art

[0002] With the development of virtual reality and augmented reality technologies, digital characters, such as digital humans, are finding increasing application in a wide range of scenarios. For example, digital humans can provide an immersive user experience on social media and virtual interactive platforms. High-quality 3D models and realistic textures are crucial for the authenticity and interactivity of digital characters. Summary of the Invention

[0003] The present disclosure provides a method, apparatus, device and storage medium for generating clothing texture maps.

[0004] According to one aspect of the present disclosure, a method for generating a clothing texture map is provided, comprising:

[0005] Obtain clothing features based on clothing reference images;

[0006] Obtaining texture features based on a coordinate map and a texture mask map associated with the clothing reference image; wherein the coordinate map includes coordinates corresponding to the texture map unfolded by the clothing model associated with the clothing reference image in the clothing model; and the texture mask map includes an image generated based on a transparent channel in the texture map unfolded by the clothing model;

[0007] A clothing texture map is generated based on the clothing feature and the texture feature.

[0008] According to another aspect of the present disclosure, a method for training a clothing texture map generation model is provided, comprising:

[0009] Obtain clothing features based on clothing reference images in training samples;

[0010] Obtaining texture features based on the coordinate map and texture mask map in the training sample and input noise; wherein the coordinate map includes coordinates corresponding to the texture map unfolded by the clothing model associated with the clothing reference image in the clothing model; and the texture mask map includes an image generated based on a transparent channel in the texture map unfolded by the clothing model;

[0011] Based on the clothing features and the texture features, the clothing texture map generation model is optimized.

[0012] According to another aspect of the present disclosure, a device for generating a clothing texture map is provided, comprising:

[0013] A clothing feature acquisition module is used to obtain clothing features based on a clothing reference image;

[0014] a texture feature acquisition module configured to obtain a texture feature according to a coordinate mapping image and a texture mask image associated with the clothing reference image; the coordinate mapping image includes coordinates corresponding to a texture map unfolded from a clothing model associated with the clothing reference image in the clothing model; and the texture mask image includes an image generated based on a transparent channel in the texture map unfolded from the clothing model;

[0015] a clothing texture map generation module configured to generate a clothing texture map based on the clothing feature and the texture feature.

[0016] According to another aspect of the present disclosure, a clothing texture map generation model training apparatus is provided, comprising:

[0017] a clothing feature acquisition module configured to obtain a clothing feature based on a clothing reference image in a training sample;

[0018] a texture feature acquisition module configured to obtain a texture feature according to a coordinate mapping image and a texture mask image in the training sample and input noise; the coordinate mapping image includes coordinates corresponding to a texture map unfolded from a clothing model associated with the clothing reference image in the clothing model; and the texture mask image includes an image generated based on a transparent channel in the texture map unfolded from the clothing model;

[0019] an optimization module configured to optimize the clothing texture map generation model based on the clothing feature and the texture feature.

[0020] According to another aspect of the present disclosure, a clothing generation method for a digital character is provided, comprising:

[0021] generating a clothing texture map of the digital character by the clothing texture map generation method of any one of the embodiments of the present disclosure;

[0022] generating clothing of the digital character by using the clothing texture map of the digital character.

[0023] According to another aspect of the present disclosure, a digital character is provided, comprising:

[0024] digital clothing generated by a clothing texture map;

[0025] the clothing texture map is generated by any one of the above.

[0026] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0027] at least one processor; and

[0028] a memory communicatively connected to the at least one processor; wherein

[0029] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any method in the embodiments of the present disclosure.

[0030] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0031] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements any one of the methods according to the embodiments of the present disclosure.

[0032] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0034] Figure 1 1 is a flow chart of a method for generating a clothing texture map according to an embodiment of the present disclosure;

[0035] Figure 2 is a flow chart of a method for generating a clothing texture map according to another embodiment of the present disclosure;

[0036] Figure 3 is a flow chart of a method for generating a clothing texture map according to another embodiment of the present disclosure;

[0037] Figure 4 1 is a flow chart of a method for training a clothing texture map generation model according to an embodiment of the present disclosure;

[0038] Figure 5 is a flow chart of a clothing texture map generation model training method according to another embodiment of the present disclosure;

[0039] Figure 6 is a flow chart of a clothing texture map generation model training method according to another embodiment of the present disclosure;

[0040] Figure 7 is a flow chart of a clothing texture map generation model training method according to another embodiment of the present disclosure;

[0041] Figures 8a to 8d is a schematic diagram of data samples according to an embodiment of the present disclosure;

[0042] Figure 9 is a schematic diagram of the network structure of an embodiment of the present disclosure;

[0043] Figure 10 is a schematic diagram of texture generation results according to an embodiment of the present disclosure;

[0044] Figure 11 1 is a structural diagram of a device for generating a clothing texture map according to an embodiment of the present disclosure;

[0045] Figure 12 is a structural diagram of a clothing texture map generating device according to another embodiment of the present disclosure;

[0046] Figure 13 1 is a structural diagram of a clothing texture map generation model training device according to an embodiment of the present disclosure;

[0047] Figure 14 is a structural diagram of a clothing texture map generation model training device according to another embodiment of the present disclosure;

[0048] Figure 15 A method for generating clothing for a digital character according to another embodiment of the present disclosure is provided;

[0049] Figure 16 It is a block diagram of an electronic device used to implement the clothing texture map generation method and clothing texture map generation model training method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0050] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0051] Figure 1 FIG. 1 is a flow chart of a method 100 for generating a clothing texture map according to an embodiment of the present disclosure, the method comprising:

[0052] S110: Obtain clothing features based on the clothing reference image.

[0053] S120: Obtaining texture features based on a coordinate map and a texture mask map associated with the clothing reference image. The coordinate map includes coordinates corresponding to the texture map unfolded on the clothing model associated with the clothing reference image, and the texture mask map includes an image generated based on a transparent channel in the texture map unfolded on the clothing model.

[0054] S130: Generate a clothing texture map based on the clothing feature and the texture feature.

[0055] In the embodiments of the present disclosure, clothing may include, but is not limited to, clothing, shoes, hats, socks, gloves, scarves, ties, accessories, and the like. A clothing reference image may also be referred to as a clothing rendering, and may include the style, color, pattern, and style of clothing that the digital character's clothing is expected to present. A clothing reference image may be a two-dimensional image. The style, color, pattern, and style of different clothing may be different. For example, clothing styles may include clothes, pants, shoes, and the like; they may also include long sleeves, short sleeves, trousers, shorts, long skirts, short skirts, coats, and the like. For another example, the style of clothing may be described by different design elements and structures, such as the style of long sleeves may include round necks, V-necks, flared sleeves, regular sleeves, and the like. Based on the clothing reference image, clothing features related to one or more clothing styles, colors, patterns, and styles may be obtained.

[0056] In an embodiment of the present disclosure, an associated coordinate map and texture mask map are obtained based on the matching of a clothing reference map. A number of three-dimensional clothing models can be pre-saved. Based on the three-dimensional clothing model, a clothing rendering map, a UV texture map (where U can be understood as a horizontal coordinate and V can be understood as a vertical coordinate), a coordinate map and a texture mask map can be generated. The coordinate map of the three-dimensional clothing model can include the coordinates corresponding to the UV texture map unfolded by the three-dimensional clothing model in the three-dimensional clothing model. The texture mask map of the three-dimensional clothing model includes an image generated based on the transparent channel in the texture map unfolded by the three-dimensional clothing model. For example, the texture mask map can be a binary image, in which the transparent channel value of the pixels in the binary image is 0 or 1, wherein a transparent channel value of a pixel is 0, representing that the pixel is completely transparent, and a transparent channel value of a pixel is 255, representing that the pixel is completely opaque. If the transparent channel value of a pixel in the UV texture map is greater than or equal to 1, the transparent channel value of the pixel in the UV texture map can be set to 1, and the other pixels can be set to 0, thereby obtaining the binary image.

[0057] In one example, by comparing the similarity between a clothing reference image and the clothing renderings of various 3D clothing models, a 3D clothing model matching the clothing reference image can be obtained. The matching 3D clothing model is then expanded to obtain a UV texture map, and based on the UV texture map, a coordinate map and texture mask map associated with the clothing reference image are obtained.

[0058] In another example, several clothing renderings, coordinate mappings, and texture mask maps of 3D clothing models are pre-saved. If a clothing reference image has a high degree of similarity to a clothing rendering of a 3D clothing model, the coordinate mapping and texture mask map of the 3D clothing model can be used as the coordinate mapping and texture mask map associated with the clothing reference image.

[0059] In the embodiment of the present disclosure, after the texture features are obtained based on the coordinate mapping image and the texture mask image, a clothing texture map corresponding to the clothing reference image can be generated based on the clothing features and the texture features.

[0060] According to the embodiments of the present disclosure, clothing features and texture features can be obtained based on the clothing reference map, coordinate mapping map and texture mask map, and then clothing texture maps reflecting the clothing features and texture features can be quickly generated, which can reduce the time and cost of clothing texture map production and improve the flexibility and diversity of clothing texture design.

[0061] Figure 2 2 is a flow chart of a method 200 for generating a clothing texture map according to another embodiment of the present disclosure. This method 200 can be used to implement steps S110 and S120 of the method 100 for generating a clothing texture map. In one embodiment, the method 200 includes: obtaining clothing features based on a clothing reference image; and further includes: S210, inputting the clothing reference image into a first network of a clothing texture map generation model to obtain the clothing features.

[0062] In one embodiment, method 200 also includes: obtaining texture features based on a coordinate mapping map and a texture mask map associated with the clothing reference image, and further includes: S220, inputting input noise, the coordinate mapping map and the texture mask map into a second network of a clothing texture map generation model to obtain the texture features.

[0063] In an embodiment of the present disclosure, the clothing texture map generation model may include a first network and a second network. The first network and the second network may use the same or similar neural network architecture (or model framework). For example, the first network and the second network may adopt a U-shaped network (UNet) architecture. UNet is a convolutional neural network (CNN) architecture that uses a U-shaped network structure and jump connections. The specific training method of the clothing texture map generation model can be found in the description of the relevant embodiments below.

[0064] An example of a UNet architecture primarily consists of an encoder and a decoder. A bridge may be included between the two, connecting the corresponding layers of the encoder and decoder to help restore lost spatial information in the decoder. The encoder (also known as the contracting path) consists of a series of convolutional and pooling layers for feature extraction and downsampling. The encoder may include multiple convolutional blocks. Each convolutional block may include a convolutional layer, batch normalization, and an activation function, helping to learn low-level features of the image. Each convolutional block may also be followed by a pooling layer. The decoder (also known as the dilating path) consists of a series of upsampling and convolutional layers for feature fusion and upsampling. The decoder may include multiple deconvolutional blocks, each of which may include a deconvolutional layer, batch normalization, and an activation function, helping to learn high-level semantic information of the image. After each deconvolutional block, an upsampling layer may be added, corresponding to the pooling layer in the encoder. The output of the upsampling layer is concatenated with the feature map of the corresponding layer in the encoder to achieve feature fusion. Each layer in the decoder is connected to the corresponding layer in the encoder through skip connections, which helps to combine the semantic information of low-level and high-level layers.

[0065] In the disclosed embodiment, inputting a clothing rendering into the first network of the clothing texture map generation model can generate clothing features for the clothing reference image. In the disclosed embodiment, input noise refers to the random noise generated during image processing by the clothing texture map generation model, and the input noise used for different clothing reference images may vary. Inputting the input noise, along with the coordinate map and texture mask associated with the clothing reference image, into the second network of the clothing texture map generation model can output texture features for the clothing reference image. Clothing features and texture features can be understood as intermediate features of the clothing texture map generation model.

[0066] According to the embodiment of the present disclosure, the clothing features and texture features of the clothing reference image are combined and processed through the first network and the second network to generate a clothing texture map that is more consistent with the characteristics of the clothing reference image. This can reduce the production time of the clothing texture map and improve the generation efficiency of the clothing texture map.

[0067] Figure 3 3 is a flow chart of a method 300 for generating a clothing texture map according to another embodiment of the present disclosure. The method 300 can be used to implement step S130 in the method 100 for generating a clothing texture map. In one embodiment, the method 300 includes: generating a clothing texture map based on the clothing features and the texture features, and further includes:

[0068] S310, splicing the clothing feature and the texture feature to obtain a splicing feature;

[0069] S320: Input the splicing features into the cross attention network of the second network to generate the clothing texture map.

[0070] In the disclosed embodiment, cross-attention can dynamically adjust the attention of each modality (or sequence) to other modalities (or sequences) by calculating the similarity between the query vector of one modality (or sequence) and the key vector and value vector of another modality (or sequence), thereby achieving comprehensive utilization of information. Clothing features and texture features can be spliced ​​and input into the cross-attention network of the second network.

[0071] In the embodiment of the present disclosure, after the clothing features and texture features are fused through the cross-attention network of the second network, a clothing texture map that can reflect the clothing features and texture features of the clothing reference image can be further generated.

[0072] According to the embodiment of the present disclosure, the cross-attention network of the second network can realize the comprehensive utilization of clothing features and texture features, and generate a clothing texture map that is more consistent with the characteristics of the clothing reference image.

[0073] In one embodiment, the method further comprises:

[0074] Matching an associated clothing model according to the clothing reference image;

[0075] Based on the texture map unfolded by the clothing model, reverse query is performed on the coordinates of each point of the clothing model to obtain the coordinate mapping associated with the clothing reference image;

[0076] Based on the transparent channel of the texture map unfolded by the clothing model, the texture mask map associated with the clothing reference map is obtained.

[0077] In an embodiment of the present disclosure, a number of three-dimensional clothing models can be pre-stored in a database. Based on a clothing reference image that can be input, a clothing model that matches the features of the clothing reference image can be searched from the database. In some examples, by comparing the similarity between the clothing reference image and the main perspective rendering of the three-dimensional clothing model, a three-dimensional clothing model that matches the clothing reference image can be obtained. For example, if the clothing reference image includes a white short-sleeved dress, a clothing model of the short-sleeved dress can be searched from the database. For another example, if the clothing reference image includes a short-sleeved round-neck T-shirt with a colorful pattern, a clothing model of the short-sleeved round-neck T-shirt can be searched from the database.

[0078] In the embodiment of the present disclosure, the three-dimensional model is composed of the vertices of all shapes in the three-dimensional space. Each vertex has unique coordinates in the three-dimensional space. By reversely querying the coordinates of each vertex in the matching three-dimensional model, a coordinate mapping diagram that can represent the shape of this three-dimensional model can be obtained.

[0079] In an embodiment of the present disclosure, an automatic UV unwrapping tool in 3D modeling software can be used to map points on the surface of a 3D model to a two-dimensional plane (e.g., UV space). The 3D model is then UV-unwrapped to obtain a two-dimensional representation of a UV texture map based on the 3D model, and then a coordinate map and texture mask map of the UV map are obtained. During the UV unwrapping process, each vertex on the 3D model surface is assigned a UV coordinate. Based on these UV coordinates, rasterization technology can be used to query the 3D coordinates of each point on the 3D model. These 3D coordinates are stored to obtain a coordinate map. Rasterization technology converts vertex data into fragments, converting an image into an image composed of rasters, with each element corresponding to a pixel in the frame buffer. The texture mask map can be a binary image generated based on a transparency channel. The transparency channel is also called an alpha channel. The transparency channel value (also known as the transparency channel value, transparency value, etc.) of each pixel ranges from 0 to 255. A transparency channel value of 0 for a pixel represents complete transparency, and a transparency channel value of 255 for a pixel represents complete opacity. The area where the value of the transparent channel in the UV texture map of the unfolded three-dimensional model is greater than a preset value, such as 1, can be regarded as the valid area of ​​the texture mask map.

[0080] According to an embodiment of the present disclosure, a coordinate mapping map and a texture mask map associated with a clothing reference image can be generated by matching a three-dimensional clothing model with a clothing reference image, and a clothing texture map that is more consistent with the characteristics of the clothing reference image can be generated.

[0081] Figure 4 FIG4 is a flow chart of a method 400 for training a clothing texture map generation model according to an embodiment of the present disclosure, the method comprising:

[0082] S410: Obtain clothing features based on clothing reference images in the training samples.

[0083] S420: Obtain texture features based on the coordinate map and texture mask map in the training sample and the input noise. The coordinate map includes coordinates corresponding to the texture map unfolded by the clothing model associated with the clothing reference image in the clothing model; and the texture mask map includes an image generated based on a transparent channel in the texture map unfolded by the clothing model.

[0084] S430: Optimizing the clothing texture map generation model based on the clothing features and the texture features.

[0085] In an embodiment of the present disclosure, a training sample may include a clothing reference image, a coordinate mapping image, and a texture mask image. The input noise may be part of the training sample. The input noise may also not be in the training sample, and may be noise randomly generated when using the training sample. After obtaining clothing features based on the clothing reference image and texture features based on the coordinate mapping image and the texture mask image, real noise and predicted noise may be obtained based on the clothing features and texture features. The loss value of the loss function of the clothing texture map generation model is calculated using the real noise and the predicted noise, and based on the loss value, it can be determined whether the clothing texture map generation model is optimized. For example, if the loss value does not converge, the loss function of the model is iterated by adjusting the parameters of the clothing texture map generation model until the clothing texture map generation model can achieve the expected training effect, such as the loss value converges.

[0086] According to the embodiments of the present disclosure, the training quality of the clothing texture map generation model can be improved by optimizing the training of the clothing texture map generation model using real noise and predicted noise.

[0087] Figure 5 is a flow chart of a clothing texture map generation model training method 500 according to another embodiment of the present disclosure. The method 500 can be used to implement steps S410 and S420 in the clothing texture map generation model training method 400. In one embodiment, the method 500 includes: in one embodiment, obtaining clothing features based on clothing reference images in training samples, further including: S510, inputting the clothing reference image into the first network of the clothing texture map generation model to obtain the clothing features;

[0088] In one embodiment, method 500 includes: obtaining texture features based on the input noise, the coordinate mapping map and the texture mask map in the training sample, and further includes: S520, inputting the input noise, the coordinate mapping map and the texture mask map into the second network of the clothing texture map generation model to obtain the texture features.

[0089] In the embodiment of the present disclosure, the method of obtaining clothing features and texture features can refer to the relevant description of the clothing texture map generation method above, and will not be repeated here.

[0090] According to the embodiment of the present disclosure, the clothing features and texture features of the clothing reference image in the sample are combined and processed through the first network and the second network, so that the trained model can generate a clothing texture map that is more consistent with the characteristics of the clothing reference image, which can reduce the production time of the clothing texture map and improve the generation efficiency of the clothing texture map.

[0091] Figure 66 is a flow chart of a clothing texture map generation model training method 600 according to another embodiment of the present disclosure. The method 600 can be used to implement step S430 in the clothing texture map generation model training method 400. In one embodiment, the method 600 includes: optimizing the clothing texture map generation model based on the clothing features and the texture features, and further includes:

[0092] S610, splicing the clothing feature and the texture feature to obtain a splicing feature;

[0093] S620, inputting the splicing features into the cross attention network of the second network;

[0094] S630, iteratively processing the splicing feature through the second network;

[0095] S640: Optimize the clothing texture map generation model based on the first noise added and the second noise removed in the iterative process.

[0096] In an embodiment of the present disclosure, the splicing features are input into the cross-attention network of the second network, and iterative processing is performed based on the splicing features through the cross-attention mechanism of the cross-attention network. In the iterative processing, the noise added to the splicing features can be considered as real noise, and the noise removed from the final noisy image can be considered as predicted noise. Based on the added noise and the removed noise, the loss value of the loss function of the clothing texture map generation model is calculated, and based on the loss value, it can be determined whether the clothing texture map generation model can be optimized. If the loss value does not converge, the loss function of the model is iterated by adjusting the parameters of the clothing texture map generation model until the loss value of the clothing texture map generation model converges.

[0097] According to an embodiment of the present disclosure, the stitching features are iteratively processed through a clothing texture map generation model to obtain a second noise, which can improve the training quality of the clothing texture map generation model.

[0098] In one embodiment, iteratively processing the splicing features by the second network includes:

[0099] The stitching feature is subjected to multi-stage iterative processing by the second network to obtain added first noise and removed second noise; wherein the multi-stage iterative processing includes a plurality of consecutive forward diffusion steps and a plurality of consecutive backward diffusion steps; the plurality of consecutive forward diffusion steps are used to iteratively add noise to the stitching feature to obtain a noisy target image, and the first noise includes the noise added in the plurality of consecutive forward diffusion steps; the plurality of consecutive backward diffusion steps are used to iteratively remove noise from the target image to obtain a restored texture map, and the second noise includes the noise removed in the plurality of consecutive backward diffusion steps.

[0100] In an embodiment of the present disclosure, a Wensheng diagram tool, such as an AI painting generation tool based on a diffusion model, can be used to process the splicing features to obtain the added first noise and the removed second noise. The splicing features are input into the second network for multi-stage iterative processing. For example, the multi-stage iterative processing can include multiple consecutive forward diffusion steps (or forward denoising steps) and multiple consecutive reverse diffusion steps (or reverse denoising steps).

[0101] For example, in a forward diffusion step, noise is first added to the splicing features to obtain a first noisy image. In the next forward diffusion step, noise is added to the first noisy image to obtain a second noisy image. This process is repeated several times until a noisy target image is obtained. During the multiple forward diffusion steps, the first noise can be obtained by summing the noise added in each forward diffusion step, or by calculating the difference between the target image and the splicing features. For another example, in a backward diffusion step, noise is first removed from the target image to obtain a first denoised image. In the next backward diffusion step, noise is removed from the first denoised image to obtain a second denoised image. This process is repeated several times until a restored texture map is obtained. During the multiple backward diffusion steps, the second noise can be obtained by summing the noise removed in each backward diffusion step, or by calculating the difference between the restored texture map and the target image. These forward diffusion steps can be used to introduce noise, such as random Gaussian noise, into the model-generated image, gradually converting the input random Gaussian noise into a noisy target image. The random Gaussian noise added is typically different from the input noise of the clothing texture generation model. The back-diffusion step described above can be used to remove noise from the image. For example, a specific sampler can be used to iteratively denoise the target image to ultimately obtain a UV texture map similar to the one in the database.

[0102] According to the embodiments of the present disclosure, the efficiency of model training can be improved by optimizing the model by adding and removing noise in multi-stage iterations.

[0103] Figure 7 is a flow chart of a clothing texture map generation model training method 700 according to another embodiment of the present disclosure. This method 700 can be used to implement step S440 in the clothing texture map generation model training method 400. In one embodiment, the method 700 includes optimizing the clothing texture map generation model based on the first noise added and the second noise removed in the iterative process, and further includes:

[0104] S710, calculating a loss function based on the first noise and the second noise;

[0105] S720, adjusting parameters of the second network based on the loss function until the loss function converges.

[0106] In the embodiments of the present disclosure, the added noise and removed noise in the iteration process are substituted into the loss function of the clothing texture map generation model to calculate the loss value. For example, the loss function can use the form of Mean Squared Error (MSE) to measure the difference between the predicted noise of the model and the real noise. The real noise can include the added noise in the forward diffusion (i.e., the positive diffusion), such as the input noise. The predicted noise can include the noise used for denoising calculated or predicted by the sampler in the reverse diffusion iteration.

[0107] An example of a loss function calculation formula is shown in the following formula:

[0108]

[0109] where t represents a given time step, x represents an image with noise received by the model, ∈ pred represents the prediction of the initial noise (i.e., the removed noise), ∈ true represents the real noise (i.e., the added noise), and the expectation E can be performed on all training samples, noise samples, and time steps. The parameters of the model are adjusted to minimize the loss function to complete the convergence of the loss function Loss.

[0110] According to the embodiments of the present disclosure, by calculating the loss function of the clothing texture map generation model to adjust the parameters of the clothing texture map generation model, the training efficiency of the clothing texture map generation model can be improved, and the trained clothing texture map generation model can generate clothing texture maps that meet the characteristics of the input image, such as clothing reference images.

[0111] The embodiments of the present disclosure provide a texture map generation scheme, which includes:

[0112] I. Dataset

[0113] According to common clothing types, a batch of three-dimensional (3D) models can be generated in advance, such as T-shirts, dresses, hooded shirts, shorts, and skirts. Each model can have a fixed UV expansion and multiple different high-quality texture maps. Model UV expansion is the process of flattening the surface of a 3D model onto a 2D plane in order to apply texture. The texture map can be a color image with a transparent channel.

[0114] Collect the required data. For example, as shown in Figures 8a to 8d Each set of data samples can include the following four types of images, and the resolution can be 512*512.

[0115] wherein, Figure 8a is the model front view render: a color image with transparency channel, rendered based on the 3D model at a fixed camera view.

[0116] Figure 8b is the UV texture map: a color image with transparency channel, a pre-designed texture.

[0117] Figure 8c is the coordinate map: a 2D image with 3D coordinates of each pixel on the 3D model, the coordinate map is generated by querying the 3D coordinates of each pixel on the 3D model from the UV map using rasterization, and storing these 3D coordinates as a separate image.

[0118] Figure 8d is the texture mask: a binary image, corresponding to the transparency channel of the UV texture map, for example, the transparency channel value of a pixel in the texture mask is 1 or 0, and the area with value 1 can correspond to the opaque area of the texture map.

[0119] II. Network model

[0120] The model structure is shown in Figure 9 , the input is the clothing reference image, random noise, coordinate map and texture mask, and the output is the UV texture map.

[0121] An example of a model network architecture includes a two-way UNet structure:

[0122] (1) UVNet, using UNet as the model framework, based on the pre-trained weights of SDXL as the base model, combining random noise, coordinate map and texture mask Figure 3 input into a UNet with a predetermined number of channels, for example, 9 channels, for training. Random noise is the standard input of the diffusion paradigm, texture mask can provide prior shape and contour for texture generation, and coordinate map strengthens the mapping relationship between 3D position semantic prior of the model and 2D texture unfolding.

[0123] (2) ClothNet, used to encode clothing features, the structure can replicate the main UNet, for example, based on the pre-trained parameters of large stable diffusion (Stable Diffusion XL, SDXL), and adjust (fine-tune) using the diffusion paradigm on a large amount of clothing data.

[0124] Loss function: Stable Diffusion is an image generation method based on a diffusion model, which gradually converts noise into an image through an iterative process. This process can be divided into multiple stages, each of which includes a forward diffusion step and a backward diffusion step. The forward diffusion step gradually introduces noise into the image, while the backward diffusion step gradually removes the noise. Moreover, the predicted noise can be obtained based on the noise removed by the backward diffusion step. When training the UNet network, the main role of the loss function is to guide the model to accurately restore the image during the inverse denoising process. For example, a loss function can be the mean square error (MSE), which is used to measure the difference between the noise predicted by the model and the actual noise.

[0125] (3) The attention mechanism is used to promote the interaction between the original UNet clothing features and texture features, and finally generate a complete texture map.

[0126] Cross attention interaction: The ClothNet network weights are fixed, the clothing features extracted by ClothNet and the texture features extracted by UVNet are connected, and then input into UVNet through the cross attention mechanism. The UVNet weights are trained and updated according to the loss function.

[0127] Some experimental results such as Figure 10 As shown. Input a short-sleeved shirt reference image, random noise, coordinate mapping map, and texture mask map into the network to output the unfolded texture map of the short-sleeved shirt. Input a long-sleeved sweater reference image, random noise, coordinate mapping map, and texture mask map into the network to output the unfolded texture map of the long-sleeved sweater top. Input a short-sleeved cheongsam reference image, random noise, coordinate mapping map, and texture mask map into the network to output the unfolded texture map of the short-sleeved cheongsam.

[0128] Figure 11 FIG. 1 is a schematic structural diagram of a clothing texture map generating device 1100 according to an embodiment of the present disclosure. The device 1100 may include:

[0129] A clothing feature acquisition module 1110 is configured to obtain clothing features based on a clothing reference image;

[0130] The texture feature acquisition module 1120 is configured to obtain texture features based on a coordinate map and a texture mask map associated with the clothing reference image; wherein the coordinate map includes coordinates corresponding to the texture map unfolded on the clothing model associated with the clothing reference image; and the texture mask map includes an image generated based on a transparent channel in the texture map unfolded on the clothing model.

[0131] The clothing texture generation module 1130 is configured to generate a clothing texture map based on the clothing features and the texture features.

[0132] Figure 12 1 is a schematic structural diagram of a clothing texture map generation device 1200 according to another embodiment of the present disclosure. The device 1200 includes: a clothing feature acquisition module 1210, a texture feature acquisition module 1220, and a clothing map generation module 1230. The functions of these modules can be referred to as the functions of the modules of the clothing texture map generation device in the above embodiment. In one embodiment, the clothing feature acquisition module 1210 is used to input the clothing reference image into the first network of the clothing texture map generation model to obtain the clothing features; the texture feature acquisition module 1220 is used to input the input noise, the coordinate map, and the texture mask map into the second network of the clothing texture map generation model to obtain the texture features.

[0133] In one embodiment, the clothing map generation module 1230 includes:

[0134] A splicing submodule 1231 is used to splice the clothing feature and the texture feature to obtain a splicing feature;

[0135] The generation submodule 1232 is used to input the splicing features into the cross attention network of the second network to generate the clothing texture map.

[0136] In one embodiment, it further includes:

[0137] a matching module 1240 for matching an associated clothing model according to the clothing reference image;

[0138] A coordinate mapping module 1250 is configured to perform a reverse query on the coordinates of each point of the clothing model based on the texture map developed by the clothing model to obtain the coordinate mapping associated with the clothing reference image;

[0139] The texture mask module 1260 is configured to obtain the texture mask image associated with the clothing reference image based on the transparent channel of the texture map unfolded by the clothing model.

[0140] Figure 13 FIG1 is a schematic structural diagram of a clothing texture map generation model training device 1300 according to an embodiment of the present disclosure. The device 1300 may include:

[0141] A clothing feature acquisition module 1310 is configured to obtain clothing features based on clothing reference images in training samples;

[0142] The texture feature acquisition module 1320 is configured to obtain texture features based on the coordinate map and texture mask map in the training sample and the input noise; wherein the coordinate map includes coordinates corresponding to the texture map unfolded by the clothing model associated with the clothing reference image in the clothing model; and the texture mask map includes an image generated based on a transparent channel in the texture map unfolded by the clothing model.

[0143] The optimization module 1330 is used to optimize the clothing texture map generation model based on the clothing features and the texture features.

[0144] Figure 14 1 is a schematic structural diagram of a clothing texture map generation model training device 1400 according to another embodiment of the present disclosure. The device 1400 includes: a clothing feature acquisition module 1410, a texture feature acquisition module 1420, and an optimization module 1430. The functions of these modules can be referred to the functions of the modules in the clothing texture map generation model training device of the above embodiment. In one embodiment, the clothing feature acquisition module 1410 is used to input the clothing reference image into the first network of the clothing texture map generation model to obtain the clothing features; the texture feature acquisition module 1420 is used to input the input noise, the coordinate map, and the texture mask map into the second network of the clothing texture map generation model to obtain the texture features.

[0145] In one embodiment, the optimization module 1430 includes:

[0146] A splicing submodule 1431 is used to splice the clothing feature and the texture feature to obtain a splicing feature;

[0147] An input submodule 1432 is used to input the concatenated features into the cross attention network of the second network;

[0148] An iterator module 1433, configured to iteratively process the splicing features through the second network;

[0149] The optimization submodule 1434 is configured to optimize the clothing texture map generation model based on the first noise added and the second noise removed in the iterative process.

[0150] In one embodiment, the iterative submodule 1433 is used to perform multi-stage iterative processing on the splicing feature through the second network to obtain added first noise and removed second noise; wherein the multi-stage iterative processing includes multiple consecutive forward diffusion steps and multiple consecutive backward diffusion steps; the multiple consecutive forward diffusion steps are used to iteratively add noise to the splicing feature to obtain a noisy target image, and the first noise includes the noise added in the multiple consecutive forward diffusion steps; the multiple consecutive backward diffusion steps are used to iteratively remove noise from the target image to obtain a restored texture map, and the second noise includes the noise removed in the multiple consecutive backward diffusion steps.

[0151] In one embodiment, the optimization submodule 1434 is configured to calculate a loss function based on the first noise and the second noise; and adjust parameters of the second network based on the loss function until the loss function converges.

[0152] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0153] Figure 15 A method for generating clothing for a digital character according to another embodiment of the present disclosure includes:

[0154] S1510: Generate a clothing texture map for the digital character using any one of the clothing texture map generation methods in the above embodiments;

[0155] S1520: Generate clothing for the digital character using the clothing texture map of the digital character.

[0156] In the disclosed embodiments, a digital character can be a digital human or another digital object. After training a model using the clothing texture map generation model training method described in the above embodiment, the model can be used to generate clothing texture maps for the digital character. The clothing texture maps can then be used to add clothing to the digital character. During this process, the size, orientation, and position of the digital character's clothing texture maps can be adjusted based on the size and shape of the digital character model to ensure that the clothing texture maps are compatible with the digital character.

[0157] The embodiments of the present disclosure further provide a digital character, comprising: digital clothing generated by clothing texture mapping; generated using any one of the clothing texture mapping generation methods of the above embodiments.

[0158] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0159] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0160] Figure 16 A schematic block diagram of an example electronic device 1600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0161] like Figure 16 As shown, device 1600 includes a computing unit 1601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1602 or a computer program loaded from a storage unit 1608 into a random access memory (RAM) 1603. Various programs and data required for the operation of device 1600 can also be stored in RAM 1603. Computing unit 1601, ROM 1602, and RAM 1603 are connected to each other via a bus 1604. An input / output (I / O) interface 1605 is also connected to bus 1604.

[0162] Various components in device 1600 are connected to I / O interface 1605, including an input unit 1606, such as a keyboard and mouse; an output unit 1607, such as various types of displays and speakers; a storage unit 1608, such as a magnetic disk and optical disk; and a communication unit 1609, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1609 allows device 1600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0163] The computing unit 1601 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 1601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1601 performs the various methods and processes described above, such as the clothing texture map generation method and the clothing texture map generation model training method. For example, in some embodiments, the clothing texture map generation method and the clothing texture map generation model training method can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 1608. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1600 via the ROM 1602 and / or the communication unit 1609. When the computer program is loaded into the RAM 1603 and executed by the computing unit 1601, one or more steps of the clothing texture map generation method and the clothing texture map generation model training method described above can be performed. Alternatively, in other embodiments, the computing unit 1601 may be configured to execute the clothing texture map generation method and the clothing texture map generation model training method in any other appropriate manner (for example, by means of firmware).

[0164] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0165] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0166] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0167] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0168] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0169] The computer system can include clients and servers. This relationship can be. The servers are typically remote from the clients with the interactions between them occurring over a communication network. The relationship between client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The servers can be cloud servers, servers of a distributed system, or servers incorporating blockchain.

[0170] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, without departing from the desired results of the technology disclosed in the present disclosure, and are not limited herein.

[0171] The specific embodiments described above are not intended to be limiting. One of skill in the art will understand that various modifications, combinations, sub-combinations, and alternatives can be made to the specific embodiments described above without departing from the principles of the present disclosure. Any such modifications, equivalents, and alternatives are intended to be included within the scope of the present disclosure.

Claims

1. A method for generating a clothing texture map, comprising: Obtain clothing features based on clothing reference images; Obtaining texture features based on a coordinate map and a texture mask map associated with the clothing reference image, comprising: inputting input noise, the coordinate map, and the texture mask map into a second network of a clothing texture map generation model to obtain the texture features; wherein the coordinate map includes coordinates corresponding to the texture map unfolded by the clothing model associated with the clothing reference image in the clothing model; and the texture mask map includes an image generated based on a transparent channel in the texture map unfolded by the clothing model; Generating a clothing texture map based on the clothing feature and the texture feature, including: Splicing the clothing feature and the texture feature to obtain a splicing feature; The splicing features are input into the cross attention network of the second network to generate the clothing texture map.

2. The method according to claim 1, wherein Obtaining clothing features based on a clothing reference image includes: inputting the clothing reference image into a first network of a clothing texture map generation model to obtain the clothing features.

3. The method according to claim 1, further comprising: Matching an associated clothing model according to the clothing reference image; Based on the texture map unfolded by the clothing model, reverse query is performed on the coordinates of each point of the clothing model to obtain the coordinate mapping map associated with the clothing reference map; The texture mask map associated with the clothing reference map is obtained based on the transparent channel of the texture map unfolded by the clothing model.

4. A clothing texture map generation model training method, comprising: Obtain clothing features based on clothing reference images in training samples; Obtaining texture features based on the coordinate map and texture mask map in the training sample and input noise, including: inputting the input noise, the coordinate map, and the texture mask map into a second network of a clothing texture map generation model to obtain the texture features; wherein the coordinate map includes coordinates corresponding to the texture map unfolded by the clothing model associated with the clothing reference image in the clothing model; and the texture mask map includes an image generated based on a transparent channel in the texture map unfolded by the clothing model; Optimizing the clothing texture map generation model based on the clothing features and the texture features, including: Splicing the clothing feature and the texture feature to obtain a splicing feature; Inputting the concatenated features into the cross-attention network of the second network; Iteratively processing the splicing features through the second network; The clothing texture map generation model is optimized based on the first noise added and the second noise removed in the iterative process.

5. The method according to claim 4, wherein Obtaining clothing features based on clothing reference images in training samples includes: inputting the clothing reference images into a first network of a clothing texture map generation model to obtain the clothing features.

6. The method according to claim 4, wherein: Iteratively processing the splicing features through the second network includes: The stitching features are subjected to multi-stage iterative processing by the second network to obtain added first noise and removed second noise; wherein the multi-stage iterative processing includes a plurality of consecutive forward diffusion steps and a plurality of consecutive backward diffusion steps; the plurality of consecutive forward diffusion steps are used to iteratively add noise to the stitching features to obtain a noisy target image, and the first noise includes the noise added in the plurality of consecutive forward diffusion steps; the plurality of consecutive backward diffusion steps are used to iteratively remove noise from the target image to obtain a restored texture map, and the second noise includes the noise removed in the plurality of consecutive backward diffusion steps.

7. The method according to claim 4 or 6, wherein: The clothing texture map generation model is optimized based on the first noise added and the second noise removed in the iterative process, including: calculating a loss function based on the first noise and the second noise; Parameters of the second network are adjusted based on the loss function until the loss function converges.

8. A device for generating a clothing texture map, comprising: A clothing feature acquisition module is used to obtain clothing features based on a clothing reference image; a texture feature acquisition module, configured to obtain texture features based on a coordinate map and a texture mask map associated with the clothing reference image, and input noise, the coordinate map, and the texture mask map into a second network of a clothing texture map generation model to obtain the texture features; wherein the coordinate map includes coordinates corresponding to the texture map unfolded by the clothing model associated with the clothing reference image on the clothing model; and the texture mask map includes an image generated based on a transparent channel in the texture map unfolded by the clothing model; A clothing map generation module is used to generate a clothing texture map based on the clothing features and the texture features. The clothing map generation module includes: A splicing submodule, configured to splice the clothing feature and the texture feature to obtain a splicing feature; A generation submodule is used to input the splicing features into the cross attention network of the second network to generate the clothing texture map.

9. The device according to claim 8, wherein The clothing feature acquisition module is used to input the clothing reference image into the first network of the clothing texture map generation model to obtain the clothing features.

10. The device according to claim 8, wherein Also includes: a matching module, configured to match an associated clothing model according to the clothing reference image; A coordinate mapping module, configured to perform a reverse query on the coordinates of each point of the clothing model based on the texture map unfolded by the clothing model, and obtain the coordinate mapping map associated with the clothing reference map; The texture mask module is configured to obtain the texture mask image associated with the clothing reference image based on a transparent channel of the texture map unfolded by the clothing model.

11. A clothing texture map generation model training device, comprising: A clothing feature acquisition module is used to obtain clothing features based on clothing reference images in training samples; a texture feature acquisition module, configured to obtain texture features based on the coordinate map and texture mask map in the training sample and input noise, and to input the input noise, the coordinate map, and the texture mask map into a second network of a clothing texture map generation model to obtain the texture features; wherein the coordinate map includes coordinates corresponding to the texture map unfolded by the clothing model associated with the clothing reference image on the clothing model; and the texture mask map includes an image generated based on a transparent channel in the texture map unfolded by the clothing model; An optimization module, configured to optimize the clothing texture map generation model based on the clothing features and the texture features, the optimization module comprising: A splicing submodule, configured to splice the clothing feature and the texture feature to obtain a splicing feature; An input submodule, configured to input the concatenated features into a cross-attention network of the second network; an iterative submodule, configured to iteratively process the splicing features through the second network; The optimization submodule is used to optimize the clothing texture map generation model based on the first noise added and the second noise removed in the iterative process.

12. The device according to claim 11, wherein The clothing feature acquisition module is used to input the clothing reference image into the first network of the clothing texture map generation model to obtain the clothing features.

13. The device according to claim 11, wherein The iterative submodule is used to perform multi-stage iterative processing on the splicing features through the second network to obtain added first noise and removed second noise; wherein the multi-stage iterative processing includes a plurality of consecutive forward diffusion steps and a plurality of consecutive backward diffusion steps; the plurality of consecutive forward diffusion steps are used to iteratively add noise to the splicing features to obtain a noisy target image, and the first noise includes the noise added in the plurality of consecutive forward diffusion steps; the plurality of consecutive backward diffusion steps are used to iteratively remove noise from the target image to obtain a restored texture map, and the second noise includes the noise removed in the plurality of consecutive backward diffusion steps.

14. The device according to claim 11 or 13, wherein The optimization submodule is used to calculate a loss function based on the first noise and the second noise; and adjust parameters of the second network based on the loss function until the loss function converges.

15. A method for generating clothing for a digital character, comprising: Generating a clothing texture map of the digital character by the method according to any one of claims 1 to 3; The clothing of the digital character is generated by using the clothing texture map of the digital character.

16. A digital character comprising: Digital clothing generated through clothing texture mapping; The clothing texture map is generated by using the method according to any one of claims 1 to 3.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

19. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images