A text-driven human body posture and texture editing method and device

Abstract

The present invention discloses a text-driven human posture and texture editing method and device in the field of computer vision. The method comprises: firstly extracting features from the text description of posture information, converting them into skeleton node coordinates and drawing them into a skeleton diagram; then extracting features from the clothing shape text, combining the skeleton diagram with the human body analysis diagram through a codec network; training texture-aware FSQ, learning different texture feature vector representations, inputting the texture text description and the human body analysis diagram into the FSQ for sampling, filling the texture and generating a relatively rough human body image that conforms to the text description. Then fine-tuning the diffusion model used for posture migration is used to further optimize the quality of the generated human body image. This method overcomes the problems of unstable generation capability and insufficient image control of existing models, ensures that the generated human body image is realistic and consistent with the semantics of the input text, and provides a new and effective way for human body image generation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and is mainly aimed at character attribute editing, and in particular to a text-driven human body posture and texture editing method and device. Background Art

[0002] The goal of character attribute editing technology is to be able to freely generate target character images using editable attribute codes. In the field of computer vision, character attribute editing has great research value and extremely broad application prospects. Current research results can be applied in many fields such as virtual reality, filmmaking, and data enhancement. However, since most current methods use image-driven or text-driven methods based on large models, these methods have flexibility problems or excessive computational complexity, which limits the progress of technology.

[0003] When implementing texture control, current methods use methods such as VAE and VQVAE. The biggest problem with VAE is that it uses continuous distribution to learn texture information. Since the distribution is too large, the effect is very unstable during the sampling process. VQVAE learns texture information through discrete codebooks, but its coding feature dimension is generally large, usually 256, and the codes in the codebook are randomly generated during initialization. The minimum distance method is used to replace the feature vector in the discrete process. Therefore, in the actual training process, the codebook collapse phenomenon is very likely to occur. That is, although the amount of code is large, the code that is finally used only occupies a small part of it, resulting in unsatisfactory texture learning results when the texture information is more complex. Summary of the invention

[0004] In order to address the deficiencies in the prior art, the present invention provides a text-driven human body posture and texture editing method and device, which not only overcomes the problem of poor texture control effect in previous methods, but also can use text description to drive the generation of target person images with a small amount of calculation, and the generated person images have reliable depth information, precise shape and clear face.

[0005] In a first aspect, the present invention provides a text-driven human body posture and texture editing method, comprising:

[0006] Acquire input text, wherein the input text includes a text description about a target posture, a text description about a clothing shape, and a text description about a texture;

[0007] The text description of the target posture, the text description of the clothing shape and the text description of the texture are respectively extracted through a feature extraction network to obtain a posture feature vector, a shape feature vector and texture category information;

[0008] Inputting the posture feature vector into a trained posture generation model to obtain corresponding skeleton coordinates, and drawing a skeleton graph according to the skeleton coordinates;

[0009] Inputting the skeleton graph in combination with the shape feature vector into a human body analysis graph generation model to generate a corresponding human body analysis graph;

[0010] Inputting the human body analysis image into a human body image generation model in combination with the texture category information to obtain a rough target human body image;

[0011] The rough target human image is input into a quality enhancement network to obtain a final human image.

[0012] Furthermore, the posture generation model is constructed based on multiple fully connected layers, with the input being the posture feature vector and the output being the skeleton coordinates;

[0013] The human body parsing graph generation model is constructed based on a multi-scale encoding and decoding network, the input is a combination of a skeleton graph and a shape feature vector, and the output is a human body parsing graph, which contains the outline of the target person and the clothing shape category information;

[0014] The human body image generation model is formed by training a finite scalar quantization model codebook for representing the vector representation of texture information in the feature space for different textures, the input is a combination of a human body analysis map and texture category information, and the output is a rough target human body image;

[0015] The quality improvement network is obtained by fine-tuning a diffusion model for posture transfer so that it has image denoising and image restoration functions. The input is a combination of a rough target human image and a human skeleton map, and the output is a target human image.

[0016] Furthermore, the method of drawing a skeleton graph according to the skeleton coordinates includes: drawing the skeleton coordinates into a skeleton graph by using a pose estimation algorithm OpenPose.

[0017] Furthermore, the training method of the posture generation model includes:

[0018] Acquire multiple human body images;

[0019] Create a training dataset based on multiple human images;

[0020] Training the posture generation model according to the training data set to obtain a trained posture generation model;

[0021] Methods for making training datasets include:

[0022] The pose estimation algorithm OpenPose is used to extract the skeleton coordinates of human images;

[0023] The posture of human images is categorized and labeled based on three attributes: human orientation, hand movements, and body inclination:

[0024] The categories of human body orientation are divided into: front, left, right and back;

[0025] The categories of hand movements include natural hanging, pocketing, raising hands, and crossing;

[0026] The categories of body orientation included no tilt, tilt to the left, and tilt to the right;

[0027] The skeleton coordinates and corresponding category labels are sorted to obtain a training data set.

[0028] Furthermore, the method for acquiring the posture feature vector includes:

[0029] Extracting category information of three attributes from the text description of the posture;

[0030] The category information of the three attributes is converted into one-hot encoding to obtain three feature vectors, which are then concatenated to obtain a posture feature vector.

[0031] Furthermore, the method for executing the posture generation model includes:

[0032] Encode each feature vector;

[0033] The encoded feature vector is fused to obtain a complete posture feature representation , the process is expressed as:

[0034] ;

[0035] in is the feature encoding module, is the feature fusion module, is the i-th posture attribute feature vector;

[0036] The pose feature representation is converted to the target skeleton coordinates based on the fully connected layer.

[0037] Furthermore, the posture generation model also includes a coordinate conversion network, which is used to replace the coordinates of the corresponding arm and leg nodes with a certain probability when encountering a non-positive orientation, and finally output 18 pairs of skeleton coordinates that meet the description , the process is expressed as:

[0038] ;

[0039] in is the coordinate transformation network, is a fully connected layer.

[0040] Furthermore, the method for obtaining the shape feature vector includes:

[0041] Extracting category information of each shape attribute from the text description of clothing shapes;

[0042] The category information of shape attributes includes hairstyle, top length, pants length, skirt, dress, and coat;

[0043] The category information of each shape attribute is converted into one-hot encoding and feature fusion is performed to obtain a shape feature vector .

[0044] Furthermore, the method for executing the human body analysis graph generation model includes:

[0045] Through the multi-scale codec network, the output of the previous decoder is used as the input of the next decoder, and the output of the encoder of the corresponding layer is provided to the decoder by jumping. The process is expressed as:

[0046] ;

[0047] ;

[0048] ;

[0049] in, is the feature encoding module, is the category information of the i-th shape attribute, To fuse the category information of each shape attribute, and are the encoder and decoder of the i-th layer, is an operation for unifying the shape of the input, is the output of the encoder at layer i, is the output of the ith decoder, is the shape feature vector.

[0050] Furthermore, the method for executing the human body image generation model includes:

[0051] After obtaining the human body analysis diagram, the human body analysis diagram is divided into parts according to the parts;

[0052] Then, the texture category information of each part is extracted from the text description of the texture; the texture category information of each part is converted into a one-hot encoding form to obtain the texture feature vector of each part;

[0053] Filling the texture feature vectors of each part into the human body analysis map according to the part to obtain a human body texture map;

[0054] Input the human body analysis map, human body texture map and full mask initial index sequence into the trained sampler to obtain an updated index sequence;

[0055] Corresponding codes are obtained from the pre-constructed codebook through the updated index sequence to form a discretized feature vector, and the discretized feature vector is input into a decoder to obtain a rough human body image.

[0056] Furthermore, the input of the sampler is a human body analysis map , human body texture map and the fully masked initial index sequence , the initial index sequence of the full mask is optimized step by step by diffusion until the index sequence corresponding to the target human image is obtained ,according to Extract feature maps from the codebook , the feature map Input decoder to get rough human image , the process is expressed as:

[0057] ;

[0058] ;

[0059] ;

[0060] in is a multi-head attention network, and , , are the learnable codebooks corresponding to the full mask initial index sequence, the human body analysis map, and the human body texture map, respectively. Indicates extracting the corresponding code in the codebook according to the index. For the decoder.

[0061] The method for obtaining the trained sampler includes:

[0062] Acquire a human body image as a source image;

[0063] Input the source image into the trained finite scalar quantization model FSQ to obtain the real index sequence;

[0064] Input the full mask initial index sequence, the human body analysis map generated according to the source image, and the human body texture map into the sampler, and sample to obtain the output index sequence;

[0065] The sampler is trained by reducing the difference between the output index sequence and the true index sequence to obtain a trained sampler.

[0066] Furthermore, the finite scalar quantization model FSQ includes:

[0067] The encoder is used to encode the input image to obtain the feature map before discretization;

[0068] The discretization module is used to scale and round off each vector in the feature map before discretization so that each dimension is an integer within a fixed range, and obtain the discretized feature map. The process is expressed as:

[0069] ;

[0070] in, For human body images The feature map obtained after passing through the encoder, It is the feature map after the feature map Z is discretized by the discretization module. is the gradient cutoff function, is the one-hot encoding of the i-th texture category, Used to connect two vectors on the feature dimension, The operation is a rounding operation. is a sigmoid operation, which converts each dimension of the feature vector to the interval (0, 1). L is a hyperparameter used to control the value range of each dimension of the vector. and are the eigenvectors before and after discretization respectively.

[0071] Furthermore, the method for obtaining the pre-constructed codebook includes:

[0072] A trained finite scalar quantization model FSQ is obtained by training each category of texture;

[0073] Extract the feature vector range that can be output by the discretization module in the trained finite scalar quantization model FSQ corresponding to each category of texture to form the codebook corresponding to the texture. The overall process is as follows:

[0074] ;

[0075] ;

[0076] ;

[0077] ;

[0078] in, is the original codebook set of different textures, is the codebook corresponding to the i-th texture, For each code in the codebook, is the codebook corresponding to the i-th texture after updating according to the texture category, n is the number of codebooks, that is, the number of texture categories, Used to connect two vectors on the feature dimension, is the one-hot encoding for the i-th texture category.

[0079] Furthermore, the training method of the finite scalar quantization model FSQ includes:

[0080] The finite scalar quantization model FSQ is trained through the loss function;

[0081] The loss function is:

[0082] ;

[0083] in is the L1 norm, I is the original image of the input finite scalar quantization model FSQ, It is the reconstructed image output by the finite scalar quantization model FSQ.

[0084] Furthermore, the method for obtaining the quality improvement network includes:

[0085] A diffusion model originally used for posture transfer is fine-tuned, with a rough human image and the corresponding skeleton image as the model input and a fine human image as the output;

[0086] The fine human image is denoised to simulate a rough human image, and the reconstruction loss is used and perceived loss As the loss function for fine-tuning the model , the function is as follows:

[0087] ;

[0088] ;

[0089] ;

[0090] in Indicates using the pre-trained VGG network to extract the features of the i-th layer, Indicates the number of layers of the VGG network used, where , is the tuning parameter used to balance different loss functions; is the L1 norm, I is the original image of the input finite scalar quantization model FSQ, It is the reconstructed image output by the finite scalar quantization model FSQ;

[0091] Based on the loss function , train the quality improvement network and obtain a trained quality improvement network.

[0092] In a second aspect, the present invention provides a text-driven human body posture and texture editing device, comprising:

[0093] Input module: used to obtain input text, wherein the input text includes a text description about the target posture, a text description about the shape of the clothing, and a text description about the texture;

[0094] Feature extraction module: used to extract the text description of the target posture, the text description of the clothing shape and the text description of the texture through the feature extraction network, and obtain the posture feature vector, the shape feature vector and the texture category information;

[0095] Posture generation module: used for inputting the posture feature vector into the trained posture generation model to obtain the corresponding skeleton coordinates, and drawing a skeleton diagram according to the skeleton coordinates;

[0096] A human body analysis graph generation module: used for inputting the skeleton graph in combination with the shape feature vector into a human body analysis graph generation model to generate a corresponding human body analysis graph;

[0097] A rough human body image generation module: used for inputting the human body analysis image in combination with the texture category information into a human body image generation model to obtain a rough target human body image;

[0098] Output module: used to input the rough target human image into the quality improvement network to obtain the final human image.

[0099] Compared with the previous human body image generation method, the present invention has the following beneficial effects:

[0100] (1) Compared with image-driven methods, text-driven methods have improved flexibility and controllability. By describing details about specific human attributes, postures, clothing, etc., the required human images can be generated more accurately.

[0101] (2) Compared with large models, the amount of calculation and training data set are greatly reduced, which is more conducive to the use and improvement of technology.

[0102] (3) Text-driven can effectively avoid copyright issues, so you don’t have to worry about infringement when using it.

[0103] (4) First, extract features from the text description of posture information, convert them into skeleton node coordinates and use the posture estimation algorithm OpenPose to draw a skeleton graph; then extract features from the clothing shape text, combine the skeleton graph and obtain the human body analysis graph through the codec network; train the texture-aware FSQ (finite scalar quantization model), learn different texture feature vector representations, input the texture text description and the human body analysis graph into the FSQ for sampling, fill the texture and generate a rough but text-compliant human body image. Then fine-tune the diffusion model used for posture transfer so that it has image denoising and restoration functions, which is used to further optimize the quality of the generated human body image. This method overcomes the problems of unstable generation capabilities and insufficient image control of existing models, ensures that the generated human body image is realistic and consistent with the input text semantics, and provides a new and effective way to generate human body images. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] Figure 1 This is the overall framework flow chart of the text-driven human posture and texture editing method;

[0105] Figure 2 This is the overall structure diagram of the text-driven human posture and texture editing method;

[0106] Figure 3 Generate a module graph for the posture;

[0107] Figure 4 It is a posture diagram - a human body analysis diagram module diagram;

[0108] Figure 5 This is a functional diagram of the texture-aware FSQ module;

[0109] Figure 6 It is a human body analysis diagram - a human body image module diagram;

[0110] Figure 7 This is the structure diagram of the rough human image sampler. DETAILED DESCRIPTION

[0111] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0112] Embodiment 1:

[0113] This embodiment discloses a text-driven human posture and texture editing method in the field of deep learning images. The flowchart and structure diagram of the overall framework are shown in FIG. Figure 1 , 2 shown.

[0114] The input of the posture generation model in the first stage is to extract posture information from the input text, obtain the corresponding skeleton node coordinates through the model, and draw a skeleton diagram using openpose. The skeleton diagram is combined with the description of clothing shape in the text as the input of the second stage. The second stage is the human body analysis diagram generation model, which outputs the corresponding human body analysis diagram. Combined with the description of texture as the input of the third stage human body image generation model, a rough target human body image is obtained, and the final human body image is obtained by inputting the quality improvement network.

[0115] The character image generation method specifically comprises the following steps:

[0116] Step 1.1: Create a training data set and annotate the posture attributes of the training data set. The posture attributes include body orientation, hand movements, and body tilt direction. The human skeleton coordinates and corresponding skeleton graph are extracted through the posture estimation algorithm OpenPose.

[0117] Step 1.2: The text description of the target posture is converted into a posture feature vector through the feature extraction network, and the feature vector is converted into the corresponding skeleton coordinates through the posture generation network, and then drawn into a skeleton graph.

[0118] Step 1.3: The text description of the target clothing shape is converted into a shape feature vector through the feature extraction network. Combined with the skeleton image obtained in step 1.2, it is input into the multi-scale encoding and decoding network (human body parsing graph generation model) to generate the corresponding human body parsing graph, which contains the outline of the target person and the clothing shape category information.

[0119] Step 1.4: Extract texture category information from the text description of human body texture through the feature extraction network, and fill in the texture category on the human body analysis graph from the three attributes of tops, bottoms and coats. Train a FSQ (Finite Scalar Quantization Model) codebook for different textures to represent the vector representation of texture information in the feature space.

[0120] Step 1.5: Based on the trained texture-aware FSQ obtained in step 1.4, train a sampler to sample a rough target human image based on the texture category, human body analysis map and codebook index.

[0121] Step 1.6: Fine-tune the pre-trained diffusion model for posture transfer as the final quality improvement network of the model. The rough human image obtained in step 1.5 and the skeleton image obtained in step 1.2 are used as the input of the network to generate a high-quality human image that meets the text description.

[0122] Specifically, the character image generation method comprises the following steps:

[0123] like Figure 3 As shown in the figure, the posture estimation algorithm OpenPose is used to extract the skeleton coordinates of the training data and draw the corresponding skeleton diagram. The posture of the human image is categorized and labeled from three attributes: human body orientation, hand movement, and body inclination. The human body orientation is divided into four categories: front, left, right, and back. The hand movement includes four categories: natural drooping, pocket, raised hand, and crossed. The body inclination includes three categories: no tilt, left tilt, and right tilt.

[0124] like Figure 3 , Figure 4 As shown in the figure, this embodiment extracts the category information of three preset attributes from the text description of the posture, obtains three feature vectors in the form of one-hot encoding, inputs these feature vectors into the posture generation network to obtain the skeleton coordinates of the target posture, and draws a skeleton graph. In the posture generation network, each feature vector is first encoded, and then feature fusion is performed to obtain a complete posture feature representation , the process can be expressed as:

[0125]

[0126] in is the feature encoding module, is the feature fusion module, is the i-th posture attribute feature vector.

[0127] After obtaining the posture feature representation, it is converted into the coordinates of the target skeleton through the fully connected layer. In order to avoid the interference of network learning due to the default value (-1, -1) of some coordinates caused by the invisible skeleton when extracting the skeleton (such as the non-positive orientation), this method adds a coordinate conversion network to replace the coordinates of the corresponding arm and leg nodes with a certain probability when encountering a non-positive orientation, so that the generalization ability of the network can be improved, and finally the skeleton coordinates that meet the description are output (18 pairs) And the corresponding skeleton diagram, the process can be expressed as:

[0128]

[0129] in is the coordinate transformation network, is a fully connected layer.

[0130] By extracting the category information of each shape attribute from the text description of the clothing shape, including hairstyle, top length, pants length, skirt, dress and coat, etc., converting it into one-hot encoding and performing feature fusion, we can get , and use it together with the skeleton map as the input of the human body parsing map generation network to obtain the target human body parsing map. The human body parsing map generation network adopts the design of the codec structure. In order to enable each decoder to obtain more comprehensive information, in addition to using the output of the previous layer decoder as input, the output of the encoder of the corresponding layer will also be provided to the decoder by jumping. The process can be expressed as:

[0131]

[0132]

[0133]

[0134] in, is the feature encoding module, is the feature fusion module, is the i-th posture attribute feature vector, To fuse the category information of each shape attribute, and are the encoder and decoder of the i-th layer, is an operation for unifying the shape of the input, is the output of the encoder at layer i, is the output of the ith decoder, is the shape feature vector.

[0135] The finite scalar quantization model FSQ is used as the basis of the texture learning network. This method is also a model of a codec structure. The original image is passed through the encoder to obtain a feature vector, and then each vector in the feature vector is scaled and rounded so that each dimension is an integer within a fixed range. Then the discretized feature vector is passed through the decoder to reconstruct the original image. This method trains an implicit codebook for subsequent sampling of the target image from the discrete space. The process can be expressed as:

[0136]

[0137] in The operation is a rounding operation. For the sigmoid operation, each dimension of the feature vector is converted to the interval (-1, 1). The hyperparameter and They are used to control the value range of each dimension of the vector and the dimension of the vector. and are the eigenvectors before and after discretization respectively.

[0138] Notice, It represents each eigenvector, and Zq represents the feature map, that is, the feature matrix.

[0139] like Figure 5 As shown in Figure 1, in order to avoid confusion between different texture learning, this method adopts a texture-aware FSQ model, that is, for the preset 18 textures (stripes, solid colors, patterns, and patterns, etc.), the corresponding codebooks are trained to learn the vector representation of texture in discrete space. Specifically, since the vectors in the original codebooks are in the same form, they are all determined by the hyperparameters and Therefore, this method connects the one-hot encoding of the current texture category after each dimensional vector. The overall process is as follows:

[0140]

[0141]

[0142]

[0143]

[0144] in, For human body images The feature vector obtained after passing through the encoder is is the codebook corresponding to the i-th texture, and It is the feature vector discretized by texture-aware FSQ , n is the number of codebooks, i.e., texture categories, which is 18 in this method. For each code in the codebook, Used to concatenate two vectors along the feature dimension.

[0145] Additional explanation: The original codebook Eti is obtained only by the hyperparameters L and d, so in fact, Eti is exactly the same, so in order to distinguish, we just ti Every vector e in i Then we concatenate the texture categories (one-hot encoding) and get .

[0146] The loss function of the entire network can be simply designed as:

[0147]

[0148] in is the L1 norm, for The reconstructed image obtained by the decoder.

[0149] Due to the generation The process is discrete, so in order to ensure that the gradient can be effectively transferred, it is necessary to To transfer the gradient, do the following:

[0150]

[0151] in is the gradient cutoff function, is the corresponding texture category.

[0152] Formula (12) is modified based on formula (10) to ensure that the gradient can be transferred during the training process (the reason why the gradient of formula (10) cannot be transferred is that the round operation is not a linear operation and cannot be derived).

[0153] Description of the finite scalar quantization model FSQ:

[0154] 1) The structure of the entire model is encoder-discretization module-decoder. The method of use is to input the source image into the encoder to obtain the feature map, and then input the feature map into the discretization module (first scale each dimension to the range of (0, L) and then round it off). The discretized feature map is obtained, and then the source image is reconstructed through the encoder.

[0155] 2) Focus on explaining the discretization module. There are two hyperparameters, one is L and the other is d. d is used to control the feature dimension of each feature vector in the feature map, and L is used to control the value range of each dimension. For example, L=2, d=3, then each dimension is in (0, 2), and after rounding, a vector may be (1, 0, 2). The number of all possible vectors is (2+1)^3. These 27 vectors constitute a codebook.

[0156] 3) Train FSQ by reducing the difference between the source image and the reconstructed image using the method in Explanation 1. In actual experiments, we trained a FSQ for each texture.

[0157] like Figure 6 , Figure 7 As shown in the figure, after obtaining the trained texture perception FSQ, the present invention trains a sampler to sample and obtain a rough target human image. The input of the sampler is the human body analysis map , target texture category and the encoding index sequence in texture-aware FSQ , the original complete mask index sequence is optimized step by step by diffusion until the index sequence corresponding to the target human image is obtained ,according to Extract feature maps from the codebook , the feature map Input decoder to get rough human image , the process can be simply expressed as:

[0158]

[0159]

[0160]

[0161] in is a multi-head attention network, and , , are the learnable codebooks corresponding to the index, parsing graph, and texture, respectively. Indicates extracting the corresponding code in the codebook according to the index. For the decoder.

[0162] The sampler model can use MOE-sampler, and the internal implementation formula can be expressed as formula (13). The three learnable codebooks are the embedding layer. First, the three inputs are input into the embedding layer to obtain the embedding vector (embedding). Then, through a layer with a self-attention mechanism (self-attention), the output is obtained.

[0163] The specific usage of the sampler is:

[0164] 1) After obtaining the human body analysis map, this human body analysis map divides the entire person's outline into multiple categories, including upper body parts, lower body parts, coat parts, hairstyle, glasses, etc.

[0165] 2) Then extract the texture categories of tops, bottoms, and coats from the original input sentence describing the target texture. Fill the category numbers into the human body analysis map by part. It can be named as a human body texture map.

[0166] 3) The human body analysis map, human body texture map, and full mask initial index sequence are input into the trained sampler as input to obtain an updated index sequence. The discretized feature map of the corresponding coding composition is obtained from the code book through the index sequence, and input into the decoder to obtain a rough human body image.

[0167] The method for obtaining the codebook specifically includes:

[0168] After training FSQ for different textures (i.e., texture-aware FSQ), we can find that due to the design of the discretization module, the contents of different codebooks are consistent. In order to distinguish them, we encode the 18 classes from 0 to 17 using a one-hot encoding method (i.e., if there are three classes, class 1 is (0, 0, 1), class 2 is (0, 1, 0), and class 3 is (1, 0, 0)) and connect them to each discretized feature vector.

[0169] During the sampling process, we convert all vectors in the codebook into continuous serial numbers according to their categories. For example, in the experiment, we set L=3 and d=5. Then the number of codes in each codebook is (L+1)^d=1024. Then, the code serial number of category 0 is 0-1023, the code serial number of category 1 is (0-1023)+1024, and the code serial number of category i is (0-1023)+1024*i.

[0170] Since there are 18 textures in total, the total amount of codes is 18*1024=18432. We need to train a sampler that can sample from 18432 codes. Its input is ① the initial index sequence of the full mask, ② the human body analysis map, ③ the texture category corresponding to each block of the human body analysis map (actually it is also a picture, obtained by directly filling the corresponding texture category (0, 1, 2...) on the human body analysis map). Then continuously sample to obtain a fully updated index sequence, extract the code from the total code book according to these index sequences, that is, obtain the discretized feature map mentioned in the previous comment, and then input it into the decoder to obtain a rough human body image.

[0171] The method for training the sampler is to first input the source image into the texture-aware FSQ to obtain a real index sequence, then sample the sampler output index sequence by the above method, and train the sampler by reducing the difference between the real index sequence and the source image.

[0172] like Figure 6 As shown, after obtaining the rough human image, the present invention additionally trains an image quality improvement network to optimize it and obtain a high-quality human image that meets the target description. Specifically, the present invention utilizes the powerful and stable generation effect of the diffusion model to fine-tune a diffusion model originally used for posture migration, that is, to learn at the image resolution of the present invention, use the rough human image and the corresponding skeleton map as model input, and output a fine human image. During the training process, the present invention adds noise to the fine human image to simulate the rough human image, and uses the reconstruction loss and perceived loss As the loss function for fine-tuning the model , the specific functions are as follows:

[0173]

[0174]

[0175]

[0176] in Indicates using the pre-trained VGG network to extract the features of the i-th layer, Indicates the number of layers of the VGG network used, where , is a tuning parameter used to balance different loss functions.

[0177] The method of this embodiment can overcome the problems of unstable generation capability and insufficient image control of existing models, ensure that the generated human body image is realistic and consistent with the semantics of the input text, and provide a new and effective way to generate human body images. Compared with the generated effect diagram of the prior art, the image generated by this method is more consistent with the text description and has higher image quality.

[0178] Embodiment 2:

[0179] Based on the same innovative idea as that of Embodiment 1, this embodiment provides a text-driven human body posture and texture editing device, including:

[0180] Input module: used to obtain input text, wherein the input text includes a text description about the target posture, a text description about the shape of the clothing, and a text description about the texture;

[0181] Feature extraction module: used to extract the text description of the target posture, the text description of the clothing shape and the text description of the texture through the feature extraction network, and obtain the posture feature vector, the shape feature vector and the texture category information;

[0182] Posture generation module: used for inputting the posture feature vector into the trained posture generation model to obtain the corresponding skeleton coordinates, and drawing a skeleton diagram according to the skeleton coordinates;

[0183] A human body analysis graph generation module: used for inputting the skeleton graph in combination with the shape feature vector into a human body analysis graph generation model to generate a corresponding human body analysis graph;

[0184] A rough human body image generation module: used for inputting the human body analysis image in combination with the texture category information into a human body image generation model to obtain a rough target human body image;

[0185] Output module: used to input the rough target human image into the quality improvement network to obtain the final human image.

[0186] The device of this embodiment can be used to implement the method described in Embodiment 1.

[0187] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0188] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0189] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0190] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0191] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A text-driven human body posture and texture editing method, characterized in that: include: Acquire input text, wherein the input text includes a text description about a target posture, a text description about a clothing shape, and a text description about a texture; The text description of the target posture, the text description of the clothing shape and the text description of the texture are respectively extracted through a feature extraction network to obtain a posture feature vector, a shape feature vector and texture category information; Inputting the posture feature vector into a trained posture generation model to obtain corresponding skeleton coordinates, and drawing a skeleton graph according to the skeleton coordinates; Inputting the skeleton graph in combination with the shape feature vector into a human body analysis graph generation model to generate a corresponding human body analysis graph; Inputting the human body analysis image into a human body image generation model in combination with the texture category information to obtain a rough target human body image; Inputting the rough target human image into a quality enhancement network to obtain a final human image; The execution method of the human body image generation model includes: After obtaining the human body analysis diagram, the human body analysis diagram is divided into multiple parts according to the parts; Then, the texture category information of each part is extracted from the text description of the texture; the texture category information of each part is converted into a one-hot encoding form to obtain a texture feature vector of each part; Fill the texture feature vectors of each part into the human body analysis map according to the part to obtain the human body texture map; Input the human body analysis map, human body texture map and full mask initial index sequence into the trained sampler to obtain an updated index sequence; Obtaining corresponding codes from a pre-constructed codebook through the updated index sequence to form a discretized feature vector, and inputting the discretized feature vector into a decoder to obtain a rough human body image; The input of the sampler is the human body analysis map T seg , human body texture map T tex And the full mask initial index sequence T code , the initial index sequence of the full mask is gradually optimized by diffusion until the index sequence T′ corresponding to the target human image is obtained code , according to T′ code The feature map Z2 is extracted from the codebook, and the feature map Z2 is input into the decoder to obtain a rough human image f. The process is expressed as: T′ code =MHA(E code (T code ),E seg (T seg ),E tex (T tex )); Z2=C(T′ code ); f = Dec(Z2); Where MHA(·) is a multi-head attention network, and E code 、E seg 、E tex are the learnable codebooks corresponding to the full mask initial index sequence, the human body analysis map, and the human body texture map, respectively. C(·) represents the extraction of the corresponding code in the codebook according to the index, and Dec(·) is the decoder; The method for obtaining the trained sampler includes: Acquire a human body image as a source image; Input the source image into the trained finite scalar quantization model FSQ to obtain the real index sequence; Input the full mask initial index sequence, the human body analysis map generated according to the source image, and the human body texture map into the sampler, and sample to obtain the output index sequence; The sampler is trained by reducing the difference between the output index sequence and the true index sequence to obtain a trained sampler; The finite scalar quantization model FSQ includes: The encoder is used to encode the input image to obtain the feature map Z before discretization; The discretization module is used to scale and round off each vector in the feature map Z before discretization so that each dimension is an integer within a fixed range, and obtain the discretized feature map. The process is expressed as: Among them, Z is the feature map obtained after the human image I passes through the encoder, Z q is the feature map after the feature map Z is discretized by the discretization module, sg(·) is the gradient cutoff function, and lb i is the unique hot encoding of the i-th texture category, cat(·) is used to connect two vectors in the feature dimension, Round(·) is a rounding operation, σ(·) is a sigmoid operation, which converts each dimension of the feature vector to the interval (0, 1), and L is a hyperparameter used to control the value range of each dimension of the vector; The method for obtaining the pre-constructed codebook includes: A trained finite scalar quantization model FSQ is obtained by training each category of texture; Extract the feature vector range that can be output by the discretization module in the trained finite scalar quantization model FSQ corresponding to each category of texture to form the codebook corresponding to the texture of this category. The overall process is as follows: Among them, E is the original codebook set of different textures, is the codebook corresponding to the i-th texture, e i For each code in the codebook, is the codebook corresponding to the i-th texture after updating according to the texture category, n is the number of codebooks, that is, the number of texture categories, cat(·) is used to connect the two vectors in the feature dimension, lb i is the unique hot encoding of the i-th texture category, d is a hyperparameter used to control the dimension, Represents the set of positive integers.

2. The text-driven human body posture and texture editing method according to claim 1, characterized in that: The posture generation model is constructed based on multiple fully connected layers, with the input being the posture feature vector and the output being the skeleton coordinates; The human body parsing map generation model is constructed based on a multi-scale codec network, with a combination of a skeleton map and a shape feature vector as input, and a human body parsing map as output, wherein the human body parsing map contains the outline of the target person and clothing shape category information; the human body image generation model is constructed based on a multi-head attention network, with a combination of a human body parsing map and texture category information as input, and a rough target human body image as output; The quality enhancement network is obtained by fine-tuning a diffusion model for posture transfer, the input of which is a combination of a rough target human image and a human skeleton map, and the output of which is a target human image.

3. The text-driven human body posture and texture editing method according to claim 2, characterized in that: The method for acquiring the posture feature vector includes: Extracting category information of three attributes from the text description of the posture; Convert the category information of the three attributes into one-hot encoding to obtain three feature vectors, and concatenate them to obtain a posture feature vector; The methods performed by the pose generation model include: Encode each feature vector; The encoded feature vector is fused to obtain the complete posture feature representation f pose , the process is expressed as: f pose =Fusion pose ([E1(a1),...,E i (to i ),...,AND n (to n )]); Where E i (·) is the feature encoding module, Fusion pose (·) is the feature fusion module, a i is the i-th posture attribute feature vector; The pose feature representation is converted to the target skeleton coordinates based on the fully connected layer.

4. The text-driven human body posture and texture editing method according to claim 3, characterized in that: The posture generation model also includes a coordinate transformation network, which is used to replace the coordinates of the corresponding arm and leg nodes with a certain probability when encountering a non-positive orientation, and finally outputs 18 pairs of skeleton coordinate sets that meet the description. coord , the process is expressed as: set coord =Trans(FCN(f pose )); Among them, Trans(·) is the coordinate transformation network and FCN(·) is the fully connected layer.

5. The text-driven human body posture and texture editing method according to claim 2, characterized in that: The method for obtaining the shape feature vector includes: Extracting category information of each shape attribute from the text description of clothing shapes; The category information of shape attributes includes: hairstyle, top length, pants length, skirt, dress, and coat; The category information of each shape attribute is converted into one-hot encoding and feature fusion is performed to obtain the shape feature vector f shape ; The method for executing the human body analysis graph generation model includes: Through the multi-scale codec network, the output of the previous decoder is used as the input of the next decoder, and the output of the encoder of the corresponding layer is provided to the decoder by skipping. The encoding and decoding are performed layer by layer to finally obtain the human body analysis map. The process is expressed as: f shape =Fusion shape ([E1(b1),...,E i (b i ),...,E n (b n )]); Among them, E i (·) is the feature encoding module, b i is the category information of the i-th shape attribute, Fusion shape (·) is the operation of fusing the category information of each shape attribute, Enc i (·) and Dec i (·) is the encoder and decoder of the i-th layer, is an operation for unifying the shape of the input, is the output of the encoder at layer i, is the output of the ith decoder, f shape is the shape feature vector.

6. The text-driven human body posture and texture editing method according to claim 2, characterized in that: The method for obtaining the quality improvement network includes: A diffusion model originally used for posture transfer is fine-tuned, with a rough human image and the corresponding skeleton image as the model input and a fine human image as the output; The fine human image is denoised to simulate a rough human image, and the reconstruction loss L is used rec And the perceptual loss L perc As the loss function L of the fine-tuned model fine-tune , the function is as follows: L fine-tune =λ rec L rec +λ perc L perc ; Among them, ||·|| is the L1 norm, I is the original image of the input finite scalar quantization model FSQ, is the reconstructed image output by the finite scalar quantization model FSQ, Indicates the use of the pre-trained VGG network to extract the features of the i-th layer, N VGG represents the number of layers of the VGG network used, where λ rec ,λ perc is the tuning parameter used to balance different loss functions; Based on the loss function L fine-tune , train the quality improvement network and obtain a trained quality improvement network.

7. A text-driven human body posture and texture editing device, characterized in that: include: Input module: used to obtain input text, wherein the input text includes a text description about the target posture, a text description about the shape of the clothing, and a text description about the texture; Feature extraction module: used to extract the text description of the target posture, the text description of the clothing shape and the text description of the texture through the feature extraction network, and obtain the posture feature vector, the shape feature vector and the texture category information; Posture generation module: used for inputting the posture feature vector into the trained posture generation model to obtain the corresponding skeleton coordinates, and drawing a skeleton diagram according to the skeleton coordinates; A human body analysis graph generation module: used for inputting the skeleton graph in combination with the shape feature vector into a human body analysis graph generation model to generate a corresponding human body analysis graph; A rough human body image generation module: used for inputting the human body analysis image in combination with the texture category information into a human body image generation model to obtain a rough target human body image; Output module: used for inputting the rough target human image into the quality improvement network to obtain the final human image; The execution method of the human body image generation model includes: After obtaining the human body analysis diagram, the human body analysis diagram is divided into multiple parts according to the parts; Then, the texture category information of each part is extracted from the text description of the texture; the texture category information of each part is converted into a one-hot encoding form to obtain a texture feature vector of each part; Fill the texture feature vectors of each part into the human body analysis map according to the part to obtain the human body texture map; Input the human body analysis map, human body texture map and full mask initial index sequence into the trained sampler to obtain an updated index sequence; Obtaining corresponding codes from a pre-constructed codebook through the updated index sequence to form a discretized feature vector, and inputting the discretized feature vector into a decoder to obtain a rough human body image; The input of the sampler is the human body analysis map T seg , human body texture map T tex And the full mask initial index sequence T code , the initial index sequence of the full mask is gradually optimized by diffusion until the index sequence T′ corresponding to the target human image is obtained code , according to T′ code The feature map Z2 is extracted from the codebook, and the feature map Z2 is input into the decoder to obtain a rough human image f. The process is expressed as: T′ code =MHA(E code (T code ),E seg (T seg ),E tex (T tex )); Z2=C(T′ code ); f = Dec(Z2); Where MHA(·) is a multi-head attention network, and E code 、E seg 、E tex are the learnable codebooks corresponding to the full mask initial index sequence, the human body analysis map, and the human body texture map, respectively. C(·) represents the extraction of the corresponding code in the codebook according to the index, and Dec(·) is the decoder; The method for obtaining the trained sampler includes: Acquire a human body image as a source image; Input the source image into the trained finite scalar quantization model FSQ to obtain the real index sequence; Input the full mask initial index sequence, the human body analysis map generated according to the source image, and the human body texture map into the sampler, and sample to obtain the output index sequence; The sampler is trained by reducing the difference between the output index sequence and the true index sequence to obtain a trained sampler; The finite scalar quantization model FSQ includes: The encoder is used to encode the input image to obtain the feature map Z before discretization; The discretization module is used to scale and round off each vector in the feature map Z before discretization so that each dimension is an integer within a fixed range, and obtain the discretized feature map. The process is expressed as: Among them, Z is the feature map obtained after the human image I passes through the encoder, Z q is the feature map after the feature map Z is discretized by the discretization module, sg(·) is the gradient cutoff function, and lb i is the unique hot encoding of the i-th texture category, cat(·) is used to connect two vectors in the feature dimension, Round(·) is a rounding operation, σ(·) is a sigmoid operation, which converts each dimension of the feature vector to the interval (0, 1), and L is a hyperparameter used to control the value range of each dimension of the vector; The method for obtaining the pre-constructed codebook includes: A trained finite scalar quantization model FSQ is obtained by training each category of texture; Extract the feature vector range that can be output by the discretization module in the trained finite scalar quantization model FSQ corresponding to each category of texture to form the codebook corresponding to the texture of this category. The overall process is as follows: Among them, E is the original codebook set of different textures, is the codebook corresponding to the i-th texture, e i For each code in the codebook, is the codebook corresponding to the i-th texture after updating according to the texture category, n is the number of codebooks, that is, the number of texture categories, cat(·) is used to connect the two vectors in the feature dimension, lb i is the unique hot encoding of the i-th texture category, d is a hyperparameter used to control the dimension, Represents the set of positive integers.

Images