Fashion figure image style transformation method

Abstract

The present application provides a fashion figure image style transformation method, comprising: a posture encoder and a style encoder respectively output posture features and style features; a cross attention module learns context information, and coarsely aligns the two features; an adaptive instance normalization module performs style normalization on the coarsely aligned features; a decoder injects style and refines texture again with position normalization, and generates a target image; and the effectiveness of the algorithm is verified according to performance indicators. Through the cross use of the cross attention mechanism, position normalization and energy distance loss function, the present application solves the problems of instability of image generation and limitation of spatial transformation capability in convolution operation, so as to realize the generation of more delicate clothing texture features.

Description

Technical Field

[0001] This invention belongs to the field of human image generation technology in computer vision, and in particular relates to a method for style transformation of fashion human images. Background Technology

[0002] Portrait image generation technology is a computer image processing technique that can generate images of people in different poses and images of clothing with different textures. This technology has great potential in fields such as fashion design, media production, online advertising, and virtual reality. Currently, portrait image generation technology based on deep learning is one of the latest technologies. Deep learning includes various machine learning techniques, such as convolutional neural networks, recurrent neural networks, and generative adversarial networks.

[0003] Generative Adversarial Networks (GANs) were first developed by Ma et al. using two U-Net-like generators to generate human images in corresponding poses, from coarse to fine. Most GANs follow an encoder-decoder design. The encoder captures the style features of the source image and the pose features of the target human's key points, then fuses the style into the pose features of the target image; the decoder reconstructs the target human image based on the deformed features. However, the instability of non-rigid deformations makes generating high-fidelity images very challenging.

[0004] Building upon the aforementioned model, the paper "Controllable person image synthesis with attribute-decomposed GAN" proposes a controllable human image generation scheme, introducing multiple adaptive instance normalization modules for style transfer. In this way, the generated person images maintain their geometric shape well. However, due to the limitations of spatial transformation capabilities in large-scale local convolution operations, the model cannot generate more refined textures.

[0005] To address these issues, many methods utilize other optimization models. However, besides the problem of image imprecision, flow-field-based approaches generate large-area artifacts. Approaches relying on additional training phases to obtain semantic segmentation maps of the target image incur significant additional time and cost. Furthermore, because accurately predicting semantic segmentation maps is extremely difficult, even minor errors in the semantic segmentation map can cause substantial deviations in the generated appearance, resulting in poor stability. Until Zhou et al. proposed a style distribution module based on cross-attention, which calculates the similarity between the source image style and the target pose to complete pose transfer, the attention mechanism can capture global and local connections, positively impacting image realism. However, the weights are often distributed across a specific part of the image, ignoring less important information, leading to a still unrealistic generated image.

[0006] In conclusion, to better develop the fashion industry, virtual try-on, and fashion design, images urgently need to generate more refined textures and obtain more realistic images. Summary of the Invention

[0007] The purpose of this invention is to provide a style transformation method for fashion portrait images. By injecting texture style features from the encoder into the decoder through position normalization, and by proposing an energy distance loss function based on optimal transmission to constrain the consistency between the generated image distribution and the real image distribution, a more refined clothing texture feature can be generated.

[0008] To achieve the objectives of this invention, a method for style transformation of fashion portrait images is provided, comprising a training phase and an image generation phase.

[0009] The training phase includes the following steps:

[0010] (1) Collect data, collect image pairs of the same person with the same clothing but different poses, source image and target image (I s I t ); Collect the source image I s Extracted semantic segmentation map S s The target image I t Extracted key point map P t ;

[0011] (2) P key point map t The attitude feature F is generated from the input attitude encoder. p ; Source image I s and semantic segmentation graph S s Style features F generated in the input style encoder s The style encoder is based on the source image I s Generate bias μ and scale σ;

[0012] (3) Aligning style features F using cross-attention mechanism s and posture features F p The coarse alignment feature F is obtained. ps ;

[0013] (4) For coarse alignment feature F ps The target feature F is obtained by performing adaptive instance normalization AdaIN(·) processing. t ;

[0014] (5) For target feature F t Upsampling is performed to update the feature values, and the updated feature values ​​are then processed by PONO(·) position normalization to finally generate a fake target image.

[0015] (6) Pair real and fake target images The input is fed into the discriminator D to determine whether it is true or false;

[0016] (7) Calculate the loss function based on the generation results of generator G and discriminator D, perform iterative optimization, and complete the training process after reaching the maximum number of training iterations;

[0017] Image generation stage: The generator G trained in the training process is applied to the above steps (1)-(5) to generate the final output target image.

[0018] Furthermore, the loss function calculated in step (7) is: Generative Adversarial Loss Function L adv Reconstruction loss function L rec Perceptual loss function L perc Energy distance loss function L MED The energy distance loss function L MED The computational requirements are met by using the target image I t As a new source image I′ s , source image I s As the new target image I′ t S t P serves as a semantic segmentation map for the new source image. s As a new keypoint map, after steps (1)-(5), a new false target image is generated. Obtain new pairs of real and fake target images

[0019] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described fashion character image style transformation method.

[0020] Compared with the prior art, the significant progress of the present invention is as follows: (1) By combining adaptive instance normalization and position normalization, the source style is further injected into the target features, solving the problem of poor spatial transformation capability of convolution operation and obtaining more detailed texture features; (2) By using the energy distance loss function, the data distribution of the generated image can be constrained to be consistent with the distribution of the real image, so that the training process has a certain robustness and the generation of the image is more realistic.

[0021] To more clearly illustrate the functional characteristics and structural parameters of the present invention, further explanation is provided below in conjunction with the accompanying drawings and specific embodiments. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0023] Figure 1 This is the overall flowchart of the present invention.

[0024] Figure 2 This is a general framework diagram of the generator for the generative adversarial network proposed in this invention.

[0025] Figure 3 This is a framework diagram of the feature alignment module based on cross-attention proposed in this invention.

[0026] Figure 4 The figure shows the experimental results of attitude transfer according to the present invention.

[0027] Figure 5 This is an experimental result diagram of the clothing editing method of the present invention. Detailed Implementation

[0028] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] To verify the effectiveness of the proposed solution, the dataset used was DeepFashion's Inshop ClothesRetrieval Benchmark, which contains 52,712 fashion clothing images with a resolution of 256×256, along with corresponding pose keypoint maps and semantic segmentation maps. Before training, all images were cropped to a resolution of 256×176, and the dataset was divided into a training set and a test set, consisting of 101,966 and 8,570 image pairs (source and target image pairs), respectively.

[0030] like Figure 2 As shown, a method for style transformation of fashion portrait images includes the following specific steps:

[0031] (1) Collect data, given image pairs of models wearing the same clothing but in different poses, source image and target image pairs (I s I t ); For source image I s Extract its 8-channel semantic segmentation map S s For target image I t Extract its 18-channel key point map P t As input to the model;

[0032] (2) Figure 2 As shown, the key point map P t The input is fed into the attitude encoder to generate attitude features F p , source image I s and S s The input is fed into the style encoder to generate style features F. s The style encoder is based on the source image I s Generate the bias μ and scale σ for later use;

[0033] (2.1) Features learned by the posture encoder:

[0034] Human key point map P of the target image t Perform downsampling twice, then feed it into a convolutional layer to obtain the pose feature F. p ;

[0035] (2.2) Style encoder learns features:

[0036] First, the source image I s The partial style features F are fed into the encoder of the trained VGG-19 model to obtain the output of its convolutional layers. s1 Simultaneously, for the source image I s Two downsampling operations are performed, and position normalization is performed after each downsampling to obtain the style feature F. s2 And simultaneously obtain the mean μ and standard deviation σ of the current feature. Finally, F s1 and F s2 Connecting them together yields the complete stylistic feature F s .

[0037] (2.3) Reconstruction style characteristics:

[0038] According to source image I s The corresponding semantic segmentation graph S s Regarding style feature F s Perform linear interpolation, according to S s Extract the style features of each channel and concatenate them together to obtain the final style feature F. s .

[0039] (3) Figure 3 As shown, firstly, based on the attitude feature F p Calculate query Q, based on style feature F s Calculate the key K and value V. Next, calculate the similarity between Q and K to obtain the attention matrix AM. Then, align the pose feature F using AM and V. p and style characteristics F s Finally, the coarse alignment feature F is obtained through the feedforward network FFN(·). ps:

[0040]

[0041] T represents the matrix transpose operation, and C represents the number of channels for the feature.

[0042] (4) For coarse alignment feature F ps Adaptive instance normalization AdaIN(·) is performed, consisting of four residual blocks. First, the style features F obtained from the style encoder are processed... s The features are fed into a multilayer perceptron to calculate the style features required for the four residual blocks, and then input into each residual block; the initial features are input into the i-th residual block. Based on the style characteristics assigned to each piece To perform adaptive instance normalization, we obtain the style features of the i-th residual block.

[0043]

[0044] Let be the style feature of the i-th residual block, σ(·) be the standard deviation of the input vector, and μ(·) be the mean of the input vector. To incorporate style features F s The i-th style feature is obtained by dividing the residual blocks equally among the numbers in the multilayer perceptron.

[0045] After four rounds of adaptive instance normalization, the target feature F is obtained. t .

[0046] (5) The target feature F obtained in step (4) t First, upsampling is performed to update the feature values, and then position normalization PONO(·) is performed to obtain the j-th result.

[0047]

[0048] j represents the j-th processing step in position normalization. According to Processed target feature value, μ j σ is the bias for the j-th position normalization. j Let be the scale for the j-th position normalization.

[0049] After two rounds of upsampling and position normalization, updated target feature values ​​are obtained, and finally, a fake target image is generated.

[0050] (6) Pair real and fake target images The input is fed into the discriminator D to determine its authenticity.

[0051] (7) Calculate the overall loss function L based on the results of generator G and discriminator D. total :

[0052] L total =λ adv L adv +λ rec L rec +λ perc L perc +λ MED L MED

[0053] Among them, L total It includes the following four parts: generating adversarial loss L adv Reconstruction loss function L rec Perceptual loss function L perc and energy distance loss L MED λ represents the weight of the corresponding loss function. adv =5,λ rec =1,λ perc =1,λ MED =1.

[0054] Generate adversarial loss L adv It is divided into two parts: a generator G and a discriminator D, and is defined as follows:

[0055]

[0056]

[0057] Reconstruction loss function L rec Calculate the difference between false and true target images using L1 distance:

[0058]

[0059] Perceived loss L perc Extracting real images using a pre-trained VGG-19 network I t The features, and then compared with the false target image. Feature comparison:

[0060]

[0061] k is the number of layers in the VGG-19 network, Φ k (·) represents the features extracted from the k-th layer of the VGG-19 network, where k = 3_2, 4_2.

[0062] Energy distance loss L MED The calculation requires generating a new set of real and fake target images. The target image I is input into the generator G. t As a new source image I′s Source Image I s As the new target image I′ t S t P serves as a semantic segmentation map for the new source image. s As a new keypoint map, after steps (1)-(5), a new false target image is generated. Based on two pairs of real and fake target images and Calculate the energy distance loss function L MED :

[0063]

[0064] E(·) is the maximum likelihood estimate, W M (·,·) represent I respectively s and I s and I t , and The Wasserstein distance between them is calculated using parameters one and two.

[0065] Next, the training process is iteratively optimized based on the calculation results until the maximum number of training iterations is reached, thus completing the entire training process.

[0066] After completing the entire training process, steps (1) to (5) are sufficient to generate the person image for the pose transfer task. The generated result is as follows: Figure 4 As shown, the first column is the source image, and starting from the second column: the first row is the human body key point image, the second row is the fake target image, and the third row is the real target image. This demonstrates that the proposed solution can effectively preserve clothing details and achieve pose transfer.

[0067] In addition to pose transfer, this invention can replace the texture and shape of clothing in a source image based on a reference image. Clothing editing can be achieved using only the previously trained model, and the generated result is as follows. Figure 5 As shown. The difference between this and the pose transfer steps (1) to (5) is that an additional reference image is required. In step two, the style features of the reference image are extracted simultaneously, and the features of the specified channels are assigned to the corresponding channels of the style features of the source image. In the figure, the first column is the source image, and starting from the second column: the first row is the reference image, the second row is the generated image with the texture of the pants referenced in the first two columns and the texture of the shirt referenced in the last two columns, and the third row is the generated image with the shape of the shirt changed according to the shirt in the target reference image. It can be seen that this scheme can better preserve the texture and shape of the clothing and effectively perform style transformation.

[0068] The specific embodiments of the present invention have been described in detail above. It should be noted that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for style transformation of fashion portrait images, comprising a training phase and an image generation phase, characterized in that, The training phase includes the following steps: (1) Collect data: collect image pairs of the same person with the same clothing but different poses, including source image and target image. Collect the source images Extracted semantic segmentation map The target image Extracted key point map ; (2) Transform the key point diagram Generate pose features from input pose encoder ; Source image and semantic segmentation graph Generating style features in the input style encoder The style encoder is based on the source image Generate bias μ and scale σ; (3) Aligning style features using cross-attention mechanism and posture features To obtain coarse alignment features ; (4) For coarse alignment features Adaptive instance normalization is performed to obtain target features. ; In step (4), adaptive instance normalization AdaIN( The processing is as follows: ； i = 1, 2, ..., n; Let i be the style feature of the i-th residual block. Input the initial features for the i-th residual block, σ( ) represents the standard deviation of the input vector, μ( () represents the mean of the input vector. To incorporate stylistic features The i-th style feature is obtained by dividing the residual block into the multilayer perceptron; (5) Target features Upsampling is performed to update feature values, and the updated feature values ​​are then normalized to their positions, ultimately generating a fake target image. ; The position normalization PONO in step (5) The processing is as follows: ； j = 1, 2, ..., m; j represents the j-th processing step in position normalization. According to Processed target feature values, The bias for the j-th position normalization is... Let j be the scale for the position normalization. (6) Pair real and fake target images The input is fed into the discriminator D to determine whether it is true or false; (7) Calculate the loss function based on the generation results of generator G and discriminator D, perform iterative optimization, and complete the training process after reaching the maximum number of training iterations; Image generation stage: The trained model generator G is applied to the above steps (1)-(5) to generate the final output target image.

2. The method for style transformation of fashion portrait images according to claim 1, characterized in that, The cross-attention mechanism in step (3) is as follows: using the feedforward network function FFN( The attention matrix AM and the value V yield coarse alignment features. , ； Q is based on posture characteristics Calculate queries Q, K, and V based on style features. Calculate the key K and value V, where T is the matrix transpose operation and C is the number of feature channels.

3. The method for changing the style of fashion portrait images according to claim 1, characterized in that, In step (4), n=4.

4. The method for style transformation of fashion portrait images according to claim 1, characterized in that, In step (5), m=2.

5. The method for style transformation of fashion portrait images according to claim 1, characterized in that, The loss function calculated in step (7) is: Generative Adversarial Loss Function Reconstruction loss function Perceptual loss function Energy distance loss function ; where the energy distance loss function The computational requirements are met by using the target image. As a new source image , source image As a new target image , As a semantic segmentation map of the new source image As a new keypoint map, after steps (1)-(5), a new false target image is generated. To obtain new pairs of real and fake target images .

6. The method for style transformation of fashion portrait images according to claim 5, characterized in that, The energy distance loss function for: ； E(·) is the maximum likelihood estimate, W M (·,·) represents the Wasserstein distance between parameter one and parameter two.

7. The method for style transformation of fashion portrait images according to claim 1, characterized in that, The dataset uses DeepFashion's Inshop Clothes Retrieval Benchmark. Before training, all images in the dataset were cropped to a resolution of 256×176, and the dataset was divided into training and test sets, with 101,966 and 8,570 pairs of images, respectively.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-7.

Images