A virtual fitting method and device based on image generation

Abstract

The present invention discloses a virtual fitting method and device based on image generation, the method comprising: taking dressed human images and standard clothing images as inputs for virtual fitting; constructing a data set for the dressed human images in combination with a parameterized human body model of a semantic body model, and using a conditional generation network to deform the two-dimensional human body according to the amount of body size change; constructing a clothing image segmentation data set for the standard clothing images based on human posture, parsing the clothing images based on posture, and deforming the clothing images in segments; fusing human body expressions unrelated to clothing and the deformed clothing images, and training a virtual fitting network to generate more realistic fitting images. The device comprises: a processor and a memory. The present invention constructs a data set for dressed human images in combination with a parameterized human body model of a semantic body model, and uses a conditional generation network to deform the two-dimensional human body according to the amount of body size change, thereby alleviating the problems of insufficient body diversity and uneven distribution in existing fitting image data sets.

Description

Technical Field

[0001] The present invention relates to the field of image generation, and in particular to a virtual fitting method and device based on image generation. Background Art

[0002] In recent years, e-commerce has rapidly grown, with an increasing number of people participating in online shopping. Apparel, due to its wide product differentiation and ease of transportation, leads the online shopping market in terms of transaction volume. However, it also has an extremely high return rate, primarily due to poor fit and unsatisfactory fit. Virtual fitting technology can not only provide size recommendations but also visually demonstrate the effects of wearing the garment, significantly enhancing the user experience and immersion in online clothing shopping, thereby reducing the return rate and the resulting financial losses.

[0003] Image-generated virtual fitting method, in terms of human body expression that is not related to clothes, there are two existing works on generating fitting image results. [1][2] The human body is represented as posture, figure, face and hair. They are based on reference [3] Extract human posture points and use references [4] The face, hair, clothing, skin and other parts are parsed from the dressed human body image. However, this kind of body representation cannot completely get rid of the influence of the original clothes. In terms of posture-guided clothing image deformation, Han et al. [1] With Wang et al. [2] All methods use thin plate spline functions to characterize clothing image deformation. However, they treat clothing as a whole and perform deformation in a single image plane (or surface), failing to characterize clothing deformation caused by rigid body pose changes. Existing fitting image datasets suffer from insufficient body diversity and uneven distribution, as models on shopping websites are mostly slender.

[0004] In summary, there are still some problems that need to be solved in the virtual fitting method based on image generation:

[0005] 1) Enhancing the body shape diversity of dressed human image datasets; 2) Human pose-guided clothing image deformation methods; 3) Construction of a virtual fitting system based on image generation. Summary of the Invention

[0006] The present invention provides a virtual fitting method and device based on image generation. The present invention uses a semantic body model to construct a data set and learns the relationship between size changes and two-dimensional body shape changes based on the data set, thereby efficiently changing the body shape in the clothing image. The clothing image is analyzed based on human posture, and each clothing segment is individually deformed so that the clothing deformation adapts to the rigid posture changes of the human body, thereby more realistically simulating the effect of trying on clothes. The present invention designs an effective virtual fitting network to integrate human body information unrelated to the user's original clothing and the deformed target clothing information, thereby generating a realistic fitting image, as described below:

[0007] A virtual fitting method based on image generation, the method comprising:

[0008] Use dressed human body images and standard clothing images as input for virtual fitting;

[0009] For images of clothed human bodies, a dataset is constructed by combining a parameterized human body model with a semantic body model, and a conditional generative network is used to deform the two-dimensional human body according to the change in body size.

[0010] For standard clothing images, we build a clothing image segmentation dataset for human posture, analyze clothing images based on posture, and deform clothing images in slices;

[0011] By fusing clothing-independent human body expressions and deformed clothing images, a virtual fitting network with more realistic images is generated.

[0012] The semantic body shape model uses the offset of the vertices of the 3D human body mesh relative to the corresponding vertices of the template mesh to describe the body shape changes. The calculation is as follows:

[0013]

[0014] Among them, Q is the number of human body parts divided according to the body shape, To explain the semantic basis of the length change of the qth human body part, To explain the semantic basis of the change in the circumference of the qth human body part, S u To explain the non-semantic basis of the changes in the remaining body size of the whole human body, are the corresponding human body parameters respectively; training samples are generated using the semantic parameterized 3D human body model.

[0015] The parameterized human body model constructed by combining the semantic body model is:

[0016] For each 3D human body in the database, a body shape change vector δS is given, and the corresponding 3D human body is generated. The 3D human body is projected to obtain the target 2D human body segmentation IS and I t ;

[0017] Based on the training samples (I S ,δS,I t ), train a conditional generative adversarial network to segment the target two-dimensional human body according to I S and body shape change vector δS to generate I t , the input of the network is I S and δS, the output is I t .

[0018] Furthermore, for the standard clothing image, a clothing image segmentation dataset for human posture is constructed, clothing images are analyzed for posture, and clothing images are deformed into pieces:

[0019] Analyze the standard clothes, segment the clothes into different parts, and then perform rigid rotation and stretching on the analyzed clothes according to the user's posture and body shape, so that the clothes are close to the user's body shape and posture;

[0020] The parsed clothing is sliced and fed into the clothing deformation network along with the user information. The clothing deformation network then outputs the generated image. The L1 loss is calculated between the generated image and the ground truth image to constrain the network.

[0021] We input a standard clothing image, automatically parse out the parts of the clothing related to the torso and limbs with the help of human key point detection, use semi-automatic annotation to annotate the clothing image, and construct the first clothing image dataset with semantic labeling for human posture. We use a user-friendly interface to interactively draw segmentation lines on the clothing image, and automatically label the clothing image by calculating the connected domain. Based on the constructed dataset, we use the image parsing framework to learn clothing image parsing guided by human posture.

[0022] Among them, the clothing deformation network is:

[0023] The training set consists of image pairs collected from shopping websites, including photos of standard clothing and photos of models wearing the clothing. In the preprocessing stage, the key points corresponding to the standard clothing images, the key points corresponding to the images of the people wearing the clothing, the two-dimensional body shape of the people wearing the clothing images, and the clothing parts segmented by parsing the photos of the people wearing the clothing.

[0024] During the training phase, clothing images are segmented into regions corresponding to human postures through a clothing parsing network. Rigid rotations and shortening / lengthening operations are performed on each part according to the target posture. The parameters of the thin plate spline function corresponding to each piece of clothing are learned based on the human body expression and clothing segmentation.

[0025] Furthermore, the clothing parsing network consists of four parts:

[0026] Two encoding networks are used to extract high-level features; a correlation layer integrates the two features into a tensor as the input of the next regression network; a regression network for predicting parameters; a TPS transformation module for transforming the image into output Where c represents the clothing segment, Represents the generated image after TPS transformation.

[0027] Using the sample triples (p, c, c t ) for training, p represents the user’s posture and body expression, c t Represents the Ground Truth image;

[0028]

[0029] The loss function of the virtual fitting network is expressed as:

[0030]

[0031]

[0032] in, Represents the deformed clothing image; represents the final result; I′ represents the rendering result; M represents the combined mask; α L1 Indicates the contribution value of L1 loss; α V Represents the contribution value of VGG loss; α i Indicates the contribution of the i-th layer to the loss in the network; α M It represents the contribution of the mask part loss.

[0033] A virtual fitting device based on image generation, comprising: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute the method steps described.

[0034] The beneficial effects of the technical solution provided by the present invention are:

[0035] 1. This paper constructs a dataset for images of dressed human bodies by combining a parameterized human body model with a semantic body model, and uses a conditional generative network to deform the two-dimensional human body according to the change in body size. This alleviates the problems of insufficient body diversity and uneven distribution in existing fitting image datasets.

[0036] 2. This paper constructs a clothing image segmentation dataset for human posture, analyzes clothing images based on posture, and deforms clothing images in slices, which helps to depict clothing deformation caused by rigid posture changes of the human body;

[0037] 3. The present invention integrates human body expressions unrelated to clothes and deformed clothing images, which helps to free the body expression from the influence of original clothes and generate more realistic virtual fitting images. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flowchart of a virtual fitting method based on image generation;

[0039] Figure 2 This is the overall flow chart of the virtual fitting method based on image generation;

[0040] Figure 3 Analytical diagram of the human body;

[0041] Figure 4 Flowchart of the method for reshaping the human body in an image;

[0042] Figure 5 Figure 1. Network structure of clothing image deformation guided by human posture.

[0043] Figure 6 Semi-automatic clothing image labeling.

[0044] Figure 7 This is the network structure diagram of virtual fitting based on image generation. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0046] Example 1

[0047] A virtual fitting method based on image generation, see Figure 1 , the method comprises the following steps:

[0048] 101: Using the dressed human body image (i.e., user photo) and the standard clothing image as input for virtual fitting;

[0049] 102: For images of clothed human bodies, a parameterized human body model based on a semantic body model is used to construct a dataset, and a conditional generative network is used to deform the 2D human body according to the change in body size.

[0050] Among them, the semantic body model uses the offset of the vertices of the three-dimensional human body mesh relative to the corresponding vertices of the template mesh to characterize body shape changes; the parametric human body model is the SMPL parametric human body model, which combines the semantic body model with the SMPL parametric human body model to construct a parametric human body model with body shape semantics.

[0051] The constructed data set is used to train the conditional generative network, and the deformation of the clothing image is guided by the deformed body shape, which will not be described in detail in the embodiment of the present invention.

[0052] 103: For standard clothing images, build a clothing image segmentation dataset for human posture, analyze clothing images based on posture, and deform clothing images in slices;

[0053] The human pose guided clothing image deformation network is trained by constructing a clothing image segmentation dataset to make it practical.

[0054] 104: Fusion of clothing-independent human body expressions and deformed clothing images to generate a virtual fitting network with more realistic images.

[0055] By integrating clothing-independent body expressions, the generated fitting images are unaffected by the user's original clothing. Clothing-independent body expressions are extracted from the user's photos. A virtual fitting network takes the user's body expressions and a posture-distorted clothing image as input to generate fitting images. The goal is to design this network so that the generated images retain both the user's original posture and figure, as well as the clothing's color and texture information, ultimately outputting a fitting image of the user in their current posture.

[0056] In summary, the embodiment of the present invention implements virtual fitting through steps 101 to 104 , thereby enhancing the realism of the virtual fitting.

[0057] Example 2

[0058] The solution in Example 1 is further introduced below with reference to specific examples and calculation formulas.

[0059] 201: Input the user photo and the standard clothing image, and perform subsequent operations based on them;

[0060] The virtual fitting method based on image generation, the overall process of the method is as follows Figure 2 The input of the whole method is the user's photo and clothing image, and the output is the user's fitting image in the current posture wearing the target clothing (the target clothing here refers to the input clothing image). The output fitting image must meet the following requirements:

[0061] (1) The user's appearance and hairstyle remain unchanged; (2) The user's posture remains unchanged; (3) The user's body shape remains unchanged; (4) The clothes fit the user more naturally.

[0062] 202: For images of clothed human bodies, a parameterized human body model based on a semantic body model is used to construct a dataset, and a conditional generative network is used to deform the two-dimensional human body according to the change in body size.

[0063] 1. User's appearance and hairstyle

[0064] For the user's appearance and hairstyle, a human body analysis model is used. [4] , to parse out the face and hairstyle of the input user image, as part of the input of the virtual fitting network, and identically map to the face and hairstyle in the final output image. Figure 3 [5] As shown in , the human body parsing model parses the human body into different parts (different colors represent different parts) and directly retains the human head (i.e., face and hairstyle), so that the user's head does not change before and after trying on clothes. That is, the face and hairstyle in the input user photo and the output virtual fitting result image are the same.

[0065] In order to keep the fitting results unaffected by the original clothes, it is necessary to extract the human body expressions that are unrelated to the original clothes from the user's photos (the human body posture, figure, face and hairstyle remain unchanged). [1][2] The human body is expressed as a face, hair, posture, and figure. Based on references [6] Extract human posture points and use references [4] The face, hair, clothing, skin and other parts are parsed from the clothed human image. The body shape is expressed as the body parts except the face and hair. In order to make it as independent of the original clothes as possible, the body shape segmentation is further downsampled to a low resolution and then enlarged back. Figure 3 For example, if we downsample the parsed human image to a low resolution and then zoom back in, the shapes of the collar and skirt won't change much. This still reflects the influence of the original clothing, preventing a clean representation of the human figure. Therefore, this representation cannot completely eliminate the influence of the original clothing.

[0066] In order to solve the above problems, the human body expression is not affected by the original clothes, using DensePose [7] Estimate the user's body shape under clothing. DensePose learns the dense correspondence between images and surface-based human representations. The result of DensePose is transferred to a 1-channel binary mask, where the number 1 represents the human body. At the same time, the body diversity enhancement operation is performed for body estimation. Specifically, the human body expression independent of clothing is divided into three parts: body shape, posture, face and hairstyle ( Figure 2 (as shown in ).

[0067] 2. Pose Estimation

[0068] The embodiment of the present invention adopts a posture estimation model [6]To estimate the 2D coordinates of 18 pose keypoints in the image, we used a pose heatmap with 18 channels, one for each keypoint, to ensure that each keypoint maintains its semantic location (e.g., shoulder, elbow, and wrist). To enhance the keypoint's effect, the 11 neighborhoods surrounding the keypoint are filled with 1s, while the remaining neighborhoods are filled with 0s. The resulting pose point map serves as input to the subsequent virtual fitting network, constraining it to generate human figures in the corresponding poses.

[0069] 3. Body size estimation

[0070] To ensure that the user's body shape remains unchanged before and after fitting (for example, when the input user photo is of a slim person, the output fitting result image still shows a slim person; when the input user photo is of an obese person, the output fitting result image still shows an obese person), an embodiment of the present invention performs a body shape diversity enhancement operation on the dressed human body image dataset. In existing datasets, the models' figures are mostly slim, so the generalization performance of the virtual fitting network trained with such datasets is weak. For example, when the user's figure is relatively slim, the output result obtained by the trained virtual fitting network is still a slim person, which meets the target requirements. However, when the input user's figure is relatively obese, the output may still be a slim person, which does not meet the preset target requirements. Therefore, it is necessary to enhance the figure of the dressed human body images and enrich the dataset to include data of various body shapes, such as height, weight, and thinness. In this way, the trained virtual fitting network performs better, ensures that the user's figure remains unchanged before and after input, and can obtain more realistic fitting effects.

[0071] Enhancing the body diversity of dressed human images is a key research topic in this embodiment of the present invention. Most models in fitting pictures on Internet stores are slim, which results in insufficient body diversity and uneven distribution in existing fitting image datasets. This embodiment of the present invention adopts a data-driven approach, such as Figure 4 As shown in FIG, first, a 2D human body region is estimated from the source image, and then a 2D human body after the body shape change is generated based on the body size change vector δS. Finally, a correspondence between the contour points of each part of the human body before and after the body shape change is established, and the source image is deformed based on the MLS (Moving Least Square) method according to the correspondence. For the 2D human body segmentation of the source image, the embodiment of the present invention adopts the method of Omran et al. [8] method.

[0072] The key issue here is how to change the body shape of the original two-dimensional human body according to the body size change. S(i.e. input human body image), body size change vector δS, target two-dimensional human body segmentation I t ), and train a conditional generative adversarial network (CGAN) based on the training samples [9] ).

[0073] The semantic body shape model uses the offset of the vertices of the 3D human body mesh relative to the corresponding vertices of the template mesh to describe the body shape changes. The calculation of the offset of all vertices relative to the corresponding vertices of the template mesh is as follows:

[0074]

[0075] Among them, Q is the number of human body parts divided according to the body shape, To explain the length variation of the qth human body part (semantic basis), S is the basis (semantic basis) for explaining the change in the circumference of the qth human body part. u A non-semantic basis is used to explain the remaining body shape variations of the entire human body. The body shape basis is obtained by principal component analysis of a 3D human body database containing different body shapes with the same posture. The semantic basis enables semantic control of the human body shape (for example, changing height and measurements), while the non-semantic basis ensures the coherence between body parts. They are the corresponding human body parameters, and the data set is used to learn the mapping relationship between body dimensions such as height and measurements and the corresponding human body parameters.

[0076] To increase the pose diversity of training samples, the semantic body model is combined with the SMPL (Skinned Multi-Person Linear Model) parameterized human body model to increase the pose diversity of training samples. By adjusting the model pose parameters and body size, 3D bodies with various poses and body shapes can be generated. For each 3D body in the database, a body shape change vector δS is given, and the corresponding 3D body can be generated. The 3D body is then projected to obtain a 2D body segmentation I S and I t Based on the training samples (I S ,δS,I t ), train a conditional generative adversarial network (CGAN [9] ), so that it is based on I S and δS generates I t (That is, the input of the network is I S and δS, the output is I t ). The objective function of the conditional generative adversarial network is as follows:

[0077]

[0078] Among them, G represents the generator, D represents the discriminator, and IS Represents the source 2D human segmentation as the input of the network CGAN, δS represents the body size change vector as conditional information, and z represents random noise. The network is trained so that the generator can generate the image according to I S And δS generate target data and obtain the target two-dimensional human segmentation I t .

[0079] 203: For standard clothing images, build a clothing image segmentation dataset for human posture, parse clothing images based on posture, and deform clothing images in pieces.

[0080] Clothing images provided by shopping websites often show neatly arranged clothing, often referred to as standard clothing images. To achieve a better fitting effect, standard clothing images need to be deformed before being synthesized with the user's body image. This embodiment of the present invention addresses the issue of poor clothing deformation by defining clothing portions associated with the torso and limbs on separate image planes (curved surfaces) for deformation, thereby enhancing the authenticity of fitting images.

[0081] The overall process is as follows Figure 5 As shown. First, the standard clothes are parsed and divided into different parts. Then, according to the user's posture and body shape, the clothes are subjected to rigid rotation and stretching, so that the clothes are as close to the user's body shape and posture as possible. Then, the parsed clothes are segmented and input into the clothes deformation network ( Figure 5 The dotted box in the middle represents the clothing deformation network. Finally, the clothing deformation network outputs the generated image. An L1 loss is calculated between the generated image and the GroundTruth image to constrain the network.

[0082] Regarding clothing analysis, we input a standard clothing image and automatically analyze the parts of the clothing related to the torso and limbs with the help of the human body key point detection method. We use a semi-automatic method to annotate clothing images and construct the first clothing image dataset with semantic labeling for human posture. Figure 6 As shown in the figure, a user-friendly interface is used to interactively draw segmentation lines on clothing images, and then the clothing images are automatically labeled by calculating connected components. Based on the constructed dataset, a common image parsing framework is used to learn human pose-guided clothing image parsing.

[0083] Human pose guided clothing image deformation network framework see Figure 5. The training set is an image pair collected from shopping websites (including: photos of standard clothes and photos of models wearing the clothes). In the preprocessing stage, the key points corresponding to the standard clothes images, the key points corresponding to the dressed human body images, the two-dimensional body shape of the dressed human body images, and the clothing parts (Ground Truth) segmented by parsing the dressed human body photos are prepared. In the training stage, the clothes image is first segmented into areas corresponding to the human body postures by the clothing parsing network, and then simple rigid rotation and shortening / lengthening operations are performed on each part according to the target posture. The rigid rotation formula is expressed as follows, where x and y represent the position coordinates of the original image data, x′ and y′ represent the position coordinates after transformation, and θ represents the rotation angle.

[0084]

[0085] Finally, the parameters of the Thin-Plate Spline (TPS) function corresponding to each piece of clothing are learned based on the human body expression and clothing segmentation. This part of the network consists of four parts: (1) two encoding networks to extract high-level features; (2) a correlation layer ( Figure 3 (shown as a diamond) integrates the two features into a tensor as the input of the next regression network; (3) the regression network used to predict parameters; (4) the TPS transformation module (abbreviated as T). It is a thin plate spline transformation module used to transform the image into an output Where c represents the clothing segment (corresponding to Figure 3 , Represents the generated image after TPS transformation (corresponding to Figure 3 The generated image on the right side of the figure). This method is end-to-end learnable and uses the sample triples (p,c,c t ) for training, p represents the user's posture and body expression (corresponding to Figure 3 User information in: posture + body shape), deformation result and c t The pixel-level loss function between (Ground Truth) is as follows, where c t Represents the Ground Truth image (corresponding to Figure 3 Ground Truth on the center right).

[0086]

[0087] 204: Fusion of clothing-independent human body expressions and deformed clothing images to generate a more realistic virtual fitting network;

[0088] Through the above steps, the embodiment of the present invention obtains the human body expression and the deformed clothing image that are not related to clothing. They are used as the input of the virtual fitting network. Then, the fitting network fuses the human body information and the clothing information consistent with the posture to generate the fitting image. Figure 7 .

[0089] Ideally, the training samples should be (I i ,c,I t ), where I i represents the input user image, c represents the clothing image, I t In reality, however, it is difficult to collect image pairs with the same posture and body shape, but only different clothes (I i ,I t ). Therefore, the training samples are organized as (p,c,I t ), where c represents the clothing image, I t represents a fitting image, and p is a human body expression that is unrelated to clothes.

[0090] First, from the dressed human image I t The human body expression p, which is independent of clothing, is extracted from the network. It consists of posture, body shape and head information. Secondly, the standard clothing image is deformed by the posture-guided clothing deformation network. Finally, the human body expression and the deformed clothing image are used as input to use Unet

[10] The rendering result and the combination mask are generated at the same time. The clothing image is combined with the rendering result according to the combination mask. The formula is as follows, where △ represents element-wise matrix multiplication.

[0091]

[0092] The loss function of the virtual fitting network is expressed as:

[0093]

[0094] in, Represents the deformed clothing image; represents the final result; I′ represents the rendering result; M represents the combined mask; α L1 Indicates the contribution value of L1 loss; α V Represents the contribution value of VGG loss; α i Indicates the contribution of the i-th layer to the loss in the network; α M It represents the contribution value of the mask part loss, where N1(.) represents the 1 norm.

[0095] In the formula, the first term Constrain the generated image to be as close to the Ground Truth as possible, the second term (α mask||1-M||1) is a regular term that constrains the generated image to utilize clothing image information as much as possible. The third term To generate the perceptual loss between the image and the Ground Truth, Ψ i (·) represents the visual perception network, which is based on ImageNet

[11] Pre-trained VGG19 network

[12] .

[0096] 205: Based on the trained virtual fitting network, the fitting image of the user in the current posture is output to meet various needs in practical applications.

[0097] A virtual fitting device based on image generation, the device comprising: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to cause the device to perform the following method steps:

[0098] Use dressed human body images and standard clothing images as input for virtual fitting;

[0099] For images of clothed human bodies, a dataset is constructed by combining a parameterized human body model with a semantic body model, and a conditional generative network is used to deform the two-dimensional human body according to the change in body size.

[0100] For standard clothing images, we build a clothing image segmentation dataset for human posture, analyze clothing images based on posture, and deform clothing images in slices;

[0101] By fusing clothing-independent human body expressions and deformed clothing images, a virtual fitting network with more realistic images is generated.

[0102] Among them, the semantic body shape model uses the offset of the 3D human body mesh vertices relative to the corresponding vertices of the template mesh to describe the body shape changes. The calculation is as follows:

[0103]

[0104] Among them, Q is the number of human body parts divided according to the body shape, To explain the semantic basis of the length change of the qth human body part, To explain the semantic basis of the change in the circumference of the qth human body part, S u To explain the non-semantic basis of the changes in the remaining body size of the whole human body, are the corresponding human body parameters respectively; training samples are generated using the semantic parameterized 3D human body model.

[0105] Among them, the parameterized human body model dataset combined with the semantic body model is constructed as follows:

[0106] For each 3D human body in the database, a body shape change vector δS is given, and the corresponding 3D human body is generated. The 3D human body is projected to obtain the target 2D human body segmentation I S and I t ;

[0107] Based on the training samples (I S ,δS,I t ), train a conditional generative adversarial network to segment the target two-dimensional human body according to I S and body shape change vector δS to generate I t , the input of the network is I S and δS, the output is I t .

[0108] Furthermore, for standard clothing images, a clothing image segmentation dataset for human posture is constructed, clothing images are analyzed based on posture, and clothing images are deformed into pieces:

[0109] Analyze the standard clothes, segment the clothes into different parts, and then perform rigid rotation and stretching on the analyzed clothes according to the user's posture and body shape, so that the clothes are close to the user's body shape and posture;

[0110] The parsed clothing is sliced and fed into the clothing deformation network along with the user information. The clothing deformation network then outputs the generated image. The L1 loss is calculated between the generated image and the ground truth image to constrain the network.

[0111] We input a standard clothing image, automatically parse out the parts of the clothing related to the torso and limbs with the help of human key point detection, use semi-automatic annotation to annotate the clothing image, and construct the first clothing image dataset with semantic labeling for human posture. We use a user-friendly interface to interactively draw segmentation lines on the clothing image, and automatically label the clothing image by calculating the connected domain. Based on the constructed dataset, we use the image parsing framework to learn clothing image parsing guided by human posture.

[0112] Among them, the clothing deformation network is:

[0113] The training set consists of image pairs collected from shopping websites, including photos of standard clothing and photos of models wearing the clothing. In the preprocessing stage, the key points corresponding to the standard clothing images, the key points corresponding to the images of the people wearing the clothing, the two-dimensional body shape of the people wearing the clothing images, and the clothing parts segmented by parsing the photos of the people wearing the clothing.

[0114] During the training phase, clothing images are segmented into regions corresponding to human postures through a clothing parsing network. Rigid rotations and shortening / lengthening operations are performed on each part according to the target posture. The parameters of the thin plate spline function corresponding to each piece of clothing are learned based on the human body expression and clothing segmentation.

[0115] Furthermore, the clothing parsing network consists of four parts:

[0116] Two encoding networks are used to extract high-level features; a correlation layer integrates the two features into a tensor as the input of the next regression network; a regression network for predicting parameters; a TPS transformation module for transforming the image into output Where c represents the clothing segment, Represents the generated image after TPS transformation.

[0117] Using the sample triples (p, c, c t ) for training, p represents the user’s posture and body expression, c t Represents the Ground Truth image;

[0118]

[0119] Among them, the loss function of the virtual fitting network is expressed as:

[0120]

[0121]

[0122] in, Represents the deformed clothing image; represents the final result; I′ represents the rendering result; M represents the combined mask; α L1 Indicates the contribution value of L1 loss; α V Represents the contribution value of VGG loss; α i Indicates the contribution of the i-th layer to the loss in the network; α M It represents the contribution of the mask part loss.

[0123] References:

[0124] [1]Han

[0125] [2]Wang B,Zheng H,Liang X,et al.Toward characteristic-preservingimage-based virtual try-on network[C] / / Proceedings of the European Conferenceon Computer Vision(ECCV).2018:589-604.

[0126] [3]Cao Z,Simon T,Wei S E,et al.Realtime multi-person 2d poseestimation using part affinity fields[C] / / Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition(CVPR).2017:7291-7299.

[0127] [4]Liang X,Gong K,Shen X,et al.Look into person:joint body parsing&pose estimation network and a new benchmark[J].IEEE Transactions on PatternAnalysis and Machine Intelligence(TPAMI),2018.

[0128] [5]Gong K,Liang X,Zhang D,et al.Look into person:Self-supervisedstructure-sensitive learning and a new benchmark for human parsing[C] / / Proceedings of the IEEE conference on computer vision and patternrecognition.2017:932-940.

[0129] [6]Cao Z,Simon T,Wei SE,Sheikh Y(2017)Realtime multi-person 2d poseestimation using part affinity fields.In:2017IEEE conference on computervision and pattern recognition(CVPR).IEEE,pp 1302–1310

[0130] [7]Neverova N,Alp Guler R,Kokkinos I(2018)Dense pose transfer.In:Proceedings of the European conference on computer vision(ECCV),pp 123–138

[0131] [8]Omran M,Lassner C,Pons-Moll G,et al.Neural body fitting:unifyingdeep learning and model based human pose and shape estimation[C] / / 2018International Conference on 3D Vision(3DV).2018:484-494.

[0132] [9]Mirza M,Osindero S.Conditional generative adversarial nets[J].arXiv preprint arXiv:1411.1784,2014.

[0133]

[10] Ronneberger O,Fischer P,Brox T.U-net:convolutional networks forbiomedical image segmentation[C] / / International Conference on Medical ImageComputing and Computer-assisted Intervention.Springer,Cham,2015:234-241

[0134]

[11] Deng J, Dong W, Socher R, et al.Imagenet: a large-scale hierarchicalimage database[C] / / Proceedings of the IEEE Conference on Computer Vision andPattern Recognition(CVPR).2009:248-255.

[0135]

[12] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[J].arXiv preprint arXiv:1409.1556,2014.

[0136] Unless otherwise specified, the embodiments of the present invention do not limit the models of the components. Any component that can perform the above functions may be used.

[0137] Those skilled in the art will understand that the accompanying drawings are only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0138] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A virtual fitting method based on image generation, characterized in that: The method comprises: Use dressed human body images and standard clothing images as input for virtual fitting; For images of clothed human bodies, a dataset is constructed by combining a parameterized human body model with a semantic body model, and a conditional generative network is used to deform the two-dimensional human body according to the change in body size. For standard clothing images, we build a clothing image segmentation dataset for human posture, analyze clothing images based on posture, and deform clothing images in slices; By fusing clothing-independent human expressions and deformed clothing images, a virtual fitting network with more realistic images is generated. For each 3D human figure in the database, a body shape change vector is given and a corresponding 3D human figure is generated. Standard clothing is parsed and segmented into its parts, and then the parsed clothing is rigidly rotated and stretched according to the user's posture and body shape. The parsed clothes are segmented and input into the clothes deformation network together with the user information. Finally, the clothes deformation network outputs the generated image.

2. The virtual fitting method based on image generation according to claim 1, characterized in that: The semantic body shape model uses the offset of the vertices of the 3D human body mesh relative to the corresponding vertices of the template mesh to describe the body shape changes. The calculation is as follows: ； Among them, Q is the number of human body parts divided according to the body shape, To explain the semantic basis of the length change of the qth human body part, To explain the semantic basis of the change in the circumference of the qth human body part, To explain the non-semantic basis of the changes in the remaining body size of the whole human body, are the corresponding human body parameters respectively; training samples are generated using the semantic parameterized 3D human body model.

3. The virtual fitting method based on image generation according to claim 2, characterized in that: The parameterized human body model constructed by combining the semantic body model is: Project the 3D human body to obtain the target 2D human body segmentation and ; Based on training samples , train a conditional generative adversarial network to segment the target two-dimensional human body and body shape change vector generate , the input of the network is and , the output is .

4. The virtual fitting method based on image generation according to claim 1, characterized in that: For standard clothing images, a clothing image segmentation dataset for human posture is constructed, clothing images are analyzed based on postures, and clothing images are deformed into pieces: The L1 loss is calculated between the image and the ground truth image to constrain the network; We input a standard clothing image, automatically parse out the parts of the clothing related to the torso and limbs with the help of human key point detection, use semi-automatic annotation of the clothing image, and construct the first clothing image dataset with semantic labeling for human posture. We use a user-friendly interface to interactively draw segmentation lines on the clothing image, and automatically label the clothing image by calculating the connected domain. Based on the constructed dataset, we use the image parsing framework to learn clothing image parsing guided by human posture.

5. The virtual fitting method based on image generation according to claim 4, characterized in that: The clothing deformation network is: The training set consists of image pairs collected from shopping websites, including photos of standard clothing and photos of models wearing the clothing. In the preprocessing stage, the key points corresponding to the standard clothing images, the key points corresponding to the images of the people wearing the clothing, the two-dimensional body shape of the people wearing the clothing images, and the clothing parts segmented by parsing the photos of the people wearing the clothing. During the training phase, clothing images are segmented into regions corresponding to human postures through a clothing parsing network. Rigid rotations and shortening / lengthening operations are performed on each part according to the target posture. The parameters of the thin plate spline function corresponding to each piece of clothing are learned based on the human body expression and clothing segmentation.

6. The virtual fitting method based on image generation according to claim 5, characterized in that: The clothing parsing network consists of four parts: Two encoding networks are used to extract high-level features; a correlation layer integrates the two features into a tensor as input to the next regression network; a regression network for predicting parameters; and a TPS transformation module for transforming the image into output. , where c represents the clothing segment, Represents the generated image after TPS transformation; Using the sample triples (p, c, ) for training, p represents the user’s posture and body shape expression, Represents the GroundTruth image; 。 7. The virtual fitting method based on image generation according to claim 1, characterized in that: The loss function of the virtual fitting network is expressed as: ； ； in, Represents the deformed clothing image; Indicates the final result; Represents the rendering result; Represents a combination mask; Represents the contribution value of VGG loss; Represents the contribution of the i-th layer in the network to the loss; Indicates the contribution of the mask part loss.

8. A virtual fitting device based on image generation, characterized in that: The device includes: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute the method steps according to any one of claims 1 to 7.

Images