A virtual fitting figure generation method with controllable clothing length

Abstract

The application discloses a kind of virtual fitting chart generation methods of controllable clothing length, it is related to virtual fitting technical field.The application method mainly includes: the picture needed in input preprocessing stage is preprocessed to remove clothing information, and the clothing unknown person representation and clothing length control vector are obtained by calculating clothing length control vector;The picture needed in input semantic prediction stage is generated by trying on semantic segmentation map generator to generate the trying on semantic segmentation map to be adjusted;The picture needed in input semantic compensation stage is pruned and compensated by compensation network based on true value to the trying on semantic segmentation map to be adjusted, and trying on semantic segmentation map is obtained;The picture needed in input clothing deformation stage is generated by clothing alignment generator to predict clothing morphing map;The picture needed in input trying on chart generation stage is generated by trying on generator to generate fitting image.The application method proposes new clothing unknown person representation and clothing length control vector, and realizes controllable clothing length virtual trying on.

Description

Technical Field

[0001] This invention relates to the field of virtual fitting technology, and more specifically, to a method for generating virtual fitting images with controllable garment length. Background Technology

[0002] Virtual image generation technology has developed rapidly in recent years, making virtual try-on technology possible. Image-based virtual try-on methods replace the clothing on the target person with clothing to try on, allowing users to directly see how the clothes would look on them, thereby improving the user's online clothing shopping experience and reducing return rates.

[0003] Virtual try-on can be achieved using both 3D and 2D technologies. 3D-based virtual try-on technology mainly involves modeling and rendering. Both human and clothing modeling require collecting vast amounts of 3D data, and both modeling and rendering require expensive equipment. Therefore, attention has turned to 2D-based virtual try-on technology, which essentially involves virtual image generation.

[0004] Virtual try-on based on 2D images is an image generation task that, given a single image of clothing, replaces one person's clothing with another. In practice, virtual try-on is achieved through image synthesis. Given an image of a person and an image of clothing, the synthesized image should meet the following criteria: (1) the person's body parts and posture are identical to the original image; (2) the clothing naturally deforms according to the person's posture and body shape; (3) the clothing pattern in the try-on result is clearly visible, including low-level features such as color and texture, as well as complex graphics such as embroidery and logos. Virtual try-on presents unique challenges. The non-rigid nature of clothing is often affected by deformation and occlusion. Generating a try-on image that satisfies correct posture, good clothing deformation effect, clear pattern, and reasonable and realistic imagery without three-dimensional information is a challenge.

[0005] GANs are among the most popular image synthesis and generation models, and they perform well in tasks such as image generation and image editing. To incorporate desired attributes into the generated samples, researchers have also utilized different signals, such as class labels, text, and attributes, as conditions for image generation.

[0006] Previous virtual try-on technologies often focused on the quality of the generated try-on images while neglecting the study of clothing length. In fact, different people have different dressing habits; some people like to wear clothes tucked into their pants, while others prefer to wear them outside. This can be visually represented by the length of the clothing worn. Previous virtual try-on technologies could only produce one effect: preserving the appearance of the clothing in the original image, without providing more options. Summary of the Invention

[0007] The purpose of this invention is to provide a method for generating virtual try-on images with controllable clothing length. Users can input different garments and different clothing length control information to obtain images of different garments and different ways of wearing them. This method has the ability to learn quickly via the network and achieve significant try-on effects, and can be used in a wide range of complex virtual try-on scenarios.

[0008] To achieve the aforementioned objectives, the specific technical solution adopted is as follows:

[0009] A method for generating virtual fitting images with controllable garment length includes the following steps:

[0010] S1: Input the original image of the target person wearing the original clothing, as well as the semantic segmentation map, density pose map and skeleton pose map corresponding to the original image. After preprocessing, remove the clothing information of the original image, and obtain the image of the person with unknown clothing and the semantic segmentation map of the person with unknown clothing based on the key points of the skeleton pose map.

[0011] S2: Input the clothing image to be tried on, the mask of the clothing image, the density pose map, the skeleton pose map, the clothing unknown character semantic segmentation map, and the clothing length control vector, and generate the try-on semantic segmentation map to be adjusted through a pre-trained end-to-end try-on semantic segmentation map generator.

[0012] S3: Input the semantic segmentation map of the original image and the semantic segmentation map of the try-on image to be adjusted. Then, use a ground truth-based compensation network to trim and compensate the semantic segmentation map of the try-on image to be adjusted to obtain the semantic segmentation map of the try-on image.

[0013] S4: Input the clothing image to be tried on, the mask of the clothing image, the density pose map, the skeleton pose map and the try-on semantic segmentation map, and obtain the clothing deformation map through a pre-trained end-to-end clothing alignment generator.

[0014] S5: Input the unknown character image of the clothing, the density pose image, the skeleton pose image, the clothing deformation image, and the try-on semantic segmentation image, and generate the final output try-on image through a pre-trained end-to-end try-on generator.

[0015] Preferably, in step S1, the character image with unknown clothing and the character semantic segmentation image with unknown clothing are the processed original image and its semantic segmentation image that do not include the top, part of the pants, and the arms, respectively. The method of obtaining them is as follows: obtain the positions of each key point from the skeleton pose image and map them onto the original image; connect the key points of the torso, arms, and hips on the original image to form a mask area covering the arms, top, and part of the pants but excluding the palm; delete the image within the mask area in the original image and its semantic segmentation image to achieve the purpose of deleting clothing information, and finally obtain the character image with unknown clothing and the character semantic segmentation image with unknown clothing.

[0016] Preferably, in step S2, the garment length control vector is a discrete vector with one dimension, used to represent the garment length;

[0017] During the training phase of the try-on semantic segmentation graph generator, the garment length control vector is calculated by standardizing the coordinates of three key points: the bottom edge of the garment, the neck, and the mid-hip. The calculation method is as follows:

[0018] The neck and mid-hip keypoints in the skeleton pose map are mapped onto the semantic segmentation map to obtain the neck keypoint P in the semantic segmentation map. n (x n y n ) and the key point P in the middle of the buttock h (x h y h ); Connecting the key point P of the neck n and the key point of the middle buttock P h Construct a straight line and iterate through the pixels along the line in an orderly manner, determining the semantic type of each pixel in semantic segmentation. Pixels on the line whose values ​​change from representing clothing to representing non-clothing are designated as keypoints P. c (x c y c ), the key point P c This represents the intersection of the straight line and the lower edge of the clothing in the semantic segmentation graph; determined by P... c and P h The difference between the ordinates and P h and P n The ratio of the differences in the ordinates is used to calculate the garment length control vector z. The specific calculation formula is as follows:

[0019]

[0020] The value of z represents the length of the clothing relative to the body. When z < 0, the clothing style or wearing effect in the image is a short garment; when z > 0, the clothing style or wearing effect in the image is a long garment.

[0021] Preferably, in step S2, during the inference phase of the try-on semantic segmentation graph generator, the clothing length control vector is either directly input by the user or obtained using the above calculation method.

[0022] Preferably, in step S2, the try-on semantic segmentation graph generator is an end-to-end network with an encoding-decoding structure. The network input is divided into three groups: the first group of inputs is the clothing image Ic to be tried on and the clothing image mask Mc; the second group of inputs is the person semantic segmentation graph P for clothing with unknown characteristics. agnostic The third set of inputs is the density attitude map I. densepose Skeleton posture diagram I pose The mapping result obtained after passing the clothing length control vector z through a fully connected layer is used to concatenate the three inputs and then pass them through the first feature extractor R1, the second feature extractor R2, and the third feature extractor R3 respectively. The three feature extractors downsample and extract shallow and deepest features. The deepest features F1, F2, and F3 extracted by the three feature extractors R1, R2, and R3 are then concatenated and input into the feature upsampling fusion module. This feature upsampling fusion module contains multiple upsampling layers for feature upsampling fusion. The first layer... The sampling layer has three cascaded feature extractors at the front end. In the feature upsampling and fusion module, each upsampling layer receives part of its input from the three deepest features F1, F2, and F3 output from the cascaded network layer at the front end, and the other part from the shallow features of the three feature extractors. The deepest features F1 and F3 are upsampled and then fused with shallow features of the same scale from the same feature extractor, resulting in two fused features F′1 and F′3. Then, the deepest feature F2 is upsampled to a shallow feature F′1 of the same scale from the same feature extractor R2. l 2. Using the same scale, F′2 is obtained. Then, F′2 and F... l 2. After concatenating F′1 and F′3, the feature F″2 is obtained by passing it through a convolutional layer. Finally, the features F′1, F″2, and F′3 obtained in the current upsampling layer are used as the inputs to the deepest features F1, F2, and F3 of the next layer. The deepest features output by the feature upsampling fusion module and the three sets of inputs from the try-on semantic segmentation map generator network are input into the normalization layer. After concatenation, the layers are reduced in dimensionality by a convolution, and the output is a try-on semantic segmentation map of the same size as the network input.

[0023] Preferably, in step S3, the ground truth-based compensation network performs trimming compensation on the semantic segmentation map of the try-on to be adjusted, specifically including two steps: trimming and compensation. First, the semantic segmentation map of the try-on to be adjusted needs to be trimmed based on the original image to remove the redundant pants parts in the semantic segmentation map of the try-on to be adjusted, and the removed area forms a semantic-free region. Then, the value of the background region is used to assign a value to the semantic-free region formed in the trimming step, so that the semantic segmentation map of the try-on to be adjusted is restored to completeness.

[0024] Preferably, in step S3, the input to the truth-based compensation network is the semantic segmentation map P of the original image. parse and the semantic segmentation graph of the try-on to be adjusted The specific processing flow within the network is as follows:

[0025] First, from the semantic segmentation map P of the original image... parse Extracting the pants channel M bottom_o Then, from the semantic segmentation map of the trial fitting that needs adjustment... Extracting the pants channel M bottom , put the pants channel M bottom With M bottom_o Multiply to get M′ bottom =M bottom ⊙M bottom_o ;

[0026] Then, from the semantic segmentation map of the try-on to be adjusted Extracting the background channel M background , background channel M background With (1-M) bottom )⊙M bottom_o Adding them together gives M′ background =M background +(1-M bottom )⊙M bottom_o ;

[0027] Finally, use M′ bottom and M′ background Replace the semantic segmentation graphs of the try-on section that need adjustment respectively. The pants channel and background channel are used to obtain the fitting semantic segmentation map P after trimming and compensation. tryon And it serves as the output of the compensation network.

[0028] Preferably, the garment alignment generator in step S4 is an end-to-end network with an encoding-decoding structure. Specifically, the network includes a deformation parameter prediction module and a garment deformation and de-overlapping module.

[0029] In the deformation parameter prediction module, the inputs are divided into three groups. The first group of inputs is the clothing diagram I. cand clothing mask image M c The second set of inputs is the density attitude map I. densepose And skeleton pose diagram I pose The third input is the semantic segmentation graph P for trying on clothes. tryon The three inputs are concatenated and then passed through the first feature extractor f1, the second feature extractor f2, and the third feature extractor f3, respectively. These three feature extractors downsample and extract shallow and deepest features. The deepest features E1, E2, and E3 extracted by the three feature extractors f1, f2, and f3 are then concatenated and passed through a convolutional layer to predict the initial flow parameters. The shallow features, deepest features, and initial flow parameters extracted by the three feature extractors are combined. The input is a feature upsampling fusion module composed of multiple cascaded upsampling layers; in the feature upsampling fusion module, three feature extractors are cascaded at the front end of the first upsampling layer, using streaming parameters. Initialize the streaming parameters F of the input feature upsampling fusion module flow Each upsampling layer receives a portion of its input as the three deepest features E1, E2, and E3 from the cascaded network layers preceding it. The remaining input consists of shallow features from three feature extractors. After upsampling, the deepest features E1, E2, and E3 are added to and fused with shallow features of the same scale from the same feature extractor to obtain new deepest features E′1, E′2, and E′3. The streaming parameters F are then input to the current upsampling layer. flow Upsampling to the same scale as the deepest features E′1, E′2, and E′3 yields new flow parameters F′. flow Using the flow parameter F′ flow The deformable feature E is obtained by sampling the deepest feature E′1. 1w Through a single convolutional mapping, the deepest features E′2 and E′3 are mapped to the deformed features E, respectively. 1w The same scale, thus corresponding to feature E 2b and feature E 3b Then E 1w E 2b and E 3b The change in flow parameters ΔF is obtained by convolution after splicing along the channels. flow The change in flow parameter ΔF flow With the current flow parameter F flow The addition yields the new flow parameter F″. flow The deepest features E′1, E′2, E′3 and the flow parameter F″ in the previous upsampling layer flow The deepest features E1, E2, E3 and the streaming parameter F are respectively used as inputs to the next upsampling layer. flowAfter passing through all upsampling layers in the feature upsampling fusion module, the final streaming parameters F are output. flow ;

[0030] In the clothing deformation and de-overlapping model, the input is clothing image I. c Clothing mask image M c Flow parameter F flow and the semantic segmentation graph P for trying on clothes Rtryon First, the final output streaming parameters F of the feature upsampling fusion module are used. flow For clothing illustration I c and clothing mask image M c Sampling was performed to obtain the deformation diagram I of the garment to be adjusted. cwarped_o And clothing mask deformation map M cwarped_o Then use the try-on semantic segmentation graph P tryon Diagram I of clothing deformation to be adjusted cwarped_o And clothing mask deformation map M cwarped_o Further trimming and compensation were performed to remove overlapping and occluded portions, resulting in an aligned and non-overlapping garment deformation diagram (I). cwarped and its mask M cwarped .

[0031] Preferably, in step S5, the try-on generator is an end-to-end network that uses the ALIAS try-on network. It takes a person image, density pose image, skeleton pose image, and clothing deformation image with unknown clothing as input, and a try-on semantic segmentation image as additional input. The network then synthesizes the final try-on image.

[0032] Preferably, the try-on semantic segmentation graph generator, clothing alignment generator, and try-on generator are all pre-trained using a conditional adversarial training framework.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0034] This invention proposes novel character representations for unknown clothing and clothing length control vectors, enabling virtual try-on with controllable clothing length by generating a controllable semantic segmentation graph. This method boasts fast network learning speed and significant try-on effects, and can be applied to a wide range of complex virtual try-on scenarios. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0036] Figure 2 This is a diagram showing the results of data preprocessing.

[0037] Figure 3 A calculation diagram for the garment length control vector;

[0038] Figure 4 This invention provides a trial structure for a semantic segmentation graph generator.

[0039] Figure 5 The training process for the semantic segmentation graph generator;

[0040] Figure 6 This describes the specific implementation process of a truth-based compensation network.

[0041] Figure 7 This is the structure of the garment alignment generator according to the method of the present invention;

[0042] Figure 8 Training process for clothing alignment generator;

[0043] Figure 9 Inputs and outputs for an ALIAS-based try-on generator;

[0044] Figure 10 For the training process of the try-on generator;

[0045] Figure 11 These are fitting photos of different garments and lengths. Detailed Implementation

[0046] The present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the described embodiments are only for the purpose of understanding the present invention and do not constitute any limitation thereof.

[0047] like Figure 1 The diagram shown is a flowchart illustrating a method for generating virtual fitting images with controllable clothing length according to a preferred embodiment of the present invention. This embodiment of the method for generating virtual fitting images with controllable clothing length includes the following steps:

[0048] S1: Input the original image of the target person wearing the original clothing, as well as the semantic segmentation map, density pose map, and skeleton pose map corresponding to the original image. After preprocessing, remove the clothing information from the original image. Based on the key points of the skeleton pose map, obtain the image of the person with unknown clothing and the semantic segmentation map of the person with unknown clothing.

[0049] This step primarily involves image data preprocessing of the dataset, aiming to obtain images and semantic segmentation maps of people with unknown clothing. It should be noted that in this invention, density pose map refers to a dense pose map, while skeleton pose map refers to a pose map composed of human keypoints (or joints). The images and semantic segmentation maps of people with unknown clothing are, respectively, the processed original images excluding the upper garment, part of the pants, and arms, and their semantic segmentation maps. Figure 2 As shown, we define the original image I and the skeleton pose map I. poseOriginal image semantic segmentation map P parse According to I pose Key points for the mid-arm, torso, left hip, mid-hip, and right hip are connected to each other on line I using thick lines. These lines, along with the quadrilateral area formed by connecting the torso key points, create a mask area. This mask area covers the arm (and possibly parts of the surrounding area), the top, and part of the pants, but excludes the palm. The purpose of this mask is to effectively eliminate the shape of the clothing, part of the pants, and the arm, preserving the remaining body parts not involved in the fitting. (1-mask) is then used in conjunction with lines I and P. parse Multiplication can remove the area of ​​clothing to be replaced while preserving the rest of the image, thus obtaining a character image with unknown clothing (Image I). agnostic Semantic segmentation image P of characters with unknown clothing agnostic The specific implementation method is as follows:

[0050] mask = f(I pose )

[0051] P agnostic =P parse ⊙(1-mask)

[0052] I agnostic =I⊙(1-mask)

[0053] In addition, the subsequent try-on semantic segmentation map generator of this invention requires a garment length control vector as input. This garment length control vector can be calculated in step S1 or specified manually. The following is a brief description of how to obtain this garment length control vector through calculation. This garment length control vector is a discrete vector with one dimension, used to represent the garment length. The garment length control vector is obtained by standardizing the coordinates of three key points: the bottom edge of the garment, the neck, and the mid-hip area. For example... Figure 3 As shown, the skeleton pose diagram I is defined. pose Original image semantic segmentation map P parse We will need I pose Some key points are placed in P through mapping parse The above steps yield the neck and mid-hip keypoints corresponding to the skeleton pose map. These keypoints are then labeled on the semantic segmentation map and represented using P... n (x n y n ), P h (x h y h ) indicates. Connect P n P h Construct a straight line l. The equation of l can be calculated from the coordinates of these two key points:

[0054]

[0055] The line l intersects the clothing region of the semantic segmentation map at the lower edge key point P of the clothing. c P c coordinates (x) c y c This represents the coordinates of the pixels in the region where the line lies, when the pixel value changes from the clothing value to the non-clothing value. In operation, y... n The lower bound is defined as the maximum height of the image, and the upper bound is defined as the maximum height of the image. y is defined from y n Traverse the line in an ordered manner along the height direction, calculating the corresponding x-coordinate in l during the traversal. During the traversal, determine the semantic type of each pixel on the line in the semantic segmentation map. When the pixel value at (x, y) on the line changes from representing clothing to representing non-clothing, (x, y) is the desired P. c coordinates (x) c y c During the training phase of the try-on semantic segmentation graph generator, the garment length control vector is determined by P. c P h The difference between the ordinates and P h P n The ratio of the differences in the ordinates is calculated as follows:

[0056]

[0057] The value of z represents the length of the clothing relative to the body. When z < 0, the clothing style or wearing effect in the image is a short garment. When z > 0, the clothing style or wearing effect in the image is a long garment.

[0058] However, it should be noted that during the inference phase of the try-on semantic segmentation graph generator, the clothing length control vector can be obtained using the same calculation method as described above, or it can be directly input by the user, thereby allowing them to change the style or wearing effect of the clothing according to their own preferences.

[0059] S2: Input the clothing image to be tried on, the mask of the clothing image, the density pose map, the skeleton pose map, the clothing unknown character semantic segmentation map, and the clothing length control vector. Generate the try-on semantic segmentation map to be adjusted through a pre-trained end-to-end try-on semantic segmentation map generator.

[0060] like Figure 4 As shown, the try-on semantic segmentation graph generator is an end-to-end network with an encoding-decoding structure. The network input to the try-on semantic segmentation graph generator is divided into three groups. The first group of input is the image I of the garment to be tried on. cand the mask M of the clothing image c The second input is a semantic segmentation map P of a person with unknown clothing. agnostic The third set of inputs is the density attitude map I. densepose Skeleton posture diagram I pose The mapping result obtained after passing the clothing length control vector z through a fully connected layer is used to concatenate the three inputs and then pass them through the first feature extractor R1, the second feature extractor R2, and the third feature extractor R3 respectively. The three feature extractors downsample and extract shallow and deepest features. The deepest features F1, F2, and F3 extracted by the three feature extractors R1, R2, and R3 are then concatenated and input into the feature upsampling fusion module. This feature upsampling fusion module contains multiple upsampling layers for feature upsampling fusion. The first layer... The sampling layer has three cascaded feature extractors at the front end. In the feature upsampling and fusion module, each upsampling layer receives part of its input from the three deepest features F1, F2, and F3 output from the cascaded network layer at the front end, and the other part from the shallow features of the three feature extractors. The deepest features F1 and F3 are upsampled and then fused with shallow features of the same scale from the same feature extractor, resulting in two fused features F′1 and F′3. Then, the deepest feature F2 is upsampled to a shallow feature F′1 of the same scale from the same feature extractor R2. l 2. Using the same scale, F′2 is obtained. Then, F′2 and F... l 2. After concatenating F′1 and F′3, the feature F″2 is obtained by passing it through a convolutional layer. Finally, the features F′1, F″2, and F′3 obtained in the current upsampling layer are used as the inputs to the deepest features F1, F2, and F3 of the next layer. The deepest features output by the feature upsampling fusion module and the three sets of inputs from the try-on semantic segmentation map generator network are input into the normalization layer. After concatenation, the layers are reduced in dimensionality by a convolution, and the output is a try-on semantic segmentation map of the same size as the network input.

[0061] The aforementioned try-on semantic segmentation graph generator can be pre-trained using a conditional adversarial training framework before being used for actual reasoning. For example... Figure 5 As shown, the original image of the garment to be tried on is defined as I during training. c The clothing mask image is M. c Density attitude diagram is I densepose The skeleton pose diagram is I pose The semantic segmentation image of the character with unknown clothing is P. agnostic The clothing length control vector is z, and the semantic segmentation map of the original image is P. parse The clothing length control vector z is mapped through two fully connected layers to a mapping result Z of the same size as the density pose map and skeleton pose map. This mapping is then applied to the training group (I...). c M cI densepose I pose P agnostic Z) is divided into (I) c M c ), (P agnostic ), (I densepose I pose (Z) Three sets of input generators. The generator extracts features from the three sets of inputs and fuses them. After several layers of feature upsampling and fusion, the predicted semantic segmentation map P is obtained. fake Under the constraints of the loss function, the trial semantic segmentation graph generator can gradually generate semantic segmentation graphs that are similar to the original semantic segmentation graph, and gradually acquire the ability to predict trial semantic segmentation graphs. The loss function of this trial semantic segmentation graph generator is:

[0062] l=λ1L CE (P fake P parse )+λ2L CGAN

[0063] In the above formula, λ1 and λ2 are the weight coefficients for the two types of losses, both of which are 1. CE For P fake P parse Pixel-level cross-entropy loss, L CGAN This is called conditional adversarial loss. Conditional adversarial loss is expressed as:

[0064] L CGAN (G, D) = E x，y [log(D(x,y))]+E x，y [log(1-D(x,-y))]+E x，y [log(1-D(G(x,y),y))]+E x，y [log(1-D(G(x,y),-y))]

[0065] Here, x represents the input, and y represents the clothing length control vector. The function of this conditional adversarial loss is to determine whether the semantic segmentation map generated by the generator is realistic, and whether the clothing is short or long. In this embodiment, the above training requires 20 rounds to obtain the desired semantic segmentation map generator. This end-to-end try-on semantic segmentation map generator allows users to input different clothing images to obtain different clothing try-on semantic segmentation maps. It also allows users to input different clothing length control vectors z to obtain clothing try-on semantic segmentation maps with different clothing lengths. In actual use, the semantic segmentation map obtained at this time is a try-on semantic segmentation map to be adjusted, and further pruning and compensation are needed to obtain the final try-on semantic segmentation map.

[0066] S3: Input the semantic segmentation map of the original image and the semantic segmentation map of the try-on image to be adjusted. Then, use a ground truth-based compensation network to trim and compensate the semantic segmentation map of the try-on image to be adjusted, and obtain the semantic segmentation map of the try-on image.

[0067] like Figure 6 As shown, the truth-based compensation network process is divided into two aspects: pruning and compensation, as detailed below:

[0068] (1) Trimming. In this step, the semantic segmentation image to be adjusted contains redundant pants parts. In the final attempt at image synthesis, the semantic segmentation image guides the network to generate the attempt at image results. However, the pants parts that are not present in the original image will provide incorrect guidance for the final attempt at image synthesis, resulting in artifacts or semantically inappropriate content. Therefore, the semantic segmentation image to be adjusted needs to undergo trimming based on the original image to remove this redundant content. This ensures that no pants patterns that should not exist are generated before or after the attempt at image synthesis, thus preventing a decrease in the quality of the attempt at image.

[0069] (2) Compensation. During the trimming step, because the excess pants portion is trimmed and deleted, the semantic segmentation map will have a small empty, semantically undefined region. The compensation involves assigning the semantic meaning of the clothing to this empty, semantically undefined region using the value of the background region, thereby making the semantic segmentation map complete and correct. After semantic compensation, this semantic part will guide the fitting image generator to generate the skin portion on the stomach during the final fitting image generation.

[0070] In an embodiment of the present invention, a semantic segmentation map P of the original image is defined. parse The semantic segmentation graph for trying on clothes needs adjustment. Try-on image semantic segmentation map P tryon The input to the ground truth-based compensation network is the semantic segmentation map P of the original image. parse and the semantic segmentation graph of the try-on to be adjusted The specific processing flow within the network is as follows:

[0071] First, from the semantic segmentation map P of the original image... parse Extracting the pants channel M bottom_o Then, from the semantic segmentation map of the trial fitting that needs adjustment... Extracting the pants channel M bottom , put the pants channel M bottom With M bottom_o Multiply to get M′ bottom =M bottom ⊙M bottom_o ;

[0072] Then, from the semantic segmentation map of the try-on to be adjusted Extracting the background channel M background, background channel M background With (1-M) bottom )⊙M bottom_o Adding them together gives M′ background =M background +(1-M bottom )⊙M bottom_o ;

[0073] Finally, use M′ bottom and M′ background Replace the semantic segmentation graphs of the try-on section that need adjustment respectively. The pants channel and background channel are used to obtain the fitting semantic segmentation map P after trimming and compensation. tryon And it serves as the output of the compensation network.

[0074] S4: Input the clothing image to be tried on, the mask of the clothing image, the density pose map, the skeleton pose map and the try-on semantic segmentation map, and obtain the clothing deformation map through a pre-trained end-to-end clothing alignment generator.

[0075] like Figure 7 As shown, the garment alignment generator is an end-to-end network with an encoding / decoding structure. The specific processing is divided into two aspects, as follows:

[0076] (1) Predicting Deformation Parameters. The deformation method uses optical flow. The input consists of a clothing image and its mask, density pose map, skeleton pose map, and try-on semantic segmentation map. The input is divided into three groups and concatenated. Each group is downsampled by three feature extractors to extract features. The extracted features are then concatenated and passed through a convolutional layer to predict the first-level flow parameters. The features from the three branches and the first-level flow parameters are input into the upsampling layer, and the features extracted by each layer by the feature extractors are input into the corresponding upsampling layer. In the upsampling layer, the features and flow parameters are upsampled and enlarged. Then, the flow parameters are used to sample the clothing features. The sampling result is concatenated with the other two features and then input into a convolutional layer to obtain the change value of the flow parameters. The change value is used to update the flow parameters to obtain the flow parameter output of this layer. Other features and the flow parameter output of this layer are used as the output of this upsampling layer and input into the next upsampling layer. After feature fusion and upsampling through several upsampling layers, the final flow parameters are obtained.

[0077] (2) Clothing Deformation and De-overlapping. The input consists of a clothing image and its mask, flow parameters, and a try-on semantic segmentation image. The clothing image is sampled using the flow parameters to obtain the clothing deformation image to be adjusted. At this point, the clothing and the try-on semantic segmentation image are not perfectly aligned, and there are still occlusions in the deformation image that overlap with the body. Therefore, the try-on semantic segmentation image is used to further trim and compensate the clothing deformation image to be adjusted, and the overlapping occlusions are removed to obtain an aligned and non-overlapping clothing deformation image and its mask.

[0078] Therefore, in the embodiments of the present invention, the above-mentioned garment alignment generator specifically includes a deformation parameter prediction module, a garment deformation and de-overlapping module, and the specific processing flow of each module is as follows:

[0079] In the above deformation parameter prediction module, the input is divided into three groups. The first group of input is the clothing diagram I. c and clothing mask image M c The second set of inputs is the density attitude map I. densepose And skeleton pose diagram I pose The third input is the semantic segmentation graph P for trying on clothes. tryon The three inputs are concatenated and then passed through the first feature extractor f1, the second feature extractor f2, and the third feature extractor f3, respectively. These three feature extractors downsample and extract shallow and deepest features. The deepest features E1, E2, and E3 extracted by the three feature extractors f1, f2, and f3 are then concatenated and passed through a convolutional layer to predict the initial flow parameters. The shallow features, deepest features, and initial flow parameters extracted by the three feature extractors are combined. The input is a feature upsampling fusion module composed of multiple cascaded upsampling layers; in the feature upsampling fusion module, three feature extractors are cascaded at the front end of the first upsampling layer, using streaming parameters. Initialize the streaming parameters F of the input feature upsampling fusion module flow Each upsampling layer receives a portion of its input as the three deepest features E1, E2, and E3 from the cascaded network layers preceding it. The remaining input consists of shallow features from three feature extractors. After upsampling, the deepest features E1, E2, and E3 are added to and fused with shallow features of the same scale from the same feature extractor to obtain new deepest features E′1, E′2, and E′3. The streaming parameters F are then input to the current upsampling layer. flow Upsampling to the same scale as the deepest features E′1, E′2, and E′3 yields new flow parameters F′. flow Using the flow parameter F′ flow The deformable feature E is obtained by sampling the deepest feature E′1. 1w Through a single convolutional mapping, the deepest features E′2 and E′3 are mapped to the deformed features E, respectively. 1w The same scale, thus corresponding to feature E 2b and feature E 3b Then E 1w E 2b and E 3b The change in flow parameters ΔF is obtained by convolution after splicing along the channels. flow The change in flow parameter ΔF flow With the current flow parameter Fflow The addition yields the new flow parameter F″. flow The deepest features E′1, E′2, E′3 and the flow parameter F″ in the previous upsampling layer flow The deepest features E1, E2, E3 and the streaming parameter F are respectively used as inputs to the next upsampling layer. flow After passing through all upsampling layers in the feature upsampling fusion module, the final streaming parameters F are output. flow ;

[0080] In the above clothing deformation and de-overlapping model, the input is clothing image I. c Clothing mask image M c Flow parameter F flow and the semantic segmentation graph P for trying on clothes tryon First, the final output streaming parameters F of the feature upsampling fusion module are used. flow For clothing illustration I c and clothing mask image M c Sampling was performed to obtain the deformation diagram I of the garment to be adjusted. cwarped_o And clothing mask deformation map M cwarped_o Then use the try-on semantic segmentation graph P tryon Diagram I of clothing deformation to be adjusted cwarped_o And clothing mask deformation map M cwarped_o Further trimming and compensation were performed to remove overlapping and occluded portions, resulting in an aligned and non-overlapping garment deformation diagram (I). cwarped and its mask M cwarped .

[0081] Similarly, the aforementioned clothing alignment generator can be pre-trained using a conditional adversarial training framework before being used for actual inference. However, it is important to note that... Figure 7 The clothing alignment generator in the inference phase takes the third set of inputs as the try-on semantic segmentation graph P. tryon However, during the training phase, the third set of inputs can be replaced with the semantic segmentation map of the original image, P. parse Therefore, as follows Figure 8 As shown, during training, the original image is defined as I, and the original clothing image is defined as I. c The clothing mask image is M. c Density attitude diagram is I densepose The skeleton pose diagram is I pose The semantic segmentation map of the original image is P. parse Using P parse Extract the upper garment region of I to obtain the target image of the upper garment region I. pc Target mask image of the upper garment area M pc As the target value, specifically, take P. parse Clothing section M top I and M topMultiplying them together gives I pc M pc The value M can be directly obtained. top The formulas involved are as follows:

[0082] I pc =M top ⊙I

[0083] M pc =M top

[0084] Using training group (I c M c I densepose I pose P parse The flow parameters F are obtained through the generator. flow . to I c With flow parameter F flow Sampling was conducted to obtain clothing sampling results I cwarped_o and its mask M cwarped_o Combining I cwarped_o With P parse Remove I cwarped_o The occluded portion that overlaps with the body, specifically P. parse Clothing section M top M top With M cwarped_o Multiply to get M cwarped Then use M cwarped with I cwarped_o Multiplying the images removes misaligned and overlapping parts, resulting in the final garment deformation diagram I. cwarped The formulas involved are as follows:

[0085] M cwarped =M top ⊙M cwarped_o

[0086] I cwarped =M cwarped ⊙I cwarped_o

[0087] Under the constraints of the loss function, the clothing alignment generator gradually generates clothing deformation maps that are similar to the clothing in the original image, and gradually acquires the ability to predict clothing deformation. The loss function of this clothing alignment generator is:

[0088] l=λ1L VGG (I cwarped I pc )+λ2L1(M cwarped M pc )+λ3L CGAN

[0089] In the above formula, λ1, λ2, and λ3 are the weight coefficients of the three types of losses, all of which are 1. VGG For I cowarped with I pc VGG visual loss, L1 is M cowarped With M pc L1 loss, L CGAN This is called conditional adversarial loss. Conditional adversarial loss is expressed as:

[0090] L CGAN (G, D) = E x [log(D(x))]+E x [log(1-D(G(x)))]

[0091] Here, x represents the input. The function of this conditional adversarial loss is to determine whether the clothing deformation map generated by the generator is realistic. In this embodiment, the above training requires 20 rounds to obtain the desired clothing alignment generator.

[0092] S5: Input the unknown character image of the clothing, the density pose image, the skeleton pose image, the clothing deformation image, and the try-on semantic segmentation image, and generate the final output try-on image through a pre-trained end-to-end try-on generator.

[0093] like Figure 9 As shown, the try-on generator is an end-to-end network, which is a try-on network based on ALIgnmentAwareSegment (ALIAS). It takes the unknown person image, density pose image, skeleton pose image and clothing deformation image as input, and uses the try-on semantic segmentation map as additional input. The network will synthesize the final try-on image.

[0094] Similarly, this try-on generator also needs to be pre-trained using a conditional adversarial training framework before it can be used for actual inference. For example... Figure 10 As shown, during training, the original image is defined as I, and the image of the person with unknown clothing is defined as I. agnostic The clothing deformation diagram is I. cwarped Density attitude diagram is I densepose The skeleton pose diagram is I pose The semantic segmentation map of the original image is P. parse Using the training group (I) cwarped I dense□□se I pose I agnostic P parse Generate fitting images using the fitting generator I fake The loss function of the try-on generator is:

[0095] l=λ2L VGG (I fake,I)+λ2L CGAN

[0096] In the above formula, λ1 and λ2 are the weight coefficients of the two types of losses, both of which are 1. Where, L VGG For I fake And VGG visual loss of I, L CGAN This is called conditional adversarial loss. Conditional adversarial loss is expressed as:

[0097] L CGAN (G, D) = E x [log(D(x))]+E x [log(1-D(G(x)))]

[0098] Here, x represents the input. The purpose of this conditional adversarial loss is to determine whether the try-on images generated by the generator are realistic.

[0099] After training, all the semantic segmentation image generators for try-on, clothing alignment generators, and try-on generators allow users to input different clothing items and different clothing length control vectors (either calculated or directly specified) and follow the above steps to obtain images of different clothing items and different wearing styles, thus achieving a controllable virtual try-on effect. In embodiments of this invention, some virtual try-on effects generated by the above method are as follows: Figure 11 As shown, this invention can effectively realize virtual try-on of clothing images, with significant results, and the length of clothing can be flexibly controlled during the virtual try-on process.

[0100] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. It will be apparent to those skilled in the art that various modifications can be made to the above embodiments, and the general principles described herein can be applied to other embodiments without inventive effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.

Claims

1. A method for generating virtual fitting images with controllable garment length, characterized in that, Includes the following steps: S1: Input the original image of the target person wearing the original clothing, as well as the semantic segmentation map, density pose map and skeleton pose map corresponding to the original image. After preprocessing, remove the clothing information of the original image, and obtain the image of the person with unknown clothing and the semantic segmentation map of the person with unknown clothing based on the key points of the skeleton pose map. S2: Input the clothing image to be tried on, the mask of the clothing image, the density pose map, the skeleton pose map, the clothing unknown character semantic segmentation map, and the clothing length control vector, and generate the try-on semantic segmentation map to be adjusted through a pre-trained end-to-end try-on semantic segmentation map generator. S3: Input the semantic segmentation map of the original image and the semantic segmentation map of the try-on image to be adjusted. Then, use a ground truth-based compensation network to trim and compensate the semantic segmentation map of the try-on image to be adjusted to obtain the semantic segmentation map of the try-on image. S4: Input the clothing image to be tried on, the mask of the clothing image, the density pose map, the skeleton pose map and the try-on semantic segmentation map, and obtain the clothing deformation map through a pre-trained end-to-end clothing alignment generator. S5: Input the unknown character image of the clothing, the density pose image, the skeleton pose image, the clothing deformation image and the try-on semantic segmentation image, and generate the final output try-on image through a pre-trained end-to-end try-on generator. In step S2, the garment length control vector is a discrete vector with one dimension, used to represent the garment length. During the training phase of the try-on semantic segmentation graph generator, the garment length control vector is calculated by standardizing the coordinates of three key points: the bottom edge of the garment, the neck, and the mid-hip. The calculation method is as follows: Map the neck and mid-hip keypoints from the skeleton pose map onto the semantic segmentation map to obtain the neck keypoints on the semantic segmentation map. ( , ) and key points of the mid-glute ( , ); connecting the key points of the neck Construct a straight line and iterate through the pixels along the line in an orderly manner, determining the semantic type of each pixel in semantic segmentation. Pixels on the line whose values ​​change from representing clothing to representing non-clothing are designated as keypoints. This key point This represents the intersection of the straight line and the lower edge of the clothing in the semantic segmentation image; [the point where the line intersects the clothing]. and The difference between the ordinates and and The ratio of the differences in the ordinates is used to calculate the garment length control vector. The specific calculation formula is as follows: The value of z represents the length of the clothing relative to the body. When z < 0, the clothing style or wearing effect in the image is a short garment; when z > 0, the clothing style or wearing effect in the image is a long garment. In step S3, the ground truth compensation network performs trimming compensation on the semantic segmentation image of the try-on to be adjusted, specifically including two steps: trimming and compensation. First, the semantic segmentation image of the try-on to be adjusted needs to be trimmed based on the original image to remove redundant pants parts from the semantic segmentation image of the try-on to be adjusted, and the removed area forms a semantically empty region. Then, the value of the background region is used to assign a value to the semantically empty region formed in the trimming step, so that the semantic segmentation image of the try-on to be adjusted is restored to completeness. In step S3, the input to the ground truth-based compensation network is the semantic segmentation map of the original image. and the semantic segmentation graph of the try-on to be adjusted The specific processing flow within the network is as follows: First, from the semantic segmentation map of the original image... Extracting the pants channel Then, from the semantic segmentation map of the trial fitting that needs adjustment... Extracting the pants channel , the pants channel and Multiply to get ; Then, from the semantic segmentation map of the try-on to be adjusted Extracting the background channel , background channel and Adding them together gives ; Finally, using and Replace the semantic segmentation graphs of the try-on section that need adjustment respectively. The pants channel and background channel are used to obtain the fitting semantic segmentation map after trimming and compensation. And it serves as the output of the compensation network.

2. The method for generating virtual fitting images with controllable clothing length according to claim 1, characterized in that, In step S1, the character image with unknown clothing and the character semantic segmentation image with unknown clothing are the original image and its semantic segmentation image after processing, excluding the top, part of the pants, and arms. The acquisition method is as follows: obtain the positions of each key point from the skeleton pose image and map them onto the original image. Connect the key points of the torso, arms, and hips on the original image to form a mask area covering the arms, top, and part of the pants, but excluding the palm. Delete the image within the mask area in the original image and its semantic segmentation image to achieve the purpose of deleting clothing information, and finally obtain the character image with unknown clothing and the character semantic segmentation image with unknown clothing.

3. The method for generating virtual fitting images with controllable clothing length according to claim 1, characterized in that, In step S2, during the inference phase of the try-on semantic segmentation graph generator, the clothing length control vector is either directly input by the user or obtained using the above calculation method.

4. The method for generating virtual fitting images with controllable clothing length according to claim 1, characterized in that, In step S2, the try-on semantic segmentation graph generator is an end-to-end network with an encoding-decoding structure. The network input is divided into three groups, the first group being the image of the clothing to be tried on. and the mask of the clothing image The second input is a semantic segmentation map of a person whose clothing is unknown. The third set of inputs is a density attitude map. Skeleton posture diagram The mapping result obtained after passing the clothing length control vector z through a fully connected layer is used to concatenate the three inputs and then pass them through the first feature extractor R1, the second feature extractor R2, and the third feature extractor R3 respectively. The three feature extractors downsample and extract shallow and deepest features. Then, the deepest features F1, F2, and F3 extracted by the three feature extractors R1, R2, and R3 are concatenated and input into the feature upsampling fusion module. The feature upsampling fusion module contains multiple upsampling layers for feature upsampling fusion. The first upsampling layer... The front end of the sample layer is cascaded with three feature extractors. In the feature upsampling and fusion module, each upsampling layer receives part of its input as the three deepest features F1, F2, and F3 output from the cascaded network layers, and the other part as shallow features from the three feature extractors. The deepest features F1 and F3 are upsampled and then fused with shallow features of the same scale from the same feature extractor, resulting in two fused features F'1 and F'3. Then, the deepest feature F2 is upsampled to a shallow feature F'1 of the same scale from the same feature extractor R2. l 2. The same scale is used to obtain F'2. Then, F'2 and F... l 2. After concatenating F'1 and F'3, feature F''2 is obtained by passing through a convolutional layer. Finally, features F'1, F''2, and F'3 obtained from the current upsampling layer are used as inputs to the deepest features F1, F2, and F3 of the next layer. The deepest features output by the feature upsampling fusion module and the three sets of inputs from the try-on semantic segmentation map generator network are input into the normalization layer. After concatenation, the layers are reduced in dimensionality by a convolution, and the output is a try-on semantic segmentation map of the same size as the network input.

5. The method for generating virtual fitting images with controllable clothing length according to claim 1, characterized in that, The garment alignment generator in step S4 is an end-to-end network with an encoding-decoding structure. Specifically, the network includes a deformation parameter prediction module and a garment deformation and de-overlapping module. In the deformation parameter prediction module, the inputs are divided into three groups. The first group of inputs is the clothing image. and clothing mask image The second set of inputs is a density attitude map. and skeleton pose diagram The third input is a semantic segmentation graph of trying on clothes. The three inputs are concatenated and then passed through the first feature extractor f1, the second feature extractor f2, and the third feature extractor f3, respectively. These three feature extractors downsample and extract shallow and deepest features. The deepest features E1, E2, and E3 extracted by the three feature extractors f1, f2, and f3 are then concatenated and passed through a convolutional layer to predict the initial flow parameters. The shallow features, deepest features, and initial flow parameters extracted by the three feature extractors are combined. The input is combined into a feature upsampling fusion module consisting of multiple cascaded upsampling layers; In the feature upsampling fusion module, three feature extractors are cascaded at the front end of the first upsampling layer, using streaming parameters. Initialize the stream parameters of the input feature upsampling fusion module Each upsampling layer receives a portion of its input as the three deepest features E1, E2, and E3 from the cascaded network layers preceding it. The remaining input consists of shallow features from three feature extractors. After upsampling, the deepest features E1, E2, and E3 are added to and fused with shallow features of the same scale from the same feature extractor to obtain new deepest features E'1, E'2, and E'3. These new features are then used as the flow parameters for the current upsampling layer. Upsampling to the same scale as the deepest features E'1, E'2, and E'3 yields new flow parameters. Using flow parameters The deformable feature E is obtained by sampling the deepest feature E'1. 1w The deepest features E'2 and E'3 are mapped to the deformed features E through a single convolution mapping. 1w The same scale, thus corresponding to feature E 2b and feature E 3b Then E 1w E 2b and E 3b The flow parameter variation values ​​are obtained by convolution after splicing along the channels. The change value of the flow parameter With current flow parameters Adding them together yields new flow parameters. The deepest features E'1, E'2, E'3 and the flow parameters in the previous upsampling layer The deepest features E1, E2, E3, and streaming parameters are respectively used as inputs to the next upsampling layer. ; After passing through all upsampling layers in the feature upsampling fusion module, the final streaming parameters are output. ; In the clothing deformation and de-overlap model, the input is a clothing drawing. Clothing mask image Flow parameters semantic segmentation graph of try-on First, the final output streaming parameters of the feature upsampling fusion module are used. clothing illustrations Clothing mask image Sampling was performed to obtain the deformation diagram of the garment to be adjusted. And clothing mask deformation diagram Then use the try-on semantic segmentation map Deformation diagram of clothing to be adjusted And clothing mask deformation diagram Further trimming and compensation are performed to remove overlapping and occluded portions, resulting in an aligned and non-overlapping garment deformation diagram. and its mask .

6. The method for generating virtual fitting images with controllable clothing length according to claim 1, characterized in that, In step S5, the try-on generator is an end-to-end network. This network uses the ALIAS try-on network, which takes the image of the person with unknown clothing, density pose image, skeleton pose image and clothing deformation image as input, and uses the try-on semantic segmentation image as additional input. The network synthesizes the final try-on image.

7. The method for generating virtual fitting images with controllable clothing length according to claim 1, characterized in that, The try-on semantic segmentation graph generator, clothing alignment generator, and try-on generator are all pre-trained using a conditional adversarial training framework.

Images