A clothing editing method based on semantic guidance

Abstract

The present invention discloses a clothing editing method based on semantic guidance, including four clothing editing functions: clothing deformation, clothing texture migration, texture printing insertion and clothing content removal. The present invention only needs to train one clothing deformation network to realize all four clothing editing functions, with low computational cost and high efficiency. The present invention applies two-dimensional image restoration technology, decomposes clothing editing into clothing restoration and character restoration, and can effectively maintain the texture information of the generated image; adopts a strategy of joint training of a restoration generator and a reconstruction device, can realize the optimal feature patch borrowing, and accurately retain the texture details of the target clothing; uses a three-step training strategy, can specifically guide the character restoration network to complete the character details when the character information is incomplete; applies a two-dimensional image virtual fitting algorithm TS-APD; adopts a semantic guidance strategy to accurately control the editing area, can maximize the guarantee of clear boundaries and accurate and reasonable shapes.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and clothing, and in particular to a clothing editing method based on semantic guidance. Background Art

[0002] Thanks to the rapid development of image generation technology and human body analysis technology, virtual fitting technology has been widely used in shopping malls, factories and mobile apps with the help of various hardware devices. With people's continuous pursuit of quality of life, private customization services based on clothing editing will gradually become popular. However, the use of traditional manual design and software tool editing has been unable to meet the private customization needs of large orders. The use of deep learning technology to achieve multifunctional clothing synthesis and editing can solve this problem.

[0003] One of the existing technologies is an attribute decomposition method that provides clothing editors with the functions of texture replacement and texture interpolation. It realizes explicit encoding of different semantic regions of the human body based on the human body analysis graph. The steps are: target posture encoding; semantic attribute decomposition; semantic style encoding and target image generation. The disadvantage of this technology is that the synthesized human image lacks spatial context information.

[0004] The second prior art is a two-stage method for synthesizing and editing human images, which includes the following steps: prediction of target human body analysis map; joint global and regional coding; spatial perception normalization and target image synthesis. This technology can realize the functions of texture migration, texture interpolation and regional editing, but its disadvantage is that it easily leads to averaging of regional textures and distortion of the face.

[0005] The third prior art is a sequential fitting method, which separates and encodes character information and clothing information in the latent space, and can freely operate the clothing synthesis process to achieve functions such as clothing layering, content removal, print insertion, texture migration, and clothing deformation. The disadvantage of this technology is that it is easy to cause texture averaging, it is difficult to maintain local texture details, and it is difficult to handle complex clothing shapes. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the existing methods and propose a clothing editing method based on semantic guidance. The present invention includes four clothing editing functions: clothing deformation, clothing texture migration, texture printing insertion and clothing content removal. Clothing deformation is to deform the top style of the character image into the style of the target clothing, and the original top texture remains unchanged; clothing texture migration is to migrate the texture of the target clothing to the character, and the original top style remains unchanged; texture printing insertion is to print textures, patterns and logos on the clothing area of ​​the character image, which should be adapted to the human body posture and occlusion; content removal is to remove the content on the clothing through a free mask and complete it according to the background.

[0007] The main problem solved by the present invention is how to edit the top of a character image in a semantically guided manner so that the shape boundary of the synthesized result is clear, the texture details are relatively intact, and the overall character image is more realistic.

[0008] In order to realize the four clothing editing functions, the technical scheme of the present invention is as follows:

[0009] A semantically guided clothing editing method, the steps are as follows:

[0010] Step 1: For clothing deformation, decompose it into clothing restoration and person restoration. First, use the TS-APD algorithm to create a clothing deformation dataset, generate top mask, distorted clothing, distorted clothing mask and fitting image from person image and clothing image, obtain the retained top image, the mask of the clothing to be restored and the retained person image through mask cropping operation, and then realize clothing deformation according to steps 2 and 3;

[0011] Step 2: Input the retained top image and the mask of the clothing to be repaired, and use the clothing repair network D to generate the repair result of the top;

[0012] Step 3: Input the character image, the restoration result of the top and the distorted clothing mask, and use the character restoration network E to output the clothing deformation result;

[0013] Step 4: For clothing texture migration, decompose it into virtual fitting and clothing deformation. Input the person image and clothing image, first use the TS-APD algorithm to generate a virtual fitting image, and then use the clothing restoration network D and the person restoration network E to output the clothing texture migration result;

[0014] Step 5: For texture printing insertion, input the person image and texture printing pattern, first use the OpenCV tool to extract the printing mask, and then use the TS-APD fitting network to output the texture printing result;

[0015] Step 6: For clothing content removal, input a person image, first use the LabelMe tool to mark the content mask that needs to be removed, and then use the clothing repair network to output the clothing content removal result.

[0016] The entire technical process only requires training the clothing repair network D and the character repair network E through steps 1, 2, and 3. No additional training is required in steps 4, 5, and 6. The corresponding results can be directly output using the pre-trained network. The results output by steps 3, 4, 5, and 6 correspond to the four clothing editing functions respectively.

[0017] The present invention adopts the TS-APD algorithm in the "Virtual Fitting Method of Two-stage Adjustable Perceptual Distillation Network" disclosed by Chen Baoyu et al. in the Journal of Graphics, which can directly generate fitting images from person images and clothing images. The data preparation stage of TS-APD includes a clothing foreground segmentation network and a person top segmentation network, and uses precise semantic segmentation to guide the fitting process. The virtual fitting stage of TS-APD includes a distortion network and a fitting network: the distortion network takes person images and clothing images as input, and can generate distorted clothing and distorted clothing masks corresponding to the person top area; the fitting network takes person images, distorted clothing and distorted clothing masks as input to generate fitting images.

[0018] Preferably, the network architecture of the clothing repair network D includes a coarse-to-fine generator and an auxiliary reconstructor, which are trained in multiple stages to generate the final clothing repair effect, as follows:

[0019] The generator includes a coarse network and a refined network, both of which adopt a gated convolutional encoder-decoder structure; taking the retained top and the mask of the clothing to be repaired as input, and the top of the original person image as the true value, the coarse network is first trained to output the coarse clothing repair result; then the parameters of the coarse network are fixed, and the coarse clothing repair result is sent to the refined network, which is weighted by the similarity of its own features and outputs the optimized clothing repair result; then, the coarse network and the refined network are jointly trained; finally, the coarse-to-fine generator and the auxiliary reconstructor are jointly trained.

[0020] The auxiliary reconstructor includes a similarity encoder and an auxiliary encoder-decoder; the similarity encoder takes the multi-scale features of the refined network decoder as input and calculates the feature similarity; the auxiliary encoder-decoder takes the optimized clothing repair result as input, the decoded features are weighted by the feature similarity, and the auxiliary clothing reconstruction result is output.

[0021] The coarse network uses L1 loss constraints, the refined network uses adversarial loss and L1 loss as constraints, and the auxiliary reconstructor uses adversarial loss and L1 loss as constraints; after multiple stages of training, a coarse-to-fine generator is used to achieve clothing repair, and the auxiliary reconstructor is only needed during the training process and can be regarded as a learnable loss function.

[0022] Preferably, the character restoration network E adopts the fitting network framework of TS-APD and is retrained through a three-step training process to enhance the character restoration capability; specifically, a fitting image is generated using a character image and distorted clothing, an original character image is generated using a retained character and original top, and a fitting image is generated using a retained fitting image and distorted clothing.

[0023] Beneficial effects of the present invention: A clothing editing method based on semantic guidance proposed by the present invention only needs to train one clothing deformation network to realize four clothing editing functions, with low computational cost and high efficiency; the image restoration technology is applied to decompose clothing editing into clothing restoration and character restoration, so as to better preserve rich texture details and enhance the realism of character images; based on the semantic guidance strategy, different clothing editing functions are guided, the boundary clarity is improved, and the synthesis ability of complex shapes is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of making clothing deformation data of the present invention;

[0025] Figure 2 is a flow chart of a clothing deformation algorithm of the present invention;

[0026] Figure 3 It is a flow chart of the clothing texture migration algorithm of the present invention.

[0027] Figure 4 It is the overall technical framework of the present invention. DETAILED DESCRIPTION

[0028] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0029] Figure 1 The garment deformation data making flow chart of the present invention comprises the following steps:

[0030] S1-1, use the TS-APD character top segmentation network to generate a top mask of the character image, and use the top mask to crop the character image to obtain the original top image and the original character image without the top;

[0031] S1-2, using the distortion network and fitting network of TS-APD, distorted clothing masks and fitting images are generated for the person image and target clothing, and the distorted clothing mask is used as the new top mask;

[0032] S1-3, using two top masks to crop the top of the person image to obtain a retained top, and using two top masks to crop the person image to obtain a retained person;

[0033] S1-4, retaining the top and the original top image as clothing restoration training data for clothing deformation, retaining the person and the original person image as character restoration training data for clothing deformation, the combination of the two simulates the entire clothing deformation process.

[0034] Figure 2 It is a flow chart of the clothing deformation algorithm of the present invention, comprising the following steps:

[0035] S2-1, use TS-APD's clothing segmentation network to generate clothing masks for character images, and use TS-APD's distortion network to generate distorted clothing and distorted clothing masks for character images and target clothing;

[0036] S2-2, using the distorted clothing mask to crop the top of the person image to obtain the retained top, and using the original top mask to crop the distorted clothing mask to obtain the clothing repair mask;

[0037] S2-3, using the clothing repair network D to generate optimized clothing repair results for retaining the top and clothing repair mask;

[0038] S2-4, use the character restoration network E to generate the final clothing deformation result for the character image, optimize the clothing restoration result and distort the clothing mask.

[0039] Figure 3 The present invention is a flow chart of clothing texture migration algorithm, comprising the following steps:

[0040] S3-1, use TS-APD's top segmentation network to generate top masks for person images, and use TS-APD's distortion network and fitting network to generate distorted clothing, distorted clothing masks, and fitting images for person images and target clothing;

[0041] S3-2, crop the fitting image with the original top mask and the distorted clothing mask to obtain the retained new top, and crop the original top mask with the distorted clothing mask to obtain the clothing repair mask;

[0042] S3-3, using the clothing repair network D to generate optimized clothing repair results for retaining the new top and clothing repair mask;

[0043] S3-4, use the character restoration network E to generate the final clothing texture migration result for the fitting image, optimize the clothing restoration result and the original top mask.

[0044] Figure 4 It is the overall technical framework of the present invention, comprising the following steps:

[0045] S4-1, the input of the clothing deformation function is the person image and the clothing image. First, the TS-APD distortion network is used to generate the distorted clothing and the distorted clothing mask. The distorted clothing mask is used to guide the repair process of the two repair networks. Then the clothing repair network is used to obtain the optimized clothing repair result. Finally, the person repair network is used to generate the final clothing deformation result.

[0046] S4-2, the clothing texture migration function is achieved by trying on clothes first and then deforming them. Its input is also person images and clothing images. First, the TS-APD distortion network and the fitting network are used to generate distorted clothing, distorted clothing masks and fitting images. Then, the clothing repair network and the person repair network are used to perform clothing deformation on the fitting images to obtain the final clothing texture migration result.

[0047] S4-3, the input of the texture printing insertion function is the person image and the texture printing pattern. First, the semantic mask of the texture printing pattern is extracted using the OpenCV tool, and then the fitting network of TS-APD is used to generate the rough printing result. Finally, the rough result and the person image are recombined through the semantic mask to obtain the final texture printing insertion result;

[0048] S4-4, the input of the clothing content removal function is a person image. First, the LabelMe tool is used to mark the content mask to be removed, and then the person image is cropped using the content mask. Finally, the final clothing content removal result is generated through the clothing repair network.

[0049] Existing clothing editing technologies either have a single function or require different training for different functions, with blurred textures, poor adaptability, and cannot be effectively promoted. The present invention is based on image restoration and semantic-guided strategies, and only needs to train one network to achieve four clothing editing functions: clothing deformation, clothing texture migration, texture print insertion, and clothing content removal. Based on image restoration technology, clothing deformation is decomposed into clothing restoration and character restoration, which are respectively achieved through two restoration networks. The clothing restoration network is jointly trained by a coarse-to-fine generator and an auxiliary context reconstructor, and the character restoration network is retrained using a three-step training strategy to improve the synthesis ability of clothing textures and character details. The other three clothing editing functions do not require additional texture training. Under the guidance of semantic information, accurate target image synthesis can be achieved using a pre-trained network.

[0050] In the description of the present invention, it is necessary to understand that the term "mask" is a one-dimensional semantic segmentation map, the term "cropping" means element-wise multiplication of an image or a semantic segmentation map, which is visually equivalent to cropping the image, and the "top" mentioned is the top area of ​​the character image, including shirts, long sleeves, coats, skirts, etc., which are collectively referred to as "tops" for the sake of ease of description.

Claims

1. A clothing editing method based on semantic guidance, characterized in that: Here are the steps: Step 1: For clothing deformation, decompose it into clothing restoration and person restoration. First, use the TS-APD algorithm to create a clothing deformation dataset, generate top mask, distorted clothing, distorted clothing mask and fitting image from person image and clothing image, obtain the retained top image, the mask of the clothing to be restored and the retained person image through mask cropping operation, and then realize clothing deformation according to steps 2 and 3; Step 2: Input the retained top image and the mask of the clothing to be repaired, and use the clothing repair network D to generate the repair result of the top; Step 3: Input the character image, the restoration result of the top and the distorted clothing mask, and use the character restoration network E to output the clothing deformation result; Step 4: For clothing texture migration, it is decomposed into virtual fitting and clothing deformation. The character image and clothing image are input. First, the TS-APD algorithm is used to generate a virtual fitting image. Then, the clothing texture migration result is output using the clothing restoration network D and the character restoration network E according to the original character's top style. Step 5: For texture printing insertion, input the person image and texture printing pattern, first use the OpenCV tool to extract the printing mask, and then use the TS-APD fitting network to output the texture printing result; Step 6: For clothing content removal, input the person image, first use the LabelMe tool to mark the content mask to be removed, and then use the clothing restoration network to output the clothing content removal result; The network architecture of the clothing repair network D includes a coarse-to-fine generator and an auxiliary reconstructor, which are trained in multiple stages to generate the final clothing repair effect, as follows: The generator includes a coarse network and a refined network, both of which adopt a gated convolutional encoder-decoder structure. The coarse network is trained first, with the retained top and the mask of the clothing to be repaired as input and the top of the original person image as the true value, and the coarse network is output to output the coarse clothing repair result. Then fix the parameters of the coarse network, send the coarse clothing repair results to the refined network, use the similarity of its own features to weight them, and output the optimized clothing repair results; then, jointly train the coarse network and the refined network; finally, jointly train the coarse-to-fine generator and the auxiliary reconstructor; The auxiliary reconstruction device includes a similarity encoder and an auxiliary encoder-decoder. The similarity encoder takes the multi-scale features of the refined network decoder as input and calculates the feature similarity. The auxiliary encoder-decoder takes the optimized clothing repair result as input, and the decoded features are weighted by the feature similarity to output the auxiliary clothing reconstruction result. The coarse network uses L1 loss constraints, the refined network uses adversarial loss and L1 loss as constraints, and the auxiliary reconstructor uses adversarial loss and L1 loss as constraints; after multiple stages of training, the coarse-to-fine generator is used to achieve clothing repair, and the auxiliary reconstructor is only needed during the training process and is regarded as a learnable loss function; The character restoration network E adopts the fitting network framework of TS-APD and is retrained through a three-step training process to enhance the character restoration capability; specifically, a fitting image is generated using a character image and distorted clothing, an original character image is generated using a retained character and original top, and a fitting image is generated using a retained fitting image and distorted clothing.

2. The semantically guided clothing editing method according to claim 1, characterized in that: The production of clothing deformation data in step 1 includes the following steps: S1-1, use the TS-APD character top segmentation network to generate a top mask of the character image, and use the top mask to crop the character image to obtain the original top image and the original character image without the top; S1-2, using the distortion network and fitting network of TS-APD, distorted clothing masks and fitting images are generated for the person image and target clothing, and the distorted clothing mask is used as the new top mask; S1-3, using two top masks to crop the top of the person image to obtain a retained top, and using two top masks to crop the person image to obtain a retained person; S1-4, retaining the top and the original top image as clothing restoration training data for clothing deformation, retaining the person and the original person image as character restoration training data for clothing deformation, the combination of the two simulates the entire clothing deformation process.

3. The semantically guided clothing editing method according to claim 1, characterized in that: The algorithm for clothing texture migration in step 4 comprises the following steps: S3-1, use TS-APD's top segmentation network to generate top masks for person images, and use TS-APD's distortion network and fitting network to generate distorted clothing, distorted clothing masks, and fitting images for person images and target clothing; S3-2, crop the fitting image with the original top mask and the distorted clothing mask to obtain the retained new top, and crop the original top mask with the distorted clothing mask to obtain the clothing repair mask; S3-3, using the clothing repair network D to generate optimized clothing repair results for retaining the new top and clothing repair mask; S3-4, use the character restoration network E to generate the final clothing texture migration result for the fitting image, optimize the clothing restoration result and the original top mask.

Images