Virtual fitting method using arm region reservation strategy

Abstract

The application discloses a virtual fitting method adopting an arm region reservation strategy, and specifically implements the following steps: step one, respectively pre-process a human body image I and a clothing image C to obtain clothing-independent human body representation h p and a mask M corresponding to the clothing image C, and obtain a pre-processing result; step two, constructing a virtual fitting network adopting an arm region reservation strategy, and constructing a loss function for network training; step three, training the virtual fitting network adopting the arm region reservation strategy by using paired image training set images and the pre-processing results obtained in step one, to obtain virtual fitting network weights adopting the arm region reservation strategy; and step four, generating lifelike fitting images based on the virtual fitting network adopting the arm region reservation strategy and the network weights, to realize a virtual fitting effect of migrating clothing to a corresponding region of a human body in an image. The method solves the problem that it is difficult to obtain a fitting result with real feeling by using an existing method.

Description

Technical Field

[0001] This invention relates to the field of virtual try-on technology, and more specifically to a virtual try-on method that employs an arm area preservation strategy. Background Technology

[0002] Virtual try-on has immense application potential in the apparel e-commerce sector. Three-dimensional virtual try-on methods simulate the effect of clothing on the human body by fitting garment pieces onto a 3D model. This method allows for multi-view physical simulation, but it typically suffers from high equipment requirements, complex models, and high time costs. In recent years, image-based virtual try-on methods have emerged with the development of deep learning. These methods do not require 3D information, are easier to deploy, and generate try-on results efficiently and quickly, making them more suitable for existing online apparel marketing scenarios and attracting widespread attention.

[0003] Current research on virtual try-on typically relies on image generation methods. Han et al. proposed the first image-based virtual try-on network, VITON (HAN X, WU Z, WU Z, et al. Viton: An image-based virtual try-on network[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2018:7543-7552.), which uses a coarse-to-fine strategy to seamlessly transmit the desired clothing to the corresponding areas of the human body. Subsequently, Wang et al. proposed CP-VTON (WANG B, ZHENG H, LIANG X, et al. Toward characteristic-preserving image-based virtual try-on network[C] / / Proceedings of the European conference on computervision(ECCV).2018:589-604.) to deform clothing by learning the parameters of thin-plate spline (TPS) transformation through a novel Geometric Matching Module (GMM), and designed a try-on module based on a U-Net structure to render the final try-on result image. Minar et al. proposed CP-VTON+ (MINAR MR, TUAN TT, AHN H, et al. Cp-vton+: Clothing shape and texture preserving image-based virtual try-on[C] / / CVPRWorkshops.2020,3:10-14.), which improved the design of human body representation and geometric matching module in CP-VTON.Yang et al. proposed ACGPN (YANG H, ZHANG R, GUO X, et al. Towards photo-realistic virtual try-on by adaptively generating-preserving image content[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2020:7850-7859.), which determines whether to generate or preserve image content based on the predicted human segmentation map. Ge et al. proposed DCTON (GE C, SONG Y, GE Y, et al. Disentangled cycle consistency for highly-realistic virtual try-on[C] / / Proceedings of the IEEE / CVFconference on computer vision and pattern recognition.2021:16928-16937.), which trains the network in a self-supervised manner through cycle consistency learning. However, CP-VTON+ neglects to preserve the human arm region, resulting in blurred hand areas and unclear boundaries between fingers in the fitting results. Furthermore, its simple U-net structure for rendering the final fitting image makes the results unnatural. ACGPN, on the other hand, ignores the influence of the original clothing in preserving image content, and the fitting results are highly dependent on the accuracy of the predicted human segmentation map. Due to the introduction of the segmentation map prediction module or the influence of the training strategy, networks such as ACGPN and DCTON have a high number of parameters and high computational resource requirements. Summary of the Invention

[0004] The purpose of this invention is to provide a virtual try-on method that employs an arm area preservation strategy, thereby solving the problem that existing methods are insufficient to obtain realistic try-on results.

[0005] The technical solution adopted in this invention is a virtual try-on method using an arm area preservation strategy, specifically implemented according to the following steps:

[0006] Step 1: Preprocess the human body image I and the clothing image C respectively to obtain a clothing-independent human body representation h. p The preprocessing result is obtained by using the mask M corresponding to the clothing image C;

[0007] Step 2: Construct a virtual try-on network that employs an arm region preservation strategy, and construct a loss function for network training;

[0008] Step 3: Use the paired image training set images and the preprocessing results obtained in Step 1 to train the virtual try-on network with the arm region preservation strategy, and obtain the weights of the virtual try-on network with the arm region preservation strategy.

[0009] Step 4: After preprocessing the images of people and clothing in the unpaired image test set in Step 1, use the obtained preprocessing results as input to generate realistic try-on images based on the virtual try-on network with an arm region preservation strategy and network weights, so as to achieve the virtual try-on effect of transferring clothing to the corresponding area of ​​the human body in the image.

[0010] The invention is further characterized in that,

[0011] Step one specifically involves: preprocessing the human body image I to obtain a clothing-independent human body representation h. p The specific preprocessing process is as follows: using the OpenPose pose estimator to estimate the pose of the person in human image I to obtain a pose heatmap; using the LIP human body parser to segment the person in human image I to obtain a body shape mask; and using the LIP human body parser to obtain the head region from human image I. For clothing image C, preprocessing is performed to obtain the clothing mask M corresponding to C. The specific preprocessing process is as follows: using the LIP human body parser to segment the clothing in image C to obtain the clothing mask M corresponding to C.

[0012] In step two, a virtual try-on network based on an arm region preservation strategy is constructed. This virtual try-on network consists of three modules: a clothing deformation module, an arm region preservation module, and a try-on generation module. The clothing deformation module is used to deform the clothing to fit the human body. The arm region preservation module is used to adaptively preserve the human arm region and obtain a corrected clothing-independent human body representation. The try-on generation module is used to generate a try-on result image from the deformed clothing and the corrected clothing-independent human body representation.

[0013] In step two, the garment deformation module includes two encoders, a relevant matching layer Co, a regressor R, and a deformation module W;

[0014] Each encoder contains four downsampling convolutional layers with a stride of 2 and two convolutional layers with a stride of 1, with filter counts of 64, 128, 256, 512, 512, and 512, respectively.

[0015] The relevant matching layer Co includes normalizing two sets of feature representations of the input to the unit L2 norm by channel and flattening them into matrices. At each position, the flattened matrices obtained from the two sets of features are multiplied to obtain the correlation map.

[0016] The regressor R consists of two downsampling convolutional layers with a stride of 2, two convolutional layers with a stride of 1, and a fully connected output layer with an output size of 50. The number of filters in the downsampling convolutional layers and the convolutional layers are 512, 256, 128, and 64, respectively.

[0017] The processing procedure of the clothing deformation module is as follows: input a clothing-independent human body representation h p The clothing mask M corresponding to the clothing image C is used by two encoders to represent the clothing-independent human body h. p Feature extraction is performed on the clothing mask M to obtain two sets of features. The correlation matching layer Co combines these two sets of features into a correlation map, which is then input into a regressor to predict the deformation parameter θ. The deformation module W, based on the TPS transform, performs deformation processing on the clothing image C and the clothing mask M according to the deformation parameter θ, resulting in the deformed clothing C. w With the transformation clothing mask M w ;

[0018] The loss function used to train the clothing deformation module is constructed as follows: Loss on pixels and deformation regularization loss Composition:

[0019]

[0020] In the formula, λ reg and λ c Loss on pixels and deformation regularization loss The corresponding hyperparameters; pixel loss Defined as transforming clothing C w Clothing area I in human body image I C The L1 norm of the model is used to encourage the network to acquire deformable clothing that approximates the truth; deformation regularization loss. Defined on the TPS deformable mesh, it is used to constrain the deformation range of the garment to maintain the garment texture and avoid excessive deformation;

[0021] The expression is:

[0022]

[0023] In the formula, p represents a control point on the TPS deformable mesh, P represents the set of control points on the TPS deformable mesh, and G... x (p) and G y (p) represent the x-axis and y-axis coordinates of the TPS deformed mesh at control point p, respectively; p0, p1, p2, and p3 represent the control points adjacent to control point p on the left, right, top, and bottom sides, respectively; G x (p0) and G x (p1) represents the x-axis coordinates of the TPS deformed mesh at control points p0 and p1, respectively; G y (p2) and G y (p3) represents the y-axis coordinates of the TPS deformed mesh at control points p2 and p3, respectively.

[0024] In step two, the arm region preservation module performs three operations: deformation of the clothing mask dilation and inversion, removal of residual pixels from the original clothing in the arm region, and acquisition of the corrected clothing-independent human body region. The final result is a corrected clothing-independent human body representation, specifically:

[0025] S1: Deformed Clothing Mask Inflation and Reversal: Using a 15*15 structuring element to inflate and invert the deformed clothing mask M obtained from the clothing deformation module. w A morphological dilation operation is performed to expand the deformable clothing mask outwards. Then, the background and foreground regions of the expanded deformable clothing mask are inverted to obtain the mask M for the region outside the deformable clothing. b =(1-dilate(M) w ,(15,15)));

[0026] S2: Removal of residual pixels from the original clothing in the arm region: First, remove the mask M corresponding to the original clothing region in the human body image I. c Perform mask expansion and flipping operations to obtain the mask M′ of the area outside the original clothing. b =(1-dilate(M) c ,(15,15))), will M′ b Arm region I in human body image I arm Multiplying by pixels yields the arm region I′, after removing residual pixels from the original clothing. arm To avoid residual pixels from the original clothing affecting the try-on generation process;

[0027] S3: Obtain the corrected clothing-independent human body region: for the arm region I′ after removing residual pixels of the original clothing. arm Mask M for areas other than the transforming costume b Perform a pixel-wise multiplication operation to obtain the arm region I″ unrelated to the clothing being tried on. arm :

[0028] I″arm =M b ⊙I′ arm

[0029] In the formula, ⊙ represents the Hadamard product; a Hadamard product is created with a width equal to the input image and a height of H. l A barcode M with random pixel values ​​and positions line And for the arm area unrelated to the clothing being tried on, I″ arm The removal process yields the final retained arm region, which is then compared with the body, head, and lower garment regions in human image I. bp The combination yields corrected clothing-independent human body regions I′ bp :

[0030]

[0031] In the formula, This represents the addition at the pixel level;

[0032] The clothing-independent human representation h obtained from step one preprocessing. p The head region extracted from human image I is replaced with a corrected, clothing-independent human body region I′. bp The corrected clothing-independent human body representation h′ is obtained. p .

[0033] In step two, the try-on generation module includes: an encoder, a decoder, and a mask synthesis module;

[0034] The encoder contains four downsampling convolutional layers with a stride of 2 and two convolutional layers with a stride of 1, with filter counts of 64, 128, 256, 512, 512, and 512, respectively.

[0035] The decoder contains six upsampling layers consisting of nearest neighbor interpolation and 1*1 convolutions. Each upsampling layer is followed by a MobileVit block. The number of filters in the upsampling layers are 512, 512, 256, 128, 64, and 4, respectively. The MobileVit block consists of three parts: local representation, global representation, and fusion.

[0036] The process of the try-on generation module is as follows: The corrected clothing-independent human body representation h′ is generated. p With the deformable costumes stacked in the passageway C w and the transformation clothing mask M wThe encoder is input to extract multi-level features; the last level of features is input to the decoder, and multi-level features are passed between the encoder and decoder through skip connections to better capture information that is easily lost during downsampling (this invention improves the decoder's ability to acquire global information by stacking MobileVit blocks after each upsampling layer of the decoder, thereby improving the quality of the generated results); the decoder outputs the predicted mapping mask M. o and rendered character images I r Finally, perform mask composition: using a pattern mask M. o Transformable Clothing C w With the rendered character image I r Combined together, the final fitting result I is obtained. o :

[0037] I o =M o ⊙C w +(1-M o )⊙I r

[0038] The loss function used to train the try-on generation module is constructed as follows:

[0039]

[0040] In the formula, λ M , λ p , λ VGG and λ FFL For loss The corresponding hyperparameters; Used to encourage composition mask M o The mask M that more closely resembles the original clothing area in human image I c Loss on pixels loss at the feature level Used to reduce the gap between the generated result and the real image in the spatial domain, φ i (I o ) and φ i (I) represent the final fitting results. o The feature map of human image I in the i-th layer of the VGG network, λ i Indicates the corresponding weights; focal frequency loss Further improve generation quality by optimizing frequency domain spacing; focus frequency loss The expression is:

[0041]

[0042] In the formula, M F N FLet (u, v) represent the width and height of the image, respectively, and let (u, v) represent the spatial frequency coordinates on the spectrum; F r (u,v) represents the frequency value of the real image at coordinates (u,v), F g (u,v) represents the frequency value of the generated image at coordinates (u,v). representing the frequency distance between the real image and the generated image, w(u,v)=|F r (u,v)-F g (u,v)| α Let α represent the spectral weighting matrix, where α is a scaling factor for flexibility. The spectral weighting matrix allows the loss to be adaptively focused on the difficult-to-handle frequency components.

[0043] In step three, the paired image training set consists of multiple images of people and images of the clothing worn by the people. During the training process, the Adam optimizer is used to optimize the network weights during backpropagation. The clothing deformation module is trained for 200,000 generations. After obtaining the corrected clothing-independent human representation through the arm region preservation module, the try-on generation module is trained for another 200,000 generations.

[0044] In step four, the unpaired image test set consists of multiple images of people and images of clothing being tried on.

[0045] The beneficial effects of this invention are:

[0046] This invention proposes a try-on network employing an arm region preservation strategy, improved upon by combining CP-VTON+. The arm region preservation module flexibly preserves the arm region from the original human image, constructing a corrected, clothing-independent human representation that more reasonably retains arm features. By combining the try-on generation module with MobileVit and introducing focus frequency loss, the quality of the generated try-on results is improved. Validation on commonly used datasets demonstrates that this invention achieves more realistic results in terms of human region features and the natural fit of clothing to the body. Attached Figure Description

[0047] Figure 1 This is a flowchart of the method of the present invention;

[0048] Figure 2 This is a diagram of the virtual try-on network structure constructed in this invention, which employs an arm region preservation strategy.

[0049] Figure 3 This is the structure of the MobileVit block in the try-on generation module of the present invention;

[0050] Figure 4A comparison chart of the fitting results of this invention with the current advanced methods CP-VTON+ and ACGPN when the first person tries on a top;

[0051] Figure 5 A comparison chart of the fitting results of this invention with the current advanced methods CP-VTON+ and ACGPN when a second person tries on a top;

[0052] Figure 6 A comparison chart of the fitting results of this invention with current advanced methods CP-VTON+ and ACGPN when a third person tries on a top.

[0053] Figure 7 A comparison chart of the fitting results of this invention with current advanced methods CP-VTON+ and ACGPN when a top is tried on by the fourth person. Detailed Implementation

[0054] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0055] This invention provides a virtual try-on method employing an arm area preservation strategy, such as... Figure 1-3 As shown, please follow these steps:

[0056] Step 1: Preprocessing. Preprocess the human body image I and the clothing image C separately to obtain a clothing-independent human body representation h. p The preprocessing result is obtained by using the mask M corresponding to the clothing image C;

[0057] Step one specifically involves: preprocessing the human body image I to obtain a clothing-independent human body representation h. p The specific preprocessing process is as follows: using the OpenPose pose estimator to estimate the pose of the person in human image I to obtain a pose heatmap; using the LIP human body parser to segment the person in human image I to obtain a body shape mask; and using the LIP human body parser to obtain the head region from human image I. For clothing image C, preprocessing is performed to obtain the clothing mask M corresponding to C. The specific preprocessing process is as follows: using the LIP human body parser to segment the clothing in image C to obtain the clothing mask M corresponding to C.

[0058] Step 2: Construct a virtual try-on network that employs an arm region preservation strategy, and construct a loss function for network training;

[0059] In step two, a virtual try-on network based on an arm region preservation strategy is constructed. This network comprises three modules: a clothing deformation module, an arm region preservation module, and a try-on generation module. The main goal of the clothing deformation module is to deform the clothing to fit the human body. The main goal of the arm region preservation module is to adaptively preserve the human arm region, obtaining a corrected clothing-independent human representation. The goal of the try-on generation module is to generate a try-on result image based on the deformed clothing and the corrected clothing-independent human representation.

[0060] The garment deformation module includes two encoders, a correlation matching layer Co, a regressor R, and a deformation module W;

[0061] Each encoder contains four downsampling convolutional layers with a stride of 2 and two convolutional layers with a stride of 1, with filter counts of 64, 128, 256, 512, 512, and 512, respectively.

[0062] The relevant matching layer Co includes normalizing two sets of feature representations of the input to the unit L2 norm by channel and flattening them into matrices. At each position, the flattened matrices obtained from the two sets of features are multiplied to obtain the correlation map.

[0063] The regressor R consists of two downsampling convolutional layers with a stride of 2, two convolutional layers with a stride of 1, and a fully connected output layer with an output size of 50. The number of filters in the downsampling convolutional layers and the convolutional layers are 512, 256, 128, and 64, respectively.

[0064] The processing procedure of the clothing deformation module is as follows: input a clothing-independent human body representation h p The clothing mask M corresponding to the clothing image C is used by two encoders to represent the clothing-independent human body h. p Feature extraction is performed on the clothing mask M to obtain two sets of features. The correlation matching layer Co combines these two sets of features into a correlation map, which is then input into a regressor to predict the deformation parameter θ. The deformation module W, based on the TPS transform, performs deformation processing on the clothing image C and the clothing mask M according to the deformation parameter θ, resulting in the deformed clothing C. w With the transformation clothing mask M w ;

[0065] The loss function used to train the clothing deformation module is constructed as follows: Loss on pixels and deformation regularization loss Composition:

[0066]

[0067] In the formula, λreg and λ c Loss on pixels and deformation regularization loss The corresponding hyperparameters; pixel loss Defined as transforming clothing C w Clothing area I in human body image I C The L1 norm of the model is used to encourage the network to acquire deformable clothing that approximates the truth; deformation regularization loss. Defined on the TPS deformable mesh, it is used to constrain the deformation range of the garment to maintain the garment texture and avoid excessive deformation;

[0068] The expression is:

[0069]

[0070] In the formula, p represents a control point on the TPS deformable mesh, P represents the set of control points on the TPS deformable mesh, and G... x (p) and G y (p) represent the x-axis and y-axis coordinates of the TPS deformed mesh at control point p, respectively; p0, p1, p2, and p3 represent the control points adjacent to control point p on the left, right, top, and bottom sides, respectively; G x (p0) and G x (p1) represents the x-axis coordinates of the TPS deformed mesh at control points p0 and p1, respectively; G y (p2) and G y (p3) represents the y-axis coordinates of the TPS deformable mesh at control points p2 and p3, respectively;

[0071] The arm region preservation module performs three steps: expanding and inverting the deformed clothing mask, removing residual pixels from the original clothing in the arm region, and obtaining the corrected clothing-independent human body region. The final result is a corrected clothing-independent human body representation, specifically:

[0072] S1: Deformed Clothing Mask Inflation and Reversal: Using a 15*15 structuring element to inflate and invert the deformed clothing mask M obtained from the clothing deformation module. w A morphological dilation operation is performed to expand the deformable clothing mask outwards. Then, the background and foreground regions of the expanded deformable clothing mask are inverted to obtain the mask M for the region outside the deformable clothing. b =(1-dilate(M) w ,(15,15)));

[0073] S2: Removal of residual pixels from the original clothing in the arm region: First, remove the mask M corresponding to the original clothing region in the human body image I. cPerform mask expansion and flipping operations to obtain the mask M′ of the area outside the original clothing. b =(1-dilate(M) c ,(15,15))), will M′ b Arm region I in human body image I arm Multiplying by pixels yields the arm region I′, after removing residual pixels from the original clothing. arm To avoid residual pixels from the original clothing affecting the try-on generation process;

[0074] S3: Obtain the corrected clothing-independent human body region: for the arm region I′ after removing residual pixels of the original clothing. arm Mask M for areas other than the transforming costume b Perform a pixel-wise multiplication operation to obtain the arm region I″ unrelated to the clothing being tried on. arm :

[0075] I″ arm =M b ⊙I′ arm

[0076] In the formula, ⊙ represents the Hadamard product; a Hadamard product is created with a width equal to the input image and a height of H. l A barcode M with random pixel values ​​and positions line And for the arm area unrelated to the clothing being tried on, I″ arm The removal process yields the final retained arm region, which is then compared with the body, head, and lower garment regions in human image I. bp The combination yields corrected clothing-independent human body regions I′ bp :

[0077]

[0078] In the formula, This represents the addition at the pixel level;

[0079] The clothing-independent human representation h obtained from step one preprocessing. p The head region extracted from human image I is replaced with a corrected, clothing-independent human body region I′. bp The corrected clothing-independent human body representation h′ is obtained. p .

[0080] The try-on generation module includes: an encoder, a decoder, and a mask synthesis module;

[0081] The encoder contains four downsampling convolutional layers with a stride of 2 and two convolutional layers with a stride of 1, with filter counts of 64, 128, 256, 512, 512, and 512, respectively.

[0082] The decoder contains six upsampling layers consisting of nearest neighbor interpolation and 1*1 convolutions. Each upsampling layer is followed by a MobileVit block. The number of filters in the upsampling layers are 512, 512, 256, 128, 64, and 4, respectively. The MobileVit block consists of three parts: local representation, global representation, and fusion.

[0083] The process of the try-on generation module is as follows: The corrected clothing-independent human body representation h′ is generated. p With the deformable costumes C stacked in the aisle w and the transformation clothing mask M w The encoder is input to extract multi-level features; the last level of features is input to the decoder, and multi-level features are passed between the encoder and decoder through skip connections to better capture information that is easily lost during downsampling (this invention improves the decoder's ability to acquire global information by stacking MobileVit blocks after each upsampling layer of the decoder, thereby improving the quality of the generated results); the decoder outputs the predicted mapping mask M. o and rendered character images I r Finally, perform mask composition: using a pattern mask M. o Transformable Clothing C w With the rendered character image I r Combined together, the final fitting result I is obtained. o :

[0084] I o =M o ⊙C w +(1-M o )⊙I r

[0085] The loss function used to train the try-on generation module is constructed as follows:

[0086]

[0087] In the formula, λ M , λ p , λ VGG and λ FFL For loss The corresponding hyperparameters; Used to encourage composition mask M o The mask M that more closely resembles the original clothing area in human image I c Loss on pixels loss at the feature level Used to reduce the gap between the generated result and the real image in the spatial domain, φ i (I o ) and φi (I) represent the final fitting results. o The feature map of human image I in the i-th layer of the VGG network, λ i Indicates the corresponding weights; focal frequency loss Further improve generation quality by optimizing frequency domain spacing; focus frequency loss The expression is:

[0088]

[0089] In the formula, M F N F Let (u, v) represent the width and height of the image, respectively, and let (u, v) represent the spatial frequency coordinates on the spectrum; F r (u,v) represents the frequency value of the real image at coordinates (u,v), F g (u,v) represents the frequency value of the generated image at coordinates (u,v). representing the frequency distance between the real image and the generated image, w(u,v)=|F r (u,v)-F g (u,v)| α Let α represent the spectral weighting matrix, where α is a scaling factor for flexibility. The spectral weighting matrix allows the loss to be adaptively focused on the difficult-to-handle frequency components.

[0090] Step 3: Use the paired image training set images and the preprocessing results obtained in Step 1 to train the virtual try-on network with the arm region preservation strategy, and obtain the weights of the virtual try-on network with the arm region preservation strategy.

[0091] In step three, the paired image training set consists of multiple images of people and images of the clothing worn by the people. During the training process, the Adam optimizer is used to optimize the network weights during backpropagation. The clothing deformation module is trained for 200,000 generations. After obtaining the corrected clothing-independent human representation through the arm region preservation module, the try-on generation module is trained for another 200,000 generations.

[0092] Step 4: The unpaired image test set consists of multiple images of people and images of clothing to be tried on. After the preprocessing operation in Step 1, the images of people and clothing to be tried on in the unpaired image test set are used as input. Based on the virtual try-on network with an arm region preservation strategy and network weights, realistic try-on images are generated to achieve the virtual try-on effect of transferring clothing to the corresponding area of ​​the human body in the image.

[0093] Example 1

[0094] Case Analysis and Method Validation

[0095] This invention was tested on the VITON dataset. This dataset contains 16,253 pairs of images of women's frontal photos and their tops. The training set contains 14,221 image pairs, and the test set contains 2,032 image pairs, with an image resolution of 256×192. The training set image pairs are paired images, and the test set images are used to evaluate the network's reconstruction ability after being paired, and to evaluate the network's performance on the virtual try-on task after being unpaired.

[0096] This invention is implemented based on the PyTorch framework and trained using a single NVIDIA Tesla V100 32GB GPU. The Adam optimizer is used for optimization during training. The clothing deformation module and the try-on generation module are each trained for 200,000 generations, with a learning rate set to 0.0001 and linearly decaying to 0 after 100,000 generations. The batch size is set to 4.

[0097] Quantitative comparisons were performed using structural similarity (SSIM), learned perceptual image patch similarity (LPIPS), Frechet inception distance (FID), and kernel inception distance (KID). This demonstrates that the present invention can achieve a more realistic virtual try-on effect and has advantages in terms of parameter count and computational cost.

[0098] The comparison results are shown in Table 1 and Table 2.

[0099] Table 1. Quantitative Comparison Table

[0100]

[0101] Table 2 Comparison of Time Cost and Computation Cost

[0102]

[0103] The application results in virtual try-on tasks are as follows: Figure 4-7 As shown.

[0104] In this invention, the arm region preservation module flexibly preserves the arm region in the original human image, constructing a corrected clothing-independent human representation, thus more reasonably preserving arm features. The combination of the try-on generation module and MobileVit, along with the introduction of focus frequency loss, improves the quality of the generated try-on results. In summary, this invention achieves more realistic results in terms of preserving human region features and ensuring a natural fit between clothing and the human body, proving the applicability of this method.

[0105] Example 2

[0106] The virtual try-on method employing an arm area preservation strategy is characterized by the following steps:

[0107] Step 1: Preprocess the human body image I and the clothing image C respectively to obtain a clothing-independent human body representation h. p The preprocessing result is obtained by using the mask M corresponding to the clothing image C;

[0108] Step one specifically involves: preprocessing the human body image I to obtain a clothing-independent human body representation h. p The specific preprocessing process is as follows: using the OpenPose pose estimator to estimate the pose of the person in human image I to obtain a pose heatmap; using the LIP human body parser to segment the person in human image I to obtain a body shape mask; and using the LIP human body parser to obtain the head region from human image I. For clothing image C, preprocessing is performed to obtain the clothing mask M corresponding to C. The specific preprocessing process is as follows: using the LIP human body parser to segment the clothing in image C to obtain the clothing mask M corresponding to C.

[0109] Step 2: Construct a virtual try-on network that employs an arm region preservation strategy, and construct a loss function for network training;

[0110] Step 3: Use the paired image training set images and the preprocessing results obtained in Step 1 to train the virtual try-on network with the arm region preservation strategy, and obtain the weights of the virtual try-on network with the arm region preservation strategy.

[0111] Step 4: After preprocessing the images of people and clothing in the unpaired image test set in Step 1, use the obtained preprocessing results as input to generate realistic try-on images based on the virtual try-on network with an arm region preservation strategy and network weights, so as to achieve the virtual try-on effect of transferring clothing to the corresponding area of ​​the human body in the image.

[0112] Example 3

[0113] The virtual try-on method employing an arm area preservation strategy is characterized by the following steps:

[0114] Step 1: Preprocess the human body image I and the clothing image C respectively to obtain a clothing-independent human body representation h. p The preprocessing result is obtained by using the mask M corresponding to the clothing image C;

[0115] Step 2: Construct a virtual try-on network that employs an arm region preservation strategy, and construct a loss function for network training;

[0116] Step 3: Use the paired image training set images and the preprocessing results obtained in Step 1 to train the virtual try-on network with the arm region preservation strategy, and obtain the weights of the virtual try-on network with the arm region preservation strategy.

[0117] Step 4: After preprocessing the images of people and clothing in the unpaired image test set in Step 1, use the obtained preprocessing results as input to generate realistic try-on images based on the virtual try-on network with an arm region preservation strategy and network weights, so as to achieve the virtual try-on effect of transferring clothing to the corresponding area of ​​the human body in the image.

Claims

1. A virtual try-on method employing an arm area preservation strategy, characterized in that, The specific steps are as follows: Step 1: Analyze the human body images separately. Images of clothing Preprocessing is performed to obtain clothing-independent human representations. Images of clothing Corresponding mask Obtain the preprocessing results; Step 2: Construct a virtual try-on network that employs an arm region preservation strategy, and construct a loss function for network training; In step two, a virtual try-on network based on an arm region preservation strategy is constructed. This virtual try-on network consists of three modules: a clothing deformation module, an arm region preservation module, and a try-on generation module. The clothing deformation module is used to deform the clothing to fit the human body. The arm region preservation module is used to adaptively preserve the human arm region and obtain a corrected clothing-independent human body representation. The try-on generation module is used to generate a try-on result image from the deformed clothing and the corrected clothing-independent human body representation. In step two, the arm region preservation module performs three operations: deformation of the clothing mask dilation and inversion, removal of residual pixels from the original clothing in the arm region, and acquisition of the corrected clothing-independent human body region. The final result is a corrected clothing-independent human body representation, specifically: S1: Deformation Clothing Mask Inflation and Reversal: Using 15 The structuring element 15 is used to obtain the deformed clothing mask from the clothing deformation module. Perform morphological expansion operations to expand the deformable clothing mask outwards; Then, invert the background and foreground areas of the expanded deformed clothing mask to obtain the mask for the area outside the deformed clothing. ; S2: Removal of residual pixels from original clothing in the arm area: First, the human body image... The mask corresponding to the original clothing area in the image. Perform mask expansion and flipping operations to obtain the mask of the area outside the original clothing. ,Will With human body images arm area Multiplying by pixels yields the arm area after removing residual pixels from the original clothing. To avoid residual pixels from the original clothing affecting the try-on generation process; S3: Obtain corrected clothing-independent human body regions: for arm regions where residual pixels of the original clothing have been removed. Area mask outside of transforming costumes Perform pixel-wise multiplication to obtain the arm area unrelated to the clothing being tried on. : In the formula, Represent the Hadamard product; create a Hadamard product with a width equal to the input image and a height of [missing information]. A barcode with random pixel values ​​and positions And for the arm area unrelated to the clothing being tried on. The removal process is performed to obtain the final preserved arm region, which is then compared with the human body image. The body, head, and lower garment areas The combination corrects clothing-independent human body areas : In the formula, This represents the addition at the pixel level; The clothing-independent human representation obtained from step one preprocessing. From human images The extracted head region is replaced with a modified, clothing-independent human body region. Corrected clothing-independent human anatomy representation ; Step 3: Use the paired image training set images and the preprocessing results obtained in Step 1 to train the virtual try-on network with the arm region preservation strategy, and obtain the weights of the virtual try-on network with the arm region preservation strategy. Step 4: After preprocessing the images of people and clothing in the unpaired image test set in Step 1, use the obtained preprocessing results as input to generate realistic try-on images based on the virtual try-on network with an arm region preservation strategy and network weights, so as to achieve the virtual try-on effect of transferring clothing to the corresponding area of ​​the human body in the image.

2. The virtual try-on method using an arm area preservation strategy according to claim 1, characterized in that, Step one specifically involves: for human body images Preprocessing is performed to obtain clothing-independent human representations. The specific preprocessing process is as follows: using the OpenPose pose estimator to process human images. Pose heatmaps obtained from pose estimation of human figures, and human images processed using the LIP human body parser. Body shape masks obtained from image segmentation of human figures, and LIP human body parsing from human images. Obtain the head region from the image; for clothing images Preprocessing is performed to obtain Corresponding clothing mask The specific preprocessing process is as follows: using a LIP human body parser to analyze the image. Image segmentation of Chinese clothing is obtained Corresponding clothing mask .

3. The virtual try-on method using an arm area preservation strategy according to claim 1, characterized in that, In step two, the clothing deformation module contains two encoders and a related matching layer. , regressor Deformation module ; Each encoder contains four downsampling convolutional layers with a stride of 2 and two convolutional layers with a stride of 1, with filter counts of 64, 128, 256, 512, 512, and 512, respectively. Related Matching Layer The process involves normalizing two sets of feature representations from the input to the unit L2 norm according to the channels and flattening them into matrices. At each position, a dot product operation is performed on the flattened matrices obtained from the two sets of features to obtain the correlation map. Regressor It contains two downsampling convolutional layers with a stride of 2, two convolutional layers with a stride of 1, and a fully connected output layer with an output size of 50. The number of filters in the downsampling convolutional layers and the convolutional layers are 512, 256, 128, and 64, respectively. The processing procedure of the clothing transformation module is as follows: input a clothing-independent human body representation. Images of clothing Corresponding clothing mask The two encoders represent the human body regardless of the clothing. With clothing mask After feature extraction and obtaining two sets of features, the relevant matching layer... The two sets of features are combined into a correlation plot, which is then input into a regressor to predict the deformation parameters. Deformation module Based on TPS transformation and deformation parameters Images of clothing and clothing mask Deformation treatment is applied to obtain deformable clothing. With deformable clothing mask ; The loss function used to train the clothing deformation module is constructed as follows: , Loss on pixels and deformation regularization loss Composition: In the formula, and Loss on pixels and deformation regularization loss The corresponding hyperparameters; pixel loss Defined as transforming clothing With human body images Clothing area The L1 norm of the model is used to encourage the network to acquire deformable clothing that approximates the truth; deformation regularization loss. Defined on the TPS deformable mesh, it is used to constrain the deformation range of the garment to maintain the garment texture and avoid excessive deformation; The expression is: In the formula, This represents a control point on the TPS deformable mesh. This represents the control point set of the TPS deformable mesh. and These represent the TPS deformed mesh at the control points. place Axis coordinates and Axis coordinates; , , , Representing control points respectively Adjacent control points on the left, right, top, and bottom sides; and These represent the TPS deformed mesh at the control points. and place Axis coordinates; and These represent the TPS deformed mesh at the control points. and place Axis coordinates.

4. The virtual try-on method using an arm area preservation strategy according to claim 1, characterized in that, In step two, the try-on generation module includes: an encoder, a decoder, and a mask synthesis module; The encoder contains four downsampling convolutional layers with a stride of 2 and two convolutional layers with a stride of 1, with filter counts of 64, 128, 256, 512, 512, and 512, respectively. The decoder contains six values ​​derived from nearest neighbor interpolation and 1. One convolutional layer forms an upsampling layer, and each upsampling layer is followed by a MobileVit block. The number of filters in the upsampling layers are 512, 512, 256, 128, 64, and 4, respectively. The MobileVit block consists of three parts: local representation, global representation, and fusion. The process of the try-on generation module is as follows: The corrected clothing-independent human body representation is... With deformable costumes stacked in the aisle and deformable clothing mask The encoder is input to extract multi-level features; the last level of features is input to the decoder, and multi-level features are passed between the encoder and decoder through skip connections to better capture information that is easily lost during downsampling; the decoder outputs the predicted mapping mask. and rendered character images Finally, perform mask composition: using a pattern mask. Transformable clothing With rendered character images The pieces are combined to obtain the final fitting result. : The loss function used to train the try-on generation module is constructed as follows: : In the formula, , , and For loss , , , The corresponding hyperparameters; Used to encourage composition masks More closely resembles human images The mask corresponding to the original clothing area in the image. Loss on pixels loss at the feature level Used to narrow the gap between the generated result and the real image in the spatial domain. and These represent the final fitting results. With human body images Feature map of layer i in the VGG network Indicates the corresponding weights; focal frequency loss Further improve generation quality by optimizing frequency domain spacing; focus frequency loss The expression is: In the formula, , These represent the width and height of the image, respectively. Coordinates representing spatial frequencies on the spectrum; Represents the true image in coordinates Frequency value at that location, Indicates the generated image in coordinates Frequency value at that location, This represents the frequency distance between the real image and the generated image. Represents the spectral weight matrix. It is a scaling factor for flexibility; the spectral weighting matrix allows the loss to adaptively focus on the difficult-to-handle frequency components.

5. The virtual try-on method using an arm area preservation strategy according to claim 4, characterized in that, In step three, the paired image training set consists of multiple images of people and images of the clothing worn by the people; during training, the Adam optimizer is used to optimize the network weights during backpropagation. The clothing deformation module is trained for 200,000 generations. After obtaining the corrected clothing-independent human representation through the arm region preservation module, the try-on generation module is trained for another 200,000 generations.

6. The virtual try-on method using an arm area preservation strategy according to claim 5, characterized in that, In step four, the unpaired image test set consists of multiple images of people and images of clothing being tried on.

Images