Virtual fitting method across body poses based on global appearance flow and deformable convolution technique

Abstract

The present application belongs to the technical field of virtual fitting, and relates to a cross-attitude virtual fitting method based on global appearance flow and variable convolution technology. * and M * ; based on the labels, feature maps are extracted and global style vectors S are calculated; attention maps and garment appearance flows are generated using S and the feature maps; through high-dimensional feature sampling and Decoder restoration, an image M wc of the model's clothes on the target person is obtained, while the reserved area T p and the to-be-generated area T m of the target person are determined; M wc , T p , and T m are input into a texture conversion network to generate a virtual fitting effect picture T tryon . The present application effectively deals with the case that the model picture and the target person picture are not in the same receptive field.

Description

Technical Field

[0001] The present invention belongs to the technical field of virtual fitting, and relates to a cross-body virtual fitting method based on global appearance flow and variable convolution technology. Background Art

[0002] 2D virtual fitting technology leverages computer vision and graphics principles to simulate the clothing fitting process on a two-dimensional surface, providing users with a shopping experience that allows them to preview the effects of different styles, colors, and sizes without actually trying them on. This technology primarily involves two types of fitting: fitting only tops and fitting the entire garment. It involves two key stages: garment distortion and texture conversion.

[0003] In the clothing distortion stage, traditional methods such as Thin Plate Spline (TPS) warping technology can handle a certain degree of clothing deformation, but are limited in their ability to handle small-scale deformation and are difficult to handle complex situations such as occlusion and dislocation. To this end, dense appearance flow technology (such as Figure 1 By capturing the overall deformation information of clothing, it can more accurately estimate and adjust the shape of clothing to adapt to target people with different body shapes and postures, thus effectively overcoming the limitations of traditional methods.

[0004] The texture conversion stage is responsible for fusing the distorted clothing with the target person's image to generate the final fitting effect image. Currently, generative adversarial networks (GANs) are a common technique for this stage. They can learn the texture mapping relationship between clothing and the target person based on training data. However, when the model image and the target person image are not within the same receptive field (i.e., when their corresponding parts are not located in the same local receptive field of the neural network), existing methods often fail to effectively align them, resulting in distorted or misaligned fitting effects.

[0005] While style-based global appearance flow technology has improved the results of top-only try-on methods to some extent, its practical application still faces challenges. This method requires a frontal image of a single top as input, which is not easily available online, limiting its practical application.

[0006] For the whole clothing fitting method, some studies have adopted a combination of TPS transformation and DF-guided Dense Warping (such as Figure 2 However, these methods are also limited by the local receptive field and cannot effectively deal with situations where the model image and the target person image are not within the same receptive field. In addition, these methods are also unable to handle complex deformations and occlusions. Summary of the Invention

[0007] The purpose of the present invention is to solve the problems existing in the prior art and provide a cross-body virtual fitting method based on global appearance flow and variable convolution technology.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0009] A cross-body virtual fitting method based on global appearance flow and variable convolution technology includes the following steps:

[0010] (1) Pretreatment;

[0011] Preprocess the target person image T to obtain the target person’s label T * , T * Including the target person's head mask T h , target person's body key points T k , target person densepose descriptor T d ;

[0012] Preprocess the model image M to obtain the model label M * , M * Including the clothes to be tried on the model M c , Model body key point M k , model densepose descriptor M p ;

[0013] (2) T * Input it into the pyramid feature extraction network PFEN for processing to obtain the feature map pv, v = 1, 2, ..., n, n is the number of layers of the pyramid feature extraction network PFEN, and M * Input into the pyramid feature extraction network PFEN for processing to obtain the feature map gv, v = 1, 2, ..., n, where M k 、M p Inputting into the pyramid feature extraction network PFEN can effectively avoid network overfitting;

[0014] (3) Obtain the global style vector S based on pn and gn (both of which contain the global human posture information required for clothing distortion);

[0015] (4) Based on pv, gv, S, the attention map m is obtained through n GL-AFWM modules n Japanese clothing appearance flow GL fn , the specific steps are as follows:

[0016] (4.1) First, input S, pn, gn into the modulated convolution network (Modulated Convolution Net) to obtain the clothing appearance flow G f1 , then G f1 The result of sampling gn is input into the deformable convolution network (Deformable ConvolutionNet) to obtain the attention map m1 and clothing appearance flow L f1 , and finally use L f1 Sampling G f1 Get the clothing appearance flow GL predicted by the first GL-AFWM module f1 ;

[0017] (4.2) Let i=n-1, m=n+1-i, p=m-1;

[0018] (4.3) First, GL fp The result of sampling gi, S, pi is input into the modulated convolutional network to obtain the clothing appearance flow Gf m , then Gf m Sample gi and then m p The multiplication result and pi are input into the variable convolution network to obtain the attention map m m and clothing appearance flow Lf m , and finally use Lf m Sampling Gf m Get the clothing appearance flow GL predicted by the mth GL-AFWM module fm ;

[0019] (4.4) Determine whether i=1 is true. If so, end the process. The last attention map m is obtained. m That is the attention map m n , the last clothing appearance flow GL fm That is clothing appearance flow GL fn Otherwise, let i=i-1 and return to step (4.3);

[0020] (5) Based on m n GL fn To M c The high-dimensional feature sampling is restored by Decoder to obtain M wc , based on T h 、T d , T gets the target person reserved area T p , based on T d , T gets the area to be generated T m ; M wc 、T p 、T m Input into the texture conversion network to generate the final effect image Ttryon .

[0021] As the preferred technical solution:

[0022] In the above-mentioned cross-body virtual fitting method based on global appearance flow and variable convolution technology, in step (2), n=3 or 5.

[0023] In the above-mentioned cross-body virtual fitting method based on global appearance flow and variable convolution technology, the specific steps of step (3) are as follows:

[0024] (3.1) Reconstruct the dimension of pn to obtain F pn , for F pn Transpose to get Will After a linear layer with a weight matrix W, we get Reconstruct the dimension of gn to get F gn ;

[0025] (3.2) Construct the correlation matrix Corr, which is as follows:

[0026]

[0027] (3.3) Use the softmax function to normalize the correlation matrix and obtain the attention map Atten between pn and gn. Each value in the x-row and y-column of the attention map represents the spatial attention weight of the x-th position of pn and the y-th position of gn. Atten is as follows:

[0028] Atten = softmax(Corr);

[0029] (3.4) F pn The product of Atten, F gn The product of Atten is connected in series to obtain S. This process can more effectively capture the global human posture information required for clothing distortion and improve the model's ability to model global features. p3 and F g3 The purpose of multiplication is to make F p3 and F g3 The information in focuses on regions that show high spatial correlation between the two.

[0030] The above process is to input pn and gn (both of which contain the global human posture information required for clothing distortion) into the VTON Co-Attention attention network for processing to obtain the global style vector S, where the VTON Co-Attention attention network is a Co-Attention attention network after the conv3d that processes three-dimensional data is replaced by the conv2d that processes two-dimensional data. Conv2d is a two-dimensional convolutional layer, and its input is usually a four-dimensional tensor, which contains the batch size (N), the number of channels (C), the height (H) and the width (W);

[0031] The present invention transfers the Co-Attention network used in the field of medical image registration to process three-dimensional medical data and applies it to the field of virtual fitting that processes two-dimensional data. The role of the Co-Attention network in the existing technology is to improve feature extraction and unsupervised motion tracking by focusing on the spatial and temporal features between different time frames, helping the model learn inter-frame dependent features, thereby better performing image registration and motion tracking. In contrast, its role in the present invention is to focus on the spatial features between the model's clothing feature map and the target person's feature map, optimize the global style vector, make it pay more attention to the correlation between different receptive fields, and thus better guide the network to distort and deform clothing.

[0032] As described above, a cross-body virtual fitting method based on global appearance flow and variable convolution technology, the specific process of step (3) is: A is obtained from pn pn , and A is obtained from gn gn After that, A pn and A gn Connect in series to get S;

[0033] From r we get A r The steps are as follows:

[0034] (a) r is passed through three 1×1 convolutional layers to obtain Q, K, and V respectively. The number of channels of Q and K is one eighth of that of r, and the number of channels of V is the same as that of r;

[0035] (b) Reconstruct the dimension of Q and transpose it to obtain proj_query T , reconstruct the dimension of K to get proj_key, and reconstruct the dimension of V to get proj_value;

[0036] (c) Using proj_query T Perform batch matrix multiplication with proj_key to obtain the energy tensor energy, and use the softmax function on the last dimension of energy to obtain the attention weight attention;

[0037] (d) Multiply attention by proj_value and reconstruct the dimension to obtain the weighted value out;

[0038] (e) Add r and out to get A r ;

[0039] Among them, r is pn, A r A pn , or r is gn, A r A gn .

[0040] As described above, a cross-body virtual fitting method based on global appearance flow and variable convolution technology, in step (5), based on m n GL fn To M c The high-dimensional feature sampling is restored by Decoder to obtain M wc Refers to: Use Encoder to extract M c The high-dimensional features of fn Sampling, the result after sampling is the same as m n Multiply and input into Decoder to get the distorted and aligned model clothing M wc .

[0041] As described above, a cross-body virtual fitting method based on global appearance flow and variable convolution technology, in step (5), based on T h 、T d , T gets the target person reserved area T p It means: T h With T d The mask values ​​of the hand and foot areas are added and multiplied by T to get T p .

[0042] As described above, a cross-body virtual fitting method based on global appearance flow and variable convolution technology, in step (5), based on T d , T to get the area to be generated T m It means: T d The mask values ​​of the arms and legs of the T are multiplied by the opposite value of the mask value of the clothes area in T to obtain T. m .

[0043] As described above, in the cross-body virtual fitting method based on global appearance flow and variable convolution technology, in step (5), M wc 、T p 、T m Input into the texture conversion network (Residual U-Net) to generate the final effect map T tryon It means: Mwc 、T p 、T m The channel layer is serially input into the U-Net residual network for training, thus obtaining T tryon .

[0044] Beneficial effects:

[0045] The present invention captures the global contextual information of the entire image by introducing a dense appearance flow prediction module (composed of a global style vector and a modulated convolutional network) with a StyleGan structure. The global style vector is extracted from the lowest-resolution feature maps of the model image and the target person image, and can contain the overall information of the entire image. In this way, even if the corresponding parts of the model image and the target person image are not in the same local receptive field, the global style vector can associate them, thereby achieving accurate alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of dense appearance flow technology;

[0047] Figure 2 Schematic diagram of the combination of TPS transformation and distance field guided dense transformation;

[0048] Figure 3 Flowchart of the cross-body virtual fitting method based on global appearance flow and variable convolution technology of the present invention;

[0049] Figure 4 This is a flowchart of the present invention for obtaining the global style vector S based on pn and gn;

[0050] Figure 5 The present invention obtains the attention map m based on pv, gv, S n Japanese clothing appearance flow GL fn Flowchart of the process. DETAILED DESCRIPTION

[0051] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content of the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0052] A cross-body virtual fitting method based on global appearance flow and variable convolution technology, such as Figure 3 The specific steps are as follows:

[0053] (1) Pretreatment;

[0054] Preprocess the target person image T to obtain the target person’s label T* , T * Including the target person's head mask T h , target person's body key points T k , target person densepose descriptor T d ;

[0055] Preprocess the model image M to obtain the model label M * , M * Including the clothes to be tried on the model M c , Model body key point M k , model densepose descriptor M p ;

[0056] (2) T * Input it into the pyramid feature extraction network PFEN for processing to obtain the feature map pv, v = 1, 2, ..., n, n is the number of layers of the pyramid feature extraction network PFEN, and M * Input into the pyramid feature extraction network PFEN for processing to obtain the feature map gv, v = 1, 2, ..., n, n can be 3 or 5;

[0057] (3) Based on pn and gn, the global style vector S is obtained, as Figure 4 As shown, the specific steps are:

[0058] (3.1) Reconstruct the dimension of pn to obtain F pn , for F pn Transpose to get Will After a linear layer with a weight matrix W, we get Reconstruct the dimension of gn to get F gn ;

[0059] (3.2) Construct the correlation matrix Corr, which is as follows:

[0060]

[0061] (3.3) Use the softmax function to normalize the correlation matrix and obtain the attention map Atten between pn and gn. Each value in the x-row and y-column of the attention map represents the spatial attention weight of the x-th position of pn and the y-th position of gn. Atten is as follows:

[0062] Atten = softmax(Corr);

[0063] (3.4) F pn The product of Atten, F gnConcatenate with the product of Atten to get S;

[0064] Alternatively, the specific process of step (3) is: obtain A from pn pn , and A is obtained from gn gn After that, A pn and A gn Connect in series to get S;

[0065] A is obtained from pn pn The steps are as follows:

[0066] (a) pn is passed through three 1×1 convolutional layers to obtain Q, K, and V respectively. The number of channels of Q and K is one eighth of that of pn, and the number of channels of V is the same as that of pn;

[0067] (b) Reconstruct the dimension of Q and transpose it to obtain proj_query T , reconstruct the dimension of K to get proj_key, and reconstruct the dimension of V to get proj_value;

[0068] (c) Using proj_query T Perform batch matrix multiplication with proj_key to obtain the energy tensor energy, and use the softmax function on the last dimension of energy to obtain the attention weight attention;

[0069] (d) Multiply attention by proj_value and reconstruct the dimension to obtain the weighted value out;

[0070] (e) Add pn and out to get A pn ;

[0071] A is obtained from gn gn The steps are as follows:

[0072] (a) Gn is passed through three 1×1 convolutional layers to obtain Q, K, and V respectively. The number of channels of Q and K is one eighth of that of Gn, and the number of channels of V is the same as that of Gn;

[0073] (b) Reconstruct the dimension of Q and transpose it to obtain proj_query T , reconstruct the dimension of K to get proj_key, and reconstruct the dimension of V to get proj_value;

[0074] (c) Using proj_query T Perform batch matrix multiplication with proj_key to obtain the energy tensor energy, and use the softmax function on the last dimension of energy to obtain the attention weight attention;

[0075] (d) Multiply attention by proj_value and reconstruct the dimension to obtain the weighted value out;

[0076] (e) Add gn and out to get A gn ;

[0077] (4) Based on pv, gv, S, the attention map m is obtained through n GL-AFWM modules n Japanese clothing appearance flow GL fn , the specific steps are as follows:

[0078] (4.1) First, input S, pn, gn into the modulated convolutional network to obtain the clothing appearance flow G f1 , then G f1 The result of sampling gn is input into the variable convolutional network to obtain the attention map m1 and clothing appearance flow L f1 , and finally use L f1 Sampling G f1 Get the clothing appearance flow GL predicted by the first GL-AFWM module f1 ;

[0079] (4.2) Let i=n-1, m=n+1-i, p=m-1;

[0080] (4.3) Figure 5 As shown, first GL fp The result of sampling gi, S, pi is input into the modulated convolutional network to obtain the clothing appearance flow Gf m , then Gf m Sample gi and then m p The multiplication result and pi are input into the variable convolution network to obtain the attention map m m and clothing appearance flow Lf m , and finally use Lf m Sampling Gf m Get the clothing appearance flow GL predicted by the mth GL-AFWM module fm ;

[0081] (4.4) Determine whether i=1 is true. If so, end the process. The last attention map m is obtained. m That is the attention map m n , the last clothing appearance flow GL fm That is clothing appearance flow GL fn Otherwise, let i=i-1 and return to step (4.3);

[0082] (5) Use Encoder to extract M c The high-dimensional features of fnSampling, the result after sampling is the same as m n Multiply and input into Decoder to get the distorted and aligned model clothing M wc , T h With T d The mask values ​​of the hand and foot areas are added and multiplied by T to get T p , T d The mask values ​​of the arms and legs of the T are multiplied by the opposite value of the mask value of the clothes area in T to obtain T. m ; M wc 、T p 、T m The channel layer is serially input into the U-Net residual network for training, thus obtaining T tryon .

Claims

1. A cross-body virtual fitting method based on global appearance flow and variable convolution technology, characterized by: The following steps are involved: (1) Pretreatment; Preprocess the target person image T to obtain the target person’s label T * , T * Including the target person's head mask T h , target person's body key points T k , target person densepose descriptor T d ; Preprocess the model image M to obtain the model label M * , M * Including the clothes to be tried on the model M c , Model body key point M k , model densepose descriptor M p ; (2) T * Input it into the pyramid feature extraction network PFEN for processing to obtain the feature map pv, v = 1, 2, ..., n, n is the number of layers of the pyramid feature extraction network PFEN, and M * Input into the pyramid feature extraction network PFEN for processing to obtain the feature map gv, v = 1, 2, ..., n; (3) Obtain the global style vector S based on pn and gn; (4) Get the attention map m based on pv, gv, S n Japanese clothing appearance flow GL fn , the specific steps are as follows: (4.1) First, input S, pn, gn into the modulated convolutional network to obtain the clothing appearance flow G f1 , then G f1 The result of sampling gn is input into the variable convolutional network to obtain the attention map m1 and clothing appearance flow L f1 , and finally use L f1 Sampling G f1 Get clothing appearance stream GL f1 ; (4.2) Let i=n-1, m=n+1-i, p=m-1; (4.3) First, GL fp The result of sampling gi, S, pi is input into the modulated convolutional network to obtain the clothing appearance flow Gf m , then Gf m Sample gi and then m p The multiplication result and pi are input into the variable convolution network to obtain the attention map m m and clothing appearance flow Lf m , and finally use Lf m Sampling Gf m Get clothing appearance stream GL fm ; (4.4) Determine whether i=1 is true. If so, end the process. The last attention map m is obtained. m That is the attention map m n , the last obtained clothing appearance flow GL fm That is clothing appearance flow GL fn Otherwise, let i=i-1 and return to step (4.3); (5) Based on m n GL fn To M c The high-dimensional feature sampling is restored by Decoder to obtain M wc , based on T h 、T d , T gets the target person reserved area T p , based on T d , T gets the area to be generated T m ; M wc 、T p 、T m Input into the texture conversion network to generate the final effect image T tryon .

2. The cross-body virtual fitting method based on global appearance flow and variable convolution technology according to claim 1, characterized in that: In step (2), n=3 or 5.

3. The cross-body virtual fitting method based on global appearance flow and variable convolution technology according to claim 1, characterized in that: The specific steps of step (3) are as follows: (3.1) Reconstruct the dimension of pn to obtain F pn , for F pn Transpose to get Will After a linear layer with a weight matrix W, we get Reconstruct the dimension of gn to get F gn ; (3.2) Construct the correlation matrix Corr, which is as follows: (3.3) Use the softmax function to normalize the correlation matrix and obtain the attention map Atten between pn and gn. Each value in the x-row and y-column of the attention map represents the spatial attention weight of the x-th position of pn and the y-th position of gn. Atten is as follows: (3.4) F pn The product of Atten, F gn Concatenated with the product of Atten, we get S.

4. The cross-body virtual fitting method based on global appearance flow and variable convolution technology according to claim 1, characterized in that: The specific process of step (3) is: obtain A from pn pn , and A is obtained from gn gn After that, A pn and A gn Connect in series to get S; From r we get A r The steps are as follows: (a) r is passed through three 1×1 convolutional layers to obtain Q, K, and V respectively. The number of channels of Q and K is one eighth of that of r, and the number of channels of V is the same as that of r; (b) Reconstruct the dimension of Q and transpose it to obtain proj_query T , reconstruct the dimension of K to get proj_key, and reconstruct the dimension of V to get proj_value; (c) Using proj_query T Perform batch matrix multiplication with proj_key to obtain the energy tensor energy, and use the softmax function on the last dimension of energy to obtain the attention weight attention; (d) Multiply attention by proj_value and reconstruct the dimension to obtain the weighted value out; (e) Add r and out to get A r ; Among them, r is pn, A r A pn , or r is gn, A r A gn .

5. The cross-body virtual fitting method based on global appearance flow and variable convolution technology according to claim 1, characterized in that: In step (5), based on m n GL fn To M c The high-dimensional feature sampling is restored by Decoder to obtain M wc Means: Use Encoder to extract M c The high-dimensional features of fn Sampling, the sampling result is the same as m n Multiply and input into Decoder to get the distorted and aligned model clothing M wc .

6. The cross-body virtual fitting method based on global appearance flow and variable convolution technology according to claim 1, characterized in that: In step (5), based on T h 、T d , T gets the target person reserved area T p It means: T h With T d The mask values ​​of the hand and foot areas are added and multiplied by T to get T p .

7. The cross-body virtual fitting method based on global appearance flow and variable convolution technology according to claim 1, characterized in that: In step (5), based on T d , T gets the area to be generated T m It means: T d The mask values ​​of the arms and legs of the T are multiplied by the opposite value of the mask value of the clothes area in T to obtain T. m .

8. The cross-body virtual fitting method based on global appearance flow and variable convolution technology according to claim 1, characterized in that: In step (5), M wc 、T p 、T m Input into the texture conversion network to generate the final effect image T tryon It means: M wc 、T p 、T m The channel layer is serially input into the U-Net residual network for training, thus obtaining T tryon .

Images