Image-based virtual fitting method and device based on explicit and implicit dual-flow clothing feature transformation

Abstract

This invention discloses an image-based virtual try-on method and apparatus based on explicit and implicit two-stream clothing feature transformation. First, a clothing network for extracting clothing features and a try-on network for generating try-on images are constructed. In the encoder stage, multi-level features are constructed based on the two networks to predict multi-scale appearance feature optical flow from clothing to the human body. In the decoder stage, explicit clothing feature warping transformation is performed using the predicted optical flow, and implicit clothing feature transformation is performed through a self-attention mechanism. Subsequently, the explicit and implicit transformation features are adaptively weighted and fused. Finally, the final virtual try-on image is output through a diffusion model denoising process and the decoder. This invention innovates in model framework, two-branch feature alignment strategy, and two-branch feature fusion strategy, effectively improving the problems of clothing deformation misalignment and texture detail distortion, and enabling the generation of high-fidelity, natural, and realistic virtual try-on images.

Description

Technical Field

[0001] This invention relates to the field of image generation, and in particular to an image virtual try-on method and apparatus based on explicit and implicit dual-stream clothing feature transformation. Background Technology

[0002] In e-commerce, virtual try-on technology aims to allow users to experience what it's like to wear specific garments without leaving home, boosting sales and reducing return rates, thus possessing significant application value. With the rapid development of generative artificial intelligence (AIGC) technology, AIGC has achieved revolutionary breakthroughs and remarkable successes in image generation, profoundly changing content creation methods and industry landscapes. Therefore, academia and industry have developed numerous image-based virtual try-on methods based on AIGC technology. Despite these positive results, existing virtual try-on technologies still face many challenges in practical applications, such as the difficulty in accurately preserving garment patterns and texture details after deformation (e.g., complex patterns and text).

[0003] Traditional virtual try-on methods are mostly based on Generative Adversarial Networks (GANs) and employ a classic deformable-re-fusion architecture. This architecture typically consists of two stages: first, a warping module predicts the appearance flow, explicitly distorting the tiled clothing image to fit the human pose; then, a generator performs pixel-level fusion between the deformed clothing image and the human image. However, this architecture has significant limitations: accurately estimating the dense, non-rigid appearance flow between clothing and the human body in high-resolution image space is extremely challenging. Explicit warping methods often struggle to handle large changes in human pose or complex limb occlusion, leading to unnatural stretching or distortion in the generated clothing.

[0004] In recent years, diffusion models have demonstrated superior performance compared to GANs in image generation, providing a new approach to solving the aforementioned problems. Existing virtual try-on methods based on diffusion models typically utilize rich generative priors from pre-trained text-to-image (T2I) models to enhance the naturalness of generated try-on images. To avoid artifacts caused by explicit distortion, most mainstream methods abandon explicit deformation processes and instead use attention mechanisms to align clothing features with the features of the person trying on the clothes, implicitly learning clothing feature transformations. Although this method improves the overall image quality, the lack of explicit geometric alignment constraints often fails to preserve fine-grained details of the clothing, resulting in texture distortion, pattern distortion, or detail deviations in the generated results, making it difficult to meet the stringent requirements of real-world scenarios.

[0005] To address the issues of detail loss and structural distortion in existing virtual try-on methods, this invention proposes an image-based virtual try-on generation method and apparatus based on explicit and implicit two-stream garment feature transformation. This method innovatively combines explicit feature transformation guided by optical flow with implicit feature transformation based on an attention mechanism: on the one hand, the explicit feature transformation branch provides precise spatial geometric guidance; on the other hand, the implicit feature transformation branch learns the flexible semantic matching relationship between clothing and the human body. By jointly learning explicit and implicit two-stream feature transformations, this invention significantly enhances the semantic and structural alignment ability between clothing and the human body, effectively preserving garment texture details while generating more accurate, natural, and realistic high-quality try-on results. Summary of the Invention

[0006] To address the aforementioned shortcomings, this invention provides a method and apparatus for generating virtual clothing try-on images based on explicit and implicit two-stream clothing feature transformation. Unlike traditional methods that rely on a single warping mode, this invention constructs an implicit two-stream clothing feature transformation architecture: on one hand, it utilizes an embedded optical flow module to predict the feature-level optical flow of clothing appearance, performing explicit geometric warping on clothing features to ensure physical alignment between clothing and human posture; on the other hand, it employs a self-attention mechanism for implicit semantic alignment to handle complex non-rigid deformations and occlusion relationships. Furthermore, an adaptive spatial gating mechanism dynamically fuses explicit and implicit features, thereby balancing structural accuracy and high-fidelity texture, achieving higher-quality and more natural virtual try-on clothing display image generation.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows: Firstly, an image-based virtual try-on method and apparatus based on explicit and implicit dual-stream clothing feature transformation, mainly comprising the following steps:

[0008] (1) Obtain a pairwise dataset containing {clothing image, clothing fitting image}, perform preprocessing, and obtain training data D1 corresponding to {clothing image, pose image, clothing mask, masked fitting image, clothing fitting image};

[0009] (2) Construct a two-branch diffusion model network model, which includes a fitting network and a clothing network; the clothing network extracts clothing features and injects them as generation conditions into the fitting network, and the fitting network extracts human body features for fitting and obtains the generated fitting image by continuously predicting the noise at each step.

[0010] (3) Construct an embedded multi-scale optical flow estimation module in the encoder of the network model, and use the multi-scale features of the virtual fitting network and the clothing network to generate a two-dimensional deformation field from clothing features to the human body to predict the feature optical flow.

[0011] (4) Construct a dual-stream feature transformation module in the decoder of the network model, including an explicit feature transformation branch and an implicit feature transformation branch, which are used to obtain explicit clothing transformation features based on feature optical flow and implicit clothing transformation features based on human body features and clothing features, respectively; fuse the results of the two branches to obtain the enhanced fused features and pass them to the next layer of the network.

[0012] (5) Use training data D1 to train the constructed model, optimize the model parameters by calculating noise prediction loss and multi-scale optical flow prediction loss, and obtain the trained virtual try-on generation model M.

[0013] (6) For the user image to be tried on and the target clothing image, after preprocessing, input them into model M and output the virtual try-on display image.

[0014] Furthermore, step (1) includes the following steps:

[0015] (1.1) Preprocess the dataset, unify the size of all images, perform format conversion and data normalization to ensure the consistency of image input;

[0016] (1.2) Use a pre-trained clothing segmentation model to extract the clothing region in the human body image, obtain the clothing segmentation mask corresponding to the original human body image, and extract the target clothing image based on the mask;

[0017] (1.3) Combine the human body image, the target clothing image and the pose image to construct a data pair (clothing image, pose image, clothing mask, masked fitting image, real clothing deformation image) to form training data D1.

[0018] Furthermore, step (2) includes the following steps:

[0019] (2.1) The virtual fitting diffusion model includes a VAE reconstruction autoencoder, a virtual fitting network and a clothing network. Both the virtual fitting network and the clothing network are built based on the U-Net network. They share the same training objective. The virtual fitting network takes the masked virtual fitting image, the clothing mask and the pose image as input, and the clothing network takes the clothing image as input.

[0020] (2.2) The VAE reconstruction variational autoencoder is an encoder-decoder structure, which includes an encoder with 22 convolutional layers and 10 residual blocks and a decoder that is symmetrical to it.

[0021] (2.3) The U-Net network adopts an encoder-decoder structure with skip connections, which includes an encoder block with 4 downsampling stages, 1 intermediate block and a decoder block with 4 upsampling stages; the main body of the network consists of stacked residual modules (ResBlocks) and spatial Transformer modules, in which the spatial Transformer module embeds a self-attention layer.

[0022] Furthermore, step (3) includes the following steps:

[0023] (3.1) The embedded multi-scale optical flow estimation module consists of a multi-level optical flow estimation block (FEB). First, the deepest-level clothing features of the clothing network and fitting network encoders are obtained. and fitting body characteristics After concatenating the two along the channel dimension, the result is input into the optical flow estimation module to obtain the initial appearance feature optical flow. The optical flow field represents the displacement of the clothing feature pixels relative to the human body feature in the horizontal and vertical directions, as shown below:

[0024]

[0025] in This represents the deepest level of clothing features in the clothing network encoder. This represents the deepest level of the virtual fitting network encoder, showing the human body features used in the virtual fitting process. Represents the corresponding number The optical flow estimation module for hierarchical feature optical flow consists of multiple neural network layers and can be simply implemented as a multilayer perceptron or a multilayer convolutional neural network.

[0026] (3.2) For each subsequent level (From 2 to N), the first Hierarchical optical flow Upsampling is performed to obtain the first Hierarchical initial feature prediction optical flow , Will Layered clothing features and fitting body characteristics After concatenation along the channel dimensions, the input is fed into the optical flow estimation module of this layer. Get the current number Hierarchical residual features predict optical flow , will the Hierarchical initial feature prediction optical flow Predicting optical flow with residual features Adding each element one by one yields the current number. Hierarchical feature prediction optical flow The specific process is defined as follows:

[0027]

[0028]

[0029] This process continues until the final appearance optical flow corresponding to the shallowest layer features of the encoder is obtained. .

[0030] (3.3) Final appearance optical flow Upsampled to the resolution of the clothing fitting image This is used for subsequent pixel-level optical flow prediction loss calculation.

[0031] Furthermore, step (4) includes the following steps:

[0032] (4.1) Construct a dual-stream feature transformation module at each level of the virtual fitting network decoder. This module includes an explicit clothing feature transformation branch and an implicit clothing feature transformation branch set in parallel, as well as an adaptive spatial gating fusion module.

[0033] (4.2) Input the features of the corresponding layers of the virtual fitting network and the clothing network decoder into the explicit clothing feature transformation branch and the implicit clothing feature transformation branch respectively, and perform the feature transformation operation in parallel;

[0034] (4.3) The output features processed by the explicit clothing feature transformation branch and the implicit clothing feature transformation branch are input together into the adaptive spatial gating fusion module for feature fusion, and the fusion result is passed to the next network layer as the final output of the current layer.

[0035] Furthermore, at each decoder feature level The corresponding optical flow field predicted using step (3.3) Regarding clothing characteristics Grid-based spatial resampling is performed, followed by warping transformation to obtain explicit clothing transformation features at each level. Specifically, the aforementioned The function is implemented using the standard PyTorch function `torch.nn.functional.grid_sample`, with the first parameter being the input features and the second parameter being the optical flow field grid. This function is based on the optical flow field grid. Each provided normalized sampling coordinate (within the range [-1, 1]) is derived from the input features. The corresponding pixel position is used to extract the neighborhood value, and the output value is calculated through bilinear interpolation. The region outside the boundary is filled with zeros. That is, the output is calculated based on the input value and the pixel position provided by the grid, completing the spatial transformation from source features to target features.

[0036] Furthermore, at each decoder feature level clothing features With the characteristics of the body when trying on clothes The features are concatenated along the spatial dimension to construct a joint feature sequence. This sequence is then input into the self-attention module of the virtual fitting network. Through the attention mechanism, a cross-regional semantic association between clothing features and human body features is implicitly established, achieving feature alignment. Let the virtual fitting human body features be... The corresponding query matrix is ​​Q, representing the human body features for trying on clothes. and clothing characteristics The corresponding key matrices are K and Fitting human body characteristics and clothing characteristics The corresponding Value matrices are V and V, respectively. The formula for calculating the self-attention layer is as follows:

[0037]

[0038]

[0039] Where concat represents the concatenation of spatial dimensions. Indicates the obtained first Hierarchical implicit clothing transformation features For feature dimensions.

[0040] Furthermore, the explicit clothing transformation features at each level are further... and implicit clothing transformation features Channel-dimensional concatenation is performed, and the data is input into a Fusion Generation Module (FGM) to predict a spatially gated graph with values ​​ranging from [0,1]. , in, , Corresponding to the current decoder's number The height and width pixel values ​​of features in the spatial dimension at the layer network level. The FGM is composed of multiple neural network layers and can be simply implemented as a multilayer perceptron or a multilayer convolutional neural network.

[0041]

[0042] in Refers to the corresponding encoder network number Hierarchical fusion generates network modules.

[0043] The features of the two branches are weighted and fused using a gating graph, and the calculation formula is as follows:

[0044]

[0045] in This indicates element-wise multiplication.

[0046] Furthermore, step (5) includes the following steps:

[0047] (5.1) In the noise addition process of the conditional diffusion model, the latent representation of the clothing fitting image obtained by the VAE encoder is... Add noise gradually After time step The latent variables after adding noise are During the denoising process, a parameter is trained as follows: Denoising network Predict exist Step-by-step noise The entire prediction process is optimized using mean squared error loss, as shown below:

[0048]

[0049] in, This represents the latent representation of the masked fitting image. This represents the mask to be generated after downsampling. This indicates the potential representation of the posture. This represents random standard Gaussian noise;

[0050] (5.2) For multi-scale appearance optical flow with N levels, to improve the accuracy of optical flow estimation, a multi-scale optical flow estimation loss is introduced. The clothing image is downsampled to the resolution of the corresponding level of appearance optical flow, and then the appearance optical flow of that level is used to perform a warping transformation on the downsampled clothing image. The warping transformation result should be consistent with the clothing part in the fitting image. Generally, L1 or L2 loss is used for calculation. The multi-scale optical flow estimation loss is obtained by weighting the losses of N levels. Specifically, as follows:

[0051]

[0052]

[0053] in, and These represent images of clothing and actual images of people trying on clothes, respectively. and These represent downsampling to Total optical flow loss between high-resolution clothing images and real-life try-on images It combines final pixel-level loss with feature-level multi-scale optical flow loss.

[0054] (5.3) Total training loss of the model Noise prediction loss from diffusion model And optical flow estimation loss Weighted composition

[0055]

[0056] in This represents the weight of the optical flow estimation loss.

[0057] (5.4) Use the Adam optimizer to optimize and adjust hyperparameters such as learning rate and batch size. After setting the parameters, start training and finally obtain the trained virtual try-on generation model M.

[0058] Furthermore, step (6) includes the following steps:

[0059] (6.1) Input the human body image and target clothing image specified by the user, and perform the same preprocessing as in step (1);

[0060] (6.2) Input the processed conditions into the network, from standard Gaussian noise Initially, noise reduction is achieved through a T-step reverse diffusion process.

[0061] (6.3) In each denoising step, the embedded multi-scale optical flow estimation module performs optical flow prediction, and the dual-flow feature transformation module performs dual clothing feature transformation and adaptive spatial gating fusion operations to finally generate the denoised latent representation. ;

[0062] (6.4) Using VAE decoder D to represent the latent representation Decoding back to pixel space yields the final high-resolution virtual fitting image. .

[0063] Secondly, the present invention also provides an image virtual fitting generation device based on explicit and implicit dual-stream clothing feature transformation, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the image virtual fitting generation method based on explicit and implicit dual-stream clothing feature transformation.

[0064] Thirdly, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the image virtual fitting generation method and apparatus based on explicit and implicit dual-stream clothing feature transformation.

[0065] Fourthly, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the image virtual fitting generation method and apparatus based on explicit and implicit dual-stream clothing feature transformation.

[0066] The beneficial effects of this invention are:

[0067] First, this invention innovatively proposes an embedded multi-scale optical flow prediction module in the virtual fitting network framework based on the diffusion model. This module utilizes the hierarchical multi-scale features of the diffusion model to predict the multi-scale appearance optical flow field at the feature level, providing more accurate deformation guidance for clothing feature transformation. This effectively improves the stretching distortion and artifact problems caused by clothing deformation and enhances the structural rationality of the generated image.

[0068] Secondly, this invention innovatively proposes a dual-stream feature transformation module and embeds an adaptive spatial gating module. By setting two clothing feature transformation branches in parallel, explicit and implicit, it effectively improves the problem that a single transformation mechanism cannot take into account both structure and texture. The explicit feature transformation branch uses optical flow field to perform warping transformation to better preserve the texture details of clothing, while the implicit feature transformation branch uses a self-attention mechanism to model cross-regional semantic associations to improve the naturalness of clothing and human body integration. The embedded adaptive spatial gating module can dynamically adjust the fusion weight of the two features pixel by pixel according to the image content, thereby generating a fitting effect with clear texture and natural fit.

[0069] In summary, this method effectively solves the problems of clothing texture distortion and misalignment in existing virtual try-on technologies, and has high application value. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 The present invention provides a structural diagram of a virtual fitting image intelligent generation method based on a diffusion model.

[0072] Figure 2 The result of the intelligent virtual try-on image generation method based on the diffusion model provided by this invention is shown in the figure.

[0073] Figure 3 This is a structural diagram of a virtual model clothing display image intelligent generation device based on a diffusion model provided by the present invention. Detailed Implementation

[0074] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described below with reference to the accompanying drawings.

[0075] This invention proposes a virtual virtual try-on image generation method and apparatus based on explicit and implicit two-stream garment feature transformation for the task of generating virtual try-on images of clothing. Given a target garment image and a user image, this method can generate high-quality virtual try-on images that preserve the garment's pattern and texture details. Traditional try-on methods based on diffusion models typically use only a single attention mechanism for garment feature transformation, often failing to achieve accurate alignment of garment geometry and high-fidelity preservation of texture details. To improve upon this, this invention adds an embedded multi-scale optical flow estimation module to the diffusion model. Utilizing the powerful semantic priors captured by the pre-trained text-based image diffusion model in generative modeling, it cascades predictions of multi-scale appearance optical flow from garment to human body, achieving relatively accurate geometric guidance at the feature level. This invention also designs a two-stream feature transformation module and an adaptive spatial gating fusion strategy. By executing the explicit feature transformation branch based on optical flow and the implicit feature transformation branch based on the attention mechanism in parallel, and adaptively fusing the features from both branches according to the feature transformation results, it generates high-fidelity virtual try-on images that better preserve garment texture details.

[0076] like Figure 1 As shown in (a) of the present invention, an image-based virtual try-on method based on explicit and implicit dual-stream clothing feature transformation is provided. The specific steps are as follows:

[0077] (1) Preprocess the paired datasets Dresscode and VITON-HD containing (clothing images, clothing display images) data pairs.

[0078] (1.1) Scale all images in the dataset to a uniform resolution (1024, 768) and normalize the pixel values ​​to the range [-1, 1].

[0079] (1.2) Use pre-trained human parsing models (Graphonomy and DensePose) to obtain human semantic segmentation map and human pose map respectively. The human semantic segmentation map is used to construct clothing mask in the absence of clothing mask in the dataset.

[0080] (1.3) Based on the clothing mask provided in the dataset The clothing area and related areas such as arms in the human body image are masked to obtain the masked fitting image. And the corresponding image of the actual clothing deformation in the masked portion. ;

[0081] (1.4) The final constructed clothing image is obtained. pose image Clothing mask The masked fitting image Realistic images of clothing deformation >Data pair D1.

[0082] (2) Use the data constructed in step (1) to train the virtual fitting model D1. The virtual fitting model mainly consists of three parts: variational autoencoder (VAE), clothing network and fitting network.

[0083] (2.1) Using a pre-trained VAE encoder The pixel-space image is compressed into a latent space. The target ground-down image of the clothing, the clothing image, the pose image, and the masked image of the clothing are each encoded as a latent representation. , , and .

[0084] (2.2) Both the clothing network and the fitting network are based on the denoised U-Net network in Stable Diffusion (SD) and the parameters are initialized based on the parameters of the Stable Diffusion pre-trained model.

[0085] (2.3) The clothing network is a U-Net network obtained by SD initialization, with the cross attention layer removed, specifically designed for extracting clothing images. The multi-scale features of the encoder, denoted as the clothing features, are denoted as... The clothing features of its decoder are recorded as ,feature superscript It increases with increasing feature resolution.

[0086] (2.4) The fitting network is a U-Net network initialized by SD, with the cross-attention layer removed. Its input is... through Latent variables after adding noise Masked fitting image encoding Human body mask after downsampling Attitude coding and time steps ,in , , as well as The network is spliced ​​along the channel dimension, and the time step t is injected into the network in the original SD method. The human body features of the encoder are denoted as follows. The decoder records the human body characteristics of the person trying on clothes as follows: Superscript of features It increases with increasing feature resolution.

[0087] (3) In the encoder part of the fitting network and the clothing network, an embedded multi-scale optical flow prediction module is constructed to predict the appearance optical flow of clothing feature deformation. The embedded multi-scale optical flow estimation module consists of a multi-level optical flow estimation block (FEB).

[0088] (3.1) Obtain the deepest level of clothing features from the clothing network and fitting network encoders. and fitting body characteristics After concatenating the two along the channel dimension, the result is input into the optical flow estimation module (composed of a three-layer convolutional neural network), as follows: Figure 1 As shown in (b) above, the initial appearance feature optical flow is obtained. The optical flow field represents the displacement of the clothing feature pixels relative to the human body feature in the horizontal and vertical directions, as shown below:

[0089]

[0090] in Represents the corresponding number The optical flow estimation module for hierarchical characteristic optical flow. This represents the deepest level of clothing features in the clothing network encoder. This represents the deepest level of human body features in the virtual try-on network encoder.

[0091] (3.2) For each subsequent level (From 2 to N), first predict the optical flow of the previous level. Perform upsampling to obtain the initial optical flow of the current layer. Then, the residual optical flow is predicted using the features of the current level. ,Will and The optical flow at the current level is obtained by adding elements one by one. The process is defined as follows:

[0092]

[0093]

[0094]

[0095] in, These are the clothing features of the clothing network encoder at level l and the human body features of the virtual fitting network encoder, respectively. This process continues until the multi-scale feature appearance optical flow of N levels is obtained. .

[0096] (3.3) Final appearance optical flow Upsampled to the resolution of the clothing fitting image This is used for subsequent pixel-level optical flow prediction loss calculation. To improve estimation accuracy, a multi-scale optical flow estimation loss is introduced. :

[0097]

[0098] in, and These represent images of clothing and actual images of people trying on clothes, respectively. and These represent scaling to... High-resolution images of clothing and real-life images of clothing in try-on, grid_sample( ) indicates with right Functions for performing warp transformations. It combines the final pixel-level optical flow loss with the feature-level multi-scale optical flow estimation loss.

[0099] (4) Construct a two-stream feature transformation module on the decoders of the clothing network and the fitting network, such as Figure 1 As shown in (c) of the diagram. This module contains parallel explicit clothing feature transformation branches and implicit clothing feature transformation branches. For the decoder's... The hierarchy defines the output features of the clothing network decoder as follows: The output characteristics of the virtual fitting network decoder are .

[0100] (4.1) Explicit clothing feature transformation branch: using the appearance optical flow estimated in step (3) at the same resolution Regarding clothing characteristics An explicit warping transformation is performed to obtain explicit clothing transformation features. To align the feature distribution, learnable linear layers similar to Attention layers are applied before and after the warping transformation. Network layer and Network layer):

[0101]

[0102] Specifically, the The function is implemented using the standard PyTorch function `torch.nn.functional.grid_sample`, with the first parameter being the input features and the second parameter being the optical flow field grid. This function is based on the optical flow field grid. Each provided normalized sampling coordinate (within the range [-1, 1]) is derived from the input features. The corresponding pixel position is used to extract the neighborhood value, and the output value is calculated through bilinear interpolation. The region outside the boundary is filled with zeros. That is, the output is calculated based on the input value and the pixel position provided by the grid, completing the spatial transformation from source features to target features.

[0103] Finally, the warped clothing features and the current-level human body features of the virtual try-on network decoder are combined. Add:

[0104]

[0105] in This is the output of the explicit feature transformation branch. This branch's garment-specific transformation is directly guided by the predicted optical flow, accurately mapping garment features to the area to be fitted.

[0106] (4.2) Implicit Clothing Feature Transformation Branch: This branch transforms the current clothing features... Layered fitting room body characteristics With clothing characteristics The data is spliced ​​in the spatial dimension and input into the self-attention layer of the virtual fitting U-Net network. Let the human body features be considered when trying on the clothing. The corresponding query matrix is ​​Q, representing the human body features for trying on clothes. and clothing characteristics The corresponding key matrices are K and Fitting human body characteristics and clothing characteristics The corresponding Value matrices are V and V, respectively. The formula for calculating the self-attention layer is as follows:

[0107]

[0108]

[0109] in Indicates the obtained first Hierarchical implicit clothing transformation features For feature dimensions.

[0110] After calculation, the implicit clothing transformation feature output is obtained through a feedforward linear layer. This flow utilizes attention-modeled semantic correspondences to drive implicit clothing feature transformations.

[0111] (5) In order to better integrate the output features of the explicit feature transformation branch and the implicit feature transformation branch, an adaptive spatial gating fusion module was designed.

[0112] (5.1) Output the explicit feature transformation branch and implicit feature transformation branch output The input is concatenated along the channel dimension and fed into a Fusion Generation Module (FGM, a neural network module consisting of three convolutional neural network layers, with the output layer being either a sigmoid activation function or an identity activation function) to generate spatial gating parameters. ,in, , Corresponding to the current decoder's number The height and width pixel values ​​of features in the network layer in terms of spatial dimension:

[0113]

[0114] in Indicates the corresponding number The network module is generated by fusing hierarchical features.

[0115] (5.2) The two features are weighted and fused according to the gating parameters to obtain the final output. And pass it to the next layer.

[0116]

[0117] (6) The model is trained using the data D1 from step (1.4) as training data.

[0118] (6.1) In the noise addition process of the diffusion model, the latent representation of the clothing fitting image obtained by the VAE encoder is... Add noise gradually Time step The latent variables are typically positive integers from 0 to T, where T is usually 1000. The latent variables after t steps of noise addition are... During the denoising process, a denoising network is trained. Predict exist step noise The training loss is optimized using mean squared error loss, as shown below:

[0119]

[0120] in, This represents the latent representation of the masked fitting image. Indicates attitude encoding. This represents the mask to be generated after downsampling. Indicates the number of noise reduction time steps. This represents random standard Gaussian noise;

[0121] (6.2) Total training loss of the model Noise prediction loss from diffusion model And optical flow estimation loss Weighted composition

[0122]

[0123] in This represents the weight of the optical flow estimation loss.

[0124] (6.3) For the dataset used, it is divided into training data and test data in a 9:1 ratio and the data is scaled to a size of (1024, 768). The training is optimized using the AdamW optimizer, with the initial learning rate adjusted to 1e-5 and the batch size set to 56.

[0125] (6.4) In the inference stage, starting with random Gaussian noise, iterative denoising is performed using the trained model. In each denoising step, the embedded multi-scale optical flow prediction module predicts the optical flow, the dual-flow feature transformation module performs explicit and implicit clothing feature transformations in parallel, and the features are fused using an adaptive spatial gating fusion module. The final latent representation is decoded by the VAE decoder to obtain the synthesized virtual fitting image, as shown in the figure. Figure 2 As shown.

[0126] Corresponding to the aforementioned embodiment of an image virtual fitting method based on explicit and implicit dual-stream clothing feature transformation, the present invention also provides an embodiment of an image virtual fitting device based on explicit and implicit dual-stream clothing feature transformation.

[0127] See Figure 3 The present invention provides an image virtual fitting device based on explicit and implicit dual-stream clothing feature transformation, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement an image virtual fitting method based on explicit and implicit dual-stream clothing feature transformation in the above embodiment.

[0128] The embodiment of the image virtual fitting device based on explicit and implicit dual-stream clothing feature transformation provided by this invention can be applied to any device with data processing capabilities, such as a personal computer, server, workstation, etc. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 3 The diagram shown is a hardware structure diagram of any data processing device, including the virtual fitting device based on a dual-flow diffusion model and multi-scale optical flow guidance provided by this invention. (Except for...) Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware (such as a high-performance graphics processing unit, GPU) depending on the actual function of the data processing device, which will not be described in detail here.

[0129] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0130] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0131] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements a virtual fitting method based on a dual-flow diffusion model and multi-scale optical flow guidance as described in the above embodiments.

[0132] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0133] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the virtual fitting method based on a dual-flow diffusion model and multi-scale optical flow guidance.

[0134] In addition to the embodiments described above, the present invention may have other implementations. All technical solutions formed by equivalent substitution or equivalent transformation fall within the protection scope claimed by the present invention.

Claims

1. A method for generating virtual clothing try-on images based on explicit and implicit dual-stream clothing feature transformation, characterized in that: include: (1) Obtain a pairwise dataset containing {clothing image, clothing fitting image} and preprocess it to obtain training data D1 corresponding to {clothing image, pose image, clothing mask, masked fitting image, clothing fitting image}; (2) Construct a two-branch diffusion model network model, which includes a fitting network and a clothing network; the clothing network extracts clothing features and injects them as generation conditions into the fitting network, and the fitting network extracts human body features for fitting and obtains the generated fitting image by continuously predicting the noise at each step. (3) Construct an embedded multi-scale optical flow estimation module in the encoder of the network model, and use the multi-scale features of the virtual fitting network and the clothing network to generate a two-dimensional deformation field from clothing features to the human body to predict the feature optical flow. (4) Construct a dual-stream feature transformation module in the decoder of the network model, including an explicit feature transformation branch and an implicit feature transformation branch, which are used to obtain explicit clothing transformation features based on feature optical flow and implicit clothing transformation features based on human body features and clothing features, respectively. The enhanced fusion features are obtained through fusion and then passed to the next layer of the network; (5) Use training data D1 to train the constructed model, optimize the model parameters by calculating noise prediction loss and multi-scale optical flow prediction loss, and obtain the trained virtual try-on generation model M. (6) For the user image to be tried on and the target clothing image, after preprocessing, input them into model M and output the virtual try-on display image.

2. The image virtual fitting generation method based on explicit and implicit dual-stream clothing feature transformation according to claim 1, characterized in that: Step (1) includes the following steps: (1.1) Preprocess the dataset, unify the size of all images, perform format conversion and data normalization to ensure the consistency of image input; (1.2) Given a preprocessed human body image and clothing image pair, use a pre-trained clothing segmentation model to segment the clothing region in the human body image to obtain a clothing mask and a masked fitting image after applying the mask to the human body image. At the same time, use a pre-trained human pose estimation network to estimate the pose of the human body image to obtain a pose image. (1.3) Construct data pairs of {human body image, clothing image, pose image, clothing mask, and masked fitting image} to form training data D1.

3. The image virtual fitting generation method based on explicit and implicit dual-stream clothing feature transformation according to claim 1, characterized in that: Step (2) includes the following steps: (2.1) The dual-branch diffusion model network model includes a VAE reconstruction autoencoder, a fitting network and a clothing network. Both the fitting network and the clothing network are built based on the U-Net network and share the same training objective. The fitting network takes the masked fitting image, the clothing mask and the pose image as input, and the clothing network takes the clothing image as input. (2.2) The U-Net network adopts an encoder-decoder structure with skip connections; the main body of the network consists of stacked residual modules and spatial Transformer modules, in which the spatial Transformer modules embed self-attention layers.

4. The image virtual fitting method based on explicit and implicit dual-stream clothing feature transformation according to claim 1, characterized in that: Step (3) involves an embedded multi-scale optical flow estimation module consisting of a multi-level optical flow estimation module (FEB) and includes the following steps: (3.1) Obtain the clothing features and fitting human body features at the deepest level of the clothing network and fitting network encoders; input both into the optical flow estimation module FEB corresponding to the feature layer to obtain the initial appearance feature optical flow, which represents the displacement of the clothing feature pixel relative to the fitting human body feature in the horizontal and vertical directions. (3.2) For each subsequent level , will the Hierarchical optical flow is upsampled to obtain the first Hierarchical initial feature prediction optical flow, The hierarchical clothing features and the human body features during fitting are input into the embedded multi-scale optical flow estimation module to obtain the current number of... Hierarchical residual features predict optical flow, and the first The initial feature prediction optical flow and the residual feature prediction optical flow are added element-wise to obtain the current optical flow. The hierarchical feature prediction optical flow is continued until N levels of multi-scale feature appearance optical flow are obtained. (3.3) Upsample the feature appearance optical flow at the maximum resolution to the resolution of the clothing fitting image to obtain the appearance optical flow at the image resolution size, which is used for subsequent pixel-level optical flow prediction loss calculation.

5. The image virtual fitting method for generating clothing based on explicit and implicit dual-stream clothing feature transformation according to claim 1, characterized in that: Step (4) includes the following steps: (4.1) Construct a dual-stream feature transformation module at each level of the virtual fitting network decoder. This module includes an explicit clothing feature transformation branch and an implicit clothing feature transformation branch set in parallel, as well as an adaptive spatial gating fusion module. (4.2) Input the features obtained from the corresponding layers of the decoder of the fitting network and the clothing network into the explicit clothing feature transformation branch and the implicit clothing feature transformation branch respectively, and perform the feature transformation operation in parallel; (4.3) The output features processed by the explicit clothing feature transformation branch and the implicit clothing feature transformation branch are input together into the adaptive spatial gating fusion module for feature fusion, and the fusion result is passed to the next network layer of the virtual fitting network decoder as the final output of the corresponding layer of the virtual fitting network decoder.

6. The image virtual fitting method for generating clothing based on explicit and implicit dual-stream clothing feature transformation according to claim 1, characterized in that: At each decoder feature level Using the predicted appearance feature optical flow at the corresponding resolution, a warping transformation is performed on the clothing features to obtain explicit clothing transformation features at each level; at each decoder feature level Clothing features and human body features for trying on clothes are concatenated along the spatial dimension to construct a joint feature sequence. This sequence is then input into the self-attention module, and by calculating the attention map matrix, the human body features for trying on clothes can focus on the relevant parts of the clothing features. Through the attention mechanism, a cross-regional semantic association between clothing features and human body features is implicitly established, achieving feature alignment and obtaining implicit clothing transformation features at each level.

7. The image virtual fitting method based on explicit and implicit dual-stream clothing feature transformation according to claim 6, characterized in that: Explicit and implicit clothing transformation features are input into a fusion generative network module (FGM) to predict two fusion parameters. Then, the explicit and implicit clothing transformation features are weighted and fused using a gating graph.

8. The image virtual fitting method for generating clothing based on explicit and implicit dual-stream clothing feature transformation according to claim 1, characterized in that: Step (5) includes the following steps: (5.1) In the process of adding noise to the conditional diffusion model, noise is gradually added to the latent representation of the clothing fitting image obtained by the VAE encoder. In the process of denoising, a denoising network is trained to predict the noise of the latent variables. The entire prediction process is optimized using mean squared error loss. (5.2) For the multi-scale feature appearance optical flow of N levels, apply the multi-scale optical flow estimation loss, downsample the clothing image to the resolution of the corresponding level appearance optical flow, and then use the level appearance optical flow to perform warping transformation on the downsampled clothing image. The warping transformation result should be consistent with the clothing part in the fitting image. Generally, L1 or L2 loss is used for calculation. The multi-scale optical flow estimation loss is obtained by weighting the loss of N levels. (5.3) The total training loss of the model is composed of a weighted average of the noise prediction loss of the diffusion model and the multi-scale optical flow estimation loss; (5.4) Use the AdamW optimizer to optimize the model parameters and adjust the learning rate and batch size hyperparameters. After the parameters are set, start training and finally obtain the trained virtual try-on generation model M.

9. The image virtual fitting method based on explicit and implicit dual-stream clothing feature transformation according to claim 1, characterized in that: Step (6) includes the following steps: (6.1) Input the human body image and target clothing image specified by the user, and perform the same preprocessing as in step (1); (6.2) Input the processed conditions into the network and gradually denoise it starting from standard Gaussian noise through a T-step back diffusion process; (6.3) In each step of the denoising process, the embedded multi-scale optical flow estimation module performs optical flow prediction, and the dual-flow feature transformation module performs dual clothing feature transformation and adaptive spatial gating fusion operation to finally generate the denoised latent representation; (6.4) Use VAE decoder D to decode the latent representation back to pixel space to obtain the final high-resolution virtual fitting image.

10. An image-based virtual fitting device for generating clothing based on explicit and implicit dual-stream clothing feature transformation, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that... When the processor executes the executable code, it implements the image virtual try-on generation method based on explicit and implicit dual-stream clothing feature transformation as described in any one of claims 1-9.

Images