Video virtual fitting method based on time sequence enhanced diffusion model

Abstract

The invention discloses a video virtual fitting method based on a time sequence enhanced diffusion model, and the method comprises the steps: firstly constructing a fashion video sequence, a posture sequence, a semantic sequence and a depth sequence data set based on a fashion data set; secondly, mapping the clothes to a corresponding deformation space frame by frame according to a posture sequence and a depth sequence by using a deformation network to obtain a deformation clothes sequence as a guide condition of a diffusion network; then, by taking generation of a fashion try-on video as a target, constructing a time sequence enhancement network and a human body feature injection module on the basis of a standard stable diffusion model, and synthesizing a deformed garment sequence and a fashion video sequence to obtain a virtual try-on video sequence; and finally, respectively using diffusion loss and perception loss, and training model parameters in the time sequence enhancement network and the character feature injection module through a back propagation algorithm. According to the invention, the user experience is improved, and the application and development of the virtual fitting technology in e-commerce and fashion industries are promoted.

Description

Technical Field

[0001] The invention belongs to the technical field of virtual fitting, and in particular relates to a video virtual fitting method based on a temporal enhanced diffusion model. Background Art

[0002] With the continuous development in the field of virtual fitting, deep learning-driven image synthesis technology has become the key to achieving a highly realistic virtual clothing fitting experience. In particular, generative diffusion models have achieved remarkable results in image synthesis tasks in recent years.

[0003] The image-based virtual fitting algorithm divides the model into two modules, the clothing deformation module and the synthesis module. The clothing deformation module gradually shifts from the thin plate spline deformation-based method to the flow-based algorithm, realizing the mapping of the target clothing to the human body posture. The purpose of the synthesis module is to naturally transfer the deformable clothing to the human body image, which needs to overcome many challenges such as occlusion relationship and boundary transition. In recent years, the diffusion model has received widespread attention due to its significant advantages over the generative adversarial network in terms of image generation quality and training process stability. The core algorithm of the synthesis module has also changed from the generative adversarial network to the diffusion model.

[0004] The video virtual fitting task involves inputting a series of video frames and images of trying on clothes, and finally outputting a new video. Due to the continuous changes in the posture, position and size of the characters in the video, video virtual fitting not only requires processing complex dynamic scenes, but also needs to accurately capture the coordinated changes of clothing and human movements, and ensure smooth and continuous visual effects between the generated video frames. If we only use the standard diffusion model to try to solve this problem, the experimental results show that the model cannot directly understand the semantic information of several input images and the relationship between them, which obviously makes it difficult to achieve the many challenges brought by the task.

[0005] (1) Fidelity of clothing transfer: Clothing transfer needs to consider the texture information of different regions and the human body posture information. It must not only ensure the accurate mapping of clothing on the human body, but also correctly handle the occlusion relationship between objects.

[0006] (2) Consistency of character details: The stable diffusion model compresses the image into a latent space representation, resulting in a loss of image semantic information. Therefore, preserving the details of the character’s visual information has become one of the challenges of video virtual fitting.

[0007] (3) Continuity of inter-frame transitions: The current mainstream diffusion model cannot process and preserve the structural information between frames. If the video sequence is simply processed frame by frame, inter-frame flickering and inconsistent context semantics will occur. Summary of the invention

[0008] The present invention provides a video virtual fitting method based on a time-series enhanced diffusion model. The method uses a two-stage network, and uses a time-series enhanced diffusion network and a character feature injection module to achieve the generation of fitting videos with natural changes and realistic effects. We conducted experiments on the VVT ​​dataset and achieved good results both quantitatively and qualitatively. The present invention mainly relates to a time-series enhancement technology applied to a diffusion model, which models time series to achieve consistency and continuity in the time dimension; and uses a cross-attention mechanism to inject semantic features to generate real and effective fashion fitting videos.

[0009] A video virtual fitting method based on a temporal enhanced diffusion model, the steps of which are as follows:

[0010] Step (1): Based on the existing fashion dataset, collect task data, preprocess the data, and construct fashion video sequence, posture sequence, semantic sequence and depth sequence data sets.

[0011] Step (2) uses an advanced deformation network to map the clothing to the corresponding deformation space frame by frame based on the posture sequence and depth sequence, and obtains the deformable clothing sequence as the guiding condition of the diffusion network. The semantic sequence is used to obtain the binary mask of the human head and lower body to prevent occlusion between the relevant area and the deformable clothing.

[0012] Step (3) aims to generate natural and accurate fashion fitting videos. Based on the standard stable diffusion model, a temporal enhancement network and a human feature injection module are constructed to synthesize the deformable clothing sequence and the fashion video sequence to obtain a virtual fitting video sequence, thereby achieving the unity of inter-frame continuity and target texture consistency.

[0013] Step (4) uses diffusion loss and perceptual loss respectively to train the model parameters in the temporal enhancement network and the character feature injection module through the back propagation algorithm until the entire model converges.

[0014] The fashion video sequence, pose sequence, semantic sequence, and depth sequence data set described in step (1) refers to the use of the most advanced pose evaluator to estimate the pose sequence of a person from a fashion image for the corresponding fashion video sequence, and the calculated pose information of the person includes 18 joint coordinate points. In addition, an advanced human body parser is used to calculate a human body semantic sequence containing 20 labels, each label representing a specific part of the human body, such as the face, hair, arms, legs, and clothing area. And, an advanced human body deep pose estimator is used to calculate a dense depth sequence of the human body. In addition, following the consistent clothing masking rules, a mask is applied to cover the original clothing and its surrounding areas, thereby obtaining a series of fashion video sequences that are unrelated to clothing.

[0015] The deformation network described in step (2) is the virtual fitting deformation network. The present invention adopts a cyclic appearance flow deformation model, which consists of three parts: a feature extraction module, a multi-scale global correlation calculation module and a cyclic appearance flow update module. The deformation effect is achieved by estimating dense flow. In short, the deformation network uses the posture sequence V in step (1) k And the depth sequence V d , the target clothing is mapped to obtain a deformable clothing sequence related to the time dimension Among them, the binary mask V obtained from the semantic sequence s It is used to constrain the occlusion relationship between regions during the deformation process.

[0016] The temporal enhancement network and human feature injection module described in step (3) realize the unification of inter-frame continuity and target texture consistency, as follows:

[0017] Part I: Timing Enhanced Networks

[0018] The temporal enhancement network proposed in the present invention is an improved model architecture based on the stable diffusion model, which ensures accurate modeling of video frame content and in-depth consideration of the relationship between frames. The stable diffusion model is essentially composed of a pair of variational autoencoders and a U-Net network. The image data is input by the encoder of the variational autoencoder, mapped into a potential representation, calculated by the U-Net network, and then handed over to the decoder of the variational autoencoder for image restoration. The U-Net network is composed of a number of downsampling blocks and a number of upsampling blocks. The upsampling and downsampling blocks are function groups constructed in series by residual blocks and attention blocks. The specific structure will be described in detail in the "Optimized Network Structure" below. The temporal enhancement network aims to improve the generalization ability of the model while leveraging the training results of large-scale image datasets, especially the generation ability in the fashion field. And by processing the time dimension information, the performance of video generation is improved. Improvements include: 1) Adjusting the processing method of the stable diffusion model input: allowing it to accept the form of A five-dimensional video tensor is prepared by combining the following steps: a five-dimensional video tensor, wherein b and f represent the batch axis and the frame time axis respectively, and c, h, and w represent the number of channels, height, and width of the video frame respectively; 2) optimizing the network structure to better capture the continuity of the time dimension: the up- and down-sampling blocks include two sets of residual blocks and attention blocks connected in series, and the residual blocks are connected in series with several nonlinear convolutional layers and residual connections of the input and output within the blocks. The present invention adds a one-dimensional convolution with a residual connection (hereinafter referred to as one-dimensional time convolution) after each nonlinear convolutional layer to enable the spatial information on different time frames to interact; the attention block is connected in series with self-attention and cross-attention, and the present invention adds one-dimensional self-attention (hereinafter referred to as one-dimensional time self-attention) after the cross-attention to realize that different time frames are taken into consideration when calculating the attention weight.

[0019] Part 2: Character Feature Injection Module

[0020] The present invention proposes a character feature injection module. The variational autoencoder is composed of a pair of encoders and decoders connected in series. The encoder maps the visual image to the latent space layer by layer, and the decoder parses the image representation in the latent space into the corresponding image layer by layer. In order to fully extract the information related to the human body, the character feature injection module transfers the features of each layer in the encoder of the variational autoencoder to the corresponding layer of the decoder, and enhances the retention of human body information during the reconstruction process through the attention mechanism.

[0021] In step (4), diffusion loss and perceptual loss are used respectively to train the model parameters in the temporal enhancement network and the character feature injection module through the back propagation algorithm until the entire model converges. The details are as follows:

[0022] Regarding the time series enhancement diffusion model, a large-scale pre-trained stable diffusion model is used as the basic model. In order to minimize the loss of prior knowledge, the weights are trained to the input layer of the open model entrance and the time series enhancement module proposed in this invention. The basic diffusion loss is used as the loss function for training this module, which is specifically defined as follows:

[0023]

[0024] Among them, z t 、z a and z w They represent the latent space representation of noise, clothing-independent image, and deformed clothing image at time t, d represents the original character depth posture information downsampled to 1 / 8 by the nearest neighbor, and I g represents the features of clothing images obtained by the image feature extractor DINO-V2, ε θ represents the noise estimation network, i.e., the time series enhanced diffusion model, and ε represents the target noise at time t.

[0025] Regarding the character feature injection module, in order to reduce the burden of training calculations, this module is designed as a decoupled pluggable module that can be trained independently from the denoising network. The module is trained through the reconstruction task of the autoencoder. The specific perceptual loss is defined as follows:

[0026]

[0027] in, is the mth feature of the VGG-19 network pre-trained on Image-Net, E is the encoding process of the variational autoencoder, and D′ is the decoding process after character feature injection.

[0028] Beneficial effects of the present invention:

[0029] The present invention proposes to introduce a temporal consistency enhancement module and a human feature injection module, while maintaining high-quality image generation, more character details can be retained in the video sequence, thereby significantly improving the quality and continuity of video generation, solving the problems of inter-frame jitter, character detail loss, poor clothing fidelity, etc. in existing methods, making the video virtual fitting task possible for practical implementation. In addition, the task of combining artificial intelligence with fashion is a current research hotspot, and the multimodal generation of information such as integrated posture, dense posture, and deformed clothing images also makes the present invention more cutting-edge and innovative in scientific research. By enhancing the continuity and detail retention of virtual fitting videos, the method helps to improve the user experience, thereby promoting the application and development of virtual fitting technology in the e-commerce and fashion industries. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a specific schematic diagram of the process of the present invention;

[0031] Figure 2 is a schematic diagram of a timing enhancement module in the method of the present invention;

[0032] Figure 3 It is a schematic diagram of a character feature injection module in the method of the present invention;

[0033] Figure 4 It is a schematic diagram of the overall framework of the present invention;

[0034] Figure 5 It is a schematic diagram of the data set of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0036] The present invention provides a video virtual fitting method based on a temporal enhanced diffusion model. Figure 1 As shown in the figure, this method uses a two-stage network, which uses a temporal enhanced diffusion network and a character feature injection module to achieve natural and realistic fitting video generation. We conducted experiments on the VVT ​​dataset and achieved good results both quantitatively and qualitatively.

[0037] A video virtual fitting method based on temporal enhancement diffusion model Figure 1 The steps are as follows:

[0038] Step (1): Based on the existing fashion dataset, collect task data, preprocess the data, and construct fashion video sequence, posture sequence, semantic sequence, and depth sequence data sets.

[0039] Step (2) uses an advanced deformation network to map the clothing to the corresponding deformation space frame by frame based on the posture sequence and depth sequence, and obtains the deformable clothing sequence as the guiding condition of the diffusion network. The semantic sequence is used to obtain the binary mask of the human head and lower body to prevent occlusion between the relevant area and the deformable clothing.

[0040] Step (3) aims to generate natural and accurate fashion fitting videos. Based on the standard stable diffusion model, a temporal enhancement network and a human feature injection module are constructed to synthesize the deformable clothing sequence and the fashion video sequence to obtain a virtual fitting video sequence, thereby achieving the unity of inter-frame continuity and target texture consistency.

[0041] Step (4) uses diffusion loss and perceptual loss respectively to train the model parameters in the temporal enhancement network and the character feature injection module through the back propagation algorithm until the entire model converges.

[0042] The step (1) of collecting task data based on an existing fashion dataset refers to evaluating the method of the present invention on the VVT ​​dataset. The VVT ​​dataset includes 791 fashion model catwalk videos, which are divided into 661 training videos and 130 test videos. Each video contains 250 to 300 frames. Data preprocessing removes frames without characters at the beginning and end of the video. In addition to the 159,170 frames of the training set and the 30,931 frames of the test set, the dataset also provides corresponding clothing images for each video. Fashion frame images and fashion clothing images are referenced Figure 5 , where the fashion frame is any unit of a fashion video sequence. The video sequence contains three parts: the model approaches, turns around, and walks out of the camera from outside the camera, and the camera is fixed.

[0043] The fashion video sequence, posture sequence, semantic sequence, and depth sequence data group described in step (1) refers to the use of the most advanced posture evaluator to estimate the posture sequence of a person from a fashion image for the corresponding fashion video sequence, and the calculated posture information of the person includes 18 joint coordinate points. In addition, an advanced human body parser is used to calculate a human body semantic sequence containing 20 labels, each label representing a specific part of the human body, such as the face, hair, arms, legs, and clothing area. And, an advanced human body deep posture estimator is used to calculate a dense depth sequence of the human body. In addition, following the consistent clothing masking rules, a mask is applied to cover the original clothing and its surrounding areas, thereby obtaining a series of fashion video sequences that are not related to clothing. Specific data group examples are as follows: Figure 5 As shown in the figure, different relevant information is obtained based on the fashion frame image, where the human body analysis image divides the parts of the human body representing different semantic information into 18 blocks of different colors, the key point image describes the 18 joint points that symbolize the human body posture information, and the clothing-independent image generates a mask based on the human body analysis image, referring to the human body arms, torso and other parts.

[0044] The deformation network described in step (2) is the virtual fitting deformation network. The present invention adopts a cyclic appearance flow deformation model, which consists of three parts: a feature extraction module, a multi-scale global correlation calculation module and a cyclic appearance flow update module. The deformation effect is achieved by estimating dense flow. In short, the deformation network uses the posture sequence V in step (1) k And the depth sequence V d , the target clothing is mapped to obtain a deformable clothing sequence related to the time dimension Among them, the binary mask V obtained from the semantic sequence s It is used to constrain the occlusion relationship between regions during the deformation process. The resulting deformable clothing frame can be referenced Figure 5 As shown in Figure 2, the deformable clothing not only retains the texture information of the clothing itself, but also maps it to a spatial position that matches the human body posture.

[0045] The temporal enhancement network and the human feature injection module described in step (3) optimize the stable diffusion model to achieve the unity of inter-frame continuity and target texture consistency. The overall framework diagram of the design of the present invention is as follows: Figure 4 The specific instructions are as follows:

[0046] Part I: Timing Enhanced Networks

[0047] In the process of video generation, it is necessary to ensure the high quality of the internal image of each frame, and also to ensure the temporal smoothness and continuity between consecutive frames. In the task of generating virtual fitting clothes in videos, the differences in various input conditions (such as human posture, human semantic analysis, deformable clothing, etc.) in each frame will cause discontinuous movement of the spatial position between consecutive frames, which is visually produced as the so-called "inter-frame jitter". In addition, due to the working principle of the probability diffusion model, random noise sequences need to be sampled and denoised, which further increases the challenge of maintaining video continuity.

[0048] like Figure 2As shown, the temporal enhancement network proposed in the present invention is an improved model architecture based on the stable diffusion model, which ensures accurate modeling of video frame anger and in-depth consideration of the relationship between frames. The stable diffusion model is essentially composed of a pair of variational autoencoders and a U-Net network. The image data is input by the encoder of the variational autoencoder, mapped into a potential representation, calculated by the U-Net network, and then handed over to the decoder of the variational autoencoder for image restoration. The U-Net network is composed of a number of downsampling blocks and a number of upsampling blocks. The upsampling and downsampling blocks are function groups constructed in series by residual blocks and attention blocks. The specific structure will be described in detail in the "Optimized Network Structure" below. This architecture aims to improve the generalization ability of the model while leveraging the training results of large-scale image datasets, especially the generation ability in the fashion field. And by processing the time dimension information, the performance of video generation is improved. Improvements include: 1) Adjusting the processing method of the stable diffusion model input to allow it to accept the form of A five-dimensional video tensor, where b and f represent the batch axis and the frame time axis respectively, and c, h, and w represent the number of channels, height, and width of the video frame respectively; 2) Optimize the network structure to better capture the continuity of the time dimension: the up and down sampling blocks contain two sets of residual blocks and attention blocks in series. The residual blocks are connected in series with several nonlinear convolutional layers and residual connections of the input and output within the blocks. The present invention adds a one-dimensional convolution with a residual connection after each nonlinear convolutional layer (hereinafter referred to as one-dimensional time convolution) to enable spatial information on different time frames to interact; the attention block is connected in series with self-attention and cross-attention. The present invention adds one-dimensional self-attention (hereinafter referred to as one-dimensional time self-attention) after the cross-attention to take into account different time frames when calculating the attention weight. The definition of one-dimensional time self-attention is as follows:

[0049]

[0050] Among them, Q t ,K t ,V t They represent the query features, key features, and value features after dimension reorganization. We jointly model the spatial dimension and the temporal dimension so that the spatial blocks can interact in the temporal dimension, thereby obtaining temporal information. k K t The dimension of the feature is normalized by scaling it, which can prevent gradient disappearance and gradient explosion and ensure the stability of feature distribution.

[0051] Specifically, when the U-Net network of the present invention processes the potential features obtained by the variational autoencoder based on the video sequence and the deformable clothing sequence, the time axis f is reshaped to the batch axis b and temporarily ignored, that is, the time relationship between the sequences is temporarily ignored. This allows the network to independently process the input video frame sequence through a standard two-dimensional convolution layer. After passing through the decoder, the feature map is reshaped into a five-dimensional tensor again to maintain the temporal integrity of the video data. This method simplifies the processing flow of the video frame without losing the prior knowledge of the model, and maintains the ability to process high-dimensional data. In addition, in order to capture the characteristics of the time dimension, the model introduces a one-dimensional time convolution layer and a one-dimensional time self-attention mechanism across time in the internal basic residual block and basic attention block. Among them, the one-dimensional time convolution layer is located after each two-dimensional convolution layer and is responsible for processing time series data, while the one-dimensional time self-attention layer enhances the network's understanding of time continuity in the video by imitating the processing method of the spatial dimension. In order to further clarify the process, the tensor transformation operation is defined as follows. First, the model rearranges the shape of the video batch tensor of shape b×n×c×h×w to (b·n)×c×h×w to process spatial information through two-dimensional convolution. For the processing of temporal information, the shape of the video batch tensor is arranged into (b·h·w)×c×n and (b·h·w)×n×c, so that the temporal convolution layer and the temporal self-attention layer can effectively process the temporal dimension of the features.

[0052] Part 2: Character Feature Injection Module

[0053] The latent space diffusion model enables the U-Net network to work in a latent space with a size lower than the pixel space by introducing a variational autoencoder. This design is crucial to reducing the parameters of the latent diffusion denoising network because it helps to achieve the optimal balance between image quality and computational efficiency. However, although the strategy of latent space compression reduces the computational burden, it also introduces data loss, especially when processing human images containing rich high-frequency details, such as hands, faces, etc., which can lead to inaccurate reconstruction.

[0054] To address the above issues, a simple and crude method is to use a binary mask of shielding clothing to perform pixel shearing, that is, to splice the retained area with the generated area. Although this method is intuitive, it often produces unnatural transition effects at the boundary between the two areas due to the pixel-level structural deviation of the generated network, especially in terms of color and shape. In addition, inaccurate human semantic analysis will also indirectly affect the image quality of the direct splicing method through the binary mask.

[0055] In order to solve this problem, the present invention proposes a character feature injection module, such as Figure 3As shown. The variational autoencoder consists of a pair of encoders and decoders connected in series. The encoder maps the visual image to the latent space layer by layer, while the decoder parses the image representation in the latent space into the corresponding image layer by layer. In order to fully extract the information related to the human body, the character feature injection module transfers the features of each layer in the encoder of the variational autoencoder to the corresponding layer of the decoder, and enhances the retention of human body information during the reconstruction process through the attention mechanism. Specifically, we express this process as the following formula:

[0056]

[0057] Among them, f represents a learnable nonlinear layer, and They represent the output and input of the portrait injection module in the i-th layer of the decoder. When i=1, is the original input fashion image, E i represents the output of the i-th layer of the encoder. Attention() is a standard cross-attention algorithm, where As the query vector, E i As the key vector and value vector. Intuitively explained, the portrait feature injection module takes into account that the temporal enhancement network is based on the downsampled latent space for generation operations, so it is hoped that the original image information without loss can be directly obtained when the synthesized latent representation is decoded, and then the encoding features of the corresponding layer are communicated with the decoding features through the attention mechanism, so that there is an opportunity to make up for the lost detail information in the decoding stage. It is worth noting that the portrait feature injection module is a pluggable component and its training is independent of the U-Net network.

[0058] In step (4), diffusion loss and perceptual loss are used respectively to train the model parameters in the temporal enhancement network and the character feature injection module through the back propagation algorithm until the entire model converges. The details are as follows:

[0059] Regarding the time series enhancement diffusion model, we use a large-scale pre-trained stable diffusion model as the basic model. In order to minimize the loss of prior knowledge, the weights are trained to the input layer of the open model entrance and the time series enhancement module proposed in this invention. We use the basic diffusion loss as the loss function for training this module, which is specifically defined as follows:

[0060]

[0061] Among them, z t 、z a and z w They represent the latent space representation of noise, clothing-independent image, and deformed clothing image at time t, d represents the original character depth posture information downsampled to 1 / 8 by the nearest neighbor, and I grepresents the features of clothing images obtained by the image feature extractor DINO-V2, ε θ represents the noise estimation network, i.e., the time series enhanced diffusion model, and ε represents the target noise at time t.

[0062] Regarding the character feature injection module, in order to reduce the burden of training calculations, this module is designed as a decoupled pluggable module that can be trained independently from the denoising network. The module is trained through the reconstruction task of the autoencoder. The specific perceptual loss is defined as follows:

[0063]

[0064] in, is the mth feature of the VGG-19 network pre-trained on Image-Net, E is the encoding process of the variational autoencoder, and D′ is the decoding process after character feature injection. The characteristic of perceptual loss is that it can be optimized based on human visual perception. Its advantage is that compared with traditional pixel-level loss, it can avoid excessive attention to pixel details, thereby better capturing the high-level semantic information of the image.

Claims

1. A video virtual fitting method based on temporal enhanced diffusion model, characterized in that: The steps include: Step 1: Based on the fashion dataset, collect task data, preprocess the data, and construct fashion video sequence, posture sequence, semantic sequence and depth sequence data sets; Step 2: Use the deformation network to map the clothing to the corresponding deformation space frame by frame according to the posture sequence and the depth sequence, and obtain the deformation clothing sequence as the guiding condition of the diffusion network; The semantic sequence is used to obtain the binary mask of the human head and lower body; Step 3: With the goal of generating fashion fitting videos, a temporal enhancement network and a human feature injection module are constructed based on the standard stable diffusion model, and the deformable clothing sequence and the fashion video sequence are synthesized to obtain a virtual fitting video sequence; Step 4: Use diffusion loss and perceptual loss respectively to train the model parameters in the temporal enhancement network and character feature injection module through the back propagation algorithm until the entire model converges.

2. The video virtual fitting method based on the temporal enhanced diffusion model according to claim 1 is characterized in that: The fashion video sequence, posture sequence, semantic sequence and depth sequence data set described in step 1 refers to estimating a posture sequence of a person from a fashion image using a posture estimator for the corresponding fashion video sequence, and the calculated posture information of the person includes 18 joint coordinate points; A human body parser is used to calculate a human body semantic sequence containing 20 labels, each label represents a part of the human body; a human body depth pose estimator is used to calculate a dense human body depth sequence; a mask is applied to cover the original clothing and its surrounding area to obtain a series of fashion video sequences that are unrelated to clothing.

3. The video virtual fitting method based on the temporal enhanced diffusion model according to claim 2 is characterized in that: The deformation network is a virtual fitting deformation network, which adopts a cyclic appearance flow deformation model. The model consists of three parts: a feature extraction module, a multi-scale global correlation calculation module, and a cyclic appearance flow update module. The deformation effect is achieved by estimating dense flow. The specific implementation is as follows: Pose sequence V used by the deformation network k And the depth sequence V d , the target clothing is mapped to obtain a deformable clothing sequence related to the time dimension Among them, the binary mask V obtained from the semantic sequence s Used to constrain the occlusion relationship between regions during the deformation process.

4. The video virtual fitting method based on the temporal enhanced diffusion model according to claim 3 is characterized in that: The timing enhancement network is specifically as follows: The temporal enhancement network is an improved model architecture based on the stable diffusion model. The stable diffusion model is composed of a pair of variational autoencoders and a U-Net network. The image data is input by the encoder of the variational autoencoder, mapped to a potential representation, calculated by the U-Net network, and then restored by the decoder of the variational autoencoder. The U-Net network consists of several downsampling blocks and several upsampling blocks. The upsampling and downsampling blocks are function groups constructed by connecting residual blocks and attention blocks in series. The improvement measures are as follows: (1) Adjust the way the stable diffusion model handles input: allow it to accept the form of A five-dimensional video tensor of , where b and f represent the batch axis and frame time axis respectively, and c, h, and w represent the number of channels, height, and width of the video frame respectively; (2) Optimized structure: The up-sampling and down-sampling blocks consist of two sets of residual blocks and attention blocks connected in series. The residual blocks are connected in series with several nonlinear convolutional layers and residual connections between the input and output of the blocks. A one-dimensional convolution with residual connection is added after each nonlinear convolutional layer to enable the spatial information on different time frames to interact. The attention block is connected in series with self-attention and cross-attention. One-dimensional self-attention is added after the cross-attention to take into account different time frames when calculating the attention weight.

5. The video virtual fitting method based on the temporal enhanced diffusion model according to claim 4 is characterized in that: The specific implementation process of the character feature injection module is as follows: The variational autoencoder consists of a pair of encoders and decoders connected in series. The encoder maps the visual image to the latent space layer by layer, while the decoder parses the image representation in the latent space into the corresponding image layer by layer. The character feature injection module transfers the features of each layer in the encoder of the variational autoencoder to the corresponding layer of the decoder, and enhances the retention of human body information during the reconstruction process through the attention mechanism. The specific process is expressed as the following formula: Among them, f represents a learnable nonlinear layer, and They represent the output and input of the portrait injection module in the i-th layer of the decoder. When i=1, is the original input fashion image, E i represents the output of the i-th layer of the encoder; Attention() is a cross-attention algorithm, where As the query vector, E i As key vector and value vector; the portrait feature injection module is a pluggable component and its training is independent of the U-Net network.

Images