A video virtual fitting method based on a time-series enhanced diffusion model

Abstract

The application discloses a kind of video virtual fitting methods based on timing enhancement diffusion model, which first based on fashion dataset, constructs fashion video sequence, posture sequence, semantic sequence and depth sequence data group.Secondly, by using morphing network, clothing is mapped to corresponding morphing space according to posture sequence and depth sequence frame by frame, and deformation clothing sequence is obtained as the guide condition of diffusion network.Then, with the goal of generating fashion fitting video, a timing enhancement network and a human feature injection module are constructed based on a standard stable diffusion model, and the deformation clothing sequence and the fashion video sequence are synthesized to obtain a virtual fitting video sequence.Finally, using diffusion loss and perception loss respectively, the model parameters in the timing enhancement network and the human feature injection module are trained through a backpropagation algorithm.The application improves the user experience and promotes the application and development of virtual fitting technology in e-commerce and fashion industry.

Description

Technical Field

[0001] The present invention belongs to the technical field of virtual fitting, and in particular relates to a video virtual fitting method based on a temporal enhancement diffusion model. Background Art

[0002] With the continuous development of the virtual fitting field, deep learning-driven image synthesis technology has become the key to achieving a highly realistic virtual clothing fitting experience. In particular, generative diffusion models have achieved remarkable results in image synthesis tasks in recent years.

[0003] The image-based virtual fitting algorithm is divided into two modules: a garment deformation module and a synthesis module. The garment deformation module gradually shifts from a thin plate spline deformation method to a flow-based algorithm, achieving the mapping of the target garment to the human pose. The synthesis module aims to naturally transfer the deformable garment to the human body image, which requires overcoming numerous challenges such as occlusion and boundary transitions. In recent years, the diffusion model has garnered widespread attention due to its significant advantages over generative adversarial networks in terms of image generation quality and training stability. The core algorithm of the synthesis module has also shifted from a generative adversarial network to a diffusion model.

[0004] The task of video virtual fitting involves inputting a series of video frames and images of people trying on clothing, and ultimately outputting a new video. Due to the continuous changes in a person's posture, position, and size in a video, video virtual fitting requires not only processing complex dynamic scenes but also accurately capturing the coordinated changes in clothing and human movement, while ensuring smooth and continuous visual effects between the generated video frames. Experimental results show that using only a standard diffusion model to address this problem indicates that the model cannot directly understand the semantic information of the multiple input images and their relationships, making it difficult to address the many challenges posed by the task.

[0005] (1) Fidelity of clothing transfer: Clothing transfer needs to consider the texture information of different regions and the human body posture information. It is necessary to ensure the accurate mapping of clothing on the human body and correctly handle the occlusion relationship between objects.

[0006] (2) Consistency of character details: The stable diffusion model compresses the image into a latent space representation, resulting in a loss of image semantic information. Therefore, preserving details about the character’s visual information has become one of the challenges of video virtual fitting.

[0007] (3) Continuity of inter-frame transitions: The current mainstream diffusion model cannot process and preserve the structural information between frames. If the video sequence is simply processed frame by frame, inter-frame flickering and inconsistent contextual semantics will occur. Summary of the Invention

[0008] This paper provides a video virtual fitting method based on a temporal enhancement diffusion model. This method uses a two-stage network, a temporal enhancement diffusion network, and a character feature injection module to generate natural, realistic fitting videos. Experiments on the VVT ​​dataset demonstrated excellent quantitative and qualitative results. This paper primarily involves a temporal enhancement technique applied to a diffusion model, modeling time series to achieve consistency and continuity across the temporal dimension; and utilizing a cross-attention mechanism to inject semantic features, generating realistic and effective fashion fitting videos.

[0009] A video virtual fitting method based on a temporal enhanced diffusion model, the steps of which are as follows:

[0010] Step (1) collects task data based on the existing fashion dataset, preprocesses the data, and constructs fashion video sequence, posture sequence, semantic sequence and depth sequence data sets.

[0011] Step (2) uses an advanced deformation network to map the clothing to the corresponding deformation space frame by frame based on the posture sequence and depth sequence, and obtains the deformable clothing sequence as the guidance condition of the diffusion network. The semantic sequence is used to obtain the binary mask of the human head and lower body to prevent occlusion between the relevant areas and the deformable clothing.

[0012] Step (3) aims to generate natural and accurate fashion fitting videos. Based on the standard stable diffusion model, a temporal enhancement network and a human feature injection module are constructed to synthesize the deformable clothing sequence and the fashion video sequence to obtain a virtual fitting video sequence, thus achieving the unity of inter-frame continuity and target texture consistency.

[0013] In step (4), diffusion loss and perceptual loss are used to train the model parameters in the temporal enhancement network and the character feature injection module through the back propagation algorithm until the entire model converges.

[0014] The fashion video sequence, pose sequence, semantic sequence, and depth sequence data set described in step (1) refers to estimating the pose sequence of a person from a fashion image using a state-of-the-art pose evaluator for the corresponding fashion video sequence, and the calculated pose information of the person includes 18 joint coordinate points. In addition, an advanced human body parser is used to calculate a human body semantic sequence containing 20 labels, each label representing a specific part of the human body, such as the face, hair, arms, legs, and clothing area. In addition, an advanced human body depth pose estimator is used to calculate a dense depth sequence of the human body. In addition, following the consistent clothing masking rules, a mask is applied to cover the original clothing and its surrounding areas, thereby obtaining a series of fashion video sequences that are unrelated to clothing.

[0015] The deformation network described in step (2) is the virtual fitting deformation network. The present invention adopts a cyclic appearance flow deformation model, which consists of three parts: a feature extraction module, a multi-scale global correlation calculation module, and a cyclic appearance flow update module. The deformation effect is achieved by estimating dense flow. In simple terms, the deformation network uses the posture sequence V in step (1) k And the depth sequence V d , the target clothing is mapped to obtain a deformable clothing sequence related to the time dimension Among them, the binary mask V obtained from the semantic sequence s It is used to constrain the occlusion relationship between regions during the deformation process.

[0016] The temporal enhancement network and human feature injection module described in step (3) achieve the unification of inter-frame continuity and target texture consistency, as follows:

[0017] Part 1: Timing Enhanced Networks

[0018] The temporal enhancement network proposed in the present invention is an improved model architecture based on the stable diffusion model, which ensures accurate modeling of video frame content and in-depth consideration of the relationship between frames. The stable diffusion model is essentially composed of a pair of variational autoencoders and a U-Net network. The image data is input by the encoder of the variational autoencoder, mapped into a potential representation, calculated by the U-Net network, and then handed over to the decoder of the variational autoencoder for image restoration. The U-Net network is composed of several downsampling blocks and several upsampling blocks. The upsampling and downsampling blocks are function groups constructed in series by residual blocks and attention blocks. The specific structure will be described in detail in the "Optimized Network Structure" below. The temporal enhancement network aims to improve the generalization ability of the model while leveraging the training results of large-scale image datasets, especially the generation ability in the fashion field. And by processing the time dimension information, the performance of video generation is improved. Improvements include: 1) Adjusting the processing method of the stable diffusion model input: allowing it to accept the form of , where b and f represent the batch axis and the frame time axis respectively, and c, h, and w represent the number of channels, height, and width of the video frame respectively; 2) Optimize the network structure to better capture the continuity of the time dimension: the up and down sampling blocks contain two sets of residual blocks and attention blocks in series. The residual blocks are connected in series with several nonlinear convolution layers and residual connections of the input and output within the blocks. The present invention adds a one-dimensional convolution with residual connection (hereinafter referred to as one-dimensional temporal convolution) after each nonlinear convolution layer to enable the spatial information on different time frames to interact; the attention block is connected in series with self-attention and cross-attention. The present invention adds one-dimensional self-attention (hereinafter referred to as one-dimensional temporal self-attention) after the cross-attention to realize that different time frames are taken into consideration when calculating the attention weight.

[0019] Part 2: Character Feature Injection Module

[0020] This paper proposes a human feature injection module. A variational autoencoder consists of a cascaded encoder and decoder. The encoder maps the visual image layer by layer into a latent space, while the decoder parses the image representation in the latent space into the corresponding image layer by layer. To fully extract information related to the human body, the human feature injection module transfers features from each layer of the variational autoencoder encoder to the corresponding layer of the decoder. It also uses an attention mechanism to enhance the preservation of human body information during the reconstruction process.

[0021] In step (4), the diffusion loss and perceptual loss are used to train the model parameters in the temporal enhancement network and the character feature injection module through the back propagation algorithm until the entire model converges. The details are as follows:

[0022] Regarding the time series enhancement diffusion model, a large-scale pre-trained stable diffusion model is used as the base model. To minimize the loss of prior knowledge, weights are trained to the input layer of the open model entrance and the time series enhancement module proposed in this invention. The basic diffusion loss is used as the loss function for training this module, which is specifically defined as follows:

[0023]

[0024] Among them, z t 、z a and z w They represent the latent space representation of noise, clothing-independent image, and deformed clothing image at time t, d represents the original character depth posture information downsampled to 1 / 8 by the nearest neighbor, and I g represents the features of clothing images obtained by the image feature extractor DINO-V2, ε θ represents the noise estimation network, i.e., the time series enhanced diffusion model, and ε represents the target noise at time t.

[0025] To reduce the computational burden of training, the character feature injection module is designed as a decoupled, pluggable module that can be trained independently from the denoising network. The module is trained using the autoencoder's reconstruction task. The specific perceptual loss is defined as follows:

[0026]

[0027] in, is the mth feature of the VGG-19 network pre-trained on Image-Net, E is the encoding process of the variational autoencoder, and D′ is the decoding process after the character feature injection.

[0028] Beneficial effects of the present invention:

[0029] The present application proposes to introduce a time consistency enhancement module and a human feature injection module, which can retain more character details in the video sequence while maintaining high-quality image generation, thereby significantly improving the quality and continuity of video generation, solving the problems of inter-frame jitter, loss of character details, poor clothing fidelity and other problems existing in the prior art, making the video virtual fitting task have the possibility of landing practice. In addition, the task of combining artificial intelligence with fashion as the current research hotspot, through the multi-modal generation of integrating posture, dense posture, morphing clothing image and other information also makes the present application more scientific and innovative. By enhancing the continuity and detail retention of virtual fitting videos, this method helps to improve the user experience, thereby promoting the application and development of virtual fitting technology in e-commerce and fashion industry. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is a specific flowchart of the method of the present application;

[0031] Figure 2 is a timing enhancement module schematic diagram in the method of the present application;

[0032] Figure 3 is a character feature injection module schematic diagram in the method of the present application;

[0033] Figure 4 is a schematic diagram of the overall framework of the present application;

[0034] Figure 5 is a data set schematic diagram of the present application. DETAILED DESCRIPTION

[0035] The present application will be further described below in conjunction with the drawings and examples.

[0036] The present application provides a video virtual fitting method based on a timing enhancement diffusion model, as shown in Figure 1 , the method uses a two-stage network, respectively using a timing enhancement diffusion network and a character feature injection module to realize natural and realistic fitting video generation, and we have carried out experiments on the VVT dataset, and good results have been achieved in quantitative and qualitative aspects.

[0037] A video virtual fitting method based on a timing enhancement diffusion model is shown in Figure 1 . The steps are as follows:

[0038] Step (1), based on the existing fashion dataset, collect the data of the task, and preprocess the data to construct fashion video sequence, posture sequence, semantic sequence, depth sequence data set.

[0039] Step (2) uses an advanced deformation network to map the clothing to the corresponding deformation space frame by frame based on the posture sequence and depth sequence, and obtains the deformable clothing sequence as the guidance condition of the diffusion network. The semantic sequence is used to obtain the binary mask of the human head and lower body to prevent occlusion between the relevant areas and the deformable clothing.

[0040] Step (3) aims to generate natural and accurate fashion fitting videos. Based on the standard stable diffusion model, a temporal enhancement network and a human feature injection module are constructed to synthesize the deformable clothing sequence and the fashion video sequence to obtain a virtual fitting video sequence, thus achieving the unity of inter-frame continuity and target texture consistency.

[0041] In step (4), diffusion loss and perceptual loss are used to train the model parameters in the temporal enhancement network and the character feature injection module through the back propagation algorithm until the entire model converges.

[0042] The step (1) of collecting task data based on the existing fashion dataset refers to evaluating the method of the present invention on the VVT ​​dataset. The VVT ​​dataset includes 791 fashion model catwalk videos, which are divided into 661 training videos and 130 test videos. Each video contains 250 to 300 frames. Data preprocessing filters out frames where no characters appear at the beginning and end of the video. In addition to the 159,170 frames of the training set and the 30,931 frames of the test set, the dataset also provides corresponding clothing images for each video. Fashion frame images and fashion clothing images reference Figure 5 ,Where the fashion frame is any unit of the fashion video sequence, the video sequence contains three parts: the model approaches from outside the camera, turns around, and walks out of the camera, and the camera is fixed.

[0043] The fashion video sequence, posture sequence, semantic sequence, and depth sequence data set described in step (1) refers to the use of the most advanced posture evaluator to estimate the posture sequence of a person from the fashion image for the corresponding fashion video sequence, and the calculated posture information of the person contains 18 joint coordinate points. In addition, an advanced human body parser is used to calculate a human body semantic sequence containing 20 labels, each label representing a specific part of the human body, such as the face, hair, arms, legs, and clothing area. And, an advanced human body depth posture estimator is used to calculate a dense depth sequence of the human body. In addition, following the consistent clothing masking rules, a mask is applied to cover the original clothing and its surrounding areas, thereby obtaining a series of fashion video sequences that are not related to clothing. Specific data group examples are as follows: Figure 5 As shown in the figure, different relevant information is obtained based on the fashion frame image. The human body analysis image divides the parts of the human body representing different semantic information into 18 blocks of different colors. The key point image describes the 18 joint points that symbolize the human body posture information. The clothing-independent image generates a mask based on the human body analysis image, referring to the human body arms, torso and other parts.

[0044] The deformation network described in step (2) is the virtual fitting deformation network. The present invention adopts a cyclic appearance flow deformation model, which consists of three parts: a feature extraction module, a multi-scale global correlation calculation module, and a cyclic appearance flow update module. The deformation effect is achieved by estimating dense flow. In simple terms, the deformation network uses the posture sequence V in step (1) k And the depth sequence V d , the target clothing is mapped to obtain a deformable clothing sequence related to the time dimension Among them, the binary mask V obtained from the semantic sequence s It is used to constrain the occlusion relationship between regions during the deformation process. The resulting deformed clothing frame can be referenced Figure 5 As shown in Figure 3, the deformable clothing not only retains the texture information of the clothing itself, but also maps it to a spatial position that conforms to the human body posture.

[0045] The temporal enhancement network and human feature injection module described in step (3) optimize the stable diffusion model to achieve the unity of inter-frame continuity and target texture consistency. The overall framework diagram of the design of the present invention is shown in FIG. Figure 4 Specific instructions are as follows:

[0046] Part 1: Timing Enhanced Networks

[0047] During video generation, it is crucial not only to ensure high-quality image generation within each frame, but also to ensure temporal smoothness and continuity between consecutive frames. In the task of generating virtual fittings in videos, the differences in various input conditions (such as human posture, human semantic parsing, and deformable clothing) within each frame can lead to discontinuous spatial motion between consecutive frames, visually creating so-called "inter-frame jitter." Furthermore, due to the working principle of the probability diffusion model, random noise sequences need to be sampled and denoised, further increasing the challenge of maintaining video continuity.

[0048] like Figure 2As shown, the temporal enhancement network proposed in the present invention is an improved model architecture based on the stable diffusion model, which ensures accurate modeling of video frame anger and in-depth consideration of the relationship between frames. The stable diffusion model is essentially composed of a pair of variational autoencoders and a U-Net network. The image data is input by the encoder of the variational autoencoder, mapped into a potential representation, calculated by the U-Net network, and then handed over to the decoder of the variational autoencoder for image restoration. The U-Net network is composed of several downsampling blocks and several upsampling blocks. The upsampling and downsampling blocks are function groups constructed in series by residual blocks and attention blocks. The specific structure will be described in detail in the "Optimized Network Structure" below. This architecture aims to improve the generalization ability of the model, especially the generation ability in the fashion field, while leveraging the training results of large-scale image datasets. And by processing time dimension information, the performance of video generation is improved. Improvements include: 1) Adjusting the processing method of the stable diffusion model input to allow it to accept the form of , where b and f represent the batch axis and the frame time axis respectively, and c, h, and w represent the number of channels, height, and width of the video frame respectively; 2) Optimize the network structure to better capture the continuity of the time dimension: the up and down sampling blocks contain two sets of residual blocks and attention blocks in series. The residual blocks are connected in series with several nonlinear convolution layers and residual connections of the input and output within the blocks. The present invention adds a one-dimensional convolution with a residual connection after each nonlinear convolution layer (hereinafter referred to as one-dimensional time convolution) to enable the spatial information on different time frames to interact; the attention block is connected in series with self-attention and cross-attention. The present invention adds one-dimensional self-attention (hereinafter referred to as one-dimensional time self-attention) after the cross-attention to realize that different time frames are taken into account when calculating the attention weight. The definition of one-dimensional time self-attention is as follows:

[0049]

[0050] Among them, Q t ,K t ,V t Represent the query features, key features, and value features after dimension reorganization. We jointly model the spatial and temporal dimensions so that spatial blocks can interact in the temporal dimension, thereby obtaining temporal information. k K t By scaling the features and normalizing them, we can prevent gradient disappearance and gradient explosion while ensuring the stability of feature distribution.

[0051] Specifically, when processing the latent features obtained by the variational autoencoder based on the video sequence and the deformable clothing sequence, the U-Net network of the present invention reshapes the time axis f to the batch axis b and temporarily ignores it, that is, temporarily ignores the temporal relationship between the sequences. This allows the network to independently process the input video frame sequence through standard two-dimensional convolutional layers. After passing through the decoder, the feature map is reshaped into a five-dimensional tensor to maintain the temporal integrity of the video data. This method not only simplifies the processing flow of video frames, but also does not lose the model's prior knowledge, maintaining the ability to process high-dimensional data. In addition, to capture the characteristics of the temporal dimension, the model introduces a cross-temporal one-dimensional temporal convolution layer and a one-dimensional temporal self-attention mechanism in the internal basic residual block and basic attention block. Among them, the one-dimensional temporal convolution layer is located after each two-dimensional convolution layer and is responsible for processing time series data, while the one-dimensional temporal self-attention layer enhances the network's understanding of temporal continuity in the video by mimicking the processing method of the spatial dimension. To further illustrate the process, the tensor transformation operation is defined as follows. First, the model rearranges the shape of the video batch tensor of shape b×n×c×h×w to (b·n)×c×h×w to perform two-dimensional convolution to process spatial information. For the processing of temporal information, the shape of the video batch tensor is rearranged to (b·h·w)×c×n and (b·h·w)×n×c, so that the temporal convolution layer and temporal self-attention layer can effectively process the temporal dimension of the features.

[0052] Part 2: Character Feature Injection Module

[0053] The latent space diffusion model introduces a variational autoencoder, enabling the U-Net network to operate in a latent space smaller than the pixel space. This design is crucial for reducing the parameters of the latent diffusion denoising network, as it helps achieve an optimal balance between image quality and computational efficiency. However, while adopting a latent space compression strategy reduces the computational burden, it also introduces data loss, especially when processing human images containing rich high-frequency details, such as hands and faces, which can lead to inaccurate reconstruction.

[0054] To address the above issues, a crude approach is to use a binary mask that blocks clothing for pixel shearing, i.e., splicing the retained area with the generated area. Although this method is intuitive, it often produces unnatural transition effects at the boundary between the two areas due to the pixel-level structural deviations of the generated network, especially in terms of color and shape. In addition, inaccurate human semantic parsing can also indirectly affect the image quality of the direct splicing method through the binary mask.

[0055] In order to solve this problem, the present invention proposes a character feature injection module, such as Figure 3As shown. The variational autoencoder consists of a pair of encoders and decoders connected in series. The encoder maps the visual image layer by layer to the latent space, while the decoder parses the image representation in the latent space into the corresponding image layer by layer. In order to fully extract the information related to the human body, the human feature injection module transfers the features of each layer in the encoder of the variational autoencoder to the corresponding layer of the decoder, and enhances the preservation of human body information during the reconstruction process through the attention mechanism. Specifically, we express this process as the following formula:

[0056]

[0057] Where f represents a learnable nonlinear layer, and They represent the output and input of the portrait injection module in the i-th layer of the decoder. When i=1, That is the original input fashion image, E i Represents the output of the encoder layer i. Attention() is a standard cross-attention algorithm, where As the query vector, E i As key vectors and value vectors. Intuitively, the portrait feature injection module takes into account that the temporal enhancement network performs generative operations based on a downsampled latent space. Therefore, it hopes to directly obtain the original image information without loss when decoding the synthesized latent representation. Furthermore, the encoded features of the corresponding layer are communicated with the decoded features through an attention mechanism, allowing for the opportunity to compensate for lost detail information during the decoding phase. It is worth noting that the portrait feature injection module is a pluggable component and is trained independently of the U-Net network.

[0058] In step (4), the diffusion loss and perceptual loss are used to train the model parameters in the temporal enhancement network and the character feature injection module through the back propagation algorithm until the entire model converges. The details are as follows:

[0059] Regarding the time series enhancement diffusion model, we use a large-scale pre-trained stable diffusion model as the base model. To minimize the loss of prior knowledge, we train weights to the input layer of the open model entrance and the time series enhancement module proposed in this invention. We use the basic diffusion loss as the loss function for training this module, which is specifically defined as follows:

[0060]

[0061] Among them, z t 、z a and z w They represent the latent space representation of noise, clothing-independent image, and deformed clothing image at time t, d represents the original character depth posture information downsampled to 1 / 8 by the nearest neighbor, and I grepresents the features of clothing images obtained by the image feature extractor DINO-V2, ε θ represents the noise estimation network, i.e., the time series enhanced diffusion model, and ε represents the target noise at time t.

[0062] To reduce the computational burden of training, the character feature injection module is designed as a decoupled, pluggable module that can be trained independently from the denoising network. The module is trained using the autoencoder's reconstruction task. The specific perceptual loss is defined as follows:

[0063]

[0064] in, is the mth feature of the VGG-19 network pre-trained on Image-Net, E is the encoding process of the variational autoencoder, and D′ is the decoding process after injecting character features. The characteristic of perceptual loss is that it can optimize based on human visual perception. Its advantage is that compared with traditional pixel-level losses, it can avoid excessive focus on pixel details, thereby better capturing the high-level semantic information of the image.

Claims

1. A video virtual fitting method based on temporal enhanced diffusion model, characterized in that: The steps include: Step 1: Based on the fashion dataset, collect task data, preprocess the data, and construct fashion video sequence, posture sequence, semantic sequence, and depth sequence data sets; Step 2: Use the deformation network to map the clothing to the corresponding deformation space frame by frame based on the posture sequence and depth sequence, and obtain the deformable clothing sequence as the guidance condition of the diffusion network; The semantic sequence is used to obtain the binary mask of the human head and lower body; Step 3: To generate fashion fitting videos, we build a temporal enhancement network and a character feature injection module based on the standard stable diffusion model. We synthesize the deformable clothing sequence with the fashion video sequence to obtain a virtual fitting video sequence. The timing enhancement network is specifically as follows: The Temporal Enhancement Network is an improved model architecture based on the Stable Diffusion Model. The Stable Diffusion Model consists of a pair of variational autoencoders and a U-Net network. Image data is input by the variational autoencoder encoder, mapped to a latent representation, calculated by the U-Net network, and then restored by the variational autoencoder decoder. The U-Net network consists of several downsampling blocks and several upsampling blocks. The upsampling and downsampling blocks are function groups constructed by connecting residual blocks and attention blocks in series. The improvement measures are as follows: (1) Adjust the way the stable diffusion model handles input: allow it to accept the form of A five-dimensional video tensor, where b and f represent the batch axis and frame time axis respectively, and c, h, and w represent the number of channels, height, and width of the video frame respectively; (2) Optimized structure: The upsampling and downsampling blocks contain two sets of residual blocks and attention blocks in series. The residual blocks are connected in series with several nonlinear convolution layers and residual connections between the input and output of the blocks. A one-dimensional convolution with residual connection is added after each nonlinear convolution layer to enable the interaction of spatial information on different time frames. The attention block is connected in series with self-attention and cross-attention. One-dimensional self-attention is added after the cross-attention to take into account different time frames when calculating the attention weight. The specific implementation process of the character feature injection module is as follows: The variational autoencoder consists of a pair of encoders and decoders connected in series. The encoder maps the visual image layer by layer to the latent space, while the decoder parses the image representation in the latent space into the corresponding image layer by layer. The character feature injection module transfers the features of each layer in the encoder of the variational autoencoder to the corresponding layer of the decoder, and enhances the preservation of human body information during the reconstruction process through the attention mechanism. The specific process is expressed as the following formula: Where f represents a learnable nonlinear layer, and They represent the output and input of the portrait injection module in the i-th layer of the decoder. When i=1, That is the original input fashion image, E i Represents the output of the encoder layer i; Attention() is a cross attention algorithm, where As the query vector, E i As key vectors and value vectors; the portrait feature injection module is a pluggable component and is trained independently of the U-Net network; Step 4: Use diffusion loss and perceptual loss respectively to train the model parameters in the temporal enhancement network and character feature injection module through the back propagation algorithm until the entire model converges.

2. The video virtual fitting method based on the temporal enhanced diffusion model according to claim 1 is characterized in that: The fashion video sequence, pose sequence, semantic sequence, and depth sequence data set described in step 1 refers to estimating a person's pose sequence from a fashion image using a pose estimator for the corresponding fashion video sequence, where the calculated pose information of the person includes 18 joint coordinate points; A human body parser is used to calculate a human body semantic sequence containing 20 labels, each label representing a part of the human body; a human body depth pose estimator is used to calculate a dense human body depth sequence; a mask is applied to cover the original clothing and its surrounding area to obtain a series of fashion video sequences unrelated to clothing.

3. The video virtual fitting method based on the temporal enhanced diffusion model according to claim 2 is characterized in that: The deformation network is a virtual fitting deformation network that uses a cyclic appearance flow deformation model. The model consists of three parts: a feature extraction module, a multi-scale global correlation calculation module, and a cyclic appearance flow update module. The deformation effect is achieved by estimating dense flow. The specific implementation is as follows: The pose sequence V used by the deformation network k And the depth sequence V d , the target clothing is mapped to obtain a deformable clothing sequence related to the time dimension Among them, the binary mask V obtained from the semantic sequence s Used to constrain the occlusion relationship between regions during the deformation process.

Images