Video virtual try-on method, device and program product based on diffusion model

Abstract

The application belongs to the technical field of computers, and provides a video virtual try-on method, device and program product based on a diffusion model. First, a target garment image and a model video are acquired, and a deep feature, a mask and a model latent feature video are extracted therefrom; meanwhile, a semantic code of the garment is extracted through a CLIP encoder. A first diffusion model processes the garment image and the CLIP code, and outputs a garment spatial attention feature vector that fuses fine-grained texture and semantics; a second diffusion model serves as a video diffusion network, receives random noise, a person feature, a mask and a deep feature, and injects a garment vector and the CLIP code into an intermediate layer, and finally outputs a try-on video that is natural and dynamically coherent. The application can efficiently generate a high-fidelity virtual try-on video according to a new garment and a model video, and has wide application value in the field of e-commerce.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, in particular to a video virtual try-on method, device and program product based on a diffusion model. BACKGROUND

[0002] Video-based virtual try-on is an important computer vision task, which aims to generate seamless videos of a person wearing a target garment by inputting a person video and a target garment image. Due to its convenience and potential for providing personalized shopping experiences for users in e-commerce, generating realistic virtual try-on videos has attracted widespread interest. The main challenge of video virtual try-on is how to naturally match the garment with the person under different poses and actions, and keep the patterns and textures of the garment from being distorted, while ensuring the coherence of the video and the natural deformation of the garment under various actions.

[0003] Existing video-based virtual try-on methods mainly rely on generative adversarial networks. These methods first warp the garment image to match a given person video, and then combine the deformed image with the person video into a generator for synthesis. However, the try-on methods based on generative adversarial networks face the following problems: (1) the warping method cannot handle challenging poses; (2) the generated video often lacks a certain degree of realism and may not generate finer details; (3) the inter-frame of the generated video is not coherent enough, and the clothing texture changes unnaturally.

[0004] Pre-trained diffusion models have shown excellent performance in various visual tasks, and compared with generative adversarial networks, diffusion models perform well in generating images with fine-grained realism. When pre-trained diffusion models are applied to the virtual try-on task, the key problems are how to ensure the controllability of the generated results, especially to maintain the complex textures and patterns of the target garment, and how to ensure the coherence of the dynamic clothing, which have not been effectively solved so far. SUMMARY

[0005] To solve the above technical problems, the present application provides a video virtual try-on method, device and program product based on a diffusion model.

[0006] The application provides a video virtual try-on method based on a diffusion model, comprising the following steps: obtaining a first garment image and a first model video wearing a garment different from the first garment image, obtaining a depth feature video and a mask video respectively according to the first model video, and obtaining a model latent feature video by combining the model video and the mask video; inputting the first garment image into a CLIP encoder to obtain a CLIP code; inputting the first garment image and the CLIP code into a first diffusion model, and outputting a garment space attention feature vector from an intermediate layer of the first diffusion model; inputting linked random noise, the model latent feature video, the mask video and the depth feature video into a second diffusion model, and injecting the garment space attention feature vector and the CLIP code into an intermediate layer of the second diffusion model, so that the second diffusion model outputs a first try-on video; training the first diffusion model and the second diffusion model by using a preset loss parameter, the preset loss parameter comprising a first preset loss, and the first preset loss being a difference loss of the output first try-on video and the model video; based on a second garment image and a second model video wearing a garment different from the second garment image, obtaining a second try-on video by using the trained first diffusion model and the second diffusion model.

[0007] The application also provides a device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the method according to any one of the preceding embodiments when executing the computer program.

[0008] The application also provides a program product having computer instructions for executing the method according to any one of the preceding embodiments.

[0009] The application realizes high-fidelity and time-sequential video virtual try-on through the cooperation of the double diffusion model and the multi-level conditional injection mechanism. Specifically, 1) the garment space attention feature vector is used to accurately maintain the garment texture and structural details, solving the texture distortion problem; 2) the combination of the depth feature, the mask and the CLIP code realizes the natural space alignment of the garment and the complex human body posture; 3) the video diffusion model and the time-sequential coherence loss ensure the dynamic consistency of the cross-frame garment deformation; 4) through the end-to-end training framework, the realistic try-on effect is generated while the identity and the background of the person are preserved, which helps to significantly improve the realism and practicality of the virtual try-on. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 A flowchart of the video virtual try-on method based on the diffusion model disclosed in the embodiments of the application.

[0011] Figure 2 A diffusion model framework for implementing the video virtual try-on method based on the diffusion model disclosed in the embodiments of the application.

[0012] Figure 3For Figure 2 The model framework diagram of the spatial self-attention module and the spatial cross-attention module in the intermediate layer of the second diffusion model.

[0013] Figure 4 For Figure 2 The model framework diagram of the clothing spatial-temporal attention module in the intermediate layer of the second diffusion model.

[0014] Figure 5 The structural diagram of the device disclosed in the embodiment of the present application. DETAILED DESCRIPTION

[0015] The technical solutions of the present application will be further described in detail below with the aid of the accompanying drawings and embodiments.

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with the aid of the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0017] To facilitate the understanding of the embodiments of the present application, the embodiments will be further explained and described below with the aid of specific embodiments combined with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present application.

[0018] As Figure 1 shown, the embodiment of the present application discloses a video virtual try-on method based on a diffusion model, including the following steps: step S100, obtaining a first clothing image and a first model video wearing a first clothing image different from the first clothing image, obtaining a depth feature video and a mask video according to the first model video, and obtaining a model latent feature video combined with the model video and the mask video.

[0019] In this step, two inputs are received: one is the static image of the target clothing, i.e., the first clothing image, such as the clothing picture to be sold displayed by the seller of the e-commerce platform; the other is the reference person video, i.e., the first model video, such as a video of the potential buyer wearing ordinary clothes (different from the target clothing) uploaded by himself.

[0020] The first model video is processed frame by frame as follows: the pre-trained depth estimation model (such as MiDaS, etc.) is used to process each frame of the video to generate a corresponding depth map sequence, i.e., to obtain a depth feature video. It can be understood that the depth map sequence provides the three-dimensional spatial structure information of the person, which helps the model to understand the front-back hierarchical relationship of the human body.

[0021] The pre-trained human parsing or instance segmentation model (such as Mask2Former, etc.) is used to process each frame of the video to generate an accurate person mask sequence, that is, to obtain a mask video. It can be understood that the mask is used to distinguish the person region from the background region in the video, to ensure that the subsequent generation process only changes the clothing region of the person and keeps the background unchanged.

[0022] The original video frame is combined with the corresponding mask, and an encoder (such as the encoder of VAE or a similar feature extraction network) is used for encoding to convert the high-dimensional pixel space image into a low-dimensional, dense latent feature representation, thereby obtaining a model latent feature video.

[0023] In step S200, the first clothing image is input into a CLIP encoder to obtain a CLIP code.

[0024] This step aims to extract the semantic information of the target clothing. Specifically, the CLIP model pre-trained on a large-scale image-text pair is used, and its image encoder is applied to the first clothing image. The CLIP encoder can extract a feature vector containing rich semantic and style information, i.e., a CLIP code. It can be understood that the CLIP code represents the style, texture, color, etc. of the clothing from a high-level semantic level, providing strong semantic guidance for the subsequent generation process.

[0025] In step S300, the first clothing image and the CLIP code are input into the first diffusion model, and a clothing space attention feature vector is output from the intermediate layer of the first diffusion model.

[0026] In this step, the first diffusion model is responsible for processing the target clothing image. As shown in Figure 2 , specifically, the first diffusion model receives the original first clothing image and the CLIP code obtained in step S200 as conditional input; in the denoising process of the model (usually in the intermediate layer of the UNet structure), a clothing space attention feature vector is generated through the internal spatial attention mechanism. The vector not only contains the semantic information of the clothing (derived from the CLIP code), but more importantly, it encodes the spatial structure and texture detail distribution relationship of the clothing image, providing fine-grained spatial guidance for subsequent precise fitting of the clothing in the video.

[0027] In step S400, the linked random noise, the model latent feature video, the mask video, and the depth feature video are input into the second diffusion model, and the clothing space attention feature vector and the CLIP code are injected into the intermediate layer of the second diffusion model, so that the second diffusion model outputs a first try-on video.

[0028] As shown in Figure 3 , Figure 4As shown, this step is performed by the second diffusion model (i.e., the video diffusion model) to realize the fusion of the garment image and the model video, i.e., to generate a try-on video. Specifically, the input of the second diffusion model includes: the linked random noise, which serves as the starting point of the generation process and provides randomness and diversity; the model latent feature video, which provides the pose, action sequence and body shape basis of the target person; the mask video, which explicitly indicates the area of the person that needs to be dressed; and the depth feature video, which provides the spatial structure information of the person and assists in generating a garment deformation with reasonable occlusion and three-dimensional sense.

[0029] To ensure that the generated garment is consistent with the target, the garment spatial attention feature vector obtained in step S300 and the CLIP code obtained in step S200 are introduced into the middle layer of the second diffusion model through a feature injection mechanism (e.g., cross-attention or feature map splicing), so that the second diffusion model can simultaneously refer to the fine-grained spatial details and high-level semantics of the garment when generating each frame.

[0030] Based on all the above conditions, the second diffusion model performs a denoising process from noise to clear video, and finally outputs a coherent first try-on video in which the person is wearing the target garment.

[0031] In step S500, the first diffusion model and the second diffusion model are trained using preset loss parameters, which include a first preset loss. The first preset loss is the difference loss between the output first try-on video and the model video.

[0032] In this step, the first diffusion model and the second diffusion model are jointly or alternately trained by optimizing the preset loss function set. The preset loss parameters include a core loss and other preset losses.

[0033] The core loss is the first preset loss, which is a reconstruction loss used to calculate the difference (e.g., using L1, L2 loss or perceptual loss) between the first try-on video generated by the second diffusion model and the original first model video in the pixel level or feature space. It can be understood that this loss can ensure that the identity, pose and background of the generated video remain consistent with the original video, and only the garment part is changed. Other preset losses may also include, for example: an adversarial loss using a discriminator to improve the realism of the generated video; a temporal coherence loss to ensure smooth transitions between video frames and natural garment dynamics; and a garment preservation loss to encourage the garment in the generated video to be consistent with the appearance of the input target garment image.

[0034] In step S600, based on a second garment image and a second model video wearing a second garment different from the second garment image, a second try-on video is obtained using the trained first diffusion model and second diffusion model.

[0035] This step is the reasoning or application stage of the present application. When the diffusion model training is completed, it can be put into actual use. For a new set of inputs: a new target clothing image (i.e. the second clothing image) and a new reference person video (i.e. the second model video), the trained first diffusion model and second diffusion model are directly called, and the processes of steps S100 to S400 (feature extraction, encoding, condition injection, generation) are repeatedly executed, and finally a second try-on video of virtually trying on the new clothing on the new person video is obtained.

[0036] In this embodiment, through the cooperation of the double diffusion model and the multi-level condition injection mechanism, high-fidelity and time-sequential video virtual try-on is achieved. Specifically, 1) the clothing space attention feature vector is used to accurately maintain the clothing texture and structural details, solving the texture distortion problem; 2) the natural space alignment of clothing and complex human poses is realized by combining depth features, masks and CLIP encoding; 3) based on the video diffusion model and the time-sequential coherence loss, the dynamic consistency of cross-frame clothing deformation is ensured; 4) through the end-to-end training framework, the identity and background of the person are preserved while generating realistic try-on effects, which helps to significantly improve the realism and practicality of virtual try-on.

[0037] As an example, the depth feature video and the mask video are obtained according to the model video respectively, and the model latent feature video is obtained by combining the model video and the mask video, including: using DensePose to extract the model video frame by frame to obtain a depth video, using a pose guide to encode the depth video to obtain the depth feature video; using SAM to extract the model video frame by frame to obtain a mask video; using a variational differential autoencoder to encode the model video and multiply it with the mask video to obtain the model latent feature video.

[0038] In this embodiment, DensePose tool is used to process the first model video frame by frame. It can be understood that DensePose can densely correspond the 2D pixel points of the human body in the video to the 3D human body surface model, thereby generating a sequence containing the depth information of each point on the human body surface, i.e. obtaining a depth video. The depth video is encoded using a pre-constructed pose guide, which is a neural network encoder that can extract compact and semantically rich human pose and spatial structure features from the above depth sequence, thereby obtaining the depth feature video.

[0039] The first model video is processed frame by frame using the SAM tool. It can be understood that, as a powerful general segmentation model, SAM can accurately identify and segment the subject area of each frame of the video, and generate a corresponding binary mask sequence of the subject, that is, obtain a mask video. The mask video accurately outlines the contour of the subject, ensuring that the subsequent generation process can strictly limit the clothing replacement within the subject area, thereby effectively protecting the background content of the original video from being modified.

[0040] The original first model video is encoded using a variational differential autoencoder. It can be understood that the variational differential autoencoder maps high-dimensional video frame data to a low-dimensional, continuous latent space, obtaining initial latent features containing information about the appearance and posture of the subject. Then, the initial latent features are subjected to matrix multiplication operation frame by frame with the mask video obtained as described above. This operation is equivalent to using the mask to spatially filter the initial latent features, retaining the features corresponding to the subject area while setting or significantly weakening the features of the background area to zero. Finally, through this mask-guided feature selection, a pure and subject-focused model latent feature video is obtained, providing an accurate subject structure prior for the second diffusion model.

[0041] As an example, the first clothing image and the CLIP code are input into the first diffusion model, and the clothing spatial attention feature vector is output from the intermediate layer of the first diffusion model, including: encoding the first clothing image using a variational differential autoencoder, and then inputting the encoded first clothing image into the first diffusion model; and inputting the CLIP code into the spatial cross-attention module of the first diffusion model; the clothing spatial attention feature vector is output from the intermediate layer of the first diffusion model.

[0042] In this embodiment, instead of directly inputting the original high-dimensional pixel first clothing image into the first diffusion model, a variational differential autoencoder is used to encode it first. It can be understood that this process converts the clothing image from the pixel space to a low-dimensional, dense latent space, which can significantly reduce the computational complexity of the subsequent first diffusion model processing, while the latent representation still retains the key visual information of the clothing. The encoded first clothing image, i.e., the clothing latent feature, is input into the first diffusion model as the main conditional signal for its denoising process.

[0043] The CLIP code containing high-level semantics obtained in step S200 is input into the spatial cross-attention module in the first diffusion model UNet architecture. In the spatial cross-attention module, the CLIP code is used as a set of keys and values, and interacts with the query representing the spatial position of the image. This allows the first diffusion model to focus on the global semantics (such as clothing style, category) described by the CLIP code during the reconstruction of the image, thereby guiding the generation process to be consistent with the target clothing in terms of semantics.

[0044] Under the combined action of the above conditions, the first diffusion model generates and outputs a clothing spatial attention feature vector at the intermediate layer of its denoising UNet (usually the feature layer at the connection between the downsampling and upsampling paths). It can be understood that this clothing spatial attention feature vector is a compact feature representation that has been processed by the diffusion model and has fused clothing fine-grained spatial structure (derived from the clothing latent feature) and high-level semantic guidance (derived from the CLIP code). It accurately encodes the spatial distribution of clothing texture and pattern, providing crucial fine-grained spatial guidance for the second diffusion model to perform video synthesis.

[0045] As an example, inputting the concatenated random noise, the model latent feature video, the mask video and the depth feature video into the second diffusion model includes: concatenating the random noise, the model latent feature video and the mask video in the spatial dimension, and then adding the depth feature video to form an input of the second diffusion model.

[0046] In this embodiment, the random noise, the model latent feature video and the mask video are concatenated in the spatial dimension. It can be understood that this concatenation operation is equivalent to stacking these information in the channel dimension to form a multi-channel fusion tensor. Among them, the random noise provides the basis for generating diversity; the model latent feature video encodes the appearance, posture and body shape information of the person; and the mask video accurately marks the spatial region of the person. Through concatenation, the second diffusion model can simultaneously perceive these different attribute guidance signals.

[0047] Subsequently, the fused features obtained by splicing are added to the depth feature video element by element. It can be understood that the depth feature video provides key three-dimensional spatial structure information. The addition operation is a feature enhancement mechanism that can directly superimpose the spatial hierarchy and geometric structure information of the human body onto the existing fused features, so that the second diffusion model can deeply integrate the three-dimensional shape of the human body when generating each frame, thereby better understanding the front and back occlusion relationship between the clothes and the human body, as well as between different parts of the human body (such as the torso covered by the arms), and finally generating a try-on video with reasonable three-dimensional spatial sense and natural occlusion effect.

[0048] As an example, the first diffusion model and the second diffusion model have the same number of intermediate layers, but the second diffusion model additionally increases the time attention module and the garment space-time attention module compared with the first diffusion model.

[0049] In this embodiment, the first diffusion model and the second diffusion model maintain parity in the basic spatial feature extraction capability. It can be understood that both of them use UNet with the same number of layers as the backbone network, which ensures that the garment space attention feature vector generated by the first diffusion model can be smoothly connected and effectively utilized with the intermediate layer of the second diffusion model in terms of feature scale, dimension and semantic level, providing a structural basis for precise feature injection. However, the second diffusion model additionally increases the time attention module and the garment space-time attention module compared with the first diffusion model.

[0050] Among them, the role of the time attention module is to establish the association between frames in the time dimension of the video. Through the time attention mechanism, the diffusion model can enable each spatial position of the current frame of the human to focus on and fuse the feature information of the corresponding position and surrounding area of the adjacent frame, which enables the diffusion model to learn and generate coherent human actions and background changes, which is the key to ensuring the temporal smoothness and flicker-free of the output video.

[0051] The garment space-time attention module is a cross-attention module specially set by the present application, and its function is to dynamically and temporally align and fuse the static garment features from the first diffusion model with the dynamic human poses in the video. Specifically, this module takes the spatio-temporal features of the second diffusion model itself as Query, and the garment space attention feature vector (after appropriate transformation) injected in step S300 as Key and Value. At each time step of the video, the module will perform a query, the purpose of which is to accurately match and apply the most suitable spatial region on the target garment texture to each part of the body at each moment in the dynamically changing human pose sequence. This ensures that the garment pattern can physically reasonably and visually naturally deform and move with the human action (such as arm swing, body twist), fundamentally solving the problem of dynamic coherence of the garment.

[0052] It should be noted that in order to ensure the stability of the training, the linear layer weight inside the garment space-time attention module is initialized to zero, ensuring that the injected module does not affect the performance of the original model at the beginning of the training, and continuously optimizing the contribution of the module to the entire model during the training process to ensure the stability of the generated video.

[0053] As an example, splicing the random noise, the model latent feature video, and the mask video in the spatial dimension includes: extracting a multi-dimensional joint feature from the first garment image and the first model video; wherein the multi-dimensional joint feature is generated by jointly fusing the CLIP encoding of the first garment image, the deep feature video of the first model video, and the mask video; based on the multi-dimensional joint feature, a set of spatial splicing points of the random noise is determined; wherein the set of spatial splicing points is used to indicate the weight distribution of the random noise spliced in the spatial dimension; based on the set of spatial splicing points, the random noise is adaptively weighted and spliced with the model latent feature video and the mask video.

[0054] This operation of directly stacking random noise, model latent features, and masks in the channel dimension provides all information sources equally to the model, which needs to learn by itself how to allocate attention in space - that is, which areas should rely on noise for creation (such as generating new clothes), and which areas should rely on conditional features for preservation (such as maintaining facial features and background). This process is burdensome for the model and is prone to confusion in critical areas, resulting in poor generation results or the need for longer training time. The present application solves the above technical problems by constructing a comprehensive guide signal needle that describes where to generate what kind of clothes.

[0055] First, a multi-dimensional joint feature is generated by fusing the features of the three key modalities, specifically including: CLIP encoding of the first garment image: providing high-level semantic information of the target garment (such as style, style, texture type), defining what to generate. Deep feature video of the first model video: providing three-dimensional pose and spatial structure information of the person, defining how the garment needs to fit the dynamic three-dimensional body. Mask video: provides accurate two-dimensional spatial region definition, clearly defining the specific location boundary of the generation operation. It can be understood that the above three are fused through a feature fusion network (e.g. composed of convolutional layers and attention mechanisms), ultimately generating a multi-dimensional joint feature that can cooperatively express garment semantics, body pose, and generation region.

[0056] The multi-dimensional joint feature obtained in the previous step is input into a weight prediction module (for example, a lightweight convolutional network). The task of the weight prediction module is to analyze the joint feature and predict a weight distribution map with the same spatial size as the video frame and a value range of [0, 1], that is, the spatial splicing point set. It can be understood that each pixel value in the weight distribution map represents the degree of dependence on random noise in the generation process at the corresponding spatial position. Specifically, a high weight value (close to 1): corresponds to the region in the mask area that needs to be greatly textured and changed according to the target clothing (for example, the main clothing covering area such as the torso, arms, etc.). These points strongly need random noise to stimulate creativity to generate a completely new clothing appearance.

[0057] A low weight value (close to 0): corresponds to the background area or the region that needs to be strictly preserved (such as the face, hands, and hair). These points should suppress the influence of random noise as much as possible to maintain the stability of the original information.

[0058] Compared with the traditional channel splicing, the application adopts a spatially adaptive weighted fusion mechanism, which can be expressed as: fusion input = random noise * spatial splicing point set + (model latent feature video || mask video) * (1 - spatial splicing point set), where || represents the conventional channel splicing.

[0059] It can be understood that this operation realizes the dynamic modulation of information sources in space, that is, in the area that needs to be highly generated, random noise dominates, while in the area that needs to be preserved, the conditional feature (model appearance and mask) is enhanced. At the same time, it can also provide accurate generation guidance for the second diffusion model, that is, the creative area and the protection area are pre-divided at the input end of the second diffusion model, providing strong spatial prior for the model, thereby effectively reducing the ambiguity and conflict in the generation process, guiding the model to converge faster, and generating a try-on video with clearer boundaries and better preserved character details.

[0060] As an example, the second diffusion model outputs a first try-on video, including: the second diffusion model outputs a result as a latent space representation, inputting the latent space representation into a decoder of the variational differential autoencoder, mapping the latent space representation to a pixel space through the decoder, and generating the first try-on video.

[0061] In this embodiment, after the second diffusion model completes the denoising process, it directly outputs a video sequence representation in a low-dimensional latent space instead of a human visible pixel video. It can be understood that in step S100 of the entire process, an encoder of the variational differential autoencoder is used to encode the original video to the latent space; here, the decoder part in the same variational differential autoencoder architecture is used to complete the reverse conversion.

[0062] The latent space representation output by the second diffusion model is input to the decoder. It can be understood that the decoder is a neural network trained to understand the correspondence between the latent space features and the pixel space images, and is responsible for upsampling and transforming the latent features rich in semantic information but cannot be directly displayed, to reconstruct a high-fidelity video frame sequence in the standard RGB pixel space, i.e., to generate the final first try-on video for users to watch and use. It can be understood that in the inference stage corresponding to step S600, the final output is the second try-on video.

[0063] As an example, when training the first diffusion model and the second diffusion model, optimization of a loss function can be used.

[0064] In this embodiment, the training of the first diffusion model and the second diffusion model can be achieved by optimizing a loss function, which is as follows: , wherein, represents the real noise; represents the predicted noise at time step t; represents the random noise; F represents feature extraction using a variational autoencoder; represents the video latent feature without the garment part; represents the model image multiplied by the mask image; represents the mask image; represents the garment image; represents the video depth pose; represents the time step length.

[0065] In this embodiment, the loss function drives the model to learn an accurate denoising process by minimizing the L2 norm between the predicted noise and the real noise, and ensures that the model can effectively use the human pose and spatial structure information to generate a try-on video with high spatial consistency and temporal coherence by specifically optimizing the pose attention and depth attention parameters.

[0066] As shown in Figure 5 , the embodiment of the present application also discloses a device comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described above.

[0067] The device is a computer device, which comprises a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a video virtual try-on method based on a diffusion model. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0068] The embodiment of the present application also discloses a program product having computer instructions for executing the method according to any one of the above.

[0069] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM), etc.

[0070] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described, however, any combination of the technical features from the above embodiments is considered to be within the scope of the present disclosure. When technical features in different embodiments are embodied in the same drawing, it can be considered that the drawing also discloses the combination of the embodiments involved.

[0071] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

Claims

1. A method for video virtual try-on based on a diffusion model, characterized in that, The method comprises the following steps: obtaining a first garment image and a first model video wearing a garment different from the first garment image, obtaining a depth feature video and a mask video from the first model video respectively, and obtaining a model latent feature video by combining the model video and the mask video; inputting the first garment image into a CLIP encoder to obtain a CLIP code; inputting the first garment image and the CLIP code into a first diffusion model, and outputting a garment space attention feature vector from an intermediate layer of the first diffusion model; inputting the linked random noise, the model latent feature video, the mask video and the depth feature video into a second diffusion model, and injecting the garment space attention feature vector and the CLIP code into an intermediate layer of the second diffusion model, so that the second diffusion model outputs a first try-on video; training the first diffusion model and the second diffusion model using preset loss parameters, wherein the preset loss parameters include a first preset loss, and the first preset loss is a difference loss between the output first try-on video and the model video; obtaining a second try-on video based on a second garment image and a second model video wearing a garment different from the second garment image using the trained first diffusion model and the second diffusion model; obtaining a depth feature video and a mask video from the model video respectively, and obtaining a model latent feature video by combining the model video and the mask video, comprising: extracting the model video frame by frame using DensePose to obtain a depth video, encoding the depth video using a pose guide to obtain the depth feature video, extracting the model video frame by frame using SAM to obtain a mask video, and encoding the model video using a variational differential autoencoder and multiplying it by the mask video to obtain a model latent feature video; inputting the first garment image and the CLIP code into the first diffusion model, and outputting a garment space attention feature vector from an intermediate layer of the first diffusion model, comprising: encoding the first garment image using a variational differential autoencoder, then inputting the encoded first garment image into the first diffusion model, and sending the CLIP code into a spatial cross-attention module of the first diffusion model; and outputting a garment space attention feature vector from an intermediate layer of the first diffusion model; inputting the linked random noise, the model latent feature video, the mask video and the depth feature video into the second diffusion model, comprising: concatenating the random noise, the model latent feature video and the mask video in the spatial dimension, and then adding the depth feature video to obtain an input of the second diffusion model; the first diffusion model and the second diffusion model have the same number of intermediate layers, but the second diffusion model additionally includes a temporal attention module and a garment space temporal attention module compared with the first diffusion model; concatenating the random noise, the model latent feature video and the mask video in the spatial dimension, comprising: extracting multi-dimensional joint features from the first garment image and the first model video; wherein the multi-dimensional joint features are generated by jointly fusing CLIP encoding of the first garment image, deep feature video of the first model video, and the mask video; determining a spatial concatenation point set of the random noise based on the multi-dimensional joint features; wherein the spatial concatenation point set is used to indicate a weight distribution of the random noise in the spatial dimension; and performing spatial adaptive weighted concatenation of the random noise, the model latent feature video, and the mask video based on the spatial concatenation point set.

2. The method of claim 1, wherein: The second diffusion model outputs a first try-on video, including: The result output by the second diffusion model is a latent space representation, and the latent space representation is input into a decoder of a variational differential autoencoder. The decoder maps the latent space representation to a pixel space to generate the first try-on video.

3. The method of claim 2, wherein: When training the first diffusion model and the second diffusion model, the training can be achieved by optimizing a loss function.

4. A computer device, comprising: The computer program is executed by the processor to implement the method of any one of claims 1-3.

5. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-3. The computer program is executed by the processor to implement the method of any one of claims 1-3.

Images