Virtual fitting method, apparatus, device, and storage medium

Abstract

The application provides a virtual try-on method, device, equipment and storage medium, the method comprising: obtaining a video containing a person, a clothing image; determining a person latent feature map according to a video obtained clothing part removed video latent feature and a depth posture in the video; determining a clothing feature map and a clothing semantic latent feature according to the clothing image; performing first self-attention calculation according to the person latent feature map and the clothing feature map to obtain a first feature map; performing first cross-attention calculation according to the clothing semantic latent feature and the first feature map to obtain a second feature map; performing spatio-temporal attention processing according to the second feature map to obtain a third feature map; and generating a virtual try-on image according to the third feature map and the clothing semantic latent feature. The method learns the properties of the clothing through spatio-temporal attention processing, realizes an efficient global spatio-temporal attention mechanism, and thus generates a more realistic and coherent try-on image, improving the visual fidelity and dynamic motion consistency of the clothing.

Description

Technical Field

[0001] This application relates to the field of virtual try-on technology, and more particularly to a virtual try-on method, apparatus, device, and storage medium. Background Technology

[0002] Virtual try-on aims to dress up a subject in a video in a realistic way. Its core challenge is to maintain the visual integrity of the clothing while dynamically adapting to the subject's movements.

[0003] Traditional video virtual try-on solutions employ a two-stage training strategy by integrating an additional temporal attention layer into a pre-trained text-to-image model. First, the model is trained on an image-based virtual try-on task to ensure high-quality single-frame images and generate visually faithful clothing appearances. Then, in the second stage, only the newly added temporal attention module is fine-tuned using video data to enhance the temporal coherence of the generated video.

[0004] Existing methods have shown some potential, but still face the challenge of simultaneously maintaining visual fidelity of clothing and consistency of dynamic motion. Specifically, the two-stage training strategy has the following key limitations: First, the parameters of the newly added temporal attention layer are only fine-tuned based on limited video data. Compared with the original parameters of the pre-trained text-to-image model, it lacks a comprehensive understanding of dynamic motion, resulting in inter-frame jitter and temporal differences in the generated video. Second, fine-tuning the temporal attention module only in the second stage may cause covariate shifts in the generated feature space, weakening or even negating the generative ability of the pre-trained model.

[0005] Therefore, an efficient global spatiotemporal attention mechanism needs to be introduced to improve the fitting performance. Summary of the Invention

[0006] To address one of the aforementioned technical deficiencies, this application provides a virtual try-on method, apparatus, device, and storage medium.

[0007] A first aspect of this application provides a virtual try-on method, the method comprising:

[0008] Get a video containing people and an image of clothing;

[0009] Based on the latent features of the video (with clothing removed) and the depth pose in the video, determine the latent feature map of the person;

[0010] Based on the clothing images, determine the clothing feature map and the clothing semantic latent features;

[0011] The first self-attention calculation is performed based on the latent feature map of the person and the feature map of the clothing to obtain the first feature map;

[0012] The first cross-attention calculation is performed based on the semantic latent features of clothing and the first feature map to obtain the second feature map;

[0013] Spatiotemporal attention processing is performed on the second feature map to obtain the third feature map;

[0014] Virtual try-on images are generated based on the third feature map and the semantic latent features of the clothing.

[0015] Optionally, a first self-attention calculation is performed based on the latent feature map of the person and the feature map of the clothing to obtain a first feature map, including:

[0016] The latent feature map of the person and the feature map of the clothing are concatenated to obtain the first concatenated feature h1 = Concat(f in ,g); where Concat() is the concatenation function, f in g represents the latent feature map of the person, and g represents the clothing feature map;

[0017] Determine the first feature map f1 = Attn1(f in ,h1,h1);

[0018] Where Attn1() is the formula for calculating the first self-attention.

[0019] Optionally, spatiotemporal attention processing is performed on the second feature map to obtain a third feature map, including:

[0020] Based on the second feature map, perform clothing-independent spatial attention processing to obtain clothing-independent spatial attention features f. spa ;

[0021] Based on the second feature map, perform clothing-independent temporal attention processing to obtain clothing-independent temporal attention features f. temp ;

[0022] Based on the second feature map, perform clothing-specific spatial attention processing to obtain clothing-specific spatial attention features f. texture ;

[0023] Determine the third feature map f out =f fused ·m+f2;

[0024] Where f2 is the second feature map, and m is the mask obtained from the video;

[0025] f fused For the second splicing feature, f fused =Linear(f combined Linear is a linear layer;

[0026] f combined For the third splicing feature, fcombined =Concat(f spa ,f temp ,f texture Concat() is a concatenation function.

[0027] Optionally, clothing-independent spatial attention processing is performed based on the second feature map to obtain clothing-independent spatial attention features f. spa ,include:

[0028] Potential library of space unrelated to clothing spa Perform positional encoding to obtain a position-encoded clothing-independent spatial latent library. Where s is the spatial location index, s = 0, 1, ..., p-1, and p is e spa The length of the feature dimension; i1 is the feature dimension index, i1 = 0, 1, ..., d1-1, d1 is the length of e. spa The number of feature dimensions, where k1 is the encoding index. e is the floor function; spa (s,i1) is e spa The value of spatial location s and feature dimension i1;

[0029] Determine clothing-independent spatial attention features f spa =Attn2(f2,E spa E spa ), where Attn2() is the formula for calculating the second cross attention.

[0030] Optionally, clothing-independent temporal attention processing is performed based on the second feature map to obtain clothing-independent temporal attention features f. temp ,include:

[0031] Potential library of time unrelated to clothing temp Perform position encoding to obtain a position-encoded clothing-independent temporal latent library. Where t1 is the time position index, s = 0, 1, ..., q-1, and q is e temp The length of ; i2 is the feature dimension index, i2 = 0, 1, ..., d2-1, d2 is the length of e ... temp The number of feature dimensions, where k2 is the encoding index. e is the floor function; temp (t1,i2) is e temp The value of i1 at time position t1 and feature dimension i2;

[0032] Determine clothing-independent spatial attention features f temp =Attn3(f2,E tempE temp ), where Attn3() is the formula for calculating the third cross attention.

[0033] Optionally, clothing-specific spatial attention processing is performed based on the second feature map to obtain clothing-specific spatial attention features f. texture ,include:

[0034] The second feature map and the clothing feature map are concatenated to obtain the fourth concatenated feature h2 = Concat(f2,g); where g is the clothing feature map;

[0035] Determine clothing-independent spatial attention features f texture =Attn4(f2,h2,h2), where Attn4() is the formula for calculating the fourth cross attention.

[0036] Optionally, the loss function when generating virtual try-on images

[0037] in, Let be the frame difference loss function. ∈ represents real noise; ∈ θ () represents the predicted noise at time step t2, where t2 is the time step identifier, and x is the noise level. t Let m be the noise at time step t2, m be the mask obtained from the video, and x be the noise at time step t2. m F() represents the product of the clothing image and the mask, g represents the feature extraction process, and d3 represents the depth pose in the video. Let it be the expected function;

[0038] Let the time-varying loss function be denoted as . N is the total number of frames in the video, and k3 is the frame identifier. For the generated virtual try-on image of frame k3+1, For the generated virtual try-on image of frame k3, This is the image in frame k3+1 of the video. This is the image in the k3rd frame of the video;

[0039] λ is the weight.

[0040] A second aspect of this application provides a virtual try-on device, the device comprising:

[0041] The acquisition module is used to acquire videos containing people and an image of clothing.

[0042] The first determining module is used to determine the latent feature map of a person based on the latent features of the video (with clothing removed) and the depth pose in the video.

[0043] The second determining module is used to determine the garment feature map and the garment semantic latent features based on the garment image;

[0044] The third determination module is used to perform the first self-attention calculation based on the latent feature map of the person and the feature map of the clothing to obtain the first feature map;

[0045] The fourth determining module is used to perform a first cross-attention calculation based on the semantic latent features of clothing and the first feature map to obtain a second feature map;

[0046] The fifth determining module is used to perform spatiotemporal attention processing on the second feature map to obtain the third feature map;

[0047] The generation module is used to generate virtual try-on images based on the third feature map and the semantic latent features of the clothing.

[0048] A third aspect of this application provides an electronic device, comprising:

[0049] Memory;

[0050] Processor; and

[0051] Computer programs;

[0052] The computer program is stored in the memory and configured to be executed by the processor to implement the method described in the first aspect above.

[0053] In a fourth aspect, this application provides a computer-readable storage medium having a computer program stored thereon; the computer program is executed by a processor to implement the method described in the first aspect above.

[0054] This application provides a virtual try-on method, apparatus, device, and storage medium. The method includes: acquiring a video containing a person and a clothing image; determining a person latent feature map based on the video latent features obtained after removing the clothing portion and the depth pose in the video; determining a clothing feature map and clothing semantic latent features based on the clothing image; performing a first self-attention calculation based on the person latent feature map and the clothing feature map to obtain a first feature map; performing a first cross-attention calculation based on the clothing semantic latent features and the first feature map to obtain a second feature map; performing spatiotemporal attention processing based on the second feature map to obtain a third feature map; and generating a virtual try-on image based on the third feature map and the clothing semantic latent features. This application's method learns about the properties of clothing through spatiotemporal attention processing, achieving an efficient global spatiotemporal attention mechanism, thereby generating more realistic and coherent try-on images and improving the visual fidelity and dynamic motion consistency of the clothing. Attached Figure Description

[0055] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0056] Figure 1 A flowchart illustrating a virtual try-on method provided in an embodiment of this application;

[0057] Figure 2 A schematic diagram illustrating the principle of a virtual try-on method provided in an embodiment of this application;

[0058] Figure 3 This is a schematic diagram of the structure of a clothing perception spatiotemporal attention module provided in an embodiment of this application;

[0059] Figure 4 This is a schematic diagram of the structure of a virtual try-on device provided in an embodiment of this application;

[0060] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0061] To make the technical solutions and advantages of the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0062] In developing this application, the inventors discovered that traditional video virtual try-on solutions employ a two-stage training strategy by integrating an additional temporal attention layer into a pre-trained text-to-image model. First, the model is trained for the image-based virtual try-on task to ensure high-quality single-frame images and generate clothing appearances with high visual fidelity. Then, in the second stage, only the newly added temporal attention module is fine-tuned using video data to enhance the temporal coherence of the generated video.

[0063] Existing methods have shown some potential, but still face the challenge of simultaneously maintaining visual fidelity of clothing and consistency of dynamic motion. Specifically, the two-stage training strategy has the following key limitations: First, the parameters of the newly added temporal attention layer are only fine-tuned based on limited video data. Compared with the original parameters of the pre-trained text-to-image model, it lacks a comprehensive understanding of dynamic motion, resulting in inter-frame jitter and temporal differences in the generated video. Second, fine-tuning the temporal attention module only in the second stage may cause covariate shifts in the generated feature space, weakening or even negating the generative ability of the pre-trained model.

[0064] Therefore, an efficient global spatiotemporal attention mechanism needs to be introduced to improve the fitting performance.

[0065] To address the aforementioned issues, this application provides a virtual try-on method, apparatus, device, and storage medium. The method includes: acquiring a video containing a person and a clothing image; determining a person latent feature map based on the video latent features (excluding clothing) and depth pose in the video; determining a clothing feature map and clothing semantic latent features based on the clothing image; performing a first self-attention calculation based on the person latent feature map and clothing feature map to obtain a first feature map; performing a first cross-attention calculation based on the clothing semantic latent features and the first feature map to obtain a second feature map; performing spatiotemporal attention processing based on the second feature map to obtain a third feature map; and generating a virtual try-on image based on the third feature map and the clothing semantic latent features. This application's method learns about clothing properties through spatiotemporal attention processing, achieving an efficient global spatiotemporal attention mechanism, thereby generating more realistic and coherent try-on images and improving the visual fidelity and dynamic motion consistency of the clothing.

[0066] See Figure 1 This embodiment provides a virtual try-on method, the implementation principle of which is as follows: Figure 2 As shown, the implementation process of this method is as follows:

[0067] 101. Obtain a video containing people and an image of clothing.

[0068] For ease of description, the video containing people obtained in step 101 will be denoted as "video", and the clothing image obtained in step 101 will be denoted as "picture".

[0069] 102. Based on the latent features of the video (with clothing removed) and the depth pose in the video, determine the latent feature map of the person.

[0070] Step 102 can be implemented using variational autoencoders (VAEs). For example, step 102 can be implemented using the following process.

[0071] 102-1, extract the depth video features of the video, and simultaneously obtain the mask m of the clothing region in the video (e.g., Figure 2 The mask (m) is used to multiply the video by the mask m to get the video with the clothing removed (e.g., video'). Figure 2 (masked images in the text).

[0072] Among them, it can be achieved through Figure 2 DensePose extracts depth video features from a video.

[0073] DensePose is a technique that maps human body parts in a 2D image to a 3D model, achieving dense pose estimation by predicting the UV coordinates (i.e., texture coordinates) of pixels.

[0074] Alternatively, the SAM (Segment Anything Model) segmentation model can be used to obtain the mask m of the clothing area in the video.

[0075] The SAM segmentation model is a general-purpose image segmentation model based on the Transformer architecture.

[0076] 102-2, Encode the depth video features extracted from the video in step 102-1 to obtain the depth pose d3 in the video, and encode the video' with the clothing removed obtained in step 102-1 to obtain the latent features of the video with the clothing removed.

[0077] Specifically, the PoseGuider module can be used to encode depth video features to obtain the depth pose d3 in the video.

[0078] PoseGuider encodes deep video features and incorporates control information into a diffusion model to obtain the deep pose d3 of the video. PoseGuider can integrate a spatiotemporal attention mechanism to ensure that the deep pose d3 of the video maintains continuity in the temporal dimension, solving problems such as discontinuous motion and loss of detail.

[0079] In addition, by encoding the video' with the clothing removed using a variational autoencoder (VAE), the latent features of the video with the clothing removed are obtained.

[0080] 102-3, random noise (such as...) Figure 2 The latent features of the video (without clothing) obtained in step 102-2 are obtained by concatenating the mask m of the clothing area in the video obtained in step 102-1 along the channel dimension to obtain a latent feature (such as a 9-channel latent feature). Then, a convolutional layer (such as a convolutional layer with 9 input channels and 4 output channels) is used to integrate the latent features.

[0081] 102-4. Combine the features obtained from step 102-3 with the depth pose d3 from the video obtained in step 102-2 to obtain the latent feature map f of the person. in .

[0082] In step 102, the video' with the clothing removed is encoded into video latent features by a variational autoencoder (VAE), and then concatenated with random noise and the mask m containing the clothing area in the video. After integrating the features through a convolutional layer, the depth pose in the video is directly added to obtain the latent feature map of the person.

[0083] It should be noted that in this embodiment and subsequent embodiments, "feature map" (e.g., a certain feature map) and "feature" (e.g., a certain feature) are the same concept. For example, a latent feature map of a person is a latent feature of a person, and a latent feature of a video is a latent feature map of a video. In other words, this embodiment and subsequent embodiments do not strictly distinguish between feature map and feature.

[0084] The latent feature map f obtained in step 102 in This will be used as the input for step 104.

[0085] 103. Based on the clothing image, determine the clothing feature map and the clothing semantic latent features.

[0086] Step 103 can be based on clothing U-Net (e.g., clothing U-Net). Figure 2 The Garment U-Net is implemented using an image diffusion model, primarily composed of multiple layers of spatial self-attention modules and spatial cross-attention modules. For example, each layer includes one spatial self-attention module and one spatial cross-attention module, and multiple layers constitute the Garment U-Net.

[0087] The implementation process of step 103 is as follows:

[0088] 103-1, the picture is encoded into clothing latent features using a variational autoencoder (VAE) and fed into the clothing UNet. Simultaneously, the picture is encoded to obtain the clothing semantic latent features e. clip .

[0089] For example, through CLIP (Contrastive Language-Image Pre-training) semantic encoders (such as...) Figure 2 The CLIP Encoder encodes the picture to obtain the semantic latent features of the clothing. clip .

[0090] CLIP is a multimodal pre-trained model based on contrastive learning that achieves cross-modal alignment of images and text through unified semantic encoding.

[0091] 103-2, the latent features of clothing are entered into the spatial self-attention module of the corresponding layer of the clothing UNet, and the feature map of the spatial self-attention module of clothing (Garment Reference Feature) is obtained.

[0092] 103-3, the feature map of the spatial self-attention module is fed into the spatial cross-attention module of the corresponding layer of the clothing UNet, and the semantic latent features of clothing are used as e clip The key-value pairs used for attention calculation are used to calculate spatial cross-attention with the feature map of the spatial self-attention module. The output latent features obtained after the calculation are then fed into the next layer until the end. The output at the end is the clothing feature map g.

[0093] In step 103, the latent features of the clothing are incorporated into the clothing U-Net, and the semantic latent features e of the clothing are extracted using the CLIP semantic encoder. clip Clothing semantic latent features e clip This is injected into each spatial cross-attention module. Specifically, fine clothing image details are extracted using the Clothing U-Net and supplemented with semantic features provided by the CLIP semantic encoder, which are then injected into the spatial cross-attention modules.

[0094] The clothing feature map g and clothing semantic latent feature e obtained in step 103 clip This will be used as input for step 104, thereby injecting the detailed features of the garment into the denoising U-Net.

[0095] 104. Perform the first self-attention calculation based on the latent feature map of the person and the feature map of the clothing to obtain the first feature map.

[0096] The implementation process of step 104 is as follows:

[0097] 104-1, the latent feature map of the person f in By concatenating the clothing feature map g with the clothing feature map, we obtain the first concatenated feature h1 = Concat(f in ,g).

[0098] Where Concat() is the concatenation function, f in g represents the latent features of the person, and g represents the clothing features.

[0099] 104-2, Determine the first feature map f1 = Attn1(f in ,h1,h1).

[0100] Where Attn1() is the formula for calculating the first self-attention.

[0101] Step 104 can be implemented based on the denoising U-Net. The denoising U-Net mainly uses a video diffusion model. The denoising U-Net is mainly composed of multiple layers of spatial self-attention modules and spatial cross-attention modules, clothing perception spatiotemporal attention modules, and temporal attention layers. For example, one layer includes a spatial self-attention module and a spatial cross-attention module, and there are multiple layers of spatial self-attention modules and spatial cross-attention modules.

[0102] In addition, the denoising U-Net also includes multiple linear layers.

[0103] Step 104 can be achieved using the spatial self-attention module.

[0104] For example, in 104-1, the spatial self-attention module of one layer of the denoising U-Net will focus on the latent feature map of the person. in Spatially concatenating the clothing feature map g with the clothing feature map g yields the first concatenated feature h1 = Concat(f in (g). Cross-attention is performed in the key-value pair computation space of 104-2 as the attention computation space to determine the first feature map.

[0105] in, To query attention, For key attention, For the sake of attention, By f in Obtained through different linear layer mappings h1 is the feature dimension, T is the transpose, and softmax() is the function to transform h1. A function that transforms into a probability distribution.

[0106] 105. Based on the semantic latent features of clothing and the first feature map, perform the first cross-attention calculation to obtain the second feature map.

[0107] Step 105 can also be implemented using a denoising U-Net, such as by using the spatial cross-attention module of the same layer of the denoising U-Net to combine the first feature map f1 with the clothing semantic latent feature e. clip Perform cross-attention calculations to integrate the semantic information of the clothing.

[0108] The implementation process of step 105 is as follows: the first feature map f1 obtained from the spatial self-attention module of the first layer (i.e., the first feature map f1 obtained in step 104) enters the spatial cross-attention module of the same layer, and is used as the clothing semantic latent feature e. clip Spatial cross-attention computation is performed on key-value pairs used for attention computation. The resulting latent features are then fed into the next layer until the end. The final output is the second feature map f2.

[0109]

[0110] Where Attn1() is the formula for calculating the fifth cross-attention, Q c To query attention, K c For key attention, V c For attention value, Q c Kc V c d is obtained by mapping f1 through different linear layers. c For e clip The feature dimension, T is the transpose, and softmax() is the function to transform the feature dimension. A function that transforms into a probability distribution.

[0111] 106. Spatiotemporal attention processing is performed on the second feature map to obtain the third feature map.

[0112] The implementation process of this step is as follows:

[0113] 106-1, based on the second feature map, perform clothing-independent spatial attention processing to obtain clothing-independent spatial attention features f. spa .

[0114] Step 106-1 can be based on a trainable clothing-independent spatial latents bank. spa Implementation. spa It is a tensor of dimension (1,512,1024), which is randomly initialized by Gaussians and optimized during training, so as to capture clothing properties of general irrelevant space, such as topology and motion patterns, from training videos while preserving important structural details.

[0115] The implementation process of step 106-1 is as follows:

[0116] 1. Potential library of spaces unrelated to clothing spa Perform positional encoding to obtain a position-encoded clothing-independent spatial latent library.

[0117] Where s is the spatial location index, s = 0, 1, ..., p-1, and p is e spa The length of the length. For example, p = 512.

[0118] i1 is the feature dimension index, i1 = 0, 1, ..., d1-1, where d1 is e spa The number of feature dimensions, where k1 is the encoding index. This is the floor function. For example, d1 = 1024.

[0119] e spa (S,i1) is e spa The value of the feature dimension i1 is given by the spatial location s.

[0120] 2. Determine clothing-independent spatial attention characteristics f spa =Attn2(f2,E spa E spa ).

[0121] Where Attn2() is the formula for calculating the second cross-attention.

[0122] 106-2, based on the second feature map, perform clothing-independent temporal attention processing to obtain clothing-independent temporal attention features f. temp .

[0123] Step 106-2 can be based on a trainable clothing-independent temporal latents bank. temp Implementation. temp It is a tensor of dimension (1, 64, 1024), which is randomly initialized by Gaussians and optimized during training, so as to capture general, time-independent clothing properties, such as topology and motion patterns, from the training video while preserving important structural details.

[0124] The implementation process of step 106-2 is as follows:

[0125] 1. Potential storage of time unrelated to clothing temp Perform position encoding to obtain a position-encoded clothing-independent temporal latent library.

[0126] Where t1 is the time position index, s = 0, 1, ..., q-1, and q is e temp The length of q. For example, q = 64.

[0127] i2 is the feature dimension index, i2 = 0, 1, ..., d2-1, where d2 is the value of e. temp The number of feature dimensions, where k2 is the encoding index. This is the floor function. For example, d2 = 1024.

[0128] e temp (t1,i2) is e temp The value is given at time position t1 and feature dimension i2.

[0129] 2. Determine clothing-independent spatial attention characteristics f temp =Attn3(f2,E temp E temp ).

[0130] Attn3() is the formula for calculating the third cross-attention.

[0131] 106-3. Based on the second feature map, perform clothing-specific spatial attention processing to obtain clothing-specific spatial attention features f. texture .

[0132] Step 106-3 involves first concatenating the second feature map and the clothing feature map g to obtain the fourth concatenated feature h2 = Concat(f2,g). Then, the clothing-independent spatial attention feature f is determined. texture =Attn4(f2,h2,h2).

[0133] Where g represents the clothing feature diagram.

[0134] Attn4() is the formula for calculating the fourth cross attention.

[0135] It should be noted that this embodiment only performs 106-1 first, and performs clothing-independent spatial attention processing based on the second feature map to obtain clothing-independent spatial attention features f. spa Then, 106-2 is performed, and clothing-independent temporal attention processing is carried out based on the second feature map to obtain clothing-independent temporal attention features f. temp Finally, 106-3 is performed, and clothing-specific spatial attention processing is carried out based on the second feature map to obtain clothing-specific spatial attention features f. texture Taking the order of steps 106-1, 106-2, and 106-3 as an example, in specific implementation, this embodiment does not limit the execution order of steps 106-1, 106-2, and 106-3. For example, steps 106-1, 106-2, and 106-3 can be executed simultaneously, or step 106-2 can be executed first and then steps 106-1 and 106-3 can be executed simultaneously, or steps 106-3, 106-1, and 106-2 can be executed in sequence, etc.

[0136] 106-4, Determine the third feature map f out =f fused ·m+f2.

[0137] Where f2 is the second feature map, and m is the mask obtained from video (i.e., obtained in step 102-1).

[0138] f fused For the second splicing feature, f fused =Linear(f combined ), Linear is a trainable linear layer.

[0139] f combined For the third splicing feature, f combined =Concat(f spa ,f temp ,f texture Concat() is a concatenation function.

[0140] Step 106 can be achieved through a denoised U-Net, such as a Garment-aware Spatio-Temporal Attention Module (GSTA) using a denoised U-Net. Figure 2 The implementation is in GSTA.

[0141] The Garment Perception Spatiotemporal Attention Module (GSTA) is located between the multi-layered spatial self-attention module, spatial cross-attention module, and temporal attention layer of the denoising U-Net.

[0142] like Figure 3 As shown, the Garment-agnostic Spatial Attention Module (GSTA) consists of three parts: the Garment-agnostic Spatial Attention Module, the Garment-agnostic Temporal Attention Module, and the Garment-specific Spatial Attention Module.

[0143] The Garment Perception Spatiotemporal Attention Module (GSTA) utilizes a trainable garment-independent spatiotemporal latent bank, which is composed of a trainable garment-independent spatial latent bank. spa Temporal Latents Bank (TLC) independent of trainable clothing temp The components are tensors of dimensions (1,512,1024) and (1,64,1024), which are randomly initialized by Gaussians and optimized during training. This allows the system to capture common clothing properties, such as topology and motion patterns, from the training videos while preserving important structural details.

[0144] In step 106, the Garment-agnostic Spatial Attention Module (GSTA) copies the second feature map f2 three times and inputs them into the Garment-agnostic Spatial Attention Module, the Garment-agnostic Temporal Attention Module, and the Garment-specific Spatial Attention Module, respectively, thereby enabling the subsequent propagation of the three parallel paths.

[0145] The first path is the clothing-independent spatial attention module. Within this module, the clothing-independent spatial latent library e is first analyzed. spa Positional encoding is performed, and the feature dimension of this spatial latent library is (0,1,…,d1-1), where d1 is e spa The number of feature dimensions (e.g., d1 = 1024), p is e spa The length (e.g., p = 512) yields the position-encoded clothing-independent spatial latent library E. spa Then, cross-attention calculation is performed.

[0146] in,

[0147] Where s is the spatial location index, s = 0, 1, ..., p-1.

[0148] d1 is e spa The number of feature dimensions, where i1 is the feature dimension index, i1 = 0, 1, ..., d1-1.

[0149] k1 is the encoding index, that is, the index of different dimension pairs in the encoding.

[0150] Attn2() is the formula for calculating the second cross-attention, Q s To query attention, K spa For key attention, V spa For attention value, Q sc K spa V spa d is obtained by mapping f2 through different linear layers of the clothing-independent spatial attention module. spa For E spa The feature dimension, T is the transpose, and softmax() is the function to transform the feature dimension. A function that transforms into a probability distribution.

[0151] By calculating cross-attention, we obtain the clothing-independent spatial attention feature f. spa .

[0152] The second path is the clothing-independent time attention module. In this module, the clothing-independent time latent library e is first... temp Positional encoding is performed, and the feature dimension of this spatial latent library is (0,1,…,d2-1), where d2 is e temp The number of feature dimensions (e.g., d² = 1024), q is e temp The length (e.g., q = 64) yields the location-encoded clothing-independent temporal latent library E. tempThen, cross-attention calculation is performed.

[0153] in,

[0154] Where t1 is the time position index, s = 0, 1, ..., q-1, and q is e temp The length.

[0155] i2 is the feature dimension index, i2 = 0, 1, ..., d2-1, where d2 is the value of e. temp The number of feature dimensions.

[0156] k2 is the encoding index, which is the index of different dimension pairs in the encoding. This is the floor function.

[0157] Attn3() is the formula for calculating the third cross-attention, Q t To query attention, K temp For key attention, V temp For attention value, Q t K temp V temp d is obtained by mapping f2 through different linear layers of the clothing-independent temporal attention module. temp For E temp The feature dimension, T is the transpose, and softmax() is the function to transform the feature dimension. A function that transforms into a probability distribution.

[0158] By calculating cross-attention, we obtain the clothing-independent spatial attention feature f. temp .

[0159] The third path is the clothing-specific spatial attention module. In the clothing-specific spatial attention module, f2 is first spatially concatenated with the clothing feature map g, and then used as a key-value pair for attention calculation to calculate spatial cross-attention, i.e., h2 = Concat(f2, g).

[0160]

[0161] Where Attn4() is the formula for calculating the fourth cross-attention. To query attention, K h For key attention, V h For the sake of attention, K h V h d is obtained by mapping f2 through different linear layers of the clothing-specific spatial attention module. h h2 is the number of feature dimensions, T is the transpose, and softmax() is the function to transform h2. A function that transforms into a probability distribution.

[0162] By calculating cross-attention, we obtain the clothing-independent spatial attention feature f. temp .

[0163] After passing through the above three paths, the data is finally concatenated and merged using a convolutional layer. Then, a mask of the clothing region is applied to focus the module on calculating the features of the clothing region. Finally, a residual block is added to integrate the original features from f2. The process is as follows:

[0164] f combined =Concat(f spa ,f temp ,f texture ).

[0165] f fused =Linear(f combined ).

[0166] f out =f fused ·m+f2.

[0167] 107. Based on the third feature map and the semantic latent features of clothing, generate virtual try-on images.

[0168] Step 107 can be implemented by the spatial cross-attention module of the clothing U-Net. The spatial cross-attention module contains multiple layers of temporal self-attention modules and temporal cross-attention modules (e.g., one layer includes one temporal self-attention module and one temporal cross-attention module). The spatial cross-attention module can integrate information between video frames to improve the coherence of the video.

[0169] The implementation process of step 107 is as follows:

[0170] 107-1, A layer of temporal self-attention module in the spatial cross-attention module for the third feature map f out After performing self-attention calculation and obtaining the output features, they are fed into the temporal cross-attention module of this layer.

[0171] 107-2, Temporal Cross-Attention Module with Clothing Semantic Latent Features e clip As key-value pairs for temporal cross-attention, they are used to perform temporal cross-attention calculations with the input features, and the results are output to the next layer.

[0172] After the features pass through multiple layers of temporal self-attention modules and temporal cross-attention modules, the generated latent features are finally obtained. These latent features are then decoded using the decoder of a variational autoencoder (VAE) to obtain high-quality virtual try-on images. All high-quality virtual try-on images constitute a high-quality virtual try-on video.

[0173] In the implementation of step 107, the loss function when generating the virtual try-on image.

[0174] in, Let be the frame difference loss function.

[0175] ∈ represents real noise, ∈ θ () represents the predicted noise at time step t2 (this noise is random noise), where t2 is the time step identifier, and x is the noise (this noise is random noise). t Let m be the noise at time step t2, m be the mask obtained from the video, and x be the noise at time step t2. m F is the product of the clothing image and the mask, F() is the feature extraction process (e.g., using a variational autoencoder), g is the clothing feature map, and d3 is the depth pose in the video. Let F(x) be the expected function. m (This refers to the latent features of the video after removing the clothing portion.)

[0176] Let the time-varying loss function be denoted as .

[0177] N is the total number of frames in the video, and k3 is the frame identifier. For the generated virtual try-on image of frame k3+1, For the generated virtual try-on image of frame k3, For the (k3+1)th frame of the video, This is the k3rd frame image in the video.

[0178] λ is the weight, such as λ = 0.1.

[0179] In other words, the frame difference loss function uses the difference between the video frame generated in step 107 and the video (i.e., video) obtained in step 101 as the loss.

[0180] To improve the temporal coherence of the final video generated in step 107, the loss function... The time variation was added to its loss function

[0181] Explicitly enforce smooth transition dynamics in the frame sequence. This is achieved by minimizing temporal differences by aligning the temporal variations of the denoising latent representation with the inherent motion patterns captured in the frame differences of the video.

[0182] The virtual try-on method provided in this embodiment introduces a unique global spatiotemporal attention module into the clothing area. By learning specifically for the properties of clothing, it generates more realistic and coherent try-on videos, improving the visual fidelity and dynamic consistency of clothing.

[0183] The virtual try-on method provided in this embodiment can generate a corresponding video of a model wearing a given garment based on a video containing a specific model and a garment image. The video should not only retain the real details of the garment, but also maintain the natural movement of the garment and the continuity of details between frames.

[0184] The virtual try-on method provided in this embodiment introduces a clothing perception spatiotemporal attention module, which can generate richer clothing details, such as textures and patterns, making the generated video more visually realistic and delicate, and improving the fine-grained details of the clothing.

[0185] The virtual try-on method provided in this embodiment introduces a clothing-aware spatiotemporal attention module, supplemented by temporal change alignment loss, which enables it to handle complex clothing dynamics and ensure smooth inter-frame details. This allows the virtual try-on method provided in this embodiment to output high-quality video results even with complex details and movements, resulting in more natural clothing dynamics and more coherent inter-frame details.

[0186] This embodiment provides a virtual try-on method, which involves acquiring a video containing a person and an image of clothing; determining a person's latent feature map based on the video's latent features (excluding clothing) and the depth pose in the video; determining a clothing feature map and clothing semantic latent features based on the clothing image; performing a first self-attention calculation based on the person's latent feature map and the clothing feature map to obtain a first feature map; performing a first cross-attention calculation based on the clothing semantic latent features and the first feature map to obtain a second feature map; performing spatiotemporal attention processing based on the second feature map to obtain a third feature map; and generating a virtual try-on image based on the third feature map and the clothing semantic latent features. The method provided in this embodiment learns about the properties of clothing through spatiotemporal attention processing, achieving an efficient global spatiotemporal attention mechanism, thereby generating more realistic and coherent try-on images and improving the visual fidelity and dynamic motion consistency of the clothing.

[0187] Based on the same inventive concept as the virtual try-on method, this embodiment provides a virtual try-on device, see [link to relevant documentation]. Figure 4 The device includes:

[0188] The acquisition module 401 is used to acquire a video containing people and an image of clothing.

[0189] The first determining module 402 is used to determine the latent feature map of a person based on the latent features of the video (with clothing removed) and the depth pose in the video.

[0190] The second determining module 403 is used to determine the clothing feature map and the clothing semantic latent features based on the clothing image.

[0191] The third determining module 404 is used to perform a first self-attention calculation based on the latent feature map of the person and the feature map of the clothing to obtain a first feature map.

[0192] The fourth determining module 405 is used to perform a first cross-attention calculation based on the semantic latent features of clothing and the first feature map to obtain a second feature map.

[0193] The fifth determining module 406 is used to perform spatiotemporal attention processing based on the second feature map to obtain the third feature map.

[0194] The generation module 407 is used to generate virtual try-on images based on the third feature map and the semantic latent features of clothing.

[0195] The third determining module 404 is used to stitch together the latent feature map of the person and the feature map of the clothing to obtain the first stitched feature h1 = Concat(f in (g). Where Concat() is the concatenation function, f in g represents the latent features of the person, and g represents the clothing features.

[0196] Determine the first feature map f1 = Attn1(f in ,h1,h1).

[0197] Where Attn1() is the formula for calculating the first self-attention.

[0198] The fifth determining module 406 is used to perform clothing-independent spatial attention processing based on the second feature map to obtain clothing-independent spatial attention features f. spa .

[0199] Based on the second feature map, perform clothing-independent temporal attention processing to obtain clothing-independent temporal attention features f. temp .

[0200] Based on the second feature map, perform clothing-specific spatial attention processing to obtain clothing-specific spatial attention features f. texture .

[0201] Determine the third feature map f out =f fused ·m+f2.

[0202] Where f2 is the second feature map and m is the mask obtained from the video.

[0203] f fused For the second splicing feature, f fused =Linear(f combined ), Linear is a linear layer.

[0204] f combined For the third splicing feature, f combined =Concat(f spa ,f temp ,f texture Concat() is a concatenation function.

[0205] Specifically, clothing-independent spatial attention processing is performed based on the second feature map to obtain clothing-independent spatial attention features f. spa ,include:

[0206] Potential library of space unrelated to clothing spa Perform positional encoding to obtain a position-encoded clothing-independent spatial latent library. Where s is the spatial location index, s = 0, 1, ..., p-1, and p is e spa The length of . i1 is the feature dimension index, i1 = 0, 1, ..., d1-1, d1 is the length of e. spa The number of feature dimensions, where k1 is the encoding index. This is the floor function. spa (s,i1) is e spa The value of the feature dimension i1 is given by the spatial location s.

[0207] Determine clothing-independent spatial attention features f spa =Attn2(f2,E spa E spa ), where Attn2() is the formula for calculating the second cross attention.

[0208] Specifically, clothing-independent temporal attention processing is performed based on the second feature map to obtain clothing-independent temporal attention features f. temp ,include:

[0209] Potential library of time unrelated to clothing temp Perform position encoding to obtain a position-encoded clothing-independent temporal latent library. Where t1 is the time position index, s = 0, 1, ..., q-1, and q is e temp The length of d2. i2 is the feature dimension index, i2 = 0, 1, ..., d2-1, where d2 is the length of e. temp The number of feature dimensions, where k2 is the encoding index. This is the floor function. temp (t1,i2) is e temp The value is given at time position t1 and feature dimension i2.

[0210] Determine clothing-independent spatial attention features f temp =Attn3(f2,E temp E temp ), where Attn3() is the formula for calculating the third cross attention.

[0211] Specifically, clothing-specific spatial attention processing is performed based on the second feature map to obtain clothing-specific spatial attention features f. texture ,include:

[0212] The second feature map and the clothing feature map are concatenated to obtain the fourth concatenated feature h2 = Concat(f2, g). Here, g is the clothing feature map.

[0213] Determine clothing-independent spatial attention features f texture =Attn4(f2,h2,h2), where Attn4() is the formula for calculating the fourth cross attention.

[0214] Among them, the loss function when generating virtual try-on images

[0215] in, Let be the frame difference loss function. ∈ represents real noise. θ () represents the predicted noise at time step t2, where t2 is the time step identifier, and x is the noise level. t Let m be the noise at time step t2, m be the mask obtained from the video, and x be the noise at time step t2. m F() represents the product of the clothing image and the mask, g represents the feature extraction process, and d3 represents the depth pose in the video. Let be the expected function.

[0216] Let the time-varying loss function be denoted as . N is the total number of frames in the video, and k3 is the frame identifier. For the generated virtual try-on image of frame k3+1, For the generated virtual try-on image of frame k3, This is the image in frame k3+1 of the video. This is the image in frame k3 of the video.

[0217] λ is the weight.

[0218] The device provided in this embodiment learns about the properties of clothing through spatiotemporal attention processing, thereby achieving an efficient global spatiotemporal attention mechanism, generating more realistic and coherent try-on images, and improving the visual fidelity and dynamic motion consistency of clothing.

[0219] Based on the same inventive concept as the virtual try-on method, this embodiment provides an electronic device, which is as follows: Figure 5 As shown, it includes: memory 501, processor 502, and computer program.

[0220] The computer program is stored in memory 501 and configured to be executed by processor 502 to implement the virtual try-on method described above.

[0221] Specifically,

[0222] Get a video containing people and an image of clothing.

[0223] Based on the latent features of the video (with clothing removed) and the depth pose in the video, the latent feature map of the person is determined.

[0224] Based on clothing images, determine clothing feature maps and clothing semantic latent features.

[0225] The first self-attention calculation is performed based on the latent feature map of the person and the feature map of the clothing to obtain the first feature map.

[0226] The first cross-attention calculation is performed based on the semantic latent features of clothing and the first feature map to obtain the second feature map.

[0227] Spatiotemporal attention processing is performed on the second feature map to obtain the third feature map.

[0228] Virtual try-on images are generated based on the third feature map and the semantic latent features of the clothing.

[0229] The first self-attention calculation is performed based on the latent feature map of the person and the feature map of the clothing to obtain the first feature map, which includes:

[0230] The latent feature map of the person and the feature map of the clothing are concatenated to obtain the first concatenated feature j1 = Concat(f in (g). Where Concat() is the concatenation function, f in g represents the latent features of the person, and g represents the clothing features.

[0231] Determine the first feature map f1 = Attn1(f in ,h1,h1).

[0232] Where Attn1() is the formula for calculating the first self-attention.

[0233] Among them, spatiotemporal attention processing is performed on the second feature map to obtain the third feature map, including:

[0234] Based on the second feature map, perform clothing-independent spatial attention processing to obtain clothing-independent spatial attention features f. spa .

[0235] Based on the second feature map, perform clothing-independent temporal attention processing to obtain clothing-independent temporal attention features f. temp .

[0236] Based on the second feature map, perform clothing-specific spatial attention processing to obtain clothing-specific spatial attention features f. texture .

[0237] Determine the third feature map f out =f fused ·m+f2.

[0238] Where f2 is the second feature map and m is the mask obtained from the video.

[0239] f fused For the second splicing feature, f fused =Linear(f combined ), Linear is a linear layer.

[0240] f combined For the third splicing feature, f combined =Concat(f spa ,f temp ,f texture Concat() is a concatenation function.

[0241] Specifically, clothing-independent spatial attention processing is performed based on the second feature map to obtain clothing-independent spatial attention features f. spa ,include:

[0242] Potential library of space unrelated to clothing spa Perform positional encoding to obtain a position-encoded clothing-independent spatial latent library. Where s is the spatial location index, s = 0, 1, ..., p-1, and p is e spa The length of . i1 is the feature dimension index, i1 = 0, 1, ..., d1-1, d1 is the length of e. spa The number of feature dimensions, where k1 is the encoding index. This is the floor function. spa (s,i1) is e spa The value of the feature dimension i1 is given by the spatial location s.

[0243] Determine clothing-independent spatial attention features fspa =Attn2(f2,E spa E spa ), where Attn2() is the formula for calculating the second cross attention.

[0244] Specifically, clothing-independent temporal attention processing is performed based on the second feature map to obtain clothing-independent temporal attention features f. temp ,include:

[0245] Potential library of time unrelated to clothing temp Perform position encoding to obtain a position-encoded clothing-independent temporal latent library. Where t1 is the time position index, s = 0, 1, ..., q-1, and q is e temp The length of d2. i2 is the feature dimension index, i2 = 0, 1, ..., d2-1, where d2 is the length of e. temp The number of feature dimensions, where k2 is the encoding index. This is the floor function. temp (t1,i2) is e temp The value is given at time position t1 and feature dimension i2.

[0246] Determine clothing-independent spatial attention features f temp =Attn3(f2,E temp E temp ), where Attn3() is the formula for calculating the third cross attention.

[0247] Specifically, clothing-specific spatial attention processing is performed based on the second feature map to obtain clothing-specific spatial attention features f. texture ,include:

[0248] The second feature map and the clothing feature map are concatenated to obtain the fourth concatenated feature h2 = Concat(f2, g). Here, g is the clothing feature map.

[0249] Determine clothing-independent spatial attention features f texture =Attn4(f2,h2,h2), where Attn4() is the formula for calculating the fourth cross attention.

[0250] Among them, the loss function when generating virtual try-on images

[0251] in, Let be the frame difference loss function. ∈ represents real noise. θ() represents the predicted noise at time step t2, t1 is the time step identifier, and x is the noise value. t Let m be the noise at time step t2, m be the mask obtained from the video, and x be the noise at time step t2. m F() represents the product of the clothing image and the mask, g represents the feature extraction process, and d3 represents the depth pose in the video. Let be the expected function.

[0252] Let the time-varying loss function be denoted as . N is the total number of frames in the video, and k3 is the frame identifier. For the generated virtual try-on image of frame k3+1, For the generated virtual try-on image of frame k3, This is the image in frame k3+1 of the video. This is the image in frame k3 of the video.

[0253] λ is the weight.

[0254] The electronic device provided in this embodiment has a computer program executed by a processor to learn about the properties of clothing through spatiotemporal attention processing, thereby achieving an efficient global spatiotemporal attention mechanism, generating more realistic and coherent try-on images, and improving the visual fidelity and dynamic motion consistency of clothing.

[0255] Based on the same inventive concept as the virtual try-on method, this embodiment provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the aforementioned virtual try-on method.

[0256] Specifically,

[0257] Get a video containing people and an image of clothing.

[0258] Based on the latent features of the video (with clothing removed) and the depth pose in the video, the latent feature map of the person is determined.

[0259] Based on clothing images, determine clothing feature maps and clothing semantic latent features.

[0260] The first self-attention calculation is performed based on the latent feature map of the person and the feature map of the clothing to obtain the first feature map.

[0261] The first cross-attention calculation is performed based on the semantic latent features of clothing and the first feature map to obtain the second feature map.

[0262] Spatiotemporal attention processing is performed on the second feature map to obtain the third feature map.

[0263] Virtual try-on images are generated based on the third feature map and the semantic latent features of the clothing.

[0264] The first self-attention calculation is performed based on the latent feature map of the person and the feature map of the clothing to obtain the first feature map, which includes:

[0265] The latent feature map of the person and the feature map of the clothing are concatenated to obtain the first concatenated feature h1 = Concat(f in (g). Where Concat() is the concatenation function, f in g represents the latent features of the person, and g represents the clothing features.

[0266] Determine the first feature map f1 = Attn1(f in ,h1,h1).

[0267] Where Attn1() is the formula for calculating the first self-attention.

[0268] Among them, spatiotemporal attention processing is performed on the second feature map to obtain the third feature map, including:

[0269] Based on the second feature map, perform clothing-independent spatial attention processing to obtain clothing-independent spatial attention features f. spa .

[0270] Based on the second feature map, perform clothing-independent temporal attention processing to obtain clothing-independent temporal attention features f. temp .

[0271] Based on the second feature map, perform clothing-specific spatial attention processing to obtain clothing-specific spatial attention features f. texture .

[0272] Determine the third feature map f out =f fused ·m+f2.

[0273] Where f2 is the second feature map and m is the mask obtained from the video.

[0274] f fused For the second splicing feature, f fused =Linear(f combined ), Linear is a linear layer.

[0275] f combined For the third splicing feature, f combined =Concat(f spa ,f temp ,f texture Concat() is a concatenation function.

[0276] Specifically, clothing-independent spatial attention processing is performed based on the second feature map to obtain clothing-independent spatial attention features f.spa ,include:

[0277] Potential library of space unrelated to clothing spa Perform positional encoding to obtain a position-encoded clothing-independent spatial latent library. Where s is the spatial location index, s = 0, 1, ..., p-1, and p is e spa The length of . i1 is the feature dimension index, i1 = 0, 1, ..., d1-1, d1 is the length of e. spa The number of feature dimensions, where k1 is the encoding index. This is the floor function. spa (s,i1) is e spa The value of the feature dimension i1 is given by the spatial location s.

[0278] Determine clothing-independent spatial attention features f spa =Attn2(f2,E spa E spa ), where Attn2() is the formula for calculating the second cross attention.

[0279] Specifically, clothing-independent temporal attention processing is performed based on the second feature map to obtain clothing-independent temporal attention features f. temp ,include:

[0280] Potential library of time unrelated to clothing temp Perform position encoding to obtain a position-encoded clothing-independent temporal latent library. Where t1 is the time position index, s = 0, 1, ..., q-1, and q is e temp The length of d2. i2 is the feature dimension index, i2 = 0, 1, ..., d2-1, where d2 is the length of e. temp The number of feature dimensions, where k2 is the encoding index. This is the floor function. temp (t1,i2) is e temp The value is given at time position t1 and feature dimension i2.

[0281] Determine clothing-independent spatial attention features f temp =Attn3(f2,E temp E temp ), where Attn3() is the formula for calculating the third cross attention.

[0282] Specifically, clothing-specific spatial attention processing is performed based on the second feature map to obtain clothing-specific spatial attention features f. texture ,include:

[0283] The second feature map and the clothing feature map are concatenated to obtain the fourth concatenated feature h2 = Concat(f2, g). Here, g is the clothing feature map.

[0284] Determine clothing-independent spatial attention features f texture =Attn4(f2,h2,h2), where Attn4() is the formula for calculating the fourth cross attention.

[0285] Among them, the loss function when generating virtual try-on images

[0286] in, Let be the frame difference loss function. ∈ represents real noise. θ () represents the predicted noise at time step t2, where t2 is the time step identifier, and x is the noise level. t Let m be the noise at time step t2, m be the mask obtained from the video, and x be the noise at time step t2. m F() represents the product of the clothing image and the mask, g represents the feature extraction process, and d3 represents the depth pose in the video. Let be the expected function.

[0287] Let the time-varying loss function be denoted as . N is the total number of frames in the video, and k3 is the frame identifier. For the generated virtual try-on image of frame k3+1, For the generated virtual try-on image of frame k3, This is the image in frame k3+1 of the video. This is the image in frame k3 of the video.

[0288] λ is the weight.

[0289] The computer-readable storage medium provided in this embodiment contains a computer program that is executed by a processor to learn about the properties of clothing through spatiotemporal attention processing, thereby achieving an efficient global spatiotemporal attention mechanism, generating more realistic and coherent try-on images, and improving the visual fidelity and dynamic motion consistency of clothing.

[0290] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0291] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0292] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0293] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0294] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0295] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0296] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A virtual try-on method, characterized in that, The method includes: Get a video containing people and an image of clothing; Based on the latent features of the video (with clothing removed) and the depth pose in the video, a latent feature map of the person is determined. Based on the clothing image, determine the clothing feature map and the clothing semantic latent features; The first self-attention calculation is performed based on the latent feature map of the person and the feature map of the clothing to obtain the first feature map; Based on the semantic latent features of the clothing and the first feature map, a first cross-attention calculation is performed to obtain a second feature map; Spatiotemporal attention processing is performed on the second feature map to obtain the third feature map; Based on the third feature map and the semantic latent features of the clothing, a virtual try-on image is generated; The step of performing spatiotemporal attention processing based on the second feature map to obtain the third feature map includes: Based on the second feature map, perform clothing-independent spatial attention processing to obtain clothing-independent spatial attention features. ; Based on the second feature map, perform clothing-independent temporal attention processing to obtain clothing-independent temporal attention features. ; Based on the second feature map, perform clothing-specific spatial attention processing to obtain clothing-specific spatial attention features. ; Determine the third feature map ; in, This is the second feature map. The mask obtained from the video; This is the second splicing feature. , Linear layer; This is the third splicing feature. , This is a concatenation function; Specifically, the step of performing clothing-specific spatial attention processing based on the second feature map to obtain clothing-specific spatial attention features. ,include: The second feature map and the clothing feature map are spliced ​​together to obtain the fourth spliced ​​feature map. ;in, For clothing feature diagrams; Determine clothing-independent spatial attention features ,in, This is the formula for calculating the fourth cross-attention.

2. The method according to claim 1, characterized in that, The step of performing a first self-attention calculation based on the latent feature map of the person and the feature map of the clothing to obtain a first feature map includes: The latent feature map of the person and the feature map of the clothing are spliced ​​together to obtain the first spliced ​​feature. ;in, For concatenation functions, For latent features of a person, For clothing feature diagrams; Determine the first feature map ; in, This is the formula for calculating the first level of self-attention.

3. The method according to claim 1, characterized in that, The clothing-independent spatial attention processing is performed based on the second feature map to obtain clothing-independent spatial attention features. ,include: Potential library of spaces unrelated to clothing Perform positional encoding to obtain a position-encoded clothing-independent spatial latent library. ;in, For spatial location index, , for Length; Indexed by feature dimensions, , for The number of feature dimensions, For encoding index, , It is the floor function; for The spatial position is The feature dimension is The value; Determine clothing-independent spatial attention features ,in, This is the formula for calculating the second cross-attention.

4. The method according to claim 1, characterized in that, The clothing-independent temporal attention processing is performed based on the second feature map to obtain clothing-independent temporal attention features. ,include: Potential library of time unrelated to clothing Perform positional encoding to obtain a position-encoded clothing-independent temporal latent library. ;in, For time location index, , for Length; Indexed by feature dimensions, , for The number of feature dimensions, For encoding index, , It is the floor function; for The time position is The feature dimension is The value; Determine clothing-independent spatial attention features ,in, This is the formula for calculating the third cross-attention.

5. The method according to claim 1, characterized in that, The loss function when generating virtual try-on images ; in, Let be the frame difference loss function. ; This is real noise; For the first Prediction noise at time steps For time step identifier, For noise, For the first Noise at time steps This is the mask obtained from the video. The product of the clothing image and the mask. For the feature extraction process, For clothing feature diagrams, The depth pose in the video. Let it be the expected function; Let the time-varying loss function be denoted as . ; The total number of frames in the video. For frame identifier, For the generated first Virtual try-on images in frames, For the generated first Virtual try-on images in frames, The video is the first Frame image, The video is the first Frame image; As weight.

6. A virtual try-on device, characterized in that, The device includes: The acquisition module is used to acquire videos containing people and an image of clothing. The first determining module is used to determine the latent feature map of a person based on the latent features of the video obtained after removing the clothing and the depth pose in the video; The second determining module is used to determine the clothing feature map and the clothing semantic latent features based on the clothing image; The third determining module is used to perform a first self-attention calculation based on the latent feature map of the person and the feature map of the clothing to obtain a first feature map; The fourth determining module is used to perform a first cross-attention calculation based on the semantic latent features of the clothing and the first feature map to obtain a second feature map; The fifth determining module is used to perform spatiotemporal attention processing on the second feature map to obtain the third feature map; The generation module is used to generate virtual try-on images based on the third feature map and the semantic latent features of the clothing; The fifth determining module is used to perform clothing-independent spatial attention processing based on the second feature map to obtain clothing-independent spatial attention features. ; Based on the second feature map, perform clothing-independent temporal attention processing to obtain clothing-independent temporal attention features. ; Based on the second feature map, perform clothing-specific spatial attention processing to obtain clothing-specific spatial attention features. ; Determine the third feature map ; in, This is the second feature map. The mask obtained from the video; This is the second splicing feature. , Linear layer; This is the third splicing feature. , This is a concatenation function; Specifically, the step of performing clothing-specific spatial attention processing based on the second feature map to obtain clothing-specific spatial attention features. ,include: The second feature map and the clothing feature map are spliced ​​together to obtain the fourth spliced ​​feature map. ;in, For clothing feature diagrams; Determine clothing-independent spatial attention features ,in, This is the formula for calculating the fourth cross-attention.

7. An electronic device, characterized in that, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, It stores a computer program thereon; the computer program is executed by a processor to implement the method as described in any one of claims 1-5.

Images