Virtual fitting video generation method and device, equipment, medium and product

Abstract

The invention provides a virtual fitting video generation method and device, equipment, a medium and a product, and relates to the technical field of artificial intelligence. The method comprises the steps of obtaining a try-on clothes image, and performing compression coding on the try-on clothes image to obtain a first clothes feature; obtaining a user face image, and performing feature fusion according to the try-on clothes image and the user face image to obtain clothes face fusion features; acquiring a user video, and extracting a video feature image from the user video; inputting the first clothes feature, the clothes face fusion feature and the video feature image into a fusion model to obtain a fusion video; and decoding the fused video to obtain a virtual fitting video. According to the method provided by the invention, the detail precision of the video frame is improved by considering the clothes and face fusion features, and the fluency of the virtual fitting video is improved by introducing technologies such as optical flow estimation and time consistency constraint into the time alignment layer in the fusion model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, medium and product for generating virtual fitting videos. Background Technology

[0002] Virtual fitting technology allows users to intuitively understand how clothing will look when shopping online, and accurately select the right size and style, thereby improving the user's shopping experience.

[0003] Existing methods for generating virtual fitting videos mainly fall into two categories: one is based on Generative Adversarial Networks (GANs), which uses deformation modules to adjust clothing images to adapt to the user's target pose, and then uses GANs to synthesize missing or occluded body parts to generate a coherent virtual fitting video; the other is based on Latent Diffusion Models (LDMs), which generates virtual fitting videos by progressively denoising the user image in the latent space and fusing features of clothing and the human body.

[0004] However, existing virtual try-on video generation methods are not precise enough in generating video frames, and the generated videos are not smooth enough, with jitter issues. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and product for generating virtual fitting videos, which improves the detail accuracy of video frames and the smoothness of virtual fitting videos.

[0006] Firstly, this application provides a method for generating virtual fitting videos, the method comprising:

[0007] Acquire images of the clothing to be tried on, and compress and encode the images to obtain the first clothing features;

[0008] Acquire the user's facial image, and perform feature fusion based on the clothing trial image and the user's facial image to obtain clothing-face fusion features;

[0009] Acquire user videos and extract video feature images from them;

[0010] The first clothing feature, clothing-face fusion feature, and video feature image are input into the fusion model to obtain the fused video. The fusion model includes a reference model and a denoising model. The reference model includes M cascaded reference units, where M is an integer greater than 1. The denoising model includes M denoising units connected end-to-end. The reference unit includes a first spatial attention layer and a first cross attention layer cascaded in sequence. The denoising unit includes a second spatial attention layer, a second cross attention layer, and a time alignment layer cascaded in sequence. The first clothing feature is used as the input to the first first spatial attention layer, the video feature image and the output of the first first spatial attention layer are used as the input to the first second spatial attention layer, and the clothing-face fusion feature is used as the input to the first cross attention layer and the second cross attention layer.

[0011] The merged video is decoded to obtain a virtual try-on video.

[0012] In one possible design, the time alignment layer includes: a frame optical flow calculation module, a motion vector smoothing module, a reverse optical flow correction module, and a timing transformation module;

[0013] The output of the second cross-attention layer serves as the input to the frame optical flow calculation module in the corresponding time alignment layer;

[0014] The frame optical flow calculation module is used to calculate the motion vectors between video frames;

[0015] The motion vector smoothing module is used to perform smoothing operations on motion vectors to obtain smoothed motion vectors.

[0016] The reverse optical flow correction module is used to remap the pixels in the current video frame to the previous video frame based on the smooth motion vector;

[0017] The timing transformation module is used to perform self-attention calculation on pixels at the same spatial location in different video frames.

[0018] In one possible design, feature fusion is performed based on the image of the clothing being tried on and the user's facial image to obtain clothing-face fusion features, including:

[0019] Semantic features are extracted from images of clothing being tried on to obtain second clothing features, and semantic features are extracted from images of the user's face to obtain facial features;

[0020] A linear transformation is performed on the second clothing feature to obtain a linear clothing feature, and a linear transformation is performed on the face feature to obtain a linear face feature;

[0021] The clothing-face fusion features are determined based on linear clothing features and linear face features.

[0022] In one possible design, clothing-face fusion features are determined based on linear clothing features and linear face features, including:

[0023] The linear clothing features and linear face features are concatenated into the first linear feature;

[0024] The second linear feature is obtained by performing a linear transformation on the first linear feature;

[0025] An embedding vector of the second linear feature is generated as the clothing-face fusion feature.

[0026] In one possible design, extracting video feature images from the user's video includes:

[0027] Extract user pose image, clothing mask image, clothing occlusion image, noise image, and first user skeleton image from user video;

[0028] Perform a convolution operation on the first user's skeletal image to obtain the second user's skeletal image;

[0029] The noisy image and the second user's skeleton image are superimposed pixel by pixel to obtain the noisy skeleton image;

[0030] The user pose image, clothing mask image, clothing occlusion image, noise image, and noise skeleton image are stitched together to form a video feature image.

[0031] In one possible design, the user pose image, clothing mask image, clothing occlusion image, noise image, and first user skeleton image are extracted from the user video, including:

[0032] Identify the locations of important body parts of the user from each video frame of the user video, and predict the user's pose based on the locations to obtain a user pose image;

[0033] The clothing area is segmented from each video frame of the user's video, and an image that distinguishes the clothing area from the rest of the area is generated as a clothing mask image. In addition, the clothing area in the video frame is occluded to obtain a clothing occlusion image.

[0034] Extract user skeletal key points from each video frame of the user's video, and generate a first user skeletal image based on the skeletal key points;

[0035] Each video frame of the user's video is input into a filter or noise model to obtain a noisy image.

[0036] Secondly, this application provides a virtual fitting video generation device, the device comprising:

[0037] The clothing acquisition module is used to acquire images of clothing to be tried on and to compress and encode the images of clothing to obtain the first clothing features;

[0038] The face acquisition module is used to acquire the user's face image and perform feature fusion based on the clothing trial image and the user's face image to obtain clothing face fusion features;

[0039] The video acquisition module is used to acquire user videos and extract video feature images from user videos;

[0040] The fusion module is used to input the first clothing feature, clothing-face fusion feature, and video feature image into the fusion model to obtain the fused video. The fusion model includes a reference model and a denoising model. The reference model includes M cascaded reference units, where M is an integer greater than 1. The denoising model includes M denoising units connected end-to-end. The reference unit includes a first spatial attention layer and a first cross attention layer cascaded in sequence. The denoising unit includes a second spatial attention layer, a second cross attention layer, and a time alignment layer cascaded in sequence. The first clothing feature is used as the input to the first first spatial attention layer, the video feature image and the output of the first first spatial attention layer are used as the input to the first second spatial attention layer, and the clothing-face fusion feature is used as the input to the first cross attention layer and the second cross attention layer.

[0041] The decoding module is used to decode the fused video to obtain the virtual try-on video.

[0042] In one possible design, the time alignment layer in the fusion module includes: an input module, a frame optical flow calculation module, a motion vector smoothing module, a reverse optical flow correction module, and a timing transformation module;

[0043] The input module is used to take the output of the second cross-attention layer as the input of the frame optical flow calculation module in the corresponding time alignment layer;

[0044] The frame optical flow calculation module is used to calculate the motion vectors between video frames;

[0045] The motion vector smoothing module is used to perform smoothing operations on motion vectors to obtain smoothed motion vectors.

[0046] The reverse optical flow correction module is used to remap the pixels in the current video frame to the previous video frame based on the smooth motion vector;

[0047] The timing transformation module is used to perform self-attention calculations on pixels at the same spatial location in different video frames.

[0048] In one possible design, the face acquisition module includes: a first extraction module, a first linear transformation module, and a feature fusion module;

[0049] The first extraction module is used to extract semantic features from the image of the clothes being tried on to obtain the second clothing features, and to extract semantic features from the image of the user's face to obtain the face features;

[0050] The first linear transformation module is used to perform a linear transformation on the second clothing features to obtain linear clothing features, and to perform a linear transformation on the face features to obtain linear face features;

[0051] The feature fusion module is used to determine the clothing-face fusion features based on linear clothing features and linear face features.

[0052] In one possible design, the feature fusion module includes: a first concatenation module, a second linear transformation module, and an embedding vector module;

[0053] The first stitching module is used to stitch the linear clothing features and the linear face features into a first linear feature;

[0054] The second linear transformation module is used to perform a linear transformation on the first linear feature to obtain the second linear feature;

[0055] The embedding vector module is used to generate embedding vectors for the second linear feature, which serves as the clothing-face fusion feature.

[0056] In one possible design, the video acquisition module includes a second extraction module, a convolution module, a pixel overlay module, and a second stitching module;

[0057] The second extraction module is used to extract user pose image, clothing mask image, clothing occlusion image, noise image, and first user skeleton image from the user video;

[0058] The convolution module is used to perform convolution operations on the first user skeleton image to obtain the second user skeleton image;

[0059] The pixel overlay module is used to overlay the noisy image and the second user skeleton image pixel by pixel to obtain the noisy skeleton image;

[0060] The second stitching module is used to stitch together the user pose image, clothing mask image, clothing occlusion image, noise image, and noise skeleton image into a video feature image.

[0061] In one possible design, the second extraction module includes a pose image module, an occlusion image module, a skeleton image module, and a noise image module;

[0062] The pose image module is used to identify the positions of important parts of the user's body in each video frame of the user's video, and predict the user's pose based on the position to obtain the user pose image;

[0063] The occlusion image module is used to segment the clothing area from each video frame of the user video and generate an image that distinguishes the clothing area from the rest of the area as a clothing mask image, and to occlude the clothing area in the video frame to obtain a clothing occlusion image.

[0064] The skeletal image module is used to extract user skeletal key points from each video frame of the user video and generate a first user skeletal image based on the skeletal key points.

[0065] The noise image module is used to input each video frame of the user's video into a filter or noise model to obtain a noise image.

[0066] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0067] The memory stores the instructions that the computer executes;

[0068] The processor executes computer execution instructions stored in memory to implement a virtual fitting video generation method according to the first aspect of the invention.

[0069] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement a virtual fitting video generation method according to the first aspect of the invention.

[0070] Fifthly, this application provides a computer program product, including a computer program, which, when executed by a processor, is used to implement a virtual try-on video generation method according to the first aspect of the invention.

[0071] This application provides a method, apparatus, device, medium, and product for generating virtual try-on videos. The method includes: acquiring an image of clothing to be tried on, and compressing and encoding the image to obtain a first clothing feature; acquiring a user's face image, and performing feature fusion based on the clothing image and the user's face image to obtain clothing-face fusion features; acquiring a user video, and extracting video feature images from the user video; inputting the first clothing feature, clothing-face fusion features, and video feature images into a fusion model to obtain a fused video. The fusion model includes a reference model and a denoising model. The reference model includes M cascaded reference units, where M is an integer greater than 1. The denoising model includes M denoising units connected end-to-end. The reference unit includes a first spatial attention layer and a first cross-attention layer cascaded sequentially. The denoising unit includes a second spatial attention layer, a second cross-attention layer, and a time alignment layer cascaded sequentially. The first clothing feature is used as the input to the first first spatial attention layer, the video feature image and the output of the first first spatial attention layer are used as the input to the first second spatial attention layer, and the clothing-face fusion features are used as the input to the first cross-attention layer and the second cross-attention layer. The fused video is decoded to obtain a virtual try-on video. The following technical effects were achieved: The VAE encoder encodes the clothing images to generate low-dimensional latent spatial features, resulting in low-dimensional first clothing features. This reduces data complexity and storage requirements, thereby lowering computational complexity. The first spatial attention layer processes these first clothing features, extracting fine-grained spatial features from the input clothing images, improving the detail accuracy of the generated video frames. By acquiring user face images and fusing features from the clothing and face images, clothing-face fusion features are obtained, preserving facial details and providing a more realistic and high-quality virtual try-on effect. Optical flow estimation and temporal consistency constraints are used to smooth motion vectors, reducing jitter between video frames and improving the smoothness and stability of consecutive video frames, thus enhancing the fluency of the virtual try-on video. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0074] Figure 1This is a schematic diagram illustrating an application scenario of a virtual try-on video generation method provided in an embodiment of this application;

[0075] Figure 2 A flowchart illustrating a virtual try-on video generation method provided in this application embodiment. Figure 1 ;

[0076] Figure 3 This is a schematic diagram of the overall architecture of the virtual fitting model provided in the embodiments of this application;

[0077] Figure 4 A flowchart illustrating a virtual try-on video generation method provided in this application embodiment. Figure 2 ;

[0078] Figure 5 This is a schematic diagram of the structure of a virtual fitting video generation device provided in an embodiment of this application;

[0079] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0080] Figure label:

[0081] 110 - Server; 120 - Terminal;

[0082] 310 - Image of clothing being tried on; 311 - First clothing feature; 320 - User face image; 321 - Clothing-face fusion feature; 330 - User video; 331 - Video feature image; 340 - Fusion model; 341 - Reference model; 342 - Reference unit; 3421 - First spatial attention layer; 3422 - First cross-attention layer; 343 - Denoising model; 344 - Denoising unit; 3441 - Second spatial attention layer; 3442 - Second cross-attention layer; 3443 - Temporal alignment layer; 350 - Fuded video; 360 - Virtual try-on video;

[0083] 510 - Clothing Acquisition Module; 520 - Face Acquisition Module; 530 - Video Acquisition Module; 540 - Fusion Module; 550 - Decoding Module;

[0084] 610 - Processor; 620 - Memory; 630 - Communication components; 640 - Bus. Detailed Implementation

[0085] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0086] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply difference. It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate that something is being used as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or design schemes. Specifically, the use of "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner. In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more.

[0087] It should be noted that the phrase "at...time" in the embodiments of this application can refer to the instant at which a certain situation occurs, or to a period of time after the occurrence of a certain situation; the embodiments of this application do not specifically limit this. Furthermore, the virtual fitting video generation method provided in the embodiments of this application is merely an example; a virtual fitting video generation method may also include more or less content.

[0088] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0089] Virtual try-on: refers to a technology that allows users to change clothes and check the effect without taking off their clothes.

[0090] Cascading refers to connecting multiple models or processing steps sequentially to form a larger system. This cascading structure allows the output of a previous model or step to serve as the input for the next model or step.

[0091] Virtual try-on technology allows users to intuitively understand how clothing looks when shopping online, and accurately select the right size and style, thus improving the shopping experience. Virtual try-on videos can dynamically showcase the texture, sheen, wrinkles, and movement of clothing, giving users a more comprehensive understanding of its characteristics. For example, the flowing effect of a long skirt while walking, or the natural sway of sleeves when waving, provides dynamic information that helps users better determine if the clothing meets their expectations. Furthermore, virtual try-on videos can rotate, move, and change poses, allowing users to observe the effect of clothing from multiple perspectives, providing a stronger sense of immersion and interactivity, thereby enhancing the shopping experience and reducing return rates for merchants.

[0092] Existing virtual try-on video generation methods mainly fall into two categories: GAN-based methods and LDM-based methods. GAN-based methods first use a warp module to adjust the clothing image to fit the user's target pose, ensuring the clothing looks natural on the user. Furthermore, since clothing may obscure certain body parts, GANs can be used to synthesize these missing or obscured portions, thus generating a coherent virtual try-on video. LDM-based methods, on the other hand, progressively denoise the user image in the latent space, gradually generating a clearer image. During the denoising process, features from both the clothing and the human body are simultaneously integrated to produce a natural and high-quality virtual try-on video.

[0093] However, while these methods have achieved some success in generating virtual try-on videos, they still have some shortcomings. For example, they are not refined enough in terms of the fine details of the generated video frames, which makes the generated try-on videos look unrealistic in some local areas. In addition, the generated videos have a jitter problem, that is, the transition between video frames is not smooth enough, which makes the final generated try-on videos not smooth.

[0094] Based on this, this application proposes an improved virtual try-on video generation method based on a diffusion model to preserve details of the user's face, hands, and clothing in the video frames, and enhance the coherence between video frames, thereby generating realistic and smooth virtual try-on videos. Specifically, this application provides a virtual try-on video generation method, apparatus, device, medium, and product, which can be used in the field of artificial intelligence technology, aiming to solve the above-mentioned technical problems of the prior art and improve the detail accuracy of video frames and the smoothness of virtual try-on videos.

[0095] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0096] To facilitate understanding of the technical solution of this application, the application scenarios of the virtual try-on video generation method provided in the embodiments of this application will be introduced first. Figure 1 This is a schematic diagram illustrating an application scenario of a virtual try-on video generation method provided in an embodiment of this application. It should be noted that... Figure 1 The examples shown are merely application scenarios that can be applied to the embodiments of this application, in order to help those skilled in the art understand the technical content of this application, but do not mean that the embodiments of this application cannot be used in other devices, systems, environments or scenarios.

[0097] like Figure 1 As shown, this application scenario includes a terminal 120 and a server 110. The terminal 120 and server 110 can be connected via a wired or wireless communication link. It should be noted that... Figure 1 The number of terminals 120 and servers 110 shown is illustrative. In other embodiments, the application scenario may include other numbers of terminals 120 and servers 110, which are not specifically limited here.

[0098] In this embodiment, terminal 120 can be understood as various electronic devices, such as wired or wireless terminals, including but not limited to smartphones, tablets, smartwatches, laptops, and desktop computers. No specific limitations are imposed here. Terminal 120 is equipped with a client. Server 110 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server, etc. No specific limitations are imposed here either.

[0099] Users can send images of clothing to try on, their facial images, and user videos to server 110 via a client installed on terminal 120, and request server 110 to generate a virtual try-on video. Server 110 analyzes and processes the received user requests and other data, and then feeds back the processing results, such as the virtual try-on video generated according to the user's request, to the client installed on terminal 120.

[0100] In some embodiments, server 110 is equipped with a virtual try-on model, which is pre-trained and can be various existing neural network models, including but not limited to feedforward neural networks, convolutional neural networks (CNNs), and recurrent neural networks (RNNs). Server 110 is the main entity executing the virtual try-on video generation method and is deployed locally or in the cloud. Server 110 is used to generate virtual try-on videos based on images of clothing to be tried on, user facial images, and user videos.

[0101] In addition, in one embodiment, the server 110 is provided with a clothing try-on image database for storing multiple clothing try-on images.

[0102] The technical solution of this application will be described in detail below with reference to specific embodiments.

[0103] Optionally, Figure 2 A flowchart illustrating a virtual try-on video generation method provided in this application embodiment. Figure 1 .like Figure 2 As shown, the method includes:

[0104] S201. Obtain images of the clothes to be tried on, and compress and encode the images of the clothes to obtain the first clothing features.

[0105] Specifically, the server can acquire images of clothing to be tried on and encode these images using the encoder in a Variational Autoencoder (VAE) to generate low-dimensional latent spatial features, i.e., obtain low-dimensional first clothing features. This reduces data complexity and storage space requirements, thereby reducing computational complexity.

[0106] S202. Obtain the user's face image, and perform feature fusion based on the clothing trial image and the user's face image to obtain clothing-face fusion features.

[0107] Specifically, in order to preserve the facial details of the user's face and provide a more realistic and high-quality virtual try-on effect, the virtual try-on video generation method provided in this application embodiment can further acquire the user's facial image and perform feature fusion based on the try-on clothing image and the user's facial image to obtain clothing-face fusion features.

[0108] S203. Acquire user videos and extract video feature images from user videos.

[0109] Specifically, the server can further acquire user videos and extract video feature images from them. These video feature images can be composed of extracted user pose images, clothing mask images, clothing occlusion images, noise images, and the first user skeleton image, stitched together.

[0110] S204. Input the first clothing feature, clothing-face fusion feature, and video feature image into the fusion model to obtain the fused video.

[0111] In this embodiment, the fusion model includes a reference model and a denoising model. The reference model includes M cascaded reference units, where M is an integer greater than 1. The denoising model includes M cascaded denoising units connected end-to-end. The reference units include a first spatial attention layer and a first cross attention layer cascaded in sequence. The denoising units include a second spatial attention layer, a second cross attention layer, and a time alignment layer cascaded in sequence. The first clothing feature is used as the input to the first first spatial attention layer. The video feature image and the output of the first first spatial attention layer are used as the input to the first second spatial attention layer. The clothing-face fusion feature is used as the input to the first cross attention layer and the second cross attention layer.

[0112] Specifically, the pre-set virtual try-on model in the server includes a fusion model. This fusion model processes the first clothing features and the clothing-face fusion features using a reference model containing multiple reference units (Reference Nets). Specifically, the first clothing features are processed through a first spatial attention layer, extracting fine-grained spatial features from the input try-on clothing image. The processing result is then output to the second spatial attention layer of the corresponding denoising unit in the first cross-attention layer and the denoising model. The first cross-attention layer processes the features output from the first spatial attention layer and the clothing-face fusion features, and outputs the result to the next reference unit. The reference unit can obtain fine-grained spatial features of the try-on clothing image from the input clothing-face fusion features.

[0113] The second spatial attention layer processes the video feature image and the features output by the first spatial attention layer in the corresponding reference unit, and outputs the processing result to the second cross-attention layer. The second cross-attention layer processes the features output by the second spatial attention layer and the clothing-face fusion features, and outputs the processing result to the temporal alignment layer. The temporal alignment layer smooths the motion vectors using techniques such as optical flow estimation and temporal consistency constraints to ensure smooth transitions and stability between consecutive video frames, and outputs the processing result to the next denoising unit. Finally, through the processing of M cascaded reference units and M cascaded denoising units, a fused video is generated.

[0114] S205. Decode the fused video to obtain a virtual try-on video.

[0115] Specifically, the server decodes the fused video output by the fusion model using a VAE decoder to obtain a virtual try-on video.

[0116] Figure 3 This is a schematic diagram of the overall architecture of the virtual fitting model provided in the embodiments of this application. Figure 3 As shown, the virtual try-on model preset in the server includes a fusion model 340. The virtual try-on model can compress and encode the acquired try-on clothing image 310 to obtain the first clothing feature 311; and can perform feature fusion based on the acquired try-on clothing image 310 and the user face image 320 to obtain clothing face fusion feature 321; and can also extract video feature image 331 from the acquired user video 330. The fusion model 340 includes a reference model 341 and a denoising model 343. The reference model 341 includes M cascaded reference units 342, where M is an integer greater than 1. The denoising model 343 includes M denoising units 344 connected end-to-end. The reference unit 342 includes a first spatial attention layer 3421 and a first cross attention layer 3422 cascaded in sequence. The denoising unit 344 includes a second spatial attention layer 3441, a second cross attention layer 3442, and a time alignment layer 3443 cascaded in sequence. The first clothing feature is used as the input to the first first spatial attention layer 3421. The video feature image and the output of the first first spatial attention layer 3421 are used as the input to the first second spatial attention layer 3441. The clothing and face fusion feature is used as the input to the first cross attention layer 3422 and the second cross attention layer 3442.

[0117] The fusion model 340 processes the first clothing feature through the first spatial attention layer 3421 included in the reference unit 342 of the reference model 341, and outputs the processing result to the second spatial attention layer 3441 of the first cross attention layer 3422 and the corresponding denoising unit 344 in the denoising model 343. The first cross attention layer 3422 processes the features output by the first spatial attention layer 3421 and the clothing-face fusion features, and outputs the processing result to the next reference unit 342. The second spatial attention layer 3441 processes the video feature image and the features output by the first spatial attention layer 3421 in the corresponding reference unit 342, and outputs the processing result to the second cross attention layer 3442; the second cross attention layer 3442 processes the features output by the second spatial attention layer 3441 and the clothing-face fusion features, and outputs the processing result to the time alignment layer 3443; the time alignment layer 3443 reduces the jitter problem between video frames through techniques such as optical flow estimation and time consistency constraints, and outputs the processing result to the next denoising unit 344. Finally, through the processing of M cascaded reference units 342 and M end-to-end denoising units 344, a fused video 350 is generated. The server then decodes the fused video 350 output by the fusion model using a VAE decoder to obtain a virtual try-on video 360.

[0118] This embodiment provides a method for generating virtual try-on videos. The method includes: acquiring images of clothing to be tried on, and compressing and encoding the images to obtain first clothing features; acquiring user face images, and fusing features based on the clothing images and user face images to obtain clothing-face fusion features; acquiring user videos, and extracting video feature images from the user videos; inputting the first clothing features, clothing-face fusion features, and video feature images into a fusion model to obtain a fused video. The fusion model includes a reference model and a denoising model. The reference model includes M cascaded reference units, where M is an integer greater than 1. The denoising model includes M cascaded denoising units. The reference units include a first spatial attention layer and a first cross-attention layer cascaded sequentially. The denoising units include a second spatial attention layer, a second cross-attention layer, and a time alignment layer cascaded sequentially. The first clothing features are used as input to the first first spatial attention layer, the video feature images and the output of the first first spatial attention layer are used as input to the first second spatial attention layer, and the clothing-face fusion features are used as input to the first and second cross-attention layers. The fused video is then decoded to obtain a virtual try-on video. The following technical effects were achieved: The VAE encoder encodes the clothing images to generate low-dimensional latent spatial features, resulting in low-dimensional first clothing features. This reduces data complexity and storage requirements, thereby lowering computational complexity. The first spatial attention layer processes these first clothing features, extracting fine-grained spatial features from the input clothing images, improving the detail accuracy of the generated video frames. By acquiring user face images and fusing features from the clothing and face images, clothing-face fusion features are obtained, preserving facial details and providing a more realistic and high-quality virtual try-on effect. Optical flow estimation and temporal consistency constraints are used to smooth motion vectors, reducing jitter between video frames and improving the smoothness and stability of consecutive video frames, thus enhancing the fluency of the virtual try-on video.

[0119] Figure 4 A flowchart illustrating a virtual try-on video generation method provided in this application embodiment. Figure 2 In one possible example, such as Figure 4 As shown, in this embodiment... Figure 2 Based on the examples, this paper provides a detailed explanation of how to obtain clothing-face fusion features and how to extract video feature images from user videos. Figure 4 As shown, the method includes:

[0120] S401. Obtain an image of the clothing to be tried on, and compress and encode the image of the clothing to obtain the first clothing feature.

[0121] Specifically, S401 is similar to S201, and will not be described again in this embodiment.

[0122] S402. Obtain the user's face image, extract the second clothing feature from the clothing trial image, and extract the face feature from the user's face image.

[0123] Specifically, after the server acquires the image of the clothing to be tried on and the image of the user's face, it can extract semantic features from the image of the clothing to be tried on using a Contrastive Language-Image Pre-Training (CLIP) image encoder to obtain the second clothing feature of the image of the clothing to be tried on; and, it can extract the facial features from the image of the user's face using a CLIP image encoder.

[0124] S403. Perform a linear transformation on the second clothing feature to obtain a linear clothing feature, and perform a linear transformation on the face feature to obtain a linear face feature.

[0125] Specifically, the server performs a linear transformation on the second clothing features in the fully connected layer (Linear Layer) of the virtual fitting model to obtain linear clothing features, and performs a linear transformation on the facial features to obtain linear facial features.

[0126] S404. Concatenate the linear clothing features and linear face features into the first linear feature.

[0127] Specifically, the server can further concatenate linear clothing features and linear facial features into a first linear feature at the fully connected layer of the virtual try-on model.

[0128] S405. Perform a linear transformation on the first linear feature to obtain the second linear feature.

[0129] Specifically, the server can further perform a linear transformation on the first linear feature to obtain a second linear feature.

[0130] S406. Generate the embedding vector of the second linear feature as the clothing-face fusion feature.

[0131] Specifically, the server can generate an embedding vector of the second linear feature based on the second linear feature, which can then be used as the clothing-face fusion feature.

[0132] S407. Acquire user video, and extract user pose image, clothing mask image, clothing occlusion image, noise image, and first user skeleton image from user video.

[0133] In this embodiment, after the server acquires the user video, it extracts the user pose image, clothing mask image, clothing occlusion image, noise image, and first user skeleton image from the user video, including:

[0134] Based on the DensePose algorithm, the positions of important parts of the user's body are identified from each video frame of the user's video, and the user's pose is predicted based on these positions to obtain the user pose image, thereby achieving accurate localization of dynamic people and estimation of user pose.

[0135] Based on the Grounding Segment Anything Model (GroundingSAM) algorithm, the clothing region is segmented from each video frame of the user video, and an image that distinguishes the clothing region from the rest of the region is generated as a clothing mask image. In addition, the clothing region in the video frame is occluded to obtain the clothing occlusion image.

[0136] In one embodiment, the clothing region can be segmented from each video frame of the user video according to the GroundingSAM algorithm, and the clothing region can be occluded by combining the paste function in the Python Imaging Library (PIL) to obtain a clothing occluded image.

[0137] Based on the Depth-Wise Pose Estimation (DWPose) algorithm, user skeletal keypoints are extracted from each video frame of the user video, and a user skeleton image is generated based on the skeletal keypoints. The user's skeletal keypoints include the user's joint and limb positions, which can provide basic skeletal information of the user.

[0138] Each frame of the user's video is input into a filter or noise model to obtain a noisy image. Specifically, depending on the type of image noise, each frame of the user's video can be input into a filter, such as a median filter or a Gaussian filter, to obtain a noisy image; alternatively, a mathematical noise model of the image and noise can be established and used to obtain the noisy image.

[0139] S408. Perform a convolution operation on the first user's skeletal image to obtain the second user's skeletal image.

[0140] Specifically, to improve the feature consistency and detail fidelity of the generated virtual try-on video, this application embodiment designs a multi-condition pose perception control module. In this module, a second user skeletal image is obtained by performing convolution operations on the first user skeletal image and encoding the sequence of the first user skeletal image using a pose encoder composed of multiple convolutional layers. The second user skeletal image and the user pose image, two complementary pose signals, are then merged into a conditional embedding. The user pose image captures the user's subtle poses and details to accurately handle the interactions between different body parts. Furthermore, to further improve the detail accuracy of the video frames generated by the virtual try-on model and enhance the smoothness of complex movements in the generated video frames, especially improving the accuracy of subtle details such as facial expressions and finger movements, the skeletal image can be used as an additional control condition. By enriching the conditions of the latent diffusion model with comprehensive pose attributes, the generation process can be finely controlled, thereby improving the detail accuracy of the video frames and the smoothness of the virtual try-on video.

[0141] In one embodiment, the attitude encoder may consist of four convolutional layers with a kernel size of 4 and a stride of 2, resulting in 16, 32, 64, and 128 output channels per layer, respectively.

[0142] S409. Superimpose the pixels of the noisy image and the second user skeleton image to obtain the noisy skeleton image.

[0143] Specifically, in the multi-condition pose perception and control module, the noisy image and the second user skeleton image can be superimposed pixel by pixel to obtain the noisy skeleton image.

[0144] S410. The user pose image, clothing mask image, clothing occlusion image, noise image, and noise skeleton image are stitched together to form a video feature image.

[0145] In this embodiment, the multi-condition posture perception and control module can stitch together the user posture image, clothing mask image, clothing occlusion image, noise image and noise skeleton image into a video feature image.

[0146] Specifically, the user's pose image, clothing mask image, clothing occlusion image, and noise image from the user video can be stitched together frame by frame to form a rich stitching feature in the latent space. Simultaneously, a noisy skeleton image is added to the stitching feature in the latent space to further improve the detail accuracy of the virtual try-on video generated by the diffusion model.

[0147] S411. Input the first clothing feature, clothing-face fusion feature, and video feature image into the fusion model to obtain the fused video.

[0148] Specifically, in each denoising unit of the denoising model included in the fusion model, the second spatial attention layer can integrate the detailed features of the clothing image output by the first spatial attention layer in the corresponding reference unit into the corresponding denoising unit. This can help the virtual try-on model learn the detailed features of the clothing image in the feature space, thereby improving the detail accuracy of the clothing in the generated virtual try-on video.

[0149] Specifically, when the detailed feature image of the clothing try-on image output by the first spatial attention layer in the reference unit is about to enter the second spatial attention layer of the corresponding denoising unit, in order to align the detailed feature image with the user video and ultimately generate a complete virtual try-on video, the number of video frames N (N should be an integer) contained in the user video can be obtained. The detailed image about to be input into the denoising unit is then copied N-1 times before being input into the second spatial attention layer of the corresponding denoising unit, ensuring that the number of detailed images matches the number of video frames. Furthermore, in the second spatial attention layer, it is concatenated with the video feature image input to that layer to obtain the first clothing video fusion feature image.

[0150] In the first spatial attention layer, self-attention operations can be performed on the detail feature images of the clothing try-on image; in the second spatial attention layer, self-attention operations can be performed on the first clothing video fusion feature image. After the self-attention operation is completed in the first spatial attention layer, the resulting detail feature image is output to the first cross-attention layer of the reference unit. After the first cross-attention layer completes the self-attention operation on the detail feature image and the clothing-face fusion feature image, the result is output to the first spatial attention layer of the next reference unit, until self-attention operations are completed in all reference units of the reference model. After the self-attention operation is completed in the second spatial attention layer to obtain the second clothing video fusion feature image, the first half of the second clothing video fusion feature image can be selected as the third clothing video fusion feature image and output to the second cross-attention layer of the denoising unit. After the self-attention operation on the third clothing video fusion feature image and the clothing-face fusion feature image is completed in the second cross-attention layer, the result is output to the time alignment layer.

[0151] The time alignment layer includes: a frame optical flow calculation module, a motion vector smoothing module, a reverse optical flow correction module, and a timing transformation module. The output of the second cross-attention layer is used as the input to the frame optical flow calculation module in the corresponding time alignment layer.

[0152] The frame optical flow calculation module is used to calculate motion vectors between video frames. For example, it estimates motion vectors between video frames using the Lucas-Kanade optical flow estimation algorithm. These motion vectors describe the positional changes of each pixel as it moves from the current video frame to the next. This process can be expressed by the following formula:

[0153] I(x,y,t)=I(x+u,y+v,t+1)

[0154] Where I(x, y, t) refers to the pixel value of pixel (x, y) at time t, u refers to the displacement of the pixel in the x direction, v refers to the displacement of the pixel in the y direction, and I(x+u, y+v, t+1) refers to the pixel value of pixel (x+u, y+v) at time t+1.

[0155] The motion vector smoothing module is used to smooth motion vectors to obtain smoothed motion vectors. Specifically, it smooths the estimated motion vectors (u, v) using a weighted average to make the motion vectors more stable.

[0156] (u s v s )=α(u pre， v pre )+(1-α)(u,v)

[0157] Among them, (u s v s () refers to the smoothed motion vector (u, v) after a smoothing operation. pre， v pre ) refers to the smoothed motion vector from the previous video frame to the current video frame, where α is the smoothing coefficient and α∈[0,1].

[0158] The reverse optical flow correction module remaps pixels in the current video frame to their positions in the previous video frame based on the smoothed motion vector. In other words, it remaps pixels in the current video frame to their positions in the previous video frame based on the smoothed motion vector, thereby promoting smooth and stable transitions between video frames. The formula is as follows:

[0159] wrap(x, y, t) = I(x + u) s y+v s ,t)

[0160] Here, wrap(x, y, t) refers to the pixel value of pixel (x, y) at time t after reverse optical flow correction.

[0161] The temporal transformation module is used to perform self-attention calculations on pixels at the same spatial location in different video frames. Specifically, for a five-dimensional video feature map, for example...

[0162] ζ∈R b×c×f×h×w

[0163] Where ζ refers to any five-dimensional video feature map, R refers to the set of real numbers, b refers to the batch size, c refers to the number of channels in the video feature map (for color videos, the initial number of channels is three, corresponding to the red, green, and blue color channels), f refers to the number of video frames, h refers to the height of the video feature map, and w refers to the width of the video feature map. By reshaping the h-axis and w-axis to the b-axis, ζ can be represented as:

[0164] ζ ′ ∈R (b×h×w)×c×f

[0165] Using the timing transformation module, along the f-axis, ζ is... ′ By performing temporal attention calculations, the temporal transformation module can perform self-attention calculations on pixel features at the same spatial location in different video frames, thereby enhancing temporal consistency and ensuring a smooth transition between video frames.

[0166] S412. Decode the fused video to obtain a virtual try-on video.

[0167] S412 is similar to S205, and will not be described again in this embodiment.

[0168] This application provides a virtual try-on video generation method. Based on a dense human pose estimation algorithm, it identifies the positions of important body parts of the user in each video frame and predicts the user's pose based on these positions, obtaining a user pose image. This achieves accurate localization of dynamic figures and user pose estimation. Furthermore, it uses a skeletal map as an additional control condition, enriching the conditions of the latent diffusion model with comprehensive pose attributes, improving the accuracy of subtle details such as facial expressions and finger movements, thereby enhancing the detail precision of video frames and the smoothness of the virtual try-on video. An optical flow estimation algorithm is used to estimate motion vectors between video frames, and then a weighted average is applied to smooth these motion vectors, resulting in smoothed motion vectors that are more stable. Based on these smoothed motion vectors, pixels in the current video frame are remapped to their positions in the previous video frame, promoting smooth and stable transitions between video frames. Finally, self-attention calculation is performed on pixels at the same spatial location in different video frames—that is, standard self-attention calculation is performed in the time dimension—to enhance temporal consistency and ensure smooth transitions between video frames.

[0169] In this embodiment of the invention, electronic devices or main control devices can be divided into functional modules according to the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional module. It should be noted that the module division in this embodiment of the invention is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0170] Figure 5 This is a schematic diagram of a virtual try-on video generation device provided in an embodiment of this application. Figure 5 As shown, the virtual fitting video generation device includes: a clothing acquisition module 510, a face acquisition module 520, a video acquisition module 530, a fusion module 540, and a decoding module 550;

[0171] The clothing acquisition module 510 is used to acquire images of clothing to be tried on and to compress and encode the images of clothing to obtain the first clothing features.

[0172] The face acquisition module 520 is used to acquire the user's face image and perform feature fusion based on the clothing image and the user's face image to obtain clothing face fusion features;

[0173] The video acquisition module 530 is used to acquire user videos and extract video feature images from user videos;

[0174] The fusion module 540 is used to input the first clothing feature, clothing-face fusion feature, and video feature image into the fusion model to obtain a fused video. The fusion model includes a reference model and a denoising model. The reference model includes M cascaded reference units, where M is an integer greater than 1. The denoising model includes M denoising units connected end-to-end. The reference unit includes a first spatial attention layer and a first cross attention layer cascaded in sequence. The denoising unit includes a second spatial attention layer, a second cross attention layer, and a time alignment layer cascaded in sequence. The first clothing feature is used as the input to the first first spatial attention layer, the video feature image and the output of the first first spatial attention layer are used as the input to the first second spatial attention layer, and the clothing-face fusion feature is used as the input to the first cross attention layer and the second cross attention layer.

[0175] The decoding module 550 is used to decode the fused video to obtain the virtual try-on video.

[0176] In one possible design, the time alignment layer in the fusion module 540 includes: an input module, a frame optical flow calculation module, a motion vector smoothing module, a reverse optical flow correction module, and a timing transformation module;

[0177] The input module is used to take the output of the second cross-attention layer as the input of the frame optical flow calculation module in the corresponding time alignment layer;

[0178] The frame optical flow calculation module is used to calculate the motion vectors between video frames;

[0179] The motion vector smoothing module is used to perform smoothing operations on motion vectors to obtain smoothed motion vectors.

[0180] The reverse optical flow correction module is used to remap the pixels in the current video frame to the previous video frame based on the smooth motion vector;

[0181] The timing transformation module is used to perform self-attention calculations on pixels at the same spatial location in different video frames.

[0182] In one possible design, the face acquisition module 520 includes: a first extraction module, a first linear transformation module, and a feature fusion module;

[0183] The first extraction module is used to extract semantic features from the image of the clothes being tried on to obtain the second clothing features, and to extract semantic features from the image of the user's face to obtain the face features;

[0184] The first linear transformation module is used to perform a linear transformation on the second clothing features to obtain linear clothing features, and to perform a linear transformation on the face features to obtain linear face features;

[0185] The feature fusion module is used to determine the clothing-face fusion features based on linear clothing features and linear face features.

[0186] In one possible design, the feature fusion module includes: a first concatenation module, a second linear transformation module, and an embedding vector module;

[0187] The first stitching module is used to stitch the linear clothing features and the linear face features into a first linear feature;

[0188] The second linear transformation module is used to perform a linear transformation on the first linear feature to obtain the second linear feature;

[0189] The embedding vector module is used to generate embedding vectors for the second linear feature, which serves as the clothing-face fusion feature.

[0190] In one possible design, the video acquisition module 530 includes a second extraction module, a convolution module, a pixel overlay module, and a second stitching module;

[0191] The second extraction module is used to extract user pose image, clothing mask image, clothing occlusion image, noise image, and first user skeleton image from the user video;

[0192] The convolution module is used to perform convolution operations on the first user skeleton image to obtain the second user skeleton image;

[0193] The pixel overlay module is used to overlay the noisy image and the second user skeleton image pixel by pixel to obtain the noisy skeleton image;

[0194] The second stitching module is used to stitch together the user pose image, clothing mask image, clothing occlusion image, noise image, and noise skeleton image into a video feature image.

[0195] In one possible design, the second extraction module includes a pose image module, an occlusion image module, a skeleton image module, and a noise image module;

[0196] The pose image module is used to identify the positions of important parts of the user's body in each video frame of the user's video, and predict the user's pose based on the position to obtain the user pose image;

[0197] The occlusion image module is used to segment the clothing area from each video frame of the user video and generate an image that distinguishes the clothing area from the rest of the area as a clothing mask image, and to occlude the clothing area in the video frame to obtain a clothing occlusion image.

[0198] The skeletal image module is used to extract user skeletal key points from each video frame of the user video and generate a first user skeletal image based on the skeletal key points.

[0199] The noise image module is used to input each video frame of the user's video into a filter or noise model to obtain a noise image.

[0200] This embodiment provides a virtual fitting video generation device that can execute a virtual fitting video generation method described in the above embodiment. Its implementation principle and technical effects are similar, and will not be repeated here.

[0201] In a specific implementation of the aforementioned virtual fitting video generation device, each module can be implemented as a processor. The processor can execute computer execution instructions stored in the memory, thereby enabling the processor to execute the aforementioned virtual fitting video generation method.

[0202] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device includes at least one processor 610 and a memory 620. The electronic device also includes a communication component 630. The processor 610, memory 620, and communication component 630 are connected via a bus 640.

[0203] In the specific implementation process, at least one processor 610 executes computer execution instructions stored in memory 620, causing at least one processor 610 to execute a virtual fitting video generation method as executed on the electronic device side as described above.

[0204] The specific implementation process of processor 610 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0205] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0206] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage.

[0207] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0208] The above description of the functions implemented by electronic devices and main control devices has introduced the solutions provided by the embodiments of the present invention. It is understood that, in order to implement the above functions, the electronic device or main control device includes hardware structures and / or software modules corresponding to the execution of each function. By combining the units and algorithm steps of the various examples described in the embodiments of the present invention, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of the present invention.

[0209] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the virtual try-on video generation method described above.

[0210] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0211] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in an electronic device or a host device.

[0212] This application also provides a computer program product, which includes a computer program stored in a readable storage medium. At least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the solution provided in any of the above embodiments.

[0213] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.

[0214] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating virtual fitting videos, characterized in that, include: Acquire images of the clothing to be tried on, and compress and encode the images to obtain the first clothing features; Acquire the user's face image, and perform feature fusion based on the clothing trial image and the user's face image to obtain clothing-face fusion features; Acquire user videos and extract video feature images from the user videos; The first clothing feature, the clothing-face fusion feature, and the video feature image are input into a fusion model to obtain a fused video. The fusion model includes a reference model and a denoising model. The reference model includes M cascaded reference units, where M is an integer greater than 1. The denoising model includes M cascaded denoising units. Each reference unit includes a first spatial attention layer and a first cross attention layer cascaded sequentially. Each denoising unit includes a second spatial attention layer, a second cross attention layer, and a time alignment layer cascaded sequentially. The first clothing feature is used as the input to the first first spatial attention layer. The video feature image and the output of the first first spatial attention layer are used as the input to the first second spatial attention layer. The clothing-face fusion feature is used as the input to the first cross attention layer and the second cross attention layer. The fused video is decoded to obtain a virtual try-on video.

2. The method according to claim 1, characterized in that, The time alignment layer includes: a frame optical flow calculation module, a motion vector smoothing module, a reverse optical flow correction module, and a timing transformation module; The output of the second cross-attention layer serves as the input to the frame optical flow calculation module in the corresponding time alignment layer; The frame optical flow calculation module is used to calculate the motion vectors between video frames; The motion vector smoothing module is used to perform smoothing operations on the motion vector to obtain a smoothed motion vector. The reverse optical flow correction module is used to remap the pixels in the current video frame to the previous video frame according to the smooth motion vector; The timing transformation module is used to perform self-attention calculation on pixels at the same spatial location in different video frames.

3. The method according to claim 1, characterized in that, The step of fusing features based on the image of the tried-on clothing and the user's facial image to obtain clothing-face fusion features includes: Semantic features are extracted from the images of the clothes being tried on to obtain second clothing features, and semantic features are extracted from the images of the user's face to obtain facial features; A linear transformation is performed on the second clothing feature to obtain a linear clothing feature, and a linear transformation is performed on the face feature to obtain a linear face feature; The clothing-face fusion feature is determined based on the linear clothing features and the linear face features.

4. The method according to claim 3, characterized in that, The step of determining the clothing-face fusion feature based on the linear clothing feature and the linear face feature includes: The linear clothing feature and the linear face feature are concatenated to form a first linear feature; The first linear feature is linearly transformed to obtain the second linear feature; An embedding vector of the second linear feature is generated as the clothing-face fusion feature.

5. The method according to any one of claims 1 to 4, characterized in that, Extracting video feature images from the user video includes: Extract user pose image, clothing mask image, clothing occlusion image, noise image, and first user skeleton image from the user video; Perform a convolution operation on the first user skeleton image to obtain the second user skeleton image; The noise image and the second user skeleton image are superimposed pixel by pixel to obtain the noise skeleton image; The user pose image, clothing mask image, clothing occlusion image, noise image, and noise skeleton image are stitched together to form the video feature image.

6. The method according to claim 5, characterized in that, The step of extracting the user pose image, clothing mask image, clothing occlusion image, noise image, and first user skeleton image from the user video includes: The positions of important parts of the user's body are identified from each video frame of the user's video, and the user's posture is predicted based on the positions to obtain a user posture image; The clothing area is segmented from each video frame of the user video, and an image is generated that distinguishes the clothing area from the remaining areas as the clothing mask image. The clothing area in the video frame is then occluded to obtain the clothing occlusion image. Extract user skeletal key points from each video frame of the user video, and generate a first user skeletal image based on the skeletal key points; Each video frame of the user video is input into a filter or noise model to obtain the noise image.

7. A virtual fitting video generation device, characterized in that, include: The clothing acquisition module is used to acquire images of clothing to be tried on and to compress and encode the images of clothing to obtain first clothing features. The face acquisition module is used to acquire the user's face image and perform feature fusion based on the clothing image and the user's face image to obtain clothing face fusion features; The video acquisition module is used to acquire user videos and extract video feature images from the user videos; A fusion module is used to input the first clothing feature, the clothing-face fusion feature, and the video feature image into a fusion model to obtain a fused video. The fusion model includes a reference model and a denoising model. The reference model includes M cascaded reference units, where M is an integer greater than 1. The denoising model includes M cascaded denoising units. Each reference unit includes a first spatial attention layer and a first cross attention layer cascaded sequentially. Each denoising unit includes a second spatial attention layer, a second cross attention layer, and a time alignment layer cascaded sequentially. The first clothing feature serves as the input to the first first spatial attention layer. The video feature image and the output of the first first spatial attention layer serve as the input to the first second spatial attention layer. The clothing-face fusion feature serves as the input to the first cross attention layer and the second cross attention layer. The decoding module is used to decode the fused video to obtain a virtual try-on video.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the virtual fitting video generation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the virtual fitting video generation method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the virtual fitting video generation method as described in any one of claims 1 to 6.

Images