Video virtual fitting framework in real scene based on controllable image diffusion model

Abstract

The invention discloses a video virtual fitting framework in a real scene based on a controllable image diffusion model. The video virtual fitting framework comprises a data preprocessing module used for constructing feature expressions of figures and clothes; the single-stage image fitting network is used for generating a virtual fitting image based on the preprocessed data and performing implicit deformation and fusion of clothes by using a cross attention mechanism; the diffusion guiding module is used for enhancing the time sequence consistency of the generated video, a self-supervision model Video MAE and a self-supervision image model DINO-V2 are introduced, the video is masked by the self-supervision model Video MAE, and a complete video is reconstructed through inter-frame information; and the long video generation module is used for generating a long video and keeping the consistency of the video, splitting the complete video into a plurality of short video clips, and reducing the jitter at the clip joints by using a time sequence common noise reduction strategy. The diffusion guiding module is designed, efficient time sequence consistent video virtual fitting is achieved through self-supervised learning, and the method has the advantages of being excellent in shielding processing, stable in long video generation and the like.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and computer vision technology, and in particular to a video virtual fitting framework in a real scene based on a controllable image diffusion model. Background Art

[0002] Video virtual fitting technology is an important research direction in the field of artificial intelligence and computer vision. Its goal is to provide users with the experience of trying on clothes in a virtual environment. This technology can not only improve the user experience of e-commerce platforms, but also provide users with personalized clothing matching suggestions. However, the application of existing video virtual fitting technology in real scenarios still faces many challenges.

[0003] At present, video virtual fitting is mainly divided into two categories: image-based virtual fitting and video-based virtual fitting. Image-based virtual fitting technology usually relies on image processing and deformation to generate static fitting effects. Video-based virtual fitting technology, on the other hand, requires processing each frame in the video sequence to generate a coherent fitting video.

[0004] In image-based virtual fitting technology, typical methods include using image generation networks (such as GANs) to deform and blend clothing to adapt it to the body shape and posture of the target person. Such methods can simulate the wearing effect of clothing to a certain extent, but they often perform poorly when dealing with complex human body movements and occlusions, and are difficult to apply to actual video fitting scenarios.

[0005] Video-based virtual fitting technology needs to deal with more complex temporal information and inter-frame consistency issues. Common methods include a two-stage processing flow: first, the clothing is deformed to adapt to the target person, and then the deformed clothing is fine-grainedly fused with the person. In order to ensure the temporal consistency of the generated video, additional timing modules are usually introduced. For example, the FW-GAN method uses the information of past frames to enhance the prediction of the current frame by introducing an optical flow-guided fusion module; the MV-TON method combines the memory module to optimize the generation result of each frame through spatiotemporal information; ClothFormer achieves realistic and spatiotemporal consistency through anti-occlusion distortion, appearance flow tracking, and dual-stream transformers.

[0006] Although existing technologies have achieved the effect of video virtual fitting to a certain extent, there are still several major defects in its application in real scenes:

[0007] 1. High data requirements: Existing video virtual fitting technology usually relies on a large amount of high-quality video data for training. The acquisition and annotation costs of this data are high, and the computing resources required for training the model are extremely high, making the popularization and application of the technology difficult.

[0008] 2. Insufficient timing consistency: Existing methods often rely on complex timing module design when dealing with inter-frame consistency. However, when processing long videos, the timing consistency of these methods is still difficult to guarantee, and jitter and incoherence are prone to occur, affecting the user experience.

[0009] 3. Occlusion and deformation problems: In real scenes, complex human body movements and limb occlusion are common phenomena. When the existing technology handles these situations, the deformation and fusion steps of clothing are easily affected by occlusion, resulting in unnatural or even ineffective fitting effects.

[0010] 4. Limited model generalization ability: As existing technologies are mainly trained on specific data sets, their generalization ability is limited. When faced with videos with different backgrounds, lighting, and character postures, the generated results are often unsatisfactory and difficult to adapt to diverse application scenarios.

[0011] Therefore, how to provide a video virtual fitting framework in a real scene based on a controllable image diffusion model is a problem that technical personnel in this field need to solve urgently. Summary of the invention

[0012] One object of the present invention is to propose a video virtual fitting framework in real scenes based on a controllable image diffusion model. The present invention designs a diffusion guidance module, and realizes efficient and time-consistent video virtual fitting through self-supervised learning and a single-stage image fitting network. It has the significant advantages of low data demand, excellent occlusion processing, strong model generalization ability and stable long video generation.

[0013] A video virtual fitting framework in a real scene based on a controllable image diffusion model according to an embodiment of the present invention includes:

[0014] Data preprocessing module: used to construct the feature expression of people and clothes, specifically including a person feature extraction unit and a clothing feature extraction unit;

[0015] Single-stage image fitting network: used to generate virtual fitting images based on preprocessed data and use cross-attention mechanism for implicit deformation and fusion of clothes;

[0016] Diffusion guidance module: used to enhance the temporal consistency of the generated video, introducing the self-supervised model Video MAE and the self-supervised image model DINO-V2. The self-supervised model Video MAE masks the video and reconstructs the complete video through inter-frame information;

[0017] Long video generation module: used to generate long videos and maintain video consistency by splitting the full video into multiple short video clips and using a temporal joint denoising strategy to reduce the jitter at the segment joints.

[0018] Furthermore, the character feature extraction unit is used to segment and perform posture detection on each frame of the video to generate a segmentation map and a posture map; the clothing feature extraction unit is used to extract image features of target clothing, including an original image and an edge map; the specific steps are:

[0019] Segment each frame of the video to obtain a segmentation map S = {S1,…,S N} and the posture graph P=P1,…,P N};

[0020] Construct a video sequence A = {A1,…,A N}, and the posture graph P = {P1, ..., P N}Together they constitute the expression of character characteristics;

[0021] Get the original image G and edge map E of the target clothes G ;

[0022] Use DINO-V2 to extract features and get the clothing feature vector F g ∈R 257×2048 , where the dimension of 257 represents the combination of 1 global feature and 256 local features.

[0023] Furthermore, the single-stage image fitting network occludes the video sequence of clothes A = {A1, ..., A N and posture graph P = {P1,…,P N} as the input of the conditional branch in the diffusion model, and the clothing features are injected into the diffusion model using a cross-attention approach.

[0024] Furthermore, the diffusion guidance module is based on an assumption that the smoother the video, the easier it is for VideoMAE to reconstruct the occluded parts from the inter-frame information, and the lower the reconstruction loss.

[0025] Furthermore, the extended guide specifically includes:

[0026] In the tth step of diffusion, the clothes region is cut out and the latent vector Z is t Decoded into video sequence I t ;

[0027] The video reconstruction loss is calculated by Video MAE:

[0028]

[0029] Among them, Ω represents the set of occluded pixels in the image, I t is the image at step t, which is obtained by passing the decoder D to the noise z t Decoded, is the image reconstructed by Video MAE at step t;

[0030] The similarity loss is calculated by DINO-V2, where L represents the video length, f represents the DINO function, and the distance function dist uses the spherical distance:

[0031]

[0032] Put the video reconstruction loss calculated by Video MAE and the similarity loss calculated by DINO-V2 into the DDIM process.

[0033]

[0034] Among them, c represents the conditional input of character features and clothing features, ∈ θ (z t ; t,c) represents the noise prediction loss of the diffusion model with parameter θ, w1 and w2 are both guide weights, and They represent the gradients of reconstruction loss and similarity loss respectively.

[0035] Furthermore, in the generation of the long video, the multiple short video segments are obtained by splitting, each segment is 16 frames, and there is an overlapping area of ​​8 frames between adjacent segments. In the noise reduction process, the latent vector of the overlapping area is the average of the latent vectors of the two adjacent video segments.

[0036] The beneficial effects of the present invention are:

[0037] The first is to reduce data requirements and computing costs: existing video virtual fitting technology usually requires a large amount of high-quality video data for training, while the present invention uses an image-based controllable diffusion model and only needs to be trained on static images. By constructing detailed character and clothing feature expressions through the data preprocessing module, the dependence on large amounts of video data and high computing resources is significantly reduced, thereby reducing the cost of data acquisition and computing.

[0038] The second is to improve temporal consistency: the diffusion guidance module designed by the present invention enhances the temporal consistency of the generated video by introducing the self-supervised model VideoMAE and the self-supervised image model DINO-V2. Video MAE uses video masking and reconstruction, based on the assumption of video smoothness, to keep the generated video features consistent between frames, thereby reducing the jitter and incoherence commonly seen in the video generation process.

[0039] The third is to improve the handling of occlusion and deformation problems: When dealing with complex human body movements and occlusion, the deformation and fusion of clothing are prone to failure in traditional methods. This invention uses a single-stage image fitting network and a cross-attention mechanism to achieve implicit deformation and fusion of clothing, avoiding the occlusion problem in the explicit deformation step, thereby generating a more natural virtual fitting effect.

[0040] Fourth, the generalization ability of the model is enhanced: the present invention uses the DINO-V2 model to extract clothing features, and by fusing global and local features, improves the adaptability of the model under different backgrounds, lighting, and character postures. Combined with the self-supervised learning method, the model maintains high generation quality and stability in a variety of application scenarios.

[0041] The fifth is to achieve stable generation of long videos: through the long video generation module, the complete video is split into multiple short video clips, and the overlapping areas of adjacent clips are processed using a temporal joint noise reduction strategy to reduce jitter at the connection of the clips and ensure the consistency and stability of long video generation. This strategy effectively solves the common problem of insufficient temporal consistency in the long video generation process.

[0042] Sixth, simplify the system architecture and improve efficiency: Compared with the traditional two-stage processing method, the present invention achieves high efficiency and simplicity of image generation through a single-stage image fitting network, avoids redundant steps and possible error accumulation in multi-stage processing, and improves the processing efficiency and reliability of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0044] Figure 1 A schematic diagram of a video virtual fitting framework in a real scene based on a controllable image diffusion model proposed by the present invention;

[0045] Figure 2 This is a schematic diagram of a single-stage virtual fitting network of a video virtual fitting framework in a real scene based on a controllable image diffusion model proposed by the present invention. DETAILED DESCRIPTION

[0046] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0047] refer to Figure 1 and Figure 2 ,A video virtual fitting framework in real scenes based on a controllable image diffusion model includes the following steps:

[0048] Data preprocessing module: used to construct the feature expression of people and clothes, specifically including a person feature extraction unit and a clothing feature extraction unit;

[0049] Single-stage image fitting network: used to generate virtual fitting images based on preprocessed data and use cross-attention mechanism for implicit deformation and fusion of clothes;

[0050] Diffusion guidance module: used to enhance the temporal consistency of the generated video, introducing the self-supervised model Video MAE and the self-supervised image model DINO-V2. The self-supervised model Video MAE masks the video and reconstructs the complete video through inter-frame information;

[0051] Long video generation module: used to generate long videos and maintain video consistency by splitting the full video into multiple short video clips and using a temporal joint denoising strategy to reduce the jitter at the segment joints.

[0052] In this embodiment, the character feature extraction unit is used to segment and perform posture detection on each frame of the video to generate a segmentation map and a posture map; the clothing feature extraction unit is used to extract image features of target clothing, including an original image and an edge map; the specific steps are:

[0053] Segment each frame of the video to obtain a segmentation map S = {S1,…,S N} and the posture graph P=P1,…,P N};

[0054] Construct a video sequence A = {A1,…,A N}, and the posture graph P = {P1, ..., P N}Together they constitute the expression of character characteristics;

[0055] Get the original image G and edge map E of the target clothes G ;

[0056] Use DINO-V2 to extract features and get the clothing feature vector F g ∈R 257×2048 , where the dimension of 257 represents the combination of 1 global feature and 256 local features.

[0057] In this embodiment, the single-stage image fitting network occludes the video sequence of clothes A=A1,…,A N} and posture graph P = {P1,…,P N} as the input of the conditional branch in the diffusion model, and the clothing features are injected into the diffusion model using a cross-attention approach.

[0058] In this implementation, the diffusion guidance module is based on an assumption that the smoother the video, the easier it is for VideoMAE to reconstruct the occluded part from the inter-frame information, and the lower the reconstruction loss.

[0059] In this implementation, the extended guidance specifically includes:

[0060] In the tth step of diffusion, the clothes region is cut out and the latent vector Z is t Decoded into video sequence I t ;

[0061] The video reconstruction loss is calculated by Video MAE:

[0062]

[0063] Among them, Ω represents the set of occluded pixels in the image, I t is the image at step t, which is obtained by passing the decoder D to the noise z t Decoded, is the image reconstructed by Video MAE at step t;

[0064] The similarity loss is calculated by DINO-V2, where L represents the video length, f represents the DINO function, and the distance function dist uses the spherical distance:

[0065]

[0066] Put the video reconstruction loss calculated by Video MAE and the similarity loss calculated by DINO-V2 into the DDIM process.

[0067]

[0068] Among them, c represents the conditional input of character features and clothing features, ∈ θ (z t ; t,c) represents the noise prediction loss of the diffusion model with parameter θ, w1 and w2 are both guide weights, and They represent the gradients of reconstruction loss and similarity loss respectively.

[0069] In this embodiment, the multiple short video segments obtained by splitting in the long video generation have each segment of 16 frames, and there is an overlapping area of ​​8 frames between adjacent segments. During the noise reduction process, the latent vector of the overlapping area is the average of the latent vectors of the two adjacent video segments.

[0070] Embodiment 1:

[0071] In order to verify the feasibility of the present invention in implementation, the present invention is applied to a large e-commerce platform. The platform has millions of users visiting it every day, and a large number of users are troubled by the inability to try on clothes during online shopping, especially when it comes to complex clothing such as dresses, sportswear, etc. This problem is particularly prominent. In order to meet user needs and improve service quality, the platform decided to adopt the video virtual fitting framework in real scenes based on the controllable image diffusion model of the present invention.

[0072] In actual application, after visiting the e-commerce platform and selecting a certain garment, the user can use a mobile phone or computer camera to record a video containing various actions, such as walking, turning and waving, etc. After the video is uploaded to the platform, the platform inputs the video into the video virtual fitting framework of the present invention.

[0073] First, the data preprocessing module processes the videos uploaded by users. Each frame of the video image is segmented to generate a segmentation map and a posture map, and the features of the target clothing are extracted, including the original image and the edge map. This step uses the DINO-V2 model to extract clothing feature vectors, which include 1 global feature and 256 local features.

[0074] Next, the single-stage image fitting network takes the person features and clothing features in the video as conditional inputs and uses the cross-attention mechanism to achieve implicit deformation and fusion of clothing. In this process, each frame in the video is processed separately and a virtual fitting image is generated that is consistent with the original video time sequence.

[0075] In order to ensure the temporal consistency of the generated video, the diffusion guidance module plays a key role. This module uses the self-supervised model Video MAE to mask the video and reconstruct the complete video through the inter-frame information. By calculating the video reconstruction loss and similarity loss, the diffusion guidance module maintains the consistency of inter-frame features during the process of image generation by the diffusion model, thereby generating a try-on video with smooth temporal sequence.

[0076] In the process of generating long videos, the entire video is split into multiple short video clips, each containing 16 frames, and there are 8 frames of overlap between adjacent clips. Through the temporal joint noise reduction strategy, the latent vector of the overlapping area is the average of the latent vectors of the two adjacent video segments, which can effectively reduce the jitter at the connection of the clips and ensure the consistency and stability of the long video.

[0077] Table 1 Comparison of experimental data between traditional method and the method of the present invention

[0078]

[0079]

[0080] From the analysis of the above table, it can be seen that in terms of data requirements, the traditional method requires each user to upload about 100MB of video data, while the method of the present invention only needs to upload about 10MB of image data, and the data requirements are reduced by about 90%. In terms of timing consistency, in the video generated by the traditional method, there are an average of 3-5 obvious inter-frame jitters for every 10 seconds of video, while in the video generated by the method of the present invention, there are only 0-1 slight inter-frame jitters for every 10 seconds of video. In terms of occlusion processing effect, when the traditional method handles occlusion and complex movements, 20% of the try-on videos have clothing deformation failure problems, while under the same circumstances, the method of the present invention reduces the incidence of clothing deformation failure problems to less than 5%. In terms of user satisfaction, the user satisfaction score of the try-on video of the traditional method is 3.8 points (out of 5 points), and the user satisfaction score of the try-on video of the method of the present invention is increased to 4.7 points (out of 5 points). In terms of the stability of long video generation, traditional methods often experience timing inconsistencies and jitter at the connections when generating videos longer than 1 minute. However, the method of the present invention can still maintain smoothness and stability without obvious jitter and inconsistency when generating videos longer than 5 minutes.

[0081] In summary, the present invention has performed well in reducing data requirements, improving timing consistency, handling occlusion problems, and improving the stability of long video generation, and is significantly better than traditional methods. After using the video virtual fitting function of the present invention, users highly praised the fitting effect and the overall shopping experience, and the user stickiness and conversion rate of the platform were also significantly improved. This fully demonstrates the huge potential and superiority of the present invention in practical applications.

[0082] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A video virtual fitting framework in real scenes based on a controllable image diffusion model, characterized by: include: Data preprocessing module: used to construct the feature expression of people and clothes, specifically including a person feature extraction unit and a clothing feature extraction unit; Single-stage image fitting network: used to generate virtual fitting images based on preprocessed data and use cross-attention mechanism for implicit deformation and fusion of clothes; Diffusion guidance module: used to enhance the temporal consistency of the generated video, introducing the self-supervised model Video MAE and the self-supervised image model DINO-V2. The self-supervised model Video MAE masks the video and reconstructs the complete video through inter-frame information; Long video generation module: used to generate long videos and maintain video consistency by splitting the full video into multiple short video clips and using a temporal joint denoising strategy to reduce the jitter at the segment joints.

2. The video virtual fitting framework in real scene based on controllable image diffusion model according to claim 1 is characterized in that: The character feature extraction unit is used to segment and perform posture detection on each frame of the video to generate a segmentation map and a posture map; the clothing feature extraction unit is used to extract image features of target clothing, including an original image and an edge map; the specific steps are: Segment each frame of the video to obtain a segmentation map S = {S1,…,S N } and the posture graph P=P1,…,P N }; Construct a video sequence A = {A1,…,A N }, and the posture graph P = {P1, ..., P N }Together they constitute the expression of character characteristics; Get the original image G and edge map E of the target clothes G ; Use DINO-V2 to extract features and get the clothing feature vector F g ∈R 257×2048 , where the dimension of 257 represents the combination of 1 global feature and 256 local features.

3. The video virtual fitting framework in real scene based on controllable image diffusion model according to claim 1, characterized in that: The single-stage image fitting network takes the video sequence A=A1,…,A N } and posture graph P = {P1,…,P N } as the input of the conditional branch in the diffusion model, and the clothing features are injected into the diffusion model using a cross-attention approach.

4. The video virtual fitting framework in real scene based on controllable image diffusion model according to claim 1, characterized in that: The diffusion guidance module is based on an assumption that the smoother the video, the easier it is for VideoMAE to reconstruct the occluded parts from the inter-frame information, and the lower the reconstruction loss.

5. The video virtual fitting framework in real scene based on controllable image diffusion model according to claim 1, characterized in that: The extended guide specifically includes: In the tth step of diffusion, the clothes region is cut out and the latent vector Z is t Decoded into video sequence I t ; The video reconstruction loss is calculated by Video MAE: Among them, Ω represents the set of occluded pixels in the image, I t is the image at step t, which is obtained by passing the decoder D to the noise z t Decoded, is the image reconstructed by Video MAE at step t; The similarity loss is calculated by DINO-V2, where L represents the video length, f represents the DINO function, and the distance function dist uses the spherical distance: Put the video reconstruction loss calculated by Video MAE and the similarity loss calculated by DINO-V2 into the DDIM process. Among them, c represents the conditional input of character features and clothing features, ∈ θ (z t ; t,c) represents the noise prediction loss of the diffusion model with parameter θ, w1 and w2 are both guide weights, and They represent the gradients of reconstruction loss and similarity loss respectively.

6. The video virtual fitting framework in real scene based on controllable image diffusion model according to claim 1, characterized in that: The multiple short video clips obtained by splitting in the long video generation are each 16 frames, and there is an overlapping area of ​​8 frames between adjacent clips. In the noise reduction process, the latent vector of the overlapping area is the average of the latent vectors of the two adjacent video segments.

Images