Video reloading system and method based on spatial-temporal feature dimension connection

Abstract

The invention discloses a video clothes changing system and method based on spatial-temporal feature dimension connection, and the system comprises a human body posture detection module, a model clothes mask module, a clothes segmentation module and a video clothes changing module which are connected in sequence. And the video can be conveniently migrated to various video generation models.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, specifically a video clothing changing system and method based on spatio-temporal feature dimension connection. Background Art

[0002] Existing video clothing changing technologies, due to relying on a parallel UNet structure and using an image model combined with an additional temporal module, have problems of excessively high model complexity and poor temporal consistency. Their resource consumption exceeds the application resource limit and it is difficult to ensure video stability. Summary of the Invention

[0003] Aiming at the deficiencies of the existing technology in the complex processing flow, especially in scenarios with severe occlusion, where it is impossible to achieve highly stable, natural and realistic clothing changing images and videos, the present invention proposes a video clothing changing system and method based on spatio-temporal feature dimension connection. By injecting conditions in a feature linking manner, it can generate high-quality clothing changing videos and is convenient to be migrated to various video generation models.

[0004] The present invention is realized through the following technical solutions:

[0005] The present invention relates to a video clothing changing system based on spatio-temporal feature dimension connection, including: a human pose detection module, a model clothing mask module, a clothing segmentation module, and a video clothing changing module connected in sequence, where: the human pose detection module performs human pose detection on the input video frames and crops out the area where the model is located; the model clothing mask module generates a corresponding area mask based on the key point connection of the human pose; the clothing segmentation module performs precise segmentation according to the clothing mask sequence of the entire video to optimize the boundary of the clothing area; the video clothing changing module replaces the clothes in the input target video after masking the clothes with the target clothing according to the input target clothing image, the segmentation map of the target clothing, the input target video after masking the clothes, and the human pose sequence, that is, the clothing changing video. The purpose of masking the clothing of the target video is to remove the influence of the original video clothing on the clothing changing result. The purpose of inputting the human pose sequence is to improve the pose fidelity of the model in the clothing changing video result.

[0006] The so-called human pose detection means: using the detected key point information, determining the central position of the human pose, and taking this as a reference to crop a fixed-size area from the upper and lower edges as the area where the model is located, so as to reduce the interference of the background and other irrelevant content on the subsequent processing steps.

[0007] The so-called area mask takes the key point connection as the skeleton, draws a corresponding rectangular area, and takes the union of all rectangular areas as a rough model mask. By masking the clothing area of each frame of the video, a clothing mask sequence of the entire video is generated, which is used to accurately identify the area of the clothing on the model.

[0008] The non-clothing area is removed, including:

[0009] a) Face removal: Generate a face mask through face analysis (such as Face Parsing) to restore the model's face and chin area to avoid loss of facial information due to accidental cropping.

[0010] b) Hand and other non-clothing area removal: Use the human body segmentation model to accurately segment the hands and other non-clothing areas and exclude these parts from the mask. After the above processing, an accurate model clothing mask is finally generated.

[0011] The human body posture sequence is obtained by extracting from other videos or generating through simulation tools (such as game engines) and extracting from model video input scenes. Technical Effects

[0012] The present invention endows mainstream video models (for example, StableVideo diffusion, CogVideoX, mixed-element video large model, etc.) with video clothing-changing capability directly by means of connecting spatial and channel features. It can synthesize highly natural and time-stable clothing-changing video results only under the prompts of three kinds of information, namely, sample clothing, input model video or image, and guiding human posture video. Compared with existing technical means, it has advantages in video quality indicators (FVD, FID and CLIP Score) and visual results. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the system of the present invention;

[0014] Figure 2 This is a structural diagram of the video clothing-changing model of the present invention;

[0015] Figure 3 Body posture encoder for the video clothing-changing model proposed by the present invention

[0016] Figure 4 Sample image of model clothing mask generated by clothing mask module;

[0017] Figure 5 It is a schematic diagram of the effect of the embodiment;

[0018] The faces in the above figures are all generated by AI. DETAILED DESCRIPTION

[0019] like Figure 1 As shown, this embodiment relates to a video dressing system based on spatiotemporal feature dimension connection, including: a human posture detection module, a model clothing masking module, a clothing segmentation module and a video dressing module connected in sequence.

[0020] like Figure 1As shown, the human body pose detection module is implemented through a pose prediction deep model, which includes: a key point heatmap prediction unit, a heatmap to key point unit, and a pose prediction unit. Among them: The key point heatmap prediction unit obtains the human body key point heatmap result through model inference based on the input human body image. The heatmap to key point unit calculates the position result of the key points by obtaining the center of the hot spots according to the human body key point heatmap information. The pose prediction unit introduces body connections between the key points under the limitation of the human body structure topology based on the obtained key point position information, and finally obtains the human body pose structure diagram.

[0021] As Figure 1 shown, the model clothing mask module includes: a limb connection width map unit, a face mask prediction unit, and a clothing segmentation unit. Among them: The limb connection width map unit locates all the limbs in the human body limb structure diagram according to the human body limb information, draws a rectangular area centered on the limb connection to obtain preliminary human body clothing mask information. The clothing segmentation unit segments the input human body video frame to obtain the mask of all clothing areas in this video frame, and then uses the obtained clothing area mask to remove the non-target clothing areas in the preliminary human body clothing mask obtained in the previous step. The face mask prediction unit segments the input human body video frame to obtain the face mask area in this video frame, and then uses the face mask to further remove the face part area in the clothing mask to prevent the face of the model from being misprocessed.

[0022] As Figure 1 shown, the clothing segmentation module is implemented through a segmentation deep model, which infers the mask area of the clothing according to the input target clothing image, and this area is used to locate the shape and position of the clothing.

[0023] As Figure 2 shown, the video clothing change module is implemented through a video diffusion model, which includes: a variational autoencoder, a variational decoder, a pose encoder module, and a UNet network. Among them: The variational autoencoder reduces the dimension and compresses the input video or image, the target clothing image, and the target clothing mask image and converts them into diffusion features. The pose encoder module converts the input human body pose diagram into features with the same scale as the diffusion feature space, and through the spatio-temporal feature dimension connection method, the connected conditional features are obtained, and then input into the UNet network. The UNet network performs denoising processing under conditional constraints according to the conditional features to generate the target clothing change video feature sequence. The variational decoder restores the features of the target clothing change video feature sequence to the video or image scale.

[0024] As Figure 2As shown in the figure, the method for generating a clothing-changing video based on the above system in this embodiment directly connects the pose features with the input image or video stream in the channel dimension while connecting the target clothing sample with each frame of the input image or video in the spatial dimension, enabling the video clothing-changing model to be adapted to the video generation model, thereby achieving a high-quality video clothing-changing effect, specifically including:

[0025] Step 1: Construct a training dataset, specifically including:

[0026] 1.1 Use web crawler means to obtain 10,000 high-definition videos and the corresponding tiled clothing of the videos from the website https: / / www.net-a-porter.com.

[0027] 1.2 Construct the corresponding modeler mask video stream and human pose sequence of the dataset through the human pose detection module and the modeler's clothing mask module.

[0028] 1.3 Extract a first video frame and the complete human pose sequence to form the data for the single-image video clothing change part, and generate the dataset D from the mixed single-image and video data.

[0029] 1.4 Initialize the parameter weights of the video clothing-changing model. In this embodiment, the initialization weights of the video clothing-changing model adopt the pre-trained weights of Stable Video Diffusion (SVD). SVD includes: a variational autoencoder and a variational decoder. Among them: the variational autoencoder converts the video sequence into diffusion features, the UNet realizes noise reduction, and the variational decoder restores the denoised feature sequence to a video sequence.

[0030] Step 2: Sample the input training data from the dataset D, specifically: sample the input target clothing or the model picture I wearing the target clothing g , provide the model video or model picture I of the target clothing to be changed t and construct the input of the guiding pose sequence P = {p 0 , p 1 , …, p N}, where: N is the total number of frames.

[0031] Step 3: As Figure 2 shown, set the number of input channels of the convolutional operator of the input convolutional layer of the UNet network of the SVD model to 5 to adapt to the conditional injection features after connection through the spatio-temporal feature dimension.

[0032] As Figure 2As shown, the connection of the spatio-temporal feature dimensions is the conditional injection method of the overall model: the input clothing mask is connected to the input model video or image through the channel dimension, and then added to the body pose features at the corresponding positions, that is, added to the channels of the model video or image while ignoring the clothing mask dimension. The above connection operation obtains a five-channel feature, and this five-channel feature is concatenated with the target reference clothing image feature in the width direction of the feature to obtain a comprehensive input feature F that is twice as wide and contains all the prompt information for changing clothes.

[0033] Step 4: Input the comprehensive input feature F into the UNet network of the video clothing change model, and predict to obtain a sequence of video features for changing clothes. At this time, half of the features on the right side of the width dimension (width) are removed from the feature sequence to obtain the final video features for changing clothes, and then the features are restored to an RGB video through the decoder part of the variational autoencoder.

[0034] As Figure 3 shown, the pose encoder includes a first two-dimensional convolutional network, two residual modules, and a second two-dimensional convolutional network connected in sequence, where: the first two-dimensional convolutional network extracts features based on a human body pose map with a size of 768*512 and 3 channels, the input end of the second residual module receives the sum of the results output by the first two-dimensional convolutional network and the first residual module, the input end of the second two-dimensional convolutional network receives the sum of the results output by the first residual module and the second residual module, and the second two-dimensional convolutional network outputs conditional features with a size of 96*64 and 8 channels.

[0035] Step 5: Calculate the reconstruction loss between the features output by the UNet network and the true value video (Ground-Truth) features (obtained by encoding the true value video sequence with a variational autoencoder), so as to train the entire video clothing change model. After training is completed, the weights of the applicable video clothing change model are obtained.

[0036] The reconstruction loss L = E θ [|||∈ - ∈ θ (z t , F, t)|], where: t is the number of steps for noise reduction, ∈ is the true noise at the (t - 1)-th step of the diffusion model, ∈ θ is the predicted noise at the (t - 1)-th step of the diffusion model, and z t is the true noise at the t-th step.

[0037] In this embodiment, during the process of training the video clothing change model, video sample points and single clothing change sample points are extracted from the dataset D at a ratio of 2:1. An overly high single-image sample ratio will cause the model to converge too slowly.

[0038] The training is preferably set to 80,000 steps to obtain a video clothing change model with stable effects.

[0039] After specific actual experiments, under the specific environment settings of 8 Nvidia A800 GPUs, CUDA version 12.2, and driver version 535.54.03, with an input image resolution of 768*512, the input video format being MP4, and the input image format being PNG, training for 80,000 generations on the video clothing change dataset D constructed in the present invention, with a batch size of 8 per GPU and a learning rate of 0.0005, generating a 768*512 clothing change video using the trained model. In a specific comparative experiment, the publicly available video model runway dataset of UBC Fashion is used. Take 100 frames of each video, and after masking the clothes of these 100 frames, use them as the input model video. The first frame is the target reference clothing picture. The specific experimental operations are as follows:

[0040] Step i, construct the input data, including providing the input target clothing or a picture I of a model wearing the target clothing g . Provide a model video or a picture I of the target clothing to be changed t , and the picture or video needs to pass through the human pose detection module and the model clothing mask module to obtain the body cropped area of the model, the body pose map or sequence, and the mask map or sequence of the body removing the original clothing. Construct the guiding pose sequence input P = {p 0 , p 1 , …, p N}), where N is the total number of frames. When the input form of the target clothing to be changed is a model video, the length is equal to that of the input video; when the input form of the target clothing to be changed is a model picture, the length is equal to the number of frames of the target generated video, and the number of frames of the target generated video is set by the user. Under this input condition, the sources of the pose sequence are mainly two types. One is to extract the pose sequence from the model runway videos on the Internet and then align the lengths of the limbs and torso of the pose sequence with those of the input model. The other is to simulate and generate through Unity or Unreal.

[0041] Step ii, as Figure 2 shown, connect the trained video clothing change model and the input clothing mask in the channel dimension with the input model video or picture, and then add them to the body pose features at the corresponding positions, that is, perform an addition operation on the channels of the model video or picture while ignoring the dimension of the clothing mask. Through this connection operation, a five-channel feature representation is obtained. Then, splice this five-channel feature with the target reference clothing picture feature in the feature width (Width) direction to generate a comprehensive input feature F with twice the original width. This input feature contains all the prompt information related to clothing change.

[0042] Step iii: Remove the right half of the output features obtained by the trained UNet, and then generate the final clothing-changing video through the pre-trained variational autoencoder.

[0043] As shown in Table 1, this method is compared with "Video Virtual Try-on using Diffusion Models[J] / / arXiv preprint arXiv:2405.11794, 2024.", "Tunnel try-on: Excavating spatial-temporal tunnels for high-quality virtual try-on in videos[C] / / Proceedings of the 32nd ACM International Conference on Multimedia. 2024:3199-3208.", and "Fashion-VDM: Video diffusion model for virtual try-on[C] / / SIGGRAPH Asia 2024 Conference Papers. 2024:1-11." on the video data selected from the UBC Fashion video dataset and YouTube. In Table 1, CLIP is the degree of semantic similarity evaluated using the CLIP model, which reflects whether the video content generated by the model is real. The smaller the value, the more real the content of the synthesized video. The values in Table 1 are the average results after frame-by-frame testing of 100-frame videos.

[0044] Table 1

[0045] As shown in Table 1, the present invention is better than the existing methods in the quantitative indicators of generating videos, which also reflects the superiority and practical value of the present invention. Figure 4 The visual effect diagram of the video clothing-changing model proposed by the present invention is shown.

[0046] Compared with the prior art, the present method does not require a complex reference model module, nor does it require complex injection mechanisms such as ControlNet and IP-Adapter. These mechanisms occupy a large amount of training resources in large-scale video models, making model training difficult or even impossible to achieve (for example, the required GPU video memory overhead exceeds 80G, and it is difficult to achieve training on top-level graphics cards such as A100 and H100). The present invention only needs to achieve conditional injection through feature connection in the channel and spatial dimensions, getting rid of the huge resource overhead brought by complex additional structure injection mechanisms (such as the reference model, also known as ReferenceNet), so as to be able to build video clothing change on existing video models at the lowest cost. Compared with the existing publicly available technologies (such as Try-tunnel-on and Vivid), it is more convenient in terms of process, input method, and training. The present invention can also be applied to various mainstream video models (such as Stable Video Diffusion, CogVideoX, and Huanyuan Video Large Model, etc.). By directly adjusting the input of the model and slightly changing the structure of the model (such as the number of channels in the input layer of the diffusion model), compared with the need to redesign the model in existing video clothing change technologies, the present invention does not require special model design, increasing the simplicity and usability of the system and method.

[0047] The above specific embodiments can be locally adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present invention. The protection scope of the present invention is subject to the claims and is not limited by the above specific embodiments. All implementation solutions within its scope are subject to the present invention.

Claims

1. A video dressing system based on spatiotemporal feature dimension connection, characterized in that: include: The human posture detection module, the model clothing masking module, the clothing segmentation module and the video clothing changing module are connected in sequence, wherein: the human posture detection module performs human posture detection on the input video frame and crops out the area where the model is located; the model clothing masking module generates the corresponding area mask based on the key point connection lines of the human posture; the clothing segmentation module performs precise segmentation according to the clothing masking sequence of the entire video to optimize the boundary of the clothing area; the video clothing changing module replaces the clothes of the input target video after masking the clothes with the target clothes, that is, the clothing changing video, according to the input target clothing image, the segmentation map of the target clothing, the input target video after masking the clothes, and the human posture sequence; the purpose of masking the clothes of the target video is to remove the influence of the clothes in the original video on the effect of the clothing changing result, and the purpose of inputting the human posture sequence is to improve the fidelity of the model posture in the clothing changing video result.

2. The video dressing system based on spatiotemporal feature dimension connection according to claim 1 is characterized in that: The human posture detection means: using the key point information obtained by the detection to determine the center position of the human posture, using this as a reference to cut a fixed size area from the upper and lower edges as the model area, thereby reducing the interference of background and other irrelevant content on subsequent processing steps; The region mask uses the key point connection line as the skeleton, draws the corresponding rectangular region, and uses the union of all rectangular regions as a rough model mask. By masking the clothing region of each frame of the video, a clothing mask sequence of the entire video is generated, which is used to accurately identify the clothing region on the model. The non-clothing area is removed, including: a) Face removal: Generate a face mask through face analysis to restore the model's face and chin area to avoid loss of facial information due to miscropping; b) Removal of hands and other non-clothing areas: Use the human body segmentation model to accurately segment the hands and other non-clothing areas, and exclude these parts from the mask. After the above processing, an accurate model clothing mask is finally generated.

3. The video dressing system based on spatiotemporal feature dimension connection according to claim 1 is characterized in that: The human posture detection module is implemented through a posture prediction deep model, which includes: a key point heat map prediction unit, a heat map to key point unit and a posture prediction unit, wherein: the key point heat map prediction unit obtains the human body key point heat map result based on the input human body image and model reasoning, the heat map to key point unit obtains the position result of the key point by calculating the hot spot center based on the human body key point heat map information, and the posture prediction unit introduces body lines between the key points based on the key point position information obtained under the restriction of the human body structure topology, and finally obtains the human body posture structure diagram.

4. The video dressing system based on spatiotemporal feature dimension connection according to claim 1 is characterized in that: The model clothing mask module includes: a limb connection wide map unit, a face mask prediction unit and a clothing segmentation unit, wherein: the limb connection wide map unit locates all limbs in the human limb structure diagram according to human limb information, draws a rectangular area with the limb connection as the center, and obtains preliminary human clothing mask information; the clothing segmentation unit obtains all clothing area masks in the video frame according to the input human video frame segmentation, and then uses the obtained clothing area mask to remove non-target clothing areas in the preliminary human clothing mask obtained in the previous step; the face mask prediction unit inputs the human video frame segmentation to obtain the face mask area in the video frame, and then uses the face mask to further remove the face part area in the clothing mask to prevent the model's face from being processed incorrectly.

5. The video dressing system based on spatiotemporal feature dimension connection according to claim 1 is characterized in that: The clothing segmentation module is implemented by a segmentation depth model, which infers and obtains a mask area of ​​clothing based on an input target clothing image, and the mask area is used to locate the shape and position of the clothing.

6. The video dressing system based on spatiotemporal feature dimension connection according to claim 1 is characterized in that: The video clothing-changing module is implemented through a video diffusion model, which includes: a variational autoencoder, a variational autodecoder, a posture encoder module and a UNet network, wherein: the variational autoencoder reduces the dimension of the input video or image, the target clothing image, and the target clothing mask image and converts them into diffusion features; the posture encoder module converts the input human posture graph into features with consistent spatial scale of the diffusion features, processes them through a spatiotemporal feature dimension connection method, obtains the connected conditional features, and then inputs them into the UNet network; the UNet network performs denoising processing under conditional constraints based on the conditional features to generate a target clothing-changing video feature sequence; the variational autodecoder restores the target clothing-changing video feature sequence features to the video or image scale.

7. A method for generating a clothing-changing video based on the system according to any one of claims 1 to 6, characterized in that: Directly connect the posture features with the input image or video stream in the channel dimension, and connect the target clothing sample with each frame of the input image or video in the spatial dimension, so that the video clothing changing model can be adapted to the video generation model, thereby achieving high-quality video clothing changing effects, including: Step 1: Build a training data set; Step 2: Sample input training data from the dataset D, specifically: sample input target clothing or model image I wearing target clothing g , provide a model video or model picture of the target clothing to be changed t And construct the guiding posture sequence input P = {p0, p1, ..., p N }, where: N is the total number of frames; Step 3, the number of input channels of the convolution operator of the input convolution layer of the Unet network of the SVD model is set to 5 to accommodate the conditional injection features after connection through the spatiotemporal feature dimensions; Step 4: Input the comprehensive input feature F into the Unet network of the video clothing-changing model to predict the video feature sequence of clothing-changing. At this time, the feature sequence removes half of the features on the right side of the width dimension to obtain the final clothing-changing video feature. Then, the feature is restored to RGB video through the decoder part of the variational autoencoder. Step 5: Calculate the reconstruction loss by combining the features output by UNet with the true video features to train the entire video clothing-changing model. After the training is completed, the applicable video clothing-changing model weights are obtained.

8. The clothing-changing video generation method according to claim 7, characterized in that: The step 1 specifically includes: 1.1 Use crawlers to obtain 10,000 high-definition videos and flat-lay clothing paired with the videos from https: / / www.net-a-porter.com; 1.2 Construct the model mask video stream and human posture sequence corresponding to the data set through the human posture detection module and the model clothing mask module; 1.3 Extract a first video image and a complete human posture sequence to form a single-image video clothing-changing data, and generate a dataset D by mixing the single-image and video data; 1.4 Initialize the parameter weights of the video clothing-changing model.

9. The method for generating a clothing-changing video according to claim 7, wherein: The initialization weights of the video clothing-changing model adopt the pre-trained weights of Stable Video Diffusion (SVD), and the SVD includes: a variational autoencoder and a variational autodecoder, wherein: the variational autoencoder converts the video sequence into diffusion features, UNet realizes denoising, and the variational autodecoder restores the denoised feature sequence to a video sequence.

10. The clothing-changing video generation method according to claim 7, characterized in that: The connection of the spatiotemporal feature dimensions is the conditional injection method of the overall model: the input clothing mask is connected to the input model video or picture through the channel dimension, and then added to the body posture feature at the corresponding position, that is, added to the channel of the model video or picture and ignoring the clothing mask dimension. The above connection operation obtains a five-channel feature, and this five-channel feature is spliced ​​with the target reference clothing picture feature in the width direction of the feature to obtain a comprehensive input feature F that is twice as wide and contains all the prompt information that needs to change clothes.

11. The clothing-changing video generation method according to claim 7, characterized in that: The posture encoder includes a first two-dimensional convolutional network, two residual modules and a second two-dimensional convolutional network connected in sequence, wherein: the first two-dimensional convolutional network performs feature extraction based on a human posture graph with a size of 768*512 and 3 channels, the input end of the second residual module receives the sum of the results output by the first two-dimensional convolutional network and the first residual module, the input end of the second two-dimensional convolutional network receives the sum of the results output by the first residual module and the second residual module, and the second two-dimensional convolutional network outputs conditional features with a size of 96*64 and 8 channels.

12. The clothing-changing video generation method according to claim 7, characterized in that: The reconstruction loss L = E θ [|||∈-∈ θ (z t ,F,t)|], where: t is the number of denoising steps, ∈ is the real noise of the diffusion model at step t-1, ∈ θ is the noise predicted by the diffusion model at step t-1, z t is the real noise of t steps.

Images