Video reloading method and device based on diffusion model and attitude estimation, and computer readable storage medium

Abstract

The invention belongs to the field of computer vision and the technical field of image processing, and particularly relates to a video reloading method and device based on a diffusion model and attitude estimation and a computer readable storage medium. The method comprises the following steps: acquiring a clothing image and a dressing video; building a clothing coding and decoding model and a posture estimation model; the clothing encoding and decoding model comprises a clothing encoding model and a clothing decoding model which are respectively used for extracting a hidden space feature vector of the image and restoring the hidden space feature vector of the image; the attitude estimation model is used for fusing the attitude key points with the original video data; training and testing a clothing coding and decoding model and a posture estimation model; building a video reloading model; and training the video reloading model. According to the method, the problems of incoordination between clothes and human limbs, distortion of action scenes, stiff clothes changing effect and the like due to lack of accurate modeling of human body dynamic characteristics in a traditional segmentation-based method are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and more specifically, relates to a video dressing method, device and computer-readable storage medium based on a diffusion model and posture estimation. Background Art

[0002] In today's rapidly developing digital and intelligent world, technologies based on computer vision and generative models have been widely applied in image and video processing, becoming effective solutions for visual synthesis and virtual presentation. The core of video dressing technology is to achieve realistic display and natural adaptation of clothing in virtual environments through precise human pose estimation and high-fidelity image generation. As a fusion of visual perception and generative modeling, video dressing methods have shown broad application prospects in a variety of fields, including virtual fitting, film and television production, and game development, and have achieved significant breakthroughs in digital human driving and dynamic interaction scenarios.

[0003] Chinese invention patent CN114565508A discloses a virtual dressing method and device, including obtaining a user image and a clothing template image; obtaining the neck key points corresponding to the user image and the clothing key points corresponding to the clothing template image; detecting whether the user's neck is occluded; if so, generating a dressing effect image using the neck area in the clothing template image based on the neck key points and clothing key points; if not, generating a dressing effect image using the neck area in the user image based on the neck key points and clothing key points. In this application, if the user's neck is not occluded, the user's neck area is repaired, the clothing area of the clothing template image is deformed, the user's hair area is deformed, and the user's neck area is used for dressing. If the user's neck is occluded, the color of the model's neck area in the clothing template image is transferred, the model's neck and clothing areas are deformed, the user's hair area is deformed, and the model's neck area is used for dressing.

[0004] Currently, the mainstream approach to costume change in videos relies on clothing segmentation and direct replacement. These methods segment the target person's clothing regions and, combined with traditional image generation techniques, map the new clothing onto the target person. However, due to a lack of accurate modeling of the human body's dynamic characteristics, these traditional segmentation-based methods often suffer from problems such as inconsistency between clothing and limbs, distorted action scenes, and abrupt costume changes, making them difficult to meet the complex and ever-changing demands of costume change in videos. Furthermore, existing methods suffer from significant deficiencies in inter-frame consistency and detail fidelity, making them unable to achieve high-quality costume changes in dynamic scenes. Summary of the Invention

[0005] The present invention aims to overcome at least one of the shortcomings of the aforementioned prior art by providing a video costume change method, apparatus, and computer-readable storage medium based on a diffusion model and posture estimation. In dynamic video scenes, this method accurately estimates human posture, capturing key points and limb structure information in real time. Combined with the diffusion model, this method generates high-quality clothing textures and details, ensuring the naturalness and realism of the costume change effect. The present invention utilizes the diffusion model to gradually generate images that closely match the clothing to the person's movements. Simultaneously, posture estimation enables dynamic adaptation of the clothing to the human body, overcoming the problems of clothing deformation, distortion, and inter-frame inconsistencies in traditional methods.

[0006] In addition, this method further enhances its adaptability to complex dynamic scenes. By processing input video frames with multiple postures and angles, it generates a visual effect in which clothing and body movements are coordinated, effectively improving inter-frame consistency and detail fidelity, making the video dressing process more in line with actual application needs, and providing efficient solutions for scenarios such as virtual fitting, film and television special effects, and digital human driving. The detailed technical solutions of the present invention are as follows: A video dressing method based on a diffusion model and posture estimation, the method comprising: S1, obtain clothing images and videos of models with different body shapes wearing clothing; S2. Build a clothing encoding and decoding model and a posture estimation model; the clothing encoding and decoding model includes a clothing encoding model and a clothing decoding model, which are respectively used to extract the latent space feature vector of the image and restore the latent space feature vector of the image; the posture estimation model is used to fuse the posture key points with the original video data to obtain a posture estimation video; S3, train and test the clothing encoding and decoding model and posture estimation model to verify the model accuracy; S4. Build a video dressing model, which integrates the clothing encoding and decoding model and the posture estimation model; S5. Use weighted KL divergence and MSE as loss functions to train the video dressing model, and test the video dressing model to verify the model accuracy.

[0007] Furthermore, the S1 specifically includes: S1-1: Data acquisition: capture image data of different clothing using a camera to obtain diverse clothing styles and detailed features, and obtain clothing images data; S1-2: For dynamic clothing scenes, capture dynamic clothing video data containing different movements to obtain dressing videos data; S1-3: Use labelme tool to label clothing images The data is used to cut out the foreground clothing and obtain the clothing mask image data; S1-4: Use labelme tool to The data is cut out for the foreground clothing and marked with 12 key nodes of the model's limbs to obtain the clothing video mask Data and motion posture key points data, where The data format is defined as: (1); In formula (1), are the key point coordinates, It is one of the 12 key points of the limbs; S1-5: Dataset It is divided into training set and test set in an 8:2 ratio.

[0008] Furthermore, the construction of the clothing encoding and decoding model and the posture estimation model specifically includes: S2-1: Build a clothing encoding model based on the Transformer architecture, build the image encoding part, perform patch slicing on clothing images and clothing mask images, then perform patch encoding and position encoding, and then perform concat feature merging. Use 4 Transformer Encoders to build the image encoder core. Transformer Encode uses the self-attention mechanism to build, and finally obtain the latent space feature vector of the image. , Represents vector dimension; S2-2: Build a clothing decoding model, using 4 Transformer Decoders to build the image decoder core. Transformer Decode uses a cross-attention mechanism to build the latent space feature vector Decoding and restoring to clothing images; S2-3: The clothing encoding model and clothing decoding model are linked in Encode-Decode mode. The features of the cross-attention mechanism at different levels are derived from the Transformer Encode at the same level. S2-4: Build a posture estimation model. Based on the YOLOv8 target detection architecture, replace its target detection decoding module with a posture key point detection module. After processing, the key point detection module is fused with the original video data features to finally obtain a posture estimation video. The posture key point detection module includes two parts: key point coordinate feature decoding and key point category feature decoding. The key point coordinate feature decoding and key point category feature decoding are used to extract key point coordinate features and key point category features, respectively.

[0009] Furthermore, the training and testing of the clothing encoding and decoding model and the posture estimation model specifically include: S3-1: The training set in the data is fed into the clothing encoding and decoding model. The training method adopts the warmup learning rate change strategy and trains for 50 rounds. The clothing encoding and decoding loss uses the weighted fusion cross entropy loss function and the Dice loss function to supervise the fine-tuning process. (2); In formula (2), CE and Dice represent the cross entropy loss function and Dice loss function respectively. is the input image, is the output image obtained by the clothing encoding and decoding model; Represents the weights of the two loss functions; Then, The test set in the data is fed into the model to verify the accuracy of the image encoding and decoding task; S3-2: The training set in the data is fed into the pose estimation model, and the learning rate change strategy adopts warmup for 50 rounds of training. The loss uses the root mean square error loss function: (3); In formula (3), is the number of key points, are the real key point coordinates, is the predicted key point coordinate; Then, The test set in the data is fed into the model to verify the accuracy of the key point detection task.

[0010] Furthermore, a video dressing model is built, specifically including: S4-1: Build a video dressing model, including a clothing encoding part and a posture estimation part. Based on the obtained clothing encoding and decoding model of the clothing encoding part and the posture estimation model of the posture estimation part, freeze the weight parameters in the two models; Clothing images Data input clothing encoding part: First, the clothing image is input into SAM2 to obtain the clothing mask image. After concat the clothing mask image obtained by SAM2 and the original clothing image, it is transmitted to the clothing encoding model to obtain the clothing encoding vector; Input the clothing video data into the pose estimation part: First, the clothing video is fed into SAM2 and the pose estimation model respectively to obtain the clothing mask video and dressing posture video , the dressing mask video and the dressing posture video are superimposed and merged to obtain the dressing posture vector with the character mask; S4-2: Add noise to the dress posture vector obtained in S4-1 to obtain a noisy dress posture vector. The noise needs to conform to the Gaussian distribution and is used to construct the data required for the diffusion model. The noisy dress posture vector is then fed into Unet for training to predict the noise distribution. At the same time, the clothing encoding vector is fed into Unet for training; S4-3: Transfer the image latent space vector obtained by Unet training to the clothing decoding model to decode the final clothing change video.

[0011] Furthermore, the S5 specifically includes: S5-1: Compare the predicted dressing videos with the real dressing videos. Use weighted KL divergence and MSE as loss functions to measure the difference between the generated sample distribution and the real data distribution. The loss function for the video dressing loss is defined as: (4); In formula (4), is the real sample data, To generate sample data, is the total amount of sample data, is the sample number, , is the dynamic weight; S5-2: Train only the Unet model, using the weighted KL divergence and MSE loss functions from S5-1. Feed the training set into the Unet model, using the warmup learning rate strategy, for 100 epochs. Then, feed the test set into the model to verify the accuracy of the video costume model.

[0012] In another aspect of the present invention, a device for a video dressing method based on a diffusion model and posture estimation is provided, the device comprising: at least one processor; and A memory storing instructions, which, when executed by the at least one processor, causes the at least one processor to execute the video dressing method based on the diffusion model and posture estimation as described above.

[0013] In another aspect of the present invention, a computer-readable storage medium is provided, which stores executable instructions. When the instructions are executed, the machine executes the video dressing method based on the diffusion model and posture estimation as described above.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a video dressing method, device, and computer-readable storage medium based on a diffusion model and posture estimation. By introducing a diffusion model, a highly consistent dressing effect is generated. First, key point information of the target person is extracted through precise posture estimation, and based on this information, the dressing object is adaptively adjusted so that the dressing object can be accurately combined with the person's posture. Second, the diffusion model is used for image generation to ensure a natural and smooth dressing effect, avoiding the problems of object incoordination or distortion in traditional dressing methods. On the basis of ensuring the accuracy of person posture estimation, the matching degree between the dressing object and the person's posture is automatically adjusted. In this way, diverse dressing effects are achieved, and the realism and visual expressiveness of the dressing are enhanced, making video dressing technology more suitable for actual application scenarios and improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of a video dressing method based on diffusion model and posture estimation described in the present invention.

[0016] Figure 2 This is a schematic diagram of key nodes of the model's limbs 12 in Example 1 of the present invention.

[0017] Figure 3 Schematic diagram of a clothing coding model in Example 1 of the present invention.

[0018] Figure 4 Schematic diagram of a clothing decoding model in Example 1 of the present invention.

[0019] Figure 5 Schematic diagram of the clothing encoding and decoding model in Example 1 of the present invention.

[0020] Figure 6 Schematic diagram of the posture estimation model in Example 1 of the present invention.

[0021] Figure 7 This is a schematic diagram of the video dressing model in Example 1 of the present invention.

[0022] Figure 8 This is a schematic diagram of the result of the dress-changing in Example 1 of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0025] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0026] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0027] Example 1 Ginseng Figure 1 This embodiment provides a video dressing method based on a diffusion model and posture estimation, the method comprising: S1. Data acquisition: Obtain clothing images and videos of models of different body shapes wearing clothing.

[0028] S1-1: Data acquisition: Use a camera to capture image data of different clothing, such as upper body clothing such as T-shirts, long sleeves, and shirts, lower body clothing such as pants and skirts, and full body clothing such as dresses and suits, to obtain diverse clothing styles and detailed features, and obtain clothing images. data; S1-2: For dynamic clothing scenes, capture clothing dynamic video data containing different actions to enhance the model's adaptability to complex scenes, thereby achieving natural transition and dynamic matching of clothing in the video dressing process, and obtaining dressing videos Data, different actions such as walking, turning, and waving; S1-3: Use labelme tool to label clothing images The data is used to cut out the foreground clothing and obtain the clothing mask image data; S1-4: Use labelme tool to analyze clothing videos The data is respectively cut out for the foreground clothing and marked with 12 key nodes of the model's limbs. The 12 key nodes include 8 key points of the limbs and 4 key points of the trunk. Figure 2 As shown, get the clothing video mask Data and motion posture key points data, where The data format is defined as: (1); in, are the key point coordinates, It is a category among the 12 key points of the limbs, including trunk, left hand, right hand, left leg, and right leg; S1-5: Dataset It is divided into training set and test set in an 8:2 ratio.

[0029] S2. Build a clothing encoding and decoding model and a posture estimation model. The specific processing steps are as follows: S2-1: Build a clothing encoding model based on the Transformer architecture and build the image encoding part, such as Figure 3 As shown in the figure, the clothing image and clothing mask image are patch sliced separately, and then patch encoding and position encoding are performed, and then concat features are merged. Four Transformer Encodes are used to build the image encoder core. Transformer Encode is constructed using the self-attention mechanism, and finally the latent space feature vector of the image is obtained. , Represents the vector dimension.

[0030] S2-2: Build a clothing decoding model, such as Figure 4 As shown, four Transformer Decoders are used to build the image decoder core. Transformer Decode is built using a cross-attention mechanism to transform the latent space feature vector Decoded and restored to clothing image.

[0031] S2-3: Encode-Decode link between clothing encoding model and clothing decoding model. The overall clothing encoding and decoding model is as follows: Figure 5 As shown in Figure 2, the features of the cross-attention mechanism at different levels are derived from the Transformer Encode at the same level, which enhances the model’s ability to learn complex postures and improves the accuracy and robustness of the model through multi-level features and relationships. Figure 5 The Encode-Decode link is the link between the first layer Transformer Encode of clothing encoding and the first layer Transformer Decode of clothing decoding, the second layer Transformer Encode of clothing encoding and the second layer Transformer Decode of clothing decoding, the third layer Transformer Encode of clothing encoding and the third layer Transformer Decode of clothing decoding, and the fourth layer Transformer Encode of clothing encoding and the fourth layer Transformer Decode of clothing decoding.

[0032] S2-4: Build a posture estimation model, such as Figure 6As shown in the figure, based on the YOLOv8 target detection architecture, the YOLOv8 target detection decoding module is replaced with a posture key point detection module. After processing by the key point detection module, the key point posture is merged with the original video data, that is, feature fusion is performed to obtain the estimated clothing posture video; The posture key point detection module includes two parts: key point coordinate feature decoding and key point category feature decoding. The key point coordinate feature decoding and key point category feature decoding are used to extract key point coordinate features and key point category features respectively.

[0033] S3: Train and test the clothing encoding and decoding model and the pose estimation model to verify the model accuracy.

[0034] Preferably, the S3 specifically includes: S3-1: The training set in the data is fed into the clothing encoding and decoding model. The training method adopts warmup as the learning rate change strategy, and the training is repeated for 50 rounds. The clothing encoding and decoding loss adopts the weighted fusion cross entropy (CE) loss function and the Dice loss function to supervise the fine-tuning process.

[0035] Clothing encoding and decoding loss function: (2); In formula (2), CE and Dice represent the cross entropy loss function and Dice loss function respectively. is the input image, The output image is obtained by the clothing encoding and decoding model, while ensuring that the input and output resolutions are the same. Represents the weight of the two loss functions to balance the influence between the two loss terms. This parameter is learnable. The test set in the data is fed into the model to verify the accuracy of the image encoding and decoding task.

[0036] S3-2: The training set in the data is fed into the pose estimation model, and the learning rate change strategy adopts warmup for 50 rounds of training. The loss adopts the root mean square error loss function, as shown in formula (3): (3); In formula (3), is the number of key points, are the real key point coordinates, is the predicted key point coordinate, and then The test set in the data is fed into the model to verify the accuracy of the key point detection task.

[0037] S4: Build a video dressing model. The specific processing steps include: S4-1: Build a video dressing model, including clothing encoding and pose estimation, such as Figure 7 As shown, based on the obtained clothing encoding and decoding model of the clothing encoding part and the posture estimation model of the posture estimation part, the weight parameters in the two models are frozen; The clothing image Data input clothing encoding part: First, the clothing image is input into SAM2 to obtain the clothing mask image. After concat the clothing mask image obtained by SAM2 and the original clothing image, it is transmitted to the clothing encoding model to obtain the clothing encoding vector; Input the clothing video data into the pose estimation part: First, the clothing video is fed into SAM2 and the pose estimation model respectively to obtain the clothing mask video and dressing posture video , the dressing mask video and the dressing posture video are superimposed and merged to obtain the dressing posture vector with the character mask.

[0038] S4-2: Add noise to the dress posture vector obtained in S4-1 to obtain a noisy dress posture vector. The noise needs to conform to the Gaussian distribution and is used to construct the data required for the diffusion model. The noisy dress posture vector is then fed into Unet for training to predict the noise distribution. At the same time, the clothing encoding vector is fed into Unet for training.

[0039] S4-3: Transfer the image latent space vector obtained by Unet training to the clothing decoding model to decode the final clothing change video.

[0040] S5: Train and test the video dressing model to verify the model accuracy.

[0041] S5-1: The predicted dressing-up video is compared with the real dressing-up video. The weighted KL divergence and MSE are used as loss functions to measure the difference between the generated sample distribution and the real data distribution. The loss function of the video dressing-up is defined as formula (4): (4); In formula (4), is the real sample data, To generate sample data, is the total amount of sample data, is the sample number, , is a dynamic weight.

[0042] S5-2: Only train the Unet model part, use the weighted KL divergence and MSE in S5-1 as the loss function, feed the training set into the model, use warmup as the learning rate change strategy, train for 100 rounds, feed the test set into the model to verify the accuracy of the video dressing model. The final effect is as follows Figure 8 As shown: This embodiment extracts key point information of the target person through precise posture estimation, and adaptively adjusts the dressing objects based on this information so that the dressing objects can be accurately combined with the person's posture. It uses a diffusion model for image generation to ensure a natural and smooth dressing effect, avoiding the problems of object incoordination or distortion in traditional dressing methods. On the basis of ensuring the accuracy of the person's posture estimation, it automatically adjusts the matching degree between the dressing objects and the person's posture.

[0043] In this way, diverse dressing effects are achieved, and the realism and visual expressiveness of dressing are enhanced, making video dressing technology more suitable for actual application scenarios and improving user experience.

[0044] Example 2 This embodiment provides a device for implementing a video dressing method based on a diffusion model and posture estimation, the device comprising: at least one processor; and A memory storing instructions, which, when executed by the at least one processor, causes the at least one processor to execute the video dressing method based on the diffusion model and posture estimation as described above.

[0045] In this embodiment, electronic devices include but are not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile computing devices, smart phones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable computing devices, consumer electronic devices, etc.

[0046] Example 3 This embodiment also provides a computer-readable storage medium storing executable instructions, which, when executed, enable the machine to perform the above-mentioned video dressing method based on diffusion model and posture estimation.

[0047] Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can read and execute instructions stored in the readable storage medium.

[0048] In this case, the program code itself read from the computer-readable medium can realize the function of any one of the above embodiments, and thus the computer-readable code and the computer-readable storage medium storing the computer-readable code constitute part of this specification.

[0049] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0050] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0051] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0052] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0053] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0054] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A video dressing method based on diffusion model and posture estimation, characterized in that: The method comprises: S1, obtain clothing images and videos of models with different body shapes wearing clothing; S2. Build a clothing encoding and decoding model and a posture estimation model; the clothing encoding and decoding model includes a clothing encoding model and a clothing decoding model, which are respectively used to extract the latent space feature vector of the image and restore the latent space feature vector of the image; the posture estimation model is used to fuse the posture key points with the original video data to obtain a posture estimation video; S3, train and test the clothing encoding and decoding model and posture estimation model to verify the model accuracy; S4. Build a video dressing model, which integrates the clothing encoding and decoding model and the posture estimation model; S5. Use weighted KL divergence and MSE as loss functions to train the video dressing model, and test the video dressing model to verify the model accuracy.

2. The video dressing method based on diffusion model and posture estimation according to claim 1, characterized in that: Said S1 specifically includes: S1-1: Capture image data of different garments using a camera to obtain diverse garment styles and detailed features, and obtain garment images data; S1-2: For dynamic clothing scenes, capture dynamic clothing video data containing different movements to obtain dressing videos data; S1-3: Use labelme tool to label clothing images The data is used to cut out the foreground clothing and obtain the clothing mask image data; S1-4: Use labelme tool to The data is cut out for the foreground clothing and marked with 12 key nodes of the model's limbs to obtain the clothing video mask Data and motion posture key points data, where The data format is defined as: （1）； In formula (1), are the key point coordinates, For each frame in the video, It is one of the 12 key points of the limbs; S1-5: Dataset It is divided into training set and test set in an 8:2 ratio.

3. The video dressing method based on diffusion model and posture estimation according to claim 2 is characterized in that: The construction of the clothing encoding and decoding model and the posture estimation model specifically includes: S2-1: Build a clothing encoding model based on the Transformer architecture, build the image encoding part, perform patch slicing on clothing images and clothing mask images, then perform patch encoding and position encoding, and then perform concat feature merging. Use 4 Transformer Encoders to build the image encoder core. Transformer Encode uses the self-attention mechanism to build, and finally obtain the latent space feature vector of the image. , Represents vector dimension; S2-2: Build a clothing decoding model, using 4 Transformer Decoders to build the image decoder core. Transformer Decode uses a cross-attention mechanism to build the latent space feature vector Decoding and restoring to clothing images; S2-3: The clothing encoding model and clothing decoding model are linked in Encode-Decode mode. The features of the cross-attention mechanism at different levels are derived from the Transformer Encode at the same level. S2-4: Build a pose estimation model. Based on the YOLOv8 target detection architecture, replace its target detection decoding module with a pose keypoint detection module. After processing by the keypoint detection module, it is fused with the original video data features to finally obtain the pose estimation video. The posture key point detection module includes two parts: key point coordinate feature decoding and key point category feature decoding. The key point coordinate feature decoding and key point category feature decoding are used to extract key point coordinate features and key point category features respectively.

4. The video dressing method based on diffusion model and posture estimation according to claim 3 is characterized in that: Training and testing clothing encoding and decoding models and pose estimation models specifically include: S3-1: The training set in the data is fed into the clothing encoding and decoding model. The training method adopts the warmup learning rate change strategy and trains for 50 rounds. The clothing encoding and decoding loss uses the weighted fusion cross entropy loss function and the Dice loss function to supervise the fine-tuning process. （2）； In formula (2), CE and Dice represent the cross entropy loss function and Dice loss function respectively. is the input image, is the output image obtained by the clothing encoding and decoding model; Represents the weights of the two loss functions; Then, The test set in the data is fed into the model to verify the accuracy of the image encoding and decoding task; S3-2: The training set in the data is fed into the pose estimation model, and the learning rate change strategy adopts warmup for 50 rounds of training. The loss uses the root mean square error loss function: （3）； In formula (3), is the number of key points, are the real key point coordinates, is the predicted key point coordinate; Then, The test set in the data is fed into the model to verify the accuracy of the key point detection task.

5. The video dressing method based on diffusion model and posture estimation according to claim 4 is characterized in that: Build a video dressing model, including: S4-1: Build a video dressing model, including clothing encoding and posture estimation. Based on the obtained clothing encoding and decoding model of the clothing encoding part and the posture estimation model of the posture estimation part, freeze the weight parameters in the two models. Clothing images Data input clothing encoding part: First, the clothing image is input into SAM2 to obtain the clothing mask image. After concat the clothing mask image obtained by SAM2 and the original clothing image, it is transmitted to the clothing encoding model to obtain the clothing encoding vector; Input the clothing video data into the pose estimation part: First, the clothing video is fed into SAM2 and the pose estimation model respectively to obtain the clothing mask video and dressing posture video , the dressing mask video and the dressing posture video are superimposed and merged to obtain the dressing posture vector with the character mask; S4-2: Add noise to the dress posture vector obtained in S4-1 to obtain a noisy dress posture vector. The noise needs to conform to the Gaussian distribution and is used to construct the data required for the diffusion model. The noisy dress posture vector is then fed into Unet for training to predict the noise distribution. At the same time, the clothing encoding vector is fed into Unet for training; S4-3: Transfer the image latent space vector obtained by Unet training to the clothing decoding model to decode the final clothing change video.

6. The video dressing method based on diffusion model and posture estimation according to claim 5, characterized in that: The S5 specifically includes: S5-1: Compare the predicted dressing videos with the real dressing videos. Use weighted KL divergence and MSE as loss functions to measure the difference between the generated sample distribution and the real data distribution. The loss function for the video dressing loss is defined as: （4）； In formula (4), is the real sample data, To generate sample data, is the total amount of sample data, is the sample number, , is the dynamic weight; S5-2: Only train the Unet model part, use the weighted KL divergence and MSE in S5-1 as the loss function, send the training set to the Unet model, use warmup as the learning rate change strategy, and send the test set to the model after 100 rounds of training to verify the accuracy of the video dressing model.

7. A device for a video dressing method based on a diffusion model and posture estimation, characterized in that: The device comprises: processor; a memory having stored thereon a computer program executable on the processor; Wherein, when the computer program is executed by the processor, the steps of the video dressing method based on diffusion model and posture estimation as claimed in any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images