Three-dimensional dynamic virtual fitting method based on motion driving

Abstract

The invention discloses a three-dimensional dynamic virtual try-on method based on motion driving, and the method comprises the steps: collecting a figure image, a clothes image and a motion posture sequence as input, and constructing a virtual try-on generation network; according to the network, firstly, multi-modal features are extracted through a feature extraction module, namely a figure encoder, a clothing encoder, a posture encoder and a CLIP model, and then semantics are aligned through a cross attention mechanism; the aligned features are input into a St-UNet network for multi-scale space-time modeling, a material stiffness extraction model is combined to realize material perception deformation, and finally, a dynamic fitting image is generated through a VAE decoder and a transposed convolutional network; a two-stage strategy is adopted in training, and a discriminator based on WGAN-GP is introduced to carry out confrontation optimization. According to the invention, through joint optimization of motion perception deformation and material constraint, a high-fidelity, cross-frame consistent and physically accurate dynamic virtual fitting effect is realized.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and virtual try-on technology, and in particular to a motion-driven three-dimensional dynamic virtual try-on method. Background Technology

[0002] Virtual try-on technology, a core innovation integrating digital fashion with consumption scenarios, achieves real-time visualization of clothing in a virtual environment through the combination of computer vision and generative models. This technology has significantly reduced online shopping return rates (platforms such as ASOS and Zalando reported a 30% reduction in return rates) and has demonstrated enormous application potential in scenarios such as live-streaming e-commerce and the metaverse.

[0003] Traditional methods utilize 2D texture synthesis and pose alignment techniques (such as CP-VTON's thin-plate spline transform and ACGPN's attention mechanism) to achieve geometric adaptation between clothing and the human body in a single frame image. However, these methods cannot simulate the physical behavior of clothing during motion (such as fabric drape and collision response). Some virtual try-on methods attempt to address these limitations using temporal Transformers and physics-aware models, but they still face key challenges: predicting nonlinear fabric deformation under rapid motion is extremely difficult. The complex dynamic behaviors caused by such motion are hard to capture accurately by current models, resulting in the final try-on effect failing to meet expectations. Summary of the Invention

[0004] This invention provides a motion-driven three-dimensional dynamic virtual try-on method to overcome the above-mentioned technical problems.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows: A motion-driven three-dimensional dynamic virtual try-on method includes: S1: Collect images of people, clothing, and motion posture sequences as training data; S2: Construct a virtual try-on generation network to output corresponding 3D dynamic virtual try-on images. This virtual try-on generation network includes: The feature extraction and fusion backbone network is used to extract and fuse multimodal features from the input data and decode them to obtain the decoded features. It includes a person encoder, clothing encoder, pose encoder, CLIP model, cross attention mechanism, St-UNet network, material stiffness extraction model and VAE decoder. The virtual try-on image generator, composed of a transposed convolutional network, is used to transpose and convolve the decoded features to output a three-dimensional dynamic virtual try-on image. The virtual try-on image discriminator, built on the WGAN-GP architecture, is used to determine whether the input image is real data or fake data generated by the virtual try-on image generator. The person encoder, clothing encoder, and posture encoder are used to encode the input person image, clothing image, and motion posture sequence, respectively, and output the initial human body features, initial clothing features, and initial posture codes. The CLIP model is used to extract initial semantic features from the input images of people and clothing. The cross-attention mechanism is used to perform cross-modal semantic alignment of the input initial character features, initial clothing features, initial pose encoding, and initial semantic features to obtain aligned features; The St-UNet network is used to upsample and downsample the initial pose encoding, aligned features and initial semantic features, and output the initial temporal coherence features. The material stiffness extraction model is used to process the initial clothing features and extract the initial material stiffness. The initial time coherence features are subjected to material-aware deformation based on the initial material stiffness to output the initial dynamic features. The VAE decoder is used to decode the initial dynamic features and output the decoded features; S3: A two-stage training strategy is used to train and optimize the virtual try-on generation network; S4: Using the optimized virtual try-on generation network, input new human images, clothing images and motion posture sequences, and output three-dimensional dynamic virtual try-on images.

[0006] Furthermore, the character encoder includes four sets of structurally identical dilated convolutional modules connected in sequence, wherein the second set of dilated convolutional modules and the fourth set of dilated convolutional modules are connected in a skip connection. Each dilated convolutional module contains a dilated convolutional layer, a batch normalization layer, and a GELU activation function layer.

[0007] Furthermore, the garment encoder includes a 4-layer CNN network and a VAE encoder connected in sequence.

[0008] Furthermore, the attitude encoder includes a 3D convolutional layer, a temporal residual block, and a max pooling layer connected in sequence; The time residual block is a residual block in the TCN network.

[0009] Furthermore, the St-UNet network includes a CA-Down-2 module, an SE module, a second spatial attention module, a cross attention module, a CA-Up-2 module, and a temporal attention module connected in sequence; The CA-Down-2 module includes a first dilated convolutional layer, a first spatial attention module, and a second dilated convolutional layer connected in sequence. The CA-Up-2 module includes a first transposed convolutional layer and a second transposed convolutional layer connected in sequence.

[0010] Furthermore, the St-UNet network is used to upsample and downsample the initial pose encoding, aligned features, and initial semantic features, outputting initial temporal coherence features, including: The aligned features and initial pose encoding are input into the CA-Down-2 module for downsampling, and the output motion features include: The aligned features are input into the first dilated convolutional layer for the first downsampling, and the first-scale feature map is output. The first-scale feature map and pose code are input into the first spatial attention module. The first spatial attention module processes the initial pose code to obtain a spatial mask. The spatial mask is then multiplied element-wise with the first-scale feature map to obtain the first-scale feature map of the focused motion region. The first-scale feature map of the focused motion region is input into the second dilated convolutional layer for a second downsampling to obtain the second-scale feature map; The second-scale feature map is input into the SE module for channel weighting to obtain the weighted features. The weighted features and the initial pose code are input into the second spatial attention module to upsample the initial pose code to obtain upsampled pose features. The upsampled pose features are concatenated with the weighted features. The concatenated features are then subjected to dimensionality reduction and linear transformation to generate a single-channel spatial mask. The single-channel spatial mask is then multiplied element-wise with the weighted features to obtain motion features. Motion features and initial semantic features are input into the cross-attention module for cross-modal semantic alignment, resulting in secondary aligned features. The features of the second alignment are input into the CA-Up-2 module, and the features of the second alignment are upsampled through two transposed convolutional layers to output the upsampled features. The upsampled features are input into the temporal attention module to encode the features and obtain the initial temporal coherent features.

[0011] Furthermore, the material stiffness extraction model comprises a three-layer MLP network connected in sequence.

[0012] Furthermore, the material-aware deformation operation is implemented by dynamically adjusting the upsampling convolution kernel, specifically including: The material stiffness is used to perform linear interpolation on the two pre-trained convolution kernels of different materials to generate dynamically adjusted upsampled convolution kernels, as shown in formula (1). (1) in, To simulate pre-trained convolutional kernels of rigid materials, Pre-trained convolutional kernels to simulate flexible materials; Material stiffness; Using the dynamically adjusted upsampling convolution kernel Upsampling is performed on the temporal coherence features to generate dynamic features that conform to the physical properties of the corresponding material; The dynamic characteristics must satisfy... Requirements Represents the pixel displacement vector. and This is a hyperparameter.

[0013] Furthermore, the two-stage training strategy includes: In the first stage, only the 4-layer CNN network in the clothing encoder is trained and the parameters of the modules other than the 4-layer CNN network are frozen. The 4-layer CNN network and the generator in the clothing encoder are optimized based on the adversarial loss function and backpropagation. The adversarial loss function is shown in Equation (2). (2) in, For mathematical expectation, For discriminator, For generator, This is an interpolation between real samples and generated samples. The gradient penalty coefficient is... These are the initial clothing features. For clothing images, For the discriminator in gradient at; The score output by the discriminator for real samples, i.e., real clothing images; The score given by the discriminator to the image output generated by the generator; In the second stage, all network modules are unfrozen, and a hybrid loss function is constructed. The generator and discriminator are jointly optimized through the hybrid loss function and backpropagation. The hybrid loss function is shown in formula (3). (3) in, These are the weighting coefficients. For the perceptual loss function, Let be the optical flow loss function, as shown in formula (4). (4) Among them, Flow( () indicates the calculation of optical flow. and These are real images and generated images, respectively.

[0014] Beneficial effects: This invention provides a motion-driven three-dimensional dynamic virtual try-on method, which has the following advantages: 1. This invention improves the effect of virtual try-on by jointly optimizing motion-sensing deformation and material constraints; 2. The St-Unet network constructed in this invention can achieve multi-scale feature sampling and accurately focus on motion regions, effectively model inter-frame motion, complete material-aware deformation, and achieve cross-semantic alignment of human body, clothing and posture by using CLIP-guided cross-attention mechanism, thereby generating high-fidelity video try-on effect. 3. This invention performs excellently in model training and evaluation, highlighting its superiority in cross-frame consistency, accuracy of physical simulation, and adaptability to complex motion. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart of a motion-driven three-dimensional dynamic virtual try-on method provided by the present invention; Figure 2 This is a model architecture diagram of the present invention; Figure 3 This is a diagram of the architecture of the character encoder. Figure 4 Architecture diagram for the dilated convolutional module; Figure 5 This is a diagram of the St-UNet network architecture. Figure 6 This is an architecture diagram of the CA-Down-2 module in the St-UNet network; Figure 7 This is an architecture diagram of the CA-Up-2 module in the St-UNet network; Figure 8 This is a schematic diagram of the final virtual dressing result. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] This embodiment provides a motion-driven three-dimensional dynamic virtual try-on method, such as... Figure 1 As shown, it includes: S1: Collect images of people, clothing, and motion posture sequences as training data; S2: Construct a virtual try-on generation network to output corresponding 3D dynamic virtual try-on images. This virtual try-on generation network includes: The feature extraction and fusion backbone network is used to extract and fuse multimodal features from the input data and decode them to obtain the decoded features. It includes a person encoder, clothing encoder, pose encoder, CLIP model, cross attention mechanism, St-UNet network, material stiffness extraction model and VAE decoder. The virtual try-on image generator, composed of a transposed convolutional network, is used to transpose and convolve the decoded features to output a three-dimensional dynamic virtual try-on image. The virtual try-on image discriminator, built on the WGAN-GP architecture, is used to determine whether the input image is real data or fake data generated by the virtual try-on image generator. The person encoder, clothing encoder, and posture encoder are used to encode the input person image, clothing image, and motion posture sequence, respectively, and output the initial human body features, initial clothing features, and initial posture codes. The CLIP model is used to extract initial semantic features from the input images of people and clothing. The cross-attention mechanism is used to perform cross-modal semantic alignment of the input initial character features, initial clothing features, initial pose encoding, and initial semantic features to obtain aligned features; The St-UNet network is used to upsample and downsample the initial pose encoding, aligned features and initial semantic features, and output the initial temporal coherence features. The material stiffness extraction model is used to process the initial clothing features and extract the initial material stiffness. The initial time coherence features are subjected to material-aware deformation based on the initial material stiffness to output the initial dynamic features. The VAE decoder is used to decode the initial dynamic features and output the decoded features; S3: A two-stage training strategy is used to train and optimize the virtual try-on generation network; S4: Using the optimized virtual try-on generation network, input new human images, clothing images and motion posture sequences, and output three-dimensional dynamic virtual try-on images.

[0019] Specifically, such as Figure 2 As shown, a virtual try-on generation network is first constructed, which includes a feature extraction and fusion backbone network, a virtual try-on image generator, and a virtual try-on image discriminator. The feature extraction and fusion backbone network includes a person encoder, a clothing encoder, a pose encoder, a CLIP model, a cross-attention mechanism, a St-UNet network, a material stiffness extraction model, and a VAE decoder. The virtual try-on image generator is composed of a transposed convolutional network, which is used to transpose the decoded features and output a 3D dynamic virtual try-on image. The virtual try-on image discriminator, which is built based on the WGAN-GP architecture, is used to determine whether the input image is real data or fake data generated by the virtual try-on image generator. During training, images of people are collected and input into the human encoder to obtain human body features, images of clothing are collected and input into the clothing encoder to obtain clothing features, and the pose sequence of the human body is collected and input into the pose encoder to obtain pose encoding. The images of people and clothing are input into the CLIP model to output semantic features. The four generated features are aligned across modalities through a cross-attention mechanism to obtain aligned features. The aligned features are input into the St-Unet network designed in this invention to achieve multi-scale feature sampling and precise focus on the motion region, effectively model the inter-frame motion, and combine the material stiffness extraction model to perform material-aware deformation on the features output by the St-UNet network. Based on the perceptually deformed features, the virtual fitting image is generated by the VAE encoder and generator. During parameter optimization, when optimizing the parameters of the clothing encoder, the virtual fitting image is input into a 4-layer CNN network to complete the adversarial loop closure. The discriminator judges the score of the output image until the score of the generator output image is consistent with the score of the real image or the error is minimized, thus completing the training and optimization. After optimization, the generator directly uses the trained parameters to generate a high-fidelity 3D fitting effect image based on the input clothing, person and pose.

[0020] In a specific embodiment, the scheme for collecting images of people, clothing, and sequences of motion postures as training data is as follows: This approach constructs a high-quality, multimodal pairing training dataset. The construction of this dataset includes the following steps: 1. Image Acquisition and Preprocessing Data source: Public dataset: We use publicly available datasets containing a rich diversity of human figures; in this approach, we use the UBCFashion dataset. Self-built dataset: In a controlled environment, images of models with different body types (covering sizes S / M / L, etc.), skin tones, hairstyles, and poses are captured using a high-resolution camera to ensure the model's generalization ability.

[0021] Preprocessing flow: Normalized cropping: Use human detection algorithms (such as YOLO, Mask R-CNN) to automatically detect and crop out the human body area and remove irrelevant background; Uniform resolution: Scale all cropped human images to a uniform fixed resolution, such as 256×256 pixels, to meet the input requirements of the subsequent encoder network.

[0022] Pixel value normalization: Normalizes the pixel values ​​of the image from the integer range of [0,255] to the interval of [-1,1] to accelerate the convergence process of model training.

[0023] 2. Acquisition and Preprocessing of Clothing Images Data source: E-commerce platform images: Obtain high-resolution product display images from partner e-commerce platforms (such as Zalando, ASOS) or publicly available clothing datasets, covering at least 20 categories such as T-shirts, dresses, coats, and pants.

[0024] Dynamic video dataset: We use video datasets containing dynamic displays of clothing, such as TikTokDress, and extract single frames from them as the source of clothing images.

[0025] Preprocessing flow: Background purification and contour extraction: The product image is preprocessed, and Gaussian blur is used to remove watermarks and complex backgrounds. Image segmentation technology (such as U-Net) is used to extract a clean clothing foreground mask to ensure the integrity of the contours.

[0026] Standardization: The clothing image is centered on the canvas and scaled to 256×256 pixels. For non-square images, edge fill or smart cropping is used to ensure the main body of the clothing remains undistorted.

[0027] Paired labeling: Each clothing image is labeled with its corresponding person image identifier and pose sequence identifier, forming a training sample of (person, clothing, pose sequence) triples.

[0028] 3. Acquisition and preprocessing of motion posture sequences Data source: Action recognition dataset: A large-scale video action dataset, such as Kinetics-400, is used. Pre-trained pose estimation algorithms (such as OpenPose and AlphaPose) are used to extract the two-dimensional coordinates (x, y) of 18 key joints of the human body (such as nose, left and right shoulders, left and right hips, etc.) from the video frame by frame.

[0029] Professional motion capture equipment: Using optical motion capture systems such as OptiTrack, the three-dimensional motion trajectory of human joints is directly and accurately captured, and then projected onto a two-dimensional plane for use.

[0030] Preprocessing flow: Sequence normalization: The length of the motion sequence is fixed, and all pose sequences are standardized to 24 frames, corresponding to 1 second of video (at a frame rate of 24fps). For sequences that are too long, sliding window sampling is used, and for sequences that are too short, cyclic padding is used.

[0031] 4. Reference Images and CLIP Feature Extraction: A set of high-quality "person-clothing" paired images is randomly selected or specified from the training samples as reference images. Before training, the deep semantic features QCLIP of these reference images are extracted offline using the image encoder of the CLIP model and stored for later use. During training, these features are input into the "cross-attention mechanism" module in the flowchart to guide cross-modal alignment.

[0032] In a specific embodiment, the scheme for constructing a virtual try-on generation network to output corresponding 3D dynamic virtual try-on images is as follows: The virtual try-on generation network includes: The feature extraction and fusion backbone network is used to extract and fuse multimodal features from the input data and decode them to obtain the decoded features. It includes a person encoder, clothing encoder, pose encoder, CLIP model, cross attention mechanism, St-UNet network, material stiffness extraction model and VAE decoder. The person encoder, clothing encoder, and pose encoder are used to encode the input person image, clothing image, and motion pose sequence, respectively, and output initial human features, initial clothing features, and initial pose codes, including: like Figure 3 As shown, the character encoder includes four groups of identical dilated convolutional modules connected in sequence, wherein the second group of dilated convolutional modules and the fourth group of dilated convolutional modules are connected in a skip connection. like Figure 4 As shown, each dilated convolutional module contains a dilated convolutional layer, a batch normalization layer, and a GELU activation function layer; Specifically, the first dilated convolution module has a 3×3 kernel in its dilated convolutional layer with a dilation rate of 2, expanding the 3 channels to 64 channels; this can expand the receptive field without increasing the number of parameters and preserve more local details. The second dilated convolutional module has a 3×3 kernel in its dilated convolutional layer, an inflation rate of 2, maintains 64 channels, and employs skip connections. The third and fourth dilated convolutional modules have the same structure as the first and second dilated convolutional modules. The output of the fourth dilated convolutional module is concatenated with the output of the second dilated convolutional module to obtain the initial human features. ; The garment encoder comprises a 4-layer CNN network and a VAE encoder connected in sequence; Specifically, the kernel size of each layer is 4×4, the stride is 2, the activation function is LeakyReLU, a small number of negative values ​​are allowed to pass through, noise is added during training to prevent the model from overfitting and enhance generalization ability; The clothing image is input into a 4-layer CNN network for feature extraction, and then the extracted features are encoded by a VAE encoder to obtain the initial clothing features. The attitude encoder includes a 3D convolutional layer, a temporal residual block, and a max pooling layer connected in sequence. The time residual block is a residual block in the TCN network; Specifically, the kernel of the 3D convolutional layer is 3×3×3, and the kernel of the max pooling layer is 1×3×3; The input is a human pose map with dimensions of 24×64×64×18 (time×height×width×number of joints). The number of channels is expanded from 18 to 64 through convolution in the 3D convolutional layer. Normalization and activation are performed through BN layer and ReLU activation layer. The output is a feature with dimensions of 24×64×64×64. The features are input into the temporal residual block, and the number of frames is reduced from 24 to 8 through a max pooling layer to output the initial pose encoding. , used to subsequently drive garment deformation; The CLIP model is used to extract initial semantic features from the input images of people and clothing. In this solution, dilated convolutional layers, CNN networks, VAE encoders, 3D convolutional layers, temporal residual blocks, and max pooling layers are all commonly used neural network models, and those skilled in the art are familiar with the specific processing procedures and output results. The cross-attention mechanism is used to perform cross-modal semantic alignment of the input initial character features, initial clothing features, initial pose encoding, and initial semantic features to obtain aligned features, including: Utilizing CLIP's semantic features and , Cross-attention computation is performed to achieve cross-modal alignment, ensuring that the semantics are matched when the model produces images; Specifically, the 512-dimensional semantic features extracted by CLIP The query vector Q is obtained by projecting it onto a 64-dimensional space using an MLP, and then the relationship between Q and human features is calculated. Clothing characteristics Attitude coding Cross attention score Where K and V are the key vector and value vector, respectively. For dimension parameters; The St-UNet network is used to upsample and downsample the initial pose encoding, aligned features, and initial semantic features, and output initial temporal coherence features, including: like Figure 5 As shown, the St-UNet network includes a CA-Down-2 module, an SE module, a second spatial attention module, a cross attention module, a CA-Up-2 module, and a temporal attention module connected in sequence. like Figure 6 As shown, the CA-Down-2 module includes a first dilated convolutional layer, a first spatial attention module, and a second dilated convolutional layer connected in sequence. like Figure 7 As shown, the CA-Up-2 module includes a first transposed convolutional layer and a second transposed convolutional layer connected in sequence; The aligned features, initial pose encoding, and semantic features are input into the St-UNet network for training, including: The aligned features and initial pose encoding are input into the CA-Down-2 module for downsampling, and the output motion features include: The aligned features are input into the first dilated convolutional layer for the first downsampling, and the first-scale feature map is output. The first-scale feature map and pose code are input into the first spatial attention module. The first spatial attention module processes the initial pose code to obtain a spatial mask. The spatial mask is then multiplied element-wise with the first-scale feature map to obtain the first-scale feature map of the focused motion region. The first-scale feature map of the focused motion region is input into the second dilated convolutional layer for a second downsampling to obtain the second-scale feature map; The second-scale feature map is input into the SE module for channel weighting to obtain the weighted features. The weighted features and the initial pose code are input into the second spatial attention module to upsample the initial pose code to obtain upsampled pose features. The upsampled pose features are concatenated with the weighted features. The concatenated features are then subjected to dimensionality reduction and linear transformation to generate a single-channel spatial mask. The single-channel spatial mask is then multiplied element-wise with the weighted features to obtain motion features. Motion features and initial semantic features are input into the cross-attention module for cross-modal semantic alignment, resulting in secondary aligned features. The features of the second alignment are input into the CA-Up-2 module, and the features of the second alignment are upsampled through two transposed convolutional layers to output the upsampled features. The upsampled features are input into the temporal attention module to encode the features and obtain the initial temporal coherence features; Specifically, the CA-Down-2 module (cascaded dilated convolution downsampling) performs multi-scale downsampling on multimodal features to capture global structure and local details. The kernels of the first and second dilated convolutional layers are both 3×3, with an inflation rate of 2 and a stride of 2. First downsampling: The aligned features are input into the first dilated convolutional layer, expanding the number of channels from 64 to 128, and downsampling the output size from 64×64 to 32×32; The output features and pose features are input into the first spatial attention module, and a spatial mask M1 is generated based on the pose encoding. The spatial mask M1 is multiplied element-wise with the first scale feature map (128×32×32) after the first downsampling to obtain the first scale feature map of the focused motion region; Second downsampling: The focused first-scale feature map is input into the second layer 3×3 dilated convolution to complete the second downsampling and output the second-scale feature map (256 channels, 16×16). The second-scale feature map is input into the SE module, and channel attention weighting is performed on the second-scale feature map to obtain Fchannel (256×16×16). The Fchannel and pose encoding are input into the second spatial attention module: a bilinear upsampling operation is performed on the pose encoding to expand the spatial size to 16×16, resulting in upsampled pose features (32×16×16). The upsampled pose features are concatenated with the Fchannel in the channel dimension to form a 288-channel feature map (288×16×16). This map is then reduced to a single channel by a 3×3 convolutional layer, and a single-channel spatial mask M2 is generated by the Sigmoid activation function. The spatial mask M2 is then multiplied element-wise with the Fchannel to obtain the motion features Fattn (256×16×16) of the focused motion region. Motion features and semantic features are input into the cross-attention module for cross-modal semantic alignment, resulting in secondary aligned features; In the CA-Up-2 module, the kernels of the first and second transposed convolutional layers are both 4×4, and the stride is 2. First upsampling: Input the 256-dimensional feature map (16×16) output by cross-attention, perform two layers of transposed convolution, reduce the number of channels from 256 to 64, increase the output size from 16×16 to 64×64, and finally output a 64-dimensional feature map (64×64 spatial size).

[0033] The output feature map is input into the temporal attention module, which processes the feature maps of three consecutive frames. , Sine and cosine positional encoding is added, and inter-frame dependencies are calculated through a multi-head self-attention mechanism. The number of heads is set to 8, and finally a temporally coherent 64-dimensional feature map (64×64 spatial size) is obtained, which is the initial temporally coherent feature. The St-Unet network constructed in this scheme combines CA-Down-2 with spatial attention blocks to achieve multi-scale feature sampling and precise focus on motion regions. It also combines CA-Up-2 with temporal attention blocks to effectively model inter-frame motion. Furthermore, it utilizes attention mechanisms to achieve cross-semantic alignment of the human body, clothing, and pose, thereby generating high-fidelity video try-on effects. The material stiffness extraction model is used to process the initial clothing features and extract the initial material stiffness. The initial temporal coherence features are subjected to material-aware deformation based on the initial material stiffness to output initial dynamic features, including: The material stiffness extraction model comprises a three-layer MLP network connected in sequence; Specifically, clothing features Input to a 3-layer MLP network, to obtain from Predicting material stiffness ; The hidden layer dimensions of the MLP are 128, 64, and 32 respectively; The initial temporal coherence features are subjected to material-aware deformation based on the initial material stiffness to output initial dynamic features, including: The material-aware deformation operation is implemented by dynamically adjusting the upsampling convolution kernel, specifically including: The material stiffness is used to perform linear interpolation on the two pre-trained convolution kernels of different materials to generate dynamically adjusted upsampled convolution kernels, as shown in formula (5). (5) in, To simulate pre-trained convolutional kernels of rigid materials, Pre-trained convolutional kernels to simulate flexible materials; Material stiffness; Using the dynamically adjusted upsampling convolution kernel Upsampling is performed on the initial temporal coherence features to generate dynamic features that conform to the physical properties of the corresponding material; The dynamic characteristics must satisfy... Requirements Represents the pixel displacement vector. and For hyperparameters; Specifically, a physical simulation dataset containing parameters of rigid (e.g., denim) and flexible (e.g., silk) materials is constructed. Supervised learning is used to initialize and train a deformation network until it converges to a state capable of distinguishing and simulating extreme material deformations. This converged network is then introduced as pre-trained weights in the subsequent main network training. When the value is close to 1, the convolution kernel becomes more rigid; if When the kernel size is close to 0, the convolution kernel becomes more flexible. This allows the model to automatically generate reasonable wrinkles and dynamic effects based on the clothing material; The VAE decoder is used to decode the initial dynamic features and output the decoded features; The virtual try-on image generator, composed of a transposed convolutional network, is used to transpose and convolve the decoded features to output a three-dimensional dynamic virtual try-on image. Finally, the 64-dimensional feature map after material-aware deformation is processed by a VAE decoder and a transposed convolutional network to obtain the initial three-dimensional dynamic virtual fitting image; The virtual try-on image discriminator, built on the WGAN-GP architecture, is used to determine whether the input image is real data or fake data generated by the virtual try-on image generator.

[0034] In a specific embodiment, the scheme for training and optimizing the virtual try-on generation network using a two-stage training strategy is as follows: The two-stage training strategy includes: In the first stage, only the 4-layer CNN network in the clothing encoder is trained and the parameters of the modules other than the 4-layer CNN network are frozen. The clothing encoder and generator are then optimized based on the adversarial loss function and backpropagation. The adversarial loss function is shown in Equation (6). (6) in, For mathematical expectation, For discriminator, For generator, This is an interpolation between real samples and generated samples. The gradient penalty coefficient is used to force the discriminator to satisfy the "Lipschitz continuity" condition, ensuring training stability. In this scheme... , These are the initial clothing features. For clothing images, For the discriminator in gradient at; The score output by the discriminator for real samples, i.e., real clothing images; The score given by the discriminator to the image output generated by the generator; In the second stage, all network modules are unfrozen, and a hybrid loss function is constructed. The generator and discriminator are jointly optimized through the hybrid loss function and backpropagation. The hybrid loss function is shown in formula (7). (7) in, These are the weighting coefficients. For the perceptual loss function, Let be the optical flow loss function, as shown in formula (8). (8) Among them, Flow( () indicates the calculation of optical flow. and These are real images and generated images, respectively. During the optimization process, gradient accumulation is used to update the parameters after accumulating the gradients of multiple mini-batches, with the effective batch size set to 16.

[0035] In a specific embodiment, the scheme of using an optimized virtual try-on generation network, inputting new character images, clothing images, and motion posture sequences, and outputting three-dimensional dynamic virtual try-on images is as follows: The new human images, clothing images, and motion posture sequences are input into the optimized virtual try-on generation network, which outputs three-dimensional dynamic virtual try-on images.

[0036] This invention demonstrates excellent performance in model training and evaluation. It was trained and tested on the UBC Fashion and TikTokDress datasets. Experimental results show that compared to other models, this invention achieves significant advantages in FID, SSIM, LPIPS, and FVD metrics, highlighting its superiority in cross-frame consistency, accuracy of physical simulation, and adaptability to complex motion. The final fitting effect is as follows: Figure 8 As shown.

[0037] In summary, this invention achieves four-layer modeling: spatial, temporal, semantic, and physical. It uses spatial attention to locate motion regions, temporal attention to ensure inter-frame coherence, CLIP to guide semantic alignment, and material parameters to constrain physical realism. This enables multi-scale feature flow, downsampling to capture global and local features, and upsampling to restore details, adapting to the multi-resolution requirements of clothing dynamics. Through layered design, this structure systematically solves core problems such as inaccurate deformation, texture breakage, and semantic contradictions in dynamic clothing try-on, providing a general technical framework for high-fidelity virtual try-on.

[0038] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A motion-driven three-dimensional dynamic virtual try-on method, characterized in that, include: S1: Collect images of people, clothing, and motion posture sequences as training data; S2: Construct a virtual try-on generation network to output corresponding 3D dynamic virtual try-on images. This virtual try-on generation network includes: The feature extraction and fusion backbone network is used to extract and fuse multimodal features from the input data and decode them to obtain the decoded features. It includes a person encoder, clothing encoder, pose encoder, CLIP model, cross attention mechanism, St-UNet network, material stiffness extraction model and VAE decoder. The virtual try-on image generator, composed of a transposed convolutional network, is used to transpose and convolve the decoded features to output a three-dimensional dynamic virtual try-on image. The virtual try-on image discriminator, built on the WGAN-GP architecture, is used to determine whether the input image is real data or fake data generated by the virtual try-on image generator. The person encoder, clothing encoder, and posture encoder are used to encode the input person image, clothing image, and motion posture sequence, respectively, and output the initial human body features, initial clothing features, and initial posture codes. The CLIP model is used to extract initial semantic features from the input images of people and clothing. The cross-attention mechanism is used to perform cross-modal semantic alignment of the input initial character features, initial clothing features, initial pose encoding, and initial semantic features to obtain aligned features; The St-UNet network is used to upsample and downsample the initial pose encoding, aligned features and initial semantic features, and output the initial temporal coherence features. The material stiffness extraction model is used to process the initial clothing features and extract the initial material stiffness. The initial time coherence features are subjected to material-aware deformation based on the initial material stiffness to output the initial dynamic features. The VAE decoder is used to decode the initial dynamic features and output the decoded features; S3: A two-stage training strategy is used to train and optimize the virtual try-on generation network; S4: Using the optimized virtual try-on generation network, input new human images, clothing images and motion posture sequences, and output three-dimensional dynamic virtual try-on images.

2. The motion-driven three-dimensional dynamic virtual try-on method according to claim 1, characterized in that, The character encoder includes four groups of identical dilated convolutional modules connected in sequence, wherein the second group of dilated convolutional modules and the fourth group of dilated convolutional modules are connected in a skip connection. Each dilated convolutional module contains a dilated convolutional layer, a batch normalization layer, and a GELU activation function layer.

3. The motion-driven three-dimensional dynamic virtual try-on method according to claim 1, characterized in that, The garment encoder comprises a 4-layer CNN network and a VAE encoder connected in sequence.

4. The motion-driven three-dimensional dynamic virtual try-on method according to claim 1, characterized in that, The attitude encoder includes a 3D convolutional layer, a temporal residual block, and a max pooling layer connected in sequence. The time residual block is a residual block in the TCN network.

5. The motion-driven three-dimensional dynamic virtual try-on method according to claim 1, characterized in that, The St-UNet network includes a CA-Down-2 module, an SE module, a second spatial attention module, a cross attention module, a CA-Up-2 module, and a temporal attention module connected in sequence. The CA-Down-2 module includes a first dilated convolutional layer, a first spatial attention module, and a second dilated convolutional layer connected in sequence. The CA-Up-2 module includes a first transposed convolutional layer and a second transposed convolutional layer connected in sequence.

6. The motion-driven three-dimensional dynamic virtual try-on method according to claim 5, characterized in that, The St-UNet network is used to upsample and downsample the initial pose encoding, aligned features, and initial semantic features, outputting initial temporal coherence features, including: The aligned features and initial pose encoding are input into the CA-Down-2 module for downsampling, and the output motion features include: The aligned features are input into the first dilated convolutional layer for the first downsampling, and the first-scale feature map is output. The first-scale feature map and pose code are input into the first spatial attention module. The first spatial attention module processes the initial pose code to obtain a spatial mask. The spatial mask is then multiplied element-wise with the first-scale feature map to obtain the first-scale feature map of the focused motion region. The first-scale feature map of the focused motion region is input into the second dilated convolutional layer for a second downsampling to obtain the second-scale feature map; The second-scale feature map is input into the SE module for channel weighting to obtain the weighted features. The weighted features and the initial pose code are input into the second spatial attention module to upsample the initial pose code to obtain upsampled pose features. The upsampled pose features are concatenated with the weighted features. The concatenated features are then subjected to dimensionality reduction and linear transformation to generate a single-channel spatial mask. The single-channel spatial mask is then multiplied element-wise with the weighted features to obtain motion features. Motion features and initial semantic features are input into the cross-attention module for cross-modal semantic alignment, resulting in secondary aligned features. The features of the second alignment are input into the CA-Up-2 module, and the features of the second alignment are upsampled through two transposed convolutional layers to output the upsampled features. The upsampled features are input into the temporal attention module to encode the features and obtain the initial temporal coherent features.

7. The motion-driven three-dimensional dynamic virtual try-on method according to claim 1, characterized in that, The material stiffness extraction model consists of three sequentially connected MLP networks.

8. The motion-driven three-dimensional dynamic virtual try-on method according to claim 1, characterized in that, The material-aware deformation operation is implemented by dynamically adjusting the upsampling convolution kernel, specifically including: The material stiffness is used to perform linear interpolation on the two pre-trained convolution kernels of different materials to generate dynamically adjusted upsampled convolution kernels, as shown in formula (1). （1） in, To simulate pre-trained convolutional kernels of rigid materials, Pre-trained convolutional kernels to simulate flexible materials; Material stiffness; Using the dynamically adjusted upsampling convolution kernel Upsampling is performed on the temporal coherence features to generate dynamic features that conform to the physical properties of the corresponding material; The dynamic characteristics must satisfy... Requirements Represents the pixel displacement vector. and This is a hyperparameter.

9. The motion-driven three-dimensional dynamic virtual try-on method according to claim 1, characterized in that, The two-stage training strategy includes: In the first stage, only the 4-layer CNN network in the clothing encoder is trained and the parameters of the modules other than the 4-layer CNN network are frozen. The 4-layer CNN network and the generator in the clothing encoder are optimized based on the adversarial loss function and backpropagation. The adversarial loss function is shown in Equation (2). （2） in, For mathematical expectation, For discriminator, For generator, This is an interpolation between real samples and generated samples. The gradient penalty coefficient is... These are the initial clothing features. For clothing images, For the discriminator in gradient at; The score output by the discriminator for real samples, i.e., real clothing images; The score given by the discriminator to the image output generated by the generator; In the second stage, all network modules are unfrozen, and a hybrid loss function is constructed. The generator and discriminator are jointly optimized through the hybrid loss function and backpropagation. The hybrid loss function is shown in formula (3). （3） in, These are the weighting coefficients. For the perceptual loss function, Let be the optical flow loss function, as shown in formula (4). （4） Among them, Flow( () indicates the calculation of optical flow. and These are real images and generated images, respectively.

Images