A Method and System for Generating Multi-View Virtual Fitting Videos Based on Facial Rigid Transformation

Abstract

This invention discloses a method and system for generating multi-view virtual try-on videos based on facial rigid transformation, relating to the fields of generative artificial intelligence and computer vision. The virtual try-on video generation method based on explicit geometric constraints includes the following steps: S1, acquiring original try-on images and action cues and constructing a multi-source input dataset; S2, segmenting the original try-on images to obtain multi-view modeling, and generating multi-view facial images and clothing texture feature parameters; S3, parsing the action cues to generate a target pose sequence, and generating the first and last frame virtual try-on images; S4, performing pairing analysis on the first and last frame virtual try-on images to generate pose transition parameters; S5, performing dynamic texture restoration on the images to generate a transition frame image sequence; S6, generating the virtual try-on video output. This invention achieves accurate segmentation and multi-view modeling of the facial and clothing regions through image segmentation and rigid geometric transformation algorithms.

Description

Technical Field

[0001] This invention relates to the fields of generative artificial intelligence and computer vision, and more particularly to a method and system for generating multi-view virtual try-on videos based on facial rigidity transformation. Background Technology

[0002] With the rapid development of virtual try-on technology, especially in e-commerce, fashion, and virtual reality, the generation of virtual try-on videos has become an important part of user experience. Traditional virtual try-on technology is mostly based on the processing of static images, which involves taking photos of products or people and then using computer graphics technology to generate try-on effects.

[0003] Explicit geometric constraints are a common technique in computer graphics, computer vision, and machine learning. They are used to enforce certain geometric conditions to make the generated models or images more consistent with the geometric rules of the real world. Unlike implicit constraints, explicit geometric constraints are directly specified in the algorithm design through explicit mathematical relationships, thereby enhancing the accuracy and interpretability of the model. This makes explicit geometric constraints particularly important in tasks such as virtual try-on, image generation, and 3D modeling, especially in ensuring the consistency and naturalness between the generated images and the actual objects or scenes.

[0004] However, existing virtual try-on video generation methods and systems based on explicit geometric constraints suffer from disjointed movements during use. Furthermore, virtual try-on methods rely on 3D modeling or preset pose templates, making it difficult to flexibly generate facial views from different angles. Traditional Fill-Redux modules are not extended to the video domain and cannot handle dynamic textures. Additionally, existing virtual try-on video generation methods and systems rely on single-view image generation, making it impossible to directly expand multi-view content through text prompts. Methods relying on 3D modeling or physical simulation require significant resources and are difficult to deploy in real-time. Cross-view identity consistency, geometric coherence, and high-frequency detail fidelity remain challenges. Moreover, current mainstream solutions are all based on single-view image generation, lacking a constraint framework for multi-view geometric consistency. When attempting to expand the viewpoint through text prompts, issues such as blurred identity features and failure to reconstruct occluded areas frequently arise. Furthermore, noticeable jagged edges exist in texture mapping of dynamically deformed areas. These factors mean that existing methods still face significant challenges in key indicators such as cross-view geometric coherence and high-frequency detail fidelity.

[0005] No effective solutions have yet been proposed to address the problems in the relevant technologies. Summary of the Invention

[0006] To address the problems in related technologies, this invention proposes a multi-view virtual fitting video generation method and system based on facial rigidity transformation, in order to overcome the aforementioned technical problems existing in the existing related technologies.

[0007] To achieve the above objectives, the specific technical solution adopted by the present invention is as follows:

[0008] According to one aspect of the present invention, a method for generating multi-view virtual fitting videos based on facial rigidity transformation is provided, comprising the following steps:

[0009] S1. Obtain the original fitting images and action prompts, perform image standardization and text vectorization respectively, and construct a multi-source input dataset;

[0010] S2. Use the SAM model to segment the original fitting images in the multi-source input dataset, obtain the face region and clothing region in the image, extract the face mask of the face region, and then perform multi-view modeling of the face mask based on the rigid geometric transformation algorithm to generate multi-view face images. At the same time, extract the clothing texture feature parameters of the clothing region.

[0011] As a preferred embodiment, the step of segmenting the original fitting images in the multi-source input dataset using the SAM model to obtain the facial and clothing regions in the images, extracting the facial mask of the facial region, and then performing multi-view modeling of the facial mask based on the rigid geometric transformation algorithm to generate multi-view facial images, while simultaneously extracting the clothing texture feature parameters of the clothing region, includes the following steps:

[0012] S21. Use the SAM model to perform semantic segmentation on the original fitting images in the multi-source input dataset to obtain the facial region and clothing region in the image, and extract the facial mask of the facial region.

[0013] S22. A head transformation map is generated by performing a similarity transformation on the facial mask using a rigid geometric transformation algorithm, and then output as a multi-view facial image.

[0014] S23. Extract the clothing texture feature parameters of the clothing area, and unify the clothing texture parameters of multi-view facial images through a material-aware style transfer algorithm.

[0015] S3. Based on the CLIP model, analyze the action prompts, generate the target pose sequence, and construct a dual-view spatial topological prior framework. Then, fuse the multi-view facial images with the clothing texture feature parameters to generate the first and last frame virtual try-on images.

[0016] As a preferred embodiment, the process of parsing action cues based on the CLIP model, generating a target pose sequence, constructing a dual-view spatial topological prior framework, and fusing multi-view facial images with clothing texture feature parameters to generate the first and last frame virtual fitting images includes the following steps:

[0017] S31. Extract the semantic feature vector of action cue words through CLIP text encoder, and generate a continuous sequence of pose key point coordinates by combining it with a predefined pose dictionary;

[0018] S32. The original fitting image and the geometrically transformed facial image are stitched together to form a left and right double image. The spatial attention mechanism is used to achieve feature alignment, and the identity consistency is maintained through the SRIM mechanism.

[0019] As a preferred embodiment, the step of stitching the original fitting image and the geometrically transformed facial image into a left-right dual image, using a spatial attention mechanism to achieve feature alignment, and maintaining identity consistency through the SRIM mechanism includes the following steps:

[0020] S321. Using the global features of the original image as identity anchors, the identity key points in the two images are aligned through a feature matching algorithm, and the original fitting image and the geometrically transformed facial image are stitched together to form a left and right double image.

[0021] S322. Employ a spatial attention mechanism and introduce context-aware masks to maintain the continuity of texture and feature alignment between the clothing area and the original image.

[0022] S323. By optimizing the spatial alignment error of the dual-view spatial topology prior frame, interpolation distortion caused by pose transformation is suppressed, and identity consistency is maintained through the SRIM mechanism.

[0023] S33. Generate a high-fidelity image of the clothing area through a diffusion repair model, optimize details using a high-frequency texture preservation strategy, and then perform pixel-level fusion with the transformed facial area to obtain the first and last frame virtual fitting images.

[0024] As a preferred embodiment, the process of generating a high-fidelity image of the clothing area using a diffusion repair model, optimizing details with a high-frequency texture preservation strategy, and then performing pixel-level fusion with the transformed facial area to obtain the first and last frame virtual fitting images includes the following steps:

[0025] S331. Align the reference image with the target pose. After the pose is aligned, perform physical rationality calibration on the generated image to ensure the coupling relationship between the clothing and the human body pose.

[0026] As a preferred embodiment, the step of aligning the reference image with the target pose, and then performing physical plausibility calibration on the generated image after pose alignment to ensure the coupling relationship between the clothing and the human body pose includes the following steps:

[0027] Let the target attitude be The pose of the reference image is The orientation alignment process is represented as:

[0028] ;

[0029] in, This represents a geometric transformation function, implemented using rotation and scaling transformations;

[0030] The generated image undergoes physical plausibility calibration to ensure the coupling relationship between the clothing and the human body posture. This process can be expressed by the following formula:

[0031] ;

[0032] in, It is a physical rationality loss function. and These represent the mechanical constraints and deformation constraints of the i-th pixel, respectively.

[0033] The S332 and Flux models guide the generation process by incorporating textual prompts, while the Flux diffusion model guides the generation of a clear image from a noisy image step by step.

[0034] S333. When repairing details and local textures in clothing areas, Vista2Vogue uses a high-frequency texture preservation strategy to repair high-frequency textures and repairs high-frequency details in non-head areas through the Fill-Redux module, while maintaining consistency of details during posture changes.

[0035] As a preferred embodiment, when repairing details and local textures in the clothing area, Vista2Vogue employs a high-frequency texture preservation strategy to repair high-frequency textures and uses the Fill-Redux module to repair high-frequency details in non-head areas, while maintaining consistency of details during pose changes, including the following steps:

[0036] The S3331 and Fill modules fill in missing details using image inpainting algorithms and combine contextual information to ensure the naturalness of the restoration effect. The process of restoring high-frequency details is optimized using the following loss function:

[0037] ;

[0038] in, It is a loss in high-frequency detail recovery. Indicates the current pixel. The restored image, It is the true value of the target image;

[0039] The S3332 and Redux modules utilize style transfer techniques to uniformly optimize the textures of both the head and non-head regions. Their goal is to minimize texture distortion during pose transformations, achieved through the following formula:

[0040] ;

[0041] in, For texture optimization results, Represents the texture after pose transformation. Used as a reference texture.

[0042] S334 and ACE-LoRA use a low-rank adaptive optimization strategy to update some parameters of the model and adjust the image content during the generation process based on alignment information.

[0043] S335 and Vista2Vogue input the statically generated results into the WAN model and achieve smooth inter-frame transitions through spatiotemporal consistency constraints.

[0044] S4. Perform pairwise analysis on the virtual try-on images of the first and last frames, calculate the change vector between adjacent poses of the virtual try-on images of the first and last frames, and generate an inter-frame motion vector field. Use the motion vector field as input to inject into the diffusion model to generate pose transition parameters.

[0045] S5. Input the virtual try-on images of the first and last frames and the posture transition parameters into the WAN video generation model, perform dynamic texture restoration on the images, and generate a continuous sequence of transition frame images based on the first and last frames.

[0046] As a preferred embodiment, the step of inputting the virtual try-on images of the first and last frames and the posture transition parameters into the WAN video generation model, performing dynamic texture restoration on the images, and generating a continuous sequence of transition frame images based on the first and last frames includes the following steps:

[0047] S51. Load the virtual fitting images of the first and last frames according to the FLF2V model, then quantize them using FP16 and set the number of frames. At the same time, use the T5 text encoder to parse the motion guidance vector obtained from the posture transition parameters. The motion guidance vector is used to assist in controlling the dynamic strategy of generating inter-frame transitions, but is not used as an explicit input to the WAN model.

[0048] After receiving the virtual try-on images of the first and last frames output by Vista2Vogue, the S52 and WAN models rely on their built-in DiT spatiotemporal attention mechanism to achieve dynamic alignment and natural transition of clothing textures. At the same time, the WAN model implicitly models the coupling relationship between clothing and human posture changes through its VCU unit, and calculates the continuous change values ​​of fabric wrinkles and motion amplitude.

[0049] S53. Through its built-in frame interpolation module, it generates transition frames in a refined manner and outputs a continuous and natural sequence of transition frame images.

[0050] S54. Output the transition sequence verified by physical constraints, trigger regeneration for frames that fail to detect collisions, and obtain a continuous sequence of transition frame images.

[0051] S6. Generate a virtual try-on video based on the transition frame image sequence, and verify the output of the virtual try-on video.

[0052] According to another aspect of the present invention, a multi-view virtual fitting video generation system based on facial rigidity transformation is provided, the system comprising:

[0053] The data acquisition and processing module acquires the original fitting images and action prompts, performs image standardization and text vectorization processing respectively, and constructs a multi-source input dataset;

[0054] The image segmentation processing module uses the SAM model to segment the original fitting images in the multi-source input dataset, obtains the facial region and clothing region in the image, extracts the facial mask of the facial region, and then performs multi-view modeling of the facial mask based on the rigid geometric transformation algorithm to generate multi-view facial images. At the same time, it extracts the clothing texture feature parameters of the clothing region.

[0055] The virtual try-on image module uses the CLIP model to parse action prompts, generate target pose sequences, and constructs a dual-view space topological prior framework. It then fuses multi-view facial images with clothing texture feature parameters to generate the first and last frame virtual try-on images.

[0056] The image pose transition module performs paired analysis on the virtual try-on images of the first and last frames, calculates the change vector between adjacent poses of the virtual try-on images of the first and last frames, and generates an inter-frame motion vector field. The motion vector field is then injected into the diffusion model as input to generate pose transition parameters.

[0057] The transition image sequence module inputs the virtual try-on images of the first and last frames and the posture transition parameters into the WAN video generation model, performs dynamic texture restoration on the images, and generates a continuous transition frame image sequence based on the first and last frames.

[0058] The video verification output module generates a virtual try-on video based on the transition frame image sequence and verifies and outputs the virtual try-on video.

[0059] The beneficial effects of this invention are as follows:

[0060] 1. This invention achieves accurate segmentation and multi-view modeling of the facial and clothing regions by combining the SAM model with a rigid geometric transformation algorithm. This makes the interaction between clothing and the character more natural, especially under multi-view changes and complex postures, maintaining a realistic fit between the face and clothing. The Flux diffusion model is used to guide the generation process, gradually generating clear images. At the same time, Vista2Vogue is used to repair high-frequency textures, ensuring the consistency of texture details of clothing under different posture changes. In addition, the Fill-Redux module is used to successfully repair high-frequency details in non-head areas during posture changes, avoiding common texture distortion problems.

[0061] 2. This invention utilizes physical rationality calibration to ensure natural coupling between clothing and human body posture changes. It also avoids abnormal deformation by introducing deformation energy parameters and fabric dynamics models, enhancing the physical consistency of generated images to better reflect actual wearing effects in different postures. Furthermore, spatiotemporal consistency constraints ensure smooth frame transitions in virtual try-on videos, maintaining consistency across multiple postures and avoiding common frame jumps or abrupt changes, thus improving the smoothness and comfort of the user experience. Combined with clothing texture feature fusion using the CLIP model, it can generate highly customized virtual try-on videos based on different user action prompts and clothing needs. The fusion of these technologies achieves high-quality simulation effects in terms of visual quality and provides users with a more personalized virtual try-on experience.

[0062] 3. This invention utilizes the Vista2Vogue method to generate the first and last frames, and reconstructs the multi-view generation process using explicit geometric constraints. Its core idea is to construct a dual-view stitching reference map using rigid transformation of the head, then lock the identity anchor point with spatial topology prior and establish cross-view geometric association. At the same time, it achieves texture restoration in non-head areas through pixel-level loss and style transfer constraints, and adopts a low-rank adaptive strategy to efficiently align pose text semantics. Vista2Vogue outperforms advanced solutions in identity consistency, clothing structure restoration and multi-view coherence, while maintaining real-time efficiency, providing a deployable solution for high-fidelity virtual try-on. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a flowchart of a multi-view virtual fitting video generation method based on facial rigidity transformation according to an embodiment of the present invention;

[0065] Figure 2 This is a system block diagram of a multi-view virtual fitting video generation system based on facial rigidity transformation according to an embodiment of the present invention.

[0066] In the picture:

[0067] 1. Data acquisition and processing module; 2. Image segmentation and processing module; 3. Virtual try-on image module; 4. Image pose transition module; 5. Transition image sequence module; 6. Video verification output module. Detailed Implementation

[0068] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0069] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0070] According to embodiments of the present invention, a method and system for generating multi-view virtual fitting videos based on facial rigidity transformation are provided.

[0071] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. According to one embodiment of the present invention, such as... Figure 1 As shown, the virtual fitting video generation method based on explicit geometric constraints according to an embodiment of the present invention includes the following steps:

[0072] S1. Obtain the original fitting images and action prompts, perform image standardization and text vectorization respectively, and construct a multi-source input dataset;

[0073] S2. Use the SAM model to segment the original fitting images in the multi-source input dataset, obtain the face region and clothing region in the image, extract the face mask of the face region, and then perform multi-view modeling of the face mask based on the rigid geometric transformation algorithm to generate multi-view face images. At the same time, extract the clothing texture feature parameters of the clothing region.

[0074] As a preferred embodiment, the step of segmenting the original fitting images in the multi-source input dataset using the SAM model to obtain the facial and clothing regions in the images, extracting the facial mask of the facial region, and then performing multi-view modeling of the facial mask based on the rigid geometric transformation algorithm to generate multi-view facial images, while simultaneously extracting the clothing texture feature parameters of the clothing region, includes the following steps:

[0075] S21. Use the SAM model to perform semantic segmentation on the original fitting images in the multi-source input dataset to obtain the facial region and clothing region in the image, and extract the facial mask of the facial region.

[0076] S22. A head transformation map is generated by performing a similarity transformation on the facial mask using a rigid geometric transformation algorithm, and then output as a multi-view facial image.

[0077] Specifically, through rotating arrays and scaling factor Adjusting head posture and proportions, the formula is:

[0078] ;

[0079] Where the translation vector Driven by the key point coordinates of the target attitude.

[0080] S23. Extract the clothing texture feature parameters of the clothing area, and unify the clothing texture parameters of multi-view facial images through a material-aware style transfer algorithm.

[0081] Specifically, this process is iterative. First, the face mask is extracted. Then, a rigid transformation, scaling, and rotation are performed on the face mask to generate a virtual fitting image from a new perspective. The same operation is then performed to generate another virtual fitting image from a new perspective. Finally, the required number of images are generated based on the usage needs, and multi-view modeling is constructed based on the generated virtual fitting images.

[0082] S3. Based on the CLIP model, analyze the action prompts, generate the target pose sequence, and construct a dual-view spatial topological prior framework. Then, fuse the multi-view facial images with the clothing texture feature parameters to generate the first and last frame virtual try-on images.

[0083] As a preferred embodiment, the process of parsing action cues based on the CLIP model, generating a target pose sequence, constructing a dual-view spatial topological prior framework, and fusing multi-view facial images with clothing texture feature parameters to generate the first and last frame virtual fitting images includes the following steps:

[0084] S31. Extract the semantic feature vector of action cue words through CLIP text encoder, and generate a continuous sequence of pose key point coordinates by combining it with a predefined pose dictionary;

[0085] S32. The original fitting image and the geometrically transformed facial image are stitched together to form a left and right double image. The spatial attention mechanism is used to achieve feature alignment, and the identity consistency is maintained through the SRIM mechanism.

[0086] As a preferred embodiment, the step of stitching the original fitting image and the geometrically transformed facial image into a left-right dual image, using a spatial attention mechanism to achieve feature alignment, and maintaining identity consistency through the SRIM mechanism includes the following steps:

[0087] S321. Using the global features of the original image as identity anchors, the identity key points in the two images are aligned through a feature matching algorithm, and the original fitting image and the geometrically transformed facial image are stitched together to form a left and right double image.

[0088] S322. Employ a spatial attention mechanism and introduce context-aware masks to maintain the continuity of texture and feature alignment between the clothing area and the original image.

[0089] Specifically, the pixel-level differences in the background region are constrained by calculating the L1 loss function:

[0090] ;

[0091] in This is a binary mask outside the head region.

[0092] S323. By optimizing the spatial alignment error of the dual-view spatial topology prior frame, interpolation distortion caused by pose transformation is suppressed, and identity consistency is maintained through the SRIM mechanism.

[0093] S33. Generate a high-fidelity image of the clothing area through a diffusion repair model, optimize details using a high-frequency texture preservation strategy, and then perform pixel-level fusion with the transformed facial area to obtain the first and last frame virtual fitting images.

[0094] As a preferred embodiment, the process of generating a high-fidelity image of the clothing area using a diffusion repair model, optimizing details with a high-frequency texture preservation strategy, and then performing pixel-level fusion with the transformed facial area to obtain the first and last frame virtual fitting images includes the following steps:

[0095] S331. Align the reference image with the target pose. After the pose is aligned, perform physical rationality calibration on the generated image to ensure the coupling relationship between the clothing and the human body pose.

[0096] As a preferred embodiment, the step of aligning the reference image with the target pose, and then performing physical plausibility calibration on the generated image after pose alignment to ensure the coupling relationship between the clothing and the human body pose includes the following steps:

[0097] Let the target attitude be The pose of the reference image is The orientation alignment process is represented as:

[0098] ;

[0099] in, This represents a geometric transformation function, implemented using rotation and scaling transformations;

[0100] The generated image undergoes physical plausibility calibration to ensure the coupling relationship between the clothing and the human body posture. This process can be expressed by the following formula:

[0101] ;

[0102] in, It is a physical rationality loss function. and Let represent the mechanical constraints and deformation constraints of the i-th pixel, respectively, and the physical rationality of the image is corrected by minimizing this loss function.

[0103] The S332 and Flux models guide the generation process by incorporating textual prompts, while the Flux diffusion model guides the generation of a clear image from a noisy image step by step.

[0104] Specifically, by combining with text prompts, the Flux model can generate more expected details. For example, given a description of a model making a victory sign with his left hand, the Flux model will automatically adjust the hand pose and texture of the relevant areas based on this information to ensure consistency of details from different perspectives.

[0105] The Flux diffusion model generates a clear image by progressively guiding a noisy image. Its generation process is represented by the following equation:

[0106] ;

[0107] in, This represents the current image during the diffusion process. For noise amplitude, To describe in the given text Below, image The gradient information is used to progressively optimize the image, making it more consistent with the text description and the target pose.

[0108] S333. When repairing details and local textures in clothing areas, Vista2Vogue uses a high-frequency texture preservation strategy to repair high-frequency textures and repairs high-frequency details in non-head areas through the Fill-Redux module, while maintaining consistency of details during posture changes.

[0109] As a preferred embodiment, when repairing details and local textures in the clothing area, Vista2Vogue employs a high-frequency texture preservation strategy to repair high-frequency textures and uses the Fill-Redux module to repair high-frequency details in non-head areas, while maintaining consistency of details during pose changes, including the following steps:

[0110] The S3331 and Fill modules fill in missing details using image inpainting algorithms and combine contextual information to ensure the naturalness of the restoration effect. The process of restoring high-frequency details is optimized using the following loss function:

[0111] ;

[0112] in, It is a loss in high-frequency detail recovery. Indicates the current pixel. The restored image, It is the true value of the target image;

[0113] The S3332 and Redux modules utilize style transfer techniques to uniformly optimize the textures of both the head and non-head regions. Their goal is to minimize texture distortion during pose transformations, achieved through the following formula:

[0114] ;

[0115] in, For texture optimization results, Represents the texture after pose transformation. The reference texture is used, and the loss function guarantees texture coherence and detail consistency during the pose transformation process.

[0116] S334 and ACE-LoRA use a low-rank adaptive optimization strategy to update some parameters of the model and adjust the image content during the generation process based on alignment information.

[0117] Specifically, ACE-LoRA uses a low-rank adaptive optimization strategy, updating only some parameters of the model, thereby achieving efficient alignment with less computational resources. The optimization objective of ACE-LoRA is expressed as:

[0118] ;

[0119] in, This is the optimized weight matrix. It is a reference weight. This represents the Frobenius norm.

[0120] The ACE-LoRA technology quickly aligns the input pose and text description, and adjusts the image content during the generation process based on this alignment information. The alignment process is represented by the following formula:

[0121] ;

[0122] in, and The generated pose and the reference pose are respectively, and the loss function minimizes the difference between the generated image and the target pose.

[0123] S335 and Vista2Vogue input the statically generated results into the WAN model and achieve smooth inter-frame transitions through spatiotemporal consistency constraints.

[0124] As a preferred embodiment, Vista2Vogue inputs the statically generated results into the WAN model and achieves smooth inter-frame transitions through spatiotemporal consistency constraints, including the following steps:

[0125] In the process of generating the first and last frames of video in the Alibaba WAN model (such as Wan2.1-FLF2V-14B), the core technology relies on the following key technologies to achieve smooth inter-frame transitions:

[0126] Conditional control branches:

[0127] First and last frames as control conditions: The model takes the first and last frames provided by the user as input conditions and compresses them into a conditional latent representation using 3DVAE. This process implicitly extracts semantic features of the first and last frames (such as object position, style, scene, etc.) without the need to manually model motion trajectories.

[0128] Binary Mask: By introducing a binary mask (1 indicates a frame to be retained, 0 indicates a frame to be generated), the model explicitly knows which frames need to be fixed (first and last frames) and which need to be dynamically generated. This avoids explicit calculation of complex motion fields.

[0129] Semantic feature injection:

[0130] CLIP Encoder and Global Context: The model extracts semantic features from the first and last frames using the CLIP image encoder and converts them into global context using an MLP. This semantic information is injected into the cross-attention mechanism of the DiT model to ensure that the generation process always revolves around the semantic content of the first and last frames, thereby achieving consistency in style, content, and structure.

[0131] Dynamic constraints: The injection of semantic features implicitly constrains the dynamic change direction of the generated frames (e.g., the position of objects and the trend of actions in the first and last frames), avoiding the problem of needing to explicitly estimate motion trajectories in traditional methods.

[0132] Spatiotemporal modeling of potential space:

[0133] Causal 3DVAE: The WAN model employs a causal 3DVAE (Spatiotemporal Variational Autoencoder), where the encoder compresses the video into a latent space, and the decoder gradually reconstructs the spatiotemporal information. Since the latent space of the VAE implicitly contains spatiotemporal continuity, the model can achieve smooth inter-frame transitions without explicitly calculating dense fields.

[0134] Spatiotemporal modeling of the Diffusion model (DiT): The DiT architecture directly learns the spatiotemporal relationship between the first and last frames through the self-attention and cross-attention mechanisms in the Transformer block. The diffusion process gradually denoises and generates intermediate frames, implicitly solving the problem of motion continuity between frames.

[0135] Data-driven implicit learning:

[0136] Large-scale training data: The WAN model is trained on a high-quality dataset containing 1.5 billion videos and 10 billion images. During training, the model has learned rich spatiotemporal dynamic patterns (such as object motion patterns, light and shadow changes, etc.). Therefore, during inference, there is no need to additionally model motion trajectories, but rather to rely on data-driven implicit learning.

[0137] The four-step data cleaning process ensures that the model can learn natural inter-frame transition patterns from the data, rather than relying on manually designed motion modeling.

[0138] Efficiency optimization and hardware adaptation:

[0139] High-compression 3DVAE: The WAN2.2 version of 3DVAE achieves a temporal and spatial compression ratio of up to 4×16×16, significantly reducing video memory usage (only 22GB of video memory is needed to generate a 5-second 720P video). This efficient compression allows models to run on consumer-grade hardware without explicitly processing high-dimensional motion fields.

[0140] MoE Architecture and Parallel Strategy: The hybrid expert (MoE) architecture and the 2D context parallel strategy (RingAttention+Ulysses) further optimize computational efficiency and avoid the high computational cost of traditional motion modeling.

[0141] S4. Perform pairwise analysis on the virtual try-on images of the first and last frames, calculate the change vector between adjacent poses of the virtual try-on images of the first and last frames, and generate an inter-frame motion vector field. Use the motion vector field as input to inject into the diffusion model to generate pose transition parameters.

[0142] Specifically, the pose changes of adjacent frames are calculated to generate inter-frame motion vector fields. Combined with ACE-LoRA technology, the motion vector fields are injected into the diffusion model to guide the generation of inter-frame pose transitions.

[0143] Then, by optimizing the L1 loss function, its formula is:

[0144] ;

[0145] Furthermore, it corrects the physical consistency between clothing and the human body between frames (such as fabric tension and joint movement).

[0146] S5. Input the virtual try-on images of the first and last frames and the posture transition parameters into the WAN video generation model, perform dynamic texture restoration on the images, and generate a continuous sequence of transition frame images based on the first and last frames.

[0147] As a preferred embodiment, the step of inputting the virtual try-on images of the first and last frames and the posture transition parameters into the WAN video generation model, performing dynamic texture restoration on the images, and generating a continuous sequence of transition frame images based on the first and last frames includes the following steps:

[0148] S51. Load the virtual fitting images of the first and last frames according to the FLF2V model, then quantize them using FP16 and set the number of frames. At the same time, use the T5 text encoder to parse the motion guidance vector obtained from the posture transition parameters. The motion guidance vector is used to assist in controlling the dynamic strategy of generating inter-frame transitions, but is not used as an explicit input to the WAN model.

[0149] Specifically, the process of parsing posture transition parameters can also be guided by input prompts describing how the first and last frames change, such as a model in a white dress making a peace sign, then smiling and moving away from the camera to sit in a chair, and outputting the parameter changes of the action enhancement words.

[0150] After receiving the virtual try-on images of the first and last frames output by Vista2Vogue, the S52 and WAN models rely on their built-in DiT spatiotemporal attention mechanism to achieve dynamic alignment and natural transition of clothing textures. At the same time, the WAN model implicitly models the coupling relationship between clothing and human posture changes through its VCU unit, and calculates the continuous change values ​​of fabric wrinkles and motion amplitude.

[0151] Specifically, a dynamic texture restoration algorithm is applied to non-head areas (such as the main body of clothing), combined with inter-frame motion vector fields, to restore details such as fabric movement and wrinkle changes. The time-aware high-frequency detail recovery loss formula is as follows: Improved repair results.

[0152] S53 uses its built-in frame interpolation module to generate transition frames in a refined manner, outputting a continuous and natural sequence of transition frame images.

[0153] Specifically, style transfer technology is then used to uniformly optimize the textures of the head and clothing areas, ensuring style consistency across frames. The formula is as follows:

[0154] ;

[0155] S54. Output the transition sequence verified by physical constraints, trigger regeneration for frames that fail to detect collisions, and obtain a continuous sequence of transition frame images.

[0156] S6. Generate a virtual try-on video based on the transition frame image sequence, and verify the output of the virtual try-on video.

[0157] Specifically, only the low-rank parameters related to pose and motion in the model are updated to reduce computational cost. The formula is as follows:

[0158] ;

[0159] Efficient video generation can be achieved by processing time series data through recurrent neural networks or Transformer architectures.

[0160] The output video consists of a complete video sequence including the first frame (front view), the last frame (back / side view), and intermediate transition frames. Video quality is then evaluated using metrics such as KID, SSIM, and LPIPS, and the optical flow consistency loss formula is applied. Quantize inter-frame coherence.

[0161] Furthermore, the WAN model has a built-in quality assurance mechanism, which is executed during the generation process of the VCU unit in Wan2.1-VACE:

[0162] Inter-frame consistency check (spatiotemporal attention weight monitoring);

[0163] Physical rationality verification (implicit constraints of fabric dynamics);

[0164] Identity drift suppression (CLIP embedding space similarity calculation);

[0165] Adding an additional verification module is equivalent to performing a second verification on the WAN output.

[0166] According to another aspect of the invention, such as Figure 2 The system shown provides a multi-view virtual try-on video generation system based on facial rigidity transformation. The system includes:

[0167] Data acquisition and processing module 1 acquires the original fitting images and action prompts, performs image standardization and text vectorization processing respectively, and constructs a multi-source input dataset;

[0168] Image segmentation processing module 2 uses the SAM model to segment the original fitting images in the multi-source input dataset, obtains the facial region and clothing region in the image, extracts the facial mask of the facial region, and then performs multi-view modeling of the facial mask based on the rigid geometric transformation algorithm to generate multi-view facial images. At the same time, it extracts the clothing texture feature parameters of the clothing region.

[0169] Virtual try-on image module 3 parses action prompts based on CLIP model, generates target pose sequence, and constructs a dual-view space topological prior framework to fuse multi-view facial images with clothing texture feature parameters to generate first and last frame virtual try-on images;

[0170] Image pose transition module 4 performs paired analysis on the virtual try-on images of the first and last frames, calculates the change vector between adjacent poses of the virtual try-on images of the first and last frames, and generates an inter-frame motion vector field. The motion vector field is then injected into the diffusion model as input to generate pose transition parameters.

[0171] The transition image sequence module 5 inputs the virtual try-on images of the first and last frames and the posture transition parameters into the WAN video generation model, performs dynamic texture restoration on the images, and generates a continuous transition frame image sequence based on the first and last frames.

[0172] The video verification output module 6 generates a virtual try-on video based on the transition frame image sequence and verifies and outputs the virtual try-on video.

[0173] In summary, by utilizing the above-mentioned technical solutions of this invention, this invention achieves accurate segmentation and multi-view modeling of the facial and clothing regions by combining image segmentation with the SAM model and rigid geometric transformation algorithm. This makes the interaction between clothing and the character more natural, especially under multi-view changes and complex postures, maintaining a realistic fit between the face and clothing. Furthermore, the Flux diffusion model is used to guide the generation process, gradually generating clear images. At the same time, Vista2Vogue is used to repair high-frequency textures, ensuring the consistency of texture details of clothing under different posture changes. In addition, the Fill-Redux module is used to successfully repair high-frequency details in non-head areas during posture changes, avoiding common texture distortion problems.

[0174] Furthermore, this invention ensures natural coupling between clothing and human body posture changes by utilizing physical rationality calibration, avoids abnormal deformation by introducing deformation energy parameters and fabric dynamics models, and enhances the physical consistency of generated images, making them more consistent with actual wearing effects under different postures. At the same time, through spatiotemporal consistency constraints, it ensures smooth transitions between frames in virtual try-on videos, allowing the character to maintain consistency during transitions between multiple postures, avoiding common frame jumps or abruptness, and improving the smoothness and comfort of the user experience. In addition, by combining clothing texture feature fusion with the CLIP model, it can generate highly customized virtual try-on videos based on different users' action prompts and clothing needs. Through the integration of these technologies, a high-quality simulation effect is achieved in terms of visual effects, and users can be provided with a more personalized virtual try-on experience.

[0175] Furthermore, this invention utilizes the Vista2Vogue method to generate the first and last frames, and reconstructs the multi-view generation process using explicit geometric constraints. Its core idea is to construct a dual-view stitching reference map using rigid transformation of the head, then lock identity anchor points and establish cross-view geometric relationships using spatial topological priors. At the same time, it achieves texture restoration in non-head areas through pixel-level loss and style transfer constraints, and adopts a low-rank adaptive strategy to efficiently align pose text semantics. Vista2Vogue outperforms advanced solutions in terms of identity consistency, clothing structure restoration, and multi-view coherence, while maintaining real-time efficiency, providing a deployable solution for high-fidelity virtual try-on.

[0176] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating multi-view virtual try-on videos based on facial rigidity transformation, characterized in that, Includes the following steps: S1. Obtain the original fitting images and action prompts, perform image standardization and text vectorization respectively, and construct a multi-source input dataset; S2. Use the SAM model to segment the original fitting images in the multi-source input dataset, obtain the face region and clothing region in the image, extract the face mask of the face region, and then perform multi-view modeling of the face mask based on the rigid geometric transformation algorithm to generate multi-view face images. At the same time, extract the clothing texture feature parameters of the clothing region. S3. Based on the CLIP model, analyze the action prompts, generate the target pose sequence, and construct a dual-view spatial topological prior framework. Then, fuse the multi-view facial images with the clothing texture feature parameters to generate the first and last frame virtual try-on images. S4. Perform pairwise analysis on the virtual try-on images of the first and last frames, calculate the change vector between adjacent poses of the virtual try-on images of the first and last frames, and generate an inter-frame motion vector field. Use the motion vector field as input to inject into the diffusion model to generate pose transition parameters. S5. Input the virtual try-on images of the first and last frames and the posture transition parameters into the WAN video generation model, perform dynamic texture restoration on the images, and generate a continuous sequence of transition frame images based on the first and last frames. S6. Generate a virtual try-on video based on the transition frame image sequence, and verify and output the virtual try-on video; The process of parsing action cues based on the CLIP model, generating a target pose sequence, constructing a dual-view spatial topological prior framework, and fusing multi-view facial images with clothing texture feature parameters to generate the first and last frame virtual try-on images includes the following steps: S31. Extract the semantic feature vector of action cue words through CLIP text encoder, and generate a continuous sequence of pose key point coordinates by combining it with a predefined pose dictionary; S32. The original fitting image and the geometrically transformed facial image are stitched together to form a left and right double image. The spatial attention mechanism is used to achieve feature alignment, and the identity consistency is maintained through the SRIM mechanism. S33. Generate a high-fidelity image of the clothing area through a diffusion repair model, optimize details using a high-frequency texture preservation strategy, and then perform pixel-level fusion with the transformed facial area to obtain the first and last frame virtual fitting images. The process of stitching the original fitting image and the geometrically transformed facial image into a left-right dual image, using a spatial attention mechanism to achieve feature alignment, and maintaining identity consistency through the SRIM mechanism includes the following steps: S321. Using the global features of the original image as identity anchors, the identity key points in the two images are aligned through a feature matching algorithm, and the original fitting image and the geometrically transformed facial image are stitched together to form a left and right double image. S322. Employ a spatial attention mechanism and introduce context-aware masks to maintain the continuity of texture and feature alignment between the clothing area and the original image. S323. By optimizing the spatial alignment error of the dual-view spatial topology prior frame, interpolation distortion caused by pose transformation is suppressed, and identity consistency is maintained through the SRIM mechanism.

2. The multi-view virtual fitting video generation method based on facial rigidity transformation according to claim 1, characterized in that, The process of segmenting the original fitting images from the multi-source input dataset using the SAM model to obtain the facial and clothing regions in the images, extracting the facial mask for the facial region, and then performing multi-view modeling on the facial mask based on the rigid geometric transformation algorithm to generate multi-view facial images, while simultaneously extracting the clothing texture feature parameters of the clothing region, includes the following steps: S21. Use the SAM model to perform semantic segmentation on the original fitting images in the multi-source input dataset to obtain the facial region and clothing region in the image, and extract the facial mask of the facial region. S22. A head transformation map is generated by performing a similarity transformation on the facial mask using a rigid geometric transformation algorithm, and then output as a multi-view facial image. S23. Extract the clothing texture feature parameters of the clothing area, and unify the clothing texture parameters of multi-view facial images through a material-aware style transfer algorithm.

3. The multi-view virtual fitting video generation method based on facial rigidity transformation according to claim 1, characterized in that, The process of inputting the virtual try-on images of the first and last frames and the posture transition parameters into the WAN video generation model, performing dynamic texture restoration on the images, and generating a continuous sequence of transition frame images based on the first and last frames includes the following steps: S51. Load the virtual fitting images of the first and last frames according to the FLF2V model, then quantize them using FP16 and set the number of frames. At the same time, use the T5 text encoder to parse the motion guidance vector obtained from the posture transition parameters. The motion guidance vector is used to assist in controlling the dynamic strategy of generating inter-frame transitions, but is not used as an explicit input to the WAN model. After receiving the virtual try-on images of the first and last frames output by Vista2Vogue, the S52 and WAN models rely on their built-in DiT spatiotemporal attention mechanism to achieve dynamic alignment and natural transition of clothing textures. At the same time, the WAN model implicitly models the coupling relationship between clothing and human posture changes through its VCU unit, and calculates the continuous change values ​​of fabric wrinkles and motion amplitude. S53 uses its built-in frame interpolation module to generate transition frames in a refined manner, outputting a continuous and natural sequence of transition frame images.

4. The multi-view virtual fitting video generation method based on facial rigidity transformation according to claim 1, characterized in that, The process of generating a high-fidelity image of the clothing area using a diffusion repair model, optimizing details with a high-frequency texture preservation strategy, and then performing pixel-level fusion with the transformed facial area to obtain the first and last frame virtual fitting images includes the following steps: S331. Align the reference image with the target pose. After the pose is aligned, perform physical rationality calibration on the generated image to ensure the coupling relationship between the clothing and the human body pose. The S332 and Flux models guide the generation process by incorporating textual prompts, while the Flux diffusion model guides the generation of a clear image from a noisy image step by step. S333. When repairing details and local textures in clothing areas, Vista2Vogue uses a high-frequency texture preservation strategy to repair high-frequency textures and repairs high-frequency details in non-head areas through the Fill-Redux module, while maintaining consistency of details during posture changes. S334 and ACE-LoRA use a low-rank adaptive optimization strategy to update some parameters of the model and adjust the image content during the generation process based on alignment information to obtain static generation results. S335 and Vista2Vogue input the statically generated results into the WAN model and achieve smooth inter-frame transitions through spatiotemporal consistency constraints.

5. The multi-view virtual fitting video generation method based on facial rigidity transformation according to claim 4, characterized in that, The process of aligning the reference image with the target pose, followed by physical calibration of the generated image to ensure the coupling relationship between the clothing and the human body pose, includes the following steps: Let the target attitude be The pose of the reference image is The orientation alignment process is represented as: ； in, This represents a geometric transformation function, implemented using rotation and scaling transformations; The generated image undergoes physical plausibility calibration to ensure the coupling relationship between the clothing and the human body posture. This process can be expressed by the following formula: ； in, It is a physical rationality loss function. and These represent the mechanical constraints and deformation constraints of the i-th pixel, respectively.

6. The multi-view virtual fitting video generation method based on facial rigidity transformation according to claim 4, characterized in that, When repairing details and local textures in the clothing area, Vista2Vogue employs a high-frequency texture preservation strategy to repair high-frequency textures and uses the Fill-Redux module to repair high-frequency details in non-head areas, maintaining consistency of details during pose changes. This includes the following steps: The S3331 and Fill modules fill in missing details using image inpainting algorithms and combine contextual information to ensure the naturalness of the restoration effect. The process of restoring high-frequency details is optimized using the following loss function: ； in, It is a loss in high-frequency detail recovery. Indicates the current pixel. The restored image, It is the true value of the target image; The S3332 and Redux modules utilize style transfer techniques to uniformly optimize the textures of both the head and non-head regions. Their goal is to minimize texture distortion during pose transformations, achieved through the following formula: ； in, For texture optimization results, Represents the texture after pose transformation. Used as a reference texture.

7. A multi-view virtual fitting video generation system based on facial rigidity transformation is used to implement the multi-view virtual fitting video generation method based on facial rigidity transformation as described in any one of claims 1-6, characterized in that, The system includes: The data acquisition and processing module acquires the original fitting images and action prompts, performs image standardization and text vectorization processing respectively, and constructs a multi-source input dataset; The image segmentation processing module uses the SAM model to segment the original fitting images in the multi-source input dataset, obtains the facial region and clothing region in the image, extracts the facial mask of the facial region, and then performs multi-view modeling of the facial mask based on the rigid geometric transformation algorithm to generate multi-view facial images. At the same time, it extracts the clothing texture feature parameters of the clothing region. The virtual try-on image module uses the CLIP model to parse action prompts, generate target pose sequences, and constructs a dual-view space topological prior framework. It then fuses multi-view facial images with clothing texture feature parameters to generate the first and last frame virtual try-on images. The image pose transition module performs paired analysis on the virtual try-on images of the first and last frames, calculates the change vector between adjacent poses of the virtual try-on images of the first and last frames, and generates an inter-frame motion vector field. The motion vector field is then injected into the diffusion model as input to generate pose transition parameters. The transition image sequence module inputs the virtual try-on images of the first and last frames and the posture transition parameters into the WAN video generation model, performs dynamic texture restoration on the images, and generates a continuous transition frame image sequence based on the first and last frames. The video verification output module generates a virtual try-on video based on the transition frame image sequence and verifies and outputs the virtual try-on video.

Images