A four-dimensional virtual fitting method and system for digital humans

Abstract

The present application belongs to the technical field of computer vision and computer graphics, and discloses a four-dimensional virtual fitting method and system for digital people. The present application obtains a source person image sequence and a target clothing image; adopts a parallel double-branch structure to generate a first and a second view-pose independent Gaussian mapping under a standard pose; generates a Gaussian parameter offset based on a driving pose and a view angle through a shared nonlinear deformation module and superimposes the offset to the Gaussian mapping; corrects a two-dimensional virtual fitting sequence for time sequence consistency to obtain a correction supervision signal; and generates a four-dimensional virtual fitting result through a differentiable rendering based on the correction supervision signal and the superimposed offset Gaussian mapping. The present application significantly surpasses the prior art in terms of multi-view consistency, dynamic continuity, rendering quality, generation speed and the like. The method not only realizes high-fidelity dynamic virtual fitting, but also achieves leading performance in visual perception indicators, physical stability and cross-clothing generalization ability.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and computer graphics technology, and relates to a four-dimensional virtual try-on method and system for digital humans. Specifically, it relates to a generative artificial intelligence technology that uses a single clothing image, Gaussian mapping decomposition and nonlinear deformation modeling to achieve dynamic, high-fidelity virtual try-on. Background Technology

[0002] With the rapid development of artificial intelligence, virtual reality (VR), and augmented reality (AR) technologies, the demand for constructing virtual avatars (i.e., "digital humans") in digital space is constantly increasing. Digital humans are not only widely used in entertainment, games, and film production, but also demonstrate enormous commercial value in e-commerce, virtual try-on, social media, and remote interaction. Especially in the apparel industry, virtual try-on technology has become a crucial element in promoting digital transformation, enabling the visualization of clothing styles and interactive try-on experiences through the combination of computer vision and generative modeling.

[0003] Traditional virtual try-on technology can be broadly categorized into three stages: two-dimensional, three-dimensional, and four-dimensional. Early two-dimensional virtual try-on methods were mostly based on image generation networks, which could generate try-on results by inputting photos of people and images of clothing. These methods have advantages such as easy data acquisition and low implementation costs, but due to the lack of three-dimensional geometric perception, the generated results often suffer from perspective errors, misaligned clothing edges, and incorrect body occlusion relationships, making it difficult to realistically reflect the dynamics of clothing in three-dimensional space. Representative studies include models such as VITON, C-VTON, and IDM-VTON, which improve image quality by introducing pose estimation, local flow alignment, and diffusion models, but are still limited to static planes.

[0004] With the maturation of 3D neural rendering technology, 3D virtual try-on has become a research hotspot. This type of method utilizes 3D human body reconstruction and techniques such as Neural Radiation Field (NeRF) and Gaussian Splatting to bind the geometry of clothing to the human body surface, thereby achieving visualized spatial deformation and angle switching. However, the construction of 3D models often requires multi-view, multi-pose acquisition equipment and precise annotation, which is costly and lacks versatility, limiting its large-scale application in commercial environments.

[0005] Building upon this foundation, researchers have further proposed the concept of four-dimensional virtual try-on, which enables dynamic interaction between a person and clothing across time. Four-dimensional virtual try-on can achieve continuous animation while maintaining visual realism, providing technical support for scenarios such as virtual model catwalks and dynamic clothing changes by virtual anchors. However, most current four-dimensional methods still rely on expensive multi-view capture equipment, complex physical simulations, or large-scale training data, making it difficult to meet the practical needs of single-view, low-cost, and rapid generation. Therefore, how to achieve high-fidelity virtual try-on with animability, controllable posture, and consistency across multiple viewpoints, relying solely on a single clothing image, has become a key technical problem urgently needing to be solved in this field.

[0006] Representative research in the field of 4D virtual try-on includes Animatable Gaussians (AG) and LayGA. These technologies achieve joint modeling of human geometry and clothing form by learning the Gaussian distribution of the human body and clothing in 3D space. AG introduces pose-driven Gaussian parameter mapping, predicting clothing deformation under different poses through implicit functions to achieve animation-level continuity. Specifically, this method embeds learnable pose-related parameters into each Gaussian point, and uses the neural implicit network StyleUNet to map the input human skeleton pose vector to the dynamic changes of the Gaussian center position, direction, and covariance matrix. Thus, when the human skeleton rotates, bends, or displaces, the system can continuously adjust the spatial distribution of each Gaussian point, generating smooth and coherent temporal animation effects. Compared to static Gaussian representations, AG has a significant advantage in spatiotemporal consistency, enabling natural deformation and continuous rendering of clothing as the human body moves. However, this method places extremely high demands on the learning ability of the pose-driven network; if the training data distribution is insufficient, artifacts such as distortion or floating may easily occur under extreme poses.

[0007] LayGA decomposes the human body model into a multi-layered structure, representing the relative positional relationship between clothing and the body using layered Gaussian volumes, and utilizes multi-view, multi-pose training data to achieve dynamic clothing transfer. In its implementation, LayGA first decomposes the human body mesh into layers, encoding the naked body, inner clothing, and outer clothing into multi-layered Gaussian volumes respectively, and then establishes a depth order based on the transparency and occlusion relationships between layers. During the training phase, the system collects RGB-D data from different viewpoints and back-optimizes the Gaussian parameters through differentiable rendering to maintain geometric and appearance consistency across multiple poses. This method can, to a certain extent, achieve automatic deformation and occlusion handling of clothing with pose, providing a feasible multi-layered modeling paradigm for virtual try-on based on Gaussian representation. However, due to its reliance on dense viewpoint sampling and a fixed number of layers, it is difficult to achieve efficient scaling in real-world applications.

[0008] However, these methods all have significant limitations. First, they heavily rely on multi-view data or depth capture equipment, making large-scale deployment in real-world environments difficult. Second, the models are often coupled with specific pose templates in their design, making them unable to flexibly adapt to new clothing or body types. Third, because the modeling methods primarily use linear transformations (such as linear hybrid skinning), they lack the ability to accurately describe the nonlinear motion of fabric (such as wrinkles, stretching, elastic deformation, etc.). Finally, during training, these systems require substantial GPU memory and time resources, failing to meet the demands of commercial real-time applications.

[0009] Therefore, there is an urgent need for an innovative virtual try-on technology solution that can achieve free pose control, multi-view rendering, dynamic clothing deformation, and temporal consistency under single-view supervision.

[0010] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0011] (1) Lack of single-view clothing driving capability: Current four-dimensional virtual fitting methods generally rely on multi-angle and multi-frame training samples, which cannot generate dynamic wearing effects from a single product image, thus limiting the application of online fitting scenarios.

[0012] (2) Insufficient dynamic consistency: Due to the lack of unified modeling of the deformation process of clothing and human body, existing methods are prone to problems such as clothing misalignment, texture drift and inconsistent deformation when posture changes.

[0013] (3) Strong dependence on physical simulation: Some methods achieve realistic deformation through physical modeling of cloth, but the physical simulation calculation is huge and sensitive to material parameters, which is not suitable for real-time generation.

[0014] (4) Limited diversity of generation: factors such as clothing style, fabric, and pattern are not fully expressed, and the generation results lack flexibility.

[0015] (5) Lack of a general framework: Existing methods are mostly designed for specific datasets or models, making it difficult to transfer to new scenarios and clothing types. Summary of the Invention

[0016] To overcome the problems existing in related technologies, the present invention discloses a four-dimensional virtual fitting method and system for digital humans, the technical solution of which is as follows:

[0017] This invention is implemented as follows: a four-dimensional virtual try-on method for digital humans, comprising the following steps:

[0018] S1, Obtain the source character image sequence and a single target clothing image ;

[0019] S2, through a dual-branch structure that operates in parallel with roles and tasks, processes the source character image sequence and a single target clothing image respectively, generating a Gaussian mapping of viewpoint-pose under standardized poses; wherein, the source branch Receive a sequence of source human images and output a standardized pose. The first Gaussian mapping Used for reconstructing the static parts of the source character; target branch Receive a single image of the target garment Output standard posture The second Gaussian mapping under Used for generating four-dimensional virtual fitting for targets;

[0020] S3, via a shared nonlinear deformer According to the driving attitude and perspective Generate view-attitude related Gaussian parameter offsets And offset the Gaussian parameters Superimposed on the first Gaussian mapping Second Gaussian mapping ;

[0021] S4, using the reciprocal optical flow correction module, the source image sequence is processed. and a single target clothing image The generated two-dimensional virtual fitting sequence Perform timing consistency correction and generate correction monitoring signals. ;

[0022] S5. Based on the corrected supervision signal and the Gaussian mapping after superposition and offset, a four-dimensional virtual fitting result is generated through a linear blending skin-driven differentiable rendering pipeline.

[0023] In step S2, generating the Gaussian mapping of viewpoint-attitude under the normalized pose includes:

[0024] Each branch Gaussian mapping Break it down into viewpoint-pose independent basic terms Offset terms related to viewpoint-attitude The expression is:

[0025] ;

[0026] ;

[0027] In the formula, For Gaussian mapping, For branches Input the standard posture Structure generated by neural network Predicted Gaussian feature map For nonlinear deformers, the attitude is driven by the input. Perspective By nonlinear deformer Predicted Gaussian characteristic residuals This refers to the process of generating a position feature map based on the pose.

[0028] Furthermore, each pixel in the Gaussian feature map encodes a set of parameters for a Gaussian sphere, including center position, scale, rotation, color, and transparency; residual map The learnable summation residuals are given for each parameter.

[0029] Furthermore, the nonlinear deformer employs differentiated activation functions in different channels:

[0030] Exponential activation is used in the geometric scale channel;

[0031] Activate the transparency channel using the Sigmoid function;

[0032] Tanh function activation is used in the shared branch to encode the offset in a zero-centrosymmetric manner.

[0033] In step S4, the temporal consistency of the two-dimensional virtual fitting sequence generated from the source person image sequence and a single target clothing image is corrected using the reciprocal optical flow correction module, including:

[0034] Frame sampling and sparsification: uniformly sample a subset of frames from the original multi-pose sequence;

[0035] Learnable optical flow and bidirectional alignment: Based on optical flow Two-dimensional supervision Differentiable grid sampling is performed to obtain the correction supervision signal. The expression is:

[0036] ;

[0037] In the formula, For a differentiable optical flow distortion operator, It is a two-dimensional virtual fitting model. For optical flow collection, Source: Image of a person Image of the target clothing.

[0038] Furthermore, reciprocal optical flow correction also includes an adversarial training step:

[0039] A discriminator is introduced to treat uncorrected frames as positive samples and rendered frames as negative samples;

[0040] Recover details and suppress blurred edges through adversarial learning.

[0041] Furthermore, the method also includes a training step, and a total loss function during the training process. for:

[0042] ;

[0043] In the formula, The L1 distance is at the pixel level and is used to measure the rendering results. Return to Correctional Supervision ; The perceptual loss is based on a pre-trained feature-aware model to maintain semantic and texture consistency; Regularization of the parameter magnitudes of the Gaussian mapping using the L2 norm; adversarial term The discriminator uses real 2D frames as positive samples and rendered frames as negative samples, while the generator minimizes the adversarial loss. The weights are the corresponding losses.

[0044] Another object of the present invention is to provide a four-dimensional virtual fitting system for digital humans, which implements a four-dimensional virtual fitting method for digital humans, including:

[0045] The data acquisition module is used to acquire source image sequences of people and single target clothing images;

[0046] The dual-branch processing module includes a source branch and a target branch, used to generate view-pose independent Gaussian mappings under the normal pose.

[0047] A shared nonlinear deformer module is used to generate Gaussian parameter offsets based on attitude and view parameters;

[0048] The reciprocal optical flow correction module is used to correct the temporal consistency of the two-dimensional virtual try-on sequence;

[0049] The rendering module is used to generate four-dimensional virtual fitting results through a linear blending skin-driven differentiable rendering pipeline based on the corrected supervision signal and the Gaussian map after superimposed offset.

[0050] Furthermore, the system also includes a geometric constraint module for introducing geometric constraints by combining the human body template SMPL and a pre-trained parametric template, and for using zero-order spherical harmonics to perform view-independent modeling of color.

[0051] Furthermore, the dual-branch processing module, the shared nonlinear deformer module, and the reciprocal optical flow correction module are integrated into the same computing device and real-time rendering is achieved through GPU parallel computing.

[0052] Combining all the above technical solutions, the beneficial effects of this invention are as follows:

[0053] First, this invention proposes a novel four-dimensional virtual try-on framework for digital humans, which can generate high-fidelity virtual try-on results with free pose control, arbitrary viewing angles, and diverse clothing selection using only a single image of clothing. Unlike existing methods that rely on multi-view or multi-frame data, this invention enables a driveable digital human to complete a high-quality dynamic try-on based on a single clothing image.

[0054] This invention employs a nonlinear deformation model and a bi-branch structure, which can accurately capture the complex deformation of clothing under various postures and perspectives. It successfully decouples the dynamic effects and decomposes the Gaussian map into a view-posture-independent branch and a view-posture-dependent offset branch, avoiding complex semantic decoupling of neural rendering, promoting knowledge transfer between dynamic fitting and character reconstruction tasks, and solving the problem of clothing deformation under different postures.

[0055] This invention achieves prior-free consistency optimization during the virtual try-on process by introducing reciprocal optical flow correction technology. Based on an optical flow correction strategy, this module ensures smooth transitions between different viewpoints and poses in the generated virtual try-on animation, reducing common problems in existing methods such as texture jumps and inconsistent deformation.

[0056] Secondly, the four-dimensional virtual try-on method for digital humans proposed in this invention significantly surpasses existing technologies in terms of multi-view consistency, dynamic continuity, rendering quality, and generation speed. Experimental verification on multiple datasets shows that this method not only achieves high-fidelity dynamic virtual try-on but also achieves leading performance in visual perception indicators, physical stability, and cross-clothing generalization ability.

[0057] Third, the four-dimensional virtual try-on framework proposed in this invention can generate high-fidelity, animated virtual try-on results from a single clothing image, supporting arbitrary posture control and multi-view rendering. This invention has promising industrialization prospects and is expected to be applied in the following fields: E-commerce and virtual try-on: Providing online clothing sales platforms with real-time, interactive four-dimensional virtual try-on functionality, significantly improving user experience and conversion rates. Digital humans and the metaverse: Generating highly realistic digital humans with freely controllable postures and perspectives, suitable for virtual idols, virtual live streaming, and immersive social interaction. Games and film production: Providing a high-quality clothing animation generation solution without physical simulation and multi-view acquisition, significantly reducing production costs and time. AR / VR and smart terminals: Realizing real-time clothing try-on and interactive display in augmented reality and virtual reality devices, promoting the integration of smart terminal hardware and virtual content. Therefore, this technical solution has significant commercial feasibility and market value, and can form a multi-layered product line (virtual try-on SDK, digital human generation platform, virtual fashion engine, etc.).

[0058] Fourth, existing virtual fitting systems are mostly limited to two-dimensional image generation or static three-dimensional reconstruction, lacking dynamic posture control and multi-view consistency. This invention achieves high-quality spatiotemporally consistent dynamic deformation of clothing under single-view supervision, breaking through the limitations of traditional reliance on multi-view acquisition, physical simulation, or parametric models, and realizing dynamic clothing deformation and spatiotemporally consistent rendering without physical simulation.

[0059] Fifth, this invention solves two core problems that have long plagued the industry: First, the inconsistency in viewpoint-pose coupling: Existing 2D or 3D virtual try-on models exhibit obvious texture breaks and temporal jumps under different viewpoints or poses. This invention, through reciprocal optical flow correction technology, achieves adaptive optimization of optical flow between consecutive frames without external supervision, ensuring animation consistency. Second, the problem of nonlinear clothing deformation modeling: Previous methods mostly rely on linear skinning (LBS) or explicit physical simulation, making it difficult to capture complex wrinkles and volume changes. The nonlinear deformer proposed in this invention can adaptively model posture- and viewpoint-related deformation offsets, achieving realistic dynamic clothing deformation. These two technological breakthroughs make single-image-driven 4D virtual try-on possible, fundamentally solving the bottleneck problems that academia and industry have long failed to overcome in this field.

[0060] Sixth, this invention overcomes two long-standing technical biases in the industry: First, high-quality virtual try-on must rely on multi-view, multi-pose 3D acquisition or physical simulation. This invention, through the design of reciprocal optical flow correction technology and a nonlinear deformer, achieves dynamic consistency and realism comparable to multi-view acquisition under single-view supervision, fundamentally breaking the "data-dependent" technical approach. Second, clothing deformation can be considered a linear or locally rigid process. This invention designs a nonlinear deformer, proving that clothing deformation can achieve superior performance through implicit high-dimensional space dynamic decoupling learning without explicit physical modeling, thus overcoming the limitations of traditional linear skinning or simplified dynamic models. Attached Figure Description

[0061] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure;

[0062] Figure 1 This is a general framework diagram of the four-dimensional virtual try-on method for digital humans provided in the embodiments of the present invention;

[0063] Figure 2 This is a schematic diagram of a four-dimensional virtual fitting framework provided in this embodiment of the invention, which uses a single image of clothing to realize a four-dimensional virtual fitting of an animated digital human.

[0064] Figure 3 This is a qualitative comparison result diagram between the embodiment of the present invention and the existing method;

[0065] Figure 4 This is an image showing the ablation experiment results in a clothing try-on task where no clothing was seen, as provided in an embodiment of the present invention.

[0066] Figure 5 This is a comparison diagram of the viewpoint-attitude consistency and baseline method provided in the embodiments of the present invention;

[0067] Figure 6 This is a user analysis result diagram of the unseen clothing fitting comparison experiment provided in an embodiment of the present invention;

[0068] Figure 7 This is a comparison result diagram of cross-type virtual try-on provided by an embodiment of the present invention. Detailed Implementation

[0069] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0070] Currently, virtual try-on systems face significant bottlenecks in dynamic consistency, nonlinear deformation, and multi-view rendering. Specifically: a) Temporal discontinuity: Most existing virtual try-on algorithms are based on static images or a limited number of video frames. When poses change rapidly or viewpoints switch, clothing textures exhibit drift, jitter, and deformation discontinuities, resulting in videos lacking temporal consistency and visual stability; b) Limited deformation modeling: Existing 3D virtual try-on systems often use linear deformation methods such as linear hybrid skinning (LBS) to simulate fabric deformation. However, actual clothing undergoes complex nonlinear dynamic changes during human movement, such as wrinkle formation, elastic recoil, and gravity-induced falls. Linear models struggle to accurately capture these details; c) Heavy data dependence: Many 4D virtual try-on models rely on multi-view acquisition or volumetric reconstruction data. Acquisition equipment is expensive and the process is complex, making it unsuitable for applications such as online e-commerce or virtual live streamers; d) Limited input: Currently, few methods can achieve dynamic try-on with only a single image of the clothing item. Most systems require multi-view clothing scanning or 3D modeling, which is complex and unsuitable for large-scale applications.

[0071] The core of this invention lies in achieving a high-fidelity, animable, controllable posture, and multi-view consistency four-dimensional virtual fitting system without multi-view data, physical simulation, or input of only a single clothing image. This invention not only focuses on the realism of the generated results but also emphasizes the dynamic coupling relationship between clothing and human movement, ensuring that the clothing moves naturally, the texture is consistent, and the posture switches freely during action transitions.

[0072] The innovation of the four-dimensional virtual fitting method and system for digital humans provided in this invention is as follows:

[0073] 1. A novel four-dimensional virtual try-on framework for digital humans based on a single garment image; This invention proposes a novel four-dimensional virtual try-on framework for digital humans that generates high-fidelity virtual try-on results with free pose control, arbitrary viewing angles, and diverse garment selections using only a single garment image. Unlike existing methods that rely on multi-view or multi-frame data, this invention enables a driveable digital human to complete high-quality dynamic try-on based on a single garment image.

[0074] 2. Nonlinear Gaussian Mapping and Clothing Deformation: Through a nonlinear deformer, this invention employs a nonlinear deformation model and a dual-branch structure, which can accurately capture the complex deformation of clothing under various postures and viewpoints. It successfully decouples the dynamic effects, decomposes the Gaussian mapping into viewpoint-posture-independent branches and viewpoint-posture-dependent offset branches, avoids complex neural rendering semantic decoupling, promotes knowledge transfer between dynamic fitting and character reconstruction tasks, and solves the problem of clothing deformation under different postures.

[0075] 3. Optical Flow Correction and Temporal Consistency: This invention achieves prior-free consistency optimization during the virtual try-on process by introducing reciprocal optical flow correction technology. Based on an optical flow correction strategy, this module ensures smooth transitions between different viewpoints and poses in the generated virtual try-on animation, reducing common problems in existing methods such as texture jumps and inconsistent deformation.

[0076] Example 1, a four-dimensional virtual try-on method for digital humans, includes the following steps:

[0077] S1, Obtain the source character image sequence and a single target clothing image ;

[0078] S2, through a dual-branch structure that operates in parallel with roles and tasks, processes the source character image sequence and a single target clothing image respectively, generating a Gaussian mapping of viewpoint-pose under standardized poses; wherein, the source branch Receive a sequence of source human images and output a standardized pose. The first Gaussian mapping Used for reconstructing the static parts of the source character; target branch Receive a single image of the target garment Output standard posture The second Gaussian mapping under Used for generating four-dimensional virtual fitting for targets;

[0079] S3, via a shared nonlinear deformer According to the driving attitude and perspective Generate view-attitude related Gaussian parameter offsets And offset the Gaussian parameters Superimposed on the first Gaussian mapping Second Gaussian mapping ;

[0080] S4, using the reciprocal optical flow correction module, the source image sequence is processed. and a single target clothing image The generated two-dimensional virtual fitting sequence Perform timing consistency correction and generate correction monitoring signals. ;

[0081] S5. Based on the corrected supervision signal and the Gaussian mapping after superposition and offset, a four-dimensional virtual fitting result is generated through a linear blending skin-driven differentiable rendering pipeline.

[0082] The core objective of the four-dimensional virtual try-on method for digital humans provided in this invention is to achieve four-dimensional virtual try-on for an animated digital human, given only a single image of clothing. Figure 1 As shown, it supports free pose control, arbitrary viewpoint rendering, and a wide selection of clothing, without relying on multi-view clothing shooting or explicit physical priors.

[0083] Specifically, the method of this invention includes the following main technical steps: receiving a user's digital human model and a target clothing image; the system automatically analyzes the human posture parameters and viewpoint information; using the nonlinear deformer proposed in this invention, the Gaussian representation of the clothing is decomposed into "viewpoint-posture invariant part" and "viewpoint-posture related offset," realizing the dynamic nonlinear deformation of the clothing; using reciprocal optical flow correction technology to adaptively correct the optical flow field of multi-frame rendering results, ensuring that the clothing shape and texture remain continuous under different postures and viewpoints; combining the linear skinning and Gaussian mapping mechanism of the digital human model, we generate a brand-new digital human model that has the same human features as the original digital human but wears the target clothing; finally, we can drive the new digital human through arbitrary input posture and achieve high-quality rendering under arbitrary viewpoints, realizing a high-fidelity, dynamically coherent four-dimensional virtual try-on effect.

[0084] I. Overall Architecture and Parallel Learning Paradigm;

[0085] like Figure 2 As shown, the system adopts a dual-branch structure with parallel character / task implementation. The "source branch" and "target branch" respectively handle the tasks of "source character reconstruction" and "target 4D fitting." A transferable nonlinear deformation channel is established between the two branches through a "nonlinear deformer." During training, all three are optimized in parallel and aligned under a normalized Gaussian representation with unified semantics. To address the temporal inconsistencies and view-pose coupling mismatch issues under single-view supervision, "reciprocal optical flow correction" is introduced to obtain a consistent and usable 2D supervision sequence. The overall output is completed through a differentiable rendering pipeline driven by linear hybrid skinning (LBS).

[0086] Specifically, given: (1) A collection of images of dressed figures from multiple perspectives and in various poses. (2) Single image of the target clothing The system employs joint learning: a) the source branch maps the position feature map under standard pose to a view-pose-independent Gaussian representation; b) the target branch learns a wearable view-pose-independent Gaussian representation under a single garment input condition; c) a nonlinear deformer receives the pose. Perspective Under the same conditions, output the view-attitude related Gaussian parameter offset as a nonlinear deformation variable that can be shared between the two tasks; d) Transform the Gaussian representation from the norm space to the current pose space through LBS and render it.

[0087] II. Decomposition Representation Based on Gaussian Mapping;

[0088] To simultaneously depict both the "stable human body and clothing structure" and the "dynamics caused by posture and perspective," this invention divides each branch... Gaussian mapping Break it down into a "view-pose independent" base term and a "view-pose dependent" offset term:

[0089] ;

[0090] ;

[0091] In the formula, For branches Input the standard posture Structure generated by neural network Predicted Gaussian feature map; For nonlinear deformers, the attitude is driven by the input. Perspective By nonlinear deformer Predicted Gaussian characteristic residuals; This is the process of generating a positional feature map based on the pose. Each pixel in the Gaussian feature map encodes a set of parameters for a Gaussian sphere, including: center position, scale, rotation, color, and transparency. Residual map. Learnable summative residuals are given for each of the above parameters, thereby realizing complex, nonlinear clothing dynamics.

[0092] III. Two-branch StyleUNet (Neural Network):

[0093] Source character reconstruction and target 4D fitting room generation;

[0094] (1) Source branch Taking multi-pose and multi-view data of a character as input, output a view-pose independent Gaussian mapping under normalized poses. This is used for the static reconstruction of the source figure. The high-frequency geometry and appearance priors learned on large-scale 3D human body data provide knowledge for subsequent clothing transfer.

[0095] (2) Target branch : Using a single image of clothing in the store For reference, output the corresponding normalized Gaussian mapping. Used to generate a four-dimensional virtual fitting room.

[0096] IV. Sharing and Activation Strategies for Nonlinear Deformers;

[0097] The nonlinear deformer, acting as a shared deformer generator for both branches, receives the position map driven by the attitude. Perspective Output the offset of the Gaussian parameters To avoid semantic mismatches caused by direct sharing, this invention employs differentiated activation functions and constraints in different channels: a) Exponential activation is used in the geometric scale channel to ensure positive values ​​and numerical stability; b) Sigmoid activation is used in the transparency channel to ensure boundedness and renderability stability; c) Tanh activation is used in the shared branch to encode the offset in a zero-centrosymmetric manner, facilitating bidirectional fine-tuning around the canonical state.

[0098] It is worth noting that the Sigmoid transparency output from the target / source task branch and the Tanh offset from the shared branch form a "learnable soft filtering" mechanism: when the combination of the two approaches zero, unstable or inconsistent Gaussians are suppressed, thereby effectively reducing "spiky" artifacts and flickering in high-frequency regions such as clothing boundaries, and improving the consistency of geometry and appearance. This mechanism significantly improves the nonlinear deformation quality and boundary stability of complex fabrics without explicit physical priors.

[0099] V. Reciprocal Optical Flow Correction: Enhancement of Prior Consistency Based on Optical Flow;

[0100] Since the training supervision is derived from frame sequences generated by 2D virtual try-on, these sequences often exhibit inconsistencies and misalignments under different viewpoints and poses. Direct supervision can lead to 3D scenes learning incorrect temporal and geometric sequences. Therefore, this invention proposes reciprocal optical flow correction:

[0101] 1) Frame Sampling and Sparsity: A subset is uniformly sampled from the original multi-pose sequence to reduce the cumulative impact of cross-frame inconsistencies while covering the viewpoint-pose distribution as much as possible. In practice, it has been found that random sampling can lead to instability, and manual selection can impair generalization ability. Therefore, uniform sampling is more robust.

[0102] 2) Learnable optical flow and bidirectional alignment: for two-dimensional supervision Based on optical flow Differentiable mesh sampling is performed to obtain correction supervision. :

[0103] ;

[0104] in, It is a two-dimensional virtual fitting model. For optical flow aggregation, For differentiable optical flow distortion operators.

[0105] The key to reciprocal optical flow correction lies in "reciprocal" optimization: the target branch, due to its fixed input, converges more easily in the early stages and provides a consistent 3D reference, thereby guiding the optical flow network to align degraded 2D frames to a more reliable early 3D representation; conversely, the corrected texture promotes a clearer and more stable appearance in the target branch output, forming a mutually reinforcing closed loop. To further constrain geometric consistency: a) the system combines the human body template SMPL and pre-trained parametric templates to introduce geometric constraints; b) zero-order spherical harmonics are used to perform view-independent color modeling.

[0106] 3) Adversarial training: Rough alignment may sacrifice texture in deformed areas. To address this, the present invention introduces a discriminator that treats uncorrected frames as real and rendered frames as fake. Through adversarial learning, details are restored and blurred edges are suppressed, thereby achieving a balance between consistency and sharpness.

[0107] VI. Learning Objectives and Loss Function

[0108] The total loss consists of four parts: pixel reconstruction, perception, regularization, and adversarial analysis.

[0109] ;

[0110] In the formula, The L1 distance is at the pixel level and is used to measure the rendering results. Return to Correctional Supervision ; The perceptual loss is based on a pre-trained feature-aware model to maintain semantic and texture consistency; Apply L2 regularization to the parameter magnitudes of the Gaussian mapping to prevent numerical explosion; adversarial terms. The discriminator uses real 2D frames as positive samples and rendered frames as negative samples, while the generator minimizes its adversarial loss to improve realism and detail quality.

[0111] Example 2: The four-dimensional virtual fitting system for digital humans provided in this embodiment of the invention includes:

[0112] The data acquisition module is used to acquire source image sequences of people and single target clothing images;

[0113] The dual-branch processing module includes a source branch and a target branch, used to generate view-pose independent Gaussian mappings under the normal pose.

[0114] A shared nonlinear deformer module is used to generate Gaussian parameter offsets based on attitude and view parameters;

[0115] The reciprocal optical flow correction module is used to correct the temporal consistency of the two-dimensional virtual try-on sequence;

[0116] The rendering module is used to generate four-dimensional virtual fitting results through a linear blending skin-driven differentiable rendering pipeline based on the corrected supervision signal and the Gaussian map after superimposed offset.

[0117] Preferably, the system also includes a geometric constraint module for introducing geometric constraints by combining the human body template SMPL and a pre-trained parametric template, and for using zero-order spherical harmonics to perform view-independent modeling of color.

[0118] Preferably, the dual-branch processing module, the shared nonlinear deformer module, and the reciprocal optical flow correction module are integrated into the same computing device and real-time rendering is achieved through GPU parallel computing.

[0119] Example 3, Alternative Temporal Consistency Optimization Strategy - Temporal Consistency Optimization Based on Self-Supervised Learning:

[0120] This invention employs reciprocal optical flow correction to enhance temporal consistency, but another possible alternative is a self-supervised learning method. Through self-supervised learning, the system can automatically optimize the training process based on inter-frame differences and image generation errors, without relying on additional manually labeled data.

[0121] Example 4, an alternative nonlinear deformation modeling method – general human body deformation modeling based on large models:

[0122] The nonlinear deformer of this invention models nonlinear clothing deformation by sharing Gaussian offset. However, with the future availability of more abundant 3D data, general human body deformation modeling can absorb more complex local and global deformation dependencies, achieving more general and efficient generation capabilities.

[0123] Example 5: Alternative Gaussian Feature Map Generation Structure – Based on Other Network Structures: This invention uses StyleUNet to generate Gaussian feature maps, but potential alternatives include ResNet, UNet, and other common encoder-decoder structures. Different alternative networks have different characteristics and limitations. Adopting a reasonable generation structure based on the actual situation can promote the performance adaptation of this invention in different tasks.

[0124] The proposed four-dimensional virtual try-on framework significantly surpasses existing technologies in terms of multi-view consistency, dynamic continuity, rendering quality, and generation speed. Experimental verification on multiple datasets demonstrates that this method not only achieves high-fidelity dynamic virtual try-on but also achieves leading performance in visual perception metrics, physical stability, and cross-clothing generalization ability.

[0125] This invention uses five characters from two 3D human datasets for evaluation: three characters (zzr, lbn1, lbn2) from the AvatarReX dataset and two characters (Actor04, Actor08) from the ActorsHQ dataset, combined with over 50 garments from the DressCode and VITON-HD datasets. Each character contains 16 to 40 capture sequences, each sequence containing approximately 2000 frames. This invention employs IDM-VTON as a 2D virtual try-on model and will be compared with video-based virtual try-on methods in subsequent sections.

[0126] This invention was compared with the following state-of-the-art methods: 1) a two-dimensional video-based virtual try-on method: ViViD; 2) a two-dimensional image-based virtual try-on method: IDM-VTON; 3) a three-dimensional editing method: GaussianEditor; and four-dimensional virtual try-on methods: IDM-VTON+SCARF and IDM-VTON+AG, where SCARF is an animable human reconstruction method based on neural radiation fields, and AG is a method based on three-dimensional Gaussian sputtering. This invention also includes LHM (a four-dimensional digital human generation method), which is similar to this invention in its digital human creation and actuation process.

[0127] Due to the lack of large-scale 3D human body datasets with various garments, traditional 3D metrics (such as ChamferDistance) are not suitable for the quantitative evaluation of 4D virtual try-on. As an alternative, this invention uses 2D metrics on rendered images. Specifically, this invention employs peak signal-to-noise ratio, structural similarity, and perceptual similarity for reference image-based evaluation.

[0128] For the unseen clothing try-on experiment, this invention uses text similarity based on the CLIP visual language model to measure the semantic alignment between the reference clothing and the rendered result, and to evaluate texture fidelity and try-on quality. Furthermore, this invention also employs subject consistency and image quality dimensions from the VBench video evaluation model as metrics to evaluate temporal consistency and frame-by-frame image fidelity. It is important to note that image quality only calculates the quality of a single frame and typically favors non-four-dimensional methods because they do not enforce temporal consistency. Therefore, image quality metrics were excluded from the main comparisons and are only used in ablation studies to analyze the contribution of different components to the quality of a single frame. Finally, this invention also includes user studies to evaluate perceived quality.

[0129] Table 1 shows the quantitative comparison results with existing methods on the AvatarReX dataset.

[0130]

[0131] Table 1 presents the quantitative comparison results with existing methods. This invention performs superiorly in all reconstructed and unseen clothing try-on settings, particularly in the following aspects: Compared to ViViD, the four-dimensional virtual try-on framework of this invention shows significantly better visual quality and temporal consistency, exposing the inherent limitations of two-dimensional video-based methods in handling complex human poses. A comparison with IDM-VTON's reconstructed try-on performance further validates the advantages of this invention, especially in mitigating the inconsistency degradation problem commonly found in existing models. In comparisons with other four-dimensional virtual try-on methods, this invention shows significant improvements in rendering quality, texture fidelity, and temporal consistency. This invention demonstrates significant improvements over existing technologies in multiple standard evaluation metrics, particularly in image quality (peak signal-to-noise ratio, structural similarity, perceptual similarity), text similarity, and subject consistency metrics. As shown in the table, the peak signal-to-noise ratio (PSNR) improved by approximately 1.3-1.5 dB, indicating a significant improvement in the quality of the generated images. Structural similarity increased by 0.014-0.02, indicating that the structure and details of the images are closer to reality. Perceptual similarity decreased by 13%, meaning that the generated images are perceptually more natural and have fewer differences. Furthermore, text similarity improved by 1.2-1.8%, indicating that the model can more accurately align the visual features of the clothing with the posture of the target person, resulting in more semantically consistent try-on results.

[0132] Table 2 shows the ablation experiments on unseen clothing on the AvatarReX dataset.

[0133]

[0134] Figure 3 Qualitative comparisons with existing methods are presented. Compared to ViViD, which is based on 2D video, this invention achieves significant improvements in temporal consistency and texture fidelity. While ViViD provides short-term continuity through frame-level temporal modeling, it lacks explicit 3D structural inference, leading to texture flickering and deformation errors in complex poses. Compared to other 4D baselines, IDM-VTON, which combines SCARF and AG, improves spatial consistency to some extent, but still suffers from viewpoint-pose inconsistencies due to missing or low-quality supervised data. SCARF is inaccurate in pose estimation, resulting in significant alignment errors. In contrast, this invention generates the most consistent and realistic results, maintaining accurate geometry, lifelike physical interactions, and high-fidelity textures.

[0135] Table 2 and Figure 4 This demonstrates the superior performance of the present invention in the unseen clothing try-on task. A significant decrease in image quality metrics occurs upon removing the nonlinear deformer, indicating compromised image quality, particularly in geometrically deformed areas where severe spike artifacts appear on the model surface. Removing reciprocal optical flow correction results in a decrease in subject consistency metrics, highlighting its crucial role in maintaining consistency. Removing adversarial loss reduces texture sharpness and color accuracy, demonstrating its important role in mitigating the instability costs introduced by other modules.

[0136] Consistency is crucial for achieving realistic four-dimensional virtual try-on. For example... Figure 5 As shown, this invention performs excellently in terms of viewpoint-pose consistency, maintaining the stability of garment deformation and texture under different viewpoints and new postures. In comparison, IDM-VTON+AG performs closest to other four-dimensional virtual try-on methods, but still exhibits texture flickering and geometric distortion issues when handling viewpoint-pose transitions.

[0137] This invention conducted a user study to evaluate the perceived quality of virtual try-on results without seeing the clothing in person. Fifty volunteers participated in the experiment (including 25 virtual try-on experts and 25 non-experts), who completed a questionnaire containing 20 sets of four-choice questions. Each question provided an image of a person and an image of clothing, along with six randomly arranged try-on videos (using this invention and five existing methods) for evaluation. Participants independently rated the following four dimensions: clothing texture fidelity, person identity fidelity, video consistency, and overall realism. Figure 6Experimental results show that the present invention outperforms all competing methods across all four evaluation dimensions. This demonstrates the significant advantages of the present method in terms of fidelity, consistency, and realism.

[0138] This experiment demonstrates two typical cross-type virtual try-on scenarios (e.g., changing from trousers to a skirt and from a shirt to underwear). These examples represent situations where the deformation prior is insufficient or overapplied between the source and target branches. Figure 7 Experimental results show that IDM-VTON+AG generates significant geometric and texture distortion in these cross-type scenes, while the present invention maintains accurate geometry and high-fidelity textures.

[0139] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention and within the spirit and principles of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A four-dimensional virtual try-on method for digital humans, characterized in that, The method includes the following steps: S1, Obtain the source character image sequence and a single target clothing image ; S2, through a dual-branch structure that operates in parallel with roles and tasks, processes the source character image sequence and a single target clothing image respectively, generating a Gaussian mapping of viewpoint-pose under standardized poses; wherein, the source branch Receive a sequence of source human images and output a standardized pose. The first Gaussian mapping Used for reconstructing the static parts of the source character; target branch Receive a single image of the target garment Output standard posture The second Gaussian mapping under Used for generating four-dimensional virtual fitting for targets; S3, via a shared nonlinear deformer According to the driving attitude and perspective Generate view-attitude related Gaussian parameter offsets And offset the Gaussian parameters Superimposed on the first Gaussian mapping Second Gaussian mapping ; S4, using the reciprocal optical flow correction module, the source image sequence is processed. and a single target clothing image The generated two-dimensional virtual fitting sequence Perform timing consistency correction and generate correction monitoring signals. ; S5. Based on the corrected supervision signal and the Gaussian mapping after superposition and offset, a four-dimensional virtual fitting result is generated through a linear blending skin-driven differentiable rendering pipeline.

2. The four-dimensional virtual try-on method for digital humans according to claim 1, characterized in that, In step S2, generating the Gaussian mapping of viewpoint-attitude under the normalized pose includes: Each branch Gaussian mapping Break it down into viewpoint-pose independent basic terms Offset terms related to viewpoint-attitude The expression is: ； ； In the formula, For Gaussian mapping, For branches Input the standard posture Structure generated by neural network Predicted Gaussian feature map For nonlinear deformers, the attitude is driven by the input. Perspective By nonlinear deformer Predicted Gaussian characteristic residuals This refers to the process of generating a position feature map based on the pose.

3. The four-dimensional virtual try-on method for digital humans according to claim 2, characterized in that, Each pixel in the Gaussian feature map encodes a set of parameters for a Gaussian sphere, including center position, scale, rotation, color, and transparency; residual map The learnable summation residuals are given for each parameter.

4. The four-dimensional virtual try-on method for digital humans according to claim 2, characterized in that, The nonlinear deformer employs differentiated activation functions in different channels: Exponential activation is used in the geometric scale channel; Activate the transparency channel using the Sigmoid function; Tanh function activation is used in the shared branch to encode the offset in a zero-centrosymmetric manner.

5. The four-dimensional virtual try-on method for digital humans according to claim 1, characterized in that, In step S4, the temporal consistency of the two-dimensional virtual fitting sequence generated from the source person image sequence and a single target clothing image is corrected using the reciprocal optical flow correction module, including: Frame sampling and sparsification: uniformly sample a subset of frames from the original multi-pose sequence; Learnable optical flow and bidirectional alignment: Based on optical flow Two-dimensional supervision Differentiable grid sampling is performed to obtain the correction supervision signal. The expression is: ； In the formula, For a differentiable optical flow distortion operator, It is a two-dimensional virtual fitting model. For optical flow collection, Source: Image of a person Image of the target clothing.

6. The four-dimensional virtual try-on method for digital humans according to claim 5, characterized in that, Reciprocal optical flow correction also includes an adversarial training step: A discriminator is introduced to treat uncorrected frames as positive samples and rendered frames as negative samples; Recover details and suppress blurred edges through adversarial learning.

7. The four-dimensional virtual try-on method for digital humans according to claim 1, characterized in that, The method also includes a training step and a total loss function during the training process. for: ； In the formula, The L1 distance is at the pixel level and is used to measure the rendering results. Return to Correctional Supervision ; The perceptual loss is based on a pre-trained feature-aware model to maintain semantic and texture consistency; Regularization of the parameter magnitudes of the Gaussian mapping using the L2 norm; adversarial term The discriminator uses real 2D frames as positive samples and rendered frames as negative samples, while the generator minimizes the adversarial loss. The weights are the corresponding losses.

8. A four-dimensional virtual fitting system for digital humans, characterized in that, This system is used to implement the four-dimensional virtual try-on method for digital humans as described in any one of claims 1-7, comprising: The data acquisition module is used to acquire source image sequences of people and single target clothing images; The dual-branch processing module includes a source branch and a target branch, used to generate view-pose independent Gaussian mappings under the normal pose. A shared nonlinear deformer module is used to generate Gaussian parameter offsets based on attitude and view parameters; The reciprocal optical flow correction module is used to correct the temporal consistency of the two-dimensional virtual try-on sequence; The rendering module is used to generate four-dimensional virtual fitting results through a linear blending skin-driven differentiable rendering pipeline based on the corrected supervision signal and the Gaussian map after superimposed offset.

9. The four-dimensional virtual fitting system for digital humans according to claim 8, characterized in that, The system also includes a geometric constraint module for introducing geometric constraints by combining the human body template SMPL and a pre-trained parametric template, and for using zero-order spherical harmonics to perform view-independent modeling of color.

10. The four-dimensional virtual fitting system for digital humans according to claim 8, characterized in that, The dual-branch processing module, the shared nonlinear deformer module, and the reciprocal optical flow correction module are integrated into the same computing device and real-time rendering is achieved through GPU parallel computing.

Images