A three-dimensional virtual clothes changing method based on physical driving and UV feature space enhancement

Abstract

The application provides a three-dimensional virtual clothes changing method based on a physical driving UV feature space enhancement, comprising the following steps: firstly, encoding multi-view image information to a standard posture UV feature plane, and realizing clothes feature replacement by using semantic information; and performing identity consistency repair in the UV space for a newly exposed skin area. The core lies in: constructing dynamic driving information according to a human body action sequence and projecting the dynamic driving information to the UV domain, combining material properties to establish a physical constraint model, and performing physical perception enhancement on the repaired feature plane; and finally, inputting the enhanced feature into a decoder to generate a three-dimensional human body representation and rendering. The application converts complex three-dimensional cloth deformation into a two-dimensional hidden space constraint optimization process, guarantees the calculation efficiency, and improves the physical reality and time sequence continuity in a dynamic scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a physical-driven UV feature space-enhanced 3D virtual clothing changing method. Background Technology

[0002] Virtual try-on is an important application in the fields of computer vision and computer graphics. Its goal is to present virtual or real clothing to a user's virtual or real human image in a realistic way. Existing virtual try-on technologies can be mainly divided into methods based on two-dimensional image space and methods based on three-dimensional space.

[0003] Clothing transfer methods based on two-dimensional image space typically perform deformation, alignment, and fusion processing on the two-dimensional image plane to generate images of the clothing after the makeover. However, because the processing is inherently limited to two-dimensional space, it is difficult to accurately depict the three-dimensional structural characteristics of clothing from different perspectives, such as its three-dimensional shape, wrinkle changes, and the fit between the clothing and the human body's three-dimensional form. Therefore, in scenes with multiple perspectives or significant pose changes, consistency issues often exist. In recent years, some methods based on generative video models have improved visual performance to some extent by training on large-scale data to learn the dynamic change patterns of clothing. However, due to the lack of explicit physical constraints, the generated results may still exhibit problems such as unstable wrinkle distribution, temporal fluctuations, or insufficient representation of differences between different materials in complex motion scenes. Virtual try-on methods based on three-dimensional space typically involve three-dimensional human body modeling, three-dimensional clothing modeling, and the geometric registration and deformation process between clothing and the human body. These methods can maintain three-dimensional structural consistency to a certain extent, but if the physical behavior of real fabric is further considered, complex processes such as fabric dynamics simulation, collision detection, and response calculation are usually required. The aforementioned physical simulation process incurs significant computational overhead and high implementation complexity, making it difficult to simultaneously guarantee real-time performance and maintain a realistic sense of dynamic physics. Therefore, some 3D methods, in practical applications, often focus on geometric alignment and topological matching, while paying insufficient attention to the physical dynamics of clothing during movement. This results in generated results lacking the necessary dynamic naturalness in scenarios involving significant human motion.

[0004] Furthermore, during virtual clothing changes, previously concealed skin areas are often revealed. How to reasonably model these newly revealed areas and ensure that the generated skin maintains consistency with the individual in terms of skin tone, texture, and overall identity characteristics is another key issue that current technologies need to address. Existing methods typically require the introduction of additional modules or multi-stage processing flows, which not only increases system complexity but may also impact identity consistency and visual coherence.

[0005] Therefore, how to simultaneously maintain identity consistency, generate skin regions reasonably, and express the dynamic physical effects of clothing in a unified representation space while ensuring computational efficiency has become a pressing technical problem in the current field of 3D virtual dress-up. Summary of the Invention

[0006] Purpose of the Invention: The purpose of this invention is to address the shortcomings of existing virtual clothing-changing technologies in terms of dynamic physical realism, computational efficiency, and cross-action identity preservation. It proposes a physics-driven UV feature space-enhanced 3D virtual clothing-changing method. This invention achieves high realism, temporal stability, and computational efficiency by performing feature decoupling and replacement in a structured UV space, utilizing a latent space content completion mechanism to ensure identity consistency, and innovatively introducing a physical perception mechanism. This results in generated results with wrinkle details and momentum effects that conform to dynamic constitutive laws, thereby achieving a highly realistic, temporally stable, and computationally efficient 3D virtual clothing-changing effect. To achieve the above objectives, this invention includes the following steps:

[0007] Step 1: For the geometric and appearance reconstruction process of a single digital human, a set of RGB images and camera parameters are used as input. A UV feature plane, i.e., a two-dimensional flattened feature plane, is initialized for each set of RGB images, where U refers to the horizontal coordinate U-coordinate and V refers to the vertical coordinate V-coordinate. The UV feature plane establishes a parameterized correspondence with the three-dimensional surface of the human body under the standard pose. By updating only the UV feature plane parameters through gradient backpropagation, a UV feature plane representation encoding the identity and clothing information of the human instance corresponding to the RGB image sample is obtained. Each pixel in the UV feature plane representation contains the semantic information of the corresponding human body region in the RGB image.

[0008] Step 2: Based on the semantic information of the corresponding human body region obtained in Step 1, the clothing features in the UV feature plane representation of the target clothing are overlaid onto the original clothing region in the UV feature plane representation of the person to be dressed, thus completing the basic clothing replacement.

[0009] Step 3: Based on the difference information between the UV feature plane representations before and after clothing replacement, candidate UV regions that are transformed from clothing areas to skin areas after clothing replacement are determined; the completeness of feature information corresponding to each UV position in the candidate UV regions is detected; in order to ensure that the edges of the repair area are natural and smooth, the candidate UV regions are appropriately expanded at the edges to obtain the mask region that needs to be repaired; using a pre-trained generative model, with the UV feature information of the mask region and its surroundings as conditional input, generative completion is performed on the UV features corresponding to the mask region to generate skin feature information consistent with the identity features of the person instance;

[0010] Step 4: Perform a UV feature enhancement process based on dynamic implicit force field and physical constraints to obtain an enhanced UV feature plane representation;

[0011] Step 5: Perform the rendering and pose mapping process.

[0012] In step 1, each RGB image is processed using an image semantic segmentation tool, and the segmentation results are divided into background, skin, and clothing to be replaced, to obtain a semantic segmentation mask corresponding to each RGB image;

[0013] In the digital human reconstruction process, a UV feature plane is initialized for each group of RGB images. The UV feature plane is used to represent a 3D Gaussian human body in canonical space. The UV feature plane representation is input into a pre-trained decoder and combined with a 3D Gaussian splash renderer for forward inference to generate the corresponding rendered image. The reconstruction loss between the rendered image and the original RGB image is calculated. By using a backpropagation optimization method, the reconstruction loss is backpropagated, and only the feature values ​​in the UV feature plane are updated and optimized, thereby obtaining the optimal UV feature plane representation in canonical space that encodes the identity, pose, and clothing information of the person instance corresponding to the image sample.

[0014] In step 3, the pre-trained generative model employs the following strategy when performing content completion: within the masked region, the denoised result predicted by the generative model based on a diffusion mechanism is used as the corresponding UV feature; in regions outside the masked region, the known feature values ​​in the original UV feature plane are kept unchanged.

[0015] ⊙ ⊙ ,

[0016] in, This represents the completed UV feature plane, where M represents the mask region. This represents the denoising result predicted by the diffusion model at step t. This represents the known eigenvalues ​​in the original UV feature plane, and the symbol ⊙ indicates element-wise multiplication.

[0017] In step 3, the pre-trained generative model includes an attention mechanism. The attention mechanism uses features extracted from the facial region represented by the UV feature plane as a conditional constraint, enabling the extended diffusion model to fuse skin color and identity-related feature information corresponding to the person instance when generating skin features of newly exposed skin regions.

[0018] Step 4 includes:

[0019] Step 4-1: Extract dynamic features and construct the external force driving field;

[0020] Obtain the pose sequence of the skinned multi-person linear model (SMPL). Where T represents the total number of frames in the attitude sequence. Let represent the human pose parameter vector corresponding to frame t. The pose sequence is used to characterize the rotational state of the human skeletal joints. Based on the pose sequence, the set of three-dimensional coordinates of the human surface vertices corresponding to each frame is obtained. Human dynamics feature vectors are calculated using the second-order time difference method:

[0021] ,

[0022] in, Let represent the set of acceleration vectors of each vertex on the human body surface in frame t; This represents the set of three-dimensional spatial coordinates of all vertices on the surface of the human body model in frame t+1;

[0023] By using a predefined correspondence between vertices and parameter domains, the three-dimensional vertex acceleration vectors are converted into two-dimensional dynamic driving fields in the parameter domain;

[0024] Step 4-2: Construct a physical constraint model;

[0025] A physical constraint model containing material constitutive constraints is constructed in the parameterized feature plane space to characterize the evolution trend of feature distribution under dynamic driving force.

[0026] Let Φ represent the overall feature distribution in the current feature plane, and Φ(s, r) represent the feature value at the parameter domain coordinate position (s, r); combined with clothing material coding... Construct a physical energy model that includes the following constraints:

[0027] The strain constraint term associated with the feature space gradient is used to characterize the intensity of feature variation in the local neighborhood;

[0028] The bending constraint term, which is related to the second-order transformation of the feature space, is used to characterize the smoothness of the feature distribution.

[0029] With dynamic driving field The relevant external force response term is used to characterize the effect of dynamic action on the characteristic distribution;

[0030] By using a physical energy model, a constraint relationship between material properties and dynamic response is established in the latent space;

[0031] Step 4-3, prediction of latent space dynamic response;

[0032] Based on the physical constraint model constructed in step 4-2, the response direction of the current feature distribution under the combined action of material constraints and dynamic driving is calculated to characterize the evolution trend of the feature in the latent space.

[0033] The response direction represents the directional information of how the eigenvalues ​​should change in the parameter domain under physical constraints and external dynamic driving forces.

[0034] A physical perception feature enhancement network can be constructed using structures such as convolutional neural networks; the physical perception feature enhancement network takes the following information as joint input:

[0035] The overall feature distribution Φ in the current feature plane;

[0036] Dynamic driving fields in the parameter domain ;

[0037] Characteristic response direction information calculated based on the physical constraint model;

[0038] The physical perception feature enhancement network predicts the dynamic displacement bias field ΔΦ in the latent space, which represents the update amount of the feature value at each parameter domain position; the dynamic displacement bias field ΔΦ represents the adjustment magnitude of the feature distribution in the latent space under the physical consistency constraint.

[0039] Step 4-4, Feature Space Reconstruction and Physical Injection;

[0040] The original UV feature plane is spatially mapped and structurally adjusted using a dynamic displacement bias field ΔΦ to generate an enhanced UV feature plane representation.

[0041] In step 4-3, in order to achieve inertial memory and momentum feedback, the physical perception feature enhancement network recursively fuses the force field of the current frame with the displacement bias of historical moments to ensure the continuity of fold evolution.

[0042] In step 4-3, the physical constraint model constructs a deformation energy constraint relationship based on the material properties of clothing. By optimizing the comprehensive response relationship between the deformation energy constraint and the dynamic driving field, the dynamic displacement bias is predicted so that the enhanced UV feature plane satisfies the physical stability constraint in spatial distribution.

[0043] Step 5 includes: inputting the enhanced UV feature plane representation obtained in step 4 into a pre-trained 3D reconstruction decoder to decode a set of 3D Gaussian points and attribute sets containing geometric and appearance compensation attributes; using the user-defined arbitrary skinned multi-person linear model SMPL pose parameters to perform linear blending skin deformation on the 3D Gaussian points, transforming the 3D Gaussian points from the normal pose space to the target pose space, and using the 3D differentiable rendering representation to render and generate the final image.

[0044] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0045] The present invention also provides a storage medium storing a computer program or instructions that, when the computer program or instructions are run on a computer, execute the steps of the method described.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. Achieving structural decoupling: This invention completes the garment feature transfer and generation process in a unified UV feature plane, avoiding the complex three-dimensional mesh topology reconstruction and explicit fabric dynamics simulation process. It achieves structural decoupling of identity information, posture information and garment appearance information in the same parameterized space, thereby reducing computational complexity and improving processing efficiency.

[0048] 2. Enhanced Dynamic Physical Consistency: This invention constructs a physical constraint model based on material properties and introduces dynamic driving information into the UV feature space to update the features, so that the generated clothing deformation satisfies the physical consistency constraint during continuous movement, thereby improving the problem of unstable clothing deformation in dynamic scenes in the prior art.

[0049] 3. Improve temporal continuity: This invention introduces historical state information for recursive fusion during feature enhancement, enabling the feature update process to have continuous response characteristics, thereby reducing detail jumps between consecutive frames and improving temporal stability in dynamic sequences.

[0050] 4. Ensure consistent identity restoration effect: For newly exposed skin areas after clothing replacement, this invention introduces an identity constraint generation mechanism in the UV feature space to fill in missing areas, so that the generated skin features are consistent with the original identity features of the person, reducing inconsistencies in skin color or texture.

[0051] 5. Maintaining geometric consistency across multiple perspectives: This invention uses a unified parametric UV space and 3D Gaussian representation for feature decoding and rendering, ensuring that the enhanced features maintain geometric consistency under different perspective conditions, thereby improving the problem of structural inconsistency that easily occurs when the perspective changes in existing 2D methods. Attached Figure Description

[0052] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0053] Figure 1 This is a flowchart of the method of the present invention.

[0054] Figure 2 This is a flowchart illustrating the optimization of the UV feature plane in an embodiment of the present invention.

[0055] Figure 3 This is a schematic diagram illustrating the exchange of different UV feature plane regions in an embodiment of the present invention.

[0056] Figure 4 This is a diagram illustrating the replacement of clothing and the driving results in an embodiment of the present invention.

[0057] Figure 5 This is a visualization of the results before and after repair in an embodiment of the present invention. Detailed Implementation

[0058] like Figure 1 As shown in the figure, this invention proposes a physical-driven UV feature space-enhanced 3D virtual clothing changing method. The overall process includes steps such as UV feature plane reconstruction, clothing replacement, mask generation and skin information completion, 3D Gaussian point decoding, pose deformation and final rendering. Figure 1 The execution order and data flow relationship of each module of this invention are clearly illustrated. The method includes the following steps:

[0059] Step 1: For the geometric and appearance reconstruction process of a single digital human, multiple RGB images and their corresponding camera parameters are used as input. First, each RGB image is processed using existing image semantic segmentation tools to divide the image region into background, skin region, and clothing region to be replaced, obtaining a semantic segmentation mask corresponding to each RGB image.

[0060] During the reconstruction process, a UV feature plane is initialized for each image sample to characterize the 3D Gaussian human appearance features in the canonical space. To visually demonstrate the UV feature plane optimization process, as shown below... Figure 2 As shown, iterative optimization is performed on a single digital human instance by backpropagating the UV feature plane through reconstruction loss until the original image can be reconstructed with high fidelity. The UV feature plane is input into a pre-trained decoder and forward inference is performed in conjunction with a 3D Gaussian splash renderer to generate the corresponding rendered image. By calculating the reconstruction loss between the rendered image and the corresponding original RGB image, and using backpropagation optimization, only the feature values ​​in the UV feature plane are updated and optimized until they can reconstruct the original image well, thus obtaining a UV feature plane representation in canonical space that encodes the identity, posture, and clothing information of the corresponding human instance. At this point, the UV feature plane mainly stores the geometric and appearance benchmarks of the human body in a standard posture, and does not yet include the dynamic deformation details under specific actions.

[0061] Step 2: Based on the semantic segmentation information obtained in Step 1, locate the original clothing area of ​​the person to be dressed up in the UV feature plane. Cover the corresponding original clothing area in the UV feature plane representation of the target clothing with the semantic features of clothing, thereby completing the basic clothing replacement operation in the UV feature plane space.

[0062] Step 3: Based on the difference between the UV feature plane representations before and after clothing replacement, identify newly exposed skin areas that lack sufficient information in the original UV feature plane representation, and generate mask regions for repair accordingly. Using a diffusion-based generative model, with the mask region and its surrounding UV feature information as input, generatively complete the UV features corresponding to the mask region. For example... Figure 3 As shown, this embodiment demonstrates the process of repairing skin loss areas caused by clothing replacement. By identifying candidate UV regions at the clothing boundary and generating a mask, a diffusion model incorporating an attention mechanism is used to fill in the mask regions, ensuring that the newly generated skin texture is visually and logically consistent with the original human body parts.

[0063] Step 4: Obtain the continuous pose sequence of the skinned multi-person linear model SMPL, extract the second-order difference acceleration features as dynamic driving information, and construct the physical energy functional in the UV space by combining material encoding. Calculate the negative gradient of this energy functional with respect to the UV features, and use it as a guiding signal to input the offset prediction network. Predict the dynamic bias in the latent space to achieve local feature enhancement (such as clothing wrinkles and pose-related shadows), and maintain inertial memory and momentum feedback in the continuous action sequence, thereby obtaining an enhanced UV feature plane representation.

[0064] Step 5: Input the enhanced UV feature plane representation into the pre-trained decoder, and obtain a set of 3D Gaussian points and their attributes in the normalized pose space through forward inference decoding. Obtain the target pose parameters to perform pose deformation on the 3D Gaussian points, and finally use a 3D Gaussian splatter renderer to render and generate the final 2D clothing change result image. The final visualization effect after completing the above steps is shown below. Figure 4 As shown.

[0065] In this embodiment, to illustrate the specific implementation process of the method of the present invention, publicly available three-dimensional human body data is selected as an example data source. This example data includes multiple three-dimensional mesh models of the human body and their corresponding parametric human body model parameters. It should be noted that the method of the present invention is not limited to using a specific dataset or parametric human body model; the relevant data is only used to illustrate the implementation method of this embodiment.

[0066] In this embodiment, to construct the multi-view image data required for reconstruction and testing, the human body 3D mesh model is rendered from multiple different camera perspectives. Specifically, multiple camera positions with azimuth and pitch angles can be set around the figure, generating an RGB image for each camera perspective and recording the corresponding camera intrinsic and extrinsic parameters. This allows for the generation of multiple RGB images from different perspectives for each human body model, thereby constructing a multi-view image dataset.

[0067] To obtain semantic information from different regions in the image, this embodiment performs semantic segmentation on each generated RGB image. The semantic segmentation tool can employ any existing segmentation network capable of outputting pixel-level semantic labels. In this embodiment, the original semantic labels in the segmentation results are further merged into the following semantic regions: 1. Background region, corresponding to areas in the image that do not contain a human body; 2. Skin region, corresponding to the exposed skin of the person not covered by clothing; 3. Clothing region, corresponding to the clothing currently worn by the person and to be replaced in the subsequent virtual clothing change process.

[0068] By classifying semantic categories as described above, we can effectively distinguish areas where skin information may need to be generated or supplemented after clothing replacement, such as the arm area exposed after replacing long-sleeved clothing with short-sleeved clothing.

[0069] In terms of data organization, for each human instance, its corresponding 3D mesh data, parameterized human model parameters, multi-view RGB images, semantic segmentation masks, and camera parameter information are stored together, so that each image sample can establish a correspondence with its corresponding human model, pose information, and semantic information, which is convenient for subsequent reconstruction and virtual clothing changing processes.

[0070] In this embodiment, a UV feature plane representation is constructed and utilized to encode the appearance information of a 3D human avatar. The UV feature plane is constructed based on the UV unfolded space of a parametric human model, which provides a 3D surface template with a unified semantic structure, ensuring that each point on the model surface can be uniquely mapped to a 2D coordinate position in the UV space. Therefore, the UV feature planes of different character instances and different clothing states have pixel-level alignment.

[0071] Each pixel location in the generative UV feature plane carries a high-dimensional feature vector, which is used to encode the 3D Gaussian point attribute information corresponding to the corresponding area on the surface of the parameterized human model. These 3D Gaussian point attributes include, but are not limited to, 3D position offset, scale, rotation, transparency, and color-related information. In this way, the geometric shape and appearance features of the person are encoded in a structured form in the 2D UV feature plane, providing a foundation for subsequent clothing feature replacement and generative completion.

[0072] In a specific implementation, the generative UV feature plane has a preset spatial resolution for encoding human appearance features in a two-dimensional UV space. Preferably, the resolution of the UV feature plane can be set to 256×256 pixels, with each pixel position corresponding to a semantically consistent region on the surface of the parameterized human model. Each pixel position stores a high-dimensional feature vector to characterize the attribute information of the three-dimensional Gaussian point corresponding to that region. In a specific embodiment, the feature vector has a dimension of 32, but is not limited to this.

[0073] The feature vectors in the UV feature plane are parsed by the decoder to obtain the corresponding three-dimensional Gaussian point attributes. The decoder is an independent neural network module whose parameters remain fixed in the method of this invention and do not participate in subsequent optimization updates. In this embodiment, the decoder can be obtained through pre-training. Specifically, before the execution of the method of this invention, the decoder is trained using three-dimensional human data containing multiple human instances. During training, the UV space features corresponding to the surface of the parameterized human model are used as input to supervise the decoder to output the corresponding three-dimensional Gaussian point attributes, enabling it to learn the mapping relationship between the UV feature space and the three-dimensional Gaussian representation. Through the above pre-training process, the decoder acquires the ability to parse the features in the UV feature plane into the geometric and appearance attributes of three-dimensional Gaussian points. During the execution phase of the method of this invention, the parameters of the decoder remain fixed and participate in the forward inference process only as a feature decoding module. This decoder can map the abstract features in the UV feature plane into the geometric and appearance attributes of three-dimensional Gaussian points, including but not limited to three-dimensional position offset, scale, rotation, transparency, and color-related parameters. During the reconstruction process, the UV feature plane parameters are optimized to minimize the reconstruction error between the decoder output and the target human image.

[0074] In a specific implementation, the reconstruction loss can be a weighted combination of various loss functions, including pixel-level reconstruction loss, structural similarity loss, and perceptual loss. Pixel-level reconstruction loss constrains the consistency between the rendered result and the original image at the pixel level; structural similarity loss constrains the consistency of image structural information; and perceptual loss constrains the consistency of high-level semantic features. The weight coefficients of each loss term can be set according to specific application requirements.

[0075] The reconstruction method described above can obtain corresponding UV feature plane representations for different character instances, different poses, and different clothing states. These UV feature plane representations simultaneously encode the character's identity features, pose information, and clothing appearance features in a structured manner within a two-dimensional UV space, providing a foundation for subsequent clothing replacement and generative completion operations based on the UV feature plane.

[0076] In this embodiment, after obtaining the UV feature plane representations corresponding to the character to be dressed and the target clothing, the clothing replacement operation is performed in the UV feature plane space. Since the UV feature planes are all constructed based on the shared parametric human model UV space, the UV feature planes of different character instances and different clothing states have a one-to-one correspondence in pixel positions, so clothing features can be directly exchanged in the UV feature plane.

[0077] Specifically, based on the semantic segmentation information obtained in step 1, the pixel positions in the UV feature plane of the target clothing that are semantically labeled as clothing are identified, and the corresponding UV features are extracted as the target clothing feature region. Simultaneously, the original clothing region is located in the UV feature plane of the person to be dressed. The UV features of the target clothing feature region are then overlaid onto the corresponding original clothing region in the UV feature plane of the person to be dressed, thereby completing the basic clothing replacement within the UV feature plane space.

[0078] After completing the garment feature replacement, it is necessary to further determine which UV regions have insufficient feature information after the replacement in order to perform subsequent generative completion processing. Therefore, this embodiment identifies candidate UV regions based on the difference information between the UV feature plane representations before and after the garment replacement. The candidate UV regions include at least the following two categories:

[0079] The first category is the boundary region between clothing and human skin, that is, the UV positions where the semantic pixels of clothing and skin are adjacent after clothing replacement. This type of region may exhibit discontinuous boundaries or unnatural transitions after direct feature coverage.

[0080] The second category consists of UV locations that transition from clothing areas to skin areas. These are areas that were covered by clothing in the original UV feature plane but are no longer covered after the replacement, such as the arm area exposed after replacing long-sleeved clothing with short-sleeved clothing. Since the original UV feature plane lacks corresponding skin feature information in these areas, it needs to be supplemented.

[0081] After identifying the candidate UV regions, the completeness of the feature information corresponding to each UV position in the candidate regions is detected. Specifically, it can be determined whether the UV position belongs to an area with insufficient information based on whether there are valid skin semantic features in the original UV feature plane, or whether its features meet a preset completeness threshold. UV positions with feature information below the completeness threshold are identified as UV positions that need to be repaired, and a mask region for subsequent generative completion is generated accordingly.

[0082] In this embodiment, a diffusion-based generative model is used to complete the features of the masked region. The diffusion model takes the UV feature plane representation after clothing replacement as input and, combined with the masked region, generatively repairs the features corresponding to the masked region in the UV feature plane space. Specifically, during the generation process of the diffusion model, the denoising result predicted by the diffusion model is used as the corresponding UV features within the masked region, while in regions outside the masked region, the known feature values ​​in the original UV feature plane remain unchanged. This can be expressed by the following formula:

[0083] ⊙ ⊙ ,

[0084] Where M represents the mask region. This represents the denoising result predicted by the diffusion model at step t. This represents the known eigenvalues ​​in the original UV feature plane, and the symbol ⊙ indicates element-wise multiplication.

[0085] In this embodiment, the diffusion model can be obtained through pre-training. Before the method of the present invention is executed, the diffusion model is trained offline using UV feature plane data containing multiple human instances, enabling it to learn the ability to generate reasonable human appearance features under given contextual UV feature conditions. During the execution phase of the method of the present invention, the parameters of the diffusion model remain fixed and are only used for generative completion of the UV features corresponding to the mask region.

[0086] To maintain consistency in person identity during skin feature completion, the diffusion model can incorporate an attention mechanism as a conditional constraint. In a specific implementation, semantic information is used to locate the UV region corresponding to the person's face from the UV feature plane, and the UV features of this facial region are extracted as conditional information. This guides the diffusion model to fuse skin color and identity-related feature information corresponding to the person instance when generating newly revealed skin region features, thereby generating a skin appearance consistent with the person instance's identity characteristics. The repair effect is as follows: Figure 5 As shown.

[0087] After completing the static skin feature completion, to further enhance the dynamic effects related to clothing and posture, a latent space feature enhancement module based on dynamics and physical constraints is introduced. In this embodiment, this module performs the following operations:

[0088] Step 4-1: Extract dynamic features and construct the external force driving field;

[0089] Obtain the pose sequence of the skinned multi-person linear model SMPL The corresponding vertex coordinates on the human body surface are: The instantaneous acceleration at the vertex of the human body surface is calculated using the second-order time difference:

[0090] ,

[0091] in, This represents the dynamic response generated by human movement.

[0092] Define a mapping operator P to project the 3D vertex acceleration vectors onto the corresponding UV parameter domains, thus obtaining the 2D dynamic driving field:

[0093] ,

[0094] in, Indicates the coordinate position in the parameter domain The corresponding dynamic driving vector at that location;

[0095] The driving force field is used to characterize the inertial effect and local compression load exerted by human body movements on the clothing area.

[0096] Step 4-2: Construct a physical constraint model;

[0097] Construct a physical energy functional with material constitutive constraints within the UV characteristic space. , where Φ represents the entire UV feature field. Let . This represents the feature value at a specific location (s, r) in the current UV feature plane, combined with the clothing material code. Define the energy function:

[0098] ,

[0099] Here, the gradient operator ∇ represents the first-order differential operator in the parameter domain, used to describe the spatial rate of change of the feature function in the coordinate direction of the UV parameterization plane; the Laplacian operator Δ represents the second-order differential operator in the parameter domain, used to describe the second-order spatial change of the feature function in the UV parameterization plane. and Here, d represents the stiffness parameter obtained from the material coding mapping; d is the differential symbol.

[0100] First item The strain energy term is used to characterize the tensile resistance of the feature in its local neighborhood by restricting the first-order gradient of the feature.

[0101] Second item This is the bending energy term, used to characterize the smoothness of the characteristic distribution and its resistance to curvature changes;

[0102] Third item The term for work done by external forces is used to characterize a two-dimensional dynamic driving force field. The guiding role of characteristic distribution.

[0103] This energy functional is used to characterize the latent space deformation trend under the combined effects of dynamic driving and material constraints.

[0104] Step 4-3, prediction of latent space dynamic response;

[0105] Variational differentiation of the physical energy functional yields the characteristic evolution direction:

[0106] ,

[0107] Among them, the biharmonic operator This represents a fourth-order differential operator that performs two consecutive Laplace operations on the characteristic field;

[0108] δ represents the variational operator used to calculate the derivative of the energy functional with respect to the characteristic function Φ;

[0109] g( ) indicates the feature's position in the parameter domain ( The direction of the physical response at point () is used to characterize the evolution trend of the eigenvalue under the current physical constraints and external driving forces.

[0110] Construct a physical perception offset prediction network, and integrate the UV feature planes. Dynamic driving force field Physical response gradient As a joint input, predict the dynamic displacement bias field in the latent space. :

[0111] ,

[0112] in For dynamic displacement bias field In the parameter domain position ( The specific value to be taken at (). This represents a network that enhances physical perception features.

[0113] The dynamic displacement offset represents the evolution direction of the feature driven by the physical force field in the latent space.

[0114] Step 4-4, Feature Space Reconstruction and Physical Injection;

[0115] The predicted latent space dynamic response is input into the feature reconstruction unit to perform physically consistent spatial mapping and structural reconstruction on the original UV feature plane, generating the enhanced parameter domain position. UV feature plane representation at ) :

[0116] ,

[0117] Where Q represents the feature space transformation operator.

[0118] This process transforms macroscopic dynamic loads into microscopic geometric deformation priors through adaptive adjustment of latent space feature distribution, enabling the subsequent decoder to naturally generate dynamic wrinkles and attitude-related details that conform to mechanical logic during the 3D reconstruction stage.

[0119] Subsequently, the enhanced UV feature plane is input into the pre-trained decoder to parse and generate three-dimensional Gaussian points and their attributes in the normalized pose space. Through forward inference decoding, a set of three-dimensional Gaussian points and their corresponding attributes in the normalized pose space are obtained. These three-dimensional Gaussian points together constitute a representation of the character's three-dimensional appearance.

[0120] To transform the 3D Gaussian points from the normalized pose space to the target pose space, this embodiment utilizes the pose parameters of the parametric human model to perform pose deformation on the 3D Gaussian points. Specifically, each 3D Gaussian point can inherit the skeletal joint influence weights of the surface point based on its correspondence with the surface of the parametric human model under the normalized pose, thereby obtaining skinning weight information associated with the human skeletal structure.

[0121] During posture deformation, the existing linear hybrid skinning method can be used. Based on the transformation parameters of each skeletal joint from the standard posture to the target posture, the position of the three-dimensional Gaussian points is weighted and transformed, so that they deform accordingly with the change of human posture. In this way, the decoded three-dimensional Gaussian points are mapped from the standard posture space to the target posture space to adapt to the required human posture.

[0122] After obtaining the three-dimensional Gaussian points in the target pose space, a three-dimensional Gaussian splatter renderer is used to render the three-dimensional Gaussian points to generate the final two-dimensional image result. During the rendering process, the three-dimensional Gaussian splatter renderer projects the three-dimensional Gaussian points onto the two-dimensional image plane according to the camera parameters, and combines the color, transparency and other attributes of each three-dimensional Gaussian point to generate the corresponding pixel color value, thereby obtaining the two-dimensional rendered image of the human body after the costume change.

[0123] Through the above posture deformation and rendering process, high-quality two-dimensional images of the human body that has completed virtual clothing change can be generated under different target postures and viewpoints.

[0124] Through the above steps, the present invention can construct a high-fidelity three-dimensional human body representation based on multi-view image data, and on this basis realize the virtual replacement of human clothing and generate corresponding two-dimensional rendering results.

[0125] The method proposed in this invention achieves efficient clothing replacement by structurally representing and editing human skin and clothing information in a unified UV space. Furthermore, when clothing changes cause skin to become visible, a generative completion mechanism automatically generates skin information that matches the character's identity, thereby enhancing the naturalness and realism of the virtual clothing change result. The 3D human body generated by this method and its rendering results can be applied to film and television production, virtual reality, games, and related fields of digital humans.

[0126] It should be noted that the above description of the present invention in conjunction with specific embodiments is only for illustrating the technical solutions of the present invention and does not constitute a limitation on the scope of protection of the present invention. For those skilled in the art, various changes, substitutions, or equivalent improvements can be made to the above method steps, module structures, and implementation methods without departing from the technical concept of the present invention, and all such changes should fall within the scope of protection of the present invention. Specific technical means not explicitly described in this embodiment can all be implemented using existing technologies known in the art.

Claims

1. A physically driven UV feature space-enhanced 3D virtual clothing changing method, characterized in that, Includes the following steps: Step 1: For the geometric and appearance reconstruction process of a single digital human, a set of RGB images and camera parameters are used as input. A UV feature plane, i.e., a two-dimensional flattened feature plane, is initialized for each set of RGB images, where U refers to the horizontal coordinate U-coordinate and V refers to the vertical coordinate V-coordinate. The UV feature plane establishes a parameterized correspondence with the three-dimensional surface of the human body under the standard pose. By updating only the UV feature plane parameters through gradient backpropagation, a UV feature plane representation encoding the identity and clothing information of the human instance corresponding to the RGB image sample is obtained. Each pixel in the UV feature plane representation contains the semantic information of the corresponding human body region in the RGB image. Step 2: Based on the semantic information of the corresponding human body region obtained in Step 1, the clothing features in the UV feature plane representation of the target clothing are overlaid onto the original clothing region in the UV feature plane representation of the person to be dressed, thus completing the basic clothing replacement. Step 3: Based on the difference information between the UV feature plane representations before and after clothing replacement, determine the candidate UV regions that are transformed from clothing areas to skin areas after clothing replacement; detect the completeness of the feature information corresponding to each UV position in the candidate UV regions, expand the candidate UV regions at the edges to obtain the mask regions that need to be repaired; use a pre-trained generative model, with the UV feature information of the mask region and its surroundings as conditional input, to perform generative completion on the UV features corresponding to the mask region, generating skin feature information consistent with the identity features of the person instance; Step 4: Perform a UV feature enhancement process based on dynamic implicit force field and physical constraints to obtain an enhanced UV feature plane representation; Step 5: Perform the rendering and pose mapping process.

2. The method according to claim 1, characterized in that, In step 1, each RGB image is processed using an image semantic segmentation tool, and the segmentation results are divided into background, skin, and clothing to be replaced, to obtain a semantic segmentation mask corresponding to each RGB image; In the digital human reconstruction process, a UV feature plane is initialized for each group of RGB images. The UV feature plane is used to represent a 3D Gaussian human body in canonical space. The UV feature plane representation is input into a pre-trained decoder and combined with a 3D Gaussian splash renderer for forward inference to generate the corresponding rendered image. The reconstruction loss between the rendered image and the original RGB image is calculated. By using a backpropagation optimization method, the reconstruction loss is backpropagated, and only the feature values ​​in the UV feature plane are updated and optimized, thereby obtaining the optimal UV feature plane representation in canonical space that encodes the identity, pose, and clothing information of the person instance corresponding to the image sample.

3. The method according to claim 2, characterized in that, In step 3, the pre-trained generative model employs the following strategy when performing content completion: within the masked region, the denoised result predicted by the generative model based on a diffusion mechanism is used as the corresponding UV feature; in regions outside the masked region, the known feature values ​​in the original UV feature plane are kept unchanged. ⊙ ⊙ ， in, This represents the completed UV feature plane, where M represents the mask region. This represents the denoising result predicted by the diffusion model at step t. This represents the known eigenvalues ​​in the original UV feature plane, and the symbol ⊙ indicates element-wise multiplication.

4. The method according to claim 3, characterized in that, In step 3, the pre-trained generative model includes an attention mechanism. The attention mechanism uses features extracted from the facial region represented by the UV feature plane as a conditional constraint, enabling the extended diffusion model to fuse skin color and identity-related feature information corresponding to the person instance when generating skin features of newly exposed skin regions.

5. The method according to claim 4, characterized in that, Step 4 includes: Step 4-1: Extract dynamic features and construct the external force driving field; Obtain the pose sequence of the skinned multi-person linear model SMPL Where T represents the total number of frames in the attitude sequence. Let represent the human pose parameter vector corresponding to frame t. The pose sequence is used to characterize the rotational state of the human skeletal joints. Based on the pose sequence, the set of three-dimensional coordinates of the human surface vertices corresponding to each frame is obtained. Human dynamics feature vectors are calculated using the second-order time difference method: ， in, Let represent the set of acceleration vectors of each vertex on the human body surface in frame t; This represents the set of three-dimensional spatial coordinates of all vertices on the surface of the human body model in frame t+1; By using a predefined correspondence between vertices and parameter domains, the three-dimensional vertex acceleration vectors are converted into two-dimensional dynamic driving fields in the parameter domain; Step 4-2: Construct a physical constraint model; A physical constraint model containing material constitutive constraints is constructed in the parameterized feature plane space to characterize the evolution trend of feature distribution under dynamic driving force. Let Φ represent the overall feature distribution in the current feature plane, and Φ(s, r) represent the feature value at the parameter domain coordinate position (s, r); combined with clothing material coding... Construct a physical energy model that includes the following constraints: The strain constraint term associated with the feature space gradient is used to characterize the intensity of feature variation in the local neighborhood; The bending constraint term, which is related to the second-order transformation of the feature space, is used to characterize the smoothness of the feature distribution. With dynamic driving field The relevant external force response term is used to characterize the effect of dynamic action on the characteristic distribution; By using a physical energy model, a constraint relationship between material properties and dynamic response is established in the latent space; Step 4-3, prediction of latent space dynamic response; Based on the physical constraint model constructed in step 4-2, the response direction of the current feature distribution under the combined action of material constraints and dynamic driving is calculated to characterize the evolution trend of the feature in the latent space. The response direction represents the directional information of how the eigenvalues ​​should change in the parameter domain under physical constraints and external dynamic driving forces. Construct a physical perception feature enhancement network, which takes the following information as joint input: The overall feature distribution Φ in the current feature plane; Dynamic driving fields in the parameter domain ; Characteristic response direction information calculated based on the physical constraint model; The physical perception feature enhancement network predicts the dynamic displacement bias field ΔΦ in the latent space, which represents the update amount of the feature value at each parameter domain position; the dynamic displacement bias field ΔΦ represents the adjustment magnitude of the feature distribution in the latent space under the physical consistency constraint. Step 4-4, Feature Space Reconstruction and Physical Injection; The original UV feature plane is spatially mapped and structurally adjusted using a dynamic displacement bias field ΔΦ to generate an enhanced UV feature plane representation.

6. The method according to claim 5, characterized in that, In step 4-3, the physical perception feature enhancement network recursively fuses the force field of the current frame with the displacement bias of historical moments.

7. The method according to claim 6, characterized in that, In step 4-3, the physical constraint model constructs a deformation energy constraint relationship based on the material properties of clothing. By optimizing the comprehensive response relationship between the deformation energy constraint and the dynamic driving field, the dynamic displacement bias is predicted so that the enhanced UV feature plane satisfies the physical stability constraint in spatial distribution.

8. The method according to claim 7, characterized in that, Step 5 includes: inputting the enhanced UV feature plane representation obtained in step 4 into a pre-trained 3D reconstruction decoder to decode a set of 3D Gaussian points and attribute sets containing geometric and appearance compensation attributes; using the user-defined arbitrary skinned multi-person linear model SMPL pose parameters to perform linear blending skin deformation on the 3D Gaussian points, transforming the 3D Gaussian points from the normal pose space to the target pose space, and using the 3D differentiable rendering representation to render and generate the final image.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, It stores a computer program or instructions that, when run on a computer, perform the steps of the method as described in any one of claims 1 to 8.

Images