3D gaussian digital human generation method and device for cross-domain action generalization

Abstract

This invention provides a method and apparatus for generating 3D Gaussian digital humans with cross-domain motion generalization capabilities, relating to the field of 3D digital human technology. The method includes: identifying the state of a pre-rendered result of a basic 3D Gaussian digital human; and generating a target motion video through a video diffusion model when the state of the pre-rendered result is abnormal and meets preset conditions. The video diffusion model is a generative artificial intelligence network based on a diffusion denoising process. This invention, by identifying the state of the pre-rendered result of a basic 3D Gaussian digital human and generating a target motion video through a video diffusion model when the state is abnormal and meets preset conditions, directly avoids the physical distortion caused by forcibly restricting input through constraint methods. It effectively achieves high-quality cross-domain motion generation, thus solving the problem of rendering instability phenomena such as human body fragmentation and outline flickering when 3D Gaussian digital humans encounter complex cross-domain motions not covered by the training set, such as large twisting movements, in existing technologies. This significantly improves the visual stability of 3D digital humans under cross-domain motion.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional digital human technology, and in particular to a method and apparatus for generating 3D Gaussian digital humans with cross-domain motion generalization. Background Technology

[0002] Existing 3D Gaussian digital humans typically involve pre-training an animated digital human model for a specific person's identity. During deployment, the input includes pose, camera parameters, or skeletal drive signals. A neural network then predicts rendering attributes such as the color, position, scale, rotation, and opacity of Gaussian points, anchor points, or neural textures, before outputting a human image via a renderer. This method generally performs well within the training motion distribution, but it is prone to instability issues such as hand and foot tremors, clothing color shifts, flickering of the human silhouette, and ray artifacts when encountering large twisting or extreme limb extension movements not covered in the training set.

[0003] To address the rendering instability issues caused by cross-domain actions, existing technologies typically employ methods such as principal component analysis projection, pose constraint, or temporal filtering during the testing phase. These methods attempt to pull the input target action back to or limit it as much as possible within the known training action distribution range of the original model, thereby preventing the model output from deviating from the stable region during training and thus maintaining the basic visual integrity of the basic 3D digital human when facing complex actions.

[0004] The aforementioned constraints can essentially only limit the motion input from deviating from the training range; they cannot truly generate new pose details that the basic digital human has not learned. When encountering complex cross-domain movements, existing methods often forcibly reduce the amplitude of movements, or still suffer from rendering quality degradation issues such as body fragmentation and ray artifacts. Therefore, in order to improve the visual stability of 3D digital humans under cross-domain movements, how to achieve high-quality cross-domain motion generation has become an urgent problem to be solved in this field. Summary of the Invention

[0005] This invention provides a 3D Gaussian digital human generation method and apparatus for cross-domain motion generalization, in order to solve the technical problem of how to achieve high-quality cross-domain motion generation.

[0006] This invention provides a method for generating 3D Gaussian digital humans with cross-domain motion generalization, comprising: Identify the state of the pre-rendered result of the basic 3D Gaussian digital human; If the state of the pre-rendered result is abnormal and the preset conditions are met, the target action video is generated through the video diffusion model. The video diffusion model is a generative artificial intelligence network based on the diffusion denoising process.

[0007] According to the present invention, a 3D Gaussian digital human generation method for cross-domain motion generalization is provided, which identifies the state of the pre-rendered result of the basic 3D Gaussian digital human, including: The pre-rendered result, target action sequence, human body contour change information, temporal consistency features, and internal state features of the basic 3D Gaussian digital human are input into the risk discrimination network to obtain the state of the pre-rendered result output by the risk discrimination network. The risk discrimination network is a deep learning model used to predict the rendering failure probability of the basic 3D Gaussian digital human under the target action and to calculate the slow-track benefit.

[0008] According to the present invention, a 3D Gaussian digital human generation method for cross-domain action generalization is provided, wherein the risk discrimination network is trained based on the following steps: The initial digital human model is trained based on the training action sequence of the sample person to obtain the basic three-dimensional Gaussian digital human. The basic 3D Gaussian digital human is used to render the action sequence to obtain the sample pre-rendering result, sample human body contour information, temporal change characteristics of adjacent frames, and internal state characteristics of the basic 3D Gaussian digital human. The sample pre-rendering results are compared with the sample real video data to determine the risk label corresponding to the reserved action sequence; Based on the sample pre-rendering results, the reserved action sequence, the sample human body contour information, the temporal change features of adjacent frames, the internal state features of the basic three-dimensional Gaussian digital human, and the risk label, the initial discrimination network is trained to obtain the risk discrimination network.

[0009] According to the present invention, a 3D Gaussian digital human generation method for cross-domain motion generalization is provided, which generates target motion videos through a video diffusion model, including: Based on the aforementioned basic 3D Gaussian digital human, identity condition information is extracted; Extract motion control information based on the target action; The identity information and the action control information are input into the video diffusion model to obtain the target action video output by the video diffusion model.

[0010] According to the present invention, a 3D Gaussian digital human generation method for cross-domain motion generalization is provided, wherein the video diffusion model is trained based on the following steps: The initial diffusion model is trained based on sample identity information containing the appearance attributes of the sample digital human and sample action control information containing the target posture sequence to obtain the video diffusion model.

[0011] According to the present invention, a 3D Gaussian digital human generation method for cross-domain motion generalization is provided, wherein the preset condition includes that no action Gaussian extension package matching the target action is retrieved in the motion memory.

[0012] According to the 3D Gaussian digital human generation method for cross-domain motion generalization provided by the present invention, after generating the target motion video through a video diffusion model, it further includes: Reconstruct a dynamic 3D Gaussian representation bound to the base 3D Gaussian digital human based on the target motion video; The dynamic 3D Gaussian representation is encapsulated as an action Gaussian extension package and saved to the action memory library.

[0013] According to the 3D Gaussian digital human generation method for cross-domain motion generalization provided by the present invention, after identifying the state of the pre-rendered result of the basic 3D Gaussian digital human, it further includes: If the pre-rendered result is in an abnormal state and does not meet the preset conditions, rendering is performed in conjunction with the retrieved Action Gaussian extension pack.

[0014] According to the 3D Gaussian digital human generation method for cross-domain motion generalization provided by the present invention, after identifying the state of the pre-rendered result of the basic 3D Gaussian digital human, it further includes: If the pre-rendered result is in a normal state, the pre-rendered result is output directly.

[0015] The present invention also provides a 3D Gaussian digital human generation device for cross-domain motion generalization, comprising: The recognition module is used to identify the state of the pre-rendered result of the basic 3D Gaussian digital human; The generation module is used to generate a target motion video through a video diffusion model when the state of the pre-rendered result is abnormal and the preset conditions are met. The video diffusion model is a generative artificial intelligence network based on the diffusion denoising process.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the 3D Gaussian digital human generation method for cross-domain action generalization as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the 3D Gaussian digital human generation method for cross-domain action generalization as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the 3D Gaussian digital human generation method for cross-domain action generalization as described above.

[0019] The present invention provides a 3D Gaussian digital human generation method and apparatus for cross-domain motion generalization. By identifying the state of the pre-rendered result of the basic 3D Gaussian digital human, and generating the target motion video through a video diffusion model when the state is abnormal and the preset conditions are met, the present invention directly avoids the physical distortion caused by the forced restriction of input by constraint means, and effectively realizes high-quality cross-domain motion generation. This solves the problem of rendering instability such as human body fragmentation and outline flickering when the 3D Gaussian digital human encounters complex cross-domain motions such as large twisting that are not covered by the training set in the prior art, and significantly improves the visual stability of the 3D digital human under cross-domain motion. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the 3D Gaussian digital human generation method for cross-domain action generalization provided by the present invention.

[0022] Figure 2 This is the overall technical roadmap for the 3D Gaussian digital human generation method oriented towards cross-domain action generalization provided by the present invention.

[0023] Figure 3 This is the training and deployment inference graph of the risk discrimination network provided by the present invention.

[0024] Figure 4 This is a training architecture diagram of the video diffusion model provided by the present invention.

[0025] Figure 5 This is a schematic diagram of the structure of the 3D Gaussian digital human generation device for cross-domain motion generalization provided by the present invention.

[0026] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] The following is combined Figures 1 to 6 This invention describes a method and apparatus for generating 3D Gaussian digital humans with cross-domain motion generalization.

[0029] Figure 1 This is a flowchart illustrating the 3D Gaussian digital human generation method for cross-domain action generalization provided by the present invention, as shown below. Figure 1 As shown, the method includes, but is not limited to, steps S1 and S2.

[0030] Step S1: Identify the state of the pre-rendered result of the basic 3D Gaussian digital human.

[0031] A basic 3D Gaussian digital human is a pre-trained, deployable 3D human digital model designed for a specific person. It can be a digital human represented using 3D Gaussian points or anchor points, primarily represented through neural networks and a rendering engine. The backbone network of a basic 3D Gaussian digital human only learns the mapping from pose to rendering attributes on a finite distribution of actions. When the input action falls outside the training distribution, the backbone network will still output numerical results, but its latent variables or Gaussian attributes may deviate from the stable region established during training. The renderer will amplify this output space abnormally, creating visible artifacts.

[0032] The pre-rendered result is a tentative rendered image generated for the input action before the formal output of the basic 3D Gaussian digital human. It can be a low-resolution keyframe image or a local rendered video clip with a short time window.

[0033] The state of the pre-rendered result refers to the performance of the pre-rendered result in terms of visual quality and physical rationality, including the normal state of a complete and stable image, as well as the abnormal state of failure phenomena such as broken human body, hand and foot shaking, outline misalignment, ray artifacts, color drift, inconsistent identity, or cross-frame flickering.

[0034] State recognition can be performed by extracting visual features from the pre-rendered results using image quality assessment algorithms, by calculating the rate of change between adjacent frames using temporal consistency detection algorithms, or by comparing the difference between the rendered results and the expected human mask to perform feature-level state recognition.

[0035] For example, combined Figure 2 When a target action is received, and this action is a cross-domain action, the system first uses the fast path (i.e., fast track) of the basic 3D Gaussian digital human to render the first few low-resolution images as pre-rendering results. Then, through mechanisms such as the fast track pre-rendering failure detection module, the jitter frequency of the hand and foot pixels in the image is calculated, thereby accurately identifying that the pre-rendering result is in an abnormal state of hand and foot jitter. Cross-domain actions are complex actions that are not covered or insufficiently covered in the training action distribution, such as yoga or large twisting movements.

[0036] Step S1 can intervene in the rendering process of digital humans in advance, and detect the risk of rendering failure of the basic model for the current unknown action in time before the error or low quality crash screen is officially output to the user, thus providing a basis for judgment on whether to switch the action generation method in the future.

[0037] Step S2: If the pre-rendered result is abnormal but meets the preset conditions, generate the target motion video using the video diffusion model.

[0038] The preset conditions are the prerequisites that must be met to invoke the video diffusion model. The video diffusion model is a generative artificial intelligence network based on the diffusion denoising process. It can be an image-to-video diffusion model, or a text- or multimodal condition-guided video diffusion model.

[0039] The target motion video is the final generated two-dimensional dynamic image that shows a specific person smoothly performing a target action. It can be a continuous frame sequence containing RGB pixel channels.

[0040] By inputting the characteristics of the target action as a guiding condition into the video diffusion model, the video diffusion model can start from random Gaussian noise, undergo multiple iterations of noise reduction, and gradually recover clear and continuous video frames that meet the action requirements and the geometric features of the person, thereby generating the required target action video.

[0041] For example, when an abnormal state of large-area body fragmentation is detected in the pre-rendered result of a certain difficult yoga pose, and this meets preset conditions, the rendering of the basic 3D Gaussian android is stopped, and instead, the yoga pose sequence is input into... Figure 2 In the slow-motion video diffusion teacher model shown, the model generates a 2D target motion video of the digital human smoothly performing the yoga pose through multi-step denoising inference. This process completely replaces the erroneous approach in existing technologies that attempt to forcibly push the target motion back into the training distribution through PCA projection, temporal filtering, or Clamp, which may still result in ray-like artifacts and fragmentation in the image. PCA stands for Principal Component Analysis projection; Clamp can be numerical truncation / pose constraint.

[0042] When the basic 3D model is unable to handle complex cross-domain actions due to the limited distribution of training data actions, step S2 can make full use of the powerful generalization ability of the generative diffusion model on out-of-distribution data to regenerate the correct visual content, avoiding the direct output of the crashed screen.

[0043] As described above, this invention identifies the state of the pre-rendered result of the basic 3D Gaussian digital human and generates the target action video through a video diffusion model when the state is abnormal and the preset conditions are met. This directly avoids the physical distortion caused by the forced restriction of input by constraint methods and effectively realizes high-quality cross-domain action generation. This solves the problem of rendering instability such as human body fragmentation and outline flickering when the 3D Gaussian digital human encounters complex cross-domain actions such as large twisting that are not covered by the training set in the prior art. It significantly improves the visual stability of the 3D digital human under cross-domain actions.

[0044] In one embodiment, step S1 may further include: The pre-rendered results, target action sequence, human body contour change information, temporal consistency features, and internal state features of the basic 3D Gaussian digital human are input into the risk discrimination network to obtain the state of the pre-rendered results output by the risk discrimination network.

[0045] The risk discriminant network is a deep learning model used to predict the rendering failure probability of a basic 3D Gaussian digitizer under target actions and to calculate slow-track gains; it is also called a frozen failure discriminator. Combined with... Figure 3 The risk discrimination network can be a cross-identity sharing model or an image sequence classification network based on a combination of convolutional neural networks and visual transformers.

[0046] The tensor data of the pre-rendered results, the features of the target action sequence, the human body contour change information, the temporal consistency features calculated based on adjacent frames, and the internal state features of the basic 3D Gaussian digital human during the rendering process can be synchronously transmitted to the input layer of the risk discrimination network for multimodal joint calculation; combined with Figure 3 Furthermore, the individualized training distribution statistics of the current basic digital human can be transmitted to the network as input, without having to retrain the entire network for each new digital human. The risk discrimination network extracts features and performs forward propagation calculations on the input multimodal data through its internal network layers, ultimately outputting a probability value or classification label as the result. Specifically, it can output a failure probability score between 0 and 1, or directly output a discrete state label of "normal" or "abnormal". Here, "normal" corresponds to low-risk actions, and "abnormal" corresponds to high-risk actions.

[0047] For example, combined with Figure 2The input target action is first subjected to a quick trial rendering, and then the pre-rendered result containing several low-resolution frames is transmitted to... Figure 2 The network employs a freeze failure discriminator. After forward computation and evaluation, the output indicates that the probability of this action window causing fast path rendering failure is 85%, thus determining that it is in an abnormal state and classifying it as a high-risk action. This directs subsequent data flow to the generation path of the slow-moving video diffusion teacher.

[0048] This invention overcomes the limitations of relying on simple action amplitude thresholds or manual rules for judgment by inputting the pre-rendered results into a risk discrimination network and using the network model to output the state of the pre-rendered results. It effectively improves the automation level and prediction accuracy of cross-domain action rendering abnormal state identification.

[0049] In one embodiment, the risk discrimination network is trained based on the following steps: The initial digital human model is trained based on the training action sequence of the sample person to obtain the basic three-dimensional Gaussian digital human. By rendering the action sequence of the basic 3D Gaussian digital human, we can obtain the sample pre-rendering results, sample human body contour information, temporal change characteristics of adjacent frames, and internal state characteristics of the basic 3D Gaussian digital human. The pre-rendered results of the samples are compared with the real video data of the samples to determine the risk labels corresponding to the reserved action sequences; Based on the sample pre-rendering results, the reserved action sequence, the sample human body contour information, the temporal change characteristics of adjacent frames, the internal state characteristics of the basic 3D Gaussian digital human, and the risk label, the initial discrimination network is trained to obtain the risk discrimination network.

[0050] The sample individuals are the subjects collected for training the base model, and can be video datasets of multiple individuals with different body types and clothing features. The training action sequences are the basic action data selected during the training phase to guide the model in learning the appearance of individuals in normal states; these can be time series of simple actions such as walking, turning, and waving. The initial digital human model is a 3D Gaussian network model with randomly initialized parameters or one that has not yet been fitted to the body features of the current sample individual; it can be a spatial point cloud architecture that has not undergone gradient descent optimization.

[0051] The photometric error between the initial rendering output of the model and the objective real video frame can be calculated by using the L1 loss function, the backpropagation algorithm can be used to calculate the network gradient, and the parameter optimizer can be used to continuously update the three-dimensional Gaussian properties of the initial digital human model, including position, scale, rotation, opacity and spherical harmonic coefficients, until the model fully converges, thereby obtaining a basic three-dimensional Gaussian digital human that can accurately reconstruct the appearance of the sample person.

[0052] The reserved motion sequences are independent motion verification segments that were deliberately retained during the construction of the basic digital human and were not used. These can be exaggerated stretching movements, rapid dance movements, etc., that differ significantly from the movements in the training. The sample pre-rendering results are image sequences with test properties generated by the basic digital human when faced with the above-mentioned unknown movements. These can be a set of test frames containing ray artifacts or limb fragmentation caused by unknown postures.

[0053] The pose parameters in the action sequence, such as the human skeleton rotation matrix, can be left out and input into the backbone network of the basic 3D Gaussian digital human to predict the Gaussian parameter attributes after deformation. Then, the 3D spatial 3D points are projected onto the 2D pixel plane through the 3D Gaussian rasterization renderer to calculate the color and transparency of each pixel, and finally obtain a series of 2D sample pre-rendered image frames.

[0054] The sample real video data is an objective, real video benchmark captured synchronously during the acquisition of the action sequence. It can be a high-definition RGB video frame, an image combination containing accurate human body masks, or precise contour information. The human body mask is a binary segmentation image used to distinguish the foreground and background of the human body. The risk label is a supervisory signal characterizing the rendering quality under a specific action window. It can be a discrete value of 0 representing low-risk samples / normal and a discrete value of 1 representing high-risk samples / abnormal, or it can be a continuously distributed quality degradation score.

[0055] The pre-rendered sample can be compared pixel-level and feature-level with real video frames, human masks, human contours, and identity references. Computer vision algorithms automatically determine if the rendered result contains artifacts such as fragmented human figures, hand and foot tremors, contour misalignment, ray artifacts, color drift, identity inconsistencies, or cross-frame flickering. If the difference in rendering quality from the real video is within a normal threshold, the leaveout action sequence is identified and labeled as low-risk; if the rendering quality significantly degrades, it is identified and labeled as high-risk. It should be noted that leaveout actions are not necessarily equivalent to high-risk actions; the true supervisory label comes from the degree of quality degradation of the base fast path rendering result relative to the real video.

[0056] The internal state features of the basic 3D Gaussian digital human are intermediate network variables during the rendering process of the digital human performing this action. These can include intermediate latent variables, Gaussian scale, opacity change rate, anchor point output change rate, or other internal network representation features. The initial discriminator network is a classification or regression network structure whose parameters have not yet been optimized. It can be a randomly initialized spatiotemporal convolutional neural network or a multilayer feedforward network.

[0057] Combination Figure 3The initial discrimination network can be fed into a large number of pre-rendered samples generated from basic fast paths, including action sequences, human body contour information, temporal variation features of adjacent frames, and intermediate latent variables, Gaussian scale, opacity, anchor point output, or other internal state features of a basic 3D Gaussian digital human. The network's output predicted probability is compared with the prepared risk labels using loss functions such as cross-entropy. The parameters of the initial discrimination network are continuously updated through backpropagation, ensuring comprehensive differences between normal and failed rendering results of the deep learning model in terms of image, temporal sequence, and internal state. After training with multi-person, multi-action, and multi-clothing data and freezing the parameters, a cross-identity risk discrimination network with high-precision predictive capabilities is finally obtained.

[0058] This invention generates supervised risk labels by constructing a rigorous comparison between real video data and pre-rendered results, and trains the initial network with multi-dimensional data such as internal state features, so that the trained risk discrimination network can learn the essential differences in failure states, which significantly improves the accuracy and reliability of the network in predicting the state of pre-rendered results under unknown actions.

[0059] In one embodiment, step S2, generating the target motion video using a video diffusion model, may further include: Extract identity information based on the basic 3D Gaussian digital human; Extract motion control information based on the target action; The identity information and motion control information are input into the video diffusion model to obtain the target motion video output by the video diffusion model.

[0060] Identity information is a set of data that can characterize the unique appearance and geometric spatial features of a specific digital person, corresponding to Figure 4 The Gaussian identity package in the image can include frontal images, side images, canonical pose images, historical frames or multiple viewpoint images of the person, and can also include auxiliary geometric information generated by rendering, such as alpha masks, depth maps, normal maps, and human body part maps, to comprehensively represent the digital person's body shape, clothing, texture, and geometric structure. Specifically, the alpha mask represents the data channel for image transparency, the depth map is an image reflecting the distance from the object's surface to the camera, and the normal map is an image recording the details of the object's surface normal vectors.

[0061] Motion control information is a set of data used to guide the generative model to produce specific body posture changes from spatial and temporal dimensions. Figure 4The motion control package in the code can be formatted information derived from the target SMPL or SMPL-X motion sequence. It includes one or more combinations of elements such as a 2D skeleton connected graph, an SMPL human forward rendering graph, a human mask, joint maps, joint velocity vectors, camera parameters, and a temporal motion window. SMPL stands for Skinned Multi-Person Linear Model, representing a vertex- and skeleton-based 3D parametric human model.

[0062] Combination Figure 4 The decoupled identity information is converted into a high-dimensional feature vector through a feature injection network and injected into the attention layer of the diffusion model. Simultaneously, motion control information is spatially concatenated and jointly fused with the video's noise tensor through an action encoding module. Based on the existing image-to-video skeleton, this video diffusion model performs denoising operations under the strong joint constraints of the aforementioned appearance and motion conditions, gradually eliminating noise and ultimately outputting a 2D target motion video that fully satisfies the specified identity features and follows the temporal changes of complex gymnastic postures.

[0063] This invention extracts identity information related to the basic model and action control information related to the target, and inputs them into the diffusion model. This achieves dual decoupling and precise control of the appearance features and limb movement patterns of the person during the video generation process, effectively ensuring that the generated target action video has both high identity consistency and action accuracy.

[0064] In one embodiment, the video diffusion model is trained based on the following steps: The initial diffusion model is trained based on sample identity information containing the appearance attributes of the sample digital human and sample motion control information containing the target posture sequence to obtain the video diffusion model.

[0065] The sample identity condition information serves as the baseline conditional data used by the network to learn various human appearance patterns during the training phase. This can be high-quality RGB images, alpha channels, and depth maps representing different body types and clothing from multiple base 3D Gaussian digital humans. The target pose sequence is the action timeline data baseline that the network must align and generate during training. This can be a real SMPL or SMPL-X skeletal motion sequence that strictly corresponds to the video. The sample motion control information is a formatted input matrix derived from the target pose sequence, convenient for convolutional networks to process. This can be a frame-by-frame mapped SMPL rendering or a combination of 2D skeleton masks. The initial diffusion model is a raw network with a standard generative denoising network architecture but not yet learned 3D Gaussian conditional injection and joint resolution capabilities. This can be a large-scale open-source image-to-video model or a base diffusion model structure pre-trained on massive amounts of natural video.

[0066] Combination Figure 4 First, based on the existing video diffusion model skeleton, independent identity encoder and motion encoder branches are added to construct an initial model. Paired sample identity condition information and sample motion control information are input into the initial model in large quantities, requiring the model to generate predicted motion videos. Then, the diffusion reconstruction loss between the predicted video and the corresponding real motion video sequence is calculated, such as noise prediction error. The network weights of the newly added identity encoder, motion encoder, and conditional injection adapter are specifically optimized using the backpropagation algorithm. After massive batches of gradient descent and iterative training, the parameters of each newly added module converge, ultimately resulting in a video diffusion model that can accurately parse input conditions in slow paths and generate highly consistent identities and actions that closely follow the target SMPL sequence.

[0067] This invention utilizes sample identity condition information and sample action control information containing specific attributes to perform targeted constraint training on the initial model, enabling the resulting video diffusion model to possess the unique ability to deeply analyze appearance attributes and target pose sequences, thereby significantly improving the physical fidelity of the image when generating complex action videos.

[0068] In one embodiment, the preset condition of the present invention may include not finding an action Gaussian extension package that matches the target action in the action memory.

[0069] An action memory is a database or file collection used for long-term storage and unified management of a collection of three-dimensional assets generated by a digital human after performing specific complex actions. The database records detailed index information such as the action signature, risk feature signature, confidence index, applicable posture range, and historical cache status of the asset.

[0070] The Action Gaussian extension package is a highly reusable collection of dynamic 3D features generated and reconstructed using slow-track methods. It can be a data packet format containing dynamic 3D Gaussian point cloud parameters that change over time, pose, or motion. This data packet has a tight geometric binding relationship with the original base digitized human and can be cached, fused, rendered, or distilled back into the base model's backbone network at any time. The dynamic 3D Gaussian point cloud parameters can include time-varying Gaussian center positions, 3D scale matrices, spherical harmonic coefficients, etc.

[0071] The action signature of the currently input target action can be extracted and mathematically similar to a large number of historical action signatures pre-stored in the action memory. If the similarity is less than a preset threshold, it is considered a hit, that is, a matching action Gaussian extended package has been retrieved. If there are no historical records with similarity below the threshold in the library, that is, the action memory has not been hit, the preset condition of not being retrieved is met, and subsequent steps are triggered.

[0072] For example, when a user drives a virtual anchor android to perform an extremely complex backflip, the pre-rendering discriminator network detects a serious anomaly in the tentative pre-rendering results. Before deciding whether to immediately invoke the computationally expensive video diffusion model to generate the video, it first searches the android's dedicated motion memory for the existence of a "backflip" 3D asset through motion signatures. Since this is the first time the android has performed a backflip, the motion memory is not matched, indicating that no matching Gaussian extension package for the motion was found, thus satisfying the preset condition. Therefore, the slow video diffusion teacher model is then officially triggered to begin high-fidelity frame-level video generation.

[0073] This invention uses "no matching Gaussian extended package of actions found in the action memory" as a preset condition, which effectively avoids redundant generation calculations for repeated identical actions, thereby reducing the computational overhead of the system and improving generation efficiency.

[0074] In one embodiment, after generating the target motion video using a video diffusion model, the method of the present invention may further include: A dynamic 3D Gaussian representation based on the target motion video and bound to a basic 3D Gaussian digital human is reconstructed. The dynamic 3D Gaussian representation is encapsulated as an Action Gaussian extension package and saved to the Action Memory.

[0075] Dynamic 3D Gaussian representation is a 3D scene representation data structure in which parameters change dynamically with time, specific actions, or continuous postures. It can be a 4D Gaussian point cloud data set that incorporates a spatial deformation field or a time dimension channel. This representation is freed from the constraints of a fixed viewpoint and can be used to render continuous 3D human body videos of target actions from a free viewpoint in full space.

[0076] Combination Figure 2 This invention goes beyond simply using the generated 2D video as a temporary replacement image. Instead, it treats each frame of the generated 2D target action video as a rigorous supervisory image. By combining the camera's physical parameters corresponding to the action sequence, accurate human mask information, and prior knowledge of the digital human's apparent identity, and employing mature 3D reconstruction algorithms or introducing a multilayer perceptron network, it reverse-optimizes and derives the 3D temporal offset and dynamic deformation matrix of the basic Gaussian points at different time steps. This successfully upscales the 2D video, reconstructing a dynamic 3D Gaussian representation with a 3D spatial structure and tightly bound to the original identity. The multilayer perceptron (MLP) is used in this process.

[0077] Then, the newly reconstructed and converged dynamic 3D Gaussian parameter matrix, along with key metadata such as the current action's specific category signature or sequence features, the applicable pose range suitable for this package's intervention, identity consistency score, and the evaluation score of the reconstruction confidence, are packaged and compressed together to generate a data format with a specific file extension, namely the Action Gaussian Extension Package. This package is then saved to the persistent storage space of the action memory and a high-efficiency cache index is established in the relational database for fast lookup. When encountering the same or similar actions later, this Action Gaussian Extension Package can be directly called for rendering, or it can be fused with the basic 3D Gaussian digital human, or it can be used as supervisory data distilled back into the basic digital human's action-driven network. Thus, the slow path not only solves the display problem of the current out-of-distribution action but also transforms new actions into reusable 3D motion capabilities for the basic digital human.

[0078] This invention reconstructs a dynamic three-dimensional Gaussian representation and encapsulates and saves it to an action memory library. This not only transforms the generated two-dimensional image into a three-dimensional asset with higher degrees of freedom, but also enables the persistent storage and accumulation of the digital human's new motion capabilities, greatly enhancing the system's ability to reuse complex motion assets.

[0079] In one embodiment, after step S1, the method of the present invention may further include: If the pre-rendered result is abnormal and does not meet the preset conditions, rendering is performed by combining the retrieved Action Gaussian extension package.

[0080] The preset conditions are not met, meaning that a successful match was found in the action memory bank. The action Gaussian extension package that is the same as or highly similar to the target action to be performed is retrieved and found.

[0081] When the engine detects an anomaly in the current motion pre-rendering and the motion memory matches the motion, it can load the newly retrieved motion Gaussian extension package file. The rendering engine then seamlessly integrates the mature dynamic 3D Gaussian point cloud parameters contained in the extension package with the base model of the basic 3D Gaussian digital human in memory space, performs efficient 3D Gaussian rasterization projection calculation, and finally renders the correct, high-quality continuous motion footage directly in the screen buffer.

[0082] This invention directly combines the retrieved extension package for rendering when the pre-rendering state is abnormal but does not meet the preset conditions, avoiding the problem of triggering a lengthy generation process every time the same or similar complex actions are encountered, and significantly improving the rendering speed and real-time response of the system in scenarios where similar complex actions are reused.

[0083] In one embodiment, after step S1, the method of the present invention may further include: If the pre-rendered result is in a normal state, output the pre-rendered result directly.

[0084] A normal status means that after evaluation by the network and algorithm, the pre-rendered result is determined to have no obvious visual degradation defects, and all quality indicators have not exceeded the attenuation threshold, and the overall process is classified as a low-risk action.

[0085] Combination Figure 2 When the pre-rendered image triggered by the target action is output as a normal state (i.e., a low-risk action) by the risk assessment network, the current action is deemed safe and does not need to enter the slow generation path. Therefore, an instruction can be issued to the rendering engine, requesting it to continue using the fast path of the basic 3D Gaussian digital human, i.e. Figure 2 The fast track in the model inputs the normal posture parameters of the current action into the base model for rapid rasterization in the original way, and sends the pre-rendered image frame or the high-resolution continuous rendering frame completed in sequence directly into the frame buffer of the graphics hardware for formal video playback.

[0086] This invention selects to directly output the pre-rendered result when the pre-rendered result is in a normal state, which ensures the simplification and directness of most routine action processing paths, and maintains the efficient real-time rendering performance and smooth interaction of the digital human when performing most routine actions to the greatest extent.

[0087] As can be seen from the foregoing, this invention, for a 3D digital human that has already been trained and deployed, when encountering complex actions not covered in the training set and experiencing rendering instability, automatically detects output space anomalies without fully retraining the basic digital human or sacrificing the real-time rendering speed of regular actions; maintains real-time fast rendering in low-risk frames; and in high-risk actions, calls a slow-speed teacher model to generate a video sequence of the identity performing the target action, and further converts the video into a dynamic 3D Gaussian representation or action Gaussian extension package bound to the original basic digital human; finally, the action capability is fused or distilled back into the basic digital human, enabling the same or similar actions to be reused faster and more stably in the future.

[0088] In the task of generating and stabilizing the out-of-distribution motion of 3D Gaussian digital humans, the advantages of the basic 3D Gaussian digital human lie in its fast rendering speed of routine motions and good consistency between identity and clothing, making it suitable for real-time interaction. Its disadvantage is the limited distribution of training motions, making it prone to human body fragmentation, jitter, outline flickering, and ray artifacts when encountering complex new motions such as large twists, occlusions, self-contact, yoga, and dance. The advantage of the video diffusion model is its ability to generate relatively natural human motion videos based on identity references and motion conditions. Its disadvantage is that the generated results are usually only two-dimensional frame sequences, unable to directly form a 3D motion capability that can be rendered from a free-viewpoint, reused, and managed uniformly with the original digital human. This invention combines the two: routine motions continue to be rendered quickly by the basic 3D Gaussian digital human, while high-risk motions are generated by the video diffusion model, which further reconstructs a dynamic 3D Gaussian representation bound to the original basic digital human.

[0089] The core of this invention lies in combining the strengths of "fast 3D Gaussian rendering" and "slow video diffusion generation." It doesn't simply replace digital human rendering with video generation results, nor does it merely use PCA, Clamp, or filtering to push complex movements back into the training distribution. Instead, it forms a fast-slow dual-track closed loop: the risk assessment module determines when the slow track is needed; the video diffusion teacher generates identity-consistent target action videos for difficult movements; the dynamic 3D Gaussian reconstruction module converts 2D videos into 3D action assets; and the action memory library saves, retrieves, merges, or distills these assets back into the base digital human. Thus, this invention simultaneously solves the challenges of real-time performance of conventional movements, the generation of out-of-distribution movements, and the 3D reusability of new movements.

[0090] Of course, a fast rendering path based on a basic 3D Gaussian digital human can be used, and during the testing phase, PCA projection, pose clapping, temporal filtering, or output space constraints can be used to pull the target motion back to the training motion distribution as much as possible. This approach is simple to implement and relatively fast, but it essentially only restricts the input or output from deviating from the training range and cannot truly generate new pose details that the basic digital human has not learned. When encountering complex movements such as large twists, occlusions, self-contact, yoga, or dance, the amplitude of the movements may be reduced, or problems such as body fragmentation, hand and foot tremors, and outline flickering may still occur. Therefore, this approach can serve as a low-cost fast path or a control scheme, but it is difficult to replace the slow video diffusion teacher and motion Gaussian memory mechanism in this invention.

[0091] Alternatively, a video generation model can be used entirely. When a high-risk action occurs, a 2D video is directly generated based on the reference image and the target action, and this video is used to replace the original rendering result of the 3D Gaussian digital human. This approach can produce relatively natural videos from a fixed perspective, but the generated result is usually just a 2D frame sequence, which cannot naturally support free-viewpoint rendering, 3D interaction, action reuse, and unified management with the original base digital human. Each time a similar action is encountered, it may still be necessary to regenerate the video, resulting in high computational costs. Therefore, this invention further reconstructs the generated video into a motion-conditional dynamic 3D Gaussian and saves it as an action Gaussian extension package, enabling a slow generation result to be transformed into reusable 3D digital human motion capabilities.

[0092] Therefore, this invention uses a risk discrimination network to identify potential failure windows for the basic digital human's actions. Within high-risk windows, it invokes a slow-motion video diffusion teacher to generate more stable target action videos, improving visual stability under out-of-distribution actions. Through fast and slow dual-track scheduling, low-risk actions are still rendered in real-time by the original basic 3D Gaussian digital human, maintaining the real-time performance of regular actions without reducing the frame rate of regular interactions due to the slow-motion teacher. Enhanced reusability of new actions: By reconstructing the generated video into a dynamic 3D Gaussian based on action conditions and encapsulating it into an action Gaussian extension package, the result of a single slow-motion generation can be repeatedly invoked in subsequent similar actions, enhancing the reusability of new actions.

[0093] The following describes the 3D Gaussian digital human generation device for cross-domain motion generalization provided by the present invention. The 3D Gaussian digital human generation device for cross-domain motion generalization described below can be referred to in correspondence with the 3D Gaussian digital human generation method for cross-domain motion generalization described above.

[0094] like Figure 5 As shown, the 3D Gaussian digital human generation device for cross-domain motion generalization provided by the present invention includes: The recognition module is used to identify the state of the pre-rendered result of the basic 3D Gaussian digital human; The generation module is used to generate target motion video using a video diffusion model when the pre-rendered result is in an abnormal state and meets preset conditions. Among them, the video diffusion model is a generative artificial intelligence network based on the diffusion denoising process.

[0095] Figure 6 A schematic diagram of the physical structure of an electronic device is provided. This device may include a processor, a communications interface, memory, and a communication bus, wherein the processor, communications interface, and memory communicate with each other via the communication bus. The processor can invoke logical instructions from the memory to execute a 3D Gaussian digital human generation method oriented towards cross-domain action generalization.

[0096] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program that can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the 3D Gaussian digital human generation method for cross-domain action generalization provided by the above methods.

[0098] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the 3D Gaussian digital human generation method for cross-domain action generalization provided by the methods described above.

[0099] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating 3D Gaussian digital humans with cross-domain motion generalization, characterized in that, include: Identify the state of the pre-rendered result of the basic 3D Gaussian digital human; If the state of the pre-rendered result is abnormal and the preset conditions are met, the target motion video is generated through the video diffusion model. The video diffusion model is a generative artificial intelligence network based on the diffusion denoising process.

2. The 3D Gaussian digital human generation method for cross-domain motion generalization according to claim 1, characterized in that, Identify the state of the pre-rendered result of the basic 3D Gaussian digital human, including: The pre-rendered result, target action sequence, human body contour change information, temporal consistency features, and internal state features of the basic 3D Gaussian digital human are input into the risk discrimination network to obtain the state of the pre-rendered result output by the risk discrimination network. The risk discrimination network is a deep learning model used to predict the rendering failure probability of the basic 3D Gaussian digital human under the target action and to calculate the slow-track benefit.

3. The 3D Gaussian digital human generation method for cross-domain motion generalization according to claim 2, characterized in that, The risk discrimination network is trained based on the following steps: The initial digital human model is trained based on the training action sequence of the sample person to obtain the basic three-dimensional Gaussian digital human. The basic 3D Gaussian digital human is used to render the action sequence to obtain the sample pre-rendering result, sample human body contour information, temporal change characteristics of adjacent frames, and internal state characteristics of the basic 3D Gaussian digital human. The sample pre-rendering results are compared with the sample real video data to determine the risk label corresponding to the reserved action sequence; Based on the sample pre-rendering results, the reserved action sequence, the sample human body contour information, the temporal change features of adjacent frames, the internal state features of the basic three-dimensional Gaussian digital human, and the risk label, the initial discrimination network is trained to obtain the risk discrimination network.

4. The 3D Gaussian digital human generation method for cross-domain motion generalization according to claim 1, characterized in that, Generating target action videos using a video diffusion model, including: Based on the aforementioned basic 3D Gaussian digital human, identity condition information is extracted; Extract motion control information based on the target action; The identity information and the action control information are input into the video diffusion model to obtain the target action video output by the video diffusion model.

5. The 3D Gaussian digital human generation method for cross-domain action generalization according to claim 4, characterized in that, The video diffusion model is trained based on the following steps: The initial diffusion model is trained based on sample identity information containing the appearance attributes of the sample digital human and sample action control information containing the target posture sequence to obtain the video diffusion model.

6. The 3D Gaussian digital human generation method for cross-domain motion generalization according to claim 1, characterized in that, The preset conditions include the absence of an action Gaussian extension package matching the target action in the action memory.

7. The 3D Gaussian digital human generation method for cross-domain motion generalization according to claim 6, characterized in that, After generating the target action video using the video diffusion model, the following is also included: Reconstruct a dynamic 3D Gaussian representation bound to the base 3D Gaussian digital human based on the target motion video; The dynamic 3D Gaussian representation is encapsulated as an action Gaussian extension package and saved to the action memory library.

8. The 3D Gaussian digital human generation method for cross-domain action generalization according to claim 6, characterized in that, After identifying the state of the pre-rendered result of the basic 3D Gaussian digital human, the process also includes: If the pre-rendered result is in an abnormal state and does not meet the preset conditions, rendering is performed in conjunction with the retrieved Action Gaussian extension pack.

9. The 3D Gaussian digital human generation method for cross-domain motion generalization according to claim 1, characterized in that, After identifying the state of the pre-rendered result of the basic 3D Gaussian digital human, the process also includes: If the pre-rendered result is in a normal state, the pre-rendered result is output directly.

10. A 3D Gaussian digital human generation device for cross-domain motion generalization, characterized in that, include: The recognition module is used to identify the state of the pre-rendered result of the basic 3D Gaussian digital human; The generation module is used to generate a target motion video through a video diffusion model when the state of the pre-rendered result is abnormal and the preset conditions are met. The video diffusion model is a generative artificial intelligence network based on the diffusion denoising process.

Images