A three-dimensional Gaussian digital human timing driving parameter optimization method based on multi-modal credibility fusion

Abstract

The present application aims to solve the problems of digital human posture deviation, local jitter and cross-frame motion discontinuity caused by noise, joint occlusion, frame loss, root node drift and other degradation factors in complex acquisition environment, and to make up for the defects of single-frame correction without timing constraints and poor single-modal adaptability. The method constructs a multi-modal timing driving parameter optimization network, fuses posture translation, pseudo-inertia, joint confidence mask and other multi-modal features, and models the inter-frame motion correlation through the Transformer causal timing encoder after unified coding to predict the frame-level posture and translation residual. At the same time, a multi-modal confidence fusion gating correction mechanism is set to combine the weighted timing prior to suppress parameter mutation and optimize only the digital human posture and global translation parameters while retaining the body shape, rotation and three-dimensional Gaussian appearance. This method can effectively repair degraded parameters, significantly reduce digital human rendering jitter and motion error, and significantly improve driving stability and timing continuity in complex abnormal scenes.

Description

Technical Field

[0001] This invention relates to the fields of digital human driving parameter processing and 3D Gaussian digital human rendering technology, and in particular to a method and system for optimizing the temporal driving parameters of 3D Gaussian digital humans based on multimodal credibility fusion. It is applicable to the intelligent repair and optimization of problems such as pose jitter, root node drift, occlusion loss and frame anomalies generated by 3D Gaussian Splatting (3DGS) digital humans during the temporal driving process. Background Technology

[0002] 3D Gaussian Digital Human is a cutting-edge technology at the intersection of computer vision and computer graphics in recent years. Unlike traditional digital human representation methods based on triangular meshes and texture mapping, 3D Gaussian Digital Human uses a large number of Gaussian Primitives distributed in 3D space as the core appearance representation units. Each Gaussian Primitive contains attribute information such as spatial center position, covariance matrix (scale and rotation), opacity, and spherical harmonic color coefficients. In the rendering stage, the system projects the 3D Gaussian Primitives onto the 2D image plane and performs transparency synthesis according to depth order, thereby generating a human image with spatial hierarchy and rich visual details. Compared with traditional mesh methods, 3D Gaussian representation has stronger appearance expression capabilities, higher rendering efficiency, and lower asset production costs; compared with implicit rendering methods based on Neural Radiance Field (NeRF), 3D Gaussian representation has a more explicit spatial primitive structure and faster training and inference speeds, making it particularly suitable for real-time digital human applications that require a balance between visual realism, rendering frame rate, and asset editability.

[0003] like Figure 4 As shown, animatable 3D Gaussian Digital Humans typically rely on parametric human models (such as SMPL / SMPL-X) for propulsion. The core mechanism involves mapping Gaussian primitives in canonical space to the target pose space via skeletal skinning transformation (such as Linear Blend Skinning, LBS), followed by rasterization rendering to generate the human image. In this process, human pose parameters (typically represented as 72-dimensional axes and angles) and root node translation parameters (3D global displacement) are the core control signals driving the digital human's movements.

[0004] However, in actual engineering deployments, the driving parameters often come from monocular human pose estimators, video motion capture systems, or inertial sensors. These upstream signals are inevitably affected by factors such as occlusion, camera shake, lighting changes, estimator instability, network transmission frame drops, and sensor noise, leading to the following typical anomalies in the driving parameters: pose jitter, i.e., joint rotation parameters fluctuate at high frequency between adjacent frames, causing flickering at the edges of the digital human's limbs and local ghosting; root node drift, i.e., global translation parameters generate cumulative errors, causing the digital human to appear to slide, float, or penetrate the ground; joint occlusion loss, i.e., when human joints are occluded by themselves or other objects, the pose components output by the estimator show significant deviations or are completely lost; frame loss and replacement, i.e., the driving parameters of a certain frame are completely wrong or repeated due to transmission or estimation failure; pose spikes, i.e., abrupt values ​​of local joint axis angle components that far exceed the normal range of motion.

[0005] To address the aforementioned issues, conventional temporal smoothing methods (such as moving average filtering, Kalman filtering, and low-pass filtering) can reduce high-frequency jitter to some extent, but they suffer from the following fundamental drawbacks: First, they cannot distinguish between low-confidence joints and confident joints—applying the same smoothing intensity to all joints leads to insufficient repair of occluded joints while over-smoothing of confident joints; Second, they easily weaken motion details in fast-moving scenes—large-scale fast motion is misjudged as noise and suppressed, resulting in a reduction in the strength and amplitude of gestures, dances, sports, and other movements; Third, the fixed parameters of the filter cannot be adaptively adjusted according to the type and degree of degradation—the optimal filtering strategy required for different types of anomalies such as occlusion, drift, and spikes differs significantly, and a fixed filter cannot simultaneously address all of them.

[0006] Furthermore, most existing methods rely solely on single-mode temporal information for restoration, failing to fully leverage the complementarity of multimodal information. Specifically, pseudo-inertial measurement features (velocity and acceleration) obtained from attitude parameter differencing can explicitly characterize motion trends and abrupt changes, helping the model distinguish between "real rapid movements" and "observed abrupt jumps." Joint validity masks can provide prior knowledge about "which joints' current observations are unreliable," preventing the model from forcibly fitting incorrect observations in occluded areas. When optional image modalities exist, color image features can provide visual contextual information such as human contours, occlusion status, and the appearance of visible areas, assisting in determining the rationality of the current posture. Existing methods fail to effectively fuse these multimodal information elements, resulting in limited restoration performance.

[0007] In summary, there is an urgent need in the existing technology for a method to adaptively repair different types of degradation by simultaneously utilizing posture temporal information, motion trend features, joint confidence information, and visual context information, without changing the driving parameters of the original 3D Gaussian digital human rendering pipeline interface. Summary of the Invention

[0008] like Figure 1 As shown, the purpose of this invention is to overcome the shortcomings of existing technologies and provide a method for optimizing the temporal driving parameters of a 3D Gaussian digital human based on multimodal credibility fusion. This method constructs pose / translation temporal sequences, pseudo-inertial measurement features, and joint validity masks, and incorporates image modalities when optional color image features are available. Each available modality is fed into the transformer encoder for self-attention fusion in a tagged form with time-position encoding and modality type embedding. The fused features predict the pose residual, translation residual, temporal gating parameters, and correction amplitude coefficients. Finally, differential correction of the driving parameters is achieved using incrementally weighted temporal priors and an adaptive gating mechanism, thereby effectively suppressing rendering instability caused by various abnormal driving parameters while maintaining the original action semantics and motion rhythm.

[0009] The technical solution of the present invention is as follows:

[0010] A time-driven parameter optimization method for 3D Gaussian digital humans based on multimodal credibility fusion includes the following steps:

[0011] S1. Construct a timing window and acquire multimodal input: for the current frame and its preceding frames... The frame history frame construction length is The time-series window is used to obtain the human pose parameters for each frame within the window. Root node translation parameters And optional, time-aligned color images for each frame. Among them, human posture parameters This is a 72-dimensional representation of the joint rotation axes and angles of a parametric human model (such as SMPL / SMPL-X) (24 joints, 3-dimensional axes and angles per joint); root node translation parameters. The root node of the human body is located in three-dimensional space in the world coordinate system, with a dimension of 3.

[0012] S2. Generate pseudo-inertial measurement features and joint validity mask: Generate pseudo-inertial measurement features by performing first-order and second-order differences on the human pose parameters and root node translation parameters of adjacent frames within the time window. ; Obtain or generate a joint validity mask representing the credibility of each pose dimension. During the training phase, the mask is generated based on the applied joint occlusion, time frame loss, random frame replacement, or pose peak position. During the deployment phase, the mask is generated based on the upstream joint confidence, timestamp continuity, or pose difference threshold.

[0013] S3. Multimodal Label Encoding and Fusion: Merging human pose parameters and root node translation parameters into a basic driving mode. (75-dimensional), with the pseudo-inertial measurement features (150-dimensional) and joint validity mask (72 dimensions) are encoded into a unified hidden dimension through a linear projection layer. (256-dimensional) After layer normalization and GELU activation function processing, sinusoidal temporal position encoding and learnable modality type embedding are added; when optional color image features exist, their 512-dimensional features are encoded into image tags through corresponding linear projection, layer normalization, and GELU activation function, and then the corresponding temporal position encoding and modality type embedding are added. The temporal tags of each available modality are concatenated along the sequence dimension and fed into an N-layer transformer encoder for multi-head self-attention fusion. The last tag of the pose branch is taken as the fused feature of the current frame. .

[0014] S4. Residual prediction and adaptive gating correction: based on the fused features of the current frame. After layer normalization, the results are fed into the attitude residual head, translation residual head, gated prediction head, and Gaussian control head, respectively, to output the attitude residual. (72-dimensional), translation residual (3D) Attitude-Time Gated Parameters (Scalar, range [0,1]), translation time gating parameter (Scalar, range [0,1]) and correction magnitude coefficient (Scalar, range [0.5, 1]). Simultaneously, based on a temporally ordered window including the current frame, an incrementally weighted temporal prior is constructed to calculate the attitude temporal prior. ̄ and translation time prior ̄, and obtain the attitude-time prior residual. Translation time prior to the defect According to the correction factor Adjusting the attitude residual Translation residual And the time prior residual adjusted by the corresponding time gating parameters, to obtain the corrected attitude parameters for the current frame. and correct the root node translation parameters .

[0015] S5. Driving Parameter Writing and Rendering: Write the corrected attitude parameters and correct the root node translation parameters The driver parameter file of the 3D Gaussian digital human rendering pipeline (such as GauHuman) is written according to the frame number. Only the pose field (poses) and root node translation field (Th) in the original parameter file are replaced, while the root node rotation field (Rh), body shape field (shapes), and other metadata fields in the original parameter file are retained. The 3D Gaussian digital human rendering pipeline then renders the digital human image according to the original reading path. This method acts as a pre-middleware for the rendering pipeline and does not modify the Gaussian assets (Gaussian center, covariance, opacity, spherical harmonic color) and skinning network parameters.

[0016] Furthermore, the pseudo-inertial measurement features in S2 The generation method is as follows: the root node translation parameters of each frame within the time window are... The translational velocity is obtained by performing a first-order difference. The translational acceleration is obtained by performing a second-order difference. Human pose parameters for each frame within the time window The attitude velocity is obtained by performing a first-order difference. The attitude acceleration is obtained by performing a second-order difference. ;Translation speed Translational acceleration Attitude and velocity and attitude acceleration splicing into pseudo-inertial measurement features The dimensions are:

[0017]

[0018] in and Each is 3-dimensional. and Each has 72 dimensions. The total dimensions are 150. The pseudo-inertial measurement features explicitly encode the motion change trend and acceleration abrupt information between adjacent frames, enabling the model to identify numerical discontinuities caused by frame replacement (even if the current frame values ​​seem reasonable, there are abrupt changes in velocity and acceleration) as well as real fast motion (large movements, but the first and second derivatives are continuous).

[0019] Joint validity mask in S2 The generation method is as follows: the 72-dimensional axis angles of human posture parameters are grouped by joints, with each joint corresponding to 3 axis angle components, sharing a validity flag bit; for joints subjected to joint occlusion degradation, time frame loss positions, random frame replacement positions, or posture peak positions, their corresponding validity flag bit is set to 0, indicating that the current observation of the joint is unreliable; the validity flag bit of the remaining joints is set to 1, indicating that they can participate in the fusion normally; finally, a 24-dimensional joint validity mask is formed, which is expanded to 72 dimensions to match the dimensions of posture parameters; when no anomalies are detected during the deployment phase, the corresponding validity flag is set to 1.

[0020] Furthermore, the specific method of multimodal tag encoding in S3 is as follows: the basic driving mode is... (75-dimensional) mapped to the hidden dimensions through the first linear projection layer. Attitude labels are obtained after layer normalization and GELU activation function; pseudo-inertial measurement features are then processed. (150-dimensional) mapped to the hidden dimensions through a second linear projection layer. After layer normalization and GELU activation function, inertial labels are obtained; joint validity masks are then applied. (72-dimensional) Mapped to hidden dimensions through a third linear projection layer After layer normalization and GELU activation, a mask label is obtained; when optional color image features exist, they (512 dimensions) are mapped to the hidden dimensions through a fourth linear projection layer. Image labels are obtained after layer normalization and GELU activation. A sinusoidal positional code is then added to each available label. and learnable modality type embedding The modal types include four types: pose, inertia, mask, and image. The modal type embeddings are learnable 256-dimensional vectors. The transformer encoder contains... Each layer uses 8 attention heads, a 512-dimensional feedforward network, a GELU activation function, a pre-normalization structure, and a dropout rate of 0.1.

[0021] Furthermore, the temporal prior in S4 is constructed as follows: in a length of ( In the time sequence window, the frames are arranged in chronological order from the earliest frame to the current frame. Each time step is recorded as ,in Assign increasing weights to each frame, including the current frame. ,satisfy Furthermore, the sum of all weights is 1. The weights increase linearly between 0.5 and 1.0 and are normalized, so that frames closer to the current frame have higher weights. Pose temporal prior. Translation time prior They are respectively:

[0022]

[0023] in and The first in the window The pose and translation parameters of the frame.

[0024] The attitude parameters are corrected in S4. and correct the root node translation parameters The calculation method is as follows: total attitude residual From the predicted attitude residual With attitude time-gated parameters The sum of the time-prior residuals of the adjusted attitude constitutes the total translation residual. Similarly:

[0025]

[0026] Correcting attitude parameters Based on current attitude parameters With correction magnitude coefficient Multiply by total attitude residual *The sum is used to correct the translation parameters. Similarly:

[0027]

[0028] in These are the time prior coefficients. and The value range is [0, 1], and the output is generated by the gated prediction head after Sigmoid activation, with correction amplitude coefficients. The value range is [0.5, 1], and it is output after being scaled by a Gaussian control head and transformed by a sigmoid transformer. The core of this mechanism lies in the transformer residual. and Responsible for learning nonlinear repair for specific degradation types; historical priors )and (This is responsible for providing stable motion trends that do not depend on training data; gating parameters) and Responsible for adaptively balancing between "trusting observation" and "trusting history"; adjusting the amplitude coefficient. It is responsible for uniformly controlling the correction magnitude, reducing the correction intensity to preserve the original action when the model is uncertain. The residual head initializes the weights and biases to zero, so the learned pose residual and translation residual are zero in the early stage of training; the final output can still include the correction amount generated by the time prior branch.

[0029] Furthermore, the method employs the following multi-source degradation simulation strategy during the training phase: applying one or more of the following degradation methods to the clean human drive parameters based on the configuration: Gaussian noise degradation, and independently adding standard deviations to the pose and root node translation. The normal distribution perturbation; joint occlusion degradation, with probability Randomly select a joint and set its axial angular components to zero, while marking the corresponding mask as invalid; discard degraded time frames with probability. Replace a specific time step with the previous degraded frame; replace degraded frames with random frames, with probability. Randomly copy a frame from the window to replace the current frame; root node drift degradation, accumulate the standard deviation of the translation parameters. Random increments; pose spike degradation, with standard deviations superimposed on local joints when this degradation scenario is enabled. The spike disturbance; hybrid degradation, which combines the above degradation types according to the configuration, is used to simulate the complex situation of occlusion, drift, local mismatch and occasional frame skipping in real monocular tracking.

[0030] A multi-objective loss function is used during training, with a total loss. It consists of the following weighted summation components:

[0031]

[0032] in , , , , , , and These are the weights for pose reconstruction, translation reconstruction, velocity, acceleration, mask awareness, jitter, residual regularization, and modal consistency loss, respectively. Pose reconstruction smoothing... loss Calculate the Smooth-L1 loss between the corrected pose and the clean pose; translate to reconstruct smoothness. loss Calculate the Smooth-L1 loss and velocity loss between the corrected translation and the clean translation. Calculate the Smooth-L1 loss between the corrected adjacent frame velocity and the clean frame velocity, constraining the first-order motion rhythm; acceleration loss. Calculate the Smooth-L1 loss between the corrected adjacent frame acceleration and the clean frame acceleration, constraining second-order motion continuity; mask-aware loss. Dimensions marked as invalid by the joint validity mask are assigned The repair weight is doubled to create a training bias that prioritizes repairing unreliable joints; jitter loss Only punish those who exceed the tolerance threshold Additional second-order difference jitter preserves the inherent acceleration characteristics of real motion; residual regularization loss Optional, used to limit overcorrection in unknown scenarios; modal consistency loss. Cosine similarity is calculated between pose labels and existing inertial, mask, and image labels to prevent complete fragmentation of modalities in the common space. Training uses the AdamW optimizer with an initial learning rate of 1e-4, a batch size of 32, and a training period of 50 epochs.

[0033] The beneficial effects of this invention are as follows:

[0034] (1) Multimodal credibility fusion: The pose / translation time sequence, pseudo-inertial measurement features, joint validity mask and optional color image features are uniformly encoded into time sequence labels. The information fusion across modalities and time steps is achieved through the self-attention mechanism of the transformer, so that the model can make comprehensive judgments by using motion trends, joint credibility and visual context at the same time, which solves the problem of insufficient repair capability of single modality in complex degradation scenarios.

[0035] (2) Adaptive gating correction: By simultaneously predicting attitude residuals, translation residuals, time gating parameters and correction magnitude coefficients, differentiated correction is achieved. Only small corrections are applied to joints and frames that are reliable in observation, and the repair intensity is automatically enhanced for abnormal areas such as occlusion, drift and spikes. This avoids the defects of traditional fixed smoothers that over-smooth reliable joints and under-repair abnormal joints.

[0036] (3) Non-intrusive deployment: As a middleware before the rendering pipeline, it only replaces the pose and translation fields in the original parameter file, without modifying the Gaussian assets and skin network. It can be seamlessly integrated with existing 3D Gaussian digital human rendering pipelines (such as GauHuman) and supports plug and play.

[0037] (4) Quantization effect: In the hybrid degradation input removal experiment at the same model checkpoint, the pose L1 loss, mask pose error, root node translation error, jitter and fast action pose error of the complete interface without input RGB image features are all lower than those of the noise input, indicating that the temporal fusion method can improve multiple driving parameter indicators. Attached Figure Description

[0038] Figure 1 The overall flowchart for multimodal reliability fusion and gating parameter correction.

[0039] Figure 2 The network structure diagram for multimodal label encoding, alignment, and residual prediction.

[0040] Figure 3This is a schematic diagram of multi-source degradation simulation and training sample construction.

[0041] Figure 4 This is a schematic diagram illustrating the relationship between human parameters and skeletal skin control in a Parametric Human Model (SMPL). Detailed Implementation

[0042] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0043] Implementation Step 1: Timing Window Construction and Multimodal Input Acquisition. For example... Figure 1 As shown, this embodiment forms a fixed-length window for the current frame and historical frames. Specifically, it takes... The time window consists of nine frames, including the current frame and its eight preceding historical frames. Each frame within the window contains the following data types: human pose parameters. The axis-angle representation is 72-dimensional SMPL, corresponding to the rotation parameters of 24 joints; the translation parameters of the root node are... A 3D vector representing the spatial position of the human root node in the world coordinate system; optional RGB image. Color images aligned with each frame's time were scaled according to the runtime configuration before being input to the image feature extractor. Data was sourced from a parameter file in ZJU-MoCap-refine format, reading the poses, root node orientation (Rh), root node translation (Th), and body shape fields. Pose was 72-dimensional (axis-angle), root node orientation was 3-dimensional, root node translation was 3-dimensional, and shape coefficients were retained only as character metadata.

[0044] By performing first- and second-order differences on the human posture and translation parameters within the window, pseudo-inertial measurement characteristics are obtained. Specifically, regarding the first Frames and Frame translation parameters and The translational velocity is obtained by performing a first-order difference. Regarding translational speed and The translational acceleration is obtained by performing a second-order difference. Similarly, the attitude velocity is obtained by performing first-order and second-order differences on the attitude parameters. and attitude acceleration .Will (3D) (3D) (72-dimensional) and (72-dimensional) stitched together to form a 150-dimensional pseudo-inertial measurement feature .

[0045] Occluded, missing, or anomalous pose dimensions are marked as low confidence, forming a joint validity mask. Specifically, the 72-dimensional pose parameters are grouped into 24 joints, each group containing 3-dimensional axes and angles, sharing a single validity flag. A joint is randomly selected as an occlusion joint with a probability of 0.15, its 3-axis and angle components are set to zero, and the corresponding validity flag is set to 0. For frame drop locations and random frame replacement locations, the validity flag of the corresponding joint is set to 0. For pose spike joints, their validity flag is set to 0; the validity flag of the remaining joints is set to 1. This results in a 24-dimensional joint validity mask, which is then expanded to 72 dimensions to match the pose parameter dimensions.

[0046] When image branching is enabled, extract RGB images from the same frame. Frame-level features. Specifically, a frozen ResNet-18 backbone network can be used to extract global feature vectors for each frame of the image and mapped to a 512-dimensional representation. This feature provides visual contextual information such as human contours, visible area appearance, and occlusion status; if no image is provided or the image branch is not enabled, this feature extraction and encoding branch is skipped.

[0047] Step 2: Multimodal tag encoding and fusion. For example... Figure 2 As shown, the basic driving modes are respectively (75-dimensional) pseudo-inertial measurement characteristics (150-dimensional) Joint validity mask (72-dimensional) and RGB image features (512-dimensional) are encoded into a time series label of uniform dimension. The encoder for each modality consists of a linear projection layer, layer normalization, and a GELU activation function, which respectively map each modality to a latent image. 256, to avoid high-dimensional color image features suppressing low-dimensional motion signals during simple stitching.

[0048] Add a sinusoidal time position code to each timing mark and learnable modality type embedding Time and location encoding The temporal position information of each frame within the encoding window is encoded using the standard sinusoidal position encoding formula. Modality type embedding. It is a learnable 256-dimensional vector containing four modalities: pose, inertia, mask, and image. For any modal category... and time step Its final label is represented as:

[0049]

[0050] After passing through the temporal fusion network, the four types of labels are concatenated along the sequence dimension. If all modalities are used, a 9-frame window forms 4×9=36 labels, which are then fed into a 4-layer transformer encoder. Each transformer encoder layer contains an 8-head self-attention mechanism and a 512-dimensional feedforward network, using the GELU activation function and a pre-normalization structure, with a random dropout rate of 0.1. The self-attention matrix can compare the temporal relationship between the current elbow pose and the elbow poses of previous frames, as well as the cross-modal relationship between the current pose anomaly and whether it is consistent with the mask, color image, and pseudo-inertial measurement features of the same frame. The last label of the pose branch in the transformer encoder output is taken as the fusion feature of the current frame. This ensures that the output is always bound to the current parameterized human model control frame, and the auxiliary modality provides evidence only through the attention mechanism, avoiding the direct misinterpretation of global color image semantics as skeleton control quantities.

[0051] The model can flexibly select modal combinations based on the operating mode: attitude-only mode (basic driving mode), using only attitude labels for temporal self-attention; attitude + inertia mode, adding pseudo-inertial measurement feature labels; attitude + inertia + mask mode, further adding joint validity mask labels; and full multimodal mode, adding labels for all four modal types. Missing optional modalities are not hard-filled with zero labels, but rather the branch is skipped, allowing the same model architecture to adapt to deployment scenarios with different data completeness.

[0052] Step 3: Residual Prediction and Adaptive Gating Correction. Increasing weights are configured based on the temporal order from each frame within the time window to the current frame to calculate the attitude time prior and translation time prior. Specifically, for... Frame windows, numbered sequentially from the earliest frame to the current frame. Weight It increases linearly between 0.5 and 1.0. , , ..., After normalization, it satisfies The current frame is used for weighting, with attitude time prior. Translation time prior Subtract the current frame input from each of them. and The attitude-time prior residuals were obtained. Translation time prior residual .

[0053] Current frame fusion features After layer normalization, the results are fed into four prediction heads: attitude residual head output. (72-dimensional), is a linear residual prediction head with an output dimension of 72. Initialization weights and biases are both zero; the output of the translation residual head... (3D), structured like the attitude residual head, output dimension 3; gated prediction head outputs attitude time-gated parameters. Translation time gating parameters Two scalars, after Sigmoid activation, have a range of [0,1] and are broadcast to the attitude and translation residual dimensions respectively; the Gaussian control head outputs 3D raw values, which are then scaled by a Sigmoid transform to obtain the correction amplitude coefficients. (Value range [0.5,1]).

[0054] The temporal gating parameters of the fused feature output control the contributions of the attitude and translation time prior residuals respectively: total attitude residual Total translation residual The correction amplitude coefficient c controls the intensity of the total residual applied to the current frame, yielding the correction result: corrected attitude. Correct translation .

[0055] This correction mechanism handles three types of behavior separately: converter residuals. and Responsible for learning nonlinear repair of specific degradation types (such as occlusion, spikes, and drift); historical prior residuals. and Responsible for providing stable motion trends independent of training data; gating parameters and It is responsible for adaptively balancing "trusting observations" and "trusting history"—when there are many invalid joints or obvious temporal anomalies in the current window, the model can increase the overall contribution of the prior at the corresponding time; when fast actions are detected, the gating value is reduced to avoid over-regression to history; the magnitude of the correction is adjusted. `c` is responsible for uniformly controlling the correction magnitude, reducing the correction strength to preserve the original action when the model is uncertain. Compared with fixed low-pass filtering, this model can increase the proportion of historical information when root drift and frame replacement occur, and reduce the risk of over-regression to history during fast actions. The residual head has its weights and biases set to zero during initialization, so the learned pose residual and translation residual are zero in the early stage of training; since the temporal prior branch still participates in the calculation, the final output is not necessarily strictly equivalent to the original input.

[0056] In addition to attitude and translation output, the Gaussian control head also predicts density scaling factors and opacity scaling factors. These are reserved as interfaces for future Gaussian attribute adaptive control and are not currently written to the Gaussian assets. The attitude residuals are retained at the deployment end. Translation residual Timing prior residuals, timing gating and and correction magnitude coefficient Intermediate values ​​provide a basis for error backtracking, threshold setting, and scenario-based parameter tuning. If the correction of a frame is too large, the engineering team can distinguish whether it is caused by network residuals, historical priors, or insufficient confidence scaling.

[0057] Step Four: Training and Deployment. For example... Figure 3 As shown, during training, multi-source degradation is applied to clean samples, and the supervised correction results restore the clean driving parameters. Specifically, clean human driving parameter sequences are read from the ZJU-MoCap-refine dataset, and one or more of the following degradation types are selected and applied to the training samples according to the configuration: Gaussian noise degradation ( ), joint obstruction degeneration ( ), Frame drop degradation ( ), Frame replacement degradation ( ), root node drift degradation ( ) and attitude peak degradation ( Each sample retains clean pose and translation, degenerate pose and translation, mask, and window frame number, so that the training process has both a clean current frame as a supervision target and knows which joints or time steps are unreliable.

[0058] Training uses the currently configured weighted multi-objective loss function:

[0059]

[0060] Residual regularization loss in the current configuration The weight is 0, reserved as an option to be enabled in training scenarios where overcorrection needs to be suppressed. The mask-aware loss... Dimensions marked as invalid by the mask are assigned a repair weight of 4 times, forming a training bias of "prioritizing the repair of untrusted joints"; jitter loss Only penalize the difference between the predicted second-order difference norm and the clean second-order difference norm exceeding [a certain threshold]. The additional jitter is reduced, but the inherent acceleration characteristics of the real motion are preserved. Training was performed using the AdamW optimizer with an initial learning rate of 1e-4, a batch size of 32, and 50 epochs.

[0061] Training logs show that the total loss of the full multimodal hybrid degradation model decreased from 0.10846 to 0.03150 in the first 50 epochs, the mask-related loss decreased from 0.09955 to 0.04521, and the jitter term decreased from 0.25938 to 0.03004, indicating that the network first learns to suppress high-frequency anomalies and then gradually reduces occlusion region errors. Model selection was not based on a single metric, but rather on the pose under hybrid degradation grouping. The loss, mask attitude error, translation error, fast motion attitude error, velocity error, acceleration error and jitter index are jointly judged, and the sub-results of each single degradation are retained.

[0062] After inference, the corrected attitude and translation are written to the original parameter file according to the frame number. Specifically, the export module reads the original parameter file (ZJU-MoCap-refine format) and writes the corrected attitude parameters according to the frame number. Write the corrected root node translation parameters into the poses field. Write to the Th field, while preserving the Rh field (root node rotation), shapes field (body size parameters), and other metadata fields from the original parameter file. This method does not change the structure and remaining fields of the original parameter file, ensuring that the 3D Gaussian digital human rendering pipeline completes rendering according to the original reading path and logic, achieving zero-intrusion integration.

[0063] In online operation, the system uses a 9-frame sliding window consisting of 8 historical frames and the current frame. With each new frame, the window advances one frame. The model only outputs the correction value of the last frame of the window, thus eliminating the need for future frames and enabling causal deployment. With a batch size of 1 and a history size of 8, the inference module averages 0.617 milliseconds per window, equivalent to approximately 1620 windows per second, with a peak CUDA memory usage of 45.14 megabytes, providing real-time rendering capabilities of 30 to 60 frames per second. Offline mode can switch the window to a bidirectional context for more stable long-term correction effects, while real-time mode retains the causal window and risk fallback interface.

[0064] Experimental Verification: As shown in Table 1, six configurations were compared in a mixed degradation scenario: Noisy input (unrepaired degradation input), Smooth baseline (non-learned low-pass filtering smoothing), Pose Only (only input pose and translation), with IMU (adding 150-dimensional pseudo-INS features on top of pose only), with mask (adding a 72-dimensional joint validity mask on top of the previous step), and Full Interface (maintaining the complete model structure but without input RGB image features in this group). All configurations used the same type checkpoint and were compared by removing the input branch during inference under the same subject, the same window length (T=8), and the same degradation type. The experimental results are shown in Table 1:

[0065] Table 1: Comparison of input removal at the same checkpoint under mixed degradation

[0066] Noise input 0.06941 0.20632 0.11257 2.95621 0.06522 1.70219 2.93874 Smooth baseline 0.05274 0.07506 0.10701 0.48786 0.05514 0.39482 0.56665 posture only 0.04576 0.04605 0.01544 0.56570 0.05122 0.34827 0.58747 Attitude + Inertia 0.03954 0.03936 0.01491 0.54608 0.04562 0.33900 0.57561 Attitude + Inertia + Mask 0.02735 0.02724 0.01291 0.42288 0.03240 0.27082 0.45559 Full interface (RGB not input) 0.02724 0.02730 0.01267 0.42211 0.03296 0.27383 0.45729

Claims

1. A method for optimizing the temporal-driven parameters of a 3D Gaussian digital human based on multimodal credibility fusion, characterized in that, include: S1, for the current frame and its preceding frames The frame history frame construction length is The process involves: S1) Obtaining human pose parameters, root node translation parameters, and color images aligned with the time of each frame within a time window, when available; S2) Performing first-order and second-order differences on the human pose parameters and root node translation parameters to generate pseudo-inertial measurement features, and obtaining or generating a joint validity mask representing the reliability of pose observation; S3) Merging the human pose parameters and root node translation parameters into a basic driving mode, encoding the basic driving mode, pseudo-inertial measurement features, and joint validity mask into time markers with a unified hidden dimension, and adding time position encoding and mode type embedding to each time marker; When the color image is available, obtaining image features corresponding to each frame through an image feature extraction network, and encoding the image features into image markers; Concatenating the markers of each available mode along the sequence dimension and feeding them into the transformer encoder for self-attention fusion, and extracting the data from the basic driving mode. The marker corresponding to the current frame is used as the current frame fusion feature; S4, the pose residual, translation residual, pose time gating parameter, translation time gating parameter and correction magnitude coefficient are predicted from the current frame fusion feature, and the pose time prior and translation time prior are constructed in ascending order of the frames in the window. Based on the pose residual, translation residual, pose time gating parameter, translation time gating parameter, correction magnitude coefficient and corresponding time prior residual, the corrected pose parameter and corrected root node translation parameter of the current frame are obtained; S5, the corrected pose parameter and corrected root node translation parameter are written into the driver parameter file used by the 3D Gaussian digital human rendering pipeline according to the frame number, only the pose field and root node translation field in the driver parameter file are replaced, and the root node rotation field, body shape field and other fields are retained. The 3D Gaussian digital human rendering pipeline reads the written driver parameter file and renders it.

2. The method according to claim 1, characterized in that, The generation of the pseudo-inertial measurement features includes: obtaining the translation velocity by performing a first-order difference on the root node translation parameters. The translational acceleration is obtained by performing a first-order difference on the translational velocity. The attitude velocity is obtained by performing first-order difference on the human posture parameters. The attitude acceleration is obtained by performing a first-order difference on the attitude velocity. ;Will , , and splicing into pseudo-inertial measurement features When the human posture parameters are 72-dimensional and the root node translation parameters are 3-dimensional, the pseudo-inertial measurement features are 150-dimensional, and the differential components of the previous frame are set to zero in the starting frame of the window.

3. The method according to claim 1, characterized in that, The joint validity mask is obtained or generated in at least one of the following ways: reading the joint confidence output by the upstream attitude estimator and marking joints below the confidence threshold as invalid; marking the corresponding frame as invalid based on the detection result of discontinuous timestamps or repetition with the previous frame; marking joints whose attitude velocity norm exceeds the velocity threshold or whose attitude acceleration norm exceeds the acceleration threshold as invalid. During the training phase, invalid markers are generated based on the location of joint occlusion, time-series frame dropping, random frame replacement, or pose spikes. Multiple pose components of each joint share a validity value, and the validity value of each joint is extended to the same dimension as the human pose parameters.

4. The method according to claim 1, characterized in that, The basic driving mode, pseudo-inertial measurement features, joint validity mask, and image features are mapped to a unified hidden dimension through independent linear projection layers, layer normalization, and GELU activation functions, respectively. The temporal position encoding is sinusoidal position encoding, and the mode type embedding includes at least four learnable embeddings: basic driving, pseudo-inertial measurement, mask, and image. The transformer encoder adopts a pre-normalization structure. The unified hidden dimension is 256, and the transformer encoder contains 4 layers, each containing 8 attention heads and a 512-dimensional feedforward network.

5. The method according to claim 1, characterized in that, The first in the timing window Weights at each time step satisfy and Attitude and time prior Translation time prior Total attitude residuals Total translation residual Correcting attitude parameters Correct the root node translation parameters Where λ is the time prior coefficient, g_p and g_x have a range of [0,1], and c is the correction magnitude coefficient with a range of [0.5,1].

6. The method according to claim 1, characterized in that, During the training phase, one or more degradation methods are selected from Gaussian noise, joint occlusion, temporal frame dropping, random frame replacement, root node drift, and pose spikes and applied to the clean human drive parameters. The hybrid degradation is composed of degradation terms enabled in the runtime configuration and with non-zero degradation intensity. The total training loss includes a weighted sum of pose reconstruction loss, translation reconstruction loss, velocity loss, acceleration loss, mask-aware loss, jitter loss, optional residual regularization loss, and modal consistency loss. The mask-aware loss assigns a higher repair weight to invalid pose dimensions than to valid pose dimensions, and the jitter loss penalizes additional second-order difference jitter exceeding a preset tolerance.

7. A time-driven parameter optimization system for a 3D Gaussian digital human based on multimodal reliability fusion, characterized in that, include: The multimodal input acquisition module is used to construct a temporal window containing the current frame and historical frames and to acquire human pose parameters, root node translation parameters, and optional color images. The feature generation module generates pseudo-inertial measurement features and acquires or generates joint validity masks; the multimodal coding and fusion module performs labeling, encoding, and self-attention fusion on the basic driving modes, pseudo-inertial measurement features, joint validity masks, and image features obtained by the image feature extraction network when color images are available; the residual prediction and gating correction module predicts the attitude residual, translation residual, attitude time gating parameters, translation time gating parameters, and correction magnitude coefficients, and obtains the corrected attitude parameters and corrected root node translation parameters by combining the incrementally weighted time priors. The drive parameter writing module is used to write the corrected pose parameters and corrected root node translation parameters to the drive parameter file used by the 3D Gaussian Digital Human rendering pipeline, according to the frame number.

Images