Third-order training method and system for dynamic gaussian role video

Abstract

The application discloses a three-order training method and system for dynamic Gaussian character video, relates to the technical field of image processing, and comprises the following steps: collecting multi-view character video from a preset system receiving end, and constructing a basic geometric representation of a character; recognizing and extracting a skeleton structure of the character; after the skeleton structure is determined, performing key frame selection based on posture difference, and simultaneously determining a key deformation field, and establishing a mapping relationship of positions of points on each frame of the multi-view character video from zero time to any time; defining a basic training posture of the character, determining final Gaussian parameters and preliminary rendering loss based on position coding and a neural network prediction mechanism; adaptively increasing the density of a 3D Gaussian distribution, and performing densification training based on a key frame set; constructing two key time constraint terms, forming a complete loss function system, and generating a high-quality and time-continuous dynamic character video from the multi-view character video collected by the system receiving end.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a three-order training method and system for dynamic Gaussian character videos. Background Technology

[0002] Dynamic character video generation technology is a novel technology aimed at reducing costs and increasing efficiency in traditional video production. It breaks through physical and resource limitations, achieving high customization and personalization. In fields such as games, advertising, and online education, it can generate customized character video content in real time based on user preferences, providing immersive interactive experiences. However, current dynamic character video generation technologies still suffer from several problems: unstable training, lack of effective phased training strategies, and susceptibility to training divergence or convergence difficulties; poor temporal continuity, with generated video sequences exhibiting flickering, jitter, and other temporal discontinuities; low reconstruction quality, with existing methods lacking sufficient reconstruction accuracy for complex, dynamically deforming characters; low computational efficiency, lacking optimization strategies for dynamic scenes, resulting in slow training and inference speeds; limited geometric representation, with mesh-based methods struggling to handle topological changes and NeRF-based methods exhibiting slow rendering speeds; and wasted resources in densification, with traditional methods densifying the entire sequence, wasting significant computational resources in non-critical areas and lacking targeted optimization. Summary of the Invention

[0003] The purpose of this invention is to provide a third-order training method and system for dynamic Gaussian character videos to solve the problems raised in the prior art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a third-order training method for dynamic Gaussian character videos, the method comprising:

[0005] Multi-view character videos are acquired from a pre-set system receiver, and a basic geometric representation of the character is constructed based on 3D modeling technology from the multi-view character videos;

[0006] Based on the character's basic geometric representation data, the character's skeletal structure is identified and extracted through skin decomposition technology. After the skeletal structure is determined, key frame selection based on pose differences is performed, and key deformation fields are determined at the same time. The mapping relationship between the position of points on each frame of the multi-view character video from time zero to any time u is established.

[0007] The system defines the basic training pose of the character, determines the basic parameters of the 3D Gaussian distribution, and determines the final Gaussian parameters and the initial rendering loss based on position encoding and neural network prediction mechanisms to complete the initial training.

[0008] Based on the gradient information obtained during the initial training process, the density of the 3D Gaussian distribution is adaptively increased, and dense training is performed based on the keyframe set to generate rendered images in real time.

[0009] The system implements a fixed parameter strategy, constructs two key time constraints, and forms a complete loss function system to generate high-quality and temporally coherent dynamic character videos from the multi-view character videos collected by the system receiver.

[0010] According to the above technical solution, the process of identifying and extracting the skeletal structure of a character through skin decomposition technology includes:

[0011] The system defines a standard space as the reference coordinate system for all deformations, ensuring that the character's pose at different times can be mapped to a unified geometric representation space.

[0012] According to the above technical solution, the keyframe selection based on pose differences includes:

[0013] Obtain the skeletal pose vector for each frame in the multi-view character video sequence. For frame t, combine all joint angles and position information of the skeleton into a high-dimensional pose vector.

[0014]

[0015] in, This represents the rotation angle of the 1st, 2nd, ..., nth joint in frame t. This represents the 3D position coordinates of the 1st, 2nd, ..., mth joints; n and m are the number of rotational joints and positional joints, respectively.

[0016] Define a pose difference metric function, where the pose difference between two frames is D(P). i ,P j The calculation is as follows:

[0017] D(P i ,P j )=α·D rot (P i ,P j )+β·D pos (P i ,P j )

[0018] Among them, D rot For rotational difference measurement:

[0019] D pos For location difference measurement:

[0020] Where i and j represent the i-th and j-th frames, respectively; P i ,P i The high-dimensional pose vectors are composed of the angles and positions of all joints of the skeleton in frames i and j, respectively; k represents the number of joints; ωk The weights for the k-joints are set according to the importance of different joints to the overall posture; α and β represent the rotation angle coefficient and the position difference coefficient, respectively.

[0021] The keyframe selection specifically includes:

[0022] Initialize the keyframe set K = {f0}, where f0 is the first frame of the video sequence;

[0023] For the remaining candidate frame set C = {f1, f2, ... f... T} Calculate the minimum pose difference between each frame in the candidate frame set and each frame in the current keyframe set:

[0024]

[0025] Select the frame corresponding to the maximum value of the minimum pose difference as the new keyframe f. new :

[0026]

[0027] f new Add to keyframe set K, and remove from candidate frame set C;

[0028] Repeat the keyframe selection process until the number of keyframes in the keyframe set reaches a preset threshold N. key Or the minimum pose difference of all frames is less than the threshold τ pose ;

[0029] Output a set of keyframes K key , where N key With τ pose These are the keyframe number threshold and minimum pose threshold set by the system, respectively.

[0030] According to the above technical solution, determining the key deformation field and establishing the mapping relationship between the positions of points in each frame of the multi-view character video from time zero to any time u includes:

[0031] The key deformation field refers to a deformation function set by the system, which is used to realize the mapping of points on each frame of the multi-view character video in standard space after deformation.

[0032] Specifically, it is expressed as follows:

[0033]

[0034] Where, x 0 This represents the coordinate position of the point at time zero in standard space. This represents the coordinate position of the point in standard space at time u.

[0035] According to the above technical solution, the system defines the basic training pose of the character, determines the basic parameters of the 3D Gaussian distribution, and determines the final Gaussian parameters and preliminary rendering loss based on position encoding and neural network prediction mechanisms, completing the initial training including:

[0036] The system initializes all basic parameters of the 3D Gaussian distribution, including the position coordinates x. 0 Rotational quaternion q 0 Anisotropic scale s 0 Transparency α 0 and color feature c 0 ;

[0037] Based on the initial parameters of all 3D Gaussian distributions, a position encoding and neural network prediction mechanism is established, with the input position coordinates x... 0 Processing is performed through positional encoding, using Fourier feature encoding: p 0 =[sin(2πBx)] 0 ),cos(2πBx 0 ]], where B is a preset frequency matrix; p 0 Refers to the encoded positional features;

[0038] The encoded position features, rotation parameters, and scale parameters are used to predict the corresponding deformation increment, denoted as Δx, through a multilayer perceptron (MLP) network. u , Δq u Δs u ;

[0039] After the network predicts the deformation increment, the system calculates the final Gaussian parameters;

[0040] The position parameter is updated to x. u =Φ 0→u (x 0 )+Δx u The rotation parameter is updated to q. u =q 0 +Δq u The scale parameter is updated to s u =s 0 +Δs u ;

[0041] Transparency and color parameters remain stable during this stage, i.e., α. u ≈α 0 ,c u ≈c 0 ;

[0042] After the parameters are updated, the system performs loss function optimization. The optimization objective includes two parts: pixel reconstruction loss and SSIM structural similarity loss.

[0043] Rendering loss L render Represented as:

[0044]

[0045] Where I u For the real target image, For the generated image to be rendered, λ1 and λ2 are the weight parameters for balancing the pixel reconstruction loss and the SSIM structural similarity loss. Indicates pixel reconstruction loss, This represents the SSIM structural similarity loss.

[0046] According to the above technical solution, the step of adaptively increasing the density of the 3D Gaussian distribution based on gradient information during the initial training process, performing denser training based on keyframes, and generating rendered images in real time includes:

[0047] Take the keyframe set K key Perform dense training and exclude other non-keyframe data;

[0048] The system continuously monitors the gradient magnitude of each 3D Gaussian distribution during the optimization process. When the gradient at a certain Gaussian point... Exceeding the preset threshold τ grad When the system determines that the region requires a more refined geometric representation, it employs different densification strategies based on the scale of the Gaussian distribution: for scale s... u >τ size The Gaussian distribution is split into several smaller Gaussian distributions using a splitting operation; for scale s u ≤τ size The Gaussian distribution is used, and a cloning operation is employed to create a new Gaussian distribution at a similar location; where τ grad The gradient threshold τ is set for the system. size The scale threshold set for the system;

[0049] The rendering process uses a tile-based algorithm, which divides the image space into multiple tiles and processes the Gaussian distribution projection and blending within each tile in parallel.

[0050] According to the above technical solution, the system implements a parameter fixing strategy including:

[0051] The system fixes the number of 3D Gaussian distributions at the current level, no longer updates the geometric parameters of the standard space, maintains the stability of the established geometric structure, and freezes the parameters of the position coding network.

[0052] According to the above technical solution, the construction of two key time constraints to form a complete loss function system, and the generation of high-quality and temporally coherent dynamic character videos from the multi-view character videos collected by the system receiver, includes:

[0053] The two key time constraints refer to the nearest neighbor loss L. smooth With time continuity loss L temp ;

[0054] The nearest neighbor loss L smooth This is used to constrain the positional changes of the Gaussian distribution between adjacent time points, ensuring the temporal smoothness of geometric deformation, specifically including:

[0055] L smooth =λ3 / NΣw G,u ‖R δ (x[Y] u-1 -x[G] u-1 )-(x[Y] u -x[G] u )||

[0056] Where N is the total number of neighborhood point pairs, x[G] u and x[Y] u Represent the positions of the adjacent Gaussian points at time u; G and Y represent the indices of the Gaussian points; R δ The relative rotation transformation matrix is ​​the product of the rotation matrix at the current time step and the transpose of the rotation matrix at the previous time step; the weight w G,u An adaptive strategy is used to calculate: when the position difference ||x u -x u-1 ||>τ threshold At that time, w G,u =0; otherwise in To control the hyperparameter of the weight decay rate, τ threshold The set position change threshold;

[0057] The time continuity loss L temp The properties used to directly constrain the smooth changes of the intrinsic properties of the Gaussian distribution over time include:

[0058] L temp =λ4w temp ||s u -s u-1 ||

[0059] Among them, w temp For weights based on location changes, s u s u-1 The scale representing times u and u-1;

[0060] The system forms a complete loss function L total :

[0061] L total =L render +L smooth +L temp .

[0062] A three-order training system for dynamic Gaussian character videos, comprising:

[0063] The 3D modeling module is used to acquire multi-view character videos from a preset system receiver and construct the basic geometric representation of the character based on 3D modeling technology.

[0064] The mapping module, based on the character's basic geometric representation data, identifies and extracts the character's skeletal structure through skin decomposition technology. After the skeletal structure is determined, it performs key frame selection based on pose differences and simultaneously determines the key deformation field, establishing a mapping relationship between the position of points on each frame of the multi-view character video from time zero to any time u.

[0065] The initial training module is used to define the basic training poses of the character, determine the basic parameters of the 3D Gaussian distribution, and determine the final Gaussian parameters and the initial rendering loss based on position encoding and neural network prediction mechanisms to complete the initial training.

[0066] The dense training module adaptively increases the density of the 3D Gaussian distribution based on the gradient information in the initial training process, performs dense training based on the keyframe set, and generates rendered images in real time.

[0067] The data analysis and aggregation module is used to implement a fixed parameter strategy for the system, construct two key time constraints, form a complete loss function system, and generate high-quality and time-coherent dynamic character videos from the multi-view character videos collected by the system receiver.

[0068] Compared with existing technologies, the beneficial effects of this invention are as follows: Compared with the NeRF dynamic modeling method, this application solves the problems of slow training speed, low rendering efficiency, and poor time consistency. The three-stage training strategy ensures the stability of model convergence, avoids training divergence, and significantly improves the training success rate and model quality. The progressive training method enables the network to gradually learn geometric representations from coarse to fine, avoiding the local optima and training instability problems that easily occur in traditional end-to-end training.

[0069] Adaptive deformation field design ensures the accuracy and consistency of geometric deformation, significantly reduces geometric reconstruction errors, and greatly improves the realism of character deformation. The parametric representation based on standard space maintains the continuity of deformation while providing flexibility in handling complex dynamic deformations.

[0070] Multi-level temporal continuity constraints effectively eliminate flickering and jitter in video sequences, significantly improving temporal consistency and resulting in smoother videos with a better viewing experience. The combination of spatial neighborhood constraints and parameter evolution constraints ensures temporal continuity at both geometric and visual levels.

[0071] The phased parameter control strategy significantly improves training efficiency, avoids unnecessary computational overhead, effectively reduces the risk of overfitting, and greatly enhances the model's generalization ability. Dynamic parameter freezing and density control make the training process more efficient and stable.

[0072] The dedicated keyframe densification strategy improves computational efficiency by 70-90%. By completely excluding non-keyframe data, the system can concentrate all computational resources on the most important pose change regions. Compared to traditional full-sequence densification methods, training time is significantly reduced, while redundant computation on secondary frames is avoided. The carefully selected keyframes ensure that the densification operation is more targeted, and the model's ability to express complex dynamic deformations is significantly improved. Attached Figure Description

[0073] Figure 1 This is a detailed flowchart illustrating the Gaussian training process of the third-order training method for dynamic Gaussian character videos according to the present invention. Detailed Implementation

[0074] Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] Example: Figure 1 As shown, this invention provides a technical solution, a third-order training method for dynamic Gaussian character videos, the method comprising:

[0076] Multi-view character videos are acquired from a pre-set system receiver, and a basic geometric representation of the character is constructed based on 3D modeling technology from the multi-view character videos;

[0077] At the beginning of the training process, comprehensive data preparation and geometric modeling of the input character video sequences are required. The core objective of this stage is to establish a stable and consistent geometric representation foundation, providing reliable input data for subsequent dynamic modeling.

[0078] The training process begins with preprocessing the 3D model sequence. The system receives multi-view character videos as input and constructs the character's basic geometric representation using 3D model event modeling techniques. This step is crucial as it determines the upper limit of the system's geometric accuracy. Next, the system performs model sequence retopology processing to ensure the mesh topology of the model remains consistent across different time points, laying the foundation for subsequent deformation field learning. Simultaneously, the Structure from Motion (SfM) method is used to estimate initial sparse point clouds from the multi-view images; these point clouds serve as the initial locations for a 3D Gaussian distribution.

[0079] Based on the preprocessed geometric data, the system extracts the skeletal structure. Using skinning decomposition technology, the system automatically identifies and extracts the character's skeletal structure, establishing the driving relationships for the character's motion. During this process, the system defines a standard space as the reference coordinate system for all deformations, ensuring that the character's posture at different times can be mapped to a unified geometric representation space. Simultaneously, a hierarchical skeletal structure is established to provide the necessary motion constraints to support complex character animations.

[0080] After the skeletal structure is determined, keyframe selection based on pose differences is performed, and key deformation fields are determined at the same time. The mapping relationship between the position of points on each frame of the multi-view character video from time zero to any time u is established.

[0081] The keyframe selection based on pose differences includes:

[0082] Obtain the skeletal pose vector for each frame in the multi-view character video sequence. For frame t, combine all joint angles and position information of the skeleton into a high-dimensional pose vector.

[0083]

[0084] in, This represents the rotation angle of the 1st, 2nd, ..., nth joint in frame t. This represents the 3D position coordinates of the 1st, 2nd, ..., mth joints; n and m are the number of rotational joints and positional joints, respectively.

[0085] Define a pose difference metric function, where the pose difference between two frames is D(P). i ,P j The calculation is as follows:

[0086] D(P i ,P j )=α·D rot (P i ,P j )+β·D pos (P i ,P j )

[0087] Among them, D rot For rotational difference measurement:

[0088] D pos For location difference measurement:

[0089] Where i and j represent the i-th and j-th frames, respectively; P i ,P i The high-dimensional pose vectors are composed of the angles and positions of all joints of the skeleton in frames i and j, respectively; k represents the number of joints; ω k The weights for the k-joints are set according to the importance of different joints to the overall posture; α and β represent the rotation angle coefficient and the position difference coefficient, respectively.

[0090] The keyframe selection specifically includes:

[0091] Initialize the keyframe set K = {f0}, where f0 is the first frame of the video sequence;

[0092] For the remaining candidate frame set C = {f1, f2, ... f... T} Calculate the minimum pose difference between each frame in the candidate frame set and each frame in the current keyframe set:

[0093]

[0094] Select the frame corresponding to the maximum value of the minimum pose difference as the new keyframe f. new :

[0095]

[0096] f new Add to keyframe set K, and remove from candidate frame set C;

[0097] Repeat the keyframe selection process until the number of keyframes in the keyframe set reaches a preset threshold N. key Or the minimum pose difference of all frames is less than the threshold τ pose ;

[0098] Output a set of keyframes K key , where N key With τ pose These are the keyframe number threshold and minimum pose threshold set by the system, respectively. These keyframes form a good coverage distribution in the pose space, providing representative training samples for subsequent denser training.

[0099] The process of determining the key deformation field and establishing the mapping relationship between the positions of points in each frame of the multi-view character video from time zero to any time u includes:

[0100] The key deformation field refers to a deformation function set by the system, which is used to realize the mapping of points on each frame of the multi-view character video in standard space after deformation.

[0101] Specifically, it is expressed as follows:

[0102]

[0103] Where, x 0 This represents the coordinate position of the point at time zero in standard space. This represents the coordinate position of the point in standard space at time u.

[0104] The system defines the character's basic training pose, determines the basic parameters of the 3D Gaussian distribution, and determines the final Gaussian parameters and initial rendering loss based on position encoding and neural network prediction mechanisms, completing the initial training including:

[0105] Here, the basic training pose of the character is defined, that is, the initial training builds the basic geometric structure. For example, in the humanoid training process, frames of representative poses such as T-pose and A-pose are used for training to train the basic humanoid geometric structure.

[0106] The system initializes all basic parameters of the 3D Gaussian distribution, including the position coordinates x. 0 Rotational quaternion q 0 Anisotropic scale s 0 Transparency α 0 and color feature c 0 ;

[0107] Based on the initial parameters of all 3D Gaussian distributions, a position encoding and neural network prediction mechanism is established, with the input position coordinates x... 0 Processing is performed through positional encoding, using Fourier feature encoding: p 0 =[sin(2πBx)] 0 ),cos(2πBx 0 ]], where B is a preset frequency matrix; p 0 Refers to the encoded positional features;

[0108] For example, in this application, assuming the encoding level is 4, the input 3D coordinate position x 0 Given a range of [0.500, 0.300, 0.700], output a 27-dimensional feature p. 0The values ​​are [0.500, 0.000, -1.000, -0.000, 1.000, -0.000, 1.000, -0.000, 1.000, 0.300, 0.951, -0.309, -0.588, -0.809, 0.951, 0.309, 0.588, -0.809, 0.700, -0.951, -0.309, 0.588, -0.809, -0.951, 0.309, -0.588, -0.809]. The actual encoding level used here is 8.

[0109] The encoded position features, rotation parameters, and scale parameters are used to predict the corresponding deformation increment, denoted as Δx, through a multilayer perceptron (MLP) network. u , Δq u Δs u MLP networks employ a fully connected structure, possessing sufficient expressive power to model complex spatial deformation relationships.

[0110] After the network predicts the deformation increment, the system calculates the final Gaussian parameters;

[0111] The position parameter is updated to x. u =Φ 0→u (x 0 )+Δx u The rotation parameter is updated to q. u =q 0 +Δq u The scale parameter is updated to s u =s 0 +Δs u ;

[0112] Transparency and color parameters remain stable during this stage, i.e., α. u ≈α 0 ,c u ≈c 0 This strategy avoids overly complex optimization problems and ensures that the network first learns basic geometric deformations.

[0113] After the parameters are updated, the system performs loss function optimization. The optimization objective includes two parts: pixel reconstruction loss and SSIM structural similarity loss.

[0114] Rendering loss L render Represented as:

[0115]

[0116] Where I u For the real target image, For the generated image to be rendered, λ1 and λ2 are the weight parameters for balancing the pixel reconstruction loss and the SSIM structural similarity loss. Indicates pixel reconstruction loss, This represents the SSIM structural similarity loss. Pixel reconstruction loss ensures pixel-level accuracy, while SSIM structural similarity loss focuses on the structural similarity of the image. Combining the two can produce visually more realistic rendering results.

[0117] Once the basic training converges to a certain extent, the training enters the adaptive densification phase. The core idea of ​​this phase is to adaptively increase the density of the 3D Gaussian distribution based on the gradient information obtained during training, thereby improving the model's ability to represent complex geometric details. The densification process is closely integrated with geometric optimization to ensure that the newly added Gaussian points effectively improve rendering quality.

[0118] The process of adaptively increasing the density of the 3D Gaussian distribution based on gradient information from the initial training process, performing denser training based on keyframes, and generating rendered images in real time includes:

[0119] Take the keyframe set K key Perform dense training and exclude other non-keyframe data;

[0120] The densification strategy is implemented based on a gradient threshold judgment mechanism. The system continuously monitors the gradient magnitude of each 3D Gaussian distribution during the optimization process. When the gradient at a certain Gaussian point... Exceeding the preset threshold τ grad When the system determines that the region requires a more refined geometric representation, it employs different densification strategies based on the scale of the Gaussian distribution: for scale s... u >τ size The Gaussian distribution is split into several smaller Gaussian distributions using a splitting operation; for scale s u ≤τ size The Gaussian distribution is used, and a cloning operation is employed to create a new Gaussian distribution at a similar location; where τ grad The gradient threshold τ is set for the system. size The scale threshold set for the system;

[0121] With the increase in the number of Gaussian distributions and the improvement in geometric representation accuracy, the system's rendering quality is significantly improved. Through an efficient rasterization module, the system can generate high-quality rendered images in real time. The rendering process employs a tile-based algorithm, dividing the image space into multiple tiles and processing the Gaussian distribution projection and blending within each tile in parallel. Simultaneously, the system supports anisotropic sputtering, accurately handling the projection of ellipsoidal Gaussian distributions to achieve more realistic rendering effects. This rendering architecture maintains real-time performance requirements while ensuring quality.

[0122] After the previous training phase, the model has achieved good geometric representation and rendering quality. The focus of the subsequent phase shifts to ensuring temporal continuity, guaranteeing that the generated dynamic videos have a smooth and coherent visual effect over time. This phase employs a strategy combining fixed parameters with specific temporal constraints.

[0123] The system implements a fixed parameter strategy, including:

[0124] The system fixes the number of 3D Gaussian distributions at the current level, stops updating the geometric parameters in the standard space, maintains the stability of the established geometric structure, and freezes the parameters of the positional encoding network. This allows the system to focus on learning the continuity constraints of the temporal dimension without being disturbed by geometric changes.

[0125] The construction of two key time constraints forms a complete loss function system, which generates high-quality and temporally coherent dynamic character videos from the multi-view character videos collected by the system receiver, including:

[0126] The two key time constraints refer to the nearest neighbor loss L. smooth With time continuity loss L temp ;

[0127] The nearest neighbor loss L smooth This is used to constrain the positional changes of the Gaussian distribution between adjacent time points, ensuring the temporal smoothness of geometric deformation, specifically including:

[0128] L smooth =λ3 / N∑w G,u ||R δ (x[Y] u-1 -x[G] u-1 )-(x[Y] u -x[G] u )||

[0129] Where N is the total number of neighborhood point pairs, x[G] u and x[Y] u Represent the positions of the adjacent Gaussian points at time u; G and Y represent the indices of the Gaussian points; R δ The relative rotation transformation matrix is ​​the product of the rotation matrix at the current time and the transpose of the rotation matrix at the previous time. The relative rotation matrix describes the rotational changes from time u-1 to time u, and is used to transform the relative position vector from the previous time to the coordinate system at the current time for comparison. The weight w... G,u An adaptive strategy is used to calculate: when the position difference ||x u -x u-1 ||>τ threshold At that time, w G,u =0; otherwise in To control the hyperparameter of the weight decay rate, τ threshold The set position change threshold;

[0130] The time continuity loss L temp The properties used to directly constrain the smooth changes of the intrinsic properties of the Gaussian distribution over time include:

[0131] L temp =λ4w temp ||s u -s u-1 ||

[0132] Among them, w temp For weights based on location changes, s u s u-1 The scale representing times u and u-1;

[0133] The system forms a complete loss function L total :

[0134] L total =L render +L smooth +L temp .

[0135] By adjusting the various weight parameters λ1, λ2, λ3, and λ4, the system can achieve an optimal balance between reconstruction accuracy, geometric continuity, and visual continuity, ensuring that all losses are on the same order of magnitude, and ultimately generating high-quality and temporally coherent dynamic character videos.

[0136] In this embodiment, a third-order training system for dynamic Gaussian character videos is also provided, the system comprising:

[0137] The 3D modeling module is used to acquire multi-view character videos from a preset system receiver and construct the basic geometric representation of the character based on 3D modeling technology.

[0138] The mapping module, based on the character's basic geometric representation data, identifies and extracts the character's skeletal structure through skin decomposition technology. After the skeletal structure is determined, it performs key frame selection based on pose differences and simultaneously determines the key deformation field, establishing a mapping relationship between the position of points on each frame of the multi-view character video from time zero to any time u.

[0139] The initial training module is used to define the basic training poses of the character, determine the basic parameters of the 3D Gaussian distribution, and determine the final Gaussian parameters and the initial rendering loss based on position encoding and neural network prediction mechanisms to complete the initial training.

[0140] The dense training module adaptively increases the density of the 3D Gaussian distribution based on the gradient information in the initial training process, performs dense training based on the keyframe set, and generates rendered images in real time.

[0141] The data analysis and aggregation module is used to implement a fixed parameter strategy for the system, construct two key time constraints, form a complete loss function system, and generate high-quality and time-coherent dynamic character videos from the multi-view character videos collected by the system receiver.

[0142] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A three-order training method for dynamic Gaussian character videos, characterized by: The method includes: Multi-view character videos are acquired from a pre-set system receiver, and a basic geometric representation of the character is constructed based on 3D modeling technology from the multi-view character videos; Based on the character's fundamental geometric representation data, skinning decomposition technology is used to identify and extract the character's skeletal structure. After the skeletal structure is determined, keyframe selection based on pose differences is performed, and key deformation fields are simultaneously determined. This establishes the position of points on each frame of the multi-view character video from time zero to any other time. Mapping relationship; The system defines the basic training pose of the character, determines the basic parameters of the 3D Gaussian distribution, and determines the final Gaussian parameters and the initial rendering loss based on position encoding and neural network prediction mechanisms to complete the initial training. Based on the gradient information obtained during the initial training process, the density of the 3D Gaussian distribution is adaptively increased, and dense training is performed based on the keyframe set to generate rendered images in real time. The system implements a fixed parameter strategy, constructs two key time constraints, forms a complete loss function system, and generates high-quality and time-coherent dynamic character videos from the multi-view character videos collected by the system receiver. The keyframe selection based on pose differences includes: Obtain the skeletal pose vector for each frame in the multi-view character video sequence. The frame combines all joint angles and position information of the skeleton into a high-dimensional pose vector: in, Indicates the first The joint in the first Frame rotation angle, Indicates the first 3D position coordinates of the joint; and The number of rotational joints and positional joints are defined, respectively; a pose difference metric function is defined, representing the pose difference between two frames. The calculation is as follows: in, For rotational difference measurement: ; For location difference measurement: ;in, Representing the first frame; Representing the first The angles and positions of all joints of the skeleton in a frame are combined into a high-dimensional pose vector. The value represents the number of joints; represent The weights set for each joint are determined based on the importance of different joints to the overall posture. and This represents the rotation angle coefficient and the position difference coefficient; the keyframe selection specifically includes: Initialize keyframe set ,in The first frame of the video sequence; For the remaining candidate frame set Calculate the minimum pose difference between each frame in the candidate frame set and each frame in the current keyframe set: Select the frame corresponding to the maximum value of the minimum pose difference as the new keyframe. : Will Add to keyframe set From the candidate frame set Remove from the keyframes; repeat the keyframe selection process until the number of keyframes in the keyframe set reaches a preset threshold. Or the minimum pose difference of all frames is less than the threshold. Output a set of keyframes. ,in, and These are the keyframe number threshold and minimum pose threshold set by the system, respectively.

2. The third-order training method for dynamic Gaussian character videos according to claim 1, characterized in that: The process of identifying and extracting the character's skeletal structure using skin decomposition technology includes: The system defines a standard space as the reference coordinate system for all deformations, ensuring that the character's pose at different times can be mapped to a unified geometric representation space.

3. The third-order training method for dynamic Gaussian character videos according to claim 2, characterized in that: The key deformation field is determined, and the positions of points on each frame of the multi-view character video are established from time zero to any other time. The mapping relationships include: The key deformation field refers to a deformation function set by the system, which is used to realize the mapping of points on each frame of the multi-view character video in standard space after deformation. Specifically, it is expressed as follows: in, This represents the coordinate position of the point at time zero in standard space. represent The coordinates of the point in standard space at that moment.

4. The third-order training method for dynamic Gaussian character videos according to claim 3, characterized in that: The system defines the character's basic training pose, determines the basic parameters of the 3D Gaussian distribution, and determines the final Gaussian parameters and initial rendering loss based on position encoding and neural network prediction mechanisms, completing the initial training including: The system initializes all basic parameters of the 3D Gaussian distribution, including position coordinates. Rotation Quaternions Anisotropic scale ,transparency and color characteristics ; Based on the initial parameters of all 3D Gaussian distributions, a position encoding and neural network prediction mechanism is established, with the input position coordinates... The process involves positional encoding, using Fourier feature encoding. ,in, It is a preset frequency matrix; The encoded positional features; the encoded positional features, rotation parameters, and scale parameters are used to predict the corresponding deformation increment through a multilayer perceptron (MLP) network, denoted as . ; After the network predicts the deformation increment, the system calculates the final Gaussian parameters; Among them, position parameter update The rotation parameters are updated to The scale parameter is updated to ; Transparency and color parameters remain stable during this stage, i.e. , After the parameters are updated, the system optimizes the loss function. The optimization objective includes two parts: pixel reconstruction loss and SSIM structural similarity loss. Rendering loss Represented as: in For the real target image, The generated image for rendering, and These are the weight parameters for the two loss terms: the pixel reconstruction loss and the SSIM structural similarity loss. Indicates pixel reconstruction loss, This represents the SSIM structural similarity loss.

5. The third-order training method for dynamic Gaussian character videos according to claim 4, characterized in that: The process of adaptively increasing the density of the 3D Gaussian distribution based on gradient information from the initial training process, performing denser training based on keyframes, and generating rendered images in real time includes: Get keyframe set Perform dense training and exclude other non-keyframe data; The system continuously monitors the gradient magnitude of each 3D Gaussian distribution during the optimization process. When the gradient at a certain Gaussian point... Exceeding the preset threshold When the system determines that the region requires a more refined geometric representation, it employs different densification strategies based on the scale of the Gaussian distribution: for different scales... The Gaussian distribution is split into several smaller Gaussian distributions using a splitting operation; for the scale... The Gaussian distribution is used, and a cloning operation is employed to create new Gaussian distributions at similar locations; where... The gradient threshold set for the system, The scale threshold set for the system; The rendering process uses The algorithm divides the image space into multiple tiles and processes the Gaussian distribution projection and mixing within each tile in parallel.

6. The third-order training method for dynamic Gaussian character videos according to claim 1, characterized in that: The system implements a fixed parameter strategy, including: The system fixes the number of 3D Gaussian distributions at the current level, no longer updates the geometric parameters of the standard space, maintains the stability of the established geometric structure, and freezes the parameters of the position coding network.

7. The third-order training method for dynamic Gaussian character videos according to claim 5, characterized in that: The construction of two key time constraints forms a complete loss function system, which generates high-quality and temporally coherent dynamic character videos from the multi-view character videos collected by the system receiver, including: The two key time constraints refer to nearest neighbor loss. With time continuity loss ; The nearest neighbor loss This is used to constrain the positional changes of the Gaussian distribution between adjacent time points, ensuring the temporal smoothness of geometric deformation, specifically including: in, The total number of neighboring point pairs. and Representing time respectively The positions of adjacent Gaussian points; , Indices representing the index of Gaussian points; The relative rotation transformation matrix refers to the product of the rotation matrix at the current time step and the transpose of the rotation matrix at the previous time step; weights An adaptive strategy is used for calculation: when the location difference hour, ;otherwise ,in To control the hyperparameters of the weight decay rate, The set position change threshold; The time continuity loss The properties used to directly constrain the smooth changes of the intrinsic properties of the Gaussian distribution over time include: in, For weights based on location changes, ; Indicates time The scale; The system forms a complete loss function. : .

8. A third-order training system for dynamic Gaussian character videos, used to implement the third-order training method for dynamic Gaussian character videos as described in claim 1, characterized in that: The system includes: The 3D modeling module is used to acquire multi-view character videos from a preset system receiver and construct the basic geometric representation of the character based on 3D modeling technology. The mapping module, based on the character's fundamental geometric representation data, identifies and extracts the character's skeletal structure using skinning decomposition technology. After the skeletal structure is determined, it performs keyframe selection based on pose differences and simultaneously determines key deformation fields, establishing the positions of points on each frame of the multi-view character video from time zero to any other time. The mapping relationship; The initial training module is used to define the basic training poses of the character, determine the basic parameters of the 3D Gaussian distribution, and determine the final Gaussian parameters and the initial rendering loss based on position encoding and neural network prediction mechanisms to complete the initial training. The dense training module adaptively increases the density of the 3D Gaussian distribution based on the gradient information in the initial training process, performs dense training based on the keyframe set, and generates rendered images in real time. The data analysis and aggregation module is used to implement a fixed parameter strategy for the system, construct two key time constraints, form a complete loss function system, and generate high-quality and time-coherent dynamic character videos from the multi-view character videos collected by the system receiver.

Images