A real-time online construction method of human head dynamic three-dimensional Gaussian model

Abstract

The application provides a real-time online construction method of a human head dynamic three-dimensional Gaussian model. The method can greatly improve the training throughput through batch rendering of the three-dimensional Gaussian, so that the model achieves almost real-time convergence rate; in addition, a local-global combined data sample sampling algorithm suitable for online modeling is designed, the new input training sample is quickly fitted through local sampling, and the global sampling is used to relieve the model forgetting problem, so that the reconstruction quality similar to offline modeling is achieved. Compared with the previous method, the application can construct the three-dimensional Gaussian face model in real time online without any preprocessing or postprocessing. In addition, online modeling can provide real-time feedback to the user while data is being acquired, and better user experience can be provided. After reconstruction, the model can be used for synthesizing high-fidelity human head animation under new expressions; the model can be widely applied in movies, animation production, network games and remote meetings.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of real-time online three-dimensional reconstruction and human face motion capture and real-time animation, and particularly relates to a real-time online construction method of a human head dynamic three-dimensional Gaussian model. BACKGROUND

[0002] In recent years, 3D Gaussian Splatting (3DGS) (Kerbl B, Kopanas G, Leimkühler T, et al. 3D Gaussian Splatting for Real-Time Radiance Field Rendering[J]. ACM Trans. Graph., 2023, 42(4): 139: 1-139: 14.) has made significant progress in scene reconstruction and inspired a series of related researches. MonoGaussianAvatar (Chen Y, Wang L, Li Q, et al. Monogaussian avatar: Monocular gaussian point-based head avatar[C] / / ACM SIGGRAPH 2024 Conference Papers. 2024: 1-9.) replaces the point cloud in PointAvatar with 3D Gaussian to improve the rendering quality. GaussianHead (Wang J, Xie J C, Li X, et al. Gaussianhead: Impressive head avatars with learnable gaussian diffusion[J]. arXiv preprint arXiv: 2312.01632, 2023.) stores the attributes related to appearance with three planes and realizes animation control through the deformation field based on MLP. Gaussian Head Avatar (Xu Y, Chen B, Li Z, et al. Gaussian head avatar: Ultra high-fidelity head avatar via dynamic gaussians[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024: 1931-1941.) realizes high-fidelity head portrait rendering through a super-resolution network. Based on the interpretability of 3DGS, some researches bind Gaussians to the template mesh to realize more convenient head expression control.GaussianAvatars (Qian S, Kirschstein T, Schoneveld L, et al. Gaussian avatars: Photorealistic head avatars with rigged 3d gaussians [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024:20299-20309.) optimize the mesh jointly to get better image alignment. FlashAvatar (Xiang J, Gao X, Guo Y, et al. FlashAvatar: High-fidelity Head Avatar with Efficient Gaussian Embedding [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024:1802-1812.) use MLP to add dynamic spatial offset for Gaussians. Another class of methods use a set of mixed bases to model head avatars. For example, HeadGaS (Dhamo H, Nie Y, Moreau A, et al. Headgas: Real-time animatable head avatars via 3d gaussian splatting [C] / / European Conference on Computer Vision. Springer, Cham, 2025:459-476.) mix learnable latent features and decode color and opacity of Gaussians through MLP. GEM compress the reconstructed Gaussian head avatar into a feature base. GaussianBlendshapes (Ma S, Weng Y, Shao T, et al. 3d gaussian blendshapes for head avatar animation [C] / / ACM SIGGRAPH 2024 Conference Papers. 2024:1-10.) directly mix all Gaussian attributes and ensure its semantic consistency with mesh blendshapes, achieving high reconstruction quality and rendering performance. Although these methods can reconstruct high-fidelity face models, they all need to be reconstructed offline after obtaining video data, limiting their practical application scenarios.

[0003] Weise et al. (Weise T, Bouaziz S, Li H, et al. Realtime performance-based facial animation [J]. ACM transactions on graphics (TOG), 2011, 30(4): 1-10.) first realized real-time facial capture by fitting a parametric blendshape model based on RGB-D data from a consumer-level depth sensor. Subsequent research focused on corrective shapes (Li H, Yu J, Ye Y, et al. Realtime facial animation with on-the-fly correctives [J]. ACM Trans. Graph., 2013, 32(4): 42: 1-42: 10.), dynamic facial expression spaces (Bouaziz S, Wang Y, Pauly M. Online modeling for real-time facial animation [J]. ACM Transactions on Graphics (ToG), 2013, 32(4): 1-10.), and non-rigid mesh deformation (Chen Y L, Wu HT, Shi F, et al. Accurate and robust 3d facial capture using a single rgbd camera [C] / / Proceedings of the IEEE International Conference on Computer Vision. 2013: 3615-3622.). Although these works demonstrated impressive results, they relied on depth data, which is generally unavailable in most video footage. Cao et al. (Cao C, Weng Y, Lin S, et al. 3D shape regression for real-time facial animation [J]. ACM Transactions on Graphics (TOG), 2013, 32(4): 1-10.) proposed a real-time facial animation system that fits a blendshape model from two-dimensional video frames.DDE (Cao C, Hou Q, Zhou K. Displaced dynamic expression regression for real-time facial tracking and animation [J]. ACM Transactions on graphics (TOG), 2014, 33(4): 1-10.) uses a general regressor, which eliminates the need for calibration for each user. These methods can track and reconstruct the face model in real time, but can only use three-dimensional grid expression, lack of realism. SUMMARY

[0004] The present application is directed to the deficiencies of the prior art, and proposes a real-time online construction method of human head dynamic three-dimensional Gaussian model. The method is suitable for general users, and does not need any special pre-treatment work for specific users. Any new user can use a home computer and a video camera directly in daily life. The present application obtains a face video stream through a monocular camera, and when the video frame removes the background and tracks the head motion parameters, the method can optimize the human head three-dimensional Gaussian model online and achieve a real-time convergence rate. After modeling, the method can drive the reconstructed human head three-dimensional Gaussian model in real time through the given head motion parameters, and synthesize high-fidelity face animation. The present application can be widely used in movies, animation production, video chat, network games and other applications, and has a wide range of application scenarios.

[0005] The present application is realized by the following technical solutions:

[0006] A real-time online construction method of human head dynamic three-dimensional Gaussian model, comprising the following steps:

[0007] (1) Initialization: calibrate the monocular camera parameters, initialize the human head dynamic three-dimensional Gaussian model; create a local sampling cache and a global sampling cache to store online training data samples, and the cache size remains unchanged during system operation;

[0008] (2) Online video stream input image acquisition and processing: use a monocular camera to shoot a face video, calculate the head foreground mask and head motion parameters for the input image; store them in the local sampling cache and the global sampling cache created in step (1);

[0009] (3) Initialize three-dimensional Gaussian color: for the first few frames of input image, project the pixel color onto the three-dimensional Gaussian according to the camera parameters, accelerate the initialization of the color attribute of the three-dimensional Gaussian, and accelerate the convergence speed in the initial training stage;

[0010] (4) Online data sample sampling: according to the number of samples in the local sampling cache and the global sampling cache during the system running process, the local data batch and the global data batch are adaptively calculated; further, the local data batch and the global data batch are respectively sampled from the local sampling cache and the global sampling cache, and finally the two are combined to obtain the training data batch for the current iteration optimization step;

[0011] (5) Dynamic three-dimensional Gaussian model batch parallel optimization: according to the training data batch obtained in step (3), first, the human head dynamic three-dimensional Gaussian model is driven according to the human head motion parameter batch to obtain the three-dimensional Gaussian under different motion parameters; second, the three-dimensional Gaussian is batch rendered; then the rendering result is calculated with the real picture in the training data batch to obtain the loss; finally, the optimization parameters of the human head dynamic three-dimensional Gaussian model are iteratively updated through the image loss;

[0012] (6) Human head animation generation: by giving the human head motion parameters and the camera view angle, the human head dynamic three-dimensional model is driven, and the head animation under the new view angle and expression is generated in real time.

[0013] Specifically, the training samples to be stored in the local sampling cache and the global sampling cache in step (1) include human head motion parameters, real images and human head foreground masks; and the capacity of the local sampling cache and the global sampling cache remains unchanged after being created and will not increase indefinitely with the increase of input data. In the system running process, the local sampling cache is maintained in the form of a first-in-first-out queue, and the global sampling cache is maintained in the form of a reservoir sampling.

[0014] Further, step (2) specifically includes the following sub-steps:

[0015] (2.1) Maintaining the global sampling cache in the form of reservoir sampling When the local sampling cache is full, the last data sample D j is taken out, and D j is stored in the global sampling cache with a probability of , where j is the sample serial number taken out, is the current capacity of the global sampling cache; when D j is stored in , a sample in is randomly discarded; reservoir sampling makes each data in the data stream have the same probability of remaining in the cache;

[0016] (2.2) Maintaining the local sampling cache in the form of a first-in-first-out queue When the local sampling cache is full, the last sample is taken out and stored in D i , otherwise it is directly stored in D i.

[0017] Further, the step (2) accelerates the convergence speed in the initial training stage by using a batch parallel optimization method of the dynamic three-dimensional Gaussian model of the human head to accelerate the convergence; specifically, the batch rendering of the three-dimensional Gaussian is used to improve the training throughput, so that the model reaches a real-time convergence rate and can be applied to online modeling.

[0018] Further, the step (3) is specifically: first, project the three-dimensional Gaussian to two dimensions through the camera parameters, then perform convolution calculation on the input image pixel color through the two-dimensional Gaussian kernel, and finally assign the calculation result to the color attribute of the Gaussian, which is expressed as follows:

[0019]

[0020] where W and H are the width and height of the image respectively, Ixy is the RGB color value of the image (x, y) pixel, and Wxy is the weight of the Gaussian splatting at the image (x, y) pixel.

[0021] Further, the step (4) includes the following sub-steps:

[0022] (4.1) According to the ratio of the sum of the sampling probability density of the current local sampling buffer and the global sampling buffer, adaptively divide the training data batch to ensure that the probability of sampling samples from the global sampling buffer is less than or equal to the probability of sampling samples from the local buffer.

[0023] (4.2) Randomly sample a local training data batch from the local sampling buffer;

[0024] (4.3) Sample a global data batch from the global sampling buffer with the time length of the sample added to the buffer as the importance sampling, and the mathematical calculation formula of the sampling density is as follows:

[0025]

[0026] W(k)=exp(w l ·k / |Ml|);

[0027] That is, when the input video frame is the i-th frame, the probability of sampling the sample D j in the j-th frame, where is the number of samples in the current , w l is a sampling coefficient set artificially, and w l is larger, the sampling weight of the new sample is higher.

[0028] Specifically, the online data sample sampling in the step (4) is performed by a data sample sampling algorithm suitable for local-global combination of online modeling, local sampling is performed first to quickly fit the new input training sample, and then global sampling is performed to relieve the model forgetting problem, so that the reconstruction quality close to offline modeling is achieved.

[0029] Further, the batch rendering of the three-dimensional Gauss in the step (5) specifically comprises the following sub-steps:

[0030] (5.1) Three-dimensional Gauss batch preprocessing: according to the camera parameters, each three-dimensional Gauss is projected to a two-dimensional plane, and the number of Gauss instances is recorded, and each group of three-dimensional Gauss in the three-dimensional Gauss batch is calculated on a different CUDA Stream;

[0031] (5.2) GPU-CPU synchronization: after all the CUDA Stream calculations in step (5.1) are completed, a GPU-CPU synchronization is performed once, the data of the number of three-dimensional Gauss instances is transmitted to the CPU, and an array of a corresponding size is created on the GPU, which is used to store the screen space depth of the Gauss instance, the corresponding pixel block serial number and the corresponding three-dimensional Gauss serial number;

[0032] (5.3) Three-dimensional Gauss batch rasterization: according to the projected three-dimensional Gauss parameters calculated in step (5.1) and the three-dimensional Gauss instances obtained in step (5.2), the contribution value of each Gauss instance to the current pixel color is calculated in turn according to the depth order and transparency mixing is performed, and the color of the current pixel is obtained; each group of three-dimensional Gauss in the three-dimensional Gauss batch is calculated on a different CUDA Stream, and finally the rendered batch image is obtained.

[0033] The beneficial effects of the present application are as follows:

[0034] The method can model a three-dimensional Gauss face model online, without any preprocessing or post-processing, simplifying the data acquisition process. In addition, online modeling can provide real-time feedback to users while data is being acquired, which is particularly suitable for ordinary users. The reconstructed model can be used for high-fidelity facial animation synthesis. Compared with the previous method, the present method greatly shortens the modeling process of the face model, provides better user experience, and completes the entire process of modeling the face model of a person in about 3 minutes. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is the running diagram after the system initialization step of the present application;

[0036] Figure 2 is the system running diagram after the fast initialization step of the three-dimensional Gauss color of the present application;

[0037] Figure 3This is a system operation diagram of the modeling process of this invention;

[0038] Figure 4 This is a system operation diagram of the generated human head animation after the modeling of this invention is completed;

[0039] Figure 5 This is a screenshot of the system running the generated human head animation after the modeling of this invention is completed. Detailed Implementation

[0040] The core of this invention lies in real-time online modeling of human face models. A monocular video stream is input, and a dynamic three-dimensional Gaussian model of the human head is optimized in real time. After reconstruction, it can be used to generate human head images and animations from new perspectives and with different expressions.

[0041] The dynamic 3D Gaussian model of the human head used in this invention is a Gaussian blend shape. The Gaussian blend shape consists of a neutral Gaussian model B0 and 50 base expression Gaussian models {B1, B2, ..., B...} corresponding to basic facial expressions. 50 The above models are all composed of a set of three-dimensional Gaussians. Each Gaussian has the following basic properties: position x∈R 3 Opacity α∈R, rotation q (quaternion representation), size s∈R 3 Spherical harmonic coefficients SH∈R 16×3 ; The Gaussian model B0 of neutral Gaussian model and the Gaussian model B of each base expression k There is a one-to-one correspondence between the Gaussians in the equations. A Gaussian model of a human face with any expression can be calculated using the following formula:

[0042]

[0043] Where ΔB k =B k -B0,{ψ k} represents the expression coefficient for each base expression. B ψ Images can be rendered using Gaussian splatting.

[0044] Based on the aforementioned Gaussian mixture shape, this invention discloses a real-time online construction method for a dynamic three-dimensional Gaussian model of the human head (including but not limited to Gaussian mixture shape), specifically including the following steps: system initialization, online video stream input image acquisition and processing, rapid initialization of three-dimensional Gaussian color, local-global combined online data sample sampling, and batch parallel optimization of the dynamic three-dimensional Gaussian model.

[0045] The following will be combined with the appendix Figure 1 Detailed explanation ( Figure 1is the running screenshot after the system initialization step of the present application, Figure 2 The left side of the middle is the control panel and the input video frame, Figure 2 The right side of the middle is the rendering result of the dynamic three-dimensional Gaussian model of the human head after color fast initialization.

[0046] 1. System initialization

[0047] 1.1 Initialize camera and sampling cache

[0048] Before modeling begins, the camera intrinsic parameters need to be calibrated first, and then the local sampling cache is created and the global sampling cache are created.

[0049] 1.2 Initialize Gaussian Mixture Shape

[0050] For the neutral Gaussian model B0, use Poisson disc sampling to uniformly sample points on the human head neutral expression mesh M0, which will be used as the initial position of the Gaussian. Initialize the Gaussian as an isotropic Gaussian, which is: initialize the size s as the average distance to the nearest three Gaussian centers, and initialize the rotation q as a unit quaternion. For the 50 base expression Gaussian models {B k}, extract the deformation gradient from the FLAME base expression mesh, that is, the deformation from the human head neutral expression mesh M0 to each base expression mesh M k , and apply the deformation gradient to the neutral Gaussian model B0 to obtain the base expression Gaussian model {B k} as the initialization. The number of initialized three-dimensional Gaussians is 60k.

[0051] 2. Online video stream input image acquisition and processing

[0052] 2.1 Online video stream image acquisition

[0053] When a user enters the camera's field of view and starts using the system, it is required that the ambient light remain roughly unchanged during the shooting process, and the user makes different head poses and facial expressions during the shooting process. Whenever the camera inputs a frame of image, first process it to obtain the training data sample by the method of step 2.2, and then update the cache and

[0054] 2.2 Input image processing

[0055] For each frame of input image I iThis invention uses Robust Video Matting (Lin S, Yang L, Saleemi I, et al. Robust high-resolution video matting with temporal guidance[C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision.2022:238-247.) to calculate the head foreground mask to remove the image background. At the same time, it uses DDE (Cao C, Hou Q, Zhou K. Displaced dynamic expression regression for real-time facial tracking and animation[J].ACM Transactions on graphics(TOG),2014,33(4):1-10.) to track the human head motion parameters. Then, the motion parameters, the image, and the foreground mask are combined to form a training data sample D. i .

[0056] 2.3 Update the local sampling cache

[0057] The local sampling buffer is maintained using a first-in, first-out (FIFO) queue: the training data samples D of the current frame are... i Add local sampling buffer like When full, then The last data sample D in j Remove it, freeing up storage space for one sample, and then put D... i Put in like If the storage is not full, directly save D. i join in That's it; step 1.4 is not required.

[0058] 2.4 Update global sampling cache

[0059] The global sampling cache is maintained using a reservoir sampling method: it will be from... Sample D was taken out j Add to global sampling buffer like If the buffer is full, then a random integer k is selected from [0, j). If k is less than the size of the global sampling buffer... Then Discard the k-th data in D and set D j Place it in an empty space, otherwise place D.j Discard. This is equivalent to discarding. The probability of D j Storing samples in a global sampling buffer ensures that each sample in the video stream has an equal probability of remaining in the buffer. middle.

[0060] 3. Fast initialization of 3D Gaussian color

[0061] like Figure 2 As shown ( Figure 2 The left side shows the control panel and the input video frames. Figure 2 The image on the right shows the rendering result of the dynamic 3D Gaussian model of the human head after rapid color initialization. This means that for the initial few frames of the input image, the pixel colors in the image can be mapped to an unoptimized 3D Gaussian model based on camera parameters, thus quickly initializing the Gaussian color (i.e., 0th order spherical harmonic) and accelerating convergence. The specific implementation steps are as follows: The 3D Gaussian model is projected onto a 2D model using the same calculation method as in step 4.1. A convolution calculation is then performed between the 2D Gaussian kernel and the input image to obtain the estimated color c for each Gaussian model. init

[0062]

[0063] Where W and H are the width and height of the image, respectively, and I... xy w represents the RGB color value of the (x, y) pixel in the image. xy This represents the weights of the Gaussian splattering on the (x, y) pixels of the image. This color initialization operation is only executed when the weights of the Gaussian splattering first exceed the threshold δ = 0.1; that is, Gaussians with poor visibility will not be color initialized. This operation is also executed only once, providing a relatively accurate initial value for subsequent gradient optimization and avoiding interference with the optimization process.

[0064] 4. Online data sampling combining local and global methods

[0065] like Figure 3 As shown ( Figure 3 The left side shows the control panel and the input video frames. Figure 3The right side of the middle is the rendering result of the dynamic three-dimensional Gaussian model of the human head in the modeling process, that is, in the traditional offline modeling scheme, since the entire video data has been preprocessed, only a sample is randomly selected from the data set in the training process, the Gaussian model of the frame is driven and obtained, the loss is calculated between the rendered image and the real image, and finally the gradient is back propagated and the model parameters are updated, and the iteration is performed. In online modeling, since the input data is streamed in, random sampling cannot obtain high-quality results. The new data sample has the least opportunity to be optimized, so more sampling weights need to be allocated, and the optimized data cannot be discarded and still needs to participate in optimization to prevent model forgetting. Therefore, the application proposes a local-global combined online data sample sampling algorithm, which iteratively optimizes the face model according to the existing data samples in the cache while processing the input data, to achieve a reconstruction quality similar to offline modeling.

[0066] 4.1 Local-global sampling

[0067] For each iteration step in the online optimization process, the application respectively samples a local sample batch and a global sample batch from the local sampling cache and the global sampling cache , combines them , and then performs batch parallel optimization (see step 5). For the global sample batch , random sampling from the global sampling cache is sufficient. For the local sample batch , in order to allocate more sampling weights to new samples, the following probability distribution is sampled from

[0068]

[0069] W(k)=exp(w l ·k / |M l |)

[0070] The meaning is that when the input video frame is the i-th frame, the probability of sampling the j-th sample D j in , where n is the number of samples in the current , w l is a manually set sampling coefficient, and the larger w l , the higher the sampling weight allocated to new samples. In specific implementation, w l =1.0.

[0071] 4.2 Divide data batches

[0072] ​In system operation, data batch size in optimization While and The number of samples will dynamically increase, the probability density ratio of sampling from and Therefore, the adaptive calculation of the division ratio and (ie The ratio of ) is required to ensure that the probability of sampling any sample from The probability of sampling any sample from Specific calculation is as follows

[0073]

[0074] Where i The specified maximum division ratio max η max = 0.7 in specific implementation. After obtaining the division ratio i η A number of random numbers between 0 and 1 are calculated, and the number of random numbers less than i η is the number of , and vice versa. After determining the division of , the data batch to be optimized in the current iteration step can be obtained according to step 4.1 sampling

[0075] 5. Batch parallel three-dimensional Gaussian model optimization

[0076] The present application optimizes a batch of data samples at each iteration step, thereby greatly improving the training throughput and enabling the model convergence speed to be applied to online real-time modeling. For each iteration step, the data sample batch to be optimized is obtained from step 4 First, the driven Gaussian model is calculated in parallel according to the batch face pose and expression parameters in , and then the three-dimensional Gaussians are batch rendered; second, the rendering result is calculated with the real picture in Loss; finally, the optimization parameters of the three-dimensional Gaussian face model are updated through image loss iteration. In batch optimization, the core of the present application is to batch render three-dimensional Gaussians in parallel, and the specific steps are as follows:

[0077] 5.1 Three-dimensional Gaussian model batch preprocessing

[0078] Each model in the batch of 3D Gaussian models includes a set of 3D Gaussians. The following steps describe the calculation process for each set of 3D Gaussians, which are executed on different CUDA Streams respectively.

[0079] First, calculate for each 3D Gaussian in parallel: project the 3D Gaussian to 2D using camera parameters, the mathematical formula is as follows,

[0080] Σ' = JW∑W T J T

[0081] x' = PWx

[0082] Where ∑' and ∑ are the 3D covariance matrix before projection and the 2D covariance matrix after projection respectively, x' and x are the position coordinates in 3D world space before projection and the coordinates on the 2D image after projection and the depth in screen space respectively. W is the View matrix of the camera, P is the camera projection matrix, J is the Jacobian matrix of the affine transformation estimation of P.∑3 D It can be calculated by the rotation q and scale s properties of the 3D Gaussian. Further, according to the 2D covariance matrix ∑', the lengths of the short axis and the long axis of the 2D Gaussian can be calculated by eigenvalue decomposition (mathematically, the Gaussian distribution has values throughout the space, but in actual implementation, the influence of the Gaussian on values outside three standard deviations can generally be ignored). The length of the long axis of the Gaussian can be used to calculate the enclosing circle of the Gaussian in the screen space, so as to obtain how many pixel blocks (Tiles) the Gaussian is in contact with. The Gaussian in contact with each pixel block is recorded as an "instance". Finally, the Gaussian spherical harmonics are converted into RGB color according to the viewing direction of the camera. Therefore, the following data can be obtained in this stage of calculation: Gaussian screen space depth, Gaussian radius length (and long axis length), Gaussian 2D coordinates on the image, Gaussian 2D covariance matrix, Gaussian RGB color, and the number of Gaussian instances generated.

[0083] Then calculate the prefix sum for the array that saves the number of all Gaussian instances. The last data of the prefix sum array is the sum of the number of Gaussian instances generated by all Gaussians, which is transferred from GPU memory to the memory for subsequent steps on the CPU.

[0084] 5.2 GPU-CPU synchronization

[0085] After waiting for the end of all batch calculations and data transfers executed on different CUDA Streams in step 5.1, allocate GPU memory of corresponding size according to the sum of the number of instances of each set of Gaussians.

[0086] 5.3 Batch rasterization of 3D Gaussian models

[0087] Each model in a batch 3D Gaussian model consists of a set of 3D Gaussians. The following steps describe the calculation process for each set of 3D Gaussians, which are executed on different CUDA Streams.

[0088] First, parallel computation is performed for each 3D Gaussian: based on the Gaussian radius and its 2D coordinates on the image, each pixel block in contact with the Gaussian, i.e., each Gaussian instance, is recalculated. Simultaneously, a key-value pair is calculated for each Gaussian instance: the key is 64 bits long, with the high 32 bits representing the pixel block number in contact and the low 32 bits representing the Gaussian's depth in screen space; the value is the Gaussian's index. Then, the key-value pair is stored in the allocated GPU memory.

[0089] Then, based on the key value, the key-value pairs are sorted using radix sort. In the sorted array, Gaussian instances of the same pixel block are stored in a contiguous block, and the Gaussians within the same pixel block are arranged from lightest to darkest screen depth. After sorting, parallel computation is needed for each 3D Gaussian instance to determine the start and end positions of the same pixel block within the array.

[0090] Finally, parallel computation is performed for each image pixel, with threads belonging to the same thread group (Block): First, the attributes of the Gaussian instances within a pixel block are loaded into the thread group's shared memory. After loading, a synchronization is performed within the thread group. Then, each Gaussian instance in the shared memory is traversed in depth-order, and the color splattered by each Gaussian instance is calculated. The mathematical formula is as follows:

[0091]

[0092] Where x p Let be the 2D coordinates of the pixel, and 'c' be the RGB color of the Gaussian pixel. After obtaining the Gaussian splash color, it is blended with the pixel's own color using transparency. If the cumulative opacity of the pixels reaches 1.0, the loop terminates prematurely. The entire image is rendered after calculating the color of each pixel.

[0093] Human head animation generation. (Example) Figure 4 As shown, Figure 4 The left side shows the control panel and the input video frames. Figure 4 The image on the right shows an animated image synthesized using a dynamic 3D Gaussian model of the human head driven by the same user's head motion parameters. This means that the completed 3D dynamic Gaussian model of the human head (Gaussian mixture shape) can be used to synthesize new images and animations from a given viewpoint and expression. Users only need to provide the human head motion parameters ψ and camera parameters to generate the Gaussian model B for each frame. ψAnd further, high realistic images and animations are obtained by using Gaussian spatter technique. Parameters can be edited manually by users, or obtained from any human head video by using face tracker.

[0094] Example 1

[0095] The inventors implemented the embodiment of the present application on a desktop computer equipped with Intel Core i7-13700KF (5.40GHz) central processor and NVidia RTX4090 graphics processor, and a webcam providing 1280x960 resolution at 30 frames per second. During the running process, for each frame, it took about 50 milliseconds to calculate the human head foreground mask and track the human head motion parameters. The throughput of three-dimensional Gaussian model training reached about 320 samples per second, making the system can model the human head model in real time.

[0096] The inventors invited different new users to test the system based on the present application. The results show that the system can model the head dynamic three-dimensional Gaussian model for any new user in real time online without any preprocessing or postprocessing work. As shown in Figure 5 Figure 5 The left side of the middle is the control panel and the input video frame, Figure 5 The right side of the middle is the animation image rendered and synthesized by driving the human head dynamic three-dimensional Gaussian model with the head motion parameters of different users, that is, after modeling is completed, the human face image and animation under different head poses and expressions can be obtained by driving this model in real time. (In addition, the human portraits used in all the figures of the present application have been used with the consent of the person)

[0097] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The specification and examples given herein are intended as illustrative only and not limiting of the present application. The application is to be considered to encompass any variation, use, or adaptation of the application following, in general, the principles of the application and including such modifications as come within the ordinary skill of the art to which the application pertains.

[0098] It should be understood that the application is not limited to the precise construction that has been described above and illustrated in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application.​

Claims

1. A real-time online construction method of a dynamic three-dimensional Gaussian model of a human head, characterized in that, The method comprises the following steps: (1) initialization: calibrate monocular camera parameters, initialize human head dynamic three-dimensional Gaussian model; create local sampling cache and global sampling cache to store online training data samples, and the cache size remains unchanged during system operation; (2) online video stream input image acquisition and processing: use a monocular camera to shoot a human face video, calculate the head foreground mask and head motion parameters for the input image, and store them in the local sampling cache and global sampling cache created in step (1); (3) initialization of three-dimensional Gaussian color: for the first few frames of input images, project the pixel color onto the three-dimensional Gaussian according to the camera parameters, accelerate the initialization of the color attribute of the three-dimensional Gaussian, and accelerate the convergence speed in the initial training stage; (4) online data sample sampling: according to the number of samples in the local sampling cache and the global sampling cache during system operation, adaptively calculate the division of local data batches and global data batches; further sample the local data batches and global data batches from the local and global sampling caches respectively, and finally combine them to obtain the training data batch for the current iteration optimization step; (5) batch parallel optimization of dynamic three-dimensional Gaussian model: according to the training data batch obtained in step (3), first batch drive the human head dynamic three-dimensional Gaussian model according to the human head motion parameters to obtain the three-dimensional Gaussian under different motion parameters; secondly, batch render the three-dimensional Gaussian; then calculate the loss between the rendering result and the real picture in the training data batch; finally, update the optimization parameters of the human head dynamic three-dimensional Gaussian model through image loss iteration; (6) human head animation generation: drive the modeled human head dynamic three-dimensional model according to the human head motion parameters and camera view angle given by the user, and real-time draw the head animation under the new view angle and expression.

2. The method of claim 1, wherein, The training samples to be stored in the local sampling cache and the global sampling cache in step (1) include human head motion parameters, real images and human head foreground masks; and the capacity of the local sampling cache and the global sampling cache remains unchanged after creation and does not increase indefinitely with the input data, and in the system operation, the local sampling cache is maintained in the form of a first-in-first-out queue, and the global sampling cache is maintained in the form of a reservoir sampling.

3. The method of claim 1, wherein the method further comprises: Step (2) specifically comprises the following sub-steps: (2.1) maintain a global sample buffer in a reservoir sampling fashion When the local sample buffer is full, the last data sample D j is taken out, and with probability D j is stored in the global sample buffer, where j is the sample number taken out, is the current capacity of the global sample buffer; when D j is stored in , a sample in is randomly discarded, reservoir sampling makes every data in the data stream have the same probability to stay in the buffer; (2.2) Maintain a local sample buffer in a first-in-first-out queue When the local sample buffer is full, take the last sample and store it in D i , otherwise store it directly in D i .

4. The method of claim 1, wherein the method further comprises: The step (2) accelerates the convergence speed in the initial training stage by using the batch parallel optimization method of the human head dynamic three-dimensional Gaussian model; specifically, it uses the batch rendering of the three-dimensional Gaussian to improve the training throughput, so that the model reaches a real-time convergence rate and can be applied to online modeling.

5. The method of claim 1, wherein the method further comprises: Step (3) specifically comprises: first, project the three-dimensional Gaussian onto two dimensions through the camera parameters, then perform convolution calculation between the two-dimensional Gaussian kernel and the input image pixel color, and finally assign the calculation result to the color attribute of the Gaussian, which is expressed as follows: where W and H are the width and height of the image, respectively, I xy is the RGB color value of the pixel of the image (x, y), w xy is the weight of the Gaussian splatting at the pixel of the image (x, y).

6. The method of claim 1, wherein the method further comprises: Step (4) comprises the following sub-steps: (4.1) According to the ratio of the sum of the sampling probability density of the current local sampling cache and the global sampling cache, the training data batch is adaptively divided to ensure that the probability of sampling samples from the global sampling cache is less than or equal to the probability of sampling samples from the local cache; (4.2) Randomly sampling a local training data batch from the local sampling cache; (4.3) Sampling a global data batch from the global sampling cache with the time length of the sample added to the cache as importance sampling, and the mathematical calculation formula of the sampling density is as follows: W(k) = exp(w l · k / |M l |); i.e. when the input video frame is the i-th frame, the probability that the sample Dj in the i-th frame of the video is sampled, where is the current is the number of samples in the i-th frame of the video, w l is a sampling coefficient set artificially, w l is larger, the sampling weight assigned to the new sample is higher.

7. The method of claim 1, wherein the method further comprises: determining a plurality of Gaussian models of the human head; and determining a plurality of Gaussian models of the human head in real time. The online data sample sampling in the step (4) is through a data sample sampling algorithm suitable for local-global combined online modeling, which locally samples the newly input training samples and globally samples to relieve the model forgetting problem, so as to achieve a similar reconstruction quality to offline modeling.

8. The method of claim 1, wherein the method further comprises: The batch rendering of the three-dimensional Gaussian in the step (5) specifically includes the following sub-steps: (5.1) Three-dimensional Gaussian batch preprocessing: according to the camera parameters, project each three-dimensional Gaussian to a two-dimensional plane, and record the number of Gaussian instances, and each group of three-dimensional Gaussians in the three-dimensional Gaussian batch is calculated on a different CUDA Stream; (5.2) GPU-CPU synchronization: after waiting for all the CUDA Stream calculations in step (5.1) to be completed, perform a GPU-CPU synchronization, transfer the data of the number of three-dimensional Gaussian instances to the CPU, and create an array of corresponding size on the GPU for storing the screen space depth of the Gaussian instance, the corresponding pixel block serial number and the corresponding three-dimensional Gaussian serial number; (5.3) Three-dimensional Gaussian batch rasterization: according to the projected three-dimensional Gaussian parameters calculated in step (5.1) and the three-dimensional Gaussian instances obtained in step (5.2), calculate the contribution value of each Gaussian instance to the current pixel color in depth order and perform transparency mixing to obtain the color of the current pixel; each group of Gaussians in the three-dimensional Gaussian batch is calculated on a different CUDA Stream, and finally the rendered batch image is obtained.

Images