Digital human modeling method, device, equipment, storage medium and program product

Abstract

The present application discloses a digital human modeling method, a digital human modeling apparatus, a digital human modeling device, a storage medium, and a program product, and relates to the technical field of human modeling. In the present application, a digital human modeling method based on multimodal driven Gaussian sputtering is proposed. First, by extracting three-dimensional geometric features from a real-time recorded video, the face recorded in the real-time recorded video is fuzzy reconstructed to obtain a static three-dimensional face model; then, by extracting multimodal features from the real-time recorded video, the static three-dimensional face model is dynamically deformed to obtain a dynamic three-dimensional model. In this way, the dynamic features of the human can be accurately captured and reproduced, and a realistic and real-time digital human can be obtained, thereby improving the expressiveness and interactivity of the digital human in various application scenarios.

Description

Technical Field

[0001] The present application relates to the technical field of human modeling, and in particular to a digital human modeling method, a digital human modeling apparatus, a digital human modeling device, a storage medium, and a program product. Background Art

[0002] Currently, traditional methods offer limited interactive features, requiring additional display equipment or complex post-processing to synthesize digital humans with real-world scenes. This increases the complexity and cost of digital human modeling, hindering real-time applications. Secondly, visual quality is poor: digital humans generated using existing technologies often exhibit pixelated, unnatural appearances, or facial distortion, impacting overall visual quality and user experience.

[0003] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a digital human modeling method, a digital human modeling device, a digital human modeling equipment, a storage medium and a program product, aiming to solve the technical problem that the digital humans currently modeled are not realistic enough.

[0005] To achieve the above objectives, this application proposes a digital human modeling method, which includes:

[0006] Acquire real-time recorded video, and extract three-dimensional geometric features and multimodal features from the real-time recorded video;

[0007] Blurred reconstruction of the human face recorded in the real-time recorded video is performed using the three-dimensional geometric features to obtain a static three-dimensional human face model;

[0008] The static three-dimensional face model is dynamically deformed using the multimodal features to obtain a dynamic three-dimensional model.

[0009] In one embodiment, the step of fuzzy reconstructing the face recorded in the real-time recorded video using the three-dimensional geometric features to obtain a static three-dimensional face model includes:

[0010] The face recorded in the real-time recorded video is subjected to fuzzy reconstruction based on 2D Gaussian sputtering using the posture parameters in the three-dimensional geometric features and the estimated point cloud parameters of the corresponding frame to obtain a static three-dimensional face model.

[0011] In one embodiment, the step of dynamically deforming the static three-dimensional face model using the multimodal features to obtain a dynamic three-dimensional model includes:

[0012] Associating the KAN-encoded multimodal features with the Gaussian position points in the static three-dimensional face model to obtain a multi-head attention weight matrix;

[0013] Based on the multi-head attention weight matrix, predicting a parameter change equivalent to the Gaussian sputtering parameter space in the static three-dimensional face model;

[0014] The original parameters of the static three-dimensional face model are adjusted according to the parameter variation to obtain a dynamic three-dimensional model.

[0015] In one embodiment, the digital human modeling method further includes:

[0016] Based on the historical static 3D face model, a static prior model, a feature prior model, and a dynamic prior model are trained;

[0017] By fine-tuning the static prior model, a real-time static model is obtained;

[0018] By fine-tuning the feature prior model, a real-time feature model is obtained;

[0019] By fine-tuning the dynamic prior model, a real-time dynamic model is obtained.

[0020] In one embodiment, the digital human modeling method further includes:

[0021] Outputting a static three-dimensional face model through the real-time static model;

[0022] Outputting multimodal features through the real-time feature model;

[0023] A dynamic three-dimensional face model is outputted through the real-time dynamic model.

[0024] In one embodiment, after the step of dynamically deforming the static three-dimensional face model using the multimodal features to obtain a dynamic three-dimensional model, the following steps are performed:

[0025] Converting the dynamic three-dimensional model into a two-dimensional image through a rasterization rendering engine;

[0026] The two-dimensional image corresponding to any video frame in the real-time recorded video is output to a display device for real-time display.

[0027] In addition, to achieve the above-mentioned purpose, the present application also proposes a digital human modeling device, which includes:

[0028] An extraction module, configured to obtain real-time recorded video and extract three-dimensional geometric features and multimodal features from the real-time recorded video;

[0029] A static module, configured to perform fuzzy reconstruction of the face recorded in the real-time video using the three-dimensional geometric features to obtain a static three-dimensional face model;

[0030] The dynamic module is used to dynamically deform the static three-dimensional face model using the multimodal features to obtain a dynamic three-dimensional model.

[0031] In addition, to achieve the above-mentioned purpose, the present application also proposes a digital human modeling device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the digital human modeling method as described above.

[0032] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps of the digital human modeling method described above are implemented.

[0033] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the digital human modeling method as described above.

[0034] One or more technical solutions proposed in this application have at least the following technical effects:

[0035] In this application, a digital human modeling method based on multimodal driven Gaussian sputtering is proposed. First, by extracting three-dimensional geometric features from real-time recorded video, the face recorded in the real-time recorded video is fuzzy reconstructed to obtain a static three-dimensional face model; then, by extracting multimodal features from the real-time recorded video, the static three-dimensional face model is dynamically deformed to obtain a dynamic three-dimensional model. Among them, by extracting multimodal features such as interactive commands of text commands, voice, expression, character code, specified script, etc. from the real-time recorded video, the static three-dimensional face model is dynamically deformed to obtain a dynamic three-dimensional model. In this way, the dynamic features of the character can be accurately captured and reproduced, and a real and real-time digital human can be obtained, thereby improving the expressiveness and interactivity of the digital human in various application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 A flowchart of the first embodiment of the digital human modeling method of this application is provided;

[0039] Figure 2 A schematic diagram of dynamic deformation processing provided in Example 1 of the digital human modeling method of this application;

[0040] Figure 3 A pre-training diagram provided for the second embodiment of the digital human modeling method of this application;

[0041] Figure 4 This is a schematic diagram of the module structure of the digital human modeling device according to an embodiment of the present application;

[0042] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the digital human modeling method in the embodiment of the present application.

[0043] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0044] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0045] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0046] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device or digital human modeling device capable of implementing the above functions. The following uses the digital human modeling device as an example to illustrate this embodiment and the following embodiments.

[0047] Based on this, the embodiment of the present application provides a digital human modeling method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the digital human modeling method of the present application.

[0048] In this embodiment, the digital human modeling method includes steps S10 to S30:

[0049] Step S10: acquiring a real-time recorded video, and extracting three-dimensional geometric features and multimodal features from the real-time recorded video;

[0050] In a feasible implementation, step S10 includes: using a hybrid expert model composed of various feature extraction models such as character analysis, semantic segmentation, pose estimation model, speech coding, and text coding to extract input features of the character object from the real-time recorded video.

[0051] In one embodiment, a hybrid expert model is used to extract 3D geometric features from real-time recorded video. Specifically, a depth estimation model is used to separate the subject from the background, ensuring that subsequent processing targets only the subject area. A semantic segmentation model is used to layer the subject image, distinguishing different body parts and details. A 3DMM (3D Morphable Model) and a pose estimation model are combined to accurately estimate the subtle features and pose of the face. The RAFT optical flow method is used to further stabilize the pose input, ensuring smoothness and accuracy of dynamic changes. The 3DMM estimation results are converted into point cloud data, providing a foundation for subsequent static modeling.

[0052] In another embodiment, multimodal features are extracted from real-time recorded video using a hybrid expert model. Specifically, the multimodal features include dynamic features such as audio signals, facial expressions, eye movements, and other features such as text commands, character identification codes, specified scripts, and the scene in which the video is recorded.

[0053] Step S20, fuzzy reconstructing the face recorded in the real-time video using three-dimensional geometric features to obtain a static three-dimensional face model;

[0054] In one embodiment, 2DGS (2D Gaussian sputtering) and 3DMM-BaseFaceModel (3DMM-based face model) are used to train and model the face to generate a static three-dimensional face model based on posture.

[0055] Among them, while obtaining a static three-dimensional face model, its three-dimensional coordinates, spherical harmonic function coefficients, transparency, scaling coefficient, rotation coefficient and other model parameters can also be determined.

[0056] Currently, existing digital human systems often fail to respond quickly during interactions, typically taking longer than a second, impacting the user's real-time experience and interaction quality. However, this embodiment uses a Gaussian sputtering model to render the human face, combined with a consumer-grade 3060Ti graphics card to achieve an inference speed of 60 FPS, ensuring a digital human response time of less than a second, meeting the requirements of real-time interaction.

[0057] Furthermore, digital humans generated with existing technologies often exhibit pixelation, unnatural appearances, or facial deformation, impacting the overall visual quality and user experience. Furthermore, most existing solutions only work effectively under specific conditions (such as when the front face or head occupies a large portion of the frame). They perform poorly when the head is tilted or when it occupies only a small portion of the frame. In this embodiment, compared to 3D Gaussian sputtering-based technologies, the 2DGS-based modeling approach effectively addresses artifacts, ensuring a consistent and natural appearance of the human figure and avoiding the ghosting artifacts that can occur after 3D reconstruction. Furthermore, because 2DGS includes both distortion and consistency losses, it allows for more accurate surface reconstruction of the 3D model, preventing distortion when viewing angles change, and ensuring consistent viewing angles.

[0058] In a feasible implementation manner, step S20 includes:

[0059] The face recorded in the real-time video is reconstructed by blurring based on 2D Gaussian sputtering using the pose parameters in the 3D geometric features and the estimated point cloud parameters of the corresponding frame to obtain a static 3D face model.

[0060] Furthermore, in one embodiment, after obtaining a static three-dimensional face model, spherical harmonics can be used to generate pixel colors in different directions to improve the lighting expression of the model; the estimated light source information can be stripped from the model by removing the light source to ensure lighting consistency; the depth distortion error and surface consistency error can be combined to perform physical error evaluation on the model, and artifacts and radial problems can be reduced through training; the SpotlessSplats (a method for 3D scene reconstruction) model can be used to generate an input image MASK (mask) to improve the model's resistance to noise and interference; and the Gaussian point cloud and its planar Gaussian distribution, basic transparency and other features can also be used to express the static model of 2DGS.

[0061] Step S30 , dynamically deforming the static three-dimensional face model using multimodal features to obtain a dynamic three-dimensional model.

[0062] In one embodiment, a KAN (Kolmogorov-Arnold Networks, a neural network architecture)-multi-head attention warping network processes input features (such as facial expressions, audio, actions, commands, and other features extracted from step S10), dynamically warping the static 3D face model constructed in step S20 to achieve real-time dynamic changes. This implements a dynamic 2DGS character rendering process. Due to the principles of KAN, this module can encode continuous and explicit input features into sparse and flexible latent features. These latent features can be expanded to higher-dimensional and simple functional relationships. Through the multi-head attention mechanism, more accurate control input and facial movement alignment can be achieved.

[0063] In a feasible implementation manner, step S30 includes:

[0064] The multimodal features encoded by KAN are associated with the Gaussian position points in the static 3D face model to obtain the multi-head attention weight matrix;

[0065] Based on the multi-head attention weight matrix, the parameter changes equivalent to the Gaussian sputtering parameter space in the static 3D face model are predicted;

[0066] The original parameters of the static three-dimensional face model are adjusted by the parameter variation to obtain a dynamic three-dimensional model.

[0067] In one embodiment, referring to Figure 2 First, the multimodal features are encoded and normalized: The multimodal features that have been highly abstracted during the extraction process have non-uniform value ranges and distributions in the hypothesis space. They need to be normalized and distributed consistently before applying the attention mechanism. Therefore, a simple MAXMIN normalization is performed before inputting into the KAN network, and then KAN encoding is performed. Then, due to the characteristics of the KAN network, it can efficiently and accurately approximate complex low-dimensional functions, which is suitable for understanding highly abstract low-dimensional features, such as the degree of eye opening (0 to 1), and map them to high-dimensional, sparser features suitable for Transformer. Therefore, the features extracted from each frame are extracted to the same dimension through the KAN network, which facilitates the subsequent calculation of the multi-head attention weight matrix. Next, the KAN-encoded features are compared with the Gaussian points ( Figure 2 The 3D coordinates are associated with the coordinate position encoding (P) of the triplane-encoded feature. This association calculates the attention weights between the features encoded by KAN and the location points (Gaussian points). The attention query is the Gaussian point encoding, the key and value are the features after KAN, and are calculated through an MLP operation. The attention weight (Score) is the dot product of Q and K after normalization. The final output feature value is the dot product of Score and Value. The attention weight expresses the relationship between the input feature and the corresponding point, and the value is the magnitude of the change in the feature relative to the point feature. The dot product of the weight matrix (Score) and Value effectively associates the areas that need to be animated with the input features. Secondly, the multi-head attention output is the implicit feature change. Finally, this feature is passed through an MLP (Multilayer Perceptron) to predict the parameter change equivalent to the Gaussian sputtering parameter space. By adding the changed parameters to the original parameters, the static 3D face model is dynamically deformed, resulting in a dynamic 3D model. Furthermore, feature synthesis rendering is performed based on the output parameters to generate the final dynamic model output.

[0068] Triplane-encode is a technique used in 3D scene generation and understanding. It involves encoding a 3D scene into three orthogonal 2D feature planes: the xy, xz, and yz planes. This encoding method can effectively capture the geometric and semantic complexity of the scene and is highly efficient when processing large-scale 3D data.

[0069] Currently, traditional methods require additional display devices or complex post-processing to synthesize digital humans with real scenes, increasing system complexity and cost. They also have limited features and are not conducive to real-time applications. Furthermore, they lack controllability and attribution analysis.

[0070] In this embodiment, a deformable network employs a multi-head attention mechanism, supporting input features such as text commands, voice, emojis, character codes, and specified scripts. This allows the deformable network to adapt its output content to these multimodal inputs. The multi-head attention mechanism and the regularization term in the loss function enable targeted tuning of the relationship between output content and input features, achieving feature decoupling.

[0071] In addition, based on the input of the MOE (Mixture of Experts) model, including the sound feature AVE, expression Blendshape, and the hidden layer output of LLM, due to the non-alignment of the input features, the traditional method uses MLP for training, but its hyperparameters are highly dependent and lack interpretability. This embodiment uses the KAN network for training feature alignment, and the trained base model has better interpretability and more comprehensive interactive performance.

[0072] In a feasible implementation manner, the step S30 includes:

[0073] Convert dynamic 3D models into 2D images through rasterization rendering engine;

[0074] The two-dimensional image corresponding to any video frame in the real-time recorded video is output to the display device for real-time display.

[0075] In this embodiment, a digital human model of a dynamic three-dimensional model in a three-dimensional space is presented in real time through rasterization rendering technology.

[0076] In one embodiment, projection matrix estimation is implemented to estimate the projection matrix from world coordinates to camera coordinates, ensuring accurate presentation of the model from the camera perspective; depth sorting of Gaussian kernels is used to divide the rendering area, reducing the amount of calculation and improving rendering efficiency, while ensuring visibility of the model based on the depth relationship; screen coordinates are converted into light directions, and tangent plane estimation is performed to avoid the problem of matrix pathology that cannot be inverted due to planarization; the final color of each pixel is calculated based on the weight and color of the Gaussian kernel, and transparency blending technology is used to process overlapping Gaussian kernels to achieve a smooth transition of the image; the processed image is output to the frame buffer, and the camera coordinate system is converted into the pixel coordinate system to complete the final image presentation.

[0077] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction and will not be repeated hereafter. On this basis, the digital human modeling method further includes:

[0078] Based on the historical static 3D face model, a static prior model, a feature prior model, and a dynamic prior model are trained;

[0079] By fine-tuning the static prior model, a real-time static model is obtained;

[0080] By fine-tuning the feature prior model, a real-time feature model is obtained;

[0081] By fine-tuning the dynamic prior model, a real-time dynamic model is obtained.

[0082] Currently, without pre-training, the training of static 3D models and dynamic 3D models, which serve as modeling tools for outputting static 3D face models and dynamic 3D models, respectively, takes 20 minutes and 3 hours. However, the addition of a priori pre-training models can effectively speed up training for the same effect, requiring only 10 minutes to complete the fine-tuning of the two steps. In addition, without pre-training, the model's external scalability is poor, and image distortion will occur when encountering unseen poses of individual characters. The addition of a priori pre-training models can facilitate all conditions for everyone and has greater robustness. Even if individual character samples are sparse during fine-tuning, the priori model can effectively complement them.

[0083] In this embodiment, the static modeling results (based on 3DMM point clouds) from multiple human faces are used to train two prior models: a static model prior and a deformable network prior. The static model prior includes the Gaussian kernel and elliptical distribution corresponding to each face. Fine-tuning only requires adjusting the corresponding spherical harmonic coefficients to complete face modeling. The deformable network prior uses multimodal input features to control the point cloud. Since all faces share a set of point cloud features (only the spherical harmonic coefficients differ), the point clouds controlled by the features and the deformable network are highly consistent across different human faces. Therefore, this prior is abstracted and trained as a pre-trained model.

[0084] In one embodiment, referring to Figure 3 The prior model consists of three pre-training models: static prior model, feature prior model and dynamic prior model.

[0085] The static prior model primarily serves as a regularization for the face, encompassing the prior probability and knowledge of facial feature distribution. This aims to prevent overfitting of the 3D model trained using standard training procedures, preserving the relevance of facial features and their similarity to real human skin. This model includes preset Gaussian sputtering parameters, eliminating the need for static training to rely on SFM results.

[0086] Among them, for the feature prior model: the feature prior includes models such as speech extractor and blendshape (a weight-based vertex deformation technology) extraction. The purpose is to unify multimodal features into the same feature space, which greatly saves time and improves the regularization effect in the training of multi-head transformers.

[0087] The dynamic prior model is used to bind the relationship between 3D model changes and features. It does not rely on experiments on a single person. Instead, it unifies static 3D models and corresponding features containing more than 200 faces and trains them. It includes the mapping relationship and inference model of the attention mechanism between common facial features and input features.

[0088] In a feasible implementation, the digital human modeling method further includes:

[0089] Through the real-time static model, a static three-dimensional face model is output;

[0090] Through the real-time feature model, multimodal features are output;

[0091] Through the real-time dynamic model, a dynamic three-dimensional face model is output.

[0092] Therefore, during the pre-training process, the analysis results of multiple characters will be integrated and put into a deformation network with a larger number of parameters for training. When fine-tuning a specific character, pruning and model distillation techniques will be added to output a smaller personal model, thereby establishing a targeted model for the specific character.

[0093] Currently, existing digital human systems often fail to achieve fast responses during interactions, often taking longer than a second, impacting the user's real-time experience and interactive quality. In this embodiment, however, the model size is small: the individual model, after pruning, is no more than 20MB, and the generalized model is only 200MB, making it easier to deploy and enable faster inference.

[0094] Furthermore, current technology typically requires trade-offs between high fidelity, efficient inference, and efficient training, with Gaussian sputtering requiring initialization using SFM. However, in this embodiment, a 3DMM is used as the initialization point cloud, while a multimodal deformation network is trained on a large number of 2DGS models derived from the 3DMM. This approach ensures strong prior support for newly added faces, ensuring robustness while reducing fine-tuning overhead and resolving the high cost of retraining each digital human. This approach achieves faster training efficiency while maintaining high fidelity and efficient inference.

[0095] In an application scenario of the digital human modeling method of the present application, a digital human modeling system is proposed, wherein a feature extraction module is responsible for extracting and analyzing features of a human object from a video.

[0096] The static modeling module receives the parsed data and generates a basic static three-dimensional face model based on 2DGS and 3DMM-BaseFaceModel.

[0097] The deformation network module processes additional input dynamic features (such as audio, expressions, etc.) to achieve dynamic deformation processing of the static three-dimensional face model and generate a dynamic three-dimensional model.

[0098] The rasterization rendering engine receives the deformed model data, converts the three-dimensional model into a two-dimensional image through rasterization technology, and outputs it to the display device to achieve real-time rendering and feature visualization.

[0099] The pre-training and fine-tuning engine trains the pre-training model through a self-built database, and the new character will be fine-tuned based on the pre-training model before practice.

[0100] Through the collaborative work of the above modules, the entire system achieves efficient, realistic and real-time presentation of digital human models, and can be widely used in multiple fields such as virtual reality, film and television animation, and online education.

[0101] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the digital human modeling method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0102] This application also provides a digital human modeling device, please refer to Figure 4 , the digital human modeling device includes:

[0103] An extraction module 10 is used to obtain real-time recorded video and extract three-dimensional geometric features and multimodal features from the real-time recorded video;

[0104] A static module 20 is used to perform fuzzy reconstruction of the face recorded in the real-time video using three-dimensional geometric features to obtain a static three-dimensional face model;

[0105] The dynamic module 30 is used to dynamically deform the static three-dimensional face model through multimodal features to obtain a dynamic three-dimensional model.

[0106] In one embodiment, the static module 20 is further configured to:

[0107] The face recorded in the real-time video is reconstructed by blurring based on 2D Gaussian sputtering using the pose parameters in the 3D geometric features and the estimated point cloud parameters of the corresponding frame to obtain a static 3D face model.

[0108] In one embodiment, the dynamic module 30 is further configured to:

[0109] The multimodal features encoded by KAN are associated with the Gaussian position points in the static 3D face model to obtain the multi-head attention weight matrix;

[0110] Based on the multi-head attention weight matrix, the parameter changes equivalent to the Gaussian sputtering parameter space in the static 3D face model are predicted;

[0111] The original parameters of the static three-dimensional face model are adjusted by the parameter variation to obtain a dynamic three-dimensional model.

[0112] In one embodiment, the digital human modeling device further includes a pre-training module for:

[0113] Based on the historical static 3D face model, a static prior model, a feature prior model, and a dynamic prior model are trained;

[0114] By fine-tuning the static prior model, a real-time static model is obtained;

[0115] By fine-tuning the feature prior model, a real-time feature model is obtained;

[0116] By fine-tuning the dynamic prior model, a real-time dynamic model is obtained.

[0117] In one embodiment, the pre-training module is further used to:

[0118] Through the real-time static model, a static three-dimensional face model is output;

[0119] Through the real-time feature model, multimodal features are output;

[0120] Through the real-time dynamic model, a dynamic three-dimensional face model is output.

[0121] In one embodiment, the digital human modeling device further includes an output module for:

[0122] After the step of dynamically deforming the static three-dimensional face model using multimodal features to obtain a dynamic three-dimensional model:

[0123] Convert dynamic 3D models into 2D images through rasterization rendering engine;

[0124] The two-dimensional image corresponding to any video frame in the real-time recorded video is output to the display device for real-time display.

[0125] The digital human modeling device provided in this application, utilizing the digital human modeling method described in the aforementioned embodiments, can address the current technical issue of digital humans lacking realism. Compared to the prior art, the digital human modeling device provided in this application offers the same beneficial effects as the digital human modeling method described in the aforementioned embodiments. Other technical features of the digital human modeling device are the same as those disclosed in the aforementioned embodiments and are not further elaborated here.

[0126] The present application provides a digital human modeling device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the digital human modeling method in the above-mentioned embodiment one.

[0127] Reference below Figure 5, which shows a schematic diagram of the structure of a digital human modeling device suitable for implementing embodiments of the present application. The digital human modeling device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The digital human modeling device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0128] like Figure 5 As shown, the digital human modeling device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the digital human modeling device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to I / O interface 1006: input devices 1007, such as a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008, such as a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003, such as a magnetic tape or hard disk; and communication device 1009. The communication device 1009 can allow the digital human modeling device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a digital human modeling device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented or provided instead.

[0129] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0130] The digital human modeling device provided in this application, utilizing the digital human modeling method described in the aforementioned embodiments, can resolve the technical issue of digital humans not being realistic enough as currently modeled. Compared to the prior art, the digital human modeling device provided in this application offers the same beneficial effects as the digital human modeling method described in the aforementioned embodiments. Other technical features of this digital human modeling device are the same as those disclosed in the aforementioned embodiments and are not further elaborated here.

[0131] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0132] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0133] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the digital human modeling method in the above-mentioned embodiment.

[0134] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0135] The computer-readable storage medium may be included in the digital human modeling device; or it may exist independently without being assembled into the digital human modeling device.

[0136] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the digital human modeling device, the digital human modeling device is enabled to: obtain real-time recorded video and extract three-dimensional geometric features and multimodal features from the real-time recorded video; fuzzy reconstruct the face recorded in the real-time recorded video using the three-dimensional geometric features to obtain a static three-dimensional face model; and dynamically deform the static three-dimensional face model using the multimodal features to obtain a dynamic three-dimensional model.

[0137] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0138] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0139] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0140] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned digital human modeling method. This computer-readable storage medium can address the current technical issue of digital humans lacking realism. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the digital human modeling method provided in the aforementioned embodiments, and are not further elaborated here.

[0141] The present application also provides a computer program product, comprising a computer program, which implements the steps of the above-mentioned digital human modeling method when executed by a processor.

[0142] The computer program product provided in this application can solve the technical problem of digital humans not being realistic enough in current modeling. Compared with the existing technology, the beneficial effects of the computer program product provided in this application are the same as those of the digital human modeling method provided in the above embodiment, and will not be elaborated here.

[0143] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A digital human modeling method, characterized in that: The digital human modeling method comprises: Acquire real-time recorded video, and extract three-dimensional geometric features and multimodal features from the real-time recorded video; Blurred reconstruction of the human face recorded in the real-time recorded video is performed using the three-dimensional geometric features to obtain a static three-dimensional human face model; Performing dynamic deformation processing on the static three-dimensional face model using the multimodal features to obtain a dynamic three-dimensional model; The step of dynamically deforming the static three-dimensional face model using the multimodal features to obtain a dynamic three-dimensional model comprises: Associating the KAN-encoded multimodal features with the Gaussian position points in the static three-dimensional face model to obtain a multi-head attention weight matrix; Based on the multi-head attention weight matrix, predicting a parameter change equivalent to the Gaussian sputtering parameter space in the static three-dimensional face model; Adjusting the original parameters of the static three-dimensional face model according to the parameter variation to obtain a dynamic three-dimensional model; The modeling object of the dynamic three-dimensional model and the recording object of the real-time recorded video are the same object.

2. The digital human modeling method according to claim 1, wherein: The step of fuzzily reconstructing the face recorded in the real-time recorded video using the three-dimensional geometric features to obtain a static three-dimensional face model includes: The face recorded in the real-time recorded video is subjected to fuzzy reconstruction based on 2D Gaussian sputtering using the posture parameters in the three-dimensional geometric features and the estimated point cloud parameters of the corresponding frame to obtain a static three-dimensional face model.

3. The digital human modeling method according to claim 1, wherein: The digital human modeling method further includes: Based on the historical static 3D face model, a static prior model, a feature prior model, and a dynamic prior model are trained; By fine-tuning the static prior model, a real-time static model is obtained; By fine-tuning the feature prior model, a real-time feature model is obtained; By fine-tuning the dynamic prior model, a real-time dynamic model is obtained.

4. The digital human modeling method according to claim 3, wherein: The digital human modeling method further includes: Outputting a static three-dimensional face model through the real-time static model; Outputting multimodal features through the real-time feature model; A dynamic three-dimensional face model is outputted through the real-time dynamic model.

5. The digital human modeling method according to claim 1, wherein: After the step of dynamically deforming the static three-dimensional face model using the multimodal features to obtain a dynamic three-dimensional model, the following steps are included: Converting the dynamic three-dimensional model into a two-dimensional image through a rasterization rendering engine; The two-dimensional image corresponding to any video frame in the real-time recorded video is output to a display device for real-time display.

6. A digital human modeling device, characterized in that: The digital human modeling device comprises: An extraction module, configured to obtain real-time recorded video and extract three-dimensional geometric features and multimodal features from the real-time recorded video; A static module, configured to perform fuzzy reconstruction of the face recorded in the real-time video using the three-dimensional geometric features to obtain a static three-dimensional face model; A dynamic module, configured to dynamically deform the static three-dimensional face model using the multimodal features to obtain a dynamic three-dimensional model; The dynamic module is also used to: The multimodal features encoded by KAN are associated with the Gaussian position points in the static 3D face model to obtain the multi-head attention weight matrix; Based on the multi-head attention weight matrix, the parameter changes equivalent to the Gaussian sputtering parameter space in the static 3D face model are predicted; By adjusting the original parameters of the static three-dimensional face model through the parameter variation, a dynamic three-dimensional model is obtained; The modeling object of the dynamic three-dimensional model and the recording object of the real-time recorded video are the same object.

7. A digital human modeling device, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the digital human modeling method according to any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the digital human modeling method according to any one of claims 1 to 5 are implemented.

9. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the digital human modeling method according to any one of claims 1 to 5 are implemented.

Images