Digital human modeling method and device, equipment, storage medium and program product

Abstract

The invention discloses a digital human modeling method, a digital human modeling device, digital human modeling equipment, a storage medium and a program product, relates to the technical field of human modeling, and provides a digital human modeling method based on multi-modal driving Gaussian sputtering. The method comprises the following steps: firstly, performing fuzzy reconstruction on a human face recorded in a real-time recorded video through three-dimensional geometric features extracted from the real-time recorded video to obtain a static three-dimensional human face model; then, dynamic deformation processing is carried out on the static three-dimensional face model through multi-modal features extracted from a real-time recorded video, and a dynamic three-dimensional model is obtained; therefore, the dynamic characteristics of the figure can be accurately captured and reproduced, the real and real-time digital person is obtained, and the expressive force and interactivity of the digital person in various application scenes are improved.

Description

Technical Field

[0001] The present application relates to the technical field of human modeling, and in particular to a digital human modeling method, a digital human modeling apparatus, a digital human modeling device, a storage medium and a program product. Background Art

[0002] At present, firstly, the interactive features of traditional methods are single, and additional display devices or complex post-processing are required to synthesize and display digital humans with real scenes, which increases the complexity and cost of digital human modeling and is not conducive to real-time application. Secondly, the visual effect is poor: digital humans generated by existing technologies often have problems such as mosaics, unnatural performance or facial deformation, which affects the overall visual effect and user experience.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present application is to provide a digital human modeling method, a digital human modeling apparatus, a digital human modeling device, a storage medium and a program product, aiming to solve the technical problem that the digital human modeled at present is not realistic enough.

[0005] To achieve the above objectives, the present application proposes a digital human modeling method, which includes: Acquire a real-time recorded video, and extract three-dimensional geometric features and multimodal features from the real-time recorded video; The human face recorded in the real-time recorded video is fuzzily reconstructed by using the three-dimensional geometric features to obtain a static three-dimensional human face model; The static three-dimensional face model is dynamically deformed by using the multimodal features to obtain a dynamic three-dimensional model.

[0006] In one embodiment, the step of fuzzily reconstructing the face recorded in the real-time recorded video by using the three-dimensional geometric features to obtain a static three-dimensional face model includes: The face recorded in the real-time recorded video is fuzzily reconstructed based on 2D Gaussian sputtering using the posture parameters in the three-dimensional geometric features and the estimated point cloud parameters of the corresponding frame to obtain a static three-dimensional face model.

[0007] In one embodiment, the step of dynamically deforming the static three-dimensional face model using the multimodal features to obtain a dynamic three-dimensional model comprises: Associating the multimodal features encoded by KAN with the Gaussian position points in the static three-dimensional face model to obtain a multi-head attention weight matrix; Based on the multi-head attention weight matrix, predicting a parameter change equivalent to the Gaussian sputtering parameter space in the static three-dimensional face model; The original parameters of the static three-dimensional face model are adjusted according to the parameter variation to obtain a dynamic three-dimensional model.

[0008] In one embodiment, the digital human modeling method further includes: Based on the historical static 3D face model, a static prior model, a feature prior model and a dynamic prior model are trained; By fine-tuning the static prior model, a real-time static model is obtained; By fine-tuning the feature prior model, a real-time feature model is obtained; By fine-tuning the dynamic prior model, a real-time dynamic model is obtained.

[0009] In one embodiment, the digital human modeling method further includes: Outputting a static three-dimensional face model through the real-time static model; Through the real-time feature model, multimodal features are output; Through the real-time dynamic model, a dynamic three-dimensional human face model is output.

[0010] In one embodiment, the step of dynamically deforming the static three-dimensional face model using the multimodal features to obtain a dynamic three-dimensional model includes: Converting the dynamic three-dimensional model into a two-dimensional image through a rasterization rendering engine; The two-dimensional image corresponding to any video frame in the real-time recorded video is output to a display device for real-time display.

[0011] In addition, to achieve the above-mentioned purpose, the present application also proposes a digital human modeling device, the digital human modeling device comprising: An extraction module, used to obtain real-time recorded video, and extract three-dimensional geometric features and multimodal features from the real-time recorded video; A static module, used for fuzzy reconstruction of the face recorded in the real-time recorded video by using the three-dimensional geometric features to obtain a static three-dimensional face model; The dynamic module is used to dynamically deform the static three-dimensional face model through the multimodal features to obtain a dynamic three-dimensional model.

[0012] In addition, to achieve the above-mentioned purpose, the present application also proposes a digital human modeling device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the digital human modeling method as described above.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the digital human modeling method described above are implemented.

[0014] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the digital human modeling method described above are implemented.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: In this application, a digital human modeling method based on multimodal driven Gaussian sputtering is proposed. First, by extracting the three-dimensional geometric features from the real-time recorded video, the face recorded in the real-time recorded video is fuzzy reconstructed to obtain a static three-dimensional face model; then, by extracting the multimodal features from the real-time recorded video, the static three-dimensional face model is dynamically deformed to obtain a dynamic three-dimensional model. Among them, by extracting multimodal features such as interactive commands of text commands, voice, expression, character code, specified script, etc. from the real-time recorded video, the static three-dimensional face model is dynamically deformed to obtain a dynamic three-dimensional model, so that the dynamic features of the character can be accurately captured and reproduced, and a real and real-time digital human can be obtained, which improves the expressiveness and interactivity of the digital human in various application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the digital human modeling method of the present application; Figure 2 A schematic diagram of dynamic deformation processing provided in Example 1 of the digital human modeling method of the present application; Figure 3 A schematic diagram of pre-training provided for the second embodiment of the digital human modeling method of the present application; Figure 4 This is a schematic diagram of the module structure of the digital human modeling device according to an embodiment of the present application; Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the digital human modeling method in the embodiment of the present application.

[0019] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0021] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0022] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a digital human modeling device, etc. The following takes the digital human modeling device as an example to illustrate this embodiment and the following embodiments.

[0023] Based on this, the present application embodiment provides a digital human modeling method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the digital human modeling method of the present application.

[0024] In this embodiment, the digital human modeling method includes steps S10 to S30: Step S10, obtaining a real-time recorded video, and extracting three-dimensional geometric features and multimodal features from the real-time recorded video; In a feasible implementation, step S10 includes: using a hybrid expert model composed of various feature extraction models such as character analysis, semantic segmentation, pose estimation model, speech coding, and text coding to extract input features of the character object from the real-time recorded video.

[0025] In one embodiment, three-dimensional geometric features are extracted from real-time recorded video through a hybrid expert model. Specifically, a depth estimation model is used to separate the person from the background, ensuring that subsequent processing is only for the person area; the person image is layered through a semantic segmentation model to distinguish different body parts and details; the 3DMM (3D MorphableModel) and pose estimation model are combined to accurately estimate the light features of the face and its pose; the RAFT optical flow method is used to further stabilize the pose input to ensure the smoothness and accuracy of dynamic changes; the 3DMM estimation results are converted into point cloud data to provide a basis for subsequent static modeling.

[0026] In another embodiment, multimodal features are extracted from real-time recorded video by using a hybrid expert model. Specifically, the multimodal features include dynamic features such as audio signals, expression changes, eye movements, and other features such as text commands, character identification codes, specified scripts, and scenes.

[0027] Step S20, fuzzy reconstruction of the face recorded in the real-time video is performed using three-dimensional geometric features to obtain a static three-dimensional face model; In one embodiment, 2DGS (2D Gaussian sputtering) and 3DMM-BaseFaceModel (3DMM-based face model) are used to train and model the face to generate a static three-dimensional face model based on posture.

[0028] Among them, while obtaining the static three-dimensional face model, its three-dimensional coordinates, spherical harmonic function coefficients, transparency, scaling coefficient, rotation coefficient and other model parameters can also be determined.

[0029] At present, the existing digital human system often cannot achieve fast response during the interaction process, usually taking more than 1 second, affecting the user's real-time experience and interactive effect. In this embodiment, the Gaussian sputtering model is used to render the face, combined with the consumer-grade graphics card 3060Ti, to achieve an inference speed of 60FPS, ensuring that the digital human response time is within 1 second, meeting the needs of real-time interaction.

[0030] In addition, digital humans generated by existing technologies often have problems such as mosaics, unnatural expressions or facial deformation, which affect the overall visual effect and user experience. Moreover, most existing solutions can only work effectively under specific conditions (such as the front face or head occupying most of the picture), and do not perform well for side heads or heads occupying only a small part of the picture. In this embodiment, compared with the technology based on 3D Gaussian sputtering, the modeling method based on 2DGS can effectively deal with artifact problems, ensure the consistency and naturalness of the character image, and avoid the ghosting phenomenon after 3D reconstruction. Moreover, because 2DGS contains distortion loss and consistency loss, the surface reconstruction of the 3D model can be made more accurate, and there will be no deformation problems when the viewing angle changes, ensuring the consistency of the viewing angle.

[0031] In a feasible implementation manner, the step S20 includes: Through the pose parameters in the three-dimensional geometric features and the estimated point cloud parameters of the corresponding frame, the face recorded in the real-time video is reconstructed by fuzzy reconstruction based on 2D Gaussian sputtering to obtain a static three-dimensional face model.

[0032] Furthermore, in one embodiment, after obtaining a static three-dimensional face model, spherical harmonics can be used to generate pixel colors in different directions to improve the lighting expression of the model; the estimated light source information can be stripped from the model by removing the model from the light source to ensure lighting consistency; the depth distortion error and surface consistency error can be combined to evaluate the model's physical error, and artifacts and radial problems can be reduced through training; the SpotlessSplats (a method for 3D scene reconstruction) model can be used to generate an input image MASK (mask) to improve the model's resistance to noise and interference; and the Gaussian point cloud and its planar Gaussian distribution, basic transparency and other features can also be used to express the static model of 2DGS.

[0033] Step S30, dynamically deforming the static three-dimensional face model using multimodal features to obtain a dynamic three-dimensional model.

[0034] In one embodiment, the input features (such as expressions, audio, actions, commands, etc. extracted from step S10) are processed by KAN (Kolmogorov-Arnold Networks, a neural network architecture)-multi-head attention deformation network, and the static three-dimensional face model constructed in step S20 is dynamically deformed to achieve real-time dynamic changes. It realizes the dynamic 2DGS character rendering process, where due to the principle of KAN, the module can encode continuous and explicit input features into sparse and flexible hidden layer features, which can be extended to higher-dimensional and simple functional relationships, and then through the multi-head attention mechanism, more accurate control input and facial action consistency can be achieved.

[0035] In a feasible implementation manner, the step S30 includes: The multimodal features encoded by KAN are associated with the Gaussian position points in the static 3D face model to obtain the multi-head attention weight matrix; Based on the multi-head attention weight matrix, the parameter changes equivalent to the Gaussian sputtering parameter space in the static 3D face model are predicted; The original parameters of the static three-dimensional face model are adjusted through the parameter variation to obtain a dynamic three-dimensional model.

[0036] In one embodiment, referring to Figure 2 First, the multimodal features are encoded and normalized: The multimodal features that have been highly abstracted during the extraction process have non-uniform value ranges and distributions in the hypothesis space. They need to be normalized and distributed consistently before applying the attention mechanism. Therefore, a simple MAXMIN normalization is performed before inputting into the KAN network, and then KAN encoding is performed. Then, due to the characteristics of the KAN network, it can efficiently and accurately approximate complex low-dimensional functions, which is suitable for the understanding of highly abstract low-dimensional features, such as the degree of eye opening (0 to 1), and map them to high-dimensional, sparser features suitable for Transformer. Therefore, the features extracted from each frame are extracted to the same latitude through the KAN network, which facilitates the subsequent calculation of the multi-head attention weight matrix. Next, the KAN-encoded features are compared with the Gaussian points ( Figure 2 The three-dimensional coordinates in the image are associated with the coordinate position encoding P of the triplane-encode. This association is to calculate the attention weight of the feature after KAN encoding and the position point (Gaussian point). Among them, the attention query is the Gaussian point encoding, the key and value are the features after KAN through MLP operation, the attention weight Score is the dot product of Q and K and normalized, and the final output feature value is the dot product of Score and Value. The attention weight expresses the relationship between the input feature and the corresponding point, and the value is the size of the feature change to the point feature change. The dot product of the weight matrix Score and Value can well associate the area that really needs to be dynamic with the input feature. Secondly, the output of multi-head attention is the implicit feature of feature change. Finally, the feature is predicted by MLP (Multilayer Perceptron) to predict the parameter change equivalent to the Gaussian sputtering parameter space. The dynamic deformation of the static three-dimensional face model is realized by adding the changed parameters to the original parameters to obtain a dynamic three-dimensional model. Further, the feature synthesis rendering is performed according to the output parameters to generate the final dynamic model output.

[0037] Among them, triplane-encode is a technology used in 3D scene generation and understanding, which mainly involves encoding the 3D scene into three orthogonal 2D feature planes, namely xy, xz and yz planes. This encoding method can effectively capture the geometric and semantic complexity of the scene and has high efficiency when processing large-scale 3D data.

[0038] At present, traditional methods require additional display devices or complex post-processing to synthesize digital humans with real scenes, which increases the complexity and cost of the system, has a single feature, and is not conducive to real-time applications. At the same time, it lacks more controllability and attribution analysis.

[0039] In this embodiment, a deformation network with a multi-head attention mechanism is used to support input of features such as text commands, voice, expressions, character codes, and specified scripts, so that the deformation network can adaptively output content based on multi-modal input. Due to the multi-head attention mechanism and the regularization term of the loss function, the relationship between the output content and the input features can be debugged in a targeted manner, while achieving feature decoupling.

[0040] In addition, according to the input of the MOE (Mixture of Experts) model, including the sound feature AVE, the expression Blendshape and the hidden layer output of the LLM, due to the non-alignment of the input features, the traditional method uses MLP for training, but its hyperparameters are highly dependent and lack interpretability. This embodiment uses the KAN network for training feature alignment, and the trained base model has better interpretability and more comprehensive interactive performance.

[0041] In a feasible implementation manner, the step S30 includes: Convert dynamic 3D models into 2D images through rasterization rendering engine; The two-dimensional image corresponding to any video frame in the real-time recorded video is output to the display device for real-time display.

[0042] In this embodiment, the digital human model of the dynamic three-dimensional model in the three-dimensional space is presented in real time through rasterization rendering technology.

[0043] In one embodiment, projection matrix estimation is implemented to estimate the projection matrix from world coordinates to camera coordinates, thereby ensuring accurate presentation of the model from the camera perspective; the rendering area is divided by depth sorting of the Gaussian kernel to reduce the amount of calculation and improve rendering efficiency, while ensuring visibility of the model based on the depth relationship; the screen coordinates are converted into the direction of light, and the cutting plane is estimated to avoid the problem of matrix pathology that cannot be inverted due to planarization; the final color of each pixel is calculated based on the weight and color of the Gaussian kernel, and the overlapping Gaussian kernels are processed using transparency blending technology to achieve a smooth transition of the image; the processed image is output to the frame buffer, and the camera coordinate system is converted into the pixel coordinate system to complete the final image presentation.

[0044] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. On this basis, the digital human modeling method also includes: Based on the historical static 3D face model, a static prior model, a feature prior model and a dynamic prior model are trained; By fine-tuning the static prior model, a real-time static model is obtained; By fine-tuning the feature prior model, a real-time feature model is obtained; By fine-tuning the dynamic prior model, a real-time dynamic model is obtained.

[0045] At present, without pre-training, the training of static 3D models and dynamic 3D models, which serve as modeling tools for outputting static 3D face models and dynamic 3D models respectively, takes 20 minutes and 3 hours respectively. The addition of a priori pre-training models can effectively improve the training speed with the same effect, and only takes 10 minutes to complete the fine-tuning of the two steps respectively. In addition, without pre-training, the model's external expansion is poor, and image distortion will occur when encountering a single person's unseen posture. The addition of a priori pre-training models can facilitate all conditions for everyone and has stronger robustness. Even if the single person samples are sparse during fine-tuning, the priori model can be effectively supplemented.

[0046] In this embodiment, the static modeling results (based on 3DMM point cloud) trained from multiple characters are used to train two prior models, namely static model prior and deformation network prior. The static model prior includes the Gaussian kernel distribution and elliptical distribution corresponding to each face. When fine-tuning, only the corresponding spherical harmonic coefficients need to be adjusted to complete face modeling. The deformation network prior includes the control of point cloud by multimodal input features. Since all faces share a set of point cloud features (only the spherical harmonic coefficients are different), different characters have a high degree of consistency in controlling point clouds through features and deformation networks. Therefore, this part is abstracted and trained as a pre-trained model.

[0047] In one embodiment, referring to Figure 3 The prior model consists of three pre-trained models: static prior model, feature prior model and dynamic prior model.

[0048] Among them, for the static prior model: the prior model is mainly a regularization of the face, that is, the prior probability and prior knowledge of the distribution of facial features, the purpose is to prevent the 3D model obtained by normal process training from overfitting, retain the correlation of facial features and the similarity with the real face skin. It contains the preset Gaussian sputtering parameters, so that static training no longer relies on the results of SFM.

[0049] Among them, for the feature prior model: the feature prior includes speech extractor, blendshape (a weight-based vertex deformation technology) extraction and other models. The purpose is to unify multimodal features into the same feature space, which has a great time-saving and effective regularization in the training of multi-head transformers.

[0050] Among them, for the dynamic prior model: the dynamic prior model is used to bind the relationship between the changes in the three-dimensional model and the features. It does not depend on the experiment of a single person. The static three-dimensional model containing 200+ faces and the corresponding features are unified and trained. It includes the mapping relationship and inference model of the attention mechanism of the universal facial features and input features.

[0051] In a feasible implementation manner, the digital human modeling method further includes: Through the real-time static model, a static three-dimensional face model is output; Through the real-time feature model, multimodal features are output; Through the real-time dynamic model, a dynamic three-dimensional face model is output.

[0052] Therefore, during the pre-training process, the analysis results of multiple characters will be integrated into a deformation network with a larger number of parameters for training. When fine-tuning a specific character, techniques such as pruning and model distillation will be added to output a smaller personal model, thereby establishing a targeted model for the specific character.

[0053] At present, existing digital human systems often cannot achieve fast response during the interaction process, usually taking more than 1 second, affecting the user's real-time experience and interactive effect. In this embodiment, the model size is small: the personal model does not exceed 20MB after pruning, and the generalized model is only 200MB, which is easy to deploy and quickly infer.

[0054] In addition, according to the current technology, trade-offs are usually made in three dimensions, namely high fidelity, efficient inference and efficient training, and Gaussian sputtering needs to be initialized using SFM. In this embodiment, 3DMM is used as the initialization point cloud, and a multi-modal deformation network is trained for a large number of characters based on the 2DGS model obtained based on 3DMM. This approach can ensure that the newly added faces have strong prior support and robustness, while reducing fine-tuning overhead and solving the high overhead problem of retraining each digital person. While ensuring high fidelity and efficient inference, faster training efficiency is achieved.

[0055] In an application scenario of the digital human modeling method of the present application, a digital human modeling system is proposed, wherein a feature extraction module is responsible for extracting and analyzing features of a human object from a video.

[0056] The static modeling module receives the parsed data and generates a basic static three-dimensional face model based on 2DGS and 3DMM-BaseFaceModel.

[0057] The deformation network module processes the dynamic features of additional input (such as audio, expressions, etc.), realizes dynamic deformation processing of the static three-dimensional face model, and generates a dynamic three-dimensional model.

[0058] The rasterization rendering engine receives the deformed model data, converts the three-dimensional model into a two-dimensional image through rasterization technology, and outputs it to the display device to achieve real-time rendering and feature visualization.

[0059] The pre-training and fine-tuning engine trains the pre-training model through a self-built database, and the new character will be fine-tuned based on the pre-training model before practice.

[0060] Through the collaborative work of the above modules, the entire system achieves efficient, realistic and real-time presentation of digital human models, and can be widely used in virtual reality, film and television animation, online education and other fields.

[0061] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the digital human modeling method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0062] This application also provides a digital human modeling device, please refer to Figure 4 , the digital human modeling device comprises: An extraction module 10 is used to obtain real-time recorded video and extract three-dimensional geometric features and multimodal features from the real-time recorded video; The static module 20 is used to perform fuzzy reconstruction of the face recorded in the real-time video by using three-dimensional geometric features to obtain a static three-dimensional face model; The dynamic module 30 is used to dynamically deform the static three-dimensional face model through multi-modal features to obtain a dynamic three-dimensional model.

[0063] In one embodiment, the static module 20 is further used for: Through the pose parameters in the three-dimensional geometric features and the estimated point cloud parameters of the corresponding frame, the face recorded in the real-time video is reconstructed by fuzzy reconstruction based on 2D Gaussian sputtering to obtain a static three-dimensional face model.

[0064] In one embodiment, the dynamic module 30 is further used for: The multimodal features encoded by KAN are associated with the Gaussian position points in the static 3D face model to obtain the multi-head attention weight matrix; Based on the multi-head attention weight matrix, the parameter changes equivalent to the Gaussian sputtering parameter space in the static 3D face model are predicted; The original parameters of the static three-dimensional face model are adjusted through the parameter variation to obtain a dynamic three-dimensional model.

[0065] In one embodiment, the digital human modeling device further includes a pre-training module for: Based on the historical static 3D face model, a static prior model, a feature prior model and a dynamic prior model are trained; By fine-tuning the static prior model, a real-time static model is obtained; By fine-tuning the feature prior model, a real-time feature model is obtained; By fine-tuning the dynamic prior model, a real-time dynamic model is obtained.

[0066] In one embodiment, the pre-training module is further used to: Through the real-time static model, a static three-dimensional face model is output; Through the real-time feature model, multimodal features are output; Through the real-time dynamic model, a dynamic three-dimensional face model is output.

[0067] In one embodiment, the digital human modeling device further includes an output module, which is used to: After the step of dynamically deforming the static three-dimensional face model through multimodal features to obtain a dynamic three-dimensional model: Convert dynamic 3D models into 2D images through rasterization rendering engine; The two-dimensional image corresponding to any video frame in the real-time recorded video is output to the display device for real-time display.

[0068] The digital human modeling device provided by the present application adopts the digital human modeling method in the above embodiment, which can solve the technical problem that the digital human modeled is not realistic enough. Compared with the prior art, the beneficial effects of the digital human modeling device provided by the present application are the same as the beneficial effects of the digital human modeling method provided by the above embodiment, and the other technical features in the digital human modeling device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0069] The present application provides a digital human modeling device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the digital human modeling method in the above-mentioned embodiment one.

[0070] Reference below Figure 5 , which shows a schematic diagram of the structure of a digital human modeling device suitable for implementing the embodiment of the present application. The digital human modeling device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The digital human modeling device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0071] like Figure 5As shown, the digital human modeling device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM) 1004. In RAM 1004, various programs and data required for the operation of the digital human modeling device are also stored. The processing device 1001, ROM 1002, and RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the digital human modeling device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a digital human modeling device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0072] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0073] The digital human modeling device provided by the present application adopts the digital human modeling method in the above embodiment, which can solve the technical problem that the digital human modeled is not realistic enough. Compared with the prior art, the beneficial effects of the digital human modeling device provided by the present application are the same as the beneficial effects of the digital human modeling method provided by the above embodiment, and the other technical features in the digital human modeling device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0074] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0075] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0076] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the digital human modeling method in the above-mentioned embodiment.

[0077] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0078] The computer-readable storage medium may be included in the digital human modeling device; or may exist independently without being assembled into the digital human modeling device.

[0079] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the digital human modeling device, the digital human modeling device: obtains real-time recorded video, and extracts three-dimensional geometric features and multimodal features from the real-time recorded video; fuzzily reconstructs the face recorded in the real-time recorded video through the three-dimensional geometric features to obtain a static three-dimensional face model; and dynamically deforms the static three-dimensional face model through the multimodal features to obtain a dynamic three-dimensional model.

[0080] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0081] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0082] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0083] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned digital human modeling method, and can solve the technical problem that the digital human modeled at present is not realistic enough. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the digital human modeling method provided in the above-mentioned embodiment, and will not be elaborated here.

[0084] The present application also provides a computer program product, including a computer program, which implements the steps of the digital human modeling method as described above when executed by a processor.

[0085] The computer program product provided by this application can solve the technical problem that the digital human modeled is not realistic enough. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as those of the digital human modeling method provided by the above embodiment, which will not be elaborated here.

[0086] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A digital human modeling method, characterized in that: The digital human modeling method comprises: Acquire a real-time recorded video, and extract three-dimensional geometric features and multimodal features from the real-time recorded video; The human face recorded in the real-time recorded video is fuzzily reconstructed by using the three-dimensional geometric features to obtain a static three-dimensional human face model; The static three-dimensional face model is dynamically deformed by using the multimodal features to obtain a dynamic three-dimensional model.

2. The digital human modeling method according to claim 1, characterized in that: The step of fuzzily reconstructing the face recorded in the real-time recorded video by using the three-dimensional geometric features to obtain a static three-dimensional face model comprises: The face recorded in the real-time recorded video is fuzzily reconstructed based on 2D Gaussian sputtering using the posture parameters in the three-dimensional geometric features and the estimated point cloud parameters of the corresponding frame to obtain a static three-dimensional face model.

3. The digital human modeling method according to claim 1, characterized in that: The step of dynamically deforming the static three-dimensional face model by using the multimodal features to obtain a dynamic three-dimensional model comprises: Associating the multimodal features encoded by KAN with the Gaussian position points in the static three-dimensional face model to obtain a multi-head attention weight matrix; Based on the multi-head attention weight matrix, predicting a parameter change equivalent to the Gaussian sputtering parameter space in the static three-dimensional face model; The original parameters of the static three-dimensional face model are adjusted according to the parameter variation to obtain a dynamic three-dimensional model.

4. The digital human modeling method according to claim 1, characterized in that: The digital human modeling method further includes: Based on the historical static 3D face model, a static prior model, a feature prior model and a dynamic prior model are trained; By fine-tuning the static prior model, a real-time static model is obtained; By fine-tuning the feature prior model, a real-time feature model is obtained; By fine-tuning the dynamic prior model, a real-time dynamic model is obtained.

5. The digital human modeling method according to claim 4, characterized in that: The digital human modeling method further includes: Outputting a static three-dimensional face model through the real-time static model; Through the real-time feature model, multimodal features are output; Through the real-time dynamic model, a dynamic three-dimensional human face model is output.

6. The digital human modeling method according to claim 1, characterized in that: The step of dynamically deforming the static three-dimensional face model by using the multimodal features to obtain a dynamic three-dimensional model comprises: Converting the dynamic three-dimensional model into a two-dimensional image through a rasterization rendering engine; The two-dimensional image corresponding to any video frame in the real-time recorded video is output to a display device for real-time display.

7. A digital human modeling device, characterized in that: The digital human modeling device comprises: An extraction module, used to obtain real-time recorded video, and extract three-dimensional geometric features and multimodal features from the real-time recorded video; A static module, used for fuzzy reconstruction of the face recorded in the real-time recorded video by using the three-dimensional geometric features to obtain a static three-dimensional face model; The dynamic module is used to dynamically deform the static three-dimensional face model through the multimodal features to obtain a dynamic three-dimensional model.

8. A digital human modeling device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the digital human modeling method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the digital human modeling method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the digital human modeling method according to any one of claims 1 to 6 are implemented.

Images