Education scene-oriented three-dimensional digital human system
By designing a three-dimensional digital human system with voice-driven facial expressions and text-driven action generation in educational scenarios, the problem of lack of efficient interaction and multi-directional display in educational scenarios is solved, and personalized teaching experience and multi-angle learning effects are achieved, reducing the effect of User threshold.
Patent Information
- Application Number
- CN202410011769.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-03
- Publication Date
- 2024-11-12
AI Technical Summary
The existing technology lacks an efficient three-dimensional digital human system in educational scenarios, and cannot realize voice interaction and multi-directional display, resulting in limited vividness and real-time teaching experience, and the production cost of three-dimensional digital human teachers is high and the user threshold is high.
A three-dimensional digital human system for educational scenarios is designed, including a digital human classroom unit and a digital human teacher action teaching unit. Through voice-driven facial expression generation and text-driven action generation, combining lip-form prediction model and motion language model to realize digital Multi-directional interaction and personalized display of human teachers.
Created a more realistic and personalized three-dimensional digital human teacher model, realizes intelligent adjustment of facial expressions and voice content, improves speech comprehension accuracy, and allows digital humans to naturally respond to text instructions through the integration of actions and text, and through high Quality rendering technology solves the problem of echoing digital verbs and body movements, achieving multi-angle learning effect.
Smart Images

Figure CN118941679A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of three-dimensional digital human, and in particular to a three-dimensional digital human system for educational scenarios. Background Art
[0002] With the rapid development of virtual 3D digital human technology, the application of 3D digital human has gradually diversified. There are many 3D digital human technologies used in visual arts, medicine and other fields, but there is still a lack of 3D digital human technologies developed for educational scenarios. The digital humans currently used in the field of education are only two-dimensional digital humans that can answer simple concepts and provide user feedback. They are relatively rough as a whole and cannot perform two-way interactions or display relatively fine operations. At the same time, the cost of traditionally producing a 3D digital human animation video is very high and requires certain skills, making it difficult to operate. This has greatly increased the user threshold, making it difficult for the user audience who most needs 3D digital human teachers to quickly get started and obtain the educational consultation they need.
[0003] Three-dimensional digital human technology includes but is not limited to voice-driven three-dimensional digital population technology, text-driven action generation technology, speech-to-text technology and three-dimensional virtual human rendering technology. However, these technologies all have some shortcomings. Voice-driven three-dimensional digital population technology can only obtain facial network output from voice input, but cannot output a complete three-dimensional human model. This limitation is not friendly to the display of digital human classrooms and three-dimensional digital human teachers; due to the complicated process of text generation actions and relatively cumbersome operations, the use of text-driven action technology in the teaching process is not convenient and efficient enough; the traditional speech-to-text model has inaccurate prediction timestamps for each utterance, cannot provide word-level timestamps, and is difficult to accurately determine the appearance time of each word in the speech, which is not easy to handle subsequent work such as text alignment, speech segmentation and reorganization, and context understanding; currently, many three-dimensional virtual human renderings are mainly displayed in the form of video, and the effect presented in this way is similar to that of 2D. When the three-dimensional virtual teacher is rendered into an animated video, the observer can only observe the teacher model from a fixed angle. At present, there is no digital person who can interact with voice in the field of education. This lack leads to a series of limitations, and the interaction between students and educators in digital learning environments is mainly limited to traditional text input and click operations. Digital characters that lack voice interaction limit the vividness and real-time nature of the teaching experience and miss the potential advantages of voice as a natural and effective way of communication. Summary of the invention
[0004] The purpose of the present invention is to enable a three-dimensional digital human to provide multi-directional interaction for educated users in a more human-like and simulated form. In order to achieve the above purpose, the present invention provides a three-dimensional digital human system for educational scenarios, including:
[0005] Digital Human Classroom Unit and Digital Human Teacher Action Teaching Unit;
[0006] The digital human classroom unit is used to create a first digital human teacher, so that the first digital human teacher makes corresponding lip movements according to the teaching video;
[0007] The digital human teacher action teaching unit is used to input action description data and generate corresponding actions according to the action description data, so that the second digital human teacher can demonstrate the actions; the action description data includes voice description data and text description data.
[0008] Preferably, the digital human classroom unit includes:
[0009] A digital human construction module, used to create the first digital human teacher using modeling face-pinching tools and 3D animation software;
[0010] A data acquisition module, used to acquire voice data and facial animation data; the voice data at least includes Mandarin Chinese voice data, English voice data and custom voice data, and the custom voice data at least includes celebrity voice data and cartoon character voice data;
[0011] A model training module, used for training a lip shape prediction model according to the speech data and the facial animation data until a loss function of the lip shape prediction model converges; the lip shape prediction model comprises an input layer, an LSTM layer, an attention layer and an output layer, and the loss function comprises a Huber loss function and a cosine distance loss function;
[0012] A lip shape inference module, used for performing lip shape inference on the audio data corresponding to the teaching video according to the lip shape prediction model, and obtaining a lip shape action sequence corresponding to the audio data;
[0013] The lip shape output module is used to bind the lip shape action sequence with the facial bones of the first digital human teacher so that the first digital human teacher can make corresponding lip shapes.
[0014] Preferably, the digital human classroom unit further includes:
[0015] A lip shape optimization module is used to optimize the lip shape action sequence; the lip shape optimization module includes:
[0016] An oral model building module, used to build an oral model;
[0017] A lip shape calculation module, used for simulating the movement of the oral model and calculating the lip shape of the intermediate frames between the key frames according to the key frames in the lip action sequence;
[0018] A smoothing processing module is used to smooth the mouth shape to optimize the mouth action sequence.
[0019] Preferably, the digital human classroom unit further includes:
[0020] A rendering module is used to render the first digital human teacher using the three-dimensional animation software and generate a rendered animation of the first digital human teacher.
[0021] Preferably, the digital human teacher action teaching unit includes:
[0022] A data input module, used for inputting action description data;
[0023] A voice conversion module, used for determining whether the action description data is voice description data, and if so, converting the voice description data to obtain text description data corresponding to the voice description data;
[0024] A text action data acquisition module, used to acquire a text action data set; the text action data set at least includes a KIT data set and a HumanML3D data set;
[0025] A text-driven action generation module, used for training a motion language model according to the text action data set, so that the motion language model generates corresponding actions according to the action description data; the motion language model includes a word segmenter and a T5 model;
[0026] The action visualization module is used to construct a second digital human teacher using a parameterized human body model so that the second digital human teacher can demonstrate the action.
[0027] Preferably, the text-driven action generation module includes:
[0028] A word segmenter training module, used for training the word segmenter according to the text action data set, so that the word segmenter learns the discrete representation of the action from the text action data set; the word segmenter adopts the architecture based on vector quantization variational autoencoder;
[0029] The T5 model training module is used to train and fine-tune the T5 model according to the text-action dataset, and pre-train the T5 model by combining unsupervised learning and supervised learning, so that the T5 model learns the association relationship between text and action from the text-action dataset.
[0030] Preferably, the text-driven action generation module further includes:
[0031] A model optimization module is used to construct a multi-task dataset according to the text action dataset, and train the motion language model according to the multi-task dataset to optimize the motion language model.
[0032] Preferably, the model optimization module includes:
[0033] A multi-task dataset construction module, used to construct the text action dataset into a multi-task dataset, and divide the multi-task dataset into a training set and a test set;
[0034] A classification module, used for classifying the tasks and instructions of the training set so that each task corresponds to one instruction;
[0035] A module for acquiring instructions to be tuned, used for training the motion language model according to the instructions, and evaluating the motion language model according to the test set, to obtain instructions to be tuned that do not meet the evaluation criteria; the evaluation criteria at least include F ID distance and R precision;
[0036] An instruction tuning module is used to tune the instruction to be tuned and retrain the motion language model according to the tuned instruction to optimize the motion language model; the tuning at least includes modifying the wording and grammatical structure of the instruction to be tuned.
[0037] Compared with the prior art, the three-dimensional digital human system for educational scenarios in the embodiment of the present invention has the following beneficial effects: the teacher model of the created three-dimensional digital human is more realistic and personalized; the intelligent adjustment of the teacher's facial expressions and voice content is realized, so that the virtual teacher can express lip movements more naturally, thereby providing a more personalized and realistic virtual teaching experience; the voice-to-text format is realized to improve the accuracy of voice comprehension; the motion language model is used to integrate actions and texts through training and instruction tuning, so that the three-dimensional digital human can respond to text instructions more naturally; the high-quality modeling and rendering technology is used to ensure the appearance of the digital character, solving the problem that the three-dimensional digital human body shape cannot echo the body movements; the three-dimensional digital human is rendered on the web page, achieving the effect of multi-angle observation and learning. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 It is a structural schematic diagram of a three-dimensional digital human system for educational scenarios according to an embodiment of the present invention;
[0039] Figure 2 is a schematic diagram of the effect of the digital human classroom unit of an embodiment of the present invention;
[0040] Figure 3 It is a structural diagram of a digital human classroom unit according to an embodiment of the present invention;
[0041] Figure 4 is a schematic diagram of the structure of the lip shape prediction model according to an embodiment of the present invention;
[0042] Figure 5 is a schematic diagram of the structure of a lip shape optimization module according to an embodiment of the present invention;
[0043] Figure 6 2 is a schematic diagram showing the effect of the digital human teacher action teaching unit according to an embodiment of the present invention;
[0044] Figure 7 This is another effect schematic diagram of the digital human teacher action teaching unit according to an embodiment of the present invention;
[0045] Figure 8 2 is a schematic diagram of the structure of a digital human teacher action teaching unit according to an embodiment of the present invention;
[0046] Fig. 9 It is a structural schematic diagram of a text-driven action generation module according to an embodiment of the present invention;
[0047] Fig.10 It is a structural diagram of the model optimization module of an embodiment of the present invention. DETAILED DESCRIPTION
[0048] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.
[0049] like Figure 1 As shown, an embodiment of the present invention provides a three-dimensional digital human system for educational scenarios, including: a digital human classroom unit 1 and a digital human teacher action teaching unit 2.
[0050] Digital Human Classroom Unit 1 simulates the classroom teaching mode including teachers, blackboards and video courseware. Users can choose virtual teacher models (simulating real people and cartoon models) and teaching videos for learning. For details, please refer to Figure 2 This unit includes digital human modeling technology, audio-driven facial technology, and rendering technology. The virtual teacher can make corresponding lip shapes according to different videos, and the image is diverse, which can increase the fun.
[0051] Specifically, the digital human classroom unit 1 is used to create a first digital human teacher so that the first digital human teacher makes corresponding lip movements according to the teaching video.
[0052] Furthermore, if Figure 3 As shown, Digital Human Classroom Unit 1 includes:
[0053] A digital human construction module 11 is used to create a first digital human teacher by using a modeling face pinching tool and 3D animation software;
[0054] Specifically, the preferred modeling and face-pinching tool in this embodiment is Metahuman Creator, and the 3D animation software is Maya. Metahuman Creator is used to create the first digital human teacher, and its appearance and features are adjusted as needed. The model is saved and exported to a common file format suitable for Maya (such as FBX).
[0055] The Metahuman model was imported into Maya, and Maya’s HumanIK tool was used to set up a proper human skeleton system by setting up the bone hierarchy, binding the character, and setting up animation controllers.
[0056] A data acquisition module 12, used to acquire voice data and facial animation data;
[0057] Specifically, the voice data includes at least Chinese Mandarin voice data, English voice data and custom voice data, and the custom voice data includes at least celebrity voice data and cartoon character voice data. The preferred Chinese Mandarin voice data in this embodiment is the Free ST Chinese Mandarin Corpus dataset, which includes about 130 hours of recording data, recorded by speakers from China, and can meet the needs of general Mandarin voice.
[0058] The preferred English speech data in this embodiment is the TED-LIUM dataset, which includes 1,495 TED speech recordings. Using this dataset allows the training results to make corresponding lip shapes according to English, increasing practicality.
[0059] The customized voice data of this embodiment includes not only celebrity voice data and cartoon character voice data, but also one's own voice data.
[0060] The preferred facial animation data of this embodiment is the facial animation data that comes with FACEGOOD, which covers a series of common facial expressions, such as smiling, frowning and blinking, and can be intelligently adjusted according to the input voice data to produce vivid and emotional facial expression animations.
[0061] A model training module 13, used for training the lip shape prediction model according to the speech data and the facial animation data until the loss function of the lip shape prediction model converges;
[0062] Specifically, the speech data is processed by linear predictive coding (LPC), the speech data is divided into frame data corresponding to the animation, and the Maya animation frame data is exported. The processed speech data is input into the lip shape prediction model, and then trained until the loss function of the model converges.
[0063] Furthermore, the lip shape prediction model is an end-to-end neural network, including an input layer, an LSTM layer, an attention layer, and an output layer. For details, see Figure 4 Among them, the forward state sequence is calculated using the following formulas: and reverse state sequence
[0064]
[0065]
[0066]
[0067] Among them, σ represents the nonlinear activation function of the hidden node, W xh and W hh denote the input-hidden weight matrix and the hidden-hidden weight matrix, respectively, and b h represents the bias vector.
[0068] Assume that the output vector generated by the LSTM layer is H = [h1,h2,...,h T ], where T represents the length of the sentence, and then the weighted sum of these output vectors (weight α) is used through the attention layer to form a sequence representation y. The specific formula is as follows:
[0069] α=softmar(W T σ(H)
[0070] y=Hα T
[0071] For the lip shape prediction model, its loss function consists of two parts, including target loss and smoothing loss. The loss function is specifically expressed by the following formula:
[0072]
[0073] Among them, y i and i p Represent the output and expected output of the model respectively, L t and L s They represent the target loss and smoothing loss respectively, ω1 and ω2 represent the target weight and smoothing weight respectively.
[0074] The target loss is used to ensure that each parameter is roughly accurate. Since the Huber loss is not as sensitive to outliers in the data set as the MSE loss, and can converge to the minimum value faster than the MSE loss and MAE loss, the Huber loss is used as the target loss. The specific formula is as follows:
[0075]
[0076] In order to generate high-quality facial animation, smoothness loss is introduced, and cosine distance is used to measure the smoothness cost between two adjacent frame samples. The specific formula is as follows:
[0077]
[0078] That is, the loss function of the lip shape prediction model in this embodiment includes the Huber loss function and the cosine distance loss function.
[0079] The lip shape inference module 14 is used to perform lip shape inference on the audio data corresponding to the teaching video according to the lip shape prediction model to obtain a lip shape action sequence corresponding to the audio data;
[0080] The lip shape output module 15 is used to bind the lip shape action sequence with the facial bones of the first digital human teacher so that the first digital human teacher can make corresponding lip shapes.
[0081] Specifically, the lip-sync action sequence is bound to the facial bones of the Metahuman model using the deformable mesh technology in Maya or other plug-ins, and applied to the Metahuman model.
[0082] In order to further adjust and optimize the lip movement sequence, Digital Human Classroom Unit 1 also includes:
[0083] The lip shape optimization module 16 is used to optimize the lip shape action sequence. Specifically, Figure 5 As shown, the lip shape optimization module 16 includes:
[0084] An oral cavity model building module 16-A is used to build an oral cavity model;
[0085] Specifically, an oral model is established based on anatomical knowledge and medical imaging data, including structures inside the oral cavity (such as the tongue, teeth, and jawbone) and related muscle models.
[0086] The lip shape calculation module 16-B is used to simulate the movement of the oral model and calculate the lip shape of the intermediate frames between the key frames according to the key frames in the lip action sequence;
[0087] Specifically, by simulating the movement of the oral model, the shape of the mouth in each frame of the mouth action sequence can be calculated according to the contraction and extension of the muscles, which can be achieved through physical simulation methods, such as using a mass spring model or finite element analysis. In the process of simulating the movement of the oral model, some physical constraints can be added, such as collision detection of the internal structure of the oral cavity, muscle length restrictions, etc., to ensure that the changes in the mouth shape conform to physiological laws. The mouth shape of the intermediate frames between the key frames in the mouth action sequence is calculated according to the key frames, which can be achieved through numerical calculation or numerical optimization methods.
[0088] The smoothing module 16-C is used to smooth the lip shape to optimize the lip movement sequence.
[0089] Furthermore, the digital human classroom unit 1 also includes:
[0090] The rendering module 17 is used to render the first digital human teacher using three-dimensional animation software and generate a rendering animation of the first digital human teacher.
[0091] Specifically, use Maya's material editor and texture mapping function to map and adjust material properties of the first digital human teacher to achieve a more realistic appearance; set appropriate lights and cameras in Maya to obtain the desired lighting and perspective effects, and make adjustments based on the teaching scene of the first digital human teacher to highlight the characteristics of the character model; use Maya renderer, adjust parameters such as pixel resolution, anti-aliasing, shadows, and global illumination in the rendering settings to obtain high-quality rendering effects, and by setting the rendering parameters, select the rendering animation sequence to generate the final rendered animation.
[0092] The main function of the Digital Human Teacher Action Teaching Unit 2 is to generate action driven by voice or text. The user inputs a piece of text or voice to describe an action, such as ballet dance action, jumping jacks in warm-up action, and three-point shooting action in basketball. The unit can generate a corresponding digital human to show the corresponding action. For details, please refer to Figure 6 and Figure 7 The user can use the mouse to drag the interface to observe the movements of the digital human teacher at various angles, which can achieve better movement teaching effects.
[0093] Specifically, the digital human teacher action teaching unit 2 is used to input action description data and generate corresponding actions according to the action description data, so that the second digital human teacher can demonstrate the actions.
[0094] Furthermore, if Figure 8 As shown, the digital human teacher action teaching unit 2 includes:
[0095] A data input module 21, used for inputting action description data;
[0096] Specifically, the action description data includes voice description data and text description data.
[0097] A voice conversion module 22, used to determine whether the action description data is voice description data, and if so, convert the voice description data to obtain text description data corresponding to the voice description data;
[0098] Specifically, when the action description data is determined to be speech description data, this embodiment uses WhisperX to convert the speech description data. WhisperX improves Whisper transcription by introducing three additional stages of pre-segmentation of input audio by voice activity detection (VAD), cutting and merging VAD segments, and forcing phoneme alignment, so as to achieve the effect of efficient speech transcription of long-format audio with accurate word-level timestamps.
[0099] A text action data acquisition module 23, used to acquire a text action data set;
[0100] Specifically, the preferred text action dataset of this embodiment includes at least the KIT dataset and the HumanML3D dataset. Among them, the KIT dataset provides 6353 text descriptions corresponding to 3911 motion sequences; the HumanML3D dataset includes a large number of 14616 motion sequences obtained from the existing large-scale motion capture dataset AMASS. In addition, this embodiment also obtains 44970 sequence-level text descriptions.
[0101] A text-driven action generation module 24 is used to train the motion language model according to the text action data set so that the motion language model generates corresponding actions according to the action description data;
[0102] Specifically, the preferred motion language model in this embodiment is the MotionGPT model, which includes a word segmenter and a T5 model.
[0103] Furthermore, if Fig. 9 As shown, the text-driven action generation module 24 includes:
[0104] A word segmenter training module 24-A, used to train the word segmenter according to the text action dataset, so that the word segmenter learns the discrete representation of the action from the text action dataset;
[0105] Specifically, the tokenizer adopts an architecture based on the Vector Quantized Variational Autoencoder (VQ-VAE). The tokenizer consists of an encoder and a decoder. The encoder first applies a one-dimensional convolution to the given frame-level motion features along the time dimension to obtain a latent vector; then, discrete quantization is used to convert the latent vector into a set of codebook entries Z, where Z consists of K latent vectors, each with a dimension of d. During the quantization process, a quantization function is used to replace each row vector with the closest codebook entry in Z. After quantization, the decoder receives the latent vector generated by the encoder, maps the latent vector back to the closest latent vector through a decoding operation, and restores the original sequence of motion features.
[0106] In order to train the word segmenter, this embodiment introduces three different loss functions to optimize the word segmenter, which are specifically expressed by the following formula:
[0107] Lv=Lr+Le+Lc
[0108] Among them, Lr represents reconstruction loss, Le represents embedding loss, and Lc represents commitment loss. In order to further improve the quality of generated actions, L1 smoothing loss and velocity regularization are used in the reconstruction loss Lr, as well as exponential moving average (EMA) and codebook reset techniques to improve the utilization of the codebook during training. By training the word segmenter, the action can be effectively represented as a language.
[0109] The T5 model training module 24-B is used to train and fine-tune the T5 model based on the text-action dataset, and pre-train the T5 model by combining unsupervised learning and supervised learning, so that the T5 model can learn the association between text and action from the text-action dataset.
[0110] Specifically, the T5 model is first trained and fine-tuned on a natural language dataset containing guiding phrases (such as the HumanML3D dataset), and then the T5 model is pre-trained through unsupervised learning and supervised learning.
[0111] In unsupervised pre-training, a corresponding target sequence is constructed by randomly replacing a certain proportion (such as 15%) of the input symbols with special markers. When constructing the target sequence, a special marker is used to define the deleted symbol segment (for example, assuming the input sequence is: "I like to eat apples, bananas, and oranges.", where the special marker is "[MASK]", and a part of the symbols are randomly replaced. The replaced input sequence may be: "I [MASK] eat apples, [MASK], oranges.", where "[MASK]" is used to represent the deleted symbol segment.), and an additional marker is added to indicate the end of the target sequence, and then the deleted symbol segment is extracted to obtain the target sequence. This embodiment realizes unsupervised training by learning the relationship between the input sequence and the corresponding target sequence. In supervised pre-training, a paired text action data set is used to supervise the relationship between action and language, so that the T5 model can understand the association between text and action. By pre-training the T5 model, the text information and action sequence can be effectively and seamlessly integrated, and a deep understanding of the relationship between text and action can be obtained.
[0112] Furthermore, the text-driven action generation module 24 further includes:
[0113] The model optimization module 24-C is used to construct a multi-task dataset based on the text action dataset, and train the motion language model based on the multi-task dataset to optimize the motion language model.
[0114] Specifically, Fig.10 As shown, the model optimization module 24-C includes:
[0115] A multi-task dataset construction module 24-C1, used to construct the text action dataset into a multi-task dataset, and divide the multi-task dataset into a training set and a test set;
[0116] Specifically, based on text-action datasets (such as KIT and HumanML3D), they are constructed as a multi-task dataset, and the multi-task dataset is divided into training and testing sets.
[0117] A classification module 24-C2, for classifying the tasks and instructions of the training set so that each task corresponds to an instruction;
[0118] Specifically, in the process of building a multi-task dataset, we first need to define the tasks and instruction templates, that is, to classify and summarize the tasks and instructions in the dataset, and then design corresponding instruction templates for each task. For each task, there are dozens of different instruction templates designed, totaling more than a thousand different tasks, each with a unique instruction (for example, the instruction for the action generation task can be "Can you generate an action sequence depicting 'a person imitating the movements of the waltz'?").
[0119] The module 24-C3 for obtaining instructions to be tuned is used to train the motion language model according to the instructions, and to evaluate the motion language model according to the test set, so as to obtain instructions to be tuned that do not meet the evaluation criteria;
[0120] Specifically, the evaluation criteria include at least FID distance and R precision. In addition, the evaluation criteria of this embodiment can also adopt: 1. Generation diversity: DIV; 2. Multimodal distance (MM Dist) measuring the distance between action and text; 3. Language quality (such as Rouge and Cider).
[0121] The instruction tuning module 24-C4 is used to tune the instructions to be tuned, and retrain the motion language model according to the tuned instructions to optimize the motion language model.
[0122] Specifically, tuning at least includes modifying the wording and grammatical structure of the instructions to be tuned, making them more accurate and clear while avoiding confusion with other instructions. The motion language model is retrained using the tuned instructions and evaluated on the test set, and it is found that the performance of the motion language model under the new instructions is improved. Through the instruction tuning module, instructions can be further optimized to improve the performance and generalization ability of the motion language model, so as to better adapt to different text action tasks.
[0123] The action visualization module 25 is used to construct a second digital human teacher using a parameterized human body model so that the second digital human teacher can demonstrate actions.
[0124] Specifically, the preferred parameterized human body model in this embodiment is the SMPL model. The SMPL model can represent different human bodies and postures. The SMPL model generated on the MotionGPT model is imported into the three-dimensional graphics software Blender to create an SMPL mesh so that the second digital human teacher can show movements.
[0125] In summary, the embodiment of the present invention is a three-dimensional digital human system for educational scenarios. The teacher model of the created three-dimensional digital human is more realistic and personalized; the intelligent adjustment of the teacher's facial expressions and voice content is realized, so that the virtual teacher can express lip movements more naturally, thereby providing a more personalized and realistic virtual teaching experience; the speech-to-text format is realized to improve the accuracy of speech comprehension; the motion language model is used to integrate actions and texts through training and instruction tuning, so that the three-dimensional digital human can respond to text instructions more naturally; the high-quality modeling and rendering technology is used to ensure the appearance of the digital character, and the problem that the three-dimensional digital human body shape cannot echo the body movements is solved; the three-dimensional digital human is rendered on the web page, achieving the effect of multi-angle observation and learning.
[0126] Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be directly referenced to each other, and each embodiment focuses on the differences from other embodiments. It should be noted that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0127] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.
Claims
1. A three-dimensional digital human system for educational scenarios, characterized in that: include: Digital Human Classroom Unit and Digital Human Teacher Action Teaching Unit; The digital human classroom unit is used to create a first digital human teacher, so that the first digital human teacher makes corresponding lip movements according to the teaching video; The digital human teacher action teaching unit is used to input action description data and generate corresponding actions according to the action description data, so that the second digital human teacher can demonstrate the actions; the action description data includes voice description data and text description data.
2. The three-dimensional digital human system according to claim 1, characterized in that: The digital human classroom unit includes: A digital human construction module, used to create the first digital human teacher using modeling face-pinching tools and 3D animation software; A data acquisition module, used to acquire voice data and facial animation data; the voice data at least includes Mandarin Chinese voice data, English voice data and custom voice data, and the custom voice data at least includes celebrity voice data and cartoon character voice data; A model training module, used for training a lip shape prediction model according to the speech data and the facial animation data until a loss function of the lip shape prediction model converges; the lip shape prediction model comprises an input layer, an LSTM layer, an attention layer and an output layer, and the loss function comprises a Huber loss function and a cosine distance loss function; A lip shape inference module, used for performing lip shape inference on the audio data corresponding to the teaching video according to the lip shape prediction model, and obtaining a lip shape action sequence corresponding to the audio data; The lip shape output module is used to bind the lip shape action sequence with the facial bones of the first digital human teacher so that the first digital human teacher can make corresponding lip shapes.
3. The three-dimensional digital human system according to claim 2, characterized in that: The digital human classroom unit also includes: A lip shape optimization module is used to optimize the lip shape action sequence; the lip shape optimization module includes: An oral model building module, used to build an oral model; A lip shape calculation module, used for simulating the movement of the oral model and calculating the lip shape of the intermediate frames between the key frames according to the key frames in the lip action sequence; A smoothing processing module is used to smooth the mouth shape to optimize the mouth action sequence.
4. The three-dimensional digital human system according to claim 2, characterized in that: The digital human classroom unit also includes: A rendering module is used to render the first digital human teacher using the three-dimensional animation software and generate a rendered animation of the first digital human teacher.
5. The three-dimensional digital human system according to claim 1, characterized in that: The digital human teacher action teaching unit includes: A data input module, used for inputting action description data; A voice conversion module, used for determining whether the action description data is voice description data, and if so, converting the voice description data to obtain text description data corresponding to the voice description data; A text action data acquisition module, used to acquire a text action data set; the text action data set at least includes a KIT data set and a HumanML3D data set; A text-driven action generation module, used for training a motion language model according to the text action data set, so that the motion language model generates corresponding actions according to the action description data; the motion language model includes a word segmenter and a T5 model; The action visualization module is used to construct a second digital human teacher using a parameterized human body model so that the second digital human teacher can demonstrate the action.
6. The three-dimensional digital human system according to claim 5, characterized in that: The text-driven action generation module includes: A word segmenter training module, used for training the word segmenter according to the text action data set, so that the word segmenter learns the discrete representation of the action from the text action data set; the word segmenter adopts the architecture based on vector quantization variational autoencoder; The T5 model training module is used to train and fine-tune the T5 model according to the text-action dataset, and pre-train the T5 model by combining unsupervised learning and supervised learning, so that the T5 model learns the association relationship between text and action from the text-action dataset.
7. The three-dimensional digital human system according to claim 6, characterized in that: The text-driven action generation module also includes: A model optimization module is used to construct a multi-task dataset according to the text action dataset, and train the motion language model according to the multi-task dataset to optimize the motion language model.
8. The three-dimensional digital human system according to claim 7, characterized in that: The model optimization module includes: A multi-task dataset construction module, used to construct the text action dataset into a multi-task dataset, and divide the multi-task dataset into a training set and a test set; A classification module, used for classifying the tasks and instructions of the training set so that each task corresponds to one instruction; The module for acquiring instructions to be tuned is used to train the motion language model according to the instructions, and evaluate the motion language model according to the test set to obtain instructions to be tuned that do not meet the evaluation criteria; the evaluation criteria at least include FID distance and R precision; An instruction tuning module is used to tune the instruction to be tuned and retrain the motion language model according to the tuned instruction to optimize the motion language model; the tuning at least includes modifying the wording and grammatical structure of the instruction to be tuned.
Citation Information
Cited By
Immersive virtual teaching system based on multi-modal perception and dynamic evaluation
CN120599153A
