Teaching-friendly digital human online MOOC construction system, method, equipment and medium

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to a teaching-friendly digital human online MOOC construction system, method and device and a medium, and the system comprises a teaching digital human intelligent construction module which constructs a teaching digital human; the teaching digital human intelligent driving action construction module is used for forming teaching digital human intelligent driving actions; the teaching digital human intelligent scene construction module is used for constructing an intelligent scene; the teaching digital person intelligent driving module is used for driving the teaching digital person in the intelligent scene based on the teaching digital person intelligent driving action; and the intelligent scene intelligent adjustment module realizes intelligent adjustment of the intelligent scene. While the teaching effectiveness is promoted, the time cost invested by a teacher in MOOC manufacturing can be reduced.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and relates to an online MOOC construction system, method, device and medium, and in particular to a teaching-friendly digital human online MOOC construction system, method, device and medium. Background Art

[0002] In recent years, the rapid development of Internet technology and continuous technological innovation have promoted further reforms in the form of education. As one of the main forms of online education in China, MOOC (Massive open online course) online education has the characteristics of large-scale, open and online. "Large-scale" means taking into account learners of different levels and needs, "open" means high-quality course standards under open and transparent peer review, and "online" means the change of student teaching media, and it is difficult for students to achieve all-round observation and interaction with teaching aids. High-quality MOOC teaching videos are an important carrier for delivering high-quality MOOC teaching content. Its construction system covers a variety of factors such as lecturers, students, recording and broadcasting systems, recording personnel, editors, and environments. The recording types include traditional studios, automatic recording in studios, real-life shooting, and classroom recording. From venue layout to personnel scheduling, from hardware systems to video communication, traditional MOOC recording methods are highly dependent on venues and manpower, and the MOOC recording cycle ranges from three months to half a year. As the core of teaching design and teaching implementation, teachers not only need to coordinate the arrangement of MOOC recording scripts and plan course content outlines, but also need to make adequate preparations for content shooting and avoid mistakes in shooting, which will greatly consume teachers' energy. To this end, some technologies have been invented to assist in the production of MOOCs:

[0003] 1. Intelligent driving of 3D digital human. The intelligent driving of digital human mainly includes two parts: facial driving and body movement driving.

[0004] Talking-face Generation (TFG) synthesizes the corresponding facial features and expressions by giving the voice of a given speaker. The facial coding part mainly includes three coding methods: landmark-based, vertices-based, and coefficient-based. Among these three coding methods, landmarks rely on the accurate location of key point features, vertices may have some missing three-dimensional coordinates, and the coefficient-based method requires a large amount of training data. Among facial movements, the movement of the mouth is strongly correlated with the content of the speaker, while the expression of the landscape face, the posture of the head, and the blinking are less correlated with the content of the speech, but affect the natural expression of the generated face. In addition, the use of pre-trained models and multimodal input or hierarchical methods to drive the face is the current development trend.

[0005] Gesture generation for digital humans generates natural and expressive body movements based on the input of multimodal information such as speech audio, text or speaker identity. The early methods for driving body movements were rule-based and relied on pre-set correspondences in states such as human conversations or speeches. However, this method has limitations. The setting of rules and the scale of predefined actions greatly affect the quality of the results and require a lot of manpower costs. In order to overcome these problems, in recent years, deep learning-based technologies have attracted widespread attention in digital human gesture generation. Recurrent neural networks (RNN), long short-term memory recursive neural networks (LSTM), generative adversarial networks (GAN), vector quantized variational autoencoders (VQ-VAE), diffusion models, etc. have been used to generate body movements from speech. Although these architectures can achieve rhythmic alignment of generated gestures and audio, the task of generating gestures with good semantic expression is still under exploration. Some of these methods use text transcription of speech as input to improve the semantics of gestures. Someone proposed a method to drive the generation of body part movements based on information such as face, body, audio, and text transcription. However, simply taking text as input does not provide a deep understanding of semantic information. Therefore, some methods use some specific structures to better learn the semantic information of audio. Some people have borrowed the idea of ​​the CLIP structure, taking text as a semantic description, using the textencoder part of the CLIP pre-trained model and the intermediate discrete embedding of the motion through the VQ-VAE structure to compare and learn the similarity, and combined with the reconstruction loss of motion to achieve the mapping of text and semantic motion sequences. Others also use contrastive learning to learn the mapping relationship between text and action sequences in the latent space, and the obtained embedding of the contrast space is used as a conditional to fuse the semantic information into the generated gestures through cross-attention. However, the effect of training is often determined by the data quality and data distribution of the data set. The semantic words in the existing digital human body motion data set are relatively sparse, and it is difficult for deep learning methods to represent semantic words with low or no frequency, while there is a many-to-many mapping relationship between semantic words and body motions. Based on the current situation and in view of LLM's ability to understand the complex relationships and metaphorical meanings between words, relevant scholars have recently used LLM to enhance the semantic expression in the task of generating gestures with audio by expanding the training dataset and externally matching the semantic action dataset.

[0006] 2. Design related to Pedagogical Agents

[0007] Researchers are focusing on virtual learning environments (Visual Learning Environments, VLE). There are already some related patents that apply digital humans to remote teaching design. A narrative-integrated teaching aid automatic generation method aims to establish a mapping relationship between text and voice and teaching aids, focusing on the integration of teaching aid objects and the environment. A digital human teacher personalized teaching device focuses on personalized design, aiming to achieve personalized support and assistance for digital humans in teaching scenarios through data collection, interaction, and personalized analysis and evaluation feedback. A teacher-side virtual human classroom recording video processing method mainly processes the teacher's recorded video, captures the teacher's facial features and body movements, reconstructs the teacher's three-dimensional model, and re-drives the teacher model to generate movements based on the recorded video to solve the problem of poor recording effect. However, they did not generate content other than the original recorded data.

[0008] Pedagogical agents (PAs), as an important part of virtual learning environments, convey teaching information through language and non-verbal behavior. Non-verbal communication refers to all elements of communication other than language, including paralanguage (tone, pitch) in voice and body language (facial expressions, eye contact, body movements, etc.) in non-voice. Many studies have shown that teachers in real classrooms and PAs in online virtual learning environments can help enhance students' learning experience and performance through some indicative, metaphorical and symbolic gestures. A working method of a virtual teaching system based on an AI assistant designs virtual avatars for teachers and students in a virtual classroom environment, and constructs a teacher action dataset, a classroom scene dataset, and a teaching activity dataset through facial recognition and avatar tracking technology of teachers, and by annotating classroom environments and teaching activities, constructs the image of an intelligent AI assistant, and constructs the spatiotemporal association of narratives, objects, and scenes. Then, teaching activities are organized through voice, gestures, virtual teaching drawings, collaborative teaching content teaching, and teaching content display. Among them, the action design of teachers and teaching environment is manifested in the movement and stay of the position in the teaching environment. The design of gestures mainly uses the gesture feature point recognition of 2D video (there may be cases where it cannot be recognized - this is just to build a data set) + Unity's hand interaction library + smoothing of fuzzy positions. It can be seen that their data quality is not enough, and they have not done a very detailed gesture design and research for adaptive teaching. Regarding fine-grained classroom gesture encoding, some scholars collected classroom teaching videos and analyzed and sorted out the behavioral representations that appeared in the classroom teaching, including symbolic actions, explanatory actions, expressive actions, adaptive actions, regulatory actions, distance actions, instrumental actions, etc. Among them, instrumental actions reflect the coordination between teachers and teaching materials in classroom teaching scenes. Some people consider teaching gestures from the perspective of demonstration, guidance, attention attraction, and emphasis based on educational purposes, and finally determine six common classroom gestures including intentional / unintentional, habitual, guiding, interactive, emphasizing, constructing, and visual gestures. Some people also focus on how PA used in holographic projection in intelligent teaching systems should design the generation of reusable standard teaching gestures. They designed a two-person collaborative task and captured and analyzed the gestures of the presenter, clustered them into 7 gesture categories including instructions, signs, metaphors, symbols, rhythms, connections, imitations, and interactive gestures, and designed corresponding hand movement descriptions. These works explored the gestures that divide the teaching effectiveness of PA in different media.

[0009] By comparing with the existing technologies, it is found that although they have realized the application of digital human in teaching scenarios, such as three-dimensional comprehensive teaching field, personalized education support, automatic generation of teaching aids integrating virtual and real, and intelligent MOOC generation, in the design of the scene, they often consider the scanning and reconstruction of the real teaching scene, and rarely consider the inherent advantages of digital human as a virtual agent in the virtual environment, that is, considering the switching of virtual scenes according to the adaptability of teaching content in real teaching; in the intelligent driving part of teaching digital human, although the existing work considers the use of language and non-language behavior (facial expression, eye contact, orientation, body movement), some work obtains the behavior of real teachers through motion capture to drive digital human in a mirror, and some work extracts audio representation as input to realize the generation and drive of digital human facial and body movements, as well as further optimization of movement generation - semantic enhancement is achieved by retrieving and matching corresponding gestures with the help of external semantic-gesture data set + LLM. The former requires the participation and cooperation of teachers, and the latter generates driving gestures, although the semantic expression of gestures is considered, but lacks further consideration of the teaching nature of gestures and the spatial attributes of teaching materials.

[0010] Therefore, in view of the defects existing in the above-mentioned prior art, it is necessary to develop a new teaching-friendly digital human online MOOC construction system, method, device and medium. Summary of the invention

[0011] In order to overcome the defects of the prior art, the present invention proposes a teaching-friendly digital human online MOOC construction system, method, device and medium, which aims to build an intelligent generation system for teaching digital humans with the help of large language model technology, text-to-speech technology, full-body generation technology of intelligent digital humans with voice, 3D point cloud reconstruction technology and UE tools, and render efficient, intelligent and accurate 3D teaching scenes through effective teaching action design and teaching space scene design, so as to reduce the time cost of teachers invested in MOOC production while promoting teaching effectiveness.

[0012] In order to achieve the above object, the present invention provides the following technical solutions:

[0013] A teaching-friendly digital human online MOOC construction system, characterized by comprising:

[0014] The intelligent construction module of teaching digital human is used to construct a teaching digital human whose image is close to that of the target teacher;

[0015] A teaching digital human intelligent driving action construction module, which is used to generate a basic action sequence based on voice and text, obtain an enhanced action sequence based on the text using a fine-tuned large language model, and fuse the basic action sequence and the enhanced action sequence to obtain an action sequence enhanced with teaching semantics for the teaching digital human. At the same time, the action sequence enhanced with teaching semantics is coordinated with the teaching materials in the teaching space to form an intelligent driving action for the teaching digital human;

[0016] A teaching digital human intelligent scene construction module, which is used to construct a teaching digital human intelligent scene;

[0017] A teaching digital human intelligent driving module, which is used to drive the teaching digital human in the intelligent scene based on the teaching digital human intelligent driving action;

[0018] The intelligent adjustment module of the intelligent scene is used to obtain the activation time of each intelligent scene and the time and angle of the digital human's direction change based on the text and time information corresponding to the intelligent driving action of the teaching digital human and the angle information corresponding to each intelligent scene, and realize the intelligent adjustment of the intelligent scene based on it.

[0019] Preferably, the teaching digital human intelligent driving action building module includes:

[0020] The audio rhythm action generation submodule is used to use the audio analysis library and the automatic speech recognition model to extract the low-dimensional and high-dimensional audio feature information in the speech, use the word vector model to extract the corresponding word embedding in the text, and generate the basic action vector z through the decoder of the pre-trained Transformer model. q *;

[0021] The teaching collaborative action enhancement submodule is used to obtain an enhanced action index based on the text using a fine-tuned large language model, retrieve matching text actions from a text action dataset based on the enhanced action index, and encode the matching text actions through an encoder of a pre-trained vector quantized variational autoencoder to obtain a quantized label action vector z e , z is achieved through weighted fusion q * and z e The fusion of the semantically enhanced label action vector z e-argu *, the semantically enhanced label action vector z e-argu * The decoder of the pre-trained vector quantized variational autoencoder decodes the action sequence of the teaching digital human with enhanced teaching semantics;

[0022] The teaching collaborative action enhancement submodule is used to set different types of collaborative actions according to the types of teaching materials, and adopt different methods based on different types of collaborative actions to achieve the coordination between the teaching semantically enhanced action sequence and the teaching materials.

[0023] Preferably, z is achieved by weighted fusion q * and z e The fusion of the semantically enhanced label action vector z e-argu *Specifically include:

[0024] Determine the time range of fusion according to the position of the enhanced action index, divide the basic action according to the time range to form multiple basic action segments, calculate the change of the movement speed of the basic action, and determine the position with the largest speed change as the fusion splicing point;

[0025] Use the label action vector to replace the corresponding basic action vector at the splicing point, and compare z before and after the replacement point. q * and z e Perform a weighted merge operation.

[0026] Preferably, the types of teaching materials include 2D plane class, 3D scene class, 3D fixed object class and 3D handheld object class. The coordinated actions corresponding to the 2D plane class, 3D scene class and 3D fixed object class are pointing actions. The coordinated actions corresponding to the 3D handheld object class are presentation actions. The pointing actions use the IK reverse settlement method to achieve the coordination between the action sequence enhanced by the teaching semantics and the teaching materials. The presentation actions use a preset action library to achieve the coordination between the action sequence enhanced by the teaching semantics and the teaching materials.

[0027] Preferably, constructing a teaching digital human that is close in appearance to the target teacher is specifically as follows: photographing the target teacher to obtain a scanned video, converting the scanned video into TIFF format and then extracting facial feature points, reconstructing the head mesh after obtaining the head point cloud and mapping the head mesh according to the facial feature points to obtain the head information of the target teacher, replacing the head information of the existing model with the head information of the target teacher, and obtaining a teaching digital human that is close in appearance to the target teacher.

[0028] In addition, the present invention also provides a teaching-friendly digital human online MOOC construction method, which is characterized by comprising:

[0029] Construct a teaching digital person whose image is similar to that of the target teacher;

[0030] Generate a basic action sequence based on speech and text, obtain an enhanced action sequence based on the text using a fine-tuned large language model, and fuse the basic action sequence and the enhanced action sequence to obtain an action sequence enhanced by the teaching semantics of the teaching digital human. At the same time, coordinate the action sequence enhanced by the teaching semantics with the teaching materials in the teaching space to form intelligent driving actions of the teaching digital human;

[0031] Construct intelligent scenarios for teaching digital humans;

[0032] Driving the teaching digital human in the intelligent scene based on the intelligent driving action of the teaching digital human;

[0033] Based on the text and time information corresponding to the intelligent driving action of the teaching digital human and the angle information corresponding to each intelligent scene, the activation time of each intelligent scene and the time and angle of the change of the digital human's direction are obtained, and intelligent adjustment of the intelligent scene is achieved based on them.

[0034] Preferably, the audio analysis library and the automatic speech recognition model are used to extract the low-dimensional and high-dimensional audio feature information in the speech respectively, the word vector model is used to extract the corresponding word embedding in the text, and the basic action vector z is generated by the decoder of the pre-trained Transformer model. q *;

[0035] Based on the text, an enhanced action index is obtained by using a fine-tuned large language model. Based on the enhanced action index, a matching text action is retrieved from a text action dataset. The matching text action is encoded by a pre-trained vector quantized variational autoencoder to obtain a quantized label action vector z. e , z is achieved through weighted fusion q * and z e The fusion of the semantically enhanced label action vector z e-argu *, the semantically enhanced label action vector z e-argu * The decoder of the pre-trained vector quantized variational autoencoder decodes the action sequence of the teaching digital human with enhanced teaching semantics;

[0036] Different types of collaborative actions are set according to the types of teaching materials, and different methods are adopted based on different types of collaborative actions to achieve the collaboration between the teaching semantics enhanced action sequence and the teaching materials.

[0037] Preferably, the types of teaching materials include 2D plane class, 3D scene class, 3D fixed object class and 3D handheld object class. The coordinated actions corresponding to the 2D plane class, 3D scene class and 3D fixed object class are pointing actions. The coordinated actions corresponding to the 3D handheld object class are presentation actions. The pointing actions use the IK reverse settlement method to achieve the coordination between the action sequence enhanced by the teaching semantics and the teaching materials. The presentation actions use a preset action library to achieve the coordination between the action sequence enhanced by the teaching semantics and the teaching materials.

[0038] Moreover, the present invention also provides a teaching-friendly digital human online MOOC construction device, which is characterized by comprising:

[0039] one or more processors;

[0040] A memory for storing one or more programs;

[0041] When the one or more programs are executed by the one or more processors, the one or more processors implement the teaching-friendly digital human online MOOC construction method as described above.

[0042] Finally, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the teaching-friendly digital human online MOOC construction method as described above are implemented.

[0043] Compared with the prior art, the teaching-friendly digital human online MOOC construction system, method, device and medium of the present invention have one or more of the following beneficial technical effects:

[0044] 1. In the intelligent driving part of the teaching digital human, the present invention takes into account the teaching nature of the action and the spatial attributes of the teaching materials, and further integrates the common teaching accompanying audio actions and the action parts that cooperate with the teaching materials in teaching.

[0045] 2. In the design of the scene, the present invention takes into account the inherent advantages of digital humans as virtual agents in virtual environments, that is, considering switching virtual scenes according to the adaptability of teaching content in real teaching. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the composition of the teaching-friendly digital human online MOOC construction system of the present invention.

[0047] Figure 2 It is the collaborative workflow of the action sequence enhanced by the teaching semantics of the present invention and the teaching materials.

[0048] Figure 3 It is an exemplary intelligent scenario of the present invention.

[0049] Figure 4 It is a flow chart of the teaching-friendly digital human online MOOC construction method of the present invention. DETAILED DESCRIPTION

[0050] Before describing in detail any embodiment of the present invention, it should be understood that the present invention is not limited in its application to the construction and arrangement details of the components described below or illustrated in the following figures. The present invention can have other embodiments and can be practiced or carried out in various ways. In addition, it should be understood that the words and terms used here are for descriptive purposes and should not be considered restrictive. The use of "including" or "having" and its variations herein is intended to cover the items and their equivalents and additional items displayed below. Unless otherwise specified or limited, the terms "install", "connect", "support" and "couple" and their variations are widely used and cover direct installation and indirect installation, connection, support and connection. In addition, "connect" and "couple" are not limited to physical or mechanical connections or connections.

[0051] Furthermore, on the first aspect, in the disclosure of the present invention, the orientation or positional relationship indicated by terms such as "longitudinal", "transverse", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" are based on the orientation or positional relationship shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore the above terms cannot be understood as limitations on the present invention; on the second aspect, the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "one" cannot be understood as a limitation on the quantity.

[0052] The present invention relates to a teaching-friendly digital human online MOOC construction system, method, device and medium. The system uses large language model technology, text-to-speech technology, full-body generation technology of intelligent digital human with speech, 3D point cloud reconstruction technology and UE tools to build an intelligent generation system for teaching digital human. Through effective teaching action design and teaching space scene design, it renders a highly efficient, intelligent and accurate 3D teaching scene, which helps to improve teaching effectiveness while reducing the time cost of teachers in MOOC production.

[0053] Figure 1 The schematic diagram of the teaching-friendly digital human online MOOC construction system of the present invention is shown. Figure 1 As shown, the teaching-friendly digital human online MOOC construction system of the present invention includes:

[0054] 1. Teaching digital human intelligent building blocks.

[0055] The teaching digital human intelligent construction module is used to construct a teaching digital human whose image is close to that of a target teacher.

[0056] In the present invention, when constructing a teaching digital human whose image is close to that of a target teacher, firstly, the target teacher is photographed to obtain a scanned video, and the scanned video is converted into a TIFF format and then facial feature points are extracted; then, after obtaining the head point cloud, the head mesh is reconstructed and a map is applied to the head mesh according to the facial feature points to obtain the head information of the target teacher; finally, the head information of the target teacher is used to replace the head information of the existing model, so as to obtain a teaching digital human whose image is close to that of the target teacher.

[0057] Specifically, the intelligent construction module of the teaching digital human is mainly implemented with the help of UE5's Metahuman. In order to ensure that the image of the teaching digital human and the target teacher are consistent, it is necessary to obtain at least the head information of the target teacher. The present invention obtains a scanned video by shooting the target teacher, then converts it into TIFF format and uses MetaShape to extract facial feature points. After obtaining the head point cloud of the target teacher, the head mesh is reconstructed and material mapping is performed based on the facial feature points to obtain the head information of the target teacher. Then, with the help of the UE Metahuman plug-in, the head information of the existing model is replaced to obtain a teaching digital human whose image is close to the target teacher.

[0058] 2. Teach the digital human intelligent driving action construction module.

[0059] The teaching digital human intelligent driving action construction module is used to generate a basic action sequence based on voice and text, obtain an enhanced action sequence based on the text using a fine-tuned large language model, and fuse the basic action sequence and the enhanced action sequence to obtain an action sequence of the teaching digital human with enhanced teaching semantics. At the same time, the action sequence with enhanced teaching semantics is coordinated with the teaching materials in the teaching space to form the teaching digital human intelligent driving action.

[0060] In the present invention, the teaching digital human intelligent driving action construction module mainly uses the deep learning network to predict and generate action sequences. Therefore, it is necessary to use a data set to pre-train the deep learning network. For this purpose, it is necessary to prepare the data set in advance.

[0061] First, the existing open source BEAT2 dataset with speech, text transcription alignment information, and Mo-Cap motion data was selected. However, the semantics of this dataset is not strong. In order to achieve the goal of teaching effectiveness of actions, it is also necessary to enhance the semantic expression of teaching actions and the design of coordination with teaching materials. Therefore, it is necessary to further construct a text action dataset. Since teaching actions are mainly gestures, only gesture actions are discussed here. According to the research on gestures by relevant scholars, common gesture types include five types: indicating gestures, beat gestures, metaphorical gestures, sign gestures, and connection gestures (Table 1). According to the research and verification of meaningful teaching gestures by relevant scholars, it includes four meaningful teaching gestures: guidance, emphasis, structure, and example (Table 2). Based on these teaching intentions, combined with the teaching behavior of teachers, and analyzing the existing MOOC teaching videos, the present invention designs teaching actions from fingers, palms, arms, whole body, and other five body parts (Table 3). After that, the teaching action files are obtained respectively by optical motion capture and motion capture gloves to construct a text action dataset.

[0062] In the text action dataset, each teaching action is stored in the form of index, main time, action description, action meaning, text example and action file (Table 4). A complete gesture includes three stages: preparation, main body and withdrawal. The main time represents the effective action time. The index consists of two parts: letters + numbers, where letters A to E represent the limbs to which the action belongs (i.e., fingers, palms, arms, whole body and others), and the number part is the word vector (WordEmbedding) obtained by the T5 model according to the corresponding action description, so as to perform hierarchical clustering according to semantics and obtain the number of each position. Actions with similar semantic information have high numerical similarity. For example, both hands cross and waving hands have negative meanings, and their indexes may be B201 and B202.

[0063] Table 1 Gesture classification

[0064] Gesture Type definition Pointing gestures Gesture pointing has the function of indicating objects and events in the concrete world, or abstract objects. Beat gesture The hands move in rhythm with the words. Metaphorical gestures The gesture depicts an abstract concept or a concrete metaphor with visual and motion images. Sign gesture The semantic content and close formal relationship between gesture and speech. Connecting gestures Gestures are used to maintain consistency with previously described content.

[0065] Table 2 Meaningful teaching gestures

[0066]

[0067] Table 3 Teaching action design (part)

[0068]

[0069]

[0070] Table 4 Data format for each teaching action storage

[0071]

[0072] With the above-mentioned existing BEAT2 dataset and the constructed text action dataset, the deep learning network used can be pre-trained. Since the present invention mainly uses the Decoder of the VQ-VAE (vector quantized variational autoencoder) and Transformer model, the Decoder of the VQ-VAE (vector quantized variational autoencoder) and Transformer model is mainly pre-trained.

[0073] First, VQ-VAE is pre-trained. The training data comes from the BEAT2 dataset and the constructed text action dataset. Training requires calculating three losses. The first one is used only to train the encoder and decoder of VQ-VAE. The gradients of the two are consistent and belong to the reconstruction loss. The second one is used to train the codebook of VQ-VAE to make it move towards the encoded Z e (Z e The third item is used to train the VQ-VAE encoder and fix the codebook gradient to ensure the output stability of the encoder. Among them, L represents the loss function, sg[·] represents the gradient stop, x represents the input, z e (x) represents the Encoder vector of input x, z q (x) represents the generated quantized vector, which is passed as input to the VQ-VAE Decoder.

[0074]

[0075] Next, the Transformer model's decoder is trained. The training data comes from the BEAT2 dataset. The data input is audio and text, and the output is the action sequence obtained after passing through the VQ-VAE encoder and codebook pre-trained in the previous step. q (Note that the parameters of the VQ-VAE encoder and decoder need to be frozen at this time to prevent further gradient descent.) For audio, the Librosa and wav2vec pre-trained models are used to extract low-dimensional and high-dimensional audio feature information respectively; for text, the FastText pre-trained model is used to obtain the corresponding word embedding. Based on the low-dimensional and high-dimensional audio feature information and the corresponding word embedding, the Transformer model’s Decoder is used to generate z q *.z q *The pre-trained VQ-VAE Decoder restores the complete action sequence x* The training loss function includes z q * and z q , x * and the reconstruction loss of both.

[0076] In the present invention, the teaching digital human intelligent driving action construction module includes an audio rhythm action generation submodule, a teaching collaborative action enhancement submodule and a teaching collaborative action enhancement submodule.

[0077] The audio rhythm action generation submodule uses an audio analysis library (e.g., Librosa) and an automatic speech recognition model (e.g., a wav2vec pre-trained model) to extract low-dimensional and high-dimensional audio feature information from the speech, respectively, uses a word vector model (e.g., a FastText pre-trained model) to extract the corresponding word vector / word embedding in the text, and generates a basic action vector z through a pre-trained Transformer model Decoder q *.

[0078] The teaching collaborative action enhancement submodule is used to obtain an enhanced action index based on the text using a fine-tuned large language model, retrieve matching text actions from a text action dataset based on the enhanced action index, and encode the matching text actions through the encoder of a pre-trained vector quantized variational autoencoder to obtain a quantized label action vector z e , z is achieved through weighted fusion q * and z e The fusion of the semantically enhanced label action vector z e-argu *, the semantically enhanced label action vector z e-argu *The decoder of the pre-trained vector quantized variational autoencoder decodes the action sequence with enhanced teaching semantics for the teaching digital human.

[0079] Since a fine-tuned large language model is to be used, it is necessary to first construct an instruction dataset for fine-tuning the large language model. In the present invention, according to the constructed text action dataset, 20 10-20 minute video clips are collected from online MOOC videos and TED speech videos, and the video audio is transcribed by ASR (Automatic Speech Recognition). After obtaining the corresponding text, the gesture expression of the video is compared, and the index is manually annotated and added to construct an instruction dataset that can be used to fine-tune the large language model. After using the instruction dataset to fine-tune the large language model, the text is input into the fine-tuned large language model to obtain an enhanced action index. With the action index, matching text actions can be retrieved from the text action dataset, including subject time, action description, action meaning, text examples, etc. Afterwards, it is input into the Decoder (encoder) of the pre-trained VQ-VAE to obtain a quantized label action vector z e .

[0080] In progress q * and z e When fusing, first determine the time range of fusion according to the position of the enhanced action index, and divide the basic action according to the time range to form multiple basic action segments, calculate the change of the movement speed of the basic action, and determine the position with the largest speed change as the fusion splicing point. Then, use the label action vector to replace the corresponding basic action vector at the splicing point, and perform z before and after the replacement point. q * and z e Perform a weighted merge operation.

[0081] Preferably, when performing the weighted merging operation, the time before the replacement point and the time after the replacement point are respectively multiplied by a half cosine curve with a weight (ws) ranging from 0.3 to 0.7 to 0.3. e-argu *The merging weight is set to 1-ws to ensure that the sum is 1. Then the spliced ​​z e-argu *The decoder of the vector quantized variational autoencoder decodes the action sequence with enhanced teaching semantics for the teaching digital human.

[0082] The teaching collaborative action enhancement submodule is used to set different types of collaborative actions according to the type of teaching materials, and adopt different methods based on different types of collaborative actions to achieve the coordination between the teaching semantically enhanced action sequence and the teaching materials.

[0083] According to the spatial properties of teaching materials, they are divided into 2D teaching materials and 3D teaching materials. The present invention emphasizes the automatic generation of collaborative actions. However, there are various forms of interaction between teachers and teaching materials, and strong interaction (grasping, playing, etc.) is not considered here. Starting from the emphasis intention in the teaching scene, the present invention focuses on the use of gestures of pointing and presentation.

[0084] The present invention does not consider the automation of the time when collaborative gestures occur. Therefore, the teacher needs to manually provide the time when collaborative gestures are performed. In the stage of preparing teaching materials and teaching scripts, the teacher indexes the key trigger words of the teaching scripts and the corresponding teaching materials and annotates the spatial attribute information. Through TTS (Text-to-Speech) and MFA (Montreal Forced Alignment), the timestamps corresponding to the keywords can be restored to obtain the time information when the collaborative action occurs. And through the scene index and spatial attribute information of the teaching materials, IK (Inverse Kinematics) is used according to the spatial position information of the target to reversely solve the action animation required for the bound skeleton to perform pointing actions within the corresponding time range.

[0085] Among them, 2D teaching materials mainly include PPT or video information presented on a 2D screen. Teaching collaborative gestures are expressed as pointing to positions in the 2D plane, guiding students to focus on the emphasized area. 3D teaching materials include the direction of the scene they belong to, the relationship with the digital human, etc. according to their spatial attributes. According to the relationship, teaching materials can be divided into the following three categories: 1. Scene type: long-distance scene teaching materials, digital humans are often placed in them, generally appearing in groups or scales, including buildings, venues, historical monuments, etc.; 2. Fixed object type: teaching materials of single 3D elements at a medium distance, generally existing side by side with digital humans in space, with a fixed spatial position; 3. Handheld object type: teaching materials of single 3D elements at a close distance, which can be held and displayed by digital humans. This category of teaching materials does not specifically belong to a certain scene and does not exist fixedly in space.

[0086] The teaching gestures are designed differently according to their different relationships with spatial objects.

[0087] Teaching materials for scenes and fixed objects are far away from the digital human and are usually fixed in the scene. When uploading, you need to first obtain the scene index, distance information, and the location information of the emphasized text and emphasized area. This part only uses the directional emphasized actions obtained by IK reverse settlement.

[0088] For hand-held objects, since their appearance is strongly related to the text content and they do not have a fixed position, and without considering complex interactions with such objects (including holding or gesturing spatial size attributes, etc.), only the automatic generation of presentation actions for such objects is achieved. In the data preparation stage, based on the previous text action data set, a standardized presentation action library is selected to pre-acquire the position information of the presentation object relative to the teaching digital human where it is expected to appear. When uploading such teaching materials, it is necessary to mark the keyword text corresponding to the appearance of the object and select the presentation action. Then complete the action splicing within the time segment corresponding to the keyword. In addition to synthesizing the presentation action of the digital human, this part will also synchronously generate animations of the appearance and disappearance of objects.

[0089] Figure 2 The specific workflow is shown. The uploading and annotation are completed by building a front-end and back-end platform with Django+Vue.js+3D.js. The key annotation information (text, keyword text, textbook index, textbook attribute, emphasized position coordinates / presentation action index) is obtained by uploading and transmitted to the back-end in the form of JSON. The blender python api is used to call blender to set the constraint information of the matching bone nodes. The absolute spatial coordinates of the emphasized position of the textbook in the space are calculated according to the preset spatial layout. The time range t (t1, t2) is obtained by using the TTS+MFA method through the keyword text. The IK tool in blender is called to solve the directional gesture action animation, and the Kalman filter smoothing algorithm is used to realize the splicing of the semantically enhanced action sequence output by the algorithm and the output directional gesture action sequence, so as to obtain the final limb action sequence.

[0090] Table 5 Quantitative information and collaborative gestures

[0091]

[0092] 3. Teaching digital human intelligent scene construction module.

[0093] The teaching digital human intelligent scene construction module is used to construct the teaching digital human intelligent scene.

[0094] The teaching materials that can be placed in teaching mainly include 2D and 3D teaching materials. Different from the fixed classroom scene mode of the real classroom, the teaching scene in the virtual world hopes to be diversified and changeable, but at the same time, it is necessary to balance the differences in the specifications of the teaching materials themselves (including dimensions, scene size, etc.). The teaching scene designed by the present invention aims to enable the digital human to complete the switching of the teaching environment displayed within the visual range with the help of the lens and other means under the condition of the minimum number of moving steps. Therefore, the present invention designs a layout of the teaching scene centered on the digital human and surrounded by scenes (such as Figure 3The intelligent scene includes a 2D screen, which is mainly used to display 2D ppt and videos in teaching materials, and other 3D scene areas (animations of large objects at a long distance / medium objects at a medium distance, and small objects at a close distance are directly added to the entire scene from the previous step). Each 3D scene corresponds to a separate camera and lighting.

[0095] 4. Teaching digital human intelligent driving module.

[0096] The teaching digital human intelligent driving module is used to drive the teaching digital human in the intelligent scene based on the teaching digital human intelligent driving action.

[0097] In the present invention, by establishing a port link through the UE, the intelligent driving action of the teaching digital human is bound to the corresponding channel of the teaching digital human, and the teaching digital human can be driven by the intelligent driving action of the teaching digital human. The driven teaching digital human, that is, the teaching digital human animation, can be imported into the intelligent scene.

[0098] 5. Intelligent scene intelligent adjustment module.

[0099] The intelligent adjustment module of the intelligent scene is used to obtain the activation time of each intelligent scene and the time and angle of the change of the digital human's direction based on the text and time information corresponding to the intelligent driving action of the teaching digital human and the angle information corresponding to each intelligent scene, and realize the intelligent adjustment of the intelligent scene based on it.

[0100] Specifically, after the teaching digital human animation is imported into the intelligent scene, the scene activation time (lens activation, lighting activation) and the time and angle of change of the digital human's orientation are obtained according to the text and time information corresponding to the original collaborative action and the fixed angle information corresponding to each intelligent scene, and are passed to the lens and lighting control and the rotation angle of the digital human respectively, and key frame information is added to realize the intelligent adjustment of the teaching scene.

[0101] like Figure 4 As shown, when using the teaching-friendly digital human online MOOC construction system of the present invention to construct a teaching-friendly digital human online MOOC, first, the teaching digital human intelligent construction module is used to construct a teaching digital human whose image is close to that of the target teacher.

[0102] Next, the teaching digital human intelligent driving action construction module is used to generate a basic action sequence based on speech and text, and an enhanced action sequence is obtained based on the text using a fine-tuned large language model, and the basic action sequence and the enhanced action sequence are fused to obtain an action sequence enhanced with teaching semantics for the teaching digital human. At the same time, the action sequence enhanced with teaching semantics is coordinated with the teaching materials in the teaching space to form the teaching digital human intelligent driving action;

[0103] Then, the teaching digital human intelligent scene construction module is used to construct the teaching digital human intelligent scene.

[0104] Afterwards, the teaching digital human intelligent driving module is used to drive the teaching digital human in the intelligent scene based on the teaching digital human intelligent driving action.

[0105] Finally, the intelligent adjustment module of the intelligent scene is used to obtain the activation time of each intelligent scene and the time and angle of the digital human's direction change based on the text and time information corresponding to the intelligent driving action of the teaching digital human and the angle information corresponding to each intelligent scene, and the intelligent adjustment of the intelligent scene is realized based on it.

[0106] The present invention also relates to a teaching-friendly digital human online MOOC construction device, which includes: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the teaching-friendly digital human online MOOC construction method as described above.

[0107] Moreover, the present invention also relates to a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the teaching-friendly digital human online MOOC construction method as described above are implemented.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Those skilled in the art can modify or replace the technical solution of the present invention according to the idea of ​​the present invention without departing from the essence and scope of the technical solution of the present invention.

Claims

1. A teaching-friendly digital human online MOOC construction system, characterized by: include: The intelligent construction module of teaching digital human is used to construct a teaching digital human whose image is close to that of the target teacher; A teaching digital human intelligent driving action construction module, which is used to generate a basic action sequence based on voice and text, obtain an enhanced action sequence based on the text using a fine-tuned large language model, and fuse the basic action sequence and the enhanced action sequence to obtain an action sequence enhanced with teaching semantics for the teaching digital human. At the same time, the action sequence enhanced with teaching semantics is coordinated with the teaching materials in the teaching space to form an intelligent driving action for the teaching digital human; A teaching digital human intelligent scene construction module, which is used to construct a teaching digital human intelligent scene; A teaching digital human intelligent driving module, which is used to drive the teaching digital human in the intelligent scene based on the teaching digital human intelligent driving action; The intelligent adjustment module of the intelligent scene is used to obtain the activation time of each intelligent scene and the time and angle of the digital human's direction change based on the text and time information corresponding to the intelligent driving action of the teaching digital human and the angle information corresponding to each intelligent scene, and realize the intelligent adjustment of the intelligent scene based on it.

2. The teaching-friendly digital human online MOOC construction system according to claim 1 is characterized in that: The teaching digital human intelligent driving action building module includes: The audio rhythm action generation submodule is used to use the audio analysis library and the automatic speech recognition model to extract the low-dimensional and high-dimensional audio feature information in the speech, use the word vector model to extract the corresponding word embedding in the text, and generate the basic action vector z through the decoder of the pre-trained Transformer model. q *; The teaching collaborative action enhancement submodule is used to obtain an enhanced action index based on the text using a fine-tuned large language model, retrieve matching text actions from a text action dataset based on the enhanced action index, and encode the matching text actions through an encoder of a pre-trained vector quantized variational autoencoder to obtain a quantized label action vector z e , z is achieved through weighted fusion q * and z e The fusion of the semantically enhanced label action vector z e-argu *, the semantically enhanced label action vector z e-argu * The decoder of the pre-trained vector quantized variational autoencoder decodes the action sequence of the teaching digital human with enhanced teaching semantics; The teaching collaborative action enhancement submodule is used to set different types of collaborative actions according to the types of teaching materials, and adopt different methods based on different types of collaborative actions to achieve the coordination between the teaching semantically enhanced action sequence and the teaching materials.

3. The teaching-friendly digital human online MOOC construction system according to claim 2 is characterized in that: Achieve z through weighted fusion q * and z e The fusion of the semantically enhanced label action vector z e-argu *Specifically include: Determine the time range of fusion according to the position of the enhanced action index, divide the basic action according to the time range to form multiple basic action segments, calculate the change of the movement speed of the basic action, and determine the position with the largest speed change as the fusion splicing point; Use the label action vector to replace the corresponding basic action vector at the splicing point, and compare z before and after the replacement point. q * and z e Perform a weighted merge operation.

4. The teaching-friendly digital human online MOOC construction system according to claim 3 is characterized in that: The types of teaching materials include 2D plane class, 3D scene class, 3D fixed object class and 3D handheld object class. The coordinated actions corresponding to the 2D plane class, 3D scene class and 3D fixed object class are pointing actions, and the coordinated actions corresponding to the 3D handheld object class are presentation actions. The pointing actions use the IK reverse settlement method to achieve the coordination between the action sequence enhanced by the teaching semantics and the teaching materials, and the presentation actions use a preset action library to achieve the coordination between the action sequence enhanced by the teaching semantics and the teaching materials.

5. The teaching-friendly digital human online MOOC construction system according to any one of claims 1 to 4, characterized in that: The specific steps of constructing a teaching digital human that is similar to the target teacher in appearance are as follows: shooting the target teacher to obtain a scanned video, converting the scanned video into TIFF format and then extracting facial feature points, reconstructing the head mesh after obtaining the head point cloud and mapping the head mesh according to the facial feature points to obtain the head information of the target teacher, replacing the head information of the existing model with the head information of the target teacher, and obtaining a teaching digital human that is similar to the target teacher in appearance.

6. A teaching-friendly digital human online MOOC construction method, characterized in that: include: Construct a teaching digital person whose image is similar to that of the target teacher; Generate a basic action sequence based on speech and text, obtain an enhanced action sequence based on the text using a fine-tuned large language model, and fuse the basic action sequence and the enhanced action sequence to obtain an action sequence enhanced by the teaching semantics of the teaching digital human. At the same time, coordinate the action sequence enhanced by the teaching semantics with the teaching materials in the teaching space to form intelligent driving actions of the teaching digital human; Construct intelligent scenarios for teaching digital humans; Driving the teaching digital human in the intelligent scene based on the intelligent driving action of the teaching digital human; Based on the text and time information corresponding to the intelligent driving action of the teaching digital human and the angle information corresponding to each intelligent scene, the activation time of each intelligent scene and the time and angle of the change of the digital human's direction are obtained, and intelligent adjustment of the intelligent scene is achieved based on them.

7. The teaching-friendly digital human online MOOC construction method according to claim 6 is characterized in that: The audio analysis library and the automatic speech recognition model are used to extract the low-dimensional and high-dimensional audio feature information in the speech respectively, the word vector model is used to extract the corresponding word embedding in the text, and the basic action vector z is generated through the decoder of the pre-trained Transformer model q *; Based on the text, an enhanced action index is obtained by using a fine-tuned large language model. Based on the enhanced action index, a matching text action is retrieved from a text action dataset. The matching text action is encoded by a pre-trained vector quantized variational autoencoder to obtain a quantized label action vector z. e , z is achieved through weighted fusion q * and z e The fusion of the semantically enhanced label action vector z e-argu *, the semantically enhanced label action vector z e-argu * The decoder of the pre-trained vector quantized variational autoencoder decodes the action sequence of the teaching digital human with enhanced teaching semantics; Different types of collaborative actions are set according to the types of teaching materials, and different methods are adopted based on different types of collaborative actions to achieve the collaboration between the teaching semantics enhanced action sequence and the teaching materials.

8. The teaching-friendly digital human online MOOC construction method according to claim 7 is characterized in that: The types of teaching materials include 2D plane class, 3D scene class, 3D fixed object class and 3D handheld object class. The coordinated actions corresponding to the 2D plane class, 3D scene class and 3D fixed object class are pointing actions, and the coordinated actions corresponding to the 3D handheld object class are presentation actions. The pointing actions use the IK reverse settlement method to achieve the coordination between the action sequence enhanced by the teaching semantics and the teaching materials, and the presentation actions use a preset action library to achieve the coordination between the action sequence enhanced by the teaching semantics and the teaching materials.

9. A teaching-friendly digital human online MOOC construction device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the teaching-friendly digital human online MOOC construction method as described in any one of claims 6 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the teaching-friendly digital human online MOOC construction method as described in any one of claims 6 to 8 are implemented.

Images