Interactive virtual digital teacher system and working method thereof

Abstract

The invention provides an interactive virtual digital teacher system and a working method thereof in order to solve the problem that real-time interaction with students cannot be carried out in traditional digital human teaching. In the digital human teaching process, the real-time question information of the user can be received, and the corresponding question and answer result text is generated; then, the audio is converted into teacher answer audio, and a corresponding question answer digital human video stream is generated based on the teacher answer audio; then, the question answering digital person video stream is pushed to the client side to be played, and real-time answering of the user question can be achieved; and finally, after the question is answered, the pushing of the course teaching audio is recovered, and then the playing of the course teaching video is recovered. Thus, the interactive virtual classroom function is realized, the students can communicate with the digital person teacher in real time during the class, the students can timely solve the problem, and the enthusiasm and the participation degree of the students are improved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer application, and in particular relates to an interactive virtual digital teacher system and a working method thereof. Background Art

[0002] With the release of OpenAI's new generation of intelligent product ChatGPT in 2022, it marks that human society has entered the era of the fourth industrial revolution represented by artificial intelligence (AI) technology; the deep integration of AI technology with various industries will become an important direction of technological innovation in the future; taking the digital human industry as an example, with the emergence of large models, digital humans can combine speech recognition, speech synthesis, speech cloning, large models and other technologies to realize functions such as digital human tour guides, digital human virtual teachers, and digital human assistants, thereby bringing a more humanized and intelligent interactive experience.

[0003] At present, in the field of education and teaching, various remote video education websites have played a great role in education and teaching. Users or students can learn knowledge by logging into the website and clicking on videos without leaving home, which has achieved a certain degree of ubiquitous education and personalized education; however, traditional digital human teaching videos have the following shortcomings: in the process of teaching and explanation, teachers and students cannot interact in real time, which not only makes it impossible for students to get timely answers to questions in class, but also reduces students' enthusiasm and participation. If questions and answers are to be conducted, only digital human assistants can be used. In this way, video teaching and lectures cannot be conducted, that is, the existing digital human teaching can only realize the teaching and lecture function, or can only realize the question and answer function. Therefore, how to provide a virtual digital teacher system that can interact in real time during video teaching and lectures has become an urgent problem to be solved. Summary of the invention

[0004] The purpose of the present invention is to provide an interactive virtual digital teacher system and a working method thereof, so as to solve the problem in the prior art that the digital human cannot interact with students in real time during teaching.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, an interactive virtual digital teacher system is provided, comprising: A question-answering interaction module, wherein the question-answering interaction module is used to obtain real-time question information in the course teaching process based on the virtual teacher, and based on the real-time question information, generate a question-answering result text and send it to the voice cloning module; A voice cloning module, wherein the voice cloning module is used to convert the question and answer result text into the teacher's answer audio, and send the teacher's answer audio to the voice management module; A voice management module, wherein a voice push queue is stored in the voice management module, wherein the voice push queue includes a plurality of course teaching audios, and the voice management module is used to interrupt the push of the course teaching audios in the voice push queue after receiving the teacher's answer audio, and transmit the teacher's answer audio to the digital virtual teacher cloning module; A digital virtual teacher cloning module is used to generate a digital human video stream of question answers based on the teacher's answer audio, and push the digital human video stream of question answers to the client to realize replying to the user's questions during the course teaching process; The voice management module is also used to resume the push of the course teaching audio in the voice push queue after transmitting the teacher's answer audio to the digital virtual teacher cloning module, so that the digital virtual teacher cloning module converts the course teaching audio pushed by the voice management module into a digital human teaching video stream, and pushes it to the client to resume the playback of the course teaching video based on the virtual teacher.

[0006] Based on the above disclosed content, the present invention is provided with a question and answer interaction module, which is used to obtain real-time question information in the course teaching process and generate a question and answer result text corresponding to the real-time question information; then, the question and answer result text is sent to the voice cloning module, and the voice cloning module can generate the teacher's answer audio based on the question and answer result text and send it to the voice management module; then, after receiving the teacher's answer audio, the voice management module will interrupt the push of the course teaching audio in the voice push queue, and transmit the teacher's answer audio to the digital virtual teacher cloning module; then, the digital virtual teacher cloning module can generate a question answering digital human video stream based on the teacher's answer audio and push it to the client to achieve real-time answers to user questions; finally, after sending the teacher's answer audio to the digital virtual teacher cloning module, the voice management module will resume the push of the course teaching audio in the voice push queue to resume the playback of the current course teaching video.

[0007] Through the above design, the interactive virtual digital teacher system provided by the present invention can receive the user's real-time question information during the digital human teaching process and generate a corresponding question and answer result text; then, convert it into the teacher's answer audio to generate the corresponding question answering digital human video stream based on the teacher's answer audio; then, push the question answering digital human video stream to the client for playback, so as to achieve real-time answers to user questions; finally, after completing the question answering, the present invention resumes the push of the course teaching audio, and then resumes the playback of the course teaching video; in this way, the present invention realizes the function of an interactive virtual classroom, so that students can communicate with the digital human teacher in real time during class, thereby getting rid of the shortcomings of the traditional digital teacher system that directly generates videos but cannot interact, not only can students' questions be answered in a timely manner, but also students' enthusiasm and participation are improved, and therefore, it is particularly suitable for large-scale application and promotion on education platforms.

[0008] In a possible design, if the real-time question information is text information, the question-answering interaction module is used to input the real-time question information into a first question-answering model to obtain the question-answering result text, wherein the first question-answering model is trained by taking a number of historical question text information as input and the question-answering result corresponding to each historical question text information as output, and the first question-answering model includes a text large language model; If the real-time question information is voice information, the question-answering interaction module is used to convert the real-time question information into text to obtain a question text, and input the question text into the first question-answering model to obtain a question-answering result text; or Inputting the real-time question information into a pre-trained second question-answering model to obtain the question-answering result text, wherein the second question-answering model includes a multimodal large language model; If the real-time question information is picture and text information, the question and answer interaction module is used to input the real-time question information into the second question and answer model to obtain the question and answer result text.

[0009] In a possible design, each course teaching audio in the voice push queue corresponds to a teaching segmentation text, wherein the teaching segmentation texts corresponding to all course teaching audios constitute the teaching text of the current digital human course teaching, and the sorting order of each course teaching audio in the voice push queue is the segmentation order of each teaching segmentation text; The voice management module is used to transmit the course teaching audios in the voice push queue to the digital virtual teacher cloning module in sequence upon receiving the course start instruction; The digital virtual teacher cloning module is used to convert the received course teaching audio into digital human teaching video streams in the order of reception, and push them to the client in sequence to realize the playback of course teaching videos based on digital virtual teachers.

[0010] In one possible design, this also includes: A teaching text generation module, wherein the teaching text generation module is used to obtain teaching prompt words, and input the teaching prompt words into a teaching text generation large language model to obtain a teaching text and send it to the voice cloning module; The voice cloning module is also used to segment the teaching text to obtain a plurality of teaching segmentation texts, and generate a plurality of course teaching audios based on each teaching segmentation text and send them to the voice management module, wherein each teaching segmentation text corresponds to a course teaching audio; The voice management module is used to receive a plurality of course teaching audios sent by the voice cloning module, and sort the course teaching audios according to the segmentation order of the teaching segmentation texts corresponding to the course teaching audios to obtain the voice push queue.

[0011] In a possible design, the teaching text generation module is also used to obtain the teaching content table and extract the teaching keywords in the teaching content table in response to the human-computer interaction operation, so as to generate the teaching prompt words using the extracted teaching keywords.

[0012] In a possible design, a voice cloning model is stored in the voice cloning module, wherein the voice cloning model is trained by taking a text set corresponding to voice information of several designated persons as input and a cloned voice corresponding to each text in each text set as output, the text set corresponding to any designated person includes texts of each audio segment of the voice information corresponding to the designated person, the designated persons include multiple designated teachers, and the timbre of the cloned voice corresponding to each text in any text set is the same as the timbre of the designated person corresponding to the text set; Among them, the voice cloning module is used to input the question and answer result text into the voice cloning model to obtain a cloned voice corresponding to the question and answer result text and with the same timbre as any designated person, and use the cloned voice corresponding to the question and answer result text output by the voice cloning model as the teacher's answer audio.

[0013] In a possible design, the digital virtual teacher cloning module stores a teacher image cloning model, wherein the teacher image cloning model is trained by taking image data sets of several designated persons as input and digital human video streams corresponding to each designated person as output, the image data set of any designated person includes the image features and audio features of the designated person, and the image features of any designated person are obtained based on the video or picture of the designated person; The digital virtual teacher cloning module is used to input the teacher's answer audio into the teacher image cloning model to obtain the question answering digital human video stream.

[0014] In a possible design, the digital virtual teacher cloning module is also used to push the question-answering digital human video stream and the digital human teaching video stream to the streaming media server, wherein the question-answering digital human video stream has a higher push priority than the digital human teaching video stream; The streaming media server is used to push the question-answering digital human video stream or the digital human teaching video stream to the client upon receiving a video pull request from the client.

[0015] In a possible design, the streaming media server is also used to preload the digital human teaching video stream before receiving the video pull request, and push the preloaded digital human teaching video stream to the client after receiving the video pull request.

[0016] In a second aspect, a working method of the interactive virtual digital teacher system according to the first aspect or any possible design of the first aspect is provided, wherein the working method comprises: The question-answering interaction module obtains real-time question information during the course teaching process based on the virtual teacher, and based on the real-time question information, generates a question-answering result text and sends it to the voice cloning module; The voice cloning module converts the question and answer result text into the teacher's answer audio, and sends the teacher's answer audio to the voice management module; After receiving the teacher's answer audio, the voice management module interrupts the push of the course teaching audio in the voice push queue, and transmits the teacher's answer audio to the digital virtual teacher cloning module; The digital virtual teacher cloning module generates a digital human video stream of the question answering based on the teacher's answering audio, and pushes the digital human video stream of the question answering to the client to realize the reply to the user's question in the course teaching process; After transmitting the teacher's answer audio to the digital virtual teacher cloning module, the voice management module resumes the push of the course teaching audio in the voice push queue, so that the digital virtual teacher cloning module converts the course teaching audio pushed by the voice management module into a digital human teaching video stream, and pushes it to the client to resume the playback of the course teaching video based on the virtual teacher.

[0017] In a third aspect, an electronic device is provided, comprising a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the working method of the interactive virtual digital teacher system as in the second aspect or any possible design of the second aspect.

[0018] In a fourth aspect, a storage medium is provided, on which instructions are stored. When the instructions are run on a computer, a working method of the interactive virtual digital teacher system as in the second aspect or any possible design of the second aspect is executed.

[0019] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, causes the computer to execute a working method of the interactive virtual digital teacher system as in the second aspect or any possible design of the second aspect.

[0020] Beneficial effects: (1) The interactive virtual digital teacher system provided by the present invention can receive the user's real-time question information during the digital human teaching process and generate the corresponding question and answer result text; then, it converts the information into the teacher's answer audio, and generates the corresponding question answering digital human video stream based on the teacher's answer audio; then, the question answering digital human video stream is pushed to the client for playback, so as to achieve real-time answering of the user's question; finally, after the question is answered, the present invention resumes the push of the course teaching audio, and then resumes the playback of the course teaching video; in this way, the present invention realizes the function of an interactive virtual classroom, so that students can communicate with the digital human teacher in real time during class, thereby getting rid of the disadvantage that the traditional digital teacher system directly generates videos but cannot interact, not only can the students' questions be answered in a timely manner, but also the students' enthusiasm and participation are improved. Therefore, it is particularly suitable for large-scale application and promotion on the education platform.

[0021] (2) The present invention pre-segments and voice clones the teaching text to obtain a number of course teaching audios; then, the number of course teaching audios are transmitted to the voice management module, and the voice management module receives the aforementioned course teaching audios and forms a voice push queue according to the segmentation order; finally, after the course starts, the number of course teaching audios in the voice push queue are pushed to the digital virtual teacher cloning module in sequence, thereby streaming the digital virtual teacher in real time to generate a corresponding digital human teaching video stream; in this way, the aforementioned method can reduce the amount of data processing during the course, thereby reducing the system processing pressure, and further improving the fluency of the teaching video.

[0022] (3) After generating the digital human teaching video stream, the present invention pushes it to the streaming media server, and adopts the streaming media server push-pull stream method (i.e., video transmission is performed after obtaining the client's pull request) to push the video to the client; in this way, the system's high concurrency capability and fluency can be improved.

[0023] (4) The present invention adopts a pre-loading method for video push, which can further improve the fluency of the teaching video.

[0024] (5) The present invention uses the video and audio, or picture and audio, of a designated teacher to train the teacher image cloning model. Thus, in actual use, only the corresponding text of the teaching needs to be input to generate a digital human teaching video stream. Based on this, compared with the traditional digital human teaching using 3D modeling, the present invention not only reduces the use cost, but also improves the data processing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of the architecture of an interactive virtual digital teacher system provided by an embodiment of the present invention; Figure 2 A schematic diagram of training a voice cloning model provided by an embodiment of the present invention; Figure 3 A schematic diagram of data processing of a voice cloning module provided in an embodiment of the present invention; Figure 4 A schematic diagram of training a teacher image cloning model provided by an embodiment of the present invention; Figure 5 A flowchart of the steps of the working method of the interactive virtual digital teacher system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0027] It should be understood that although the terms first, second, etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another unit. For example, a first unit can be referred to as a second unit, and similarly, a second unit can be referred to as a first unit without departing from the scope of the exemplary embodiments of the present invention.

[0028] It should be understood that the term "and / or" that may appear in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" that may appear in this article describes another type of association object relationship, indicating that two relationships may exist. For example, A / and B can represent two situations: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this article generally indicates that the previous and next associated objects are in an "or" relationship.

[0029] Example: See also Figure 1 As shown, the interactive virtual digital teacher system provided in this embodiment is intended to realize the function of an interactive virtual classroom, that is, to enable students to communicate with the digital human teacher in real time during class, thereby getting rid of the shortcomings of the traditional digital teacher system that directly generates videos but cannot interact, and then enabling students' questions to be answered in a timely manner, so as to improve students' enthusiasm and participation; wherein, the detailed system architecture of the interactive virtual digital teacher system is disclosed below.

[0030] In specific implementation, for example, the aforementioned interactive virtual digital teacher system may include, but is not limited to: a question-and-answer interaction module, a voice cloning module, a voice management module, and a digital virtual teacher cloning module; wherein the question-and-answer interaction module is used to obtain real-time question information in the course teaching process based on the virtual teacher, and based on the real-time question information, generate a question-and-answer result text and send it to the voice cloning module; in this embodiment, the question-and-answer interaction module is used to interact with the user in the course teaching process, that is, to collect the user's questions in real time, which can be in the form of text, voice, or text and pictures, and after collecting the real-time question information, the question-and-answer interaction module can provide targeted answers to the questions and generate corresponding question-and-answer result texts; of course, the generation process of the question-and-answer result text is described in detail below.

[0031] After generating the question and answer result text of the user's real-time question information, the voice cloning module can be used to convert the question and answer result text into audio information of a designated person, thereby realizing question answering based on a virtual teacher, that is: the voice cloning module is used to convert the question and answer result text into the teacher's answer audio, and send the teacher's answer audio to the voice management module; then, based on the voice management module, switching between question answering and course teaching can be performed.

[0032] Specifically, in this embodiment, the priority of question answering is higher than that of course teaching, that is, the voice management module is responsible for reading, managing and pushing the course teaching audio and the teacher's answer audio, that is, realizing audio management, which can realize the sorting, insertion and pushing functions of the course teaching audio; optionally, a voice push queue is stored in the voice management module, wherein the voice push queue includes a number of course teaching audios, each course teaching audio corresponds to a teaching segmentation text, and the teaching segmentation texts corresponding to all course teaching audios constitute the teaching text of the current digital human course teaching, and the sorting order of each course teaching audio in the voice push queue is the segmentation order of each teaching segmentation text; thus, by pushing the aforementioned course teaching audios in sequence, the audio playback of the entire course teaching can be realized.

[0033] However, as mentioned above, answering questions has a higher priority than course teaching. Therefore, when the voice management module receives the teacher's answer audio, it will interrupt the push of the course teaching audio in the voice push queue; then, it will give priority to transmitting the teacher's answer audio to the digital virtual teacher cloning module to achieve real-time answering of questions.

[0034] Furthermore, for example, the priority transmission of the teacher's answer audio can be but is not limited to the insertion method, that is: the teacher's answer audio is inserted into the first position of the voice push queue, and then the push of the course teaching audio is interrupted, and the audio information of the first position in the queue is transmitted first, so that the priority push of the teacher's answer audio can be achieved; of course, data transmission interfaces with different priorities can also be set, that is, the data transmission interface of the question-and-answer interaction module can be divided according to the type of transmitted data, that is, the course teaching audio corresponds to one data transmission interface (such as interface A), and the teacher's answer audio corresponds to another data transmission interface (such as interface B); then, the priority of the teacher's answer audio corresponding to the data transmission interface is set to the highest priority; thus, when the voice management module receives the data transmitted by interface B, it will immediately push the data transmitted by interface B; in this embodiment, the above example is only for illustration, and the specific priority push method can be set according to actual use, and is not limited to this.

[0035] After completing the priority push of the teacher's answer audio, the digital virtual teacher cloning module is used to generate a question-answering digital human video stream based on the teacher's answer audio; then, the question-answering digital human video stream is pushed to the client for playback, so that real-time responses to users' questions during the course teaching process can be achieved.

[0036] Finally, the voice management module, after transmitting the teacher's answer audio to the digital virtual teacher cloning module, resumes the push of the course teaching audio in the voice push queue, so that the digital virtual teacher cloning module converts the course teaching audio pushed by the voice management module into a digital human teaching video stream, and pushes it to the client to resume the playback of the course teaching video based on the virtual teacher; for example, after receiving the teacher's answer audio, the voice management module is transmitting the course teaching audio ranked first in the voice push queue. At this time, the push of the course teaching audio ranked first is interrupted, and then the teacher's answer audio is pushed first. At this time, the digital virtual teacher cloning module can first generate a digital human video stream of question answering, and push it to the client for playback, so as to achieve real-time answering of questions; then, after completing the push of the aforementioned teacher's answer audio, the voice management module continues to transmit the course teaching audio ranked first; at this time, the digital virtual teacher cloning module continues to drive the digital virtual teacher in real time, and then generates a digital human teaching video stream to resume course teaching.

[0037] Through the above-mentioned design, the digital virtual teacher system provided in this embodiment is different from the traditional digital human system. The system is based on the real-time drive of the digital virtual teacher, and during the digital virtual teacher's lecture, it can answer students' questions in real time in a dynamic and interactive manner, so that students' questions can be answered in time, and at the same time, students' enthusiasm and participation can be improved. Therefore, a new solution is provided for the real-time interactive explanation digital human system.

[0038] In a possible design, the present embodiment provides the specific working process of the aforementioned question-answer interaction module as follows: Among them, the question-answering interaction module uses different methods to generate question-answering result text according to the data type of real-time question information.

[0039] In this embodiment, if the real-time question information is text information, the question-and-answer interaction module is used to input the real-time question information into the first question-and-answer model to obtain the question-and-answer result text.

[0040] Optionally, the aforementioned first question-and-answer model is trained by taking a number of historical question text information as input (i.e., different question information collected during the history teaching process) and outputting question-and-answer results corresponding to each historical question text information; wherein, for example, the first question-and-answer model includes a text large language model, and in this embodiment, the text large language model can adopt an open source large model, such as the ChatGLM series large model, the LLaMA series large model or the Qwen series large model, etc., and these open source large models can also be incrementally pre-trained, fine-tuned, retrieval enhanced, etc., so as to obtain an optimized model, and then, the optimized model is used as the first question-and-answer model.

[0041] Similarly, the aforementioned text large language model can also adopt a closed source large model, such as ChatGPT, Wenxin Yiyan, Zhipu Qingyan, etc.; of course, the aforementioned closed source large model can also be incrementally pre-trained, fine-tuned, and search enhanced, so as to further optimize the model capabilities and obtain a higher-performance closed source large model as the first question-answering model.

[0042] Meanwhile, in this embodiment, if the real-time question information is voice information, the question-answering interaction module may generate a corresponding question-answering result text in any one of the following two ways.

[0043] Among them, the first method is: the question and answer interaction module is used to convert the real-time question information into text to obtain a question text; then, the question and answer interaction module inputs the question text into a first question and answer model to obtain a question and answer result text; in this embodiment, for example, converting voice information into text can be implemented but not limited to using a speech recognition model, such as using Whisper, FunASR, Kaldi and other speech recognition models, or directly using closed-source Tencent Cloud speech recognition, iFlytek speech recognition models, etc.; of course, the specific speech recognition model can be specifically set according to actual use, and is not limited to the aforementioned examples, and the process of generating question and answer results based on the question text has been explained in the aforementioned description and will not be repeated here.

[0044] The second method is: the question and answer interaction module directly inputs the real-time question information into the pre-trained second question and answer model to obtain the question and answer result text; in this embodiment, the second question and answer model is taken as an example to include a multimodal large language model; wherein, the multimodal large language model can also adopt an open source large language model or a closed source large language model, and of course, the model can also be incrementally pre-trained, fine-tuned, retrieval enhanced and other processes to further improve the model performance.

[0045] Furthermore, for example, open source multimodal large language models may include but are not limited to NExT-GPT, MoE-LlaVA, InternLM-XComposer2 or GLM-4-Voice models; and closed source multimodal large language models may include but are not limited to GPT-4, GPT-4o, Wenxin Yiyan, Zhipu Qingyan, etc.; of course, the specific type of model can be selected according to actual use, and is not limited to the aforementioned examples.

[0046] Therefore, through the above detailed explanation of the question-answer interaction module, the question-answer result text corresponding to the real-time question information can be generated based on the text large language model or the multimodal large language model.

[0047] After completing the description of the working process of the question-answer interaction module, the specific working process of the public voice cloning module is as follows: In a specific application, for example, a voice cloning module stores a voice cloning model, wherein the voice cloning model is trained by taking a text set corresponding to voice information of several designated persons as input and a cloned voice corresponding to each text in each text set as output, the text set corresponding to any designated person includes the text of each audio segment of the voice information corresponding to the designated person, the designated persons include multiple designated teachers, and the timbre of the cloned voice corresponding to each text in any text set is the same as the timbre of the designated person corresponding to the text set.

[0048] In the specific implementation, see Figure 2 As shown below, the detailed training process of the public voice cloning model is as follows: (1) Acquire the voice information of several designated persons, and separate the human voice and background sound of each voice information to obtain each processed voice information.

[0049] (2) Performing speech segmentation on each processed speech information to obtain a speech segmentation set corresponding to each processed speech information, that is, a speech segmentation set of any processed speech information includes corresponding audio segment 1, audio segment 2, audio segment 3, etc.

[0050] (3) Perform speech recognition on each audio segment in each speech cutting set to obtain the text corresponding to each audio segment, so as to use the text corresponding to each audio segment in each speech cutting set to form a text set of speech information of each designated person; of course, the text corresponding to each audio segment can also be visualized for manual proofreading to obtain the proofread text, and then use the proofread text to form a text set of speech information of each designated person.

[0051] (4) Inputting the text set of the voice information of each designated person into the neural network model for training, thereby obtaining a voice cloning model; finally, the voice cloning model can be stored in the model library for standby; in this embodiment, the initial model of the voice cloning model (i.e., the model before training) can be selected from, but not limited to, ChatTTS, Seed-TTS, Fish Speech models, etc.; at the same time, the example model library can also include, but not limited to, third-party text-to-speech library functions in Python such as pyttsx3, gtts, AipSpeech, etc., and can call existing text-to-speech API services on the Internet, such as Volcano Engine speech synthesis, Microsoft Text-to-Speech Service, iFlytek Spark and other API services, that is, calling the aforementioned API services to implement the voice cloning function.

[0052] In this way, based on the above steps (1) to (4), the training of the voice cloning model can be realized, and then, the teacher's answer audio corresponding to the question and answer result text can be generated based on the voice cloning model, that is: The voice cloning module is used to input the question and answer result text into the voice cloning model to obtain a cloned voice corresponding to the question and answer result text and having the same timbre as any designated person, and use the cloned voice corresponding to the question and answer result text output by the voice cloning model as the teacher's answer audio. In this embodiment, since the voice cloning model can clone the timbre of different designated teachers, in actual use, teacher's answer audio with corresponding timbre of different designated teachers can be generated as needed.

[0053] Thus, after completing the detailed working description of the aforementioned voice cloning module, this embodiment discloses the detailed working process of the digital virtual teacher cloning module.

[0054] In actual use, for example, the digital virtual teacher cloning module stores a teacher image cloning model, wherein the teacher image cloning model is trained with image data sets of several designated persons as input and digital human video streams corresponding to each designated person as output, and the image data set of any designated person includes the image features and audio features of the designated person, and the image features of any designated person are obtained based on the video or picture of the designated person.

[0055] Optional, see Figure 4 As shown in Figure 1, one of the training processes of the following public teacher image cloning model: (1) Obtaining initial data sets of several designated persons, wherein the initial data set of any designated person includes the person video and audio, or the person picture and audio of the designated person.

[0056] (2) Preprocessing the initial data set of each designated person to obtain an image data set of each designated person after the data preprocessing, wherein the data preprocessing includes extracting image features from the person video and the person picture, and extracting audio features from the audio; therefore, the image data set of any designated person includes the image features and audio features of the designated person; in this embodiment, for example, a convolutional neural network model, a recurrent neural network model and other models may be used, but are not limited to, to extract image features from videos and pictures; of course, different models may be selected to extract image features according to actual needs, and are not limited to the aforementioned examples; at the same time, the audio features may be used, but are not limited to, frequency domain features, time domain features and / or time-frequency domain features, and the image features may include, but are not limited to, the mouth shape features of the designated person.

[0057] (3) Inputting the image data sets of each designated person into the neural network model for training, that is, taking the image data sets of each designated person as input and the digital human video stream of each designated person as output, to train the neural network model, so that after the training is completed, a teacher image cloning model is obtained; finally, it is stored in the model library for standby; in this embodiment, the teacher image cloning model can be, but is not limited to, a GAN network model. Of course, open source models such as ER-Nerf, musetalk, wav2lip, sadtalker, etc. can also be used, or closed source commercial digital human software can be used, such as iFlytek Zhizao, Silicon Intelligent Digital Human and other software to drive the digital human in real time.

[0058] In this way, after completing the training of the teacher image cloning model based on the aforementioned steps, the real-time driving of the digital virtual teacher can be performed, that is: the digital virtual teacher cloning module is used to input the teacher's answer audio into the teacher image cloning model to obtain the question answering digital human video stream; in this embodiment, since image data sets of different designated teachers are used in training, in actual use, different designated teachers' question answering digital human video streams can be selected to be generated according to needs, such as teacher A's question answering digital human video stream, teacher B's question answering digital human video stream, etc.

[0059] Therefore, through the above-mentioned detailed explanation of the question-answer interaction module, the voice cloning module and the digital virtual teacher cloning module, the interactive virtual teacher system composed of the above-mentioned modules can be used to realize the function of an interactive virtual classroom; in this way, during the digital virtual teacher's lecture, the student's questions can be answered in real time in a dynamic and interactive manner, so that the student's questions can be answered in time, and at the same time the student's enthusiasm and participation can be improved.

[0060] In a possible design, the second aspect of this embodiment discloses richer functions of the entire system based on the first aspect of the embodiment.

[0061] Among them, in this embodiment, the aforementioned interactive virtual teacher system may also include but is not limited to: a teaching text generation module; wherein the teaching text generation module is used to generate teaching texts corresponding to course teaching, so that the subsequent voice cloning module and the digital virtual teacher cloning module can drive the digital virtual teacher in real time based on this, thereby realizing course teaching based on the digital virtual teacher.

[0062] In the specific implementation, see Figure 1 As shown, the teaching text generation module is used to obtain teaching prompt words, and input the teaching prompt words into the teaching text generation large language model to obtain the teaching text and send it to the voice cloning module; in specific implementation, the teaching text can include but is not limited to lesson plans or lectures.

[0063] Optionally, when the teaching prompt words are generated, the teaching text generation module responds to human-computer interaction operations by obtaining the teaching content table and extracting the teaching keywords in the teaching content table, and then using the extracted teaching keywords to generate the teaching text; specifically, the teaching content table can be but not limited to the form filling plus text description method, wherein the table content can be but not limited to including age, subject, textbook, volume, required class hours, subject, teaching method, teaching framework and other options, such as age can be but not limited to set as elementary school, junior high school, high school, secondary vocational school, university and other options; in the subject, a series of options such as Chinese, mathematics, English, geography, history, chemistry, information technology, etc. can be set.

[0064] Similarly, in textbooks, you can set options such as People's Education Press, Jiangsu Education Press, Beijing Press, and Zhejiang Education Press; in volume type, you can set options such as first grade volume 1, first grade volume 2, second grade volume 1, high school volume 1, compulsory courses, and elective courses; in required class hours, you can set options such as one class hour and two class hours.

[0065] In the subject, users can fill in the name of the subject that needs to be taught by the digital human virtual teacher in this column; in the teaching method, options such as lecture method, case teaching method, situational teaching method, etc. can be set, and users can choose one or more according to their needs; in the teaching framework, users can fill in the teaching framework that they want the model to generate content to follow in this column; in addition, there is a text box for filling in other requirements. When the content of the above table is not sufficient to support the user's requirements, the user can fill in additional requirements in this column.

[0066] In this way, the user interacts with the teaching text generation module, and the teaching text generation module can generate a corresponding teaching content table; then, the teaching content table is parsed and the internal keywords are extracted to obtain teaching prompt words.

[0067] Furthermore, after obtaining the teaching prompt words, they can be input into the teaching text to generate a large language model, so as to generate the corresponding teaching text; wherein, for example, the teaching text to generate a large language model can be but not limited to the use of open source models such as the ChatGLM series large model, the LLaMA series large model or the Qwen series large model, and closed source models such as ChatGPT, Wenxin Yiyan, Zhipu Qingyan, etc. can also be used; of course, the aforementioned models can also be incrementally pre-trained, fine-tuned, and search enhanced to obtain an optimized model, and finally, the optimized model is used as the teaching text to generate a large language model.

[0068] Furthermore, for example, after using the teaching text to generate a large language model to generate a teaching text, it can also be transmitted to the text editing module for content modification, that is, the text editing module visualizes the teaching text so that the teacher can edit the teaching text and upload the edited teaching text. At this time, the teaching text generation module can transmit the edited teaching text to the voice cloning module for voice cloning.

[0069] In addition, in this embodiment, the user can also directly upload the teaching text; of course, the specific method can be selected according to actual use and is not specifically limited here.

[0070] In this way, after generating the teaching text using the teaching text generation module, the voice cloning module can be used to generate the course teaching audio corresponding to the teaching text.

[0071] Specifically, the process of generating course teaching audio is as follows: See also Figure 3 As shown, the voice cloning module is also used to segment the teaching text to obtain a plurality of teaching segmentation texts, and based on each teaching segmentation text ( Figure 3 1, 2 and 3 in the text segment), generate several course teaching audios and send them to the voice management module, where each teaching segment text corresponds to a course teaching audio, that is, Figure 3 The audio in.

[0072] In this embodiment, since the teaching texts are mostly long texts, they need to be segmented to improve processing efficiency; optionally, different segmentation methods can be used to segment the teaching text, such as choosing to segment based on punctuation marks (i.e., segmenting based on commas or periods), segmenting by a fixed number of words (such as segmenting the long text every 10 words), segmenting according to semantics, etc.; of course, the specific segmentation method can be set according to actual use, and is not limited to the aforementioned examples.

[0073] In this way, after completing the segmentation of the teaching text, each segmented text (i.e., the teaching segmented text) can be sent to the voice cloning module to generate the corresponding course teaching audio; then, the course teaching audio can be sent to the voice management module to form a voice push queue, that is, the voice management module is used to receive a number of course teaching audios sent by the voice cloning module, and sort each course teaching audio according to the segmentation order of each course teaching audio corresponding to the teaching segmented text, so as to obtain the voice push queue; finally, the course teaching audios in the voice push queue are pushed to the digital virtual teacher cloning module in sequence to realize the real-time driving of the virtual digital teacher, thereby obtaining the digital human teaching video stream.

[0074] At the same time, in this embodiment, the processing of the aforementioned teaching course audio is performed before the course starts, that is, the aforementioned processes are completed in advance, namely: the voice management module is used to transmit the various course teaching audios in the voice push queue to the digital virtual teacher cloning module in sequence when receiving the course start instruction; and the digital virtual teacher cloning module is used to convert the received various course teaching audios into digital human teaching video streams in the order of reception, and push them to the client in sequence to realize the playback of course teaching videos based on digital virtual teachers.

[0075] Therefore, through the above explanation, this embodiment completes the functions of long text cutting and voice cloning before the course starts, and then pushes the voices corresponding to a series of sentences generated in advance by long text slicing in real time, so as to generate the corresponding digital human teaching video stream; in this way, the computing power pressure of the system is reduced, and the fluency of the digital virtual teacher's teaching and the real-time interactive response ability are improved.

[0076] In a possible design, the third aspect of this embodiment is further optimized on the basis of the second and first aspects of the embodiment, that is, to improve the fluency of the virtual teacher's teaching from the perspective of video stream transmission.

[0077] In specific applications, for example, the digital virtual teacher cloning module is also used to push the question-answering digital human video stream and the digital human teaching video stream to the streaming media server; and the streaming media server is used to push the question-answering digital human video stream or the digital human teaching video stream to the client upon receiving the video pull request from the client; of course, in this embodiment, the push priority of the question-answering digital human video stream is higher than the digital human teaching video stream, that is, the question-answering digital human video stream is transmitted first to achieve real-time answers to questions when the virtual teacher is teaching.

[0078] In this embodiment, for example, the system may also include a video stream push module, that is, the digital virtual teacher cloning module sends the question-answering digital human video stream and the digital human teaching video stream to the video stream push module; then the video stream push module pushes it to the streaming media server, and finally sends it to the client through the streaming media server.

[0079] Optionally, for example, it can be but not limited to using B / S mode for streaming media push, that is, adopting the working mode of browser request and server response; further, the specific transmission protocol can select rtmp, webrtc, hls, http-flv, srt and other real-time streaming media protocols, and can be but not limited to selecting srs, nginx and other streaming media servers; of course, the specific transmission protocol and streaming media server can be specifically set according to actual use, and is not limited to the aforementioned examples.

[0080] In this way, this embodiment is equivalent to pushing the real-time driven digital virtual teacher video stream (that is, the digital human teaching video stream) to the streaming media server and then the user end pulls the stream for playback to push the video stream; in this way, the system's high concurrency capability can be enhanced and the pressure on the back-end server can be alleviated, thereby further improving the smoothness of the video.

[0081] In addition, in this embodiment, the streaming media server is also used to preload the digital human teaching video stream before receiving the video pull request, and push the preloaded digital human teaching video stream to the client after receiving the video pull request; in this way, the streaming media server is equivalent to preloading one or several digital virtual teacher video streams to increase the fluency of the digital virtual classroom; of course, the length of the preloaded video stream can also be one or several audio sentence lengths, which can be specifically set according to actual use and is not limited to the above examples; in addition, the example client can be a dedicated software program client or a web browser client, that is, the streaming media server is accessed through the above two methods to achieve video stream pulling.

[0082] In this way, the real-time driven digital virtual teacher video stream is pushed to the streaming media server and then played by the user end, combined with the pre-loaded cache method, which can greatly improve the smoothness of the digital virtual classroom, thereby improving the user experience.

[0083] Therefore, the working process of the entire interactive virtual digital teacher system is: Teaching text generation module: This module uses the method of cue words and large language model to generate the lesson plans or teaching words required for the digital virtual teacher to give lectures, that is, to generate teaching text. Question and answer interaction module: This module receives questions from users (students), and then uses the text large language model or a multimodal large model to answer students' questions and obtain the question and answer result text. Voice cloning module: This module supports users (teachers) to upload their own voices, clone the voices through neural network training, and obtain a voice cloning model; wherein, this module receives the teaching text generated by the large language model in the teaching text generation module, and receives the question and answer result text generated in the question and answer interaction module; at the same time, it can realize automatic segmentation of long texts, and then select the above-mentioned voice cloning model for reasoning or use the text-to-speech conversion component to convert text into sound, and save it locally in sequence, that is, to obtain multiple course teaching audios or teacher answer audios. Voice management module: This module is responsible for reading, managing and pushing the text generated by the teaching text generation module and multiple voices (i.e., course teaching audio) generated by the voice cloning module, as well as the question and answer result text generated by the question and answer interaction module and the teacher's answer audio generated by the voice cloning module; wherein, this module can realize the function of audio management, and can realize the functions of sorting, inserting and pushing the above-mentioned audio; of course, in this embodiment, the priority of the teacher's answer audio is the highest priority, that is, the teacher's answer audio is pushed to the digital virtual teacher image cloning module first. Digital virtual teacher image cloning module: This module can realize image cloning by the teacher uploading his own video + audio, or picture + audio for neural network training, and can realize receiving the voice pushed by the voice management module, and drive the digital virtual teacher in real time, thereby generating the corresponding digital human teaching video stream or question answering digital human video stream. Video stream push module: This module receives the video stream generated by the digital virtual teacher image cloning module and pushes it to the streaming media server. Client; This module continuously requests and pulls the video stream pushed by the video stream push module in the streaming media server, and conducts real-time classroom teaching and answering questions. Of course, before the course starts, it is necessary to: Upload voice to the voice cloning module to realize the training of the teacher voice cloning model. The voice cloning module can realize the function of converting text into audio based on the voice cloning model. At the same time, upload video or photo to the digital virtual teacher cloning module to realize the training of the teacher image cloning model. The digital virtual teacher image cloning module can realize the generation of audio-driven virtual teacher video stream based on the teacher image cloning model.

[0084] Secondly, the teaching text generation module is used to generate teaching text based on the large language model plus prompt words, and the generated teaching text is transmitted to the text editing module for editing; of course, you can also choose to directly copy and paste the completed teaching text into the text editing module, and then transmit it to the voice cloning module.

[0085] Next, the teaching text is segmented to obtain a number of teaching segmentation texts, and based on each teaching segmentation text, a number of course teaching audios are generated and sent to the voice management module; and the voice management module receives the several course teaching audios sent by the voice cloning module, and sorts each course teaching audio according to the segmentation order of each course teaching audio corresponding to the teaching segmentation text, so as to obtain the voice push queue.

[0086] At this point, the course data preprocessing has been completed and the course teaching can be carried out, namely: Once the virtual course begins: The voice management module pushes the course teaching audio in the voice push queue to the digital virtual teacher cloning module in sequence according to the order. The digital virtual teacher cloning module generates a course teaching video stream based on the course teaching audio and pushes it to the streaming media server. The client will pull the relevant course teaching video for playback as needed.

[0087] At the same time, during the course teaching process, the question-answering interaction module obtains students' real-time question information, and based on the real-time question information, generates a question-answering result text and sends it to the voice cloning module; the voice cloning module converts the question-answering result text into the teacher's answer audio, and sends the teacher's answer audio to the voice management module.

[0088] Then, after receiving the teacher's answer audio, the voice management module interrupts the push of the course teaching audio in the voice push queue, and transmits the teacher's answer audio to the digital virtual teacher cloning module with priority.

[0089] Then, the digital virtual teacher cloning module generates a digital human video stream of the question answering based on the teacher's answering audio, and pushes the digital human video stream of the question answering to the streaming media server, and the client pulls the digital human video stream of the question answering to realize the reply to the user's question in the course teaching process; Finally, after transmitting the teacher's answer audio to the digital virtual teacher cloning module, the voice management module resumes the push of the course teaching audio in the voice push queue, so that the digital virtual teacher cloning module converts the course teaching audio pushed by the voice management module into a digital human teaching video stream, and pushes it to the client to resume the playback of the course teaching video based on the virtual teacher.

[0090] Thus, through the above design, the present invention pre-segments and voice clones the long teaching text, and forms a voice push queue with the obtained cloned voice in order, wherein, if the voice generated by the question-answer interaction module is received in the middle, the voice will be inserted into the front end of the voice push queue in real time to achieve real-time answering of the question; at the same time, after the course starts, the voice management module will continuously send the voice in the voice push queue, and drive the digital virtual teacher in real time; and after the digital virtual teacher video stream is generated, the streaming media server push-pull stream method can be selected, and it can be directly pushed to the client using the Webrtc p2p method; in this way, the present invention realizes the function of an interactive virtual classroom, and gets rid of the disadvantage that the traditional digital teacher system directly plays the video generated by the relevant neural network model and cannot interact; the system combines advanced artificial intelligence technology and adopts a real-time streaming dynamic drive method, which is a great improvement in the function of the traditional online teaching platform headed by the traditional digital human teacher system. During class, students can communicate with the digital human teacher in real time, which is conducive to promoting the digitization, intelligence, personalization and ubiquity of education, and is particularly suitable for online education platforms.

[0091] In addition, the system architecture can also be used as digital human tour guides or online shopping mall tour guides in scenes such as trains, trains, and airplanes, and is therefore not limited to the aforementioned application scenarios.

[0092] See also Figure 5 As shown, in a possible design, the fourth aspect of this embodiment provides a working method of the interactive virtual digital teacher system described in the first to third aspects of the embodiments, wherein the operating steps of the method may be but are not limited to the following steps S1 to S5.

[0093] S1. The question-answering interaction module obtains real-time question information during the course teaching process based on the virtual teacher, and based on the real-time question information, generates a question-answering result text and sends it to the voice cloning module.

[0094] S2. The voice cloning module converts the question and answer result text into the teacher's answer audio, and sends the teacher's answer audio to the voice management module.

[0095] S3. After receiving the teacher's answer audio, the voice management module interrupts the push of the course teaching audio in the voice push queue and transmits the teacher's answer audio to the digital virtual teacher cloning module.

[0096] S4. The digital virtual teacher cloning module generates a question-answering digital human video stream based on the teacher's answer audio, and pushes the question-answering digital human video stream to the client to respond to the user's questions during the course teaching process.

[0097] S5. After transmitting the teacher's answer audio to the digital virtual teacher cloning module, the voice management module resumes the push of the course teaching audio in the voice push queue, so that the digital virtual teacher cloning module converts the course teaching audio pushed by the voice management module into a digital human teaching video stream and pushes it to the client to resume the playback of the course teaching video based on the virtual teacher.

[0098] The working details and technical effects of the method provided in this embodiment can be found in the first to third aspects of the embodiments, and will not be elaborated here.

[0099] The fifth aspect of this embodiment provides an electronic device, comprising: a memory, a processor and a transceiver that are communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the working method of the interactive virtual digital teacher system as described in the fourth aspect of the embodiment.

[0100] For example, the memory may include, but is not limited to, random access memory (RAM), read only memory (ROM), flash memory, first input first output (FIFO) and / or first in last out (FILO), etc. Specifically, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). At the same time, the processor may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state.

[0101] In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. For example, the processor may not be limited to a microprocessor of the STM32F105 series, a reduced instruction set computer (RISC) microprocessor, an X86 or other architecture processor, or a processor with an integrated embedded neural network processing unit (NPU); the transceiver may be, but is not limited to, a wireless fidelity (WIFI) wireless transceiver, a Bluetooth wireless transceiver, a general packet radio service technology (General Packet Radio Service, GPRS) wireless transceiver, a ZigBee protocol (a low-power local area network protocol based on the IEEE802.15.4 standard, ZigBee) wireless transceiver, a 3G transceiver, a 4G transceiver and / or a 5G transceiver, etc. In addition, the device may also include, but is not limited to, a power module, a display screen and other necessary components.

[0102] The working process, working details and technical effects of the electronic device provided in this embodiment can be found in the first to third aspects of the embodiments, and will not be elaborated here.

[0103] The sixth aspect of this embodiment provides a storage medium that stores instructions for the working method of the interactive virtual digital teacher system described in the fourth aspect of the embodiment, that is, the storage medium stores instructions, and when the instructions are run on a computer, the working method of the interactive virtual digital teacher system described in the fourth aspect of the embodiment is executed.

[0104] The storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, a CD, a hard disk, a flash memory, a USB flash drive and / or a memory stick, and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0105] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the first to third aspects of the embodiments, and will not be described in detail here.

[0106] The seventh aspect of this embodiment provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the working method of the interactive virtual digital teacher system as described in the fourth aspect of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0107] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An interactive virtual digital teacher system, characterized in that: include: A question-answering interaction module, wherein the question-answering interaction module is used to obtain real-time question information during the course teaching process based on the virtual teacher, and based on the real-time question information, generate a question-answering result text and send it to the voice cloning module; A voice cloning module, wherein the voice cloning module is used to convert the question and answer result text into the teacher's answer audio, and send the teacher's answer audio to the voice management module; A voice management module, wherein a voice push queue is stored in the voice management module, wherein the voice push queue includes a plurality of course teaching audios, and the voice management module is used to interrupt the push of the course teaching audios in the voice push queue after receiving the teacher's answer audio, and transmit the teacher's answer audio to the digital virtual teacher cloning module; A digital virtual teacher cloning module is used to generate a digital human video stream of question answers based on the teacher's answer audio, and push the digital human video stream of question answers to the client to realize replying to the user's questions during the course teaching process; The voice management module is also used to resume the push of the course teaching audio in the voice push queue after transmitting the teacher's answer audio to the digital virtual teacher cloning module, so that the digital virtual teacher cloning module converts the course teaching audio pushed by the voice management module into a digital human teaching video stream, and pushes it to the client to resume the playback of the course teaching video based on the virtual teacher.

2. An interactive virtual digital teacher system according to claim 1, characterized in that: If the real-time question information is text information, the question-answering interaction module is used to input the real-time question information into a first question-answering model to obtain the question-answering result text, wherein the first question-answering model is trained by taking a number of historical question text information as input and the question-answering results corresponding to each historical question text information as output, and the first question-answering model includes a text large language model; If the real-time question information is voice information, the question-answering interaction module is used to convert the real-time question information into text to obtain a question text, and input the question text into the first question-answering model to obtain a question-answering result text; or Inputting the real-time question information into a pre-trained second question-answering model to obtain the question-answering result text, wherein the second question-answering model includes a multimodal large language model; If the real-time question information is picture and text information, the question and answer interaction module is used to input the real-time question information into the second question and answer model to obtain the question and answer result text.

3. An interactive virtual digital teacher system according to claim 1, characterized in that: Each course teaching audio in the voice push queue corresponds to a teaching segmentation text, wherein the teaching segmentation texts corresponding to all course teaching audios constitute the teaching text of the current digital human course teaching, and the sorting order of each course teaching audio in the voice push queue is the segmentation order of each teaching segmentation text; The voice management module is used to transmit the course teaching audios in the voice push queue to the digital virtual teacher cloning module in sequence upon receiving the course start instruction; The digital virtual teacher cloning module is used to convert the received course teaching audio into digital human teaching video streams in the order of reception, and push them to the client in sequence to realize the playback of course teaching videos based on digital virtual teachers.

4. An interactive virtual digital teacher system according to claim 3, characterized in that: Also includes: A teaching text generation module, wherein the teaching text generation module is used to obtain teaching prompt words, and input the teaching prompt words into a teaching text generation large language model to obtain a teaching text and send it to the voice cloning module; The voice cloning module is also used to segment the teaching text to obtain a plurality of teaching segmentation texts, and generate a plurality of course teaching audios based on each teaching segmentation text and send them to the voice management module, wherein each teaching segmentation text corresponds to a course teaching audio; The voice management module is used to receive a plurality of course teaching audios sent by the voice cloning module, and sort the course teaching audios according to the segmentation order of the teaching segmentation texts corresponding to the course teaching audios to obtain the voice push queue.

5. An interactive virtual digital teacher system according to claim 4, characterized in that: The teaching text generation module is also used to obtain the teaching content table and extract the teaching keywords in the teaching content table in response to the human-computer interaction operation, so as to generate the teaching prompt words using the extracted teaching keywords.

6. An interactive virtual digital teacher system according to claim 1, characterized in that: The voice cloning module stores a voice cloning model, wherein the voice cloning model is trained by taking a text set corresponding to voice information of several designated persons as input and a cloned voice corresponding to each text in each text set as output, wherein the text set corresponding to any designated person includes texts of each audio segment of the voice information corresponding to the designated person, the designated persons include multiple designated teachers, and the timbre of the cloned voice corresponding to each text in any text set is the same as the timbre of the designated person corresponding to the text set; Among them, the voice cloning module is used to input the question and answer result text into the voice cloning model to obtain a cloned voice corresponding to the question and answer result text and with the same timbre as any designated person, and use the cloned voice corresponding to the question and answer result text output by the voice cloning model as the teacher's answer audio.

7. An interactive virtual digital teacher system according to claim 1, characterized in that: The digital virtual teacher cloning module stores a teacher image cloning model, wherein the teacher image cloning model is trained by taking image data sets of several designated persons as input and digital human video streams corresponding to each designated person as output, the image data set of any designated person includes the image features and audio features of any designated person, and the image features of any designated person are obtained based on the video or picture of any designated person; The digital virtual teacher cloning module is used to input the teacher's answer audio into the teacher image cloning model to obtain the question answering digital human video stream.

8. An interactive virtual digital teacher system according to claim 1, characterized in that: The digital virtual teacher cloning module is also used to push the question-answering digital human video stream and the digital human teaching video stream to the streaming media server, wherein the question-answering digital human video stream has a higher push priority than the digital human teaching video stream; The streaming media server is used to push the question-answering digital human video stream or the digital human teaching video stream to the client upon receiving a video pull request from the client.

9. An interactive virtual digital teacher system according to claim 8, characterized in that: The streaming media server is also used to preload the digital human teaching video stream before receiving the video pull request, and push the preloaded digital human teaching video stream to the client after receiving the video pull request.

10. A working method of the interactive virtual digital teacher system according to any one of claims 1 to 9, characterized in that: include: The question-answering interaction module obtains real-time question information during the course teaching process based on the virtual teacher, and based on the real-time question information, generates a question-answering result text and sends it to the voice cloning module; The voice cloning module converts the question and answer result text into the teacher's answer audio, and sends the teacher's answer audio to the voice management module; After receiving the teacher's answer audio, the voice management module interrupts the push of the course teaching audio in the voice push queue, and transmits the teacher's answer audio to the digital virtual teacher cloning module; The digital virtual teacher cloning module generates a digital human video stream of the question answering based on the teacher's answering audio, and pushes the digital human video stream of the question answering to the client to realize the reply to the user's question in the course teaching process; After transmitting the teacher's answer audio to the digital virtual teacher cloning module, the voice management module resumes the push of the course teaching audio in the voice push queue, so that the digital virtual teacher cloning module converts the course teaching audio pushed by the voice management module into a digital human teaching video stream, and pushes it to the client to resume the playback of the course teaching video based on the virtual teacher.

Images