An intelligent course recording system based on AI digital human and its working method

Abstract

The present invention discloses an intelligent class recording system based on AI digital human and a working method thereof, and the present invention relates to the field of teaching technology of artificial intelligence information technology. The system includes a video acquisition module, a voice processing module, an image processing module, a digital human training module, a digital human generation module and a class recording video synthesis module. The working method comprises the following steps: S1: video recording and processing; S2: voice data processing; S3: human image data processing; S4: digital human training; S5: digital human generation; S6: class recording video synthesis. The present invention constructs an intelligent class recording system based on AI digital human and a working method thereof, which can intelligently generate a teacher image and synthesize a class recording video according to the input text, thereby reducing the time and labor cost spent by the teacher on class recording. The present invention provides a new way for teachers to record classes and promotes the intelligence of teaching activities.

Description

Technical Field

[0001] The present invention relates to the field of teaching technology of artificial intelligence information technology, and specifically to an intelligent course recording system based on AI digital human and a working method thereof. Background Art

[0002] Recording classes is an important part of teaching activities and has great significance. Students are able to study freely according to their own schedule and watch specific parts repeatedly when needed. This flexibility enables students to better grasp the course content and review it when needed. Recording classes can also create high-quality teaching resources that can be shared with a wider group of students. In this way, educational resources can be more easily disseminated in different places and times, providing learning opportunities for more students.

[0003] However, recording high-quality courses requires appropriate equipment and technology, such as cameras, microphones, video editing software, etc., and requires certain technical skills to operate these equipment and software. In addition, recording courses takes a lot of time, including preparing courseware, recording, editing, and publishing, which may add extra workload for teachers. Therefore, a flexible and intelligent course recording solution is urgently needed. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide an intelligent course recording system based on AI digital human and its working method. Combining the latest digital human technology, the neural radiation field ER-NeRF technology is used to customize the teacher's head and upper body, the intelligent voice replication technology Bert-VIT2 is used to customize the teacher's timbre, and the image processing technology is used to generate a complete digital human.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: an intelligent course recording system based on AI digital human, the system comprising a video acquisition module, a voice processing module, an image processing module, a digital human training module, a digital human generation module and a course recording synthesis module;

[0006] The video acquisition module cuts the recorded short video and separates the audio and video files;

[0007] The speech processing module uses the UVR5 model to separate the front and background sounds of the speech, uses the audio cutting tool Audio-Slice to perform audio segmentation, and uses the HuBERT model to extract audio features;

[0008] The image processing module uses the face parsing model face-parsing to segment the head and torso of the person, and uses the OpenFace framework to extract the coding features of AU45 binocular blinking;

[0009] The digital human training module iterates the training of the head, torso and voice of the customized character respectively; the head and torso of the character are trained by the ER-NeRF model, and the voice of the character is trained by the Bert-VITS2 model;

[0010] The digital human generation module outputs customized character recorded class voices based on the input text through the Bert-VITS2 model inference, and outputs customized character lecture videos through the ER-NeRF model inference;

[0011] The recorded lesson synthesis module cuts and pastes the digital human, combines it with the original character's body, and combines the audio and video to generate the final recorded lesson video.

[0012] The present invention also discloses a working method of an intelligent course recording system based on AI digital human, which comprises the following steps:

[0013] Step S1: Video recording and processing, recording the teacher's lecture video, cropping the video and modifying the frame rate, and separating the audio from the video;

[0014] Step S2: voice data processing, processing the audio separated in step S1 to extract voice features;

[0015] Step S3: human body image data processing, performing head image segmentation, torso image segmentation and blink feature extraction on each frame in the video;

[0016] Step S4: Digital human training, using the neural radiation field-based algorithm ER-NeRF for digital human training, and using Bert-VITS2 for character voice reproduction;

[0017] Step S5: Digital human generation, using the ER-NeRF model for reasoning to generate a customized digital human teacher’s video, and using the Bert-VITS2 model for reasoning to generate a customized digital human teacher’s audio;

[0018] Step S6: synthesize the recorded class video, cut and paste the generated digital human back into the original video, splice the original video of the person, synthesize the voice and video, and form the final recorded class video.

[0019] Preferably, the video recording and processing method in step S1 is specifically as follows:

[0020] Step S1-1: Shoot a teacher's lecture video with a duration of 5 minutes. In order to ensure the final effect, the subject is required to be always in the camera without any obstruction;

[0021] Step S1-2: After recording is completed, separate the audio from the video and crop the video to ensure that the video image above the shoulders covers the entire video as much as possible, scale the image and modify the frame rate to ensure that the video resolution is 512*512 and the frame rate is 25fps, and record the scaling factor.

[0022] Preferably, the voice data processing method in step S2 is specifically:

[0023] Step S2-1: Use the UVR5 model to separate the human voice from the background sound in step S1, remove the background noise, and obtain clean human voice audio;

[0024] Step S2-2: Obtain the audio cutting tool Audio-Slice from the official website, use the tool to cut the above human voice audio into multiple audio segments, then use the FunASR Chinese model to identify and annotate the audio segments, and integrate the annotated audio segments with the annotated files as a speech training set;

[0025] Step S2-3: Use the HuBERT model to process the clean vocal audio in step S2-1 and extract features.

[0026] Preferably, the human body image processing method in step S3 is specifically:

[0027] Step S3-1: extract all frames in the video of step S1, use the trained face parsing model face-parsing to perform image segmentation, and extract the head part of the person in each frame;

[0028] Step S3-2: Use the face-parsing model again to segment the torso of the person in each frame;

[0029] Step S3-3: Use the OpenFace framework to extract the encoding features of AU45 binocular blink defined by FACES.

[0030] Preferably, the digital human training method in step S4 is specifically:

[0031] Step S4-1: Build an ER-NeRF model and train the character's head. The input of the model is the voice feature and the blink feature. The output of the model is the color and density of the head. The loss is calculated with the head part transformed in step S3-1. After the iteration is completed, save the trained head weight file, load the trained head weight, apply ER-NeRF to fine-tune the character's lips, and after the iteration is completed, save the optimized head weight.

[0032] Step S4-2: Load the head weights optimized in the above step S4-1, apply ER-NeRF to train the character's torso, the output of the network is the color and density of the torso, and the loss is calculated with the torso transformed in step S3-2. After the training is completed, save the weights;

[0033] Step S4-3: Build the Bert-VITS2 model, train the character voice, use the tools provided by Bert-VITS2 to process the voice training set in step S2-2, generate relevant feature files, and perform training after completion. After the training iteration is completed, save the voice weight file.

[0034] Preferably, the digital human synthesis method in step S5 is specifically as follows:

[0035] Step S5-1: Prepare the course content to be recorded and save it as text;

[0036] Step S5-2: Input the text of the recorded course content into the Bert-VITS2 model for inference, convert the text into speech, and output customized character audio;

[0037] Step S5-3: Use the HuBERT model to extract the customized character audio features in step S5-2, input the audio features and blink features into the ER-NeRF model for inference, and drive the generation of digital humans.

[0038] Preferably, the method for synthesizing the recorded class video in step S6 is specifically as follows:

[0039] Step S6-1: scaling the resolution of the digital human video in step S5-3, and restoring the initial resolution of the video according to the scaling factor in step S1-2. To ensure that the duration of the digital human video is consistent with that of the original video, the following operations are performed on the original video: if the duration of the digital human video is longer than that of the original video, the original video is repeated to complete the remaining duration. If the duration of the digital human video is shorter than that of the original video, the original video is edited to remove the redundant duration at the end of the original video. After the duration alignment is completed, the restored digital human video is pasted back to the original video.

[0040] Step S6-2: Combine the clipped digital human video with the character audio in step S5-2 to synthesize the audio and video as the final recorded class video.

[0041] The present invention provides an intelligent course recording system based on AI digital human and its working method. Compared with the prior art, it has the following beneficial effects: the system intelligently generates corresponding lecture videos according to the input course text, realizes the customization of the teacher's voice and character image, reduces the time cost required for teachers to record lessons, and improves the flexibility and convenience of students' learning. In addition, the present invention also provides a new solution for teachers to record lessons, which promotes the automation and intelligence of teaching. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The present invention is a flowchart of the system operation in the embodiment of the present invention. DETAILED DESCRIPTION

[0043] Some knowledge is not explained in this specification, which belongs to the prior art known to the professional and technical personnel and is easy to understand. The technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention.

[0044] An intelligent course recording system based on AI digital human, the system comprises a video acquisition module, a voice processing module, an image processing module, a digital human training module, a digital human generation module and a course recording synthesis module;

[0045] The video acquisition module cuts the recorded short video and separates the audio and video files;

[0046] The speech processing module uses the UVR5 model to separate the front and background sounds of the speech, uses the audio cutting tool Audio-Slice to perform audio segmentation, and uses the HuBERT model to extract audio features;

[0047] The image processing module uses the face parsing model face-parsing to segment the head and torso of the person, and uses the OpenFace framework to extract the coding features of AU45 binocular blinking;

[0048] The digital human training module iterates the training of the head, torso and voice of the customized character respectively; the head and torso of the character are trained by the ER-NeRF model, and the voice of the character is trained by the Bert-VITS2 model;

[0049] The digital human generation module outputs customized character recorded class voices based on the input text through the Bert-VITS2 model inference, and outputs customized character lecture videos through the ER-NeRF model inference;

[0050] The recorded lesson synthesis module cuts and pastes the digital human, combines it with the original character's body, and combines the audio and video to generate the final recorded lesson video.

[0051] The present invention also discloses a working method of an intelligent course recording system based on AI digital human, which comprises the following steps:

[0052] Step S1: Video recording and processing: Record the teacher's lecture video, crop the video, modify the frame rate, and separate the audio from the video.

[0053] Step S1-1: Shoot a teacher's lecture video with a duration of 5 minutes. In order to ensure the final effect, the person is required to be always in the camera without any obstruction.

[0054] Step S1-2: After recording, separate the audio from the video and crop the video to ensure that the video above the shoulders covers the entire video as much as possible. Scale the image and modify the frame rate to ensure that the video resolution is 512*512 and the frame rate is 25fps. Record the scaling factor. d .

[0055] Step S2: Voice data processing: The audio separated in step S1 is processed to extract voice features.

[0056] Step S2-1: Use the UVR5 model to separate the human voice from the background sound in step S1, remove the background noise, and obtain clean human voice audio.

[0057] Step S2-2: Obtain the audio cutting tool Audio-Slice from the official website, use the tool to cut the above human voice audio into multiple audio segments, and then use the FunASR Chinese model to identify and annotate the audio segments. The annotated files are in .lab format. The annotated audio segments and the annotated files are integrated as the speech training set.

[0058] Step S2-3: Use the HuBERT model to process the clean vocal audio in step S2-1, extract features, and save them as hubert.npy file.

[0059] Step S3: Human body image data processing: Perform head image segmentation, torso image segmentation and blink feature extraction on each frame in the video.

[0060] Step S3-1: Extract all frames in the video of step S1, use the trained face parsing model face-parsing (https: / / github.com / zllrunning / face-parsing.PyTorch) to perform image segmentation, and extract the head part of the person in each frame.

[0061] Step S3-2: Use the face-parsing model again to segment the torso of the person in each frame.

[0062] Step S3-3: Use the OpenFace framework to extract the coding features of AU45 binocular blink defined by FACES (Facial Action Coding System) and save them as au.csv file.

[0063] Step S4: Digital human training. Use the neural radiation field-based algorithm ER-NeRF to train the digital human. Use Bert-VITS2 to reproduce the character's voice.

[0064] Step S4-1: Build an ER-NeRF model and train the character's head. The input of the model is the speech features in the hubert.npy file and the blink features in the au.csv file. The output of the model is the color and density of the head, which is used to calculate the loss of the head part after the transformation in step S3-1. After the iteration is completed, save the trained head weight file. Load the trained head weight, apply ER-NeRF to fine-tune the character's lips, and save the optimized head weight after the iteration is completed.

[0065] Step S4-2: Load the head weights optimized in step S4-1 above, and use ER-NeRF to train the torso. The output of the network is the color and density of the torso, which is calculated with the torso transformed in step S3-2. After training, save the weights.

[0066] Step S4-3: Build the Bert-VITS2 model and train the character voice. Use the tools provided by Bert-VITS2 to process the voice training set in step S2-2, generate relevant feature files, and then perform training. After the training iteration is completed, save the voice weight file.

[0067] Step S5: Digital human generation. Use the ER-NeRF model for inference to generate a customized digital human teacher video. Use the Bert-VITS2 model for inference to generate a customized digital human teacher audio.

[0068] Step S5-1: Prepare the course content to be recorded and save it as text.

[0069] Step S5-2: Input the text of the recorded course content into the Bert-VITS2 model for inference, convert the text into speech, and output customized character audio.

[0070] Step S5-3: Use the HuBERT model to extract the audio features of the customized person in step S5-2 and save them as the custom.npy audio feature file. Input the custom.npy file and the au.csv file into the ER-NeRF model for inference to drive the generation of digital humans.

[0071] Step S6: Synthesis of recorded class video. Paste the generated digital human back into the original video and splice the original video of the person. Synthesize the voice and video to form the final recorded class video.

[0072] Step S6-1: Scaling the resolution of the digital human video in step S5-3 according to the scaling factor in step S1-2 d , restore the original resolution of the video. To ensure that the duration of the digital human video is consistent with that of the original video, the following operations are performed on the original video: If the duration of the digital human video is longer than that of the original video, the original video is repeated to make up the remaining duration. If the duration of the digital human video is shorter than that of the original video, the original video is edited to remove the excess duration at the end of the original video. After the duration alignment is completed, the restored digital human video is pasted back to the original video.

[0073] Step S6-2: Combine the clipped digital human video with the character audio in step S5-2 to synthesize the audio and video as the final recorded class video.

[0074] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent course recording system based on AI digital human, characterized by: The system includes a video acquisition module, a voice processing module, an image processing module, a digital human training module, a digital human generation module and a recorded lesson synthesis module; The video acquisition module cuts the recorded short video and separates the audio and video files; The speech processing module uses the UVR5 model to separate the front and background sounds of the speech, uses the audio cutting tool Audio-Slice to perform audio segmentation, and uses the HuBERT model to extract audio features; The image processing module uses the face parsing model face-parsing to segment the head and torso of the person, and uses the OpenFace framework to extract the coding features of AU45 binocular blinking; The digital human training module iterates the training of the head, torso and voice of the customized character respectively; the head and torso of the character are trained by the ER-NeRF model, and the voice of the character is trained by the Bert-VITS2 model; The digital human generation module outputs customized character recorded class voices based on the input text through the Bert-VITS2 model inference, and outputs customized character lecture videos through the ER-NeRF model inference; The recorded lesson synthesis module cuts and pastes the digital human, combines it with the original character's body, and combines the audio and video to generate the final recorded lesson video.

2. A working method of an intelligent course recording system based on AI digital human, comprising an intelligent course recording system based on AI digital human as claimed in claim 1, characterized in that: The method comprises the following steps: Step S1: Video recording and processing, recording the teacher's lecture video, cropping the video and modifying the frame rate, and separating the audio from the video; Step S2: voice data processing, processing the audio separated in step S1 to extract voice features; Step S3: human body image data processing, performing head image segmentation, torso image segmentation and blink feature extraction on each frame in the video; Step S4: Digital human training, using the neural radiation field-based algorithm ER-NeRF for digital human training, and using Bert-VITS2 for character voice reproduction; Step S5: Digital human generation, using the ER-NeRF model for reasoning to generate a customized digital human teacher’s video, and using the Bert-VITS2 model for reasoning to generate a customized digital human teacher’s audio; Step S6: synthesize the recorded class video, cut and paste the generated digital human back into the original video, splice the original video of the person, synthesize the voice and video, and form the final recorded class video.

3. The working method of the intelligent course recording system based on AI digital human according to claim 2 is characterized by: The video recording and processing method in step S1 is specifically as follows: Step S1-1: Shoot a teacher's lecture video with a duration of 5 minutes. In order to ensure the final effect, the subject is required to be always in the camera without any obstruction; Step S1-2: After recording is completed, separate the audio from the video and crop the video to ensure that the video image above the shoulders covers the entire video as much as possible, scale the image and modify the frame rate to ensure that the video resolution is 512*512 and the frame rate is 25fps, and record the scaling factor.

4. The working method of the intelligent course recording system based on AI digital human according to claim 2 is characterized by: The voice data processing method in step S2 is specifically as follows: Step S2-1: Use the UVR5 model to separate the human voice from the background sound in step S1, remove the background noise, and obtain clean human voice audio; Step S2-2: Obtain the audio cutting tool Audio-Slice from the official website, use the tool to cut the above human voice audio into multiple audio segments, then use the FunASR Chinese model to identify and annotate the audio segments, and integrate the annotated audio segments with the annotated files as a speech training set; Step S2-3: Use the HuBERT model to process the clean vocal audio in step S2-1 and extract features.

5. The working method of the intelligent course recording system based on AI digital human according to claim 2 is characterized by: The human body image processing method in step S3 is specifically as follows: Step S3-1: extract all frames in the video of step S1, use the trained face parsing model face-parsing to perform image segmentation, and extract the head part of the person in each frame; Step S3-2: Use the face-parsing model again to segment the torso of the person in each frame; Step S3-3: Use the OpenFace framework to extract the encoding features of AU45 binocular blink defined by FACES.

6. The working method of the intelligent course recording system based on AI digital human according to claim 2 is characterized by: The digital human training method in step S4 is specifically as follows: Step S4-1: Build an ER-NeRF model and train the character's head. The input of the model is the voice feature and the blink feature. The output of the model is the color and density of the head. The loss is calculated with the head part transformed in step S3-1. After the iteration is completed, save the trained head weight file, load the trained head weight, apply ER-NeRF to fine-tune the character's lips, and after the iteration is completed, save the optimized head weight. Step S4-2: Load the head weights optimized in the above step S4-1, apply ER-NeRF to train the character's torso, the output of the network is the color and density of the torso, and the loss is calculated with the torso transformed in step S3-2. After the training is completed, save the weights; Step S4-3: Build the Bert-VITS2 model, train the character voice, use the tools provided by Bert-VITS2 to process the voice training set in step S2-2, generate relevant feature files, and perform training after completion. After the training iteration is completed, save the voice weight file.

7. The working method of the intelligent course recording system based on AI digital human according to claim 2 is characterized by: The digital human synthesis method in step S5 is specifically as follows: Step S5-1: Prepare the course content to be recorded and save it as text; Step S5-2: Input the text of the recorded course content into the Bert-VITS2 model for inference, convert the text into speech, and output customized character audio; Step S5-3: Use the HuBERT model to extract the customized character audio features in step S5-2, input the audio features and blink features into the ER-NeRF model for inference, and drive the generation of digital humans.

8. The working method of the intelligent course recording system based on AI digital human according to claim 2 is characterized by: The method for synthesizing the recorded class video in step S6 is specifically as follows: Step S6-1: scaling the resolution of the digital human video in step S5-3, and restoring the initial resolution of the video according to the scaling factor in step S1-2. To ensure that the duration of the digital human video is consistent with that of the original video, the following operations are performed on the original video: if the duration of the digital human video is longer than that of the original video, the original video is repeated to complete the remaining duration. If the duration of the digital human video is shorter than that of the original video, the original video is edited to remove the redundant duration at the end of the original video. After the duration alignment is completed, the restored digital human video is pasted back to the original video. Step S6-2: Combine the clipped digital human video with the character audio in step S5-2 to synthesize the audio and video as the final recorded class video.

Images