A method, device, equipment, medium and product for automatically generating micro-course videos

Abstract

This application discloses a method, device, equipment, medium, and product for automatically generating micro-class videos, relating to the field of micro-class video production. The method includes cloning the teacher's voice based on a teaching video; extracting key facial features of the teacher in the teaching video to construct a virtual digital human of the teacher; extracting the teacher's lecture notes to generate a teaching document; extracting PPT content from the teaching video to reconstruct the PPT page; and synthesizing the cloned teacher's voice, the virtual digital human, the teaching document, and the PPT page to create a micro-class teaching video. This application can reduce technical difficulty and improve the efficiency of generating micro-class videos.

Description

Technical Field

[0001] The present application relates to the field of micro-course video production, and in particular to a method, device, equipment, medium and product for automatically generating micro-course videos. Background Art

[0002] Although China's educational informatization has achieved phased success, the integration of digital technology and education still faces many challenges. This is especially true for the development of digital educational resources, which still requires greater efforts in developing high-quality course resources. For ordinary teachers, courseware, lesson plans, practical cases, code, data, homework, test papers, question banks, and other essential teaching resources are relatively easy to implement with extensive experience in digital development. However, the development of video resources, such as micro-courses, presents an insurmountable gap. This is because video production has a high technical threshold and requires high standards in teaching design, language expression, personal image, on-camera performance, and post-production editing techniques. It generally requires professional companies to participate in the recording, which is not only costly, but also time-consuming and technically difficult, leading to physical and mental fatigue for teachers, thus hindering the speed and quality of course video resource development. Summary of the Invention

[0003] The purpose of this application is to provide a method, device, equipment, medium and product for automatically generating micro-course videos to solve the problems of high technical difficulty and low efficiency in generating micro-course videos.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a method for automatically generating micro-course videos, which is implemented based on artificial intelligence technology and includes:

[0006] Clone the teacher's voice based on the teacher's teaching video;

[0007] Extracting key points of the teacher's face in the teaching video and constructing a virtual digital human of the teacher;

[0008] Extract teacher's lecture notes and generate teaching copy;

[0009] Extract PPT content from the teaching video and reconstruct the PPT page;

[0010] The cloned teacher's voice, the virtual digital human, the teaching text and the PPT page are synthesized to create a micro-class teaching video.

[0011] Optionally, cloning the teacher's voice based on the teacher's teaching video specifically includes:

[0012] Based on adaptive noise cancellation technology, the denoised speech signal is extracted from the teaching video containing noisy speech through an adaptive filter;

[0013] Extracting speech features from the denoised speech signal using an acoustic model; the speech features include Mel-frequency cepstral coefficient features;

[0014] Based on a variational autoencoder, the speech features are mapped to a latent space, and the decoder reconstructs the speech to generate a cloned teacher voice.

[0015] Optionally, extracting key points of the teacher's face in the teaching video and constructing a virtual digital human of the teacher specifically includes:

[0016] A deep learning module is used to extract the teacher's facial key points and body shape in the teaching video;

[0017] Performing 3D facial reconstruction based on the facial key points to generate a 3D facial model;

[0018] Mapping the teacher's facial texture onto the 3D facial model using UV mapping and image fusion technology;

[0019] Based on the mapped 3D facial model, a virtual digital human image is constructed using a generative adversarial network.

[0020] Animating and driving the virtual digital human image to generate a dynamic virtual digital human;

[0021] Performing a 3D rendering engine on the dynamic virtual digital human to generate a rendered virtual digital human;

[0022] The rendered virtual digital human is optimized to construct a virtual digital human.

[0023] Optionally, extract teacher's lecture notes and generate teaching copy, including:

[0024] Preprocessing and correcting the text of the teacher's lecture notes to generate a corrected text;

[0025] Using text summarization technology to screen key points in the corrected text, and using a dictionary or knowledge base to perform terminology verification on the corrected text;

[0026] Adjust the language style of the verified text to generate an adjusted text;

[0027] Optimizing the adjusted text using a reinforcement learning method to generate a text optimization strategy;

[0028] A sequence generation model is used to generate teaching texts according to the text optimization strategy; the sequence generation model includes a recurrent neural network and a Transformer model.

[0029] Optionally, extracting PPT content from the teaching video and reconstructing the PPT page specifically includes:

[0030] Extracting key frames from the teaching video according to timestamps and content changes of each video frame in the teaching video; the key frames contain PPT content;

[0031] Using a target detection algorithm to locate the display area of the PPT content in the key frame;

[0032] Based on the display area, the image segmentation technology is used to separate the PPT content and the background;

[0033] Use optical character recognition technology to identify and separate text from PPT content;

[0034] Use image restoration technology to repair obscured PPT content and generate restored PPT content;

[0035] Recombine the text in the separated PPT content and the repaired PPT content, adjust the PPT layout and style, and reconstruct the PPT page.

[0036] Optionally, synthesizing the cloned teacher's voice, the virtual digital human, the teaching text, and the PPT page to create a micro-class teaching video specifically includes:

[0037] Using speech analysis technology, extracting audio features of the cloned teacher's voice; the audio features include phonemes, volume, and pitch;

[0038] Inputting the audio features into a combined model of a convolutional neural network and a recurrent neural network to output a lip movement sequence; the combined model is a lip synchronization model;

[0039] Utilize Blendshape technology to deform the vertices of the 3D facial model to generate a deformed 3D facial model;

[0040] Applying the lip movement sequence to the deformed 3D facial model to generate a virtual digital human with dynamic lip movements;

[0041] Rendering the virtual digital human with dynamic lip movements using a 3D rendering engine to generate a rendered virtual digital human;

[0042] The rendered virtual digital human, the cloned teacher's voice, the teaching text and the PPT page are synchronously synthesized to create a micro-class teaching video.

[0043] A micro-course video automatic generation system, which is implemented based on artificial intelligence technology and includes:

[0044] The cloning module is used to clone the teacher's voice based on the teacher's teaching video;

[0045] A virtual digital human construction module is used to extract key points of the teacher's face in the teaching video and construct a virtual digital human of the teacher;

[0046] Teaching copy generation module, used to extract teacher's lecture notes and generate teaching copy;

[0047] A PPT page reconstruction module, used to extract PPT content from the teaching video and reconstruct the PPT page;

[0048] The micro-class teaching video creation module is used to synthesize the cloned teacher's voice, the virtual digital human, the teaching text and the PPT page to create a micro-class teaching video.

[0049] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any one of the above-described methods for automatically generating micro-course videos.

[0050] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the above-mentioned methods for automatically generating micro-course videos.

[0051] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any one of the above-mentioned methods for automatically generating micro-course videos.

[0052] According to the specific embodiments provided in this application, the following technical effects are disclosed: To achieve the goal of easily producing high-quality micro-lessons for all teachers, two goals must be achieved: first, minimizing teacher input, and second, minimizing technical difficulty. Regarding goal one, this application only requires recording a traditional teaching video of a teacher in a regular classroom. Using artificial intelligence technology, the application clones the teacher's voice, constructs a virtual digital human of the teacher, extracts the teacher's lecture notes, and reconstructs the PowerPoint presentation. The cloned teacher's voice, the virtual digital human, the teaching materials, and the PowerPoint presentation are synthesized to create a high-quality micro-lesson video. In subsequent micro-lesson video production, teachers do not need to re-record or record the video; simply providing the lecture PowerPoint presentation and the teacher's lecture notes allows for the synthesis of a new micro-lesson video. This application achieves "one-time acquisition, multiple generation," eliminating the need for teachers to record videos multiple times, even for different teaching content, significantly accelerating the speed and quality of digital teaching resource development. Regarding goal two, this application utilizes a fully automatic generation mode. Users only need to check and confirm the lecture notes, PowerPoint content, and the digital human image in advance. The micro-lesson video generation process requires no human intervention, thus reducing technical difficulty and ensuring efficient micro-lesson video generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 This is a flow chart of a method for automatically generating micro-course videos in one embodiment of the present application;

[0055] Figure 2 A schematic diagram of a system for automatically generating micro-course videos according to an embodiment of the present application;

[0056] Figure 3 This is a schematic diagram of generating new micro-class videos from original micro-class videos based on artificial intelligence technology in this application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0058] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0059] In an exemplary embodiment, Figure 1 As shown, a method for automatically generating micro-course videos is provided, which is implemented based on artificial intelligence technology and includes:

[0060] Step 101: Based on the teacher's teaching video, clone the teacher's voice.

[0061] Use a high-definition camera to record the teacher's teaching process in a standard classroom environment. The teacher only needs to record once and it can be used for subsequent micro-class video generation.

[0062] In an exemplary embodiment, step 101 specifically includes: based on adaptive noise cancellation technology, extracting a denoised speech signal from a teaching video containing noisy speech through an adaptive filter; using an acoustic model to extract speech features from the denoised speech signal; the speech features include Mel-frequency cepstral coefficient features; based on a variational autoencoder, mapping the speech features to a latent space, and reconstructing the speech by a decoder to generate a cloned teacher's voice.

[0063] Furthermore, voice cloning methods include voice extraction and voice cloning.

[0064] Speech extraction includes the following steps:

[0065] 1) Adaptive noise cancellation (ANC).

[0066] Principle: ANC technology uses an adaptive filter to adjust parameters in real time to maximize the signal-to-noise ratio, thereby extracting pure speech from noisy speech.

[0067] Formula: The filter weight update can be expressed as: Among them, w t+1 is the filter weight vector at time t+1, w t is the filter weight vector at time t, μ is the learning rate, e t is the error at the current moment, x t is the input signal and T is its transpose.

[0068] This application uses ANC technology to adaptively process classroom ambient noise and improve the quality of speech extraction.

[0069] 2) Automatic speech recognition (ASR).

[0070] Principle: The ASR system converts speech into text and uses acoustic and language models for recognition.

[0071] Formula: The acoustic model can be expressed as: P(O|A)=2πσ21e-2σ2(OA)2, where O is the observation sequence, A is the audio feature, and σ is the standard deviation.

[0072] Voice cloning involves the following steps:

[0073] 1) Acoustic feature extraction.

[0074] Principle: Extract features such as Mel-Frequency Cepstral Coefficients (MFCC) from speech signals for acoustic model training.

[0075] Formula: MFCC extraction involves Fourier transform, triangular filter bank and discrete cosine transform.

[0076] 2) Variational auto-encoder (VAE).

[0077] Principle: VAE is a generative model that maps speech features to a latent space through an encoder and reconstructs speech through a decoder.

[0078] Formula: The formula for VAE is Vc = Va·CLONE(Vt); where Vc is the cloned voice, Va is the feature representation of the audio, Vt is the teacher's original voice, and CLONE is the clone network.

[0079] The objective function of a VAE is to minimize the reconstruction error and the KL divergence L between the distribution of the latent space and the prior distribution: L = -∑logp(x|z) + KL(q(z|x)||p(z)). The VAE objective function consists of two parts: a. Reconstruction error: -∑logp(x|z), where p(x|z) is the probability of reconstructing the input data x given the latent variable z. This part encourages the model to reconstruct data as similar as possible to the original data. b. KL divergence between the distribution of the latent space and the prior distribution: (q(z|x)||p(z)), where q(z|x) is the posterior distribution of the latent variable z and p(z) is the prior distribution (usually a standard normal distribution). The KL divergence measures the difference between two probability distributions and encourages the model's latent space distribution to be close to the prior distribution.

[0080] 3) Adversarial training.

[0081] Principle: Using adversarial networks for voice cloning, the generator network learns to generate voice features similar to the target voice, while the discriminator network tries to distinguish between real and generated voices.

[0082] Formula: The loss function LD of the discriminator can be expressed as:

[0083] LD=-Eq(z)[logD(G(z))]-Epdata(x)[log(1-D(x))]

[0084] Among them, Eq(z) is the expectation of the latent variables generated by the generator, D(G(z)) is the judgment of the discriminator on the generated data, Epdata(x) is the expectation of the true data distribution pdata, and D(x) is the expectation of the true data distribution pdata.

[0085] This application uses VAE to clone voices, maintaining the diversity and naturalness of the voices.

[0086] Step 102: Extract key points of the teacher's face in the teaching video and construct a virtual digital human of the teacher.

[0087] In an exemplary embodiment, step 102 specifically includes: using a deep learning module to extract the teacher's facial key points and body shape in the teaching video; performing 3D facial reconstruction based on the facial key points to generate a 3D facial model; using UV mapping and image fusion technology to map the teacher's facial texture to the 3D facial model; based on the mapped 3D facial model, using a generative adversarial network to construct a virtual digital human image; animating and driving the virtual digital human image to generate a dynamic virtual digital human; performing a 3D rendering engine on the dynamic virtual digital human to generate a rendered virtual digital human; optimizing the rendered virtual digital human to construct a virtual digital human.

[0088] In an exemplary embodiment, constructing a virtual digital human includes the following steps:

[0089] 1) Facial key point detection.

[0090] Technology: Use deep learning models (such as MTCNN, FaceNet).

[0091] Formula: The model outputs the key point vector KP = [(x1, y1), (x2, y2), ..., (xN, yN)], where (xN, yN) is the coordinate of the facial key point and N is the number of facial key points.

[0092] 2) 3D facial model reconstruction.

[0093] Technology: 3D facial reconstruction using deep learning models and facial key points.

[0094] Formula: The 3D facial model M is obtained by the key point vector KP through the deformation model M=f(KP).

[0095] 3) Facial texture mapping.

[0096] Technology: Use UV mapping and image fusion technology.

[0097] Formula: Texture T is mapped to the 3D facial model M through T = g(M, I), where I is the input 2D facial image.

[0098] 4) Generative Adversarial Network (GAN) Training

[0099] Technology: Using conditional generative adversarial networks (cGAN).

[0100] Formula: Generator G: DI = G(z,c), where z is random noise and c is conditional encoding (such as facial features).

[0101] Discriminator D: DI is classified as real or generated.

[0102] 5) Generation of digital human image.

[0103] Technology: Use the trained GAN to generate digital human images DI.

[0104] Formula: Dt=GAN(Ft) where Dt is the digital teacher image and Ft is the teacher's morphological features.

[0105] 6) Animation and driving.

[0106] Technology: Using Blendshape and Facial Action Coding System (FACS).

[0107] Formula: A=h(DI, P), where h is the animation function and P is the facial expression parameter.

[0108] 7) Rendering and output.

[0109] Technology: Use a 3D rendering engine (such as Three.js, Unity).

[0110] Formula: VF = r(A, L, C), where r is the rendering function, L is the lighting condition, and C is the camera parameter.

[0111] 8) Post-processing and optimization.

[0112] Technology: Use image and video editing software.

[0113] Formula: VF′=p(VF), where p represents post-processing operations such as color correction and denoising.

[0114] This application combines 3D morphology estimation and GAN to generate digital teacher images, improving the realism of the images.

[0115] Step 103: Extract the teacher's lecture notes and generate teaching documents.

[0116] In an exemplary embodiment, step 103 specifically includes: preprocessing and correcting the text of the teacher's lecture notes to generate a corrected text; using text summarization technology to screen key points in the corrected text, and using a dictionary or knowledge base to perform terminology verification on the corrected text; adjusting the language style of the verified text to generate an adjusted text; using a reinforcement learning method to optimize the adjusted text to generate a text optimization strategy; using a sequence generation model to generate a teaching copy according to the text optimization strategy; the sequence generation model includes a recurrent neural network and a Transformer model.

[0117] In an exemplary embodiment, extracting teacher lecture notes and generating teaching copy is a teaching copy optimization process. The purpose of teaching copy optimization is to improve the teaching effect and viewing experience of the generated micro-class video, which usually involves grammar correction, content refinement, terminology accuracy and language style adjustment.

[0118] Teaching copy optimization can be achieved through natural language processing (NLP) technology, including machine learning, deep learning, and reinforcement learning.

[0119] The teaching copy optimization process includes the following steps:

[0120] 1) Text preprocessing

[0121] Purpose: Clean and standardize text data in preparation for subsequent processing.

[0122] Method: Delete extra spaces and punctuation marks, perform word segmentation, lowercase, etc.

[0123] 2) Grammar and spelling correction.

[0124] Purpose: To correct grammatical errors and spelling mistakes in the copy.

[0125] Approach: Use rule-based systems or deep learning models to identify and correct errors.

[0126] 3) Concise content.

[0127] Purpose: To simplify redundant information and highlight key points.

[0128] Methods: Text summarization techniques were used, including extractive and generative summarization.

[0129] 4) Terminology accuracy.

[0130] Purpose: To ensure correct terminology is used.

[0131] Method: Use domain-specific dictionaries or knowledge bases to perform terminology verification.

[0132] 5) Language style adjustment.

[0133] Purpose: To adapt language style to the teaching environment.

[0134] Approach: Adjust the level of formality, complexity, and vocabulary used according to the characteristics of the target audience.

[0135] 6) Reinforcement Learning Optimization

[0136] Objective: To use reinforcement learning to automatically optimize copywriting to improve teaching effectiveness.

[0137] Method: Define a reward function and train the agent to generate more effective teaching copy.

[0138] Example of a formula in reinforcement learning:

[0139] Policy function π: π(a|s, θ), which represents the probability of taking action a in state s, and θ is the policy parameter.

[0140] Reward function R: R(s,a,s′), which represents the reward obtained by taking action a from state s to state s′.

[0141] 7) Generate new copy.

[0142] Purpose: To generate new teaching copy using the optimized strategy.

[0143] Method: Generate text according to an optimization strategy π, which may involve sequence generation models such as recurrent neural networks (RNNs) or Transformers.

[0144] 8) Post-evaluation and iteration.

[0145] Purpose: To evaluate the effectiveness of copywriting optimization and make iterative improvements.

[0146] Method: Use manual evaluation or automatic evaluation metrics (such as BLEU score) to evaluate the generated copy and iteratively optimize based on the feedback.

[0147] This application uses reinforcement learning algorithms to optimize teaching texts and improve teaching effectiveness.

[0148] Step 104: extracting PPT content from the teaching video and reconstructing the PPT page.

[0149] In an exemplary embodiment, step 104 specifically includes: extracting key frames from the teaching video based on the timestamps and content changes of each video frame in the teaching video; the key frames contain PPT content; using a target detection algorithm to locate the display area of the PPT content in the key frames; based on the display area, using image segmentation technology to separate the PPT content and the background; using optical character recognition technology to identify the text separated from the PPT content; using image restoration technology to restore the obscured PPT content and generate restored PPT content; recombining the text in the separated PPT content and the restored PPT content, adjusting the PPT layout and style, and reconstructing the PPT page.

[0150] In an exemplary embodiment, target detection and image segmentation techniques are used to extract PPT and blackboard content from a video; image restoration techniques are used to supplement occluded PPT areas; and style transfer techniques are used to unify the visual style of the PPT.

[0151] Formula: P′=SMOOTH(P,Ω) where P′ is the reconstructed PPT page, P is the extracted PPT content, and Ω is the occluded area.

[0152] The goal of PPT content extraction and reconstruction is to extract the PPT slide content from the teaching video and complete the obscured parts to generate a clear and complete PPT page. This step involves image processing, object detection, image segmentation, and image restoration technologies.

[0153] Reconstructing a PPT page includes the following steps:

[0154] 1) Video frame extraction.

[0155] Purpose: Extract key frames from the video, which contain images for PPT presentation.

[0156] Method: Select keyframes based on their timestamps and content changes.

[0157] 2) PPT area detection.

[0158] Purpose: To locate the display area of the PPT slide in the video.

[0159] Method: Use object detection algorithms, such as YOLO or SSD, to identify and locate PPT slides.

[0160] 3) Frame segmentation and correction.

[0161] Purpose: Separate the PPT content from the background and correct the deformation caused by the shooting angle.

[0162] Method: Apply image segmentation technology, such as GrabCut or DeepLab, to perform region segmentation; use perspective correction algorithm to correct deformation.

[0163] 4) Text and graphic recognition.

[0164] Purpose: To identify text and graphic elements in PPT.

[0165] Method: Use optical character recognition (OCR) technology to recognize text; for graphics, specific recognition algorithms may be required.

[0166] 5) Image restoration.

[0167] Purpose: To supplement PPT content that is blocked by the teacher's body or other objects.

[0168] Method: Use image restoration techniques, such as Inpainting or GAN-based restoration, to fill in the missing areas.

[0169] 6) Content Reconstruction

[0170] Purpose: Reconstruct the extracted and repaired PPT content into complete slide pages.

[0171] Method: Reassemble the segmented, corrected, and repaired PPT content, and adjust the layout and style to match the style of the original PPT.

[0172] 7) Quality enhancement

[0173] Purpose: To improve the image quality of reconstructed PPT pages and ensure that text and graphics are clear and readable.

[0174] This application uses image restoration technology and style transfer technology to reconstruct PPT to ensure the integrity of the content and the unity of the visual effects.

[0175] Step 105: Synthesize the cloned teacher's voice, the virtual digital human, the teaching text, and the PPT page to create a micro-class teaching video.

[0176] In an exemplary embodiment, step 105 specifically includes: using speech analysis technology to extract audio features of the cloned teacher's voice; the audio features include phonemes, volume and pitch; inputting the audio features into a combined model of a convolutional neural network and a recurrent neural network to output a lip movement sequence; the combined model is a lip synchronization model; using Blendshape technology to deform the vertices of the 3D facial model to generate a deformed 3D facial model; applying the lip movement sequence to the deformed 3D facial model to generate a virtual digital human with dynamic lip movements; using a 3D rendering engine to render the virtual digital human with dynamic lip movements to generate a rendered virtual digital human; synchronously synthesizing the rendered virtual digital human, the cloned teacher's voice, the teaching text and the PPT page to create a micro-class teaching video.

[0177] In an exemplary embodiment, the technical details of audio and video synthesis include: using deep learning-driven lip synchronization technology to synthesize the teacher's lecture audio; and synchronizing the digital teacher's lip movements with the audio through 3D facial reenactment technology.

[0178] The core of audio and video synthesis lies in synchronizing the audio signal with the digital human video, specifically ensuring that the digital human's lip movements match the audio speech, thereby creating a natural and realistic lecture video. This process involves extracting audio features, lip synchronization technology, and generating 3D facial animation.

[0179] Formula: Vf = SYNTHESIZE(Lo, Dt, Vc), where Vf is the final micro-lecture video, SYNTHESIZE is the synthesis function responsible for synchronizing audio and video and generating the final video; Lo is the optimized audio (lecture content); Dt is the 3D facial model of the digital teacher; and Vc is the lip movement sequence driven by audio.

[0180] The audio and video synthesis technology includes the following steps:

[0181] 1) Audio feature extraction.

[0182] Purpose: To extract features from lecture audio that are helpful for lip synchronization, such as phonemes, volume, and pitch.

[0183] Method: Use speech analysis techniques such as Mel-Frequency Cepstral Coefficient (MFCC) extraction.

[0184] 2) Lip synchronization model training.

[0185] Objective: Train a deep learning model to map audio features to lip movements.

[0186] Method: A combined model of convolutional neural network (CNN) and recurrent neural network (RNN) is used to input audio features and output lip movement sequences.

[0187] 3) 3D facial model animation.

[0188] Purpose: To animate a 3D facial model based on the output of a lip sync model.

[0189] Method: Use Blendshape technology or deform directly on the vertices of the 3D model.

[0190] 4) Audio-driven lip movements.

[0191] Purpose: To use extracted audio features to drive the lip movements of a digital teacher.

[0192] Method: Audio features are input into the lip synchronization model to obtain lip movement sequences and apply them to the 3D facial model.

[0193] 5) Video rendering.

[0194] Purpose: To render a digital teacher video with dynamic lip movements.

[0195] Method: Use a 3D rendering engine such as Unity or Unreal Engine to render a 3D facial model with animation.

[0196] 6) Audio and video synchronization synthesis.

[0197] Purpose: To synthesize the rendered animation video synchronously with the original audio.

[0198] Method: Use video editing software or programming libraries (such as FFmpeg) to synchronize audio and video.

[0199] 7) Post-processing.

[0200] Purpose: To perform post-processing such as color correction and denoising on the synthesized video to improve the video quality.

[0201] Method: Use image processing technology to improve video quality.

[0202] This application uses deep learning technology to achieve high synchronization between lip shape and audio.

[0203] Repeat the above method of automatically generating micro-course videos until the micro-course video production for the entire course is completed.

[0204] Based on the same inventive concept, the embodiment of the present application further provides a micro-course video automatic generation system, which is implemented based on artificial intelligence technology and includes:

[0205] The cloning module is used to clone the teacher's voice based on the teacher's teaching video;

[0206] A virtual digital human construction module is used to extract key points of the teacher's face in the teaching video and construct a virtual digital human of the teacher;

[0207] Teaching copy generation module, used to extract teacher's lecture notes and generate teaching copy;

[0208] A PPT page reconstruction module, used to extract PPT content from the teaching video and reconstruct the PPT page;

[0209] The micro-class teaching video creation module is used to synthesize the cloned teacher's voice, the virtual digital human, the teaching text and the PPT page to create a micro-class teaching video.

[0210] In an exemplary embodiment, the system structure diagram is as follows Figure 2 As shown in the figure, it consists of a shooting system and a computer system. The shooting system mainly shoots a single video of the teacher teaching, and the computer system is responsible for extracting the sound and image of the video. Then, through artificial intelligence technology, the corresponding sound and image information is processed to finally generate a new micro-class video. The generation effect diagram is shown in the figure. Figure 3 shown.

[0211] Processing unit of a computer system

[0212] Voice Processing Unit: This unit processes and generates the teacher's original voice, and is divided into pre-processing and post-processing. Pre-processing includes voice extraction, voice denoising, voice cloning, and speech-to-text conversion; post-processing includes text-to-speech and voice imitation.

[0213] Image processing unit: The image processing unit is mainly responsible for processing the teacher's image and PPT content. The teacher's image processing includes extracting the teacher's original image from the video, extracting facial features and movement features, and generating a new teacher digital person; PPT content processing includes identifying the PPT content in the original video, removing the occlusion of the teacher's body on the PPT through key frame comparison, and generating complete PPT content.

[0214] Content Processing Unit: Utilizing a large AI model, the unit automatically corrects the text generated by speech through dialogue, prompts, and training, thereby generating a micro-lecture video based on the teacher's lecture content. Furthermore, the unit can generate a micro-lecture script and generate a PowerPoint presentation based on the script, enabling the creation of micro-lecture videos even for content not previously taught or produced by the teacher.

[0215] Video generation unit: Through the information pre-processed by the sound processing unit, image processing unit, and content processing unit, use text to drive sound, sound to drive digital human lectures (especially lip repair and movement repair), combined with the generated PPT pictures, automatically synthesize micro-class videos and generate subtitles corresponding to the time.

[0216] This application fully automates micro-course shooting, information extraction, and video generation, greatly reducing the technical difficulty of the micro-course production process and facilitating the efficient generation of high-quality micro-course videos.

[0217] The production of micro-class videos is no longer based on shooting and editing, but is mainly computer-generated through artificial intelligence technology. Whether it is the teacher's voice, teacher image, or teaching PPT, they are all generated by the system and automatically synthesized into micro-class videos.

[0218] Teachers only need to record a short lecture video to generate high-quality teacher images, high-quality lecture voices, and high-quality PPTs based on this video, and finally generate high-quality micro-class videos. Based on this, they can efficiently generate micro-class videos with other content, eliminating the defects of conventional micro-class videos such as insufficient lighting, noisy sounds, unclear PPTs, and low video quality.

[0219] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for automatically generating micro-course videos is implemented.

[0220] In an exemplary embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0221] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0222] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0223] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0224] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0225] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0226] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0227] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for automatically generating micro-course videos, characterized in that: The method for automatically generating micro-course videos is implemented based on artificial intelligence technology and includes: Clone the teacher's voice based on the teacher's teaching video; Extracting key points of the teacher's face in the teaching video and constructing a virtual digital human of the teacher; Building a virtual digital human involves the following steps: 1) Facial key point detection: Use a deep learning model to output a key point vector KP=[(x1, y1), (x2, y2), ..., (xN, yN)], where (xN, yN) is the coordinate of the facial key point and N is the number of facial key points; 2) 3D facial model reconstruction: The 3D facial model M is constructed by the key point vector KP through the deformation model M= f (KP) obtained; 3) Facial texture mapping: The texture T is mapped to the 3D facial model M via T = g(M, I), where I is the input 2D facial image; 4) Generative Adversarial Network Training: Generator G: DI = G(z,c), where z is random noise and c is the conditional code; Discriminator D: DI is classified as real or generated; 5) Digital human image generation: Dt=GAN(Ft) where Dt is the digital teacher image and Ft is the teacher’s morphological features; 6) Animation and driving; 7) Rendering and output; 8) Post-processing and optimization; Extract teacher's lecture notes and generate teaching copy; Extracting PPT content from the teaching video and reconstructing the PPT page specifically includes: Extracting key frames from the teaching video according to timestamps and content changes of each video frame in the teaching video; the key frames contain PPT content; Using a target detection algorithm to locate the display area of the PPT content in the key frame; Based on the display area, the image segmentation technology is used to separate the PPT content and the background; Use optical character recognition technology to identify and separate text from PPT content; Use image restoration technology to repair obscured PPT content and generate restored PPT content; Recombine the text in the separated PPT content and the repaired PPT content, adjust the PPT layout and style, and reconstruct the PPT page; Synthesizing the cloned teacher's voice, the virtual digital human, the teaching text, and the PPT page to create a micro-class teaching video specifically includes: Using speech analysis technology, extracting audio features of the cloned teacher's voice; the audio features include phonemes, volume, and pitch; Inputting the audio features into a combined model of a convolutional neural network and a recurrent neural network to output a lip movement sequence; the combined model is a lip synchronization model; Utilize Blendshape technology to deform the vertices of the 3D facial model to generate a deformed 3D facial model; Applying the lip movement sequence to the deformed 3D facial model to generate a virtual digital human with dynamic lip movements; Rendering the virtual digital human with dynamic lip movements using a 3D rendering engine to generate a rendered virtual digital human; The rendered virtual digital human, the cloned teacher's voice, the teaching text and the PPT page are synchronously synthesized to create a micro-class teaching video.

2. The method for automatically generating micro-course videos according to claim 1, wherein: The method of cloning the teacher's voice based on the teacher's teaching video specifically includes: Based on adaptive noise cancellation technology, the denoised speech signal is extracted from the teaching video containing noisy speech through an adaptive filter; Extracting speech features from the denoised speech signal using an acoustic model; the speech features include Mel-frequency cepstral coefficient features; Based on a variational autoencoder, the speech features are mapped to a latent space, and the decoder reconstructs the speech to generate a cloned teacher voice.

3. The method for automatically generating micro-course videos according to claim 1, wherein: The extracting of key points of the teacher's face in the teaching video and constructing a virtual digital human of the teacher specifically includes: A deep learning module is used to extract the teacher's facial key points and body shape in the teaching video; Performing 3D facial reconstruction based on the facial key points to generate a 3D facial model; Mapping the teacher's facial texture onto the 3D facial model using UV mapping and image fusion technology; Based on the mapped 3D facial model, a virtual digital human image is constructed using a generative adversarial network. Animating and driving the virtual digital human image to generate a dynamic virtual digital human; Performing a 3D rendering engine on the dynamic virtual digital human to generate a rendered virtual digital human; The rendered virtual digital human is optimized to construct a virtual digital human.

4. The method for automatically generating micro-course videos according to claim 1, wherein: Extract teacher's lecture notes and generate teaching documents, including: Preprocessing and correcting the text of the teacher's lecture notes to generate a corrected text; Using text summarization technology to screen key points in the corrected text, and using a dictionary or knowledge base to perform terminology verification on the corrected text; Adjust the language style of the verified text to generate an adjusted text; Optimizing the adjusted text using a reinforcement learning method to generate a text optimization strategy; A sequence generation model is used to generate teaching texts according to the text optimization strategy; the sequence generation model includes a recurrent neural network and a Transformer model.

5. A micro-course video automatic generation system, characterized in that: The micro-course video automatic generation system is applied to the micro-course video automatic generation method according to any one of claims 1 to 4. The micro-course video automatic generation system is implemented based on artificial intelligence technology and includes: The cloning module is used to clone the teacher's voice based on the teacher's teaching video; A virtual digital human construction module is used to extract key points of the teacher's face in the teaching video and construct a virtual digital human of the teacher; Building a virtual digital human involves the following steps: 1) Facial key point detection: Use a deep learning model to output a key point vector KP=[(x1, y1), (x2, y2), ..., (xN, yN)], where (xN, yN) is the coordinate of the facial key point and N is the number of facial key points; 2) 3D facial model reconstruction: The 3D facial model M is constructed by the key point vector KP through the deformation model M= f (KP) obtained; 3) Facial texture mapping: The texture T is mapped to the 3D facial model M via T = g(M, I), where I is the input 2D facial image; 4) Generative Adversarial Network Training: Generator G: DI = G(z,c), where z is random noise and c is the conditional code; Discriminator D: DI is classified as real or generated; 5) Digital human image generation: Dt=GAN(Ft) where Dt is the digital teacher image and Ft is the teacher’s morphological features; 6) Animation and driving; 7) Rendering and output; 8) Post-processing and optimization; Teaching copy generation module, used to extract teacher's lecture notes and generate teaching copy; A PPT page reconstruction module, used to extract PPT content from the teaching video and reconstruct the PPT page; The micro-class teaching video creation module is used to synthesize the cloned teacher's voice, the virtual digital human, the teaching text and the PPT page to create a micro-class teaching video.

6. A computer device comprising: The memory and the processor are characterized in that the computer program is stored in the memory and can be run on the processor, and the processor executes the computer program to implement the steps of the micro-course video automatic generation method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for automatically generating micro-course videos according to any one of claims 1 to 4 are implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for automatically generating micro-course videos according to any one of claims 1 to 4 are implemented.

Images