Teaching video generation method and device, equipment, storage medium and program product

Abstract

The invention discloses a teaching video generation method and device, equipment, a storage medium and a program product, and relates to the technical field of video processing, and the method comprises the steps: analyzing an original teaching video, and obtaining a teacher voice content text and teaching courseware; based on the text style vector of the target teacher, performing text reconstruction processing on the teacher voice content text to obtain a target teacher lecture text; generating a target teacher style sound based on the text style vector and the voice style vector of the target teacher in combination with the lecture text of the target teacher; for each page of teaching courseware, based on the text style vector, performing text reconstruction on the courseware content text of the teaching courseware to obtain a target teaching courseware; and generating a teaching video based on the target teaching courseware, the target teacher style sound and the virtual image of the target teacher. According to the method, the logic integrity of course content and the accuracy of knowledge points are ensured, and the generated video fits the teaching style of a target teacher in the aspects of voice rhythm, language expression and courseware style.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to methods, apparatus, devices, storage media, and program products for generating instructional videos. Background Technology

[0002] With the rapid development of artificial intelligence technology, the education sector is gradually introducing new technologies such as digital human lecturers, speech synthesis, and automatic course content generation to improve teaching efficiency and course reuse rates. Through virtual teacher-driven course delivery, AI-synthesized speech-assisted instruction, and multilingual course video generation, educational platforms can quickly update course content and adapt it to multiple scenarios. These technologies have already been applied in online university courses, digital human vendor products, and AIGC content creation tools, and are gradually forming initial industrialization paths.

[0003] Among related technologies, the mainstream solutions mainly fall into four categories: first, inputting lecture notes and courseware to directly generate virtual teacher course videos; second, using script-driven video production platforms to generate courses through pre-made lecturer images and voices; third, using face-swapping and speech synthesis technologies to replace the lecturer in existing course videos with a new teacher; and fourth, generating multilingual course videos through speech recognition and translation. While these technologies have made some progress in automatic course generation and multilingual adaptation, they largely remain at the level of simple image and sound replacement, neglecting the transfer of teacher language style and expression; they cannot personalize the rhythm, logic, and expression according to the target teacher's style; they lack adaptive teaching structure and style quantitative control mechanisms; and they struggle to meet the needs of universities or platforms for large-scale lecturer replacement and style consistency maintenance of existing course videos. Therefore, when facing the more complex application scenario of "replacing the lecturer in existing high-quality course videos," existing technologies still have significant shortcomings. Summary of the Invention

[0004] The main purpose of this application is to provide a method, apparatus, device, storage medium, and computer product for generating teaching videos, aiming to solve the technical problem that related technologies have insufficient ability to transfer the consistency of the target teacher's language style and expression rhythm in the generation of teaching videos.

[0005] To achieve the above objectives, this application proposes a method for generating instructional videos, the method comprising: Analyze the original teaching videos to obtain the text of the teacher's audio content and teaching materials; Based on the text style vector of the target teacher, the text of the teacher's speech content is reconstructed to obtain the text of the target teacher's lecture notes; Based on the target teacher's text style vector and speech style vector, combined with the target teacher's lecture text, the target teacher's style voice is generated; the text style vector includes speech rate features, pause features, sentence structure features and language style features, and the speech style vector includes fundamental frequency features; For each page of teaching courseware, the text content of the teaching courseware is reconstructed based on text style vectors to obtain the target teaching courseware; Teaching videos are generated based on the target teaching materials, the target teacher's style and voice, and the target teacher's virtual image.

[0006] In one embodiment, the steps of parsing the original teaching video to obtain the teacher's voice content text and teaching courseware include: Speech recognition processing is performed on the teacher's voice in the original teaching video to obtain the text of the teacher's voice content; From the original teaching video, select the video frames containing the teaching materials, sort the video frames based on the page number information of the teaching materials, and obtain the teaching materials. For each page of the teaching slides, extract the text content to obtain the slide content text.

[0007] In one embodiment, before obtaining the target teacher's lecture text by reconstructing the teacher's speech content based on the target teacher's text style vector, the following steps are also included: The target teacher's speech sample is divided into frames, and the fundamental frequency of each frame is calculated using a fundamental frequency estimation algorithm to obtain the fundamental frequency sequence of the target teacher's speech sample; Calculate the fundamental frequency mean and fundamental frequency standard deviation of the fundamental frequency sequence to obtain the fundamental frequency characteristics; Based on the total number of words in the text content of the target teacher's speech sample and the total duration of the speech sample, the speech rate of the target teacher is calculated to obtain speech rate features. Silence segments are detected in the target teacher's speech samples. The pause ratio is determined based on the ratio of the duration of silence segments in the target teacher's speech samples to the total duration of the speech samples, and pause features are obtained. The text content of the target teacher's voice sample is used to identify the length of sentences, determine the ratio of long and short sentences in the text content, and obtain sentence features; Based on the BERT language model, features are extracted from the text content to obtain language style features.

[0008] In one embodiment, the step of reconstructing the teacher's speech text based on the target teacher's text style vector to obtain the target teacher's lecture script text includes: The teacher's speech content text and text style vector are combined with the lecture rewriting prompts and input into the text rewriting model to obtain multiple candidate lecture texts; among them, the lecture rewriting prompts include content preservation constraints and style fit constraints; Calculate the style fit score and content retention score between each candidate lecture text and the teacher's audio content text; Based on style fit scores and content retention scores, the target teacher's lecture text was selected from each candidate lecture text.

[0009] In one embodiment, the step of generating a target teacher's style voice based on the target teacher's text style vector and speech style vector, combined with the target teacher's lecture text, includes: Based on fundamental frequency features, the sentiment tags of the target teacher's lecture text are determined; The target teacher's lecture text, combined with prompt words, is input into a large language model, which outputs the potential pause points in the target teacher's lecture text. Add pause tags at potential pause points in the target teacher's lecture text; the pause tags contain pause duration information determined based on pause features; The target teacher's lecture text, which includes emotion and pause tags, is input into the speech synthesis model to generate a voice in the target teacher's style; the speech duration parameter of the speech synthesis model is determined based on speech rate features.

[0010] In one embodiment, for each page of teaching courseware, the steps of reconstructing the text content of the teaching courseware based on text style vectors to obtain the target teaching courseware include: For each page of teaching materials, the content text and language style features of the materials, along with the prompts for rewriting the materials, are input into the text rewriting model to obtain the rewritten text of the materials. Rewrite the text of all teaching materials and arrange them according to page numbers to obtain the target teaching materials.

[0011] Secondly, to achieve the above objectives, this application further provides an instructional video generation device, the device comprising: The parsing module is used to parse the original teaching video to obtain the text of the teacher's voice content and teaching courseware; The lecture script reconstruction module is used to reconstruct the text of the teacher's speech content based on the target teacher's text style vector to obtain the target teacher's lecture script text. The sound generation module is used to generate the target teacher's style voice based on the target teacher's text style vector and speech style vector, combined with the target teacher's lecture text; the text style vector includes speech rate features, pause features, sentence structure features and language style features, and the speech style vector includes fundamental frequency features; The courseware reconstruction module is used to reconstruct the text of each page of teaching courseware based on text style vectors to obtain the target teaching courseware. The video generation module is used to generate teaching videos based on the target teaching materials, the target teacher's style and voice, and the target teacher's virtual image.

[0012] Thirdly, to achieve the above objectives, this application further provides an instructional video generation device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the instructional video generation method described above.

[0013] Fourthly, to achieve the above objectives, this application further provides a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the above-described teaching video generation method.

[0014] Fifthly, to achieve the above objectives, this application further provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the above-described instructional video generation method.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: This application analyzes the audio and courseware content of the original teaching videos, and combines the target teacher's text style vector and audio style vector to achieve personalized style transfer at three levels: script generation, courseware content reconstruction, and sound synthesis. This ensures that the generated videos match the target teacher's teaching style in terms of audio rhythm, language expression, and courseware style, while maintaining the logical integrity of the course content and the accuracy of the knowledge points. Ultimately, this results in highly immersive and realistic personalized teaching videos, significantly improving course reuse efficiency and learning experience. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the teaching video generation method in the embodiments of this application.

[0019] Figure 2 This is a schematic diagram of the module connections for the instructional video generation device.

[0020] Figure 3 This is a schematic diagram of the equipment used to generate instructional videos.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The main solution of this application embodiment is as follows: First, the original teaching video is parsed to extract the teacher's speech content text and teaching courseware, and style features such as speech rate, pauses, sentence structure, language style, and fundamental frequency are extracted from the speech. Then, the original lecture text and courseware content are reconstructed by combining the target teacher's text style vector and speech style vector to obtain a lecture text and courseware that conform to the target teacher's expression style. On this basis, through sentiment annotation, potential pause prediction, and duration control, the target teacher's lecture text is input into a speech synthesis model to generate a voice with the target teacher's style. Finally, the target teacher's style voice, the reconstructed courseware, and the target teacher's virtual image are combined to synthesize a complete teaching video, thereby achieving the reproduction of the teaching video in the style of the target teacher while maintaining the course knowledge content and structure.

[0025] Specifically, this application provides a method for generating instructional videos, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the teaching video generation method of this application.

[0026] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or teaching video generation device capable of performing the above functions. The following description uses a teaching video generation device as an example to illustrate this embodiment and the subsequent embodiments.

[0027] In this embodiment, the teaching video generation method includes steps S10 to S50: Step S10: Analyze the original teaching video to obtain the teacher's voice content text and teaching courseware.

[0028] It should be noted that the original teaching video, as a real recorded course video, serves as a reference for generating the target teaching video and as the input source for the solution. It is used to extract the teacher's lecture notes and courseware to support subsequent style transfer and video reconstruction.

[0029] The teacher's audio content is the text obtained by speech recognition and processing of the teacher's voice in the original teaching video. The teaching courseware is the teaching courseware that appears in the original teaching video, such as the PPT courseware shown by the teacher during the lecture.

[0030] In a feasible real-time manner, step S10 includes steps A10 to A30: Step A10: Perform speech recognition processing on the teacher's voice in the original teaching video to obtain the text of the teacher's voice content.

[0031] Step A20: Select courseware video frames containing teaching materials from each video frame of the original teaching video, and sort each courseware video frame based on the courseware page number information contained in the courseware video frame to obtain the teaching materials.

[0032] Step A30: For each page of the teaching materials, extract the text content of the teaching materials to obtain the text content of the teaching materials.

[0033] Table 1 Courseware and Audio Schedule In one example, the original teaching video can be parsed by using the Whisper model to automatically recognize the audio portion of the video and obtain the teacher's spoken text. OpenCV is then used to identify video frames containing PPT slides within the original teaching video. These PPT slides are then extracted sequentially into individual PPT images, and the PaddleOCR model is used to extract the content from these images and convert it into the courseware text.

[0034] Subsequently, a dynamic time warping algorithm is used to align the teaching audio content text with the PPT slides for easier subsequent processing. Specifically, the original teaching video's video frame sequence is V={V_1,V_2,....,V_n}. From this sequence, the video frame sequence containing the PPT slides is identified and converted into a PPT image sequence P={p_1,p_2,....,p_m}. The teacher's audio content text is then divided into segments, forming a text sequence T={t_1,t_2,....,t_k}.

[0035] Determine the start and end times of the PPT slides appearing in the original teaching video, as well as the start and end times of the audio corresponding to each paragraph in the teacher's audio content text in the original teaching video, as shown in Table 1, the slides and audio timetable.

[0036] The timestamp intersection of the start-end timestamp of the PPT slide page and the timestamp of the teacher's audio content text is determined. The correspondence shown in Table 2, the timestamp alignment table, is established based on whether the time periods overlap, thus completing the alignment process between the teaching audio content text and the PPT slide page.

[0037] Table 2 Timestamp Alignment Table Step S20: Based on the target teacher's text style vector, perform text reconstruction processing on the teacher's speech content text to obtain the target teacher's lecture text.

[0038] Step S30: Based on the target teacher's text style vector and speech style vector, and combined with the target teacher's lecture text, generate the target teacher's style voice.

[0039] Step S40: For each page of teaching courseware, based on text style vectors, the text content of the teaching courseware is reconstructed to obtain the target teaching courseware.

[0040] The main purpose of introducing style vectors in this embodiment is to transfer the teacher's voice content text in the original teaching video to the target teacher's script text in the style of the target teacher based on the text style vector, and then combine the voice style vector to convert the target teacher's script text into the target teacher's audio.

[0041] It should be noted that text style vectors include speech rate features, pause features, sentence structure features, and language style features, which are used to provide a reference for text processing, while speech style vectors include fundamental frequency features, which are used to provide a reference for speech signal processing.

[0042] In one feasible implementation, the determination of text style vectors and speech style vectors may include steps B10 to B60: Step B10: The target teacher's speech sample is divided into frames, and the fundamental frequency of each frame is calculated using a fundamental frequency estimation algorithm to obtain the fundamental frequency sequence of the target teacher's speech sample.

[0043] Step B20: Calculate the fundamental frequency mean and fundamental frequency standard deviation of the fundamental frequency sequence to obtain the fundamental frequency characteristics.

[0044] Step B30: Based on the total number of words in the text content of the target teacher's speech sample and the total duration of the speech in the target teacher's speech sample, calculate the speech rate of the target teacher to obtain speech rate features.

[0045] Step B40: Detect silence segments in the target teacher's speech samples. Based on the ratio of the silence segment duration to the total speech duration of the target teacher's speech samples, determine the pause ratio and obtain pause features.

[0046] Step B50 identifies the length of sentences in the text content of the target teacher's voice sample, determines the ratio of long and short sentences in the text content, and obtains sentence features.

[0047] Step B60: Extract features from the text content based on the BERT language model to obtain language style features.

[0048] Specifically, this implementation method mainly models the style of the target teacher, performs multi-dimensional feature analysis on the historical speech samples of the target teacher, and extracts style vectors that can comprehensively represent the teacher's personal expression style, including voice-level features and text-level features, providing a data foundation for subsequent lecture reconstruction, speech synthesis and style transfer.

[0049] For example, historical speech data of the target teacher is collected as speech samples of the target teacher, and the unique expression style is extracted from the speech samples of the target teacher to construct a style vector S. teacher In this example, we use newteacher.wav as the target teacher's voice sample and extract its corresponding text content, which is "Hello everyone, today we will learn the first chapter. First, we will look at the overall structure of the course...". The main methods for extracting the target teacher's style vector include: Fundamental frequency feature extraction: The PYIN algorithm is used to extract the fundamental frequency of each frame of the target teacher's speech sample, resulting in a fundamental frequency sequence F_0(t) = [123Hz, 120Hz, 118Hz, 125Hz, …]. The mean fundamental frequency of this sequence F_0(t) is then calculated. μF 0 and fundamental frequency standard deviation σF 0. The mean fundamental frequency is used as the average fundamental frequency characteristic of the target teachers, and the standard deviation of the fundamental frequency is used as the fundamental frequency volatility characteristic of the target teachers. In this example, the average fundamental frequency is 122.5Hz, and the fundamental frequency volatility is 3.4.

[0050] Speech rate feature extraction: In this example, the text content of the target teacher's speech sample contains a total of 41 Chinese characters, approximately 19 words, and the total duration of the target teacher's speech sample is 11.7 seconds (i.e., 0.195 minutes). The calculated speech rate feature WMP is as follows: Pause feature extraction involves detecting silence segments in the target teacher's speech sample and calculating the duration of each silence segment. In this example, two significant pauses were found: the first pause lasted 4.6–5.0 seconds (0.4 seconds); the second pause lasted 8.1–8.5 seconds (0.4 seconds). Therefore, the duration of the silence segment is 0.8 seconds. The total duration of the target teacher's speech sample is 11.7 seconds, and the pause ratio ρ is: Sentence structure feature extraction involves identifying the length of sentences in the text content of the target teacher's voice sample, determining the ratio of long to short sentences in the text content, and obtaining sentence structure features. In this example, the ratio of long sentences to short sentences is 0.7, indicating a preference for shorter sentences.

[0051] The steps for extracting language style features can be as follows: the text content of the target teacher's speech sample is represented by the BERT language model embedder. The text content is input into the BERT language model embedder to obtain the sentence vector L_s. BERT will map a sentence to a high-dimensional dense vector space, usually 512-dimensional, 768-dimensional, etc. In this embodiment, the first 512 dimensions are extracted as the language style feature representation.

[0052] Ultimately, the style vector of the target teacher is obtained. S teacher = [Average fundamental frequency, fundamental frequency variability, speech rate characteristics, sentence structure characteristics, language style characteristics] = [122.5, 2.4, 97.44, 0.06, 0.7, L] _S ].

[0053] After obtaining the style vector of the target teacher, the teacher's speech content text rewriting, target teacher style voice synthesis, and teaching courseware reconstruction are performed based on the style vector. Specifically, step S20 includes steps C10 to C30: Step C10 involves inputting the teacher's speech content text and text style vector, along with the script rewriting prompts, into the text rewriting model to obtain multiple candidate script texts. The script rewriting prompts include content preservation constraints and style fit constraints.

[0054] Step C20: Calculate the style fit score and content retention score between each candidate lecture text and the teacher's audio content text.

[0055] Step C30: Based on the style fit score and content retention score, the target teacher's lecture text is determined from each candidate lecture text.

[0056] In this implementation, the text rewriting model can be a general generative large language model. The core of constructing the target teacher's lecture text is to use a large language model for content-preserving style transfer, and to introduce control variables to guide style factors such as sentence complexity, tone, and keyword selection.

[0057] For example, based on the previously extracted style vector of the target teacher S teacher For the teacher's voice content text T srcRewrite to obtain the target teacher's lecture text. T style The method can be expressed as: T style =StyleRewrite(T src, S teacher ) in, StyleRewrite() This can be understood as a text rewriting model, whose input is the style vector of the target teacher. S teacher and teacher's voice content text T src The output is the target teacher's lecture text. T style .

[0058] In the specific rewriting process, content preservation constraints and style similarity constraints can be used to ensure the quality of the rewritten teacher's lecture text.

[0059] Specifically, the content retention constraint is mainly used to ensure that the transferred lecture notes maintain consistency with the original text in terms of knowledge structure, teaching logic, and core points. It can be expressed as: ContentSim(T style, T src )≥δ c in, For teacher's voice content text, T style For target teacher lecture text, The content retention score, which can be calculated using a text similarity function, measures whether the transferred target teacher's lecture text faithfully matches the teacher's spoken content. Specifically, a vector model is first used to... and T style Convert to a high-dimensional vector, then use a text similarity calculation function to calculate. and T style The cosine angle is used as the content retention score, and the content retention score can be calculated as follows: Its value ranges from 0 to 1, and the closer it is to 1, the more consistent the content is.

[0060] δ c This is the constraint threshold for content retention; a higher threshold indicates stricter requirements. Preferably, it is set to 0.85 in this example.

[0061] Style fit constraint, mainly used to ensure that the style of the transferred lecture notes is consistent with the teaching style of the target teacher, can be expressed as: in, For the style vector of the target teacher, T style For target teacher lecture text, StyleSim ( T style , S teacher This is a style similarity calculation function used to measure whether the style of the transferred text matches the style of the new teacher. It's worth noting that the style similarity calculation uses [function name - likely a specific parameter or parameter]. The style vectors that can be used to process text, also known as text style vectors. S gen , S gen = [Speech rate features, pause features, sentence structure features, language style features], specifically, firstly, a vector model is used to... S gen and T style Convert to a high-dimensional vector and then calculate. S gen and T style The cosine angle is used to obtain the style fit score. δ s This is a constraint threshold for style fit. Preferably, δ s It can be set to 0.8.

[0062] Understandably, introducing content retention constraints during text rewriting, and using constraint thresholds to determine the content consistency of the rewritten target teacher's lecture text, ensures that the teaching structure and core points are not weakened while style transfer is achieved, thereby maintaining the stability and consistency of teaching quality. Introducing style fit constraints ensures that the output text closely resembles the target teacher's expression habits in terms of sentence structure, colloquialism, transition logic, and keyword style.

[0063] In this example, the specific steps for processing the teacher's spoken content text are as follows: By using explicit Prompt instructions that encompass constraints on content retention and style fit, multiple candidate speech texts [T1, T2, ..., T] are generated through a large model. n ].

[0064] For each candidate speech text, calculate the content similarity score and style fit score.

[0065] The similarity score and style fit score are combined to obtain a comprehensive evaluation score. The candidate lecture text with the highest comprehensive evaluation score is selected as the target teacher's lecture text.

[0066] The above steps can be expressed in the following way: in, T style This refers to a teacher's lecture notes document. λ c and λ s This represents weight information. In this example, it is set... λ c =0.7, λ s =0.3 indicates a greater emphasis on content preservation.

[0067] Step S30 includes steps D10 to D40: Step D10: Based on fundamental frequency features, determine the sentiment labels of the target teacher's lecture text.

[0068] Step D20: Input the target teacher's lecture text along with the prompt words into the large language model and output the potential pause points of the target teacher's lecture text.

[0069] Step D30: Add pause tags at potential pause points in the target teacher's lecture text; the pause tags contain pause duration information determined based on pause features.

[0070] Step D40: Input the target teacher's lecture text, which includes emotion tags and pause tags, into the speech synthesis model to generate the target teacher's style voice.

[0071] The speech duration parameter of the speech synthesis model is determined based on speech rate features.

[0072] Specifically, the speech synthesis model can be a large open-source speech generation model, such as CosyVoice 2.0, which first starts from the style vector S teacher The features that meet the input control requirements of the large speech generation model are identified. In this example, these features include fundamental frequency features (average fundamental frequency and fundamental frequency variability), pause features, and speech rate features. By analyzing the average fundamental frequency and fundamental frequency variability, the emotional label classification of the generated teacher's speech can be selected. The speech rate feature determines the duration of the speech output by the model, and the pause feature determines the insertion position of the pause label.

[0073] For example, the emotional labels of the TTS speech synthesis model are first determined based on the average fundamental frequency and its fluctuation. The average fundamental frequency is the average value of the fundamental frequency of each frame in the speech, measured in Hz. The natural speaking pitch of men and women differs significantly, thus requiring gender-specific thresholds. In this example, the target teacher is detected as male, with an average fundamental frequency of 122.5 Hz, which basically falls within the lower bound of the average fundamental frequency range for men (120–150 Hz) (see Table 3, Pitch Evaluation Table), and can be considered a slightly lower pitch style.

[0074] Table 3 Pitch Evaluation Table Fundamental frequency variability reflects the range of pitch variation. Generally, speaking styles with pronounced intonation tend to have higher fundamental frequency variability. Table 4 shows the general empirical thresholds for style evaluation (applicable to both men and women).

[0075] Table 4 Style Evaluation Table In this example, the fundamental frequency fluctuation value is 3.4Hz, which falls exactly in the "normal fluctuation" range.

[0076] By comprehensively evaluating the average fundamental frequency and fundamental frequency fluctuation value of the target teachers, emotion labels (which can be "neutral" or "serious") that meet the requirements of the CosyVoice2.0 speech synthesis model are obtained, so that the mapped emotion labels can be included during speech synthesis.

[0077] Subsequently, the speech duration parameter of the speech synthesis model is determined. Specifically, in this example, the target teacher's speech rate feature WMP is 97.44, while the model's default is 110. This means the default speech duration parameter needs to be reduced by a factor of 1.13 to make the output speech signal closer to the target teacher's speech rate style. When inputting the target teacher's lecture text into the speech synthesis model, pause tags need to be added to the text. In this example, the pause tag positions are determined by inputting the target teacher's lecture text into the large language model, combining prompt words, and based on the text's syntactic structure, punctuation, and keywords, causing the large language model to output a set of pause points in the target teacher's lecture text. Pause tags are then added to these pause points. <pause>By dynamically adjusting the insertion method, the duration of each pause (breaktime) is determined (each breaktime is in the range of 150ms to 500ms) so that the pause ratio of the target teacher's lecture text is the same as the pause feature ρ (6.8%).

[0078] Specifically, in this example, the target teacher's pause rate ρ is 6.8%, which means that there is an average of about 0.068 seconds of pause per second of speech.

[0079] First, determine the initial pause duration (breaktime) for each pause point based on the character type preceding each pause point in the pause point set.

[0080] Determine the initial pause ratio based on the initial pause duration.

[0081] Calculate the difference between the initial pause ratio and the target teacher's pause ratio.

[0082] If the initial pause ratio is less than the target teacher's pause ratio, the pause duration at each pause point will be dynamically increased based on the character type preceding each pause point.

[0083] If the initial pause ratio is greater than the target teacher's pause ratio, the pause duration at each pause point will be dynamically reduced based on the character type preceding each pause point.

[0084] Specifically, pauses with the character type of period or semicolon are marked as mandatory pauses, while pauses with commas or enumeration commas are marked as weak pauses. If a pause is preceded by a pre-defined keyword (such as "firstly," "summarize," or "therefore"), it is marked as an emphasis pause. If the initial pause ratio is less than the target teacher's pause ratio, the pause duration is extended according to the priority order of weak pauses > emphasis pauses > mandatory pauses, ensuring that the difference between the initial and target teacher's pause ratios does not exceed a preset range (±0.005 in this example). Conversely, if the initial pause ratio is greater than the target teacher's pause ratio, the pause duration is shortened according to the priority order of weak pauses > emphasis pauses > mandatory pauses, ensuring that the difference between the initial and target teacher's pause ratios does not exceed a preset range (±0.005 in this example).

[0085] Finally, the target teacher's lecture text containing emotion and pause labels is input into the speech synthesis model, so that the speech synthesis model has a high degree of matching with the target teacher's speech style and rhythm.

[0086] Step S40 includes steps E10 to E20: Step E10: For each page of teaching materials, input the text content and language style features of the materials, along with the prompts for rewriting the materials, into the text rewriting model to obtain the rewritten text of the materials.

[0087] Step E20: Rewrite the text of all teaching courseware and arrange it according to page numbers to obtain the target teaching courseware.

[0088] Specifically, firstly, a text rewriting model (which can be a general generative large language model) is used in conjunction with cue words and the target teacher's language style characteristics to stylize and rewrite the text content of each PPT slide, resulting in PPT slide text that matches the target teacher's style. Then, by combining the PPT slide page numbers obtained from parsing the original teaching video and the time each page appears in the video, a new course PPT is generated for use in compositing new videos.

[0089] For example, the Prompt instructions are set according to the style of the target teacher and input into the text rewriting model, such as: Please rewrite the following PPT slides with the original knowledge points unchanged into a more organized and formal PPT presentation that conforms to teaching standards.

[0090] Original PPT content: Chapter 1 Introduction Course objectives: To master the basic framework, understand the background, and introduce the methods.

[0091] The content includes: history, methodology, and application cases. Current teacher preference style: Emphasis on clear parallel structure I like to use leading phrases. Rewritten output: Chapter 1: Course Introduction Objective of this chapter: Understanding the overall structure of the course Mastering core research methods Understanding the relevant historical background and typical applications Main contents of this chapter: 1. History of the Discipline's Development 2. Methodological Foundations 3. Case Study Analysis The rewritten content will be automatically formatted into the new course template PPT, creating a new PPT presentation.

[0092] The current teacher preference style in the example is obtained by cluster analysis of 512-dimensional language style features.

[0093] Step S50: Generate a teaching video based on the target teaching courseware, the target teacher's style and voice, and the target teacher's virtual image.

[0094] It should be noted that during the generation of the teaching video, the lip movements of the virtual avatar need to be confirmed in conjunction with the target teacher's voice style.

[0095] Specifically, the target teacher's shot exemption generated by the speech synthesis model is used as input to drive the teacher's digital avatar to generate a lip-synced video with an alpha channel. Then, the target teacher's virtual avatar, the rewritten PPT slides, and the target teacher's voice are input into the rendering and synthesis engine and merged into the final output course video.

[0096] In summary, this embodiment introduces the target teacher's text style vector and speech style vector to perform multi-dimensional style transfer and reconstruction of the original teaching video's script, speech, and courseware content. This achieves the transformation of the course video into a teaching video that matches the target teacher's personalized style without altering the teaching knowledge structure and core content. This ensures the integrity of the teaching logic while enhancing the personalized expression of the content. This method not only makes the generated video closely resemble the target teacher's natural teaching habits in terms of speech rate, pauses, sentence structure, and language style, but also maintains the logical coherence and consistency of the courseware presentation. Ultimately, it significantly improves the acceptability and immersive experience of the teaching video, facilitating the efficient reuse and differentiated customization of teaching resources.

[0097] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the teaching video generation method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0098] This application also provides a teaching video generation device, please refer to... Figure 2 The teaching video generation device includes: The parsing module 10 is used to parse the original teaching video to obtain the teacher's voice content text and teaching courseware.

[0099] The lecture manuscript reconstruction module 20 is used to reconstruct the text of the teacher's speech content based on the target teacher's text style vector to obtain the target teacher's lecture manuscript text.

[0100] The sound generation module 30 is used to generate the target teacher's style voice based on the target teacher's text style vector and speech style vector, combined with the target teacher's lecture text; the text style vector includes speech rate features, pause features, sentence structure features and language style features, and the speech style vector includes fundamental frequency features.

[0101] The courseware reconstruction module 40 is used to reconstruct the text of the courseware content based on text style vectors for each page of the teaching courseware to obtain the target teaching courseware.

[0102] The video generation module 50 is used to generate teaching videos based on the target teaching courseware, the target teacher's style and voice, and the target teacher's virtual image.

[0103] The teaching video generation device provided in this application, employing the teaching video generation method in the above embodiments, can solve the technical problem of insufficient consistency transferability of target teacher's language style and expression rhythm in related technologies. Compared with related technologies, the beneficial effects of the teaching video generation device provided in this application are the same as those of the teaching video generation method provided in the above embodiments, and other technical features in the teaching video generation device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0104] This application provides an instructional video generation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the instructional video generation method in the above embodiments.

[0105] The following is for reference. Figure 3 The diagram illustrates a structural schematic of a teaching video generation device suitable for implementing embodiments of this application. The teaching video generation device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 3 The instructional video generation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0106] like Figure 3 As shown, the instructional video generation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 (ROM) or a program loaded from a storage device 1003 into a random access memory 1004 (RAM). The random access memory 1004 also stores various programs and data required for the operation of the instructional video generation device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 (I / O interface) is also connected to the bus 1005. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the instructional video generation device to communicate wirelessly or wiredly with other devices to exchange data. Although instructional video generation devices with various systems are shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0107] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0108] The teaching video generation device provided in this application, employing the teaching video generation method described in the above embodiments, can solve the technical problem of insufficient consistency transferability of target teachers' language style and expression rhythm in related technologies. Compared with related technologies, the beneficial effects of the teaching video generation device provided in this application are the same as those of the teaching video generation method provided in the above embodiments, and other technical features in this teaching video generation device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0109] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0110] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0111] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the teaching video generation method in the above embodiments.

[0112] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0113] The aforementioned computer-readable storage medium may be included in the instructional video generation device; or it may exist independently and not be assembled into the instructional video generation device.

[0114] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the teaching video generation device, the teaching video generation device performs the following actions: parses the original teaching video to obtain the teacher's speech content text and teaching courseware; performs text reconstruction processing on the teacher's speech content text based on the target teacher's text style vector to obtain the target teacher's lecture script text; generates a target teacher's style voice based on the target teacher's text style vector and speech style vector, combined with the target teacher's lecture script text; the text style vector includes speech rate features, pause features, sentence structure features, and language style features, and the speech style vector includes fundamental frequency features; for each page of the teaching courseware, performs text reconstruction on the courseware content text based on the text style vector to obtain the target teaching courseware; and generates a teaching video based on the target teaching courseware, the target teacher's style voice, and the target teacher's virtual image.

[0115] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0117] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0118] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described teaching video generation method. This addresses the technical problem of insufficient transferability of the target teacher's language style and expression rhythm in related technologies. Compared with related technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the teaching video generation method provided in the above embodiments, and will not be elaborated upon here.

[0119] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the instructional video generation method described above.

[0120] The computer program product provided in this application can solve the technical problem of insufficient transferability of the target teacher's language style and expression rhythm in related technologies. Compared with related technologies, the beneficial effects of the computer program product provided in this application are the same as those of the teaching video generation method provided in the above embodiments, and will not be repeated here.

[0121] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.< / pause>

Claims

1. A method for generating instructional videos, characterized in that, The method includes: Analyze the original teaching videos to obtain the text of the teacher's audio content and teaching materials; Based on the text style vector of the target teacher, the text of the teacher's speech content is reconstructed to obtain the text of the target teacher's lecture script; Based on the target teacher's text style vector and speech style vector, and combined with the target teacher's lecture text, a target teacher's style voice is generated; the text style vector includes speech rate features, pause features, sentence structure features, and language style features, and the speech style vector includes fundamental frequency features; For each page of the teaching courseware, the courseware content text is reconstructed based on the text style vector to obtain the target teaching courseware; Based on the target teaching materials, the target teacher's voice style, and the target teacher's virtual image, a teaching video is generated.

2. The teaching video generation method as described in claim 1, characterized in that, The steps for analyzing the original teaching video to obtain the teacher's audio content text and teaching courseware include: The teacher's voice in the original teaching video is subjected to speech recognition processing to obtain the text of the teacher's voice content; From each video frame of the original teaching video, select the courseware video frames containing the teaching courseware, and sort each courseware video frame based on the courseware page number information contained in the courseware video frames to obtain the teaching courseware; For each page of the teaching materials, extract the text content of the teaching materials to obtain the text content of the teaching materials.

3. The teaching video generation method as described in claim 1, characterized in that, Before performing text reconstruction processing on the teacher's speech content text based on the target teacher's text style vector to obtain the target teacher's lecture script text, the following steps are also included: The target teacher's speech sample is divided into frames, and the fundamental frequency of each frame is calculated using a fundamental frequency estimation algorithm to obtain the fundamental frequency sequence of the target teacher's speech sample; The fundamental frequency mean and fundamental frequency standard deviation of the fundamental frequency sequence are calculated to obtain the fundamental frequency characteristics; Based on the total number of words in the text content of the target teacher's speech sample and the total duration of the speech in the target teacher's speech sample, the speech rate of the target teacher is calculated to obtain speech rate features; Silence segments are detected in the target teacher's speech sample. The pause ratio is determined based on the ratio of the duration of silence segments in the target teacher's speech sample to the total duration of the speech in the target teacher's speech sample, and pause features are obtained. The text content of the target teacher's voice sample is subjected to sentence length and short sentence recognition to determine the ratio of long and short sentences in the text content and obtain sentence structure features; Based on the BERT language model, feature extraction is performed on the text content to obtain language style features.

4. The teaching video generation method as described in claim 1, characterized in that, The step of reconstructing the teacher's speech text based on the target teacher's text style vector to obtain the target teacher's lecture script includes: The teacher's speech content text and the text style vector are combined with the lecture rewriting prompts and input into the text rewriting model to obtain multiple candidate lecture texts; wherein, the lecture rewriting prompts include content preservation constraints and style fit constraints; Calculate the style fit score and content retention score between each candidate lecture text and the teacher's voice content text; Based on the style fit score and the content retention score, the target teacher's lecture text is determined from each of the candidate lecture texts.

5. The teaching video generation method as described in claim 1, characterized in that, The step of generating the target teacher's style voice based on the target teacher's text style vector and speech style vector, combined with the target teacher's lecture text, includes: Based on the fundamental frequency features, the sentiment tags of the target teacher's lecture text are determined; The target teacher's lecture text, combined with prompt words, is input into a large language model, which outputs the potential pause points of the target teacher's lecture text. Add pause tags at potential pause points in the target teacher's lecture text; the pause tags contain pause duration information determined based on the pause features; The target teacher's lecture text, containing the emotion tags and the pause tags, is input into the speech synthesis model to generate the target teacher's style voice; the speech duration parameter of the speech synthesis model is determined based on the speech rate feature.

6. The teaching video generation method as described in claim 2, characterized in that, The steps of reconstructing the text content of each page of the teaching courseware based on the text style vector to obtain the target teaching courseware include: For each page of the teaching materials, the text content of the materials and the language style features, combined with the rewriting prompts, are input into the text rewriting model to obtain the rewritten text of the materials. The text of all the teaching courseware is rewritten and formatted according to page numbers to obtain the target teaching courseware.

7. A teaching video generation device, characterized in that, The device includes: The parsing module is used to parse the original teaching video to obtain the text of the teacher's voice content and teaching courseware; The lecture manuscript reconstruction module is used to reconstruct the text of the teacher's speech content based on the target teacher's text style vector to obtain the target teacher's lecture manuscript text. The sound generation module is used to generate a voice in the style of the target teacher based on the text style vector and speech style vector of the target teacher, combined with the text of the target teacher's lecture; the text style vector includes speech rate features, pause features, sentence structure features and language style features, and the speech style vector includes fundamental frequency features; The courseware reconstruction module is used to reconstruct the text content of each page of the teaching courseware based on the text style vector to obtain the target teaching courseware. The video generation module is used to generate teaching videos based on the target teaching courseware, the target teacher's style and voice, and the target teacher's virtual image.

8. A teaching video generation device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the instructional video generation method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the teaching video generation method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the instructional video generation method as described in any one of claims 1 to 6.

Images