Teaching video generation method, device, equipment, storage medium and program product

Abstract

The application discloses a teaching video generation method and device, equipment, a storage medium and a program product, relates to the technical field of video processing, and the method comprises the following steps: analyzing an original teaching video, obtaining teacher voice content text and teaching courseware; based on a text style vector of a target teacher, performing text reconstruction processing on the teacher voice content text to obtain a target teacher script text; based on the text style vector and a voice style vector of the target teacher, combining the target teacher script text to generate a target teacher style voice; for each page of teaching courseware, based on the text style vector, performing text reconstruction on the courseware content text of the teaching courseware to obtain target teaching courseware; and based on the target teaching courseware, the target teacher style voice and a virtual image of the target teacher, generating a teaching video. The application guarantees the logical integrity and knowledge point accuracy of course content, so that the generated video is consistent with the teaching style of the target teacher in terms of voice rhythm, language expression and courseware style.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a teaching video generation method and device, equipment, a storage medium and a program product. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, new technical means such as digital human instructors, speech synthesis, and automatic generation of course content are gradually introduced into the education field to improve teaching efficiency and course reuse rate. Through virtual teacher-driven course explanation, AI-synthesized speech-assisted teaching, and multi-language course video generation, education platforms can quickly update course content and adapt to multiple scenarios in a short period of time. These technologies have been applied in online courses in colleges and universities, digital human manufacturer products, and AIGC content creation tools, and have gradually formed a preliminary industrialization path.

[0003] In related technologies, the mainstream solutions mainly include four categories: one is to input the lecture and courseware and directly generate a virtual teacher course video; two is a script-driven video production platform that generates courses by pre-producing instructor images and voices; three is to use face swapping and speech synthesis technology to replace the existing course video instructors with new teachers; four is to generate multi-language version course videos through speech recognition and translation. Although these technologies have made some progress in course automatic generation and multi-language adaptation, these more only stay at the level of simple replacement of images and voices, ignoring the migration of teacher language style and expression method; unable to make individualized adjustments to rhythm, logic and expression according to the style of the target teacher; lack of teaching structure adaptation and style quantitative control mechanism; difficult to meet the needs of universities or platforms for replacing the lecture subject of existing high-quality course videos and maintaining the consistency of the style. Therefore, when facing the more complex application scenario of "replacement of the lecture subject of existing high-quality course videos", the existing technology still has obvious deficiencies. SUMMARY

[0004] The main purpose of the present application is to provide a teaching video generation method, device, equipment, storage medium and computer product, which aims to solve the technical problem of insufficient consistency migration ability of target teacher language style and expression rhythm in related art in teaching video generation.

[0005] To achieve the above-mentioned purpose, the present application provides a teaching video generation method, the method comprising:

[0006] parsing an original teaching video to obtain teacher speech content text and teaching courseware;

[0007] performing text reconstruction processing on the teacher speech content text based on a text style vector of a target teacher to obtain a target teacher script text;

[0008] generate the target teacher style sound based on the target teacher text style vector and the target teacher speech style vector and in combination with the target teacher speech text; the text style vector comprises a speech speed feature, a pause feature, a sentence pattern feature and a language style feature, and the speech style vector comprises a fundamental frequency feature;

[0009] reconstruct the text of the course content of the teaching courseware based on the text style vector to obtain the target teaching courseware for each page of the teaching courseware;

[0010] generate the teaching video based on the target teaching courseware, the target teacher style sound and the virtual image of the target teacher.

[0011] In an embodiment, the step of analyzing the original teaching video to obtain the teacher speech content text and the teaching courseware comprises:

[0012] perform speech recognition processing on the teacher speech in the original teaching video to obtain the teacher speech content text;

[0013] select courseware video frames containing the teaching courseware from the video frames of the original teaching video, sort the courseware video frames based on the courseware page code information contained in the courseware video frames to obtain the teaching courseware;

[0014] extract the text content of the teaching courseware to obtain the course content text for each page of the teaching courseware.

[0015] In an embodiment, before the step of reconstructing the text of the teacher speech content text based on the target teacher text style vector to obtain the target teacher speech text, the method further comprises the step of:

[0016] frame the target teacher speech sample, calculate the fundamental frequency of each frame by a fundamental frequency estimation algorithm to obtain the fundamental frequency sequence of the target teacher speech sample;

[0017] calculate the mean and standard deviation of the fundamental frequency sequence to obtain the fundamental frequency feature;

[0018] calculate the speech speed of the target teacher based on the total number of words of the text content of the target teacher speech sample and the total duration of the speech of the target teacher speech sample to obtain the speech speed feature;

[0019] detect the silence segment of the target teacher speech sample, determine the pause ratio based on the ratio of the silence segment duration of the target teacher speech sample to the total duration of the speech of the target teacher speech sample to obtain the pause feature;

[0020] identify the long and short sentences of the text content of the target teacher speech sample to determine the long and short sentence ratio of the text content to obtain the sentence pattern feature;

[0021] extract the language style feature based on the BERT language model.

[0022] In an embodiment, the step of obtaining the target teacher's lecture text based on the target teacher's text style vector and the text reconstruction of the teacher's voice content text comprises:

[0023] The teacher's voice content text and the text style vector are combined with the lecture rewriting prompt word to input into the text rewriting model to obtain a plurality of candidate lecture texts; wherein the lecture rewriting prompt word comprises a content maintenance constraint and a style fitting constraint;

[0024] The style fitting degree score and the content maintenance degree score between each candidate lecture text and the teacher's voice content text are calculated respectively;

[0025] Based on the style fitting degree score and the content maintenance degree score, the target teacher's lecture text is determined from the candidate lecture texts.

[0026] In an embodiment, the step of generating the target teacher's style sound based on the target teacher's text style vector and voice style vector and combining the target teacher's lecture text comprises:

[0027] Based on the fundamental frequency feature, the emotional label of the target teacher's lecture text is determined;

[0028] The target teacher's lecture text is combined with the prompt word and input into the large language model to output the potential pause point of the target teacher's lecture text;

[0029] Pause labels are added at the potential pause points of the target teacher's lecture text; the pause labels contain pause duration information determined based on the pause feature;

[0030] The target teacher's lecture text containing the emotional label and the pause label is input into the speech synthesis model to generate the target teacher's style sound; the speech duration parameter of the speech synthesis model is determined based on the speech speed feature.

[0031] In an embodiment, the step of obtaining the target teaching course based on the text style vector and the text reconstruction of the course content text of the teaching course for each page of the teaching course comprises:

[0032] For each page of the teaching course, the course content text and the language style feature are combined with the course rewriting prompt word to input into the text rewriting model to obtain the course rewriting text;

[0033] The course rewriting texts of all teaching courses are paginated according to the page number to obtain the target teaching course.

[0034] In a second aspect, to achieve the above object, the present application further provides a teaching video generation device, the device comprising:

[0035] The parsing module is configured to parse the original teaching video to obtain a teacher speech content text and a teaching courseware;

[0036] The speech generation module is configured to generate a target teacher style sound based on the text style vector and the speech style vector of the target teacher and in combination with the target teacher speech text, wherein the text style vector comprises a speech speed feature, a pause feature, a sentence pattern feature and a language style feature, and the speech style vector comprises a fundamental frequency feature.

[0037] The speech generation module is configured to generate a target teacher style sound based on the text style vector and the speech style vector of the target teacher and in combination with the target teacher speech text, wherein the text style vector comprises a speech speed feature, a pause feature, a sentence pattern feature and a language style feature, and the speech style vector comprises a fundamental frequency feature.

[0038] The courseware reconstruction module is configured to, for each page of the teaching courseware, perform text reconstruction on a courseware content text of the teaching courseware based on the text style vector to obtain a target teaching courseware.

[0039] The video generation module is configured to generate a teaching video based on the target teaching courseware, the target teacher style sound and a virtual image of the target teacher.

[0040] In a third aspect, to achieve the above object, the present application further provides a teaching video generation device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the teaching video generation method.

[0041] In a fourth aspect, to achieve the above object, the present application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the teaching video generation method.

[0042] In a fifth aspect, to achieve the above object, the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the teaching video generation method.

[0043] The one or more technical solutions provided by the present application have at least the following technical effects:

[0044] The present application analyzes the speech and courseware content of the original teaching video, combines the text style vector and the speech style vector of the target teacher, and realizes personalized style transfer in the aspects of speech generation, courseware content reconstruction and sound synthesis, so as to make the generated video conform to the teaching style of the target teacher in terms of speech rhythm, language expression and courseware style under the premise of ensuring the logical integrity and accuracy of the course content, and finally form a personalized teaching video with high immersion and reality, thereby significantly improving the course reuse efficiency and learning experience. BRIEF DESCRIPTION OF DRAWINGS

[0045] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate preferred embodiments of the present application and, together with the description, serve to explain the principles of the application.

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, those skilled in the art can obtain other drawings from these drawings without any creative effort.

[0047] Figure 1 The flowchart of the teaching video generation method in the embodiments of the present application.

[0048] Figure 2 The module connection diagram of the teaching video generation device.

[0049] Figure 3 The structure diagram of the teaching video generation device.

[0050] The purposes, functional features and advantages of the present application will be further explained with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0052] In order to better understand the technical solutions of the present application, the following will be described in detail in combination with the drawings in the specification and the specific embodiments.

[0053] The main solution of the embodiments of the present application is: first, parse the original teaching video, extract the teacher's speech content text and teaching courseware, and extract the speech speed, pause, sentence pattern, language style and fundamental frequency and other style features from the speech; then, combine the target teacher's text style vector and the speech style vector to reconstruct the original lecture text and courseware content, and obtain the lecture and courseware that conform to the target teacher's expression style; on this basis, through emotion labeling, latent pause prediction and time length control, input the target teacher's lecture text into the speech synthesis model to generate the sound with the target teacher's style; finally, combine the target teacher's style sound, reconstructed courseware and target teacher's virtual image to synthesize a complete teaching video, which realizes the reproduction of the teaching video in the target teacher's style while keeping the course knowledge content and structure unchanged.

[0054] Specifically, the embodiments of the present application provide a teaching video generation method, which refers to Figure 1 , Figure 1 The flowchart of the first embodiment of the teaching video generation method of the present application.

[0055] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a teaching video generation device, etc. capable of realizing the above functions. The following takes the teaching video generation device as an example to describe the embodiment and the following embodiments.

[0056] In the embodiment, the teaching video generation method includes steps S10-S50:

[0057] Step S10, analyzing the original teaching video to obtain the teacher's speech content text and the teaching courseware.

[0058] It should be noted that the original teaching video, as a real recorded course video, is a reference object for generating the target teaching video, and as an input source of the scheme, it is used to extract the teacher's script and courseware to support subsequent style transfer and video reconstruction.

[0059] The teacher's speech content is a text obtained by speech recognition and processing of the teacher's lecture sound in the original teaching video, and the teaching courseware is the teaching courseware appearing in the original teaching video, such as the PPT courseware displayed by the teacher during the lecture.

[0060] In a feasible real-time manner, step S10 includes steps A10-A30:

[0061] Step A10, performing speech recognition processing on the teacher's speech in the original teaching video to obtain the teacher's speech content text.

[0062] Step A20, screening the courseware video frames containing the teaching courseware from the video frames of the original teaching video, sorting the courseware video frames based on the courseware page information contained in the courseware video frames, and obtaining the teaching courseware.

[0063] Step A30, for each page of the teaching courseware, extracting the text content of the teaching courseware to obtain the courseware content text.

[0064] Table 1 Courseware and speech schedule

[0065]

[0066] In an example, the way of analyzing the original teaching video can be to use the Whisper model to perform automatic speech recognition on the audio part of the original teaching video to obtain the teacher's speech content text. Use OpenCV to determine the video frames in which the PPT courseware exists in the original teaching video, extract the PPT courseware in the video frames into one by one PPT pictures in order, and at the same time call the PaddleOCR model to extract the content in the PPT pictures into courseware content text.

[0067] Subsequently, the dynamic time warping algorithm is used to align the teaching voice content text and the PPT courseware page, facilitating subsequent processing. Specifically, the video frame sequence of the original teaching video is V={V_1, V_2,...., V_n}, from which the video frame sequence containing the PPT courseware is determined, which is converted into the PPT image sequence P={p_1, p_2,...., p_m}. The content of the teacher's voice content text is divided by section to form the text sequence T={t_1, t_2,...., t_k}.

[0068] The start time and end time of the PPT courseware page in the original teaching video and the start time and end time of the voice corresponding to each paragraph in the teacher's voice content text in the original teaching video are determined, as shown in Table 1 Courseware and Voice Time Table.

[0069] The start-end time stamp of the PPT courseware page and the time stamp of the teacher's voice content text are determined by time stamp intersection judgment, and the corresponding relationship shown in Table 2 Time Stamp Alignment Table is established, and the alignment of the teaching voice content text and the PPT courseware page is completed.

[0070] Table 2 Time Stamp Alignment Table

[0071]

[0072] Step S20, based on the text style vector of the target teacher, the text reconstruction processing of the teacher's voice content text is performed, and the target teacher's lecture text is obtained.

[0073] Step S30, based on the text style vector and the voice style vector of the target teacher, the target teacher's lecture text is combined to generate the target teacher's style sound.

[0074] Step S40, for each page of teaching courseware, based on the text style vector, the text reconstruction of the courseware content text of the teaching courseware is performed, and the target teaching courseware is obtained.

[0075] The main purpose of introducing the style vector in this embodiment is to transfer the teacher's voice content text in the original teaching video to the target teacher's lecture text of the target teacher's style according to the text style vector, and then convert the target teacher's lecture text to the target teacher's audio combined with the voice style vector.

[0076] It should be noted that the text style vector includes speech rate features, pause features, sentence pattern features, and language style features, which are used to provide reference for text processing, and the voice style vector includes fundamental frequency features, which are used to provide reference for voice signal processing.

[0077] In a feasible implementation, the manner of determining the text style vector and the speech style vector can include steps B10-B60.

[0078] Step B10, frame the target teacher speech sample, calculate the fundamental frequency of each frame by the fundamental frequency estimation algorithm, and obtain the fundamental frequency sequence of the target teacher speech sample.

[0079] Step B20, calculate the mean and standard deviation of the fundamental frequency sequence to obtain the fundamental frequency feature.

[0080] Step B30, calculate the speech rate of the target teacher based on the total number of words in the text content of the target teacher speech sample and the total duration of the target teacher speech sample, and obtain the speech rate feature.

[0081] Step B40, detect the silent section of the target teacher speech sample, determine the pause ratio based on the ratio of the silent section duration of the target teacher speech sample to the total duration of the target teacher speech sample, and obtain the pause feature.

[0082] Step B50, identify the long and short sentences of the text content of the target teacher speech sample to determine the long and short sentence ratio of the text content, and obtain the sentence style feature.

[0083] Step B60, extract features from the text content based on the BERT language model to obtain the language style feature.

[0084] Specifically, the embodiment mainly models the style of the target teacher, analyzes the multi-dimensional features of the historical speech sample of the target teacher, extracts a style vector that can fully represent the personal expression style of the teacher, which includes sound level features and text level features, and provides a data basis for subsequent speech reconstruction, speech synthesis and style transfer.

[0085] For example, collect the historical speech data of the target teacher as the target teacher speech sample, extract the unique expression style of the target teacher from the target teacher speech sample, and construct a style vector S teacher In this example, newteacher.wav is used as the target teacher speech sample and the corresponding text content is extracted, which is "Hello everyone, today we will learn the first chapter. First we see the overall structure of the course...". The way to extract the style vector of the target teacher mainly includes:

[0086] Fundamental frequency feature extraction: use the PYIN algorithm to extract the fundamental frequency of each frame of the target teacher speech sample to obtain the fundamental frequency sequence F_0(t)=[123Hz, 120Hz, 118Hz, 125Hz, …], and calculate the mean pF 0 and the standard deviation of the fundamental frequency sequence F_0(t) pF0. The mean fundamental frequency is used as the average fundamental frequency characteristic of the target teachers, and the standard deviation of the fundamental frequency is used as the fundamental frequency volatility characteristic of the target teachers. In this example, the average fundamental frequency is 122.5Hz, and the fundamental frequency volatility is 3.4.

[0087] Speech rate feature extraction: In this example, the text content of the target teacher's speech sample contains a total of 41 Chinese characters, approximately 19 words, and the total duration of the target teacher's speech sample is 11.7 seconds (i.e., 0.195 minutes). The calculated speech rate feature WMP is as follows:

[0088]

[0089] Pause feature extraction involves detecting silence segments in the target teacher's speech sample and calculating the duration of each silence segment. In this example, two significant pauses were found: the first pause lasted 4.6–5.0 seconds (0.4 seconds); the second pause lasted 8.1–8.5 seconds (0.4 seconds). Therefore, the duration of the silence segment is 0.8 seconds. The total duration of the target teacher's speech sample is 11.7 seconds, and the pause ratio ρ is:

[0090]

[0091] Sentence structure feature extraction involves identifying the length of sentences in the text content of the target teacher's voice sample, determining the ratio of long to short sentences in the text content, and obtaining sentence structure features. In this example, the ratio of long sentences to short sentences is 0.7, indicating a preference for shorter sentences.

[0092] The steps for extracting language style features can be as follows: the text content of the target teacher's speech sample is represented by the BERT language model embedder. The text content is input into the BERT language model embedder to obtain the sentence vector L_s. BERT will map a sentence to a high-dimensional dense vector space, usually 512-dimensional, 768-dimensional, etc. In this embodiment, the first 512 dimensions are extracted as the language style feature representation.

[0093] Ultimately, the style vector of the target teacher is obtained. S teacher = [Average fundamental frequency, fundamental frequency variability, speech rate characteristics, sentence structure characteristics, language style characteristics] = [122.5, 2.4, 97.44, 0.06, 0.7, L] _S ].

[0094] After obtaining the style vector of the target teacher, the teacher's speech content text rewriting, target teacher style voice synthesis, and teaching courseware reconstruction are performed based on the style vector. Specifically, step S20 includes steps C10 to C30:

[0095] Step C10, input the teacher voice content text and the text style vector into a text rewriting model to obtain a plurality of candidate lecture text; wherein the lecture rewriting prompt word includes a content retention constraint and a style fitting constraint.

[0096] Step C20, respectively calculate the style fitting degree score and the content retention degree score between each candidate lecture text and the teacher voice content text.

[0097] Step C30, based on the style fitting degree score and the content retention degree score, determine the target teacher lecture text from the candidate lecture texts.

[0098] In the embodiment, the text rewriting model can be a general generative large language model, and the main core of the target teacher lecture text construction is to use the large language model for content retention type style transfer, and introduce control variables to guide the style factors such as sentence complexity, tone, keyword selection, etc.

[0099] For example, according to the style vector of the target teacher extracted before S teacher , the teacher voice content text T src is rewritten to obtain the target teacher lecture text T style The way can be expressed as:

[0100] T style =StyleRewrite(T src, S teacher )

[0101] Among them, StyleRewrite() It can be understood as a text rewriting model, the input of which is the style vector of the target teacher S teacher and the teacher voice content text T src , and the output is the target teacher lecture text T style .

[0102] In the specific rewriting process, content retention constraints and style similarity constraints can be used to ensure the rewriting quality of the target teacher lecture text.

[0103] Specifically, the content retention degree constraint is mainly used to ensure that the migrated lecture is consistent with the original text in terms of knowledge structure, teaching logic and core points, which can be expressed as:

[0104] ContentSim(T style,T src )≥δ c

[0105] wherein, is the teacher speech content text, T style is the target teacher lecture text, represents the content preservation score, which can be calculated by a text similarity calculation function to measure whether the target teacher lecture text after migration is faithful to the teacher speech content text. Specifically, first, the vector model is used to convert and T style into high-dimensional vectors, and then the cosine angle between and T style is calculated as the content preservation score. The calculation method of the content preservation score can be represented as:

[0106]

[0107] The value range is 0~1, and the closer to 1 represents the more consistent content.

[0108] δ c is the content preservation constraint threshold, which is higher, the stricter, and is preferably set to 0.85 in this example.

[0109] The style fit degree constraint is mainly used to keep the style of the lecture after migration consistent with the teaching style of the target teacher, which can be represented as:

[0110]

[0111] wherein, is the style vector of the target teacher, T style is the target teacher lecture text, StyleSim ( T style , S teacher ) is a style similarity calculation function used to measure whether the text style after style migration is consistent with the style of the new teacher. It is worth noting that when calculating the style similarity, the part of the style vector that can be processed in the text is selected, that is, the text style vector S gen , S gen = [Speech speed feature, pause feature, sentence feature, language style feature], specifically, first, the vector model is used to convert S ​gen and T style Convert into high-dimensional vectors, and then calculate S gen and T style The cosine of the included angle of δ s is the constraint threshold of style fit degree. Preferably, δ s It can be set to 0.8.

[0112] It can be understood that the content retention degree constraint is introduced in the text rewriting process, and the content consistency of the target teacher's lecture text obtained after rewriting is judged in combination with the constraint threshold, so as to ensure that the teaching structure and core points are not weakened while the style is migrated, so as to maintain the stability and consistency of the teaching quality. And the introduction of the style fit degree constraint can ensure that the output text is close to the expression habits of the target teacher in terms of sentence structure, spoken language degree, transition logic and keyword style.

[0113] In the present example, the specific steps of the teacher's voice content text are as follows:

[0114] Through the Prompt explicit instruction including the content retention degree and style fit degree constraints, the large model generates a plurality of candidate lecture texts [T1, T2,...., T n ].

[0115] The content similarity score and the style fit degree score are calculated for each candidate lecture text, respectively.

[0116] The similarity score and the style fit degree score are combined to obtain a comprehensive evaluation score, and the candidate lecture text with the highest comprehensive evaluation score is taken as the target teacher's lecture text.

[0117] The above steps can be expressed in the following way:

[0118]

[0119] wherein, T style represents the teacher's lecture file, λ c and λ s represents the weight information. In the present example, λ c = 0.7, λ s = 0.3, indicating a greater bias towards content retention.

[0120] Step S30 includes steps D10-D40:

[0121] Step D10, determine the emotional label of the target teacher's lecture text based on the fundamental frequency features.

[0122] Step D20, input the target teacher's lecture text combined with the prompt words into the large language model, and output the potential pause points of the target teacher's lecture text.

[0123] Step D30, add pause labels at the potential pause points of the target teacher's lecture text; the pause labels contain pause duration information determined based on the pause features.

[0124] Step D40, input the target teacher's lecture text containing emotional labels and pause labels into the speech synthesis model to generate the target teacher style voice.

[0125] wherein the speech duration parameters of the speech synthesis model are determined based on the speech rate features.

[0126] Specifically, the speech synthesis model can be an open source speech generation large model, such as CosyVoice2.0, which first determines the features that meet the input control requirements of the speech generation large model from the style vector S teacher The features include fundamental frequency features (average fundamental frequency and fundamental frequency fluctuation), pause features, and speech rate features. By analyzing the average fundamental frequency and fundamental frequency fluctuation, the emotional label classification of the generated teacher voice can be selected. By the speech rate feature, the speech duration of the model output is determined, and by the pause feature, the insertion position of the pause label is determined.

[0127] For example, first, determine the emotional label of the speech synthesis model TTS according to the average fundamental frequency and the fundamental frequency fluctuation. The average fundamental frequency is the average value of the fundamental frequency of each frame in the speech, with the unit of Hz. The natural speaking pitch of men and women differs greatly, so a gender judgment threshold is needed. In this example, the gender of the target teacher is detected to be male, and the average fundamental frequency is 122.5 Hz, which basically meets the lower limit of the male average fundamental frequency range (120-150 Hz) (see Table 3 for pitch evaluation table), which can be considered as a slightly low pitch style.

[0128] Table 3 Pitch Evaluation Table

[0129]

[0130] The fundamental frequency fluctuation can reflect the amplitude of the intonation. Generally, the speaking style with obvious intonation will have a higher fundamental frequency fluctuation. Table 4 Style Evaluation Table shows the general experience threshold for style evaluation (common for men and women).

[0131] Table 4 Style Evaluation Table

[0132]

[0133] In the present example, the fundamental frequency fluctuation value = 3.4 Hz, which falls exactly in the "normal fluctuation" interval.

[0134] The average fundamental frequency and the fundamental frequency fluctuation value of the target teacher are comprehensively evaluated to obtain an emotion label (which can be "neutral" or "serious") conforming to the format requirements of the speech synthesis model CosyVoice2.0, so as to bring the mapped emotion label when synthesizing speech.

[0135] Subsequently, the speech duration parameter of the speech synthesis model is determined. Specifically, in the present example, the speech rate feature WMP of the target teacher is 97.44, while the default of the model is 110, that is, the default speech duration parameter needs to be reduced by 1.13 times to make the output speech signal close to the speech rate style of the target teacher. When inputting the lecture text of the target teacher into the speech synthesis model, it is necessary to add a pause label in the text. In the present example, the confirmation method of the pause label position is to input the lecture text of the target teacher into the large language model, combine the prompt words, and make the large language model output the pause point set of the lecture text of the target teacher based on the text syntax structure, punctuation, and keywords, and add a pause label at these pause points <pause>, and the break time of each pause point (each break time is in the range of 150 ms to 500 ms) is determined by dynamically adjusting the inserted pause time, so that the pause ratio of the target teacher's speech text is the same as the pause feature p (6.8%).

[0136] Specifically, in this example, the pause ratio p of the target teacher is 6.8%, which means that there is about 0.068 seconds of pause in every 1 second of speech.

[0137] First, the initial pause time of each pause point is determined according to the character type before each pause point in the pause point set.

[0138] According to the initial pause time, the initial pause ratio is determined.

[0139] The difference between the initial pause ratio and the pause ratio of the target teacher is calculated.

[0140] If the initial pause ratio is less than the pause ratio of the target teacher, the pause time of each pause point is dynamically increased according to the character type before each pause point.

[0141] If the initial pause ratio is greater than the pause ratio of the target teacher, the pause time of each pause point is dynamically reduced according to the character type before each pause point.

[0142] Specifically, pause points with character types of periods and semicolons are marked as mandatory pause points, pause points with character types of commas and colons are marked as weak pause points, and if the pause point is not a punctuation mark but a preset keyword (such as "firstly", "conclusively", "therefore"), the pause point is marked as an emphasized pause point. If the initial pause ratio is less than the pause ratio of the target teacher, the pause time of each pause point is extended according to the priority order of weak pause points-emphasized pause points-mandatory pause points, so that the difference between the initial pause ratio and the pause ratio of the target teacher does not exceed the preset range (±0.005 in this example). Correspondingly, if the initial pause ratio is greater than the pause ratio of the target teacher, the pause time of each pause point is shortened according to the priority order of weak pause points-emphasized pause points-mandatory pause points, so that the difference between the initial pause ratio and the pause ratio of the target teacher does not exceed the preset range (±0.005 in this example).

[0143] Finally, the target teacher's speech text containing emotion labels and pause labels is input into a speech synthesis model, so that the speech synthesis model has a higher matching degree with the target teacher's speech style rhythm and the target teacher's style sound.

[0144] Step S40 includes steps E10-E20:

[0145] Step E10, for each page of the teaching courseware, input the courseware content text and language style features combined with courseware rewriting prompts into the text rewriting model to obtain courseware rewritten text.

[0146] Step E20, arrange the courseware rewritten text of all teaching courseware according to the page number to obtain the target teaching courseware.

[0147] Specifically, first, the text rewriting model (which can be a general generative large language model) is used to rewrite the text content of each page of the PPT courseware in style according to the prompt and the language style features of the target teacher, to obtain PPT courseware text that conforms to the style of the target teacher. Then, combined with the PPT page number obtained by analyzing the original teaching video and the time each page appears in the video, a new course PPT is generated for new video synthesis.

[0148] Illustratively, the inheritance target teacher style setting Prompt explicit instruction is input into the text rewriting model, such as: please keep the following PPT page content unchanged, rewrite it into a more clear and clear, formal PPT expression method that conforms to the teaching specification.

[0149] Original PPT content:

[0150] {Chapter 1: Introduction

[0151] Course objectives: master the basic framework, understand the background, and introduce the method.

[0152] Contents include: history, methodology, application cases

[0153] Current teacher preferred style:

[0154] Pay attention to parallel structure and clearness

[0155] Like to use guiding sentences

[0156] Rewritten output:

[0157] {Chapter 1: Course Introduction

[0158] Chapter objectives:

[0159] Understand the overall structure of the course

[0160] Master the core research method

[0161] Understand the relevant historical background and typical applications

[0162] Main contents of this chapter:

[0163] 1. History of subject development

[0164] 2. Methodology foundation

[0165] 3. Practice case analysis

[0166] The rewritten content is automatically typeset into a new course template PPT to form a new PPT.

[0167] The current teacher preference style in the example is obtained by clustering analysis on 512-dimensional language style features.

[0168] Step S50, based on the target teaching courseware, the target teacher style voice and the virtual image of the target teacher, a teaching video is generated.

[0169] It should be noted that in the process of generating a teaching video, the virtual image needs to be confirmed in combination with the target teacher style voice.

[0170] Specifically, the shot exemption of the target teacher generated by the speech synthesis model is taken as input to drive the digital image of the teacher to generate a video with transparent channels that match the mouth shape. Then the virtual image of the target teacher, the rewritten PPT courseware and the voice of the target teacher are input into the rendering synthesis engine to be fused into the final output course video.

[0171] In summary, the embodiment introduces the text style vector and the speech style vector of the target teacher to perform multi-dimensional style migration and reconstruction on the script, speech and courseware content of the original teaching video, realizes the conversion of the course video into a teaching video that meets the personalized style of the target teacher without changing the teaching knowledge structure and core content, and ensures the integrity of the teaching logic while improving the personalized expression effect of the content. The method not only makes the generated video close to the natural teaching habits of the target teacher in terms of speech speed, pause, sentence pattern and language style, but also maintains the orderliness and consistency in courseware expression, and finally significantly improves the acceptability and immersion of the teaching video, facilitating efficient reuse and differentiated customization of teaching resources.

[0172] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the teaching video generation method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0173] The present application also provides a teaching video generation device, please refer to Figure 2 The teaching video generation device comprises:

[0174] The analysis module 10 is used for analyzing the original teaching video to obtain the teacher speech content text and the teaching courseware.

[0175] The script reconstruction module 20 is used for performing text reconstruction processing on the teacher speech content text based on the text style vector of the target teacher to obtain the target teacher script text.

[0176] The sound generation module 30 is configured to generate a target teacher style sound based on the target teacher's text style vector and speech style vector in combination with the target teacher's speech text. The text style vector includes a speech speed feature, a pause feature, a sentence pattern feature and a language style feature. The speech style vector includes a fundamental frequency feature.

[0177] The course reconstruction module 40 is configured to perform text reconstruction on the course content text of the teaching course based on the text style vector for each page of the teaching course, to obtain a target teaching course.

[0178] The video generation module 50 is configured to generate a teaching video based on the target teaching course, the target teacher style sound and the virtual image of the target teacher.

[0179] The teaching video generation device provided in the present application adopts the teaching video generation method in the above-mentioned embodiments, and can solve the technical problem of insufficient consistency migration capability of the language style and expression rhythm of the target teacher in the related art. Compared with the related art, the teaching video generation device provided in the present application has the same beneficial effects as the teaching video generation method provided in the above-mentioned embodiments, and other technical features in the teaching video generation device are the same as the features disclosed in the above-mentioned embodiments, which will not be repeated here.

[0180] The present application provides a teaching video generation device, which comprises at least one processor and a memory in communication connection with the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the teaching video generation method in the above-mentioned embodiments.

[0181] Reference will now be made to the drawings, and specific examples thereof will be described hereinbelow. Figure 3 which shows a structural schematic diagram of a teaching video generation device suitable for implementing the embodiments of the present application. The teaching video generation device in the embodiments of the present application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (for example, vehicle-mounted navigation terminals) and the like, and fixed terminals such as digital TVs, desktop computers and the like. Figure 3 The teaching video generation device shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0182] As Figure 3 As shown, the teaching video generation device can include a processing apparatus 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read-only memory 1002 (ROM) or loaded from a storage apparatus 1003 into a random access memory 1004 (RAM). Various programs and data required for the operation of the teaching video generation device are also stored in the random access memory 1004. The processing apparatus 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. An input / output interface 1006 (I / O interface) is also connected to the bus 1005. Generally, the following systems can be connected to the input / output interface 1006: input apparatuses 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output apparatuses 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage apparatus 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication apparatus 1009. The communication apparatus 1009 can allow the teaching video generation device to communicate with other devices wirelessly or by wire to exchange data. Although the teaching video generation device with various systems is shown in the figure, it should be understood that all the systems shown are not required to be implemented or possessed. More or fewer systems can be alternatively implemented or possessed.

[0183] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication apparatus, or installed from the storage apparatus 1003, or installed from the ROM 1002. When the computer program is executed by the processing apparatus 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.

[0184] The teaching video generation device provided by the present application adopts the teaching video generation method in the above-mentioned embodiments, and can solve the technical problem of insufficient consistency migration capability of the target teacher's language style and expression rhythm in the related art. Compared with the related art, the teaching video generation device provided by the present application has the same beneficial effects as the teaching video generation method provided by the above-mentioned embodiments, and other technical features in the teaching video generation device are the same as the features disclosed in the above-mentioned embodiments, which will not be repeated here.

[0185] It should be understood that various aspects of the disclosure can be implemented in hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any appropriate manner in any one or more embodiments or examples.

[0186] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0187] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer programs) for performing the teaching video generation method in the above embodiments.

[0188] The computer readable storage medium provided by the present application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to: an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, system or device. The program code contained on the computer readable storage medium can be transmitted by any appropriate medium, including but not limited to: electric wire, optical cable, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0189] The above computer readable storage medium can be contained in the teaching video generation device; or can exist separately without being assembled into the teaching video generation device.

[0190] The computer readable storage medium described above carries one or more programs, when the one or more programs are executed by the teaching video generation device, the teaching video generation device is caused to: parse an original teaching video to obtain teacher speech content text and teaching courseware; perform text reconstruction processing on the teacher speech content text based on a text style vector of a target teacher to obtain a target teacher script text; generate a target teacher style sound based on the text style vector and a speech style vector of the target teacher in combination with the target teacher script text; the text style vector includes speech speed characteristics, pause characteristics, sentence pattern characteristics and language style characteristics, and the speech style vector includes fundamental frequency characteristics; for each page of the teaching courseware, perform text reconstruction on courseware content text of the teaching courseware based on the text style vector to obtain a target teaching courseware; and generate a teaching video based on the target teaching courseware, the target teacher style sound and a virtual image of the target teacher.

[0191] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0192] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may be executed in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or combinations of hardware and software.

[0193] The modules involved in the embodiments of the present application can be implemented in software or in hardware. In some cases, the names of the modules do not limit the modules themselves.

[0194] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., a computer program) for executing the above teaching video generation method, and can solve the technical problem of insufficient consistency migration ability of the target teacher's language style and expression rhythm in related technologies. Compared with related technologies, the computer readable storage medium provided by the present application has the same beneficial effects as the teaching video generation method provided by the above embodiments, which will not be repeated here.

[0195] The present application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the teaching video generation method as described above.

[0196] The computer program product provided by the present application can solve the technical problem of insufficient consistency migration ability of the target teacher's language style and expression rhythm in related technologies. Compared with related technologies, the computer program product provided by the present application has the same beneficial effects as the teaching video generation method provided by the above embodiments, which will not be repeated here.

[0197] The above is only some embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation made by using the content of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.< / pause>

Claims

1. A method for generating instructional videos, characterized in that, The method includes: Analyze the original teaching videos to obtain the text of the teacher's audio content and teaching materials; Based on the text style vector of the target teacher, the text of the teacher's speech content is reconstructed to obtain the text of the target teacher's lecture script; Based on the target teacher's text style vector and speech style vector, and combined with the target teacher's lecture text, a target teacher's style voice is generated; the text style vector includes speech rate features, pause features, sentence structure features, and language style features, and the speech style vector includes fundamental frequency features; For each page of the teaching courseware, the courseware content text is reconstructed based on the text style vector to obtain the target teaching courseware; Based on the target teaching materials, the target teacher's voice style, and the target teacher's virtual image, a teaching video is generated; The step of reconstructing the teacher's speech text based on the target teacher's text style vector to obtain the target teacher's lecture script includes: The teacher's speech content text and the text style vector are combined with the lecture rewriting prompts and input into the text rewriting model to obtain multiple candidate lecture texts; wherein, the lecture rewriting prompts include content preservation constraints and style fit constraints; Calculate the style fit score and content retention score between each candidate lecture text and the teacher's voice content text; Based on the style fit score and the content retention score, the target teacher's lecture text is determined from each of the candidate lecture texts; The style fit constraint can be expressed as: in, For the style vector of the target teacher, T style For target teacher lecture text, StyleSim ( T style , S teacher This is a style similarity calculation function used to measure whether the style of the transferred text matches the style of the new teacher. δ s This is the constraint threshold for style fit.

2. The teaching video generation method as described in claim 1, characterized in that, The steps for analyzing the original teaching video to obtain the teacher's audio content text and teaching courseware include: The teacher's voice in the original teaching video is subjected to speech recognition processing to obtain the text of the teacher's voice content; From each video frame of the original teaching video, select the courseware video frames containing the teaching courseware, and sort each courseware video frame based on the courseware page number information contained in the courseware video frames to obtain the teaching courseware; For each page of the teaching materials, extract the text content of the teaching materials to obtain the text content of the teaching materials.

3. The teaching video generation method as described in claim 1, characterized in that, Before performing text reconstruction processing on the teacher's speech content text based on the target teacher's text style vector to obtain the target teacher's lecture script text, the following steps are also included: The target teacher's speech sample is divided into frames, and the fundamental frequency of each frame is calculated using a fundamental frequency estimation algorithm to obtain the fundamental frequency sequence of the target teacher's speech sample; The fundamental frequency mean and fundamental frequency standard deviation of the fundamental frequency sequence are calculated to obtain the fundamental frequency characteristics; Based on the total number of words in the text content of the target teacher's speech sample and the total duration of the speech in the target teacher's speech sample, the speech rate of the target teacher is calculated to obtain speech rate features; Silence segments are detected in the target teacher's speech sample. The pause ratio is determined based on the ratio of the duration of silence segments in the target teacher's speech sample to the total duration of the speech in the target teacher's speech sample, and pause features are obtained. The text content of the target teacher's voice sample is subjected to sentence length and short sentence recognition to determine the ratio of long and short sentences in the text content and obtain sentence structure features; Based on the BERT language model, feature extraction is performed on the text content to obtain language style features.

4. The teaching video generation method as described in claim 1, characterized in that, The step of generating the target teacher's style voice based on the target teacher's text style vector and speech style vector, combined with the target teacher's lecture text, includes: Based on the fundamental frequency features, the sentiment tags of the target teacher's lecture text are determined; The target teacher's lecture text, combined with prompt words, is input into a large language model, which outputs the potential pause points of the target teacher's lecture text. Add pause tags at potential pause points in the target teacher's lecture text; the pause tags contain pause duration information determined based on the pause features; The target teacher's lecture text, containing the emotion tags and the pause tags, is input into the speech synthesis model to generate the target teacher's style voice; the speech duration parameter of the speech synthesis model is determined based on the speech rate feature.

5. The teaching video generation method as described in claim 2, characterized in that, The steps of reconstructing the text content of each page of the teaching courseware based on the text style vector to obtain the target teaching courseware include: For each page of the teaching materials, the text content of the materials and the language style features, combined with the rewriting prompts, are input into the text rewriting model to obtain the rewritten text of the materials. The text of all the teaching courseware is rewritten and formatted according to page numbers to obtain the target teaching courseware.

6. A teaching video generation device, characterized in that, The device includes: The parsing module is used to parse the original teaching video to obtain the text of the teacher's voice content and teaching courseware; The lecture manuscript reconstruction module is used to reconstruct the text of the teacher's speech content based on the target teacher's text style vector to obtain the target teacher's lecture manuscript text. The sound generation module is used to generate a voice in the style of the target teacher based on the text style vector and speech style vector of the target teacher, combined with the text of the target teacher's lecture; the text style vector includes speech rate features, pause features, sentence structure features and language style features, and the speech style vector includes fundamental frequency features; The courseware reconstruction module is used to reconstruct the text content of each page of the teaching courseware based on the text style vector to obtain the target teaching courseware. The video generation module is used to generate teaching videos based on the target teaching courseware, the target teacher's voice style, and the target teacher's virtual image; The lecture reconstructing module is also used for: The teacher's speech content text and the text style vector are combined with the lecture rewriting prompts and input into the text rewriting model to obtain multiple candidate lecture texts; wherein, the lecture rewriting prompts include content preservation constraints and style fit constraints; Calculate the style fit score and content retention score between each candidate lecture text and the teacher's voice content text; Based on the style fit score and the content retention score, the target teacher's lecture text is determined from each of the candidate lecture texts; The style fit constraint can be expressed as: in, For the style vector of the target teacher, T style For target teacher lecture text, StyleSim ( T style , S teacher This is a style similarity calculation function used to measure whether the style of the transferred text matches the style of the new teacher. δ s This is the constraint threshold for style fit.

7. A teaching video generation device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the instructional video generation method as described in any one of claims 1 to 6.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the teaching video generation method as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the instructional video generation method as described in any one of claims 1 to 6.

Images