Foreign language teaching video generation method and device and computer program product

Abstract

The invention provides a foreign language teaching video generation method and device and a computer program product. The foreign language teaching video generation method comprises the steps of obtaining a video subtitle text corresponding to a to-be-processed video; based on the video subtitle text and a preset explanation generation rule, a foreign language teaching explanation text of the video subtitle text is generated by adopting a large language model; generating a corresponding explanation audio based on the foreign language teaching explanation text; determining a time information relationship between the foreign language teaching explanation text and the explanation audio; according to the time information relationship between the foreign language teaching explanation text and the explanation audio, generating a display style corresponding to the explanation audio; and generating an explanation video corresponding to the to-be-processed video according to the explanation audio, the display style corresponding to the explanation audio and the time information relationship between the explanation text and the explanation audio. According to the invention, the target teaching video corresponding to the to-be-processed video can be automatically generated, the generated teaching video is highly targeted, and personalized requirements of users can be met.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a method, a device and a computer program product for generating a foreign language teaching video based on a large language model. Background Art

[0002] The current foreign language learning market faces a series of challenges, especially for learners who seek personalized, interactive and interesting learning experiences. Traditional teaching videos are often based on fixed teaching materials, which, although structured and systematic, do not meet the interests and needs of all learners. Due to the universality of teaching materials, they often fail to take into account the specific levels of different learners, resulting in some learners feeling that the content is too simple and boring, while other learners may feel that the content is too difficult and feel frustrated. This one-size-fits-all approach limits the flexibility and efficiency of foreign language learning, making it difficult for many learners to find teaching videos that are both suitable for their current level and can stimulate their interest, ultimately affecting the continuity and effectiveness of learning.

[0003] In addition, although automatic subtitles and translation tools have made up for the obstacles in language understanding to a certain extent, their functions are still relatively limited. These tools mainly focus on the literal translation of vocabulary, while ignoring the more complex and subtle parts of language learning, such as the analysis of grammatical structure, the ambiguity of vocabulary, and the understanding of cultural background. As a result, learners often only get the literal meaning of the surface, but cannot deeply understand the deeper meaning and usage scenarios behind the language, which is far from enough to cultivate real language application ability.

[0004] Therefore, it is very necessary to automatically generate personalized foreign language teaching videos according to learners' actual needs and learning progress. Summary of the invention

[0005] The present invention provides a foreign language teaching video generation method, a generation device and a computer program product to generate personalized teaching videos to meet the actual needs of learners.

[0006] The present invention provides a method for generating a foreign language teaching video, which comprises the following steps:

[0007] Get the video subtitle text corresponding to the video to be processed;

[0008] Based on the video subtitle text and the preset explanation generation rules, a large language model is used to generate the foreign language teaching explanation text of the video subtitle text;

[0009] Generate corresponding explanation audio based on foreign language teaching explanation text;

[0010] Determine the time information relationship between foreign language teaching text and audio;

[0011] According to the time information relationship between the foreign language teaching explanation text and the explanation audio, a display style corresponding to the explanation audio is generated;

[0012] An explanation video corresponding to the video to be processed is generated according to the explanation audio, the display style corresponding to the explanation audio, and the time information relationship between the explanation text and the explanation audio.

[0013] According to the foreign language teaching video generation method provided by the present invention, the obtaining of the video subtitle text corresponding to the video to be processed specifically includes: directly obtaining the corresponding video subtitle text from the video to be processed; or

[0014] Get the video to be processed provided by the user;

[0015] Perform segmentation preprocessing on the video to be processed;

[0016] Get the subtitle text corresponding to the preprocessed video and store it.

[0017] According to the foreign language teaching video generation method provided by the present invention, the preset explanation generation rules include system role identity settings, generated content requirements and format requirements of generated results; the generated content requirements include the translation sense style, sentence explanation rules and explanation style.

[0018] According to the foreign language teaching video generation method provided by the present invention, the generation of corresponding explanation audio based on the foreign language teaching explanation text specifically includes:

[0019] The foreign language teaching explanation text is processed into a language style acceptable to the TTS model, and then speech synthesis technology is used to generate the explanation audio.

[0020] Furthermore, the processing of the foreign language teaching explanation text into a language style acceptable to the TTS model includes: marking the foreign language teaching explanation text with different languages, and selecting a suitable TTS model according to different languages;

[0021] The process of language marking of foreign language teaching explanation text is as follows:

[0022] Through the prompt words, the large language model is required to distinguish the explanation text by language and insert separators at the position where the language changes;

[0023] Get the text after being cut by the delimiter, replace the delimiter with the SSML format tag, and generate the SSML document;

[0024] The generated SSML document is passed to the TTS model to generate the explanation audio.

[0025] According to the foreign language teaching video generation method provided by the present invention, the determining of the time information relationship between the foreign language teaching explanation text and the explanation audio comprises:

[0026] The speech recognition technology is used to generate audio subtitles for the explanation content, and the similarity algorithm is used to calculate the matching relationship between the subtitle text and the original explanation text, and the time information of the subtitle text with high similarity is matched to the explanation text.

[0027] Furthermore, the specific process of generating the display style corresponding to the explanation audio according to the time information relationship between the foreign language teaching explanation text and the explanation audio is:

[0028] Calling ASR to convert the explanation audio into text to obtain the explanation audio subtitle file, wherein the explanation audio subtitle file includes text content and timestamp;

[0029] Perform similarity matching on the audio subtitle file and the foreign language teaching explanation text according to the text content, and obtain a format file according to the matching result and the timestamp, wherein the format file includes the foreign language teaching explanation text and its timestamp;

[0030] Generate the display style of foreign language teaching explanation text according to the format file.

[0031] According to the foreign language teaching video generation method provided by the present invention, the method further includes:

[0032] Splicing the explanation video with the video to be processed to generate an integrated video;

[0033] Splice at least two integrated videos together to generate a set of foreign language teaching videos.

[0034] The present invention also provides a foreign language teaching video generation device, which includes a video processing module, a content generation module, a TTS voice generation module, a time information determination module, a subtitle and visual content generation module and a video generation module;

[0035] The video processing module is used to process the acquired video to be processed to obtain a video subtitle text; the content generation module is used to generate a foreign language teaching explanation text of the video subtitle text using a large language model according to the video subtitle text and a preset explanation generation rule; the TTS voice generation module is used to generate a corresponding explanation audio according to the foreign language teaching explanation text; the time information determination module is used to determine the time information relationship between the foreign language teaching explanation text and the explanation audio; the subtitle and visual content generation module is used to generate a display style corresponding to the explanation audio according to the time information relationship between the foreign language teaching explanation text and the explanation audio; the video generation module is used to generate an explanation video corresponding to the video to be processed according to the explanation audio, the display style corresponding to the explanation audio, and the time information relationship between the explanation text and the explanation audio.

[0036] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, it implements any of the above-mentioned foreign language teaching video generation methods.

[0037] According to the above specific implementation of the present invention, it can be known that at least the following beneficial effects are achieved: by adopting the foreign language teaching video generation method provided by the present invention, users can use any video of their interest to generate foreign language teaching material videos without manual operation, and can automatically generate target teaching video clips corresponding to the video to be processed. The generated teaching videos are highly targeted and can meet the personalized needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 is a flow chart of the method for generating a foreign language teaching video provided by the present invention;

[0040] Figure 2 It is a schematic diagram of a style of generating explanation video slices based on the text content of the original video slices in the foreign language teaching video generation method provided by the present invention;

[0041] Figure 3 It is a flow chart of splicing original video slices and explanation video slices in the foreign language teaching video generation method provided by the present invention;

[0042] Figure 4It is a schematic diagram of a specific teaching explanation video generation process in the foreign language teaching video generation method provided by the present invention;

[0043] Figure 5 It is a structural schematic diagram of the foreign language teaching video generating device provided by the present invention.

[0044] Description of reference numerals:

[0045] 11. Video processing module; 12. Content generation module; 13. TTS speech generation module; 14. Time information determination module; 15. Subtitle and visual content generation module; 16. Video generation module. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Combine the following Figure 1 to Figure 5 The invention describes a method and a device for generating a foreign language teaching video.

[0048] like Figure 1 As shown, the foreign language teaching video generation method provided by the present invention comprises the following steps:

[0049] S 1. Obtain the video subtitle text corresponding to the video to be processed.

[0050] S2. Based on the video subtitle text and preset explanation generation rules, a large language model is used to generate a foreign language teaching explanation text of the video subtitle text.

[0051] S3. Generate corresponding explanation audio based on the foreign language teaching explanation text.

[0052] S4. Determine the time information relationship between the foreign language teaching explanation text and the explanation audio.

[0053] S5. Generate a display style corresponding to the foreign language teaching explanation audio according to the time information relationship between the foreign language teaching explanation text and the explanation audio.

[0054] S6. Generate an explanation video corresponding to the video to be processed according to the explanation audio, the display style corresponding to the explanation audio, and the time information relationship between the explanation text and the explanation audio.

[0055] In the above step S1, after obtaining the video to be processed provided by the user, the video may be segmented and pre-processed first, and then the subtitle text corresponding to the segmented video to be processed is obtained and stored. The subtitle text includes text content and time information.

[0056] The video to be processed refers to any video for which an explanation video segment needs to be generated. For example, the video to be processed may be a movie, a TV series, an animation video, etc. The present invention does not limit the type of the video to be processed.

[0057] The present invention does not limit the preprocessing method of the video to be processed.

[0058] In a possible implementation of the present invention, if the video to be processed is a long video file, the long video can be cut into two or more short-duration video segments based on a preset duration. For example, each short-duration video segment can be 3 minutes; or based on the time information of the subtitle text corresponding to the video to be processed, the long video can be accurately sliced ​​to ensure the integrity of the content of each video slice.

[0059] The present invention does not limit the method for obtaining the subtitle text corresponding to the video to be processed.

[0060] In a possible implementation of the present invention, the audio of the video to be processed can be separated by audio extraction, and the audio file can be speech recognized by ASR speech-to-text technology to obtain the corresponding audio text, and the audio text can be used as the video subtitle text of the video to be processed. The corresponding video subtitle text can also be directly obtained according to the subtitle file of the video to be processed. The present invention does not limit the audio extraction tool and speech recognition technology used.

[0061] In a possible implementation of the present invention, the original video can be segmented based on SRT (SubRip Text, subtitle file) using text and corresponding timestamp information, and the segmented segments can be used as material files for subsequent splicing with the explanation video. As shown in Table 1, after processing, the correspondence between the text sentence, the timestamp, and the corresponding original video slice can be obtained, and the correspondence information can be stored as subsequent data input.

[0062] Table 1 Correspondence between original video subtitle text and timestamp

[0063]

[0064]

[0065] In the above step S2, the video subtitle text is processed by calling a large language model, and the detailed explanation content for each sentence is automatically generated according to the preset explanation generation rules.

[0066] There is no restriction on the preset explanation generation rules. The preset explanation generation rules may include analysis of sentence structure, grammar and word meaning, and may also be combined in a configurable form according to the user's learning preferences.

[0067] Specifically, the preset explanation generation rules may include, but are not limited to: system role identity settings, generated content requirements, and generated result format requirements. Among them, the generated content requirements include the sense of language style of translation, sentence explanation rules, and explanation style.

[0068] In a possible implementation of the present invention, the mother tongue and target learning language can be selected on the user front-end page, or the target learning language can be automatically obtained by sampling the video, and then the selected content is passed as a parameter to the generation rule. The system supports multiple language cross-combinations. For example, the user's mother tongue can be Chinese and the target learning language can be Egyptian dialect, or the user's mother tongue can be Korean and the target learning language can be English. The support range is jointly affected by the support range of ASR (Automatic Speech Recognition), large language model, and TTS (Text To Speech).

[0069] In a possible implementation of the present invention, based on preset explanation generation rules, prompt words can be set, wherein the prompt words may include but are not limited to role setting sentences, translation sense style, sentence explanation rules, explanation style and format rule setting sentences.

[0070] Role setting: The system role identity can be set as a language teaching expert, or the rules can be superimposed by obtaining the user's selected native language and target language.

[0071] For example, "You are a [Japanese] language teaching expert for [Chinese] native-speaking students. Please consider the language sense and habits of the students' native language when generating explanations", where [Chinese] and [Japanese] are the received transmission parameters. This allows the large language model to generate explanations that are more suitable for the language users to understand based on the language characteristics of the student's native language when outputting.

[0072] Translation style: The meaning of the sentence is translated so that the user has an overall understanding of the meaning of the sentence to be learned. The language model can be set to ensure that the translation is faithful to the meaning and sense of the original text. In a possible implementation of the present invention, the context of the sentence can also be passed to the language model as background information to make the translation more in line with the original context.

[0073] The sentence explanation rule is to set a teaching explanation method based on the original text in the prompt words to guide users to learn the grammatical knowledge points involved in the video content in a certain way and angle. The present invention does not limit the specific explanation rule design.

[0074] In a possible implementation method, the explanation rules may include word order interpretation rules, grammatical function interpretation rules, context interpretation rules, example and metaphor rules, etc.

[0075] Among them, the word order interpretation rules refer to: explaining the order of words in a sentence and explaining the influence of the position of different words in a sentence on the meaning of the sentence. For example, the word order of subject, predicate and object may have different importance in different languages, and its logic needs to be explained clearly.

[0076] Grammatical function interpretation rules refer to: explaining the specific grammatical functions played by words, phrases or sentences in a sentence, which helps users understand the structure and logic of the sentence.

[0077] The contextual interpretation rule means that the specific context of the sentence is considered, including the context and the intention of the language user. When interpreting, relevant background information can be provided to help understand the meaning and usage of the sentence.

[0078] The example and metaphor rule means that by using examples or metaphors to illustrate language phenomena, abstract concepts can be made more concrete and easier to understand. This method helps listeners or readers connect abstract concepts with concrete situations.

[0079] As a possible implementation method, rule routing matching can also be performed for different target learning languages. Since the structures of different languages ​​are different, they can also be preset in advance according to the language type to achieve a more refined explanation effect. Then, according to the target learning language selected by the user, the sentence explanation rules in the database are called and spliced ​​into the overall prompt words to be generated.

[0080] Explanation style: directly affects the user's understanding of the information. A clear and concise explanation style can be more easily understood by the recipient.

[0081] Format rules for generated content: Since audio will be generated later, the large language model can be required to generate the format at this stage to facilitate the subsequent TTS text preprocessing. The main purpose of the format requirement is to optimize the auditory effect after the audio is generated. Affected by the training data structure of the large language model used, when no format requirement is made, the model will generate the format required during training by default, which may be inconsistent with the required format style. For example, a sequence format such as 1., 2., 3. may be generated in front of each paragraph of content, or unexpected symbols may be used to wrap some text parts, such as "" to wrap foreign language parts. These additional contents may cause interruptions, sudden reading of symbol names, and other unexpected effects after TTS generates audio. Therefore, it is necessary to set the format requirements for the generated content in advance at this stage, or use a large language model that has been trained and can generate content according to the expected format rules. And before generating audio, the data format should be cleaned to remove characters or format styles that do not meet the requirements to prevent auditory interruptions.

[0082] In a possible implementation of the present invention, when a general large language model without fine-tuning is used for generation, a few-shot (a small number of sample prompts) or a hard requirement method can be used to constrain the format. The few-shot method can be used in the prompt words to provide the large language model with the expected generation format and format samples; or a hard requirement method can be used, such as "do not generate text with xx symbols", to prevent the large language model from continuing to generate incorrect format content according to the training style. In actual operation, both methods usually appear.

[0083] In a possible implementation of the present invention, a large language model can be trained according to the expected formatting rules. In the long run, this method is more stable and controllable, but it is still necessary to prepare enough formatted data in the early stage of the project to fine-tune the model.

[0084] In a possible implementation manner of the present invention, the prompt words may also be packaged in a componentized manner, and support the user to define the combination range of each prompt word.

[0085] In the above step S3, the generated explanation text is processed into a language style acceptable to the TTS model, and then the speech synthesis technology (TTS) is called to generate the explanation audio. In order to enhance the accuracy effect, the speech synthesis technology for multi-language effect enhancement can be used for texts where multiple languages ​​frequently appear. The present invention does not limit the selected speech synthesis technology.

[0086] In a possible implementation of the present invention, the text is firstly annotated with the language, and a suitable TTS model is selected according to different languages ​​to improve the pronunciation accuracy of the final audio.

[0087] Due to the need for refined audio generation, the explanation text needs to be processed into a more refined markup language acceptable to TTS before calling the TTS model. In text-to-speech (TTS) technology, SSML (Speech Synthesis Markup Language) is a common markup language used to define various attributes of speech, such as sound model, language, speech speed, etc. The selected markup language is related to the selected TTS model. Here we explain the processing steps for converting the explanation text into the markup language, which includes:

[0088] (1) Prompt words are used to require the large language model to distinguish the explanation text by language and insert separators at the position where the language changes. For example, if the explanation text is a mixture of Chinese and Japanese, the prompt words can be used to require that the non-Chinese or Japanese parts be wrapped with separators.

[0089] (2) After obtaining the text divided by the delimiter, the delimiter is replaced with an SSML format tag to generate an SSML document.

[0090] (3) Pass the generated SSML document to the TTS model to generate the explanation audio.

[0091] Whether such language-based formatting is needed depends on the capabilities of the selected TTS model. In a possible implementation of the present invention, a TTS model that supports multi-language mixing can also be used. If a TTS model that supports multi-language mixed generation is selected, it is more simplified, and only the required parameters need to be declared in the table header, without the need for specific label or parameter declaration processing based on the language in the text content.

[0092] In the above step S4, after obtaining the audio file of the explanation audio, the speech recognition technology (ASR) is used to generate the explanation audio subtitle text for the explanation audio, and the similarity algorithm is used to calculate the matching relationship between the subtitle text and the original explanation text, and the time information of the subtitle text with high similarity is matched to the explanation text to ensure the accuracy of the displayed content. The main purpose of this step is to obtain the time information of the explanation text in the explanation audio.

[0093] The explanation audio is converted into text using speech recognition technology (ASR) to generate a text file with explanation audio subtitle text and timestamp. In the text file, the explanation audio subtitle text has a timestamp per sentence, and the timestamp expresses the start time information and end time information of the sentence. The explanation audio subtitle text is not used to be displayed on the screen, but is used for similarity matching with the explanation text to match the timestamp of the explanation text.

[0094] In a possible implementation of the present invention, a Levenshtein distance algorithm may be used when processing text matching. The algorithm calculates the Levenshtein distance between two strings and then converts the distance into a similarity score using the following formula:

[0095]

[0096] Among them, max_length represents the length of the longer string of the two strings, and distance represents the Levenshtein distance between the two strings.

[0097] The basic idea of ​​this algorithm is that the smaller the edit distance between two strings (i.e. the higher the similarity), the higher their similarity score. Edit distance refers to the minimum number of editing operations required to convert from one string to another, including operations such as inserting, deleting, and replacing characters. It is suitable for comparing the similarity between two texts, for example, in scenarios such as spelling correction and search suggestions, and can consider character-level editing operations. It is suitable for short texts and character-level matching.

[0098] In actual scenarios, a suitable similarity calculation method can be selected according to data characteristics, and is not limited in the embodiments of the present invention.

[0099] In the above step S5, after obtaining the correspondence between the explanation text and the time information, the presentation style of the explanation text can be created. The text presentation is intended to enhance understanding and memory and keep the audio and explanation content synchronized. After generating the presentation content, the correspondence between the text, presentation content and time information can be obtained. The correspondence can be stored here, so that the explanation text can be presented according to time when the video is generated later, that is, the explanation text content is presented within the audio playback time corresponding to the text. The presentation style can be a subtitle style or a picture style, which is not limited in the present invention.

[0100] Specifically, according to the time information relationship between the foreign language teaching explanation text and the explanation audio, the specific process of generating the display style corresponding to the explanation audio is as follows:

[0101] S51, calling ASR to convert the explanation audio into text to obtain an explanation audio subtitle file, wherein the explanation audio subtitle file includes text content and a timestamp.

[0102] S52, performing similarity matching on the explanation audio subtitle file and the foreign language teaching explanation text according to the text content, and obtaining a format file according to the matching result and the timestamp, wherein the format file includes the foreign language teaching explanation text and its start time and end time.

[0103] S 53. Generate a display style of the foreign language teaching explanation text according to the format file, which display style may be a subtitle style or a picture style, etc. As a preferred implementation, the original video subtitle text and the corresponding original video screenshot may be used as a background display of the explanation video, and the display style is superimposed on the background display.

[0104] As a specific example, Figure 2 As shown, the presentation style is a grammar explanation text, which is displayed on the background of the original video subtitle text.

[0105] In a possible implementation of the present invention, the generated subtitles or pictures can be burned into the video, which is convenient for users to view at any time without the need for a network environment.

[0106] In a possible implementation of the present invention, a front-end mask layer can also be added to the page to allow the user to set the control to turn on or off subtitles (or other visual styles), so that when the user is learning advanced, he can view the foreign language explanation content on demand to enhance the connection memory.

[0107] In the above step S6, the explanation audio and the display style are merged based on the time information relationship between the explanation text and the explanation audio to generate an explanation video clip. In this way, each sentence of the explanation text is accompanied by detailed linguistic explanation audio and screen display content, which is conducive to improving the efficiency of language learning.

[0108] In a possible implementation of the present invention, the explanation video clips can be spliced ​​with the original video slices, so that the original video slices are played first, and then the linguistic explanations for the sub-sections in the video are played, thereby enhancing user memory.

[0109] For ease of understanding, Figure 3 As shown, the original video slice and the explanation video generated based on the text in the original video slice can be spliced ​​to obtain a spliced ​​video. Among them, the front part of the spliced ​​video is the original video slice, and the back part is the explanation video, which can enhance user memory.

[0110] In a possible implementation of the present invention, after a spliced ​​video is generated, a plurality of spliced ​​videos may be spliced ​​together to output a set of foreign language teaching videos.

[0111] It can be seen from the entire process of the present invention that the user only needs to provide the video to be processed during the entire process, and the remaining process will be automatically processed by the present invention, and finally a video accompanied by original sound slices and detailed explanations will be output, and no user intervention is required in the entire process.

[0112] In order to facilitate the understanding of the whole process, Figure 4As shown, a specific teaching explanation video generation process is: extract audio from the video to be processed to obtain an audio file; perform speech recognition on the audio file to obtain video subtitle text; input the video subtitle text into the large language text generation model to obtain the teaching explanation text; perform speech synthesis on the teaching explanation text to obtain the teaching explanation audio; generate the teaching explanation video based on the teaching explanation audio.

[0113] By adopting the foreign language teaching video generation method provided by the present invention, users can use any video of their interest to generate foreign language teaching material videos, without manual operation, and can automatically generate target teaching video clips corresponding to the video to be processed. The user's native language and the target learning language can be cross-selected arbitrarily, so the scalability is extremely high. The user can choose the teaching structure combination by himself to expand more personalized learning content. When generating the explanation voice, the effect is optimized for high-frequency cross-mixed languages, which can ensure that the explanation content and the pronunciation of the original sound are accurate and clear. The explanation content is automatically displayed in the video following the timeline, which is convenient for users to intuitively understand the text content when listening to the explanation. The present invention can be applied to mobile terminals and adapted according to the size of the mobile terminal. The overall end-to-end accuracy of the present invention is controllable.

[0114] In an exemplary embodiment, based on the foreign language teaching video generation method provided by the present invention, as Figure 5 As shown, the present invention also provides a foreign language teaching video generation device, which includes a video processing module 11, a content generation module 12, a TTS voice generation module 13, a time information determination module 14, a subtitle and visual content generation module 15 and a video generation module 16.

[0115] Among them, the video processing module 11 is used to process the acquired video to be processed to obtain the video subtitle text. The content generation module 12 is used to generate the foreign language teaching explanation text of the video subtitle text using a large language model according to the video subtitle text and the preset explanation generation rules. The TTS speech generation module 13 is used to generate the corresponding explanation audio according to the foreign language teaching explanation text. The time information determination module 14 is used to determine the time information relationship between the foreign language teaching explanation text and the explanation audio. The subtitle and visual content generation module 15 is used to generate the display style corresponding to the explanation audio according to the time information relationship between the foreign language teaching explanation text and the explanation audio. The video generation module 16 is used to generate the explanation video corresponding to the video to be processed according to the explanation audio, the display style corresponding to the explanation audio, and the time information relationship between the explanation text and the explanation audio.

[0116] Specifically, the video processing module 11 cuts the video into short video clips, decodes each video clip, extracts the video stream and the audio stream, and converts the audio file into a subtitle text with a timestamp using the ASR speech-to-text technology. The video processing module can improve the conversion accuracy by segmenting the speech, filtering the noise, and recognizing the dialect.

[0117] The content generation module 12 uses a large language model to process the extracted subtitle text and automatically generates detailed explanation content for each sentence according to preset explanation generation rules. The preset explanation generation rules may include analyzing the structure, grammar and meaning of the sentence, or may be combined in a configurable form according to the user's learning preferences.

[0118] The TTS speech generation module 13 generates the explanation audio using text-to-speech technology (TTS); the generated explanation content is language-labeled to provide the accuracy of subsequent TTS speech synthesis. When multilingual pronunciation is involved, a suitable TTS model can be selected according to different languages ​​to improve accuracy, or a multilingual mixed TTS model with good effect can be used.

[0119] The time information determination module 14 obtains the corresponding explanation audio subtitle file based on the audio file of the explanation audio, and determines the time information of the explanation text in the explanation audio based on the correspondence between the subtitle text of the explanation audio and the explanation text, so as to achieve time synchronization between the explanation text and the explanation audio.

[0120] The subtitle and visual content generation module 15 generates synchronized subtitles according to the explanation content, or displays them in other visual styles, such as images, front-end masks, etc. The design of visual content is intended to enhance understanding and memory while maintaining synchronization with the explanation content.

[0121] The video generation module 16 splices the processed audio, subtitles and visual content with the original video slices and outputs the final teaching video, in which each sentence is accompanied by detailed linguistic explanations and visual aids.

[0122] It should be noted that the foreign language teaching video generating device and the foreign language teaching video generating method provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0123] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the processor can execute the foreign language teaching video generation method provided by the above methods, the method includes:

[0124] Get the video subtitle text corresponding to the video to be processed;

[0125] Based on the video subtitle text and the preset explanation generation rules, a large language model is used to generate the foreign language teaching explanation text of the video subtitle text;

[0126] Generate corresponding explanation audio based on foreign language teaching explanation text;

[0127] Determine the time information relationship between foreign language teaching text and audio;

[0128] According to the time information relationship between the foreign language teaching explanation text and the explanation audio, a display style corresponding to the explanation audio is generated;

[0129] An explanation video corresponding to the video to be processed is generated according to the explanation audio, the display style corresponding to the explanation audio, and the time information relationship between the explanation text and the explanation audio.

[0130] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0131] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a foreign language teaching video, characterized in that: The following steps are involved: Get the video subtitle text corresponding to the video to be processed; Based on the video subtitle text and the preset explanation generation rules, a large language model is used to generate the foreign language teaching explanation text of the video subtitle text; Generate corresponding explanation audio based on foreign language teaching explanation text; Determine the time information relationship between foreign language teaching text and audio; According to the time information relationship between the foreign language teaching explanation text and the explanation audio, a display style corresponding to the explanation audio is generated; An explanation video corresponding to the video to be processed is generated according to the explanation audio, the display style corresponding to the explanation audio, and the time information relationship between the explanation text and the explanation audio.

2. The method for generating foreign language teaching videos according to claim 1, characterized in that: The obtaining of the video subtitle text corresponding to the video to be processed specifically includes: directly obtaining the corresponding video subtitle text from the video to be processed; or Get the video to be processed provided by the user; Perform segmentation preprocessing on the video to be processed; Get the subtitle text corresponding to the preprocessed video and store it.

3. The method for generating foreign language teaching videos according to claim 1, characterized in that: The preset explanation generation rules include system role identity settings, generated content requirements and format requirements of generated results; the generated content requirements include the language sense style of translation, sentence explanation rules and explanation style.

4. The method for generating foreign language teaching videos according to claim 1, characterized in that: The generating of corresponding explanation audio based on the foreign language teaching explanation text specifically includes: The foreign language teaching explanation text is processed into a language style acceptable to the TTS model, and then speech synthesis technology is used to generate the explanation audio.

5. The method for generating foreign language teaching videos according to claim 4, characterized in that: The processing of the foreign language teaching explanation text into a language style acceptable to the TTS model includes: marking the foreign language teaching explanation text with different languages ​​and selecting appropriate TTS models according to different languages; The process of language marking of foreign language teaching explanation text is as follows: Through the prompt words, the large language model is required to distinguish the explanation text by language and insert separators at the position where the language changes; Get the text after being cut by the delimiter, replace the delimiter with the SSML format tag, and generate the SSML document; The generated SSML document is passed to the TTS model to generate the explanation audio.

6. The method for generating foreign language teaching videos according to claim 1, characterized in that: Determining the time information relationship between the foreign language teaching explanation text and the explanation audio includes: The speech recognition technology is used to generate audio subtitles for the explanation content, and the similarity algorithm is used to calculate the matching relationship between the subtitle text and the original explanation text, and the time information of the subtitle text with high similarity is matched to the explanation text.

7. The method for generating foreign language teaching videos according to claim 1, characterized in that: The specific process of generating the display style corresponding to the explanation audio according to the time information relationship between the foreign language teaching explanation text and the explanation audio is as follows: Calling ASR to convert the explanation audio into text to obtain the explanation audio subtitle file, wherein the explanation audio subtitle file includes text content and timestamp; Perform similarity matching on the audio subtitle file and the foreign language teaching explanation text according to the text content, and obtain a format file according to the matching result and the timestamp, wherein the format file includes the foreign language teaching explanation text and its timestamp; Generate the display style of foreign language teaching explanation text according to the format file.

8. The method for generating foreign language teaching videos according to claim 1, characterized in that: The method further includes: Splicing the explanation video with the video to be processed to generate an integrated video; Splice at least two integrated videos together to generate a set of foreign language teaching videos.

9. A device for generating foreign language teaching videos, characterized in that: It includes a video processing module, a content generation module, a TTS voice generation module, a time information determination module, a subtitle and visual content generation module, and a video generation module; The video processing module is used to process the acquired video to be processed to obtain a video subtitle text; the content generation module is used to generate a foreign language teaching explanation text of the video subtitle text using a large language model according to the video subtitle text and a preset explanation generation rule; the TTS voice generation module is used to generate a corresponding explanation audio according to the foreign language teaching explanation text; the time information determination module is used to determine the time information relationship between the foreign language teaching explanation text and the explanation audio; the subtitle and visual content generation module is used to generate a display style corresponding to the explanation audio according to the time information relationship between the foreign language teaching explanation text and the explanation audio; the video generation module is used to generate an explanation video corresponding to the video to be processed according to the explanation audio, the display style corresponding to the explanation audio, and the time information relationship between the explanation text and the explanation audio.

10. A computer program product, characterized in that It includes a computer program, which can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, it implements the foreign language teaching video generation method according to any one of claims 1 to 8.

Images