A method, apparatus, electronic device, and storage medium for generating teaching materials and videos.

Abstract

This invention discloses a method, apparatus, electronic device, and storage medium for generating teaching scripts and teaching videos. The method includes: generating script text corresponding to each slide image based on the text content of the presentation data; generating a basic video with spoken audio based on the script text corresponding to each slide image; aligning the script text with the ASR word timeline corresponding to the basic video character by character to obtain a subtitle timeline, and burning subtitles into the basic video based on the subtitle timeline and the script text to obtain an intermediate video; determining the precise location result of the target word based on the original text box location result and the OCR recognition location result, and performing visual annotation processing on the target word appearing in the intermediate video based on the precise location result to obtain the finished video. This solution can significantly improve the subtitle time accuracy and target word annotation location accuracy of the generated finished video, and greatly reduce the iterative update cost.

Description

Technical Field

[0001] This invention relates to the field of computer processing technology, and in particular to a method, apparatus, electronic device, and storage medium for generating teaching materials and teaching videos. Background Technology

[0002] With the rapid growth of the online education and corporate training market, instructional videos have become a core medium for knowledge dissemination. In existing instructional video production technologies, if any step is modified (such as adjusting the lecture script), all subsequent steps must be redone, and the entire video must be re-rendered. Furthermore, the accuracy of subtitle timing is insufficient: existing speech recognition tools typically output timestamps at the sentence level, with a typical accuracy of ±300 to 500 milliseconds, but such sentence-level timestamps are far from meeting practical needs. Therefore, existing instructional video production technologies have many problems that require improvement and solutions. Summary of the Invention

[0003] This invention provides a method, apparatus, electronic device, and storage medium for generating teaching scripts and teaching videos, which can realize end-to-end automated teaching video generation. At the same time, the subtitle timing accuracy of the generated videos is significantly improved, the accuracy of target word annotation and positioning in the screen is significantly improved, and the cost of iterative updates can be greatly reduced.

[0004] In a first aspect, the present invention provides a method for generating teaching scripts and teaching videos, comprising: Based on the text content of the presentation data, a script text corresponding to each slide image is generated, and the script text is used to verbally explain the text content. Based on the lecture text corresponding to each slide image, a basic video with spoken audio is generated. The script text is aligned word-by-word with the ASR word timetable corresponding to the base video to obtain the subtitle timeline. The base video is then burned with subtitles based on the subtitle timeline and the script text to obtain the intermediate video. The ASR word timetable is generated based on the audio data contained in the base video, and the subtitle timeline is used to indicate the start and end times of the display of each sentence in the script text in the subtitles of the intermediate video. Based on the localization results of the original text box corresponding to the target word and the OCR recognition results, the precise localization result of the target word is determined, and the target word appearing in the intermediate video frame is visually annotated based on the precise localization result to obtain the finished video.

[0005] Secondly, the present invention also provides a teaching script and teaching video generation device, comprising: The presentation text generation module is used to generate presentation text corresponding to each slide image based on the text content of the presentation data. The presentation text is used to verbally explain the text content. The basic video generation module is used to generate a basic video with spoken audio based on the lecture text corresponding to each slide image; The intermediate video generation module is used to align the lecture text with the ASR word timetable corresponding to the base video word by word to obtain the subtitle timeline, and to burn subtitles into the base video based on the subtitle timeline and the lecture text to obtain the intermediate video; wherein, the ASR word timetable is generated based on the audio data contained in the base video, and the subtitle timeline is used to indicate the start and end times of the display of each lecture sentence in the lecture text in the subtitles of the intermediate video; The finished video generation module is used to determine the precise positioning result of the target word based on the original text box positioning result and the OCR recognition positioning result, and to perform visual annotation processing on the target word appearing in the intermediate video frame based on the precise positioning result to obtain the finished video.

[0006] Thirdly, this invention also provides an electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the teaching script and teaching video generation method provided in any embodiment of the present invention.

[0007] Fourthly, embodiments of the present invention also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the teaching script and teaching video generation method provided in any embodiment of the present invention.

[0008] The technical solution of this invention involves generating a script text corresponding to each slide image based on the text content of the presentation data. This script text is used for oral explanation of the text content. Then, based on the script text corresponding to each slide image, a base video with oral audio is generated. Next, the script text and the ASR word timeline corresponding to the base video are aligned character by character to obtain a subtitle timeline. Subtitles are then burned into the base video based on the subtitle timeline and the script text to obtain an intermediate video. The ASR word timeline is generated based on the audio data contained in the base video, and the subtitle timeline indicates the start and end times of the display of each sentence in the script text in the subtitles of the intermediate video, thereby achieving millisecond-level (±50ms) character-by-character time alignment accuracy, resulting in a several-fold improvement in alignment accuracy. Finally, based on the original text box positioning result and OCR recognition positioning result corresponding to the target word, the precise positioning result of the target word is determined. The precise positioning result of the target word is determined by a dual-source fusion positioning method, and the target word appearing in the intermediate video is visually labeled based on the precise positioning result to obtain the finished video. By decoupling the processing content at each stage of the video generation process, this solution enables end-to-end automated teaching video generation. At the same time, the subtitle timing accuracy of the generated video is significantly improved, the accuracy of target word annotation and positioning in the picture is significantly improved, and the cost of iteration and update is also greatly reduced.

[0009] The above description of the invention is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0011] Figure 1 This is a flowchart illustrating a method for generating teaching materials and teaching videos according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating another method for generating teaching materials and teaching videos provided in an embodiment of the present invention; Figure 3 A schematic diagram of a teaching script and teaching video generation device provided in an embodiment of the present invention; Figure 4This is a schematic diagram of the structure of an electronic device for implementing a method for generating teaching manuscripts and teaching videos, as provided in an embodiment of the present invention. Detailed Implementation

[0012] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0013] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0014] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0015] It should be noted that the concepts of "first" and "second" mentioned in this invention are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0016] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0017] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0018] Figure 1 This is a flowchart illustrating a method for generating teaching materials and videos according to an embodiment of the present invention. This embodiment is applicable to generating teaching videos based on presentation text data. The method can be executed by a teaching material and video generation device, which can be implemented in software and / or hardware and is generally integrated into any electronic device with network communication capabilities, such as a mobile terminal, PC, or server. Figure 1 As shown, the teaching script and teaching video generation method of this invention embodiment may include the following process: S110. Based on the text content of the presentation data, generate the script text corresponding to each slide image. The script text is used to verbally explain the text content.

[0019] Presentation data consists of multiple slide images, each containing specific text content. For example, presentation data could be a PowerPoint presentation, which could consist of multiple slide images, each containing specific text content related to medical knowledge.

[0020] Correspondingly, the lecture script can be understood as the text used for verbal explanation of the text content in the presentation data; in other words, it can serve as a teaching script. Compared to the text content in the presentation data, the lecture script, in addition to the original text content, also includes supplementary text such as connecting phrases, explanations, and transitional phrases needed for verbal explanation. In other words, the lecture script contains richer text content than the original presentation data, making it more suitable for verbal explanation.

[0021] Specifically, based on the text content of the presentation data, the script text corresponding to each slide image can be generated separately. For example, for a single slide image, the title, body text, structure, and optional notes of that slide image can be used as constraints, and a large language model can be called to generate the corresponding script text for that slide image. This script text will then be used as the teaching script for subsequent oral presentations. This script text not only retains the key terms and knowledge points originally present in the slide image but also adds necessary connecting phrases, explanations, and transitions. Following the same principle, script text corresponding to each slide image can be generated separately.

[0022] As an optional but non-limiting implementation, the presentation text for each slide image is generated based on the text content of the presentation data. This includes: breaking down the presentation data page by page to obtain multiple slide images; for a single slide image, generating the corresponding presentation text based on the text box coordinates and content of the text boxes; wherein the text box coordinates are used to determine the position of the text box content within the slide image. Using this optional approach, the corresponding presentation text can be generated based on the text box coordinates and content of the slide images.

[0023] The text box coordinates include its left edge, top edge, width, and height. These coordinates are used to determine the position of the text box content within the slide image.

[0024] Specifically, the presentation data can be broken down page by page to obtain multiple slide images. It's important to note that presentation data stores text content at the text box level, with each text box containing its corresponding text content. Therefore, the coordinate information of the text boxes can be used to accurately locate them and determine their content. In other words, for a single slide image, the text content of that slide image can be determined based on its text box coordinates and content, thus generating the corresponding presentation text. The same principle can be applied to generate the presentation text for each slide image.

[0025] For example, given received presentation data (such as PowerPoint slides), the python-pptx library (a standard library in the Python ecosystem for automating PPT processing and extraction) can be used to break down the presentation data page by page, resulting in multiple slide images. Then, for a single slide image, python-pptx can extract the coordinate information (left, top, width, height) of all text boxes and their text content for each slide image, and further call a large language model to generate the corresponding lecture text for that slide image. The same principle applies to the processing of other slide images.

[0026] S120. Generate a basic video with spoken audio based on the text of the presentation corresponding to each slide image.

[0027] The spoken audio refers to the audio used to verbally explain the content of the lecture text. Correspondingly, the basic video can be understood as a preliminarily generated video, which still needs further processing in subsequent steps. Optionally, in this embodiment, the basic video may include footage of a virtual character giving the spoken explanation. The specific virtual character can be set to different roles based on actual needs; for example, the virtual character could be a virtual character acting as a teacher. It is understood that having such a scene-appropriate virtual character in the generated video can make the video content and teaching effect more vivid and engaging.

[0028] Specifically, the text of the speech corresponding to each slide image can be sent to the digital human synthesis platform to generate a base video with spoken audio. This base video can also include footage of a virtual character speaking.

[0029] As an optional but non-limiting implementation, the base video includes a virtual character speaking on screen. Correspondingly, based on the script text corresponding to each slide image, a base video with spoken audio is generated, including: for a single slide image, generating corresponding audio data based on the script text corresponding to the slide image, and rendering and controlling the virtual character based on the generated audio data to obtain a video segment corresponding to the slide image; and splicing the video segments corresponding to each slide image to obtain the base video.

[0030] Specifically, for a single slide image, corresponding audio data can be generated based on the corresponding text in the presentation script. This can be achieved, for example, using a text-to-speech (TTS) engine. Furthermore, the generated audio data is used to render and control a virtual character. For instance, facial movements can be controlled to mimic lip movements during speech, resulting in a video clip corresponding to that slide image. This video clip includes the virtual character's narration, creating a visual effect of the virtual character giving a presentation. The same process applies to other slide images. Then, the video clips corresponding to each slide image can be stitched together to obtain a base video. For example, the FFmpeg concat protocol can be used to stitch all the video clips together into a complete base video (base_video.mp4).

[0031] S130. Align the script text with the ASR word timeline corresponding to the base video word timeline word by word to obtain the subtitle timeline, and burn subtitles into the base video based on the subtitle timeline and the script text to obtain the intermediate video; wherein, the ASR word timeline is generated based on the audio data contained in the base video, and the subtitle timeline is used to indicate the start and end times of the display of each script sentence in the subtitles of the intermediate video.

[0032] The ASR word timetable is a time-stamped transcription of audio into text, and is a type of structured data. This ASR word timetable contains text (characters or words) identified from the speech, as well as word-level timestamp information. The word-level timestamp information indicates the start and end times (i.e., the appearance and end times) of each identified word. In this embodiment, the ASR word timetable is generated based on the audio data contained in the underlying video.

[0033] Intermediate video can be understood as the video obtained after subtitle burning into the basic video.

[0034] The subtitle timeline indicates the start and end times for each sentence in the speech text to appear as subtitles in the video. Using the subtitle timeline, each sentence can be precisely recorded into the video, ensuring synchronization between the audio data and the displayed subtitles.

[0035] Understandably, the script text and the audio data of the base video are not yet linked at this point. It is impossible to know which sentence in the script corresponds to which scene or audio in the base video. Therefore, in order to accurately record each sentence in the script text into the video, it is necessary to determine the subtitle timeline so as to accurately record the subtitles and achieve synchronization between the audio data and the displayed subtitles.

[0036] Specifically, since the ASR word timetable is generated based on the audio data contained in the base video, and the base video is generated based on the script text, the script text and the corresponding ASR word timetable of the base video can be aligned word by word. That is, using the script text as a baseline, each sentence or word in the script is matched word by word with the ASR word timetable corresponding to the synthesized audio (i.e., the ASR word timetable of the base video). This process removes pauses, omissions, rewrites, homophone substitutions, and punctuation differences, thereby determining the start and end times of each script sentence. After the alignment is complete, a subtitle timeline is obtained. This subtitle timeline indicates the start and end times when each script sentence will be displayed in the subtitles of the intermediate video. Then, subtitles can be burned into the base video based on the subtitle timeline and the script text to obtain the intermediate video.

[0037] S140. Based on the localization results of the original text box corresponding to the target word and the OCR recognition localization results, determine the precise localization result of the target word, and perform visual annotation processing on the target word appearing in the intermediate video frame based on the precise localization result to obtain the finished video.

[0038] In this context, target words can be understood as keywords that need to be highlighted. For example, in the case of an instructional video, target words such as "central nervous system," "spinal cord," and "synapse" can be highlighted.

[0039] Native text box location results refer to the location results of the target word obtained by using the relevant information of the text boxes in the presentation data. In contrast, OCR recognition location results refer to the location results of the target word obtained by using OCR (Optical Character Recognition) technology. These two are essentially location results obtained for the same target word using different methods.

[0040] The finished video can be understood as the final video produced, which can be used for teaching and explanation, etc.

[0041] Specifically, for the target word, the relevant information of the text boxes in the presentation data can be used to determine the original text box location result; alternatively, OCR recognition technology can be used to determine the OCR recognition location result. Then, based on the original text box location result and the OCR recognition location result corresponding to the target word, the precise location result of the target word can be determined. This precise location result can be understood as the final determined location result of the target word, which is relatively more accurate and can indicate the precise pixel coordinates of the target word in order to locate the target word. Subsequently, visual annotation processing can be performed on the target word appearing in the intermediate video frame based on the precise location result, thereby obtaining the finished video. In this embodiment, the visual annotation processing includes, but is not limited to: underlining processing, highlighting processing, and bounding box processing (such as using circles or ellipses to circle the target word), so that the target word can be highlighted and achieve a striking highlighting effect. Of course, other types of visual annotation processing can also be performed based on actual needs, which are not limited here.

[0042] It should be noted that the four-stage processing of generating the finished video in this embodiment is as follows: courseware parsing and script generation, digital human video synthesis, subtitle timeline generation and burning, and visual annotation processing of target words. These four stages are decoupled pipelines, the state of each stage can be recorded, and intermediate states can be passed between stages through standardized manifest files, supporting independent recalculation of any stage. This architecture breaks through the limitations of existing solutions that require "one-time generation, full redo". For example, when it is necessary to modify the subtitle timeline, adjust the subtitle style, or annotate target words, partial recalculation can be performed by stage / page / object. This allows the video segments, subtitles, and annotation results of unchanged pages to be reused. Only the subtitle stage or annotation stage needs to be reprocessed, without re-synthesizing the digital human video (the most time-consuming stage), significantly improving iteration efficiency by several times. At the same time, four pipeline modes are provided: video_only (generate only the basic video), subtitle_burn_only (burn only the subtitles), annotation_only (add annotation only), and video_and_annotation (full process), allowing users to choose the execution granularity as needed.

[0043] The technical solution of this invention involves generating a script text corresponding to each slide image based on the text content of the presentation data. This script text is used for oral explanation of the text content. Then, based on the script text corresponding to each slide image, a base video with oral audio is generated. Next, the script text and the ASR word timeline corresponding to the base video are aligned character by character to obtain a subtitle timeline. Subtitles are then burned into the base video based on the subtitle timeline and the script text to obtain an intermediate video. The ASR word timeline is generated based on the audio data contained in the base video, and the subtitle timeline indicates the start and end times of the display of each sentence in the script text in the subtitles of the intermediate video, thereby achieving millisecond-level (±50ms) character-by-character time alignment accuracy, resulting in a several-fold improvement in alignment accuracy. Finally, based on the original text box positioning result and OCR recognition positioning result corresponding to the target word, the precise positioning result of the target word is determined. The precise positioning result of the target word is determined by a dual-source fusion positioning method, and the target word appearing in the intermediate video is visually labeled based on the precise positioning result to obtain the finished video. By decoupling the processing content at each stage of the video generation process, this solution enables end-to-end automated teaching video generation. At the same time, the subtitle timing accuracy of the generated video is significantly improved, the accuracy of target word annotation and positioning in the picture is significantly improved, and the cost of iteration and update is also greatly reduced.

[0044] Figure 2 This is a flowchart illustrating another method for generating teaching materials and videos according to an embodiment of the present invention. The technical solution of this embodiment further optimizes the process of determining the precise location result of the target word based on the original text box location result and OCR recognition location result corresponding to the target word in the aforementioned embodiments, building upon the technical solutions of the above embodiments. This embodiment can be combined with various optional solutions in one or more of the above embodiments. For example... Figure 2 As shown, the teaching script and teaching video generation method of this invention embodiment may include the following process: S210. Based on the text content of the presentation data, generate the script text corresponding to each slide image. The script text is used to verbally explain the text content.

[0045] S220. Generate a basic video with spoken audio based on the text of the presentation corresponding to each slide image.

[0046] S230. Performing character-by-character alignment between the manuscript text and the ASR word time schedule corresponding to the base video to obtain a subtitle timeline, and performing subtitle burning on the base video based on the subtitle timeline and the manuscript text to obtain an intermediate video; wherein the ASR word time schedule is generated based on the audio data contained in the base video, and the subtitle timeline is used to indicate the start time and end time at which each manuscript sentence in the manuscript text is displayed in the subtitles of the intermediate video respectively.

[0047] As an optional but non-limiting implementation, performing character-by-character alignment between the manuscript text and the ASR word time schedule corresponding to the base video to obtain a subtitle timeline includes: transcribing the audio data contained in the base video based on a multi-engine transcription service to obtain the ASR word time schedule; performing text normalization processing on the manuscript text and the ASR word time schedule respectively to obtain a to-be-aligned manuscript text and a to-be-aligned ASR word time schedule; wherein the text normalization processing includes at least one of the following: traditional-simplified Chinese character conversion, confusing word replacement, case difference elimination, blank difference elimination and punctuation difference elimination; using a beam search method to align and match each sequentially appearing manuscript sentence in the to-be-aligned manuscript text with the to-be-aligned ASR word time schedule respectively to obtain a preset number of candidate alignment paths; wherein the candidate alignment paths record the alignment matching results corresponding to each manuscript sentence in sequence; determining a target alignment path from the respective candidate alignment paths, and determining the subtitle timeline based on the target alignment path. By adopting this optional solution, character-by-character alignment can be performed between the manuscript text and the ASR word time schedule corresponding to the base video based on the beam search method to obtain the subtitle timeline, which can achieve millisecond-level (±50ms) character-by-character time alignment accuracy, so that the alignment accuracy is improved several times.

[0048] Wherein the text normalization processing includes at least one of the following: traditional-simplified Chinese character conversion, confusing word replacement, case difference elimination, blank difference elimination and punctuation difference elimination. For example, the OpenCC library and a manual ASR confusing word mapping table may be used to perform relevant text normalization processing on the manuscript text and the ASR word time schedule respectively, such as converting traditional Chinese characters therein to simplified Chinese characters, replacing high-frequency confusing words such as "qiu→qiu" (autumn→hill) and "jian→jian" (cut→space), eliminating differences in case, blanks, punctuation marks, etc., so that the processed manuscript text and ASR word time schedule can meet the same normalization standard, which facilitates the subsequent solution to perform character-by-character alignment matching between the two.

[0049] The candidate alignment path can be understood as a possible matching result obtained in the alignment matching process, and the number of candidate alignment paths may be at least one. In contrast, the target alignment path can be understood as the optimal alignment path screened out from the multiple candidate alignment paths, which is equivalent to the globally optimal matching result.

[0050] Specifically, the audio data contained in the basic video can be transcribed using a multi-engine transcription service to obtain an ASR word timetable. Then, the script text and the ASR word timetable are processed using text normalization to obtain the script text to be aligned and the ASR word timetable to be aligned, respectively. Next, a beam search method can be used to align each sentence appearing sequentially in the script text to be aligned with the ASR word timetable to be aligned, thereby obtaining a preset number of candidate alignment paths. Each candidate alignment path records the alignment matching result corresponding to each sentence in sequence; for example, for a single sentence, the alignment matching result can be "successfully matched" or "skipped". In addition, each candidate alignment path can also record the following information: the number of matched sentences, the cumulative score, and the ASR word index at the end of the path. In this embodiment, the preset number associated with the candidate alignment paths can be determined by setting the total number of beams based on the beam search method. Next, the target alignment path can be determined from the candidate alignment paths. For example, the candidate alignment path with the most matched speech sentences and the highest cumulative score can be selected as the target alignment path. Subsequently, the subtitle timeline can be determined based on the target alignment path. For example, for speech sentences that have been successfully matched in the target alignment path, the start and end times of the speech sentence can be determined directly using the relevant timestamp information associated with the alignment matching results. For speech sentences that have not been successfully matched in the target alignment path (i.e., skipped speech sentences), the time window corresponding to the speech sentence can be calculated by using the time range of other speech sentences that are adjacent to the speech sentence and have been successfully matched. Thus, the subtitle timeline can be determined.

[0051] As an optional but non-limiting implementation, each sentence appearing sequentially in the lecture text to be aligned is matched with the ASR word timetable to be aligned, resulting in a preset number of candidate alignment paths. This includes: for the current lecture sentence to be aligned, matching the current lecture sentence with the ASR word timetable to determine the alignment matching result corresponding to the current lecture sentence; wherein the alignment matching result is used to indicate at least a preset number of top-ranking candidate intervals; based on each candidate interval, updating a preset number of initial candidate sequences associated with the current lecture sentence, resulting in a preset number of updated candidate sequences; and iteratively updating the updated candidate sequences based on the alignment matching results corresponding to each lecture sentence appearing after the current lecture sentence, until all lecture sentences are matched, thus obtaining the preset number of candidate alignment paths. Using this optional scheme, each lecture sentence appearing in the lecture text can be matched with the ASR word timetable corresponding to the underlying video, thereby obtaining a preset number of candidate alignment paths.

[0052] The initial candidate sequence can be understood as the initial sequence used when aligning and matching the current speech sentence. This initial candidate sequence contains the alignment and matching results of each speech sentence preceding the current speech sentence. During the alignment and matching process, each speech sentence is equivalent to a node, and an alignment and matching operation is performed at each such node. For example, if the current speech sentence to be aligned is the fourth speech sentence in the entire speech text, then the initial candidate sequence associated with this current speech sentence will sequentially contain the alignment and matching results corresponding to the three speech sentences preceding it. Relatedly, the updated candidate sequence can be understood as the sequence obtained by updating the initial candidate sequence based on the alignment and matching results of the current speech sentence.

[0053] Specifically, for the current lecture sentence to be aligned, the current lecture sentence can be aligned and matched with the ASR word timetable to determine the alignment matching result corresponding to the current lecture sentence. This alignment matching result is also used to indicate at least a preset number of candidate intervals with high ranking. Among them, the candidate interval can indicate the ASR word start-end index pair that can match the current lecture sentence (i.e., it includes: ASR word start index and ASR word end index). The preset number of high ranking is the total number of beams set by the beam search method. Then, based on each candidate interval, the preset number of initial candidate sequences associated with the current lecture sentence can be updated to obtain a preset number of updated candidate sequences. These updated candidate sequences will then serve as the initial candidate sequences associated with the next lecture sentence that appears immediately after the current lecture sentence. It's important to note that when using the beam search method, the total number of beams is pre-defined. During the search process, this beam is maintained. For example, if the total beam size is set to 24, a beam with a width of 24 is maintained. In each alignment matching process, the top 24 candidate intervals are selected and retained to update the sequences in the maintained beam. Following the same principle, based on the alignment matching results of each subsequent sentence in the lecture, these candidate sequences are iteratively updated until all lecture sentences are matched, thus obtaining the predetermined number of candidate alignment paths.

[0054] For example, using a cluster search method can avoid local errors caused by greedy matching sentence by sentence. Specifically, several possible global alignment paths can be maintained simultaneously. Each path records the number of matched sentences, the cumulative score, and the last used ASR word index. When processing each lecture sentence, each path can choose to match a candidate interval or temporarily skip the sentence. In each round, only the top few paths with the most matches, high cumulative scores, and reasonable chronological order are retained. For example, if a beam with a width of 24 is maintained, for each lecture sentence, each state in the beam can choose "match a candidate" or "skip". When the state is "match a candidate", the cumulative score can be the current score plus the candidate score and minus the word gap penalty score; when the state is "skip", the cumulative score can be the current score minus the penalty score for unmatched sentences and minus the penalty score for the number of reading units. Candidate paths are sorted in descending order of (number of matches, cumulative score, last ending index) triples, and the top 24 paths are retained in each round. The target alignment path can then be determined from this, which is essentially the globally optimal matching path.

[0055] As an optional but non-limiting implementation, the current speech sentence is aligned with the ASR word timetable to be aligned, determining the alignment result corresponding to the current speech sentence. This includes: constructing a search window for the current speech sentence in the ASR word timetable based on its relative position in the text to be aligned; enumerating all possible candidate intervals within the search window and concatenating the ASR words appearing in each candidate interval to obtain multiple candidate ASR texts; calculating the similarity score between the current speech sentence and each candidate ASR text, and selecting a predetermined number of candidate intervals with the highest scores from all possible candidate intervals based on the similarity scores. Using this optional approach, local alignment matching can be performed within the search window constructed in the ASR word timetable to be aligned, thereby determining a predetermined number of candidate intervals that match the current speech sentence.

[0056] The relative position information can be represented by dividing the "cumulative number of words before the current sentence in the speech" by the "cumulative total number of words in the speech text".

[0057] The search window can be understood as a local alignment region used to facilitate matching within the ASR term timeline to be aligned. The search window helps limit the search scope and avoids global traversal.

[0058] Specifically, a search window for the current speech sentence within the ASR word timetable to be aligned can be constructed based on its relative position within the text to be aligned. For example, the expected start and end indices (expected_start_index and expected_end_index) of the current speech sentence within the ASR word timetable can be calculated based on its relative position, resulting in an expected interval. This expected interval is then expanded before and after a preset dynamic redundancy radius to accommodate variations in speaking speed, pauses, omissions, or differences in ASR word segmentation, thus forming a search window (corresponding to start_min_index and start_max_index). The preset dynamic redundancy radius can be differentiated based on the total number of words, such as: preset dynamic redundancy radius radius=max(12,ceil(total number of words × 12%)). Then, all possible candidate intervals within the search window can be enumerated, and the ASR words appearing in each candidate interval can be concatenated to obtain multiple candidate ASR texts. Next, the similarity score between the current speech sentence and each candidate ASR text can be calculated, and a predetermined number of candidate intervals with the highest scores can be selected from all possible candidate intervals based on the similarity score. For example, in calculating the similarity score, in addition to the basic similarity between the two text strings, length deviation penalty, start offset penalty, and end offset penalty can be deducted, such as: score = confidence - length_gap × 0.22 - start_shift × 0.01 - end_shift × 0.003. Among them, the length deviation penalty length_gap is used to measure the length difference between the candidate ASR text and the speech sentence, and the start offset penalty start_shift and end offset penalty end_shift are used to measure the degree to which the candidate interval deviates from the expected position.

[0059] Optionally, after subtitle burning into the video, the quality of the subtitle burning effect can be evaluated in the following aspects: coverage, reading speed control, page coverage, uniformity of subtitle display time, proportion of unnatural sentence breaks, and minimum single subtitle coverage. The specific threshold conditions / values ​​used in the evaluation can be differentiated based on actual needs. For example, it can be determined whether the coverage is not less than 70% (i.e., whether ≥70% of the speech sentences are successfully aligned), whether the reading speed control CPS-P95 is not greater than 16 characters / second (i.e., whether it is ≤16 characters / second), whether the page coverage is not less than 60% (i.e., ≥60% of the page has effective subtitles), calculate the uniformity of subtitle display time (e.g., duration variation coefficient CV), whether the proportion of unnatural sentence breaks is not greater than 6%, and calculate the minimum single subtitle coverage, etc. If the subtitle burning effect meets the requirements, subsequent processing can continue; if the subtitle burning effect does not meet the requirements, the reconstruction of the subtitle timeline and re-burning of the subtitles can be automatically triggered.

[0060] S240. Search for target text boxes containing target words in each slide image, and determine the region coordinate information of the target words based on the local relative position information of the target words in the target text boxes, and use the region coordinate information as the original text box positioning result.

[0061] In this context, the target text box refers to a text box containing the target word. In this embodiment, at least one target text box can be identified.

[0062] Local relative position information can be represented using the relative position of the target word within the text box content of the target text box, as well as the line segmentation results of the target text box.

[0063] Specifically, when locating target words, we can search for target text boxes containing the target word in each slide image. For example, we can search for text boxes containing the target word in `native_text_boxes` using SequenceMatcher similarity matching, and select text boxes with a SequenceMatcher similarity exceeding a set threshold (e.g., 0.68) as target text boxes. `native_text_boxes` stores the coordinates of all text boxes and their corresponding content. Then, based on the local relative position of the target word within the target text boxes, we can determine the region coordinates of the target word and use these region coordinates as the native text box location result.

[0064] S250. Perform OCR recognition on each slide image to obtain each OCR recognition word and the bounding box coordinate information of each OCR recognition word. Search for target words in each OCR recognition word to determine the target bounding box coordinate information associated with the target word, and use the target bounding box coordinate information as the OCR recognition localization result.

[0065] Specifically, OCR recognition can be performed on each slide image separately to obtain each OCR-recognized word and its bounding box coordinates. Then, target words can be searched among these OCR-recognized words. For example, target words can be searched by matching them against SequenceMatcher similarity. The bounding box coordinates associated with OCR-recognized words whose SequenceMatcher similarity exceeds a set threshold are used as the target bounding box coordinates associated with that target word, and these target bounding box coordinates are used as the OCR recognition and localization result.

[0066] S260. The region coordinate information and the target bounding box coordinate information are scored from the preset evaluation dimensions, and the precise positioning result of the target word is determined based on the scoring results.

[0067] Among them, the preset evaluation dimensions can be set differently based on actual needs.

[0068] Specifically, for the target word, the determined positioning results include region coordinate information (corresponding to the original text box positioning results) and target bounding box coordinate information (corresponding to the OCR recognition positioning results). Then, the region coordinate information and target bounding box coordinate information can be scored from preset evaluation dimensions to determine which positioning result is more reliable and accurate. Furthermore, the precise positioning result of the target word can be determined based on the scoring results. For example, the one with the higher score can be taken as the precise positioning result of the target word.

[0069] As an optional but non-limiting implementation, the preset evaluation dimensions include at least one of the following dimensions: similarity dimension, aspect ratio constraint dimension, minimum area constraint dimension, line span score dimension, and expected deviation dimension. Correspondingly, the region coordinate information and target bounding box coordinate information are scored according to the preset evaluation dimensions, and the precise positioning result of the target word is determined based on the scoring results. This includes: comprehensively scoring the region coordinate information based on the preset evaluation dimensions to obtain a comprehensive text box score result; comprehensively scoring the target bounding box coordinate information based on the preset evaluation dimensions to obtain an OCR comprehensive score result; and determining the precise positioning result based on the comprehensive text box score result and the comprehensive OCR score result. Using this optional scheme, the two positioning results of the target word can be evaluated from multiple preset evaluation dimensions to determine the precise positioning result.

[0070] The similarity dimension assesses the proximity of the target word to related candidate text. The aspect ratio constraint filters entire page areas or minor noise (it can be set from 0.24 to 18.0, and up to 26.0 for long terms). The minimum area constraint excludes invalid boxes that are too small. The line span score evaluates the handling of target words displayed across multiple lines. The expected deviation dimension measures the deviation between the candidate region of the target word and the native coordinate mapping result of the PPT. Of course, an evaluation can also be performed using the confidence dimension; for example, the final decision on whether to accept a location result can be based on the relationship between the confidence level associated with a specific location result and a confidence threshold.

[0071] Specifically, the text box coordinate information can be comprehensively scored based on preset evaluation dimensions to obtain a comprehensive score result; and the target bounding box coordinate information can be comprehensively scored based on preset evaluation dimensions to obtain an OCR comprehensive score result; then, based on the text box comprehensive score result and the OCR comprehensive score result, the one with the higher comprehensive score is selected as the accurate positioning result.

[0072] S270. Based on the accurate positioning results, visual annotation processing is performed on the target words appearing in the intermediate video frame to obtain the finished video.

[0073] As an optional but non-limiting implementation, visual annotation processing of target words appearing in the intermediate video frame is performed based on precise positioning results to obtain the final video. This includes: determining the target word sub-region and glyph anchor points in the intermediate video frame based on precise positioning results and coordinate mapping relationships; wherein, the coordinate mapping relationship is used to indicate the mapping relationship between the slide image size and the intermediate video frame size; determining the horizontal range of the target underline based on the left and right boundary information of the target word sub-region, the font size of the target word, and the scaling ratio of the intermediate video frame, the target underline is used for visual annotation processing of the target word; determining the vertical position of the target underline based on the glyph anchor points and preset offset position information; determining the line width of the target underline based on the resolution information of the intermediate video frame and the font height of the target word; and drawing and rendering the target underline in the intermediate video frame based on the horizontal range, vertical position, and line width to obtain the final video. Using this optional solution, target words can be underlined, while ensuring that the visually annotated underlines accurately fit the target words without obscuring the content.

[0074] Here, the target word sub-region refers to the location of the target word within the video frame. The glyph anchor point can be understood as a visual reference point used for visual annotation. Since the bottom edge of a text box typically includes line spacing and white space, visual annotation cannot be performed directly on the bottom of the target word's bounding box. Instead, a glyph anchor point that closely matches the target word's position needs to be determined for accurate subsequent visual annotation. In this embodiment, the glyph anchor point can be determined based on the height, aspect ratio, and multi-line configuration of the text box containing the target word, so that the glyph anchor point is closer to the bottom position of the target word's glyph. For example, when the height and aspect ratio are both high, approximately 82% of the text box height can be used as the glyph anchor point.

[0075] Specifically, a mapping relationship can be constructed between the slide image size and the intermediate video frame size to obtain a coordinate mapping relationship. For example, the PPT coordinate system information (x, y, width, height) can be mapped to the 1920×1080 video frame coordinates according to the horizontal and vertical scaling ratios to construct the coordinate mapping relationship between the original page size of the slide image and the target video frame size. Then, based on the precise positioning results and coordinate mapping relationship, the target word sub-region and glyph anchor point in the intermediate video frame are determined. Next, based on the left and right boundary information of the target word sub-region, the font size of the target word, and the scaling ratio of the intermediate video frame, the horizontal range of the target underline is determined. This target underline is used for visual annotation of the target word. Based on the glyph anchor point and preset offset position information, the vertical position of the target underline is determined. This preset offset position information can be differentiated according to actual needs. For example, it can start from below the glyph anchor point and offset according to the preset offset position information to determine the vertical position of the target underline. Based on the resolution information of the intermediate video frame and the font height of the target word, the line width of the target underline is determined. Finally, based on the determined horizontal range, vertical position, and line width of the target underline, the target underline can be drawn and rendered in the middle video frame to obtain the finished video.

[0076] It's important to note that after determining the horizontal range, vertical position, and line width of the target underline, its shape can be fine-tuned based on the target word's width, the text box's aspect ratio, and visual balance. This ensures the rendered underline has a better visual effect. For example, for extra-wide text (aspect ratio ≥ 10.0 and width ≥ 1100 pixels), the target underline width can be reduced to 94% to avoid visual imbalance. Similarly, for multi-line text, independent `line_segments` arrays can be generated, with independent start and end coordinates and dynamic padding calculated for each line to prevent the underline from covering the entire line and causing visual imbalance.

[0077] Of course, besides underlining target words, you can also circle or highlight them. When performing these visual annotations, the target word sub-region is still used, only the rendering style differs. For example, a circle can be drawn with the center of the target word sub-region as its center, and the radius is calculated by adding padding to the width and height. Similarly, highlighting can be done by expanding outwards from the target word sub-region to form a semi-transparent rectangle or a rounded rectangle. Specific processing details can be differentiated based on actual needs, and will not be detailed here.

[0078] The technical solution of this invention involves generating script text corresponding to each slide image based on the text content of the presentation data. This script text is used for oral explanation of the text content. Then, based on the script text corresponding to each slide image, a base video with oral audio is generated. Next, the script text and the ASR word timetable corresponding to the base video are aligned word by word to obtain a subtitle timeline. Subtitles are then burned into the base video based on the subtitle timeline and the script text to obtain an intermediate video. The ASR word timeline is generated based on the audio data contained in the base video. The subtitle timeline indicates the start and end times of the display of each sentence in the script text in the subtitles of the intermediate video, thereby achieving millisecond-level (±50ms) word-by-word time alignment accuracy, resulting in a several-fold improvement in alignment precision. Then, on each slide... The process involves several steps: First, searching for target text boxes containing target words within an image. Based on the local relative position of the target word within the text box, the region coordinates of the target word are determined and used as the original text box location result. Second, performing OCR recognition on each slide image yields each OCR-recognized word and its bounding box coordinates. The target word is then searched within each OCR-recognized word to determine its associated bounding box coordinates, which are used as the OCR recognition location result. Third, scoring the region coordinates and target bounding box coordinates across preset evaluation dimensions, and determining the precise location of the target word based on the scores. This precise location is then determined using a dual-source fusion localization method. Finally, visual annotation is applied to the target words appearing in intermediate video frames based on the precise location results, resulting in the final video. This approach decouples the processing at each stage of video generation, enabling end-to-end automated teaching video generation. The resulting video exhibits significantly improved subtitle timing accuracy, significantly improved target word annotation accuracy, and significantly reduced iteration update costs.

[0079] Figure 3This is a schematic diagram of a teaching script and teaching video generation device provided in an embodiment of the present invention. This embodiment is applicable to generating teaching videos based on presentation text data. The teaching script and teaching video generation device can be implemented in software and / or hardware, and is generally integrated into any electronic device with network communication capabilities, such as a mobile terminal, PC, or server. Figure 3 As shown, the teaching script and teaching video generation device of this embodiment of the invention may include a script text generation module 310, a basic video generation module 320, an intermediate video generation module 330, and a finished video generation module 340. Wherein: The presentation text generation module 310 is used to generate presentation text corresponding to each slide image based on the text content of the presentation data. The presentation text is used to verbally explain the text content. The basic video generation module 320 is used to generate a basic video with spoken audio based on the script text corresponding to each slide image; The intermediate video generation module 330 is used to align the script text with the ASR word timetable corresponding to the base video word by word to obtain the subtitle timeline, and to burn subtitles into the base video based on the subtitle timeline and the script text to obtain the intermediate video; wherein, the ASR word timetable is generated based on the audio data contained in the base video, and the subtitle timeline is used to indicate the start and end times of the display of each script sentence in the script text in the subtitles of the intermediate video; The finished video generation module 340 is used to determine the precise positioning result of the target word based on the original text box positioning result and the OCR recognition positioning result corresponding to the target word, and to perform visual annotation processing on the target word appearing in the intermediate video frame based on the precise positioning result to obtain the finished video.

[0080] The technical solution of this invention involves a script text generation module generating script text corresponding to each slide image based on the text content of the presentation data. This script text is used for narrating the text content. Then, a basic video generation module generates a basic video with narrated audio based on the script text corresponding to each slide image. Next, an intermediate video generation module aligns the script text with the ASR word timeline corresponding to the basic video word timeline character by character to obtain a subtitle timeline. Subtitles are then burned into the basic video based on the subtitle timeline and the script text to obtain the intermediate video. The ASR word timeline is based on the basic video word timeline. The audio data generated within the video is used to indicate the start and end times of each sentence in the lecture text when it appears in the subtitles of the intermediate video, achieving millisecond-level (±50ms) word-by-word time alignment accuracy, resulting in a several-fold improvement in alignment precision. Finally, the final video generation module determines the precise location of the target word based on the original text box location result and OCR recognition location result. A dual-source fusion positioning method is used to determine the precise location of the target word, and visual annotation processing is performed on the target word appearing in the intermediate video frame based on the precise location result, resulting in the final video. This solution, by decoupling the processing content of each stage in the video generation process, enables end-to-end automated teaching video generation. Simultaneously, the subtitle time accuracy and the accuracy of target word annotation and location in the generated final video are significantly improved, while the iteration update cost is also greatly reduced.

[0081] As an optional but non-limiting implementation, the intermediate video generation module 330 includes a transcription submodule, a normalization processing submodule, an alignment and matching submodule, and a subtitle timeline determination submodule. Wherein: The transcription submodule is used to transcribe the audio data contained in the base video based on the multi-engine transcription service to obtain the ASR word time schedule; The normalization processing submodule is used to perform text normalization processing on the lecture text and the ASR term timetable respectively to obtain the lecture text to be aligned and the ASR term timetable to be aligned; wherein, the text normalization processing includes at least one of the following: conversion between simplified and traditional Chinese characters, replacement of confusing words, elimination of capitalization differences, elimination of whitespace differences, and elimination of punctuation differences; The alignment matching submodule is used to perform a cluster search method to align each sentence in the speech text to be aligned with the ASR term timetable to be aligned, thereby obtaining a preset number of candidate alignment paths; wherein, the candidate alignment paths record the alignment matching results corresponding to each sentence in turn. The subtitle timeline determination submodule is used to determine the target alignment path from each of the candidate alignment paths, and to determine the subtitle timeline based on the target alignment path.

[0082] As an optional but non-limiting implementation, the alignment matching submodule includes an alignment matching result determination unit, a sequence update unit, and a candidate alignment path determination unit. Wherein: The alignment matching result determination unit is used to perform alignment matching between the current speech sentence to be aligned and the ASR term timetable to be aligned, and determine the alignment matching result corresponding to the current speech sentence; wherein, the alignment matching result is used to indicate at least a preset number of candidate intervals with the highest ranking. The sequence update unit is used to update a preset number of the initial candidate sequences associated with the current speech sentence based on each of the candidate intervals, so as to obtain a preset number of updated candidate sequences; The candidate alignment path determination unit is used to iteratively update the update candidate sequence based on the alignment matching results corresponding to each speech sentence that appears after the current speech sentence, until all speech sentences are matched, so as to obtain a preset number of candidate alignment paths.

[0083] As an optional but non-limiting implementation, the alignment matching result determination unit includes a search window construction subunit, a candidate ASR text determination subunit, and a candidate interval filtering subunit. Wherein: The search window construction subunit is used to construct a search window for the current speech sentence in the ASR term timetable to be aligned, based on the relative position information of the current speech sentence in the speech text to be aligned; The candidate ASR text determination subunit is used to enumerate all possible candidate intervals within the search window and concatenate the ASR words appearing in each candidate interval to obtain multiple candidate ASR texts. The candidate interval filtering subunit is used to calculate the similarity score between the current speech sentence and each of the candidate ASR texts, and to filter out a preset number of candidate intervals with the highest scores from all possible candidate intervals based on the similarity score.

[0084] As an optional but non-limiting implementation, the finished video generation module 340 includes a native text box positioning result determination submodule, an OCR recognition positioning result determination submodule, and a precise positioning result determination submodule. Wherein: The native text box location result determination submodule is used to search for target text boxes containing the target word in each slide image, and determine the region coordinate information of the target word based on the local relative position information of the target word in the target text box, and use the region coordinate information as the native text box location result; The OCR recognition and localization result determination submodule is used to perform OCR recognition on each slide image to obtain each OCR recognition word and the bounding box coordinate information of each OCR recognition word, and to search for the target word in each OCR recognition word to determine the target bounding box coordinate information associated with the target word, and to use the target bounding box coordinate information as the OCR recognition and localization result. The precise positioning result determination submodule is used to score the region coordinate information and the target bounding box coordinate information from preset evaluation dimensions, and determine the precise positioning result of the target word based on the scoring results.

[0085] As an optional but non-limiting implementation, the preset evaluation dimensions include at least one of the following dimensions: similarity dimension, aspect ratio constraint dimension, minimum area constraint dimension, cross-line score dimension, and expected deviation dimension; correspondingly, the precise positioning result determination submodule includes a text box comprehensive scoring unit, an OCR comprehensive scoring unit, and a precise positioning result determination unit, wherein: The text box comprehensive scoring unit is used to comprehensively score the region coordinate information based on the preset evaluation dimensions to obtain the text box comprehensive scoring result. The OCR comprehensive scoring unit is used to comprehensively score the target bounding box coordinate information based on the preset evaluation dimensions to obtain the OCR comprehensive scoring result. The precise positioning result determination unit is used to determine the precise positioning result based on the comprehensive scoring result of the text box and the comprehensive scoring result of the OCR.

[0086] As an optional but non-limiting implementation, the finished video generation module 340 further includes a parameter information determination submodule, a horizontal range determination submodule, a vertical position determination submodule, a line width determination submodule, and an underline drawing submodule. Wherein: The parameter information determination submodule is used to determine the target word sub-region and character anchor point in the intermediate video frame based on the precise positioning result and coordinate mapping relationship; wherein, the coordinate mapping relationship is used to indicate the mapping relationship between the slide image size and the intermediate video frame size; The horizontal range determination submodule is used to determine the horizontal range of the target underline based on the left and right boundary information of the target word sub-region, the font size of the target word, and the scaling ratio of the intermediate video frame. The target underline is used to perform visual annotation processing on the target word. The vertical position determination submodule is used to determine the vertical position of the target underline based on the character anchor point and the preset offset position information; The line width determination submodule is used to determine the line width of the target underline based on the resolution information of the intermediate video frame and the font height of the target word; The underline drawing submodule is used to draw and render the target underline in the intermediate video frame based on the horizontal range, the vertical position, and the line width to obtain the finished video.

[0087] The teaching script and teaching video generation device provided in this embodiment of the invention can be used to execute the teaching script and teaching video generation method, and has the corresponding functional modules and beneficial effects of executing the teaching script and teaching video generation method.

[0088] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of the present invention.

[0089] Figure 4 This is a schematic diagram of an electronic device for implementing a method for generating teaching materials and teaching videos, as provided in an embodiment of the present invention. (Refer to the following...) Figure 4 The diagram illustrates a structural schematic of an electronic device 410 suitable for implementing embodiments of the present invention. The terminal devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0090] like Figure 4As shown, the electronic device 410 includes at least one processor 411 and a memory, such as a read-only memory (ROM) 412 and a random access memory (RAM) 413, communicatively connected to the at least one processor 411. The memory stores computer programs executable by the at least one processor. The processor 411 can perform various appropriate actions and processes based on the computer program stored in the ROM 412 or loaded from storage unit 418 into the RAM 413. The RAM 413 may also store various programs and data required for the operation of the electronic device 410. The processor 411, ROM 412, and RAM 413 are interconnected via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0091] Multiple components in electronic device 410 are connected to input / output (I / O) interface 415, including: input unit 416, such as keyboard, mouse, etc.; output unit 417, such as various types of monitors, speakers, etc.; storage unit 418, such as disk, optical disk, etc.; and communication unit 419, such as network card, modem, wireless transceiver, etc. Communication unit 419 allows electronic device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0092] Processor 411 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 411 executes the teaching script and teaching video generation methods provided in any embodiment of the present invention.

[0093] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the teaching script and teaching video generation method shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 419, or installed from storage unit 418, or installed from read-only memory (ROM) 412. When the computer program is executed by processor 411, it performs the functions defined in the teaching script and teaching video generation method of the embodiments of the present invention.

[0094] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0095] The electronic device provided in this embodiment of the invention belongs to the same inventive concept as the teaching manuscript and teaching video generation method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0096] This invention provides a computer storage medium storing a computer program that, when executed by a processor, implements the teaching script and teaching video generation method provided in the above embodiments.

[0097] It should be noted that the computer-readable medium described above in this invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0098] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0099] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0100] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0101] The units described in the embodiments of the present invention can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0102] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0103] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0104] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0105] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0106] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for generating teaching scripts and teaching videos, characterized in that, The method includes: Based on the text content of the presentation data, a script text corresponding to each slide image is generated, and the script text is used to verbally explain the text content. Based on the lecture text corresponding to each slide image, a basic video with spoken audio is generated. The script text is aligned word-by-word with the ASR word timetable corresponding to the base video to obtain the subtitle timeline. The base video is then burned with subtitles based on the subtitle timeline and the script text to obtain the intermediate video. The ASR word timetable is generated based on the audio data contained in the base video, and the subtitle timeline is used to indicate the start and end times of the display of each sentence in the script text in the subtitles of the intermediate video. Based on the localization results of the original text box corresponding to the target word and the OCR recognition localization results, the precise localization result of the target word is determined, and the target word appearing in the intermediate video frame is visually annotated based on the precise localization result to obtain the finished video.

2. The method according to claim 1, characterized in that, The step of aligning the script text with the ASR word timeline corresponding to the base video, word by word, to obtain the subtitle timeline includes: The audio data contained in the base video is transcribed using a multi-engine transcription service to obtain the ASR word time schedule; The lecture text and the ASR terminology timetable are respectively subjected to text normalization processing to obtain the lecture text to be aligned and the ASR terminology timetable to be aligned; wherein, the text normalization processing includes at least one of the following: conversion between simplified and traditional Chinese characters, replacement of confusing words, elimination of capitalization differences, elimination of whitespace differences, and elimination of punctuation differences; A cluster search method is used to match each sentence in the speech text to be aligned with the ASR term timetable to be aligned, thereby obtaining a preset number of candidate alignment paths; wherein, each candidate alignment path records the alignment matching results corresponding to each sentence in turn. A target alignment path is determined from each of the candidate alignment paths, and the subtitle timeline is determined based on the target alignment path.

3. The method according to claim 2, characterized in that, The step involves aligning and matching each sentence in the lecture text to be aligned with the ASR term timetable to obtain a preset number of candidate alignment paths, including: For the current speech sentence to be aligned, the current speech sentence is aligned and matched with the ASR term timetable to be aligned, and the alignment matching result corresponding to the current speech sentence is determined; wherein, the alignment matching result is used to indicate at least a preset number of candidate intervals with the highest ranking; Based on each of the candidate intervals, the preset number of initial candidate sequences associated with the current speech sentence are updated to obtain the preset number of updated candidate sequences; Based on the alignment matching results corresponding to each speech sentence that appears after the current speech sentence, the update candidate sequence is iteratively updated in sequence until all speech sentences that appear are matched, so as to obtain a preset number of candidate alignment paths.

4. The method according to claim 3, characterized in that, The step of aligning and matching the current speech sentence with the ASR term timetable to be aligned, and determining the alignment and matching result corresponding to the current speech sentence, includes: Based on the relative position information of the current speech sentence in the speech text to be aligned, a search window for the current speech sentence in the ASR term timetable to be aligned is constructed; Enumerate all possible candidate intervals within the search window, and concatenate the ASR words appearing in each candidate interval to obtain multiple candidate ASR texts; Calculate the similarity score between the current speech sentence and each of the candidate ASR texts, and select a preset number of candidate intervals with the highest scores from all possible candidate intervals based on the similarity scores.

5. The method according to claim 1, characterized in that, The determination of the precise location result of the target word based on the local text box location result and the OCR recognition location result includes: Search for target text boxes containing the target word in each slide image, and determine the region coordinate information of the target word based on the local relative position information of the target word in the target text box, and use the region coordinate information as the original text box positioning result; OCR recognition is performed on each slide image to obtain each OCR recognition word and the bounding box coordinate information of each OCR recognition word. The target word is then searched in each OCR recognition word to determine the target bounding box coordinate information associated with the target word. The target bounding box coordinate information is then used as the OCR recognition and localization result. The region coordinate information and the target bounding box coordinate information are scored according to preset evaluation dimensions, and the precise positioning result of the target word is determined based on the scoring results.

6. The method according to claim 5, characterized in that, The preset evaluation dimensions include at least one of the following dimensions: similarity dimension, aspect ratio constraint dimension, minimum area constraint dimension, cross-line score dimension, and expected deviation dimension. Accordingly, the step of scoring the region coordinate information and the target bounding box coordinate information from preset evaluation dimensions, and determining the precise positioning result of the target word based on the scoring results, includes: Based on the preset evaluation dimensions, the region coordinate information is comprehensively scored to obtain the comprehensive score result of the text box. Based on the preset evaluation dimensions, the target bounding box coordinate information is comprehensively scored to obtain the OCR comprehensive score result; The precise positioning result is determined based on the comprehensive score of the text box and the comprehensive score of the OCR.

7. The method according to claim 1, characterized in that, The step of visually annotating the target words appearing in the intermediate video frame based on the precise positioning result to obtain the finished video includes: Based on the precise positioning results and coordinate mapping relationship, the target word sub-region and glyph anchor point in the intermediate video frame are determined; wherein, the coordinate mapping relationship is used to indicate the mapping relationship between the slide image size and the intermediate video frame size; Based on the left and right boundary information of the target word sub-region, the font size of the target word, and the scaling ratio of the intermediate video frame, the horizontal range of the target underline is determined, and the target underline is used to perform visual annotation processing on the target word; Based on the character anchor points and the preset offset position information, the vertical position of the target underline is determined; Based on the resolution information of the intermediate video frame and the font height of the target word, the line width of the target underline is determined; Based on the horizontal range, the vertical position, and the line width, the target underline is drawn and rendered in the intermediate video frame to obtain the finished video.

8. A device for generating teaching materials and teaching videos, characterized in that, The device includes: The presentation text generation module is used to generate presentation text corresponding to each slide image based on the text content of the presentation data. The presentation text is used to verbally explain the text content. The basic video generation module is used to generate a basic video with spoken audio based on the lecture text corresponding to each slide image; The intermediate video generation module is used to align the lecture text with the ASR word timetable corresponding to the base video word by word to obtain the subtitle timeline, and to burn subtitles into the base video based on the subtitle timeline and the lecture text to obtain the intermediate video; wherein, the ASR word timetable is generated based on the audio data contained in the base video, and the subtitle timeline is used to indicate the start and end times of the display of each lecture sentence in the lecture text in the subtitles of the intermediate video; The finished video generation module is used to determine the precise positioning result of the target word based on the original text box positioning result and the OCR recognition positioning result, and to perform visual annotation processing on the target word appearing in the intermediate video frame based on the precise positioning result to obtain the finished video.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the teaching script and teaching video generation method as described in any one of claims 1-7.

10. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the teaching script and teaching video generation method as described in any one of claims 1-7.

Images