Multimodal automatic teaching video generation method and system based on intermediate file

Abstract

The invention discloses a multi-modal automatic teaching video generation method and system based on an intermediate file. The analysis module is used for extracting texts and pictures from documents input by a user and constructing a searchable knowledge base. The content generation module retrieves related teaching content texts according to knowledge points specified by a user through calling of a two-stage large language model, and generates an explanation first draft in a free format; and retrieving related pictures, performing format conversion, and combining with the explaining first draft to generate a structured JSON file. The format conversion module is used for converting the JSON file into an intermediate file which can be read by human and can be analyzed by a machine through a predefined template engine; the collaborative review module presents the intermediate file to a user and receives a modification instruction; and the synchronous updating module performs reverse analysis and grammar verification, and updates the original structured data according to the changed content. The material generation module generates corresponding audio and image materials; and the video synthesis module synthesizes the materials into a video file and outputs the video file.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of video generation, and relates to artificial intelligence and digital content creation, in particular to a multi-modal automated teaching video generation method and system based on intermediate files. BACKGROUND

[0002] With the popularization of online education, video has become a key carrier for knowledge dissemination. However, the traditional video production process requires high professional skills of content creators, including script writing, courseware design, audio recording, video editing, etc., resulting in long production cycle, low efficiency and high technical threshold.

[0003] With the development of language models and visual language models, artificial intelligence assisted video generation methods have emerged. Many online tools provide one-click text-to-video generation services, but their multi-modal content processing capabilities are limited, making it difficult to achieve dynamic and high-definition presentation. At the same time, due to the opaque internal generation logic of the model and the lack of human review nodes in the video generation process, users have limited ability to fine-tune the picture content, layout, and narration rhythm, resulting in weak customization control. Users can only find problems after the complete video is generated, at which point they can only make post-production modifications using professional software, missing the opportunity to correct errors at the lowest cost in the text stage. In addition, when using a single language model instruction to directly generate a final file containing complex syntax, it is easy to fail due to model output errors or context length restrictions, and the stability and success rate of the generation process need to be improved.

[0004] In particular for the generation of teaching videos, in addition to text content, the input document content of the user usually includes a large amount of flowcharts, statistical charts, tables, formulas and other high diversity and non-standardized drawing content. Existing models lack the ability to automatically analyze and generate these multi-modal elements, and are all treated as static pictures, which will increase the difficulty of layout and may cause loss of structured information, making it difficult to meet the accuracy requirements of video content in teaching scenarios. Moreover, the automatically generated narration may have inconsistent tone and unnatural tone, which seriously affects the overall presentation of the video and the experience of the viewer, and further adversely affects the teaching results.

[0005] In summary, the existing technology lacks an end-to-end automated method that can handle complex graphic content while providing an intuitive and controllable review and modification interface to stably generate high-quality teaching videos. SUMMARY

[0006] To address the shortcomings of existing technologies, this invention proposes a method and system for generating multimodal automated teaching videos based on intermediate files. By introducing a human-readable and machine-parseable intermediate file as the process hub, and combining content and format decoupling, intelligent parsing of multimodal elements, and a human-machine collaborative review mechanism, an automated video generation method for multimodal documents with both robustness and controllability is achieved. This method is particularly suitable for generating knowledge dissemination videos for courses, technical training, compliance presentations, and other purposes where high accuracy and controllability of content are required.

[0007] A multimodal automated teaching video generation method based on intermediate files includes the following steps: Step 1: Multimodal source document parsing The document parsing library is used to preprocess the Word document input by the user, extracting the text and images. Descriptive titles are generated for the images, building a searchable knowledge base containing text content, images, and their descriptive titles.

[0008] Step 2: Structured Content Generation The user specifies a particular knowledge point from a Word document. First, the text of the knowledge point is submitted as a prompt to the language model, generating a free-formatted initial explanation. Then, relevant images are retrieved based on the knowledge point text, and the visual language model is used to convert the images to their correct formats. Finally, the initial explanation and the converted images are used as input to the language model to obtain a structured JSON file.

[0009] Step 3: Intermediate file generation The structured JSON file obtained in step 2 is converted into an intermediate presentation file that conforms to the syntax of a specific presentation framework using a predefined template engine.

[0010] Step 4: Human-machine collaborative review Based on the intermediate file generated in step 3, a human-machine collaborative review is conducted. First, the user previews and edits the file, and then the system performs rule-based parsing and syntax validation. For intermediate files that pass validation, an incremental update is performed in the JSON file, meaning only the changed parts are regenerated. For intermediate files that fail validation, error location feedback is provided to the user.

[0011] Step 5: Material Generation The intermediate file content, approved in step 4, is parsed to separate the text from the script fields and the page content. A text-to-speech function is then used to generate an audio file from the script field text. The page content is then rendered to generate an image file.

[0012] Step 6: Video Compositing The audio file and the image file generated in step 5 are first generated into independent video clips according to a pairing relationship, and then a complete video file is synthesized according to the order, and is packaged and output.

[0013] The multi-modal automated teaching video generation system based on the intermediate file comprises: The analysis module is used for extracting text and pictures from the Word document input by the user and constructing a searchable knowledge base.

[0014] The content generation module is used for generating a preliminary explanation according to the knowledge point specified by the user, matching the picture, and generating a structured JSON file based on the preliminary explanation and the picture.

[0015] The format conversion module is used for converting the structured JSON file into a presentation intermediate file through a predefined template engine.

[0016] The collaborative review module is used for presenting the intermediate file to the user and receiving modification instructions.

[0017] The synchronous updating module is used for reverse parsing and syntax verification of the modified intermediate file, identifying the content change part, and updating the original structured data corresponding to the content change part, The material generation module is used for generating corresponding audio and image materials according to the updated structured data. The video synthesis module is used for synthesizing the materials into a final video.

[0018] The present application has the following beneficial effects: 1. The robustness and stability of the generation process are improved: by dividing the complex generation task into multiple stages of knowledge point selection, free format preliminary draft and structured data, the risk of directly generating complex syntax format by a single model is avoided, and the success rate and stability of the automated process are improved.

[0019] 2. Controllability and flexibility of the generation process are realized: the introduction of the presentation intermediate file is both an instruction for machine execution and a draft readable by the user. The user can intervene by selecting knowledge points and modifying the intermediate file in the middle stage, realizing convenient content iteration and adjustment.

[0020] 3. Incremental updating and collaborative modification are supported: by introducing the intermediate file of man-machine cooperation and designing the reverse synchronization mechanism based on rule analysis, incremental modification of the generated content is supported. The user's modification in the text stage can be accurately identified, and the system only needs to regenerate the changed part, avoiding the global reprocessing of the entire video, improving the modification efficiency, and realizing the low-cost man-machine cooperative workflow.

[0021] 4. Extended processing capabilities for multi-modal content and diversified charts: not only can visual elements such as flowcharts and formulas be converted into code in a specific format through visual language models, but also diversified chart processing mechanisms are introduced. By identifying chart types and calling external chart generation services, complex charts in various professional fields can be automatically processed and converted into high-definition visual images. This expands the types of content that can be processed and improves the applicability and visual presentation of generated videos in different disciplines.

[0022] 5. Improved quality and consistency of generated content: by locking timbre parameters in audio generation and controlling rhythm in scripts, the consistency and naturalness of narration are ensured; by rendering visual elements in high definition, the clarity of visual content is ensured, thereby improving the overall quality of the final video.

[0023] 6. Lower technical threshold for users: the entire process hides the underlying technical implementation details from the user, who can adjust the video content by selecting knowledge points and modifying the note-like markup file, thereby reducing the technical threshold for non-professional users. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 Schematic diagram of a multi-modal automated teaching video generation system based on intermediate files; Figure 2 Schematic diagram of a user input document; Figure 3 A corpus segment retrieved based on a specified knowledge point; Figure 4 Schematic diagram of intermediate file one; Figure 5 Schematic diagram of intermediate file two; Figure 6 Schematic diagram of intermediate file three; Figure 7 Video screenshot generated based on intermediate file one; Figure 8 Video screenshot generated based on intermediate file two; Figure 9 Video screenshot generated based on intermediate file three; DETAILED DESCRIPTION

[0025] The invention will be further explained and described in conjunction with the accompanying drawings; This embodiment takes the course of auditing as an example to illustrate Figure 1 how the system shown can convert a document containing text and different pictures such as "Principles of Auditing" into a video file with accurate content, clear vision, and professional listening experience through a highly automated, intelligent, and controllable process: Step 1, Multi-modal source document parsing The user input Word document "Principles of Auditing.docx" is preprocessed using python-docx or other document parsing libraries to extract the text and pictures in the document.

[0026] The text extraction includes extracting all chapter and paragraph texts in the document. The picture extraction first exports all embedded pictures in the document in order of position and records the relative position, then uses a visual language model to generate a descriptive title for each picture, building a picture description knowledge base for subsequent picture retrieval. Table 1 shows an example of prompt words when using a visual language model (VLM) to generate a descriptive title for a picture: Table 1 During preprocessing, only picture extraction and picture description knowledge base establishment are performed, and high-cost picture-to-code conversion is not performed.

[0027] Step 2, Structured content generation To ensure the controllable scope of the generation task, the user specifies specific knowledge points from the Word document as input, so that the language model focuses on independent knowledge units, avoiding increased error rates due to processing long texts. Based on the user-specified knowledge points, structured content is generated in stages, breaking down complex generation tasks and reducing the difficulty of directly generating final strict format data. The specific steps are as follows: s2.1, For the content of Chapter 4 "Audit Objectives, Audit Evidence, and Audit Working Papers" of "Principles of Auditing", first give a specified knowledge point "Recognition of various transactions, matters and related disclosures during the audit period", submit the knowledge point text to the language model as a prompt, and make the language model generate a draft of the explanation around the knowledge point. The draft of the explanation uses a flexible, semi-structured format to break down the content into multiple slides, each slide containing a title, key points, and a script for the explanation.

[0028] In an optional embodiment, to ensure that the language model can effectively use the original text information and make the generated explanation draft content more accurate and relevant, retrieval-augmented generation (RAG) is performed through the following steps: s2.1.1, During the multi-modal source document parsing process, the user input Word document is divided by paragraph, and the vector representation of each paragraph is calculated by an embedding model to build a vector knowledge base.

[0029] s2.1.2, When the user specifies the knowledge point, use the knowledge point text as a query to retrieve the most relevant paragraphs from the vector knowledge base. For example,Figure 2 The user inputs the Word document, queries the knowledge point "determination of various transactions, matters and related disclosures during the audit period", and retrieves the corpus fragments as shown in the following figure: Figure 3 The contents are used as reference materials, and the knowledge point text is used to form the prompt words of the language model. Table 2 shows an example of prompt words for generating the initial draft of the explanation using the language model: Table 2 To ensure the retrieval quality of long documents, a double-layer index of chapter title vectors and paragraph content vectors is established when building the vector knowledge base. During retrieval, the chapters are matched first, and then the paragraphs are finely retrieved within the relevant chapters to improve the efficiency and accuracy of long text processing. In addition, a confidence score is calculated for each retrieval result. When the scores of all relevant fragments are below a preset threshold, the system can actively prompt the user to "lack reliable reference materials, suggest uploading supplementary files" to avoid generating low-quality or irrelevant content in the initial draft of the explanation due to inaccurate prompt words.

[0030] s2.2, according to the user-specified knowledge point text, perform semantic retrieval in the picture description knowledge base to find the most relevant pictures. For pictures to be used, only use the visual language large model for deep analysis to avoid unnecessary high-cost preprocessing of all pictures.

[0031] For pictures with flowchart content, convert their topological structure into code in the graph description language. For pictures with mathematical formula content, call the mathematical formula OCR service to convert them into code in the specific formula typesetting language.

[0032] In an optional embodiment, considering the uncertainty of the visual language large model, for flowchart conversion, first use the visual language large model to generate a textual description of the graph, and then use the language model to generate Mermaid code based on the textual description to describe its topological structure, to improve accuracy. For Figure 2 The Mermaid code of the logical relationship diagram in is shown in Table 3: Table 3 For complex UML diagrams that cannot be generated with high confidence, the system can issue a "manual correction required" prompt to the user interface, or directly output an SVG vector diagram as an alternative. For complex mathematical formulas, if the confidence after OCR processing is low, do not perform conversion, but directly retain the original picture format and the corresponding LaTeX source code as alt text for manual review.

[0033] For complex charts other than flowcharts or mathematical formulas, first identify their specific types, such as bar charts, line charts, pie charts, etc., and extract their core data, including titles, axes, data series, and numerical values. To reduce the technical difficulty of reverse derivation of structured data from bitmaps, limit the scope of automated processing to several common chart types, and guide users to provide source data files for generating the charts, or directly fill in the data that needs to be corrected by the visual language large model, thereby improving the success rate and data accuracy of complex chart conversion. Then format the source data extracted by the model or provided by the user, call the AI chart generation tool, and generate professional chart pictures. Finally, save the chart link returned by the AI chart generation tool.

[0034] s2.3, the first draft of the explanation generated in s2.1 and the converted picture in s2.2 are used as input for the language model, and the language model outputs a strict, pure content structured JSON file. The JSON file contains fields such as the title "title", key points "key_points", and script "script" of each slide, as well as the converted picture structure code "mermaid_code", formula code "latex_code", or chart link "chart_url", but does not contain formatting syntax. Table 4 shows an example of a structured JSON file: Table 4 Step 3, intermediate file generation The structured JSON file obtained in step 2 is converted into a Markdown format presentation intermediate file that conforms to the syntax of a specific presentation framework using a pre-defined template engine as shown in Table 5. The structural rules defined in the template engine are the basis for the subsequent step of regularized reverse parsing, ensuring that the mapping process from markup language to structured data is stable and predictable.

[0035] Table 5 The specific conversion process is: traverse the JSON array, generate a slide for each object, and mark it with a separator "---". The title field "title" is used as the first-level title of the slide, the key points "key_points" are converted into an unordered list, and the script "script" is wrapped in a specific comment as a voiceover script. For the converted picture structure code, formula code, or chart link, embed it in the corresponding code block or image tag.

[0036] For the structured JSON file shown in Table 4, the pre-defined template engine is converted into Figures 4~6The presentation intermediate file shown in the presentation can be seen in Table 4, the three objects are converted into three slides.

[0037] Step 4, human-computer collaborative review Based on the intermediate file generated in step 3, the human-computer collaborative review is carried out, first by the user for preview and editing, and then by the system for analysis and verification, the specific steps are as follows: s4.1, show the presentation intermediate file generated in step 3 to the user, the user can choose to edit the presentation intermediate file in a local editor or in an online integrated development environment. Including the voiceover script content in the symbol " <!--... --> ", adjusting the flow of chart script, and controlling the rhythm of voice, etc.

[0038] s4.2, in order to balance the flexibility of user modification and the robustness of system processing, the following incremental update mechanism based on regularized analysis and change range awareness is adopted to update the presentation intermediate file edited by the user: s4.2.1, use the structural rules of the template engine defined when generating the presentation intermediate file in step 3, during the user editing process, reverse parse the presentation intermediate file edited by the user, preferentially identify and extract the structural rules defined in the template engine, including the first title, unordered list and voiceover wrapped in specific annotations. When the user performs regular content editing, the intermediate file can be accurately parsed back to a structured JSON file.

[0039] The matching logic of the reverse parsing is: each slide is separated by `---`, the first line after the separator symbol `---` is usually `# title`, the line starting with `--` after `# title` is `key_points`, and the voiceover script is extracted by regular expression `<!-- script\s*\n(.*?)\n\s*-->`.

[0040] In an optional embodiment, before reverse parsing the presentation intermediate file edited by the user, the presentation intermediate file can be precompiled by calling the build command. When the precompilation fails, the error information and code location are fed back to the user to guide them to correct it, which can effectively capture syntax errors introduced by manual modification, thereby ensuring the input quality of the analysis program.

[0041] ​​s4.2.2, For the user saved modified presentation intermediate file, syntax verification is performed, and for syntax errors that do not conform to the presentation framework or embedded code, the user is provided with explicit error positioning and modification suggestions. For the presentation intermediate file that passes the verification, compare the newly parsed structured JSON file with the version before modification, and for content-level changes such as narration text, title text modification, etc., perform incremental update, i.e. only regenerate the corresponding audio or update the text data of that part.

[0042] s4.2.3, When the user makes a complex modification beyond the preset rules, resulting in reverse parsing failure or ambiguity, the system will prompt the user that the modification cannot be incrementally processed, and provide a rollback option, such as choosing to abandon the modification, or performing a one-time full re-rendering to ensure that the final result is completely consistent with the intermediate file.

[0043] Through this hierarchical processing and rollback mechanism, flexibility is ensured while ensuring the stability and reliability of data synchronization.

[0044] Step 5, material generation Generate audio and pictures according to the presentation intermediate file after step 4 review, the specific steps are as follows: s5.1, Traverse the slide item marked for update in the structured JSON file parsed in step 4, extract the text in the script field, replace the custom mark with a pause instruction recognized by the text-to-speech TTS engine, then send the script of each slide along with the fixed speaker embedding vector parameter to the TTS engine to generate an audio file corresponding to each slide.

[0045] To ensure the uniformity of the narration tone, the system can prompt the user to record a reference audio at the first start of the project. Then, extract the acoustic features of the reference audio through the text-to-speech TTS engine, and achieve tone uniformity by saving and reusing the same speaker embedding vector at the project level.

[0046] s5.2, Call the rendering service of the corresponding presentation framework, start the Web service to render the presentation intermediate file. After the page is loaded, first use the DOM selector to identify and traverse the HTML element representing each slide, then perform the waiting logic to ensure that all dynamic content has been fully rendered. Then for each HTML element located, call the headless browser instance to take a screenshot of the specific rendering optimization page. This method can specify the screenshot area as the bounding box of the element to ensure the accuracy of the screenshot. At the same time, a high resolution parameter can be configured to output high-definition pictures suitable for video production. Finally, after the screenshot task is completed, automatically close the headless browser instance and terminate the local rendering service to release system resources.

[0047] Step 6: Video Compositing The materials generated in step 5 are then composited and packaged to generate the final video file. The specific steps are as follows: s6.1 Iterate through each audio and image pair generated in step 5 and obtain the audio duration. For each audio and image pair, call the video processing tool to generate an independent video clip. Key instruction parameters include: the video frame is a looping image, the video clip duration is exactly the same as the audio duration, and a standard encoding format is used to ensure compatibility.

[0048] Since the generation of each video segment is independent, the synthesis of multiple segments can be processed in parallel to shorten the total production time of long videos.

[0049] s6.2 Generate a list containing all video clip paths from s6.1, and then use the splicing function of the video processing tool to combine them in sequence into a complete video stream and audio.

[0050] In one alternative embodiment, video filters are applied during the stitching process to add smooth fade-in / fade-out, slide, or other configurable visual transition effects between video clips.

[0051] In one alternative embodiment, the user can input a background music file, and the system uses audio processing filters during the splicing process to mix the background music with the narration. When the narration plays, the background music volume is automatically reduced to ensure the narration is clear.

[0052] s6.3 encapsulates the video stream and audio synthesized in s6.2 into a specified standard video container to generate the final video file, provides the user with the corresponding download link, and can automatically clean up temporary files generated during the process. Figures 7~9 Based on Figures 4~6 The screenshot shown is a video frame generated from the intermediate file.

[0053] In an optional embodiment, the system can overlay an image watermark at a specified position on the video frame according to a preset template.

[0054] Those skilled in the art will understand that, in addition to the course instruction videos described above, the core ideas of this invention are also applicable to other video production fields that require precise proofreading of generated content, such as product function explanations and legal clause interpretations. The implementation methods are similar to those in this embodiment, and will not be repeated here.

Claims

1. A multimodal automated teaching video generation method based on intermediate files, characterized in that: Includes the following steps: Step 1: Use a language model to convert the text and image content related to the specified knowledge points in the user-input document into a structured JSON file; Step 2: Using a predefined template engine, the structured JSON file is converted into an intermediate presentation file that conforms to the syntax of a specific presentation framework and is both human-readable and machine-parseable. Step 3: Conduct human-computer collaborative review of intermediate files. First, the user previews and edits the files, and then the system performs rule-based reverse parsing and syntax verification. For content that passes verification, incremental updates are performed, and for content that fails verification, error location feedback is provided to the user. Step 4: Based on the incrementally updated structured JSON file, generate image material files and narration audio files to synthesize the final teaching video.

2. The multimodal automated teaching video generation method based on intermediate files as described in claim 1, characterized in that: Before step 1, a two-stage large language model is invoked. First, the text content related to the specified knowledge point in the user input document is retrieved and converted into a free-format teaching content draft. Then, the draft is combined with the formatted image content to generate a structured JSON file.

3. The multimodal automated teaching video generation method based on intermediate files as described in claim 2, characterized in that: Use a document parsing library to preprocess the user-input Word document and extract the text and images from the document; The text is segmented into paragraphs, and the vector representation of each chapter title and each paragraph is calculated through an embedding model to build a vector knowledge base with a two-level index, which is used to retrieve text content related to a specified knowledge point. The images are first exported in order and their relative positions are recorded. Then, a visual language model is used to generate a descriptive title for each image, thus building an image description knowledge base for retrieving image content related to specified knowledge points.

4. The multimodal automated teaching video generation method based on intermediate files as described in claim 1 or 3, characterized in that: Use a large visual language model to perform in-depth analysis of image content related to specified knowledge points; For images containing flowcharts, the code converts their topology into a chart description language; for images containing mathematical formulas, the code calls a mathematical formula OCR service to convert them into a specific formula typesetting language; for other complex charts, the core data is extracted, and an external professional chart generation service is called to generate a visualization image and return a chart link.

5. The multimodal automated teaching video generation method based on intermediate files as described in claim 1, characterized in that: Before performing rule-based reverse parsing in step 3, the build command is invoked to precompile the intermediate files of the presentation. If the precompilation fails, the error message and code location are fed back to guide the correction.

6. The multimodal automated teaching video generation method based on intermediate files as described in claim 1, characterized in that: Before generating the narration audio file, a reference audio is specified, and the acoustic features of the reference audio are extracted using a text-to-speech (TTS) engine. The acoustic features of the reference audio are saved and reused in the project to ensure the consistency of the timbre in the narration audio file.

7. The multimodal automated teaching video generation method based on intermediate files as described in claim 1, characterized in that: The web service is started to render the intermediate files of the presentation, outputting high-definition images as image material files, which are then combined with the corresponding narration audio files to generate a video clip; then all video clips are spliced ​​together in sequence to synthesize the final teaching video.

8. A multimodal automated teaching video generation method system based on intermediate files, characterized in that: include: The content generation module is used to generate a draft explanation based on the knowledge points specified by the user, parse and match the images, and generate a structured JSON file based on the draft explanation and the images. The format conversion module is used to convert the structured JSON file into a human-readable and machine-parseable markup language-based intermediate file using a predefined template engine; The collaborative review module is used to present the intermediate files to the user and receive modification instructions; The synchronous update module is used to reverse-parse and validate the modified intermediate file, identify the content changes, and update the original structured data corresponding to the content changes. The material generation module is used to generate corresponding audio and image materials based on the updated structured data; The video compositing module is used to combine the materials into a final video.

9. The multimodal automated teaching video generation method system based on intermediate files as described in claim 8, characterized in that: It also includes a parsing module, which extracts text and images from user-input Word documents to build a searchable knowledge base.

10. The multimodal automated teaching video generation method system based on intermediate files as described in claim 9, characterized in that: The content generation module uses a two-stage large language model call. First, it calls the large language model to retrieve relevant teaching content text from the searchable knowledge base based on the knowledge points specified by the user, and generates a free-format explanatory draft. Next, relevant images are retrieved and their formats are converted. Then, a large language model is called to combine the initial draft of the explanation with the converted image content to generate a structured JSON file.

Images