Teaching broadcast video generation method and apparatus

Abstract

The application discloses a teaching broadcast video generation method and device, and relates to the technical field of digital people, and the teaching broadcast video generation method comprises the following steps: receiving an input target file, a broadcast tone parameter and a digital person material resource, performing analysis processing on the target file, splitting to obtain multiple groups of single-page structured data, and processing each group of single-page structured data to generate multiple single-page explanation manuscripts; for each single-page explanation manuscript, processing the single-page explanation manuscript based on the broadcast tone parameter to generate a broadcast audio stream corresponding to the single-page explanation manuscript; based on the broadcast audio stream and the digital person material resource, constructing a digital person driving video stream; and rendering all single-page pictures, all broadcast audio streams and all digital person driving video streams to generate a teaching broadcast video corresponding to the target file. The application solves the technical problem that a file cannot be converted into a teaching video broadcast by a digital person in the related art.

Description

Technical Field

[0001] This invention relates to the field of digital human technology, and more specifically, to a method and apparatus for generating instructional broadcast videos. Background Technology

[0002] As corporate training, policy dissemination, product introductions, and solution explanations gradually shift online and to a larger scale, the traditional video production method of "human scriptwriting—human voice-over—live-on-camera recording—post-editing" has revealed problems such as long production cycles, high costs, inconsistent styles, and difficulty in rapid batch updates. Especially in scenarios such as customer training, industry course launches, and frequent iterations of internal policies, there is often a need to quickly convert documents such as PPTs, PDFs, and Word documents into video content with explanations. Manual workflows can no longer meet the high-frequency, standardized production demands.

[0003] In related technologies, document-to-video solutions mainly fall into the following categories: The first category is automatic video generation solutions based on template splicing, which usually rely on fixed text rules, preset shots, and material libraries to convert input text into simple video clips, but have limited ability to understand complex charts, page structures, and professional semantics; The second category is dubbed video solutions based on TTS (Text-to-Speech) and subtitle synthesis, which can generate audio and subtitles, but usually lack the ability to generate intelligent explanations for page content, and it is also difficult to achieve high consistency with digital human images, actions, and lip movements; The third category is end-to-end content generation solutions based on multimodal large models, which can understand document images and text to a certain extent, but if the entire document is directly subjected to unified reasoning, problems such as missing page information, misunderstanding of charts, lengthy and uncontrolled generated content, and excessive inference resource consumption are likely to occur.

[0004] Meanwhile, with the rapid development of digital human broadcasting technology, it has evolved from static image-driven to real-time or offline generation methods that support audio-driven lip-syncing, facial expressions, and motion synthesis. For example, a broadcasting solution based on a single digital human generates an explanatory video based on text or audio using a fixed digital human image. This solution typically lacks document understanding and page-by-page script generation capabilities, and it is also difficult to simultaneously support both built-in images and user-defined digital avatars as driving modes.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This invention provides a method and apparatus for generating instructional broadcast videos, which at least solves the technical problem in related technologies that files cannot be converted into instructional videos broadcast by digital humans.

[0007] According to one aspect of the present invention, a method for generating instructional broadcast videos is provided, comprising: receiving an input target file, broadcast timbre parameters, and digital human material resources; parsing the target file to obtain multiple sets of single-page structured data, and processing each set of single-page structured data to generate multiple single-page explanatory texts, wherein the single-page structured data includes at least: single-page images; for each single-page explanatory text, processing the single-page explanatory text based on the broadcast timbre parameters to generate a broadcast audio stream corresponding to the single-page explanatory text; constructing a digital human-driven video stream based on the broadcast audio stream and digital human material resources; and rendering all single-page images, all broadcast audio streams, and all digital human-driven video streams to generate an instructional broadcast video corresponding to the target file.

[0008] Further, the steps of parsing and processing the target file to obtain multiple sets of single-page structured data include: determining the file format of the target file, and converting the target file format to obtain a standardized file if the file format is not a preset format, wherein the format of the standardized file is a preset format; splitting the standardized file into multiple page sub-files, wherein each page sub-file corresponds to page number information; rendering each page sub-file to obtain multiple single-page images; extracting text from each page sub-file to obtain multiple text data; extracting the page structure from each page sub-file to obtain multiple page structure data; and for each page sub-file, aggregating the single-page images, text data, and page structure data of the sub-file to obtain the single-page structured data corresponding to the sub-file.

[0009] Further, the steps for processing each set of single-page structured data to generate multiple single-page explanatory texts include: performing layout detection on the single-page images in the single-page structured data to obtain the detection results; if the detection results indicate that the single-page images include non-text content, processing the single-page images and text data in the single-page structured data based on contextual constraints to obtain single-page explanatory texts, wherein the contextual constraints include at least: duration constraints, terminology consistency constraints, style constraints, and coherence constraints; if the detection results indicate that the single-page images do not include non-text content, processing the text data based on contextual constraints to obtain single-page explanatory texts.

[0010] Furthermore, the steps of processing the single-page explanation text based on the broadcast timbre parameters to generate the corresponding broadcast audio stream include: processing the single-page explanation text to obtain standardized broadcast text; determining the timbre type of the broadcast timbre parameters; if the timbre type is a preset timbre type, obtaining the preset text and preset voiceprint features corresponding to the preset timbre type from the preset timbre library, wherein the preset timbre library includes: multiple preset timbre types and preset text and preset voiceprint features corresponding to each preset timbre type; performing sound synthesis based on the preset text, preset voiceprint features, and standardized broadcast text to generate the broadcast audio stream; if the timbre type is not a preset timbre type, identifying the broadcast timbre parameters to obtain reference text; and performing sound synthesis based on the reference text, broadcast timbre parameters, and standardized broadcast text to generate the broadcast audio stream.

[0011] Further, the steps of processing the single-page explanation text to obtain a standardized broadcast text include: pausing the single-page explanation text to obtain a first initial broadcast text; standardizing the first initial broadcast text to obtain a second initial broadcast text; performing speech processing on the second initial broadcast text to obtain a third initial broadcast text; and performing audio continuity processing on the third initial broadcast text to obtain a standardized broadcast text.

[0012] Furthermore, the steps for constructing a digital human-driven video stream based on the broadcast audio stream and digital human material resources include: determining the resource type of the digital human material resources, wherein the resource type includes: preset video type, external video type, and image type; and using a driving strategy based on the resource type to process the broadcast audio stream and digital human material resources to construct a digital human-driven video stream.

[0013] Furthermore, a resource-type-driven strategy is adopted to process the broadcast audio stream and digital human material resources, constructing a digital human-driven video stream. This includes: when the resource type of the digital human material resources is a preset video type, obtaining a preset video template corresponding to the preset video type; aligning the video length of the preset video template with the audio length of the broadcast audio stream to obtain a target video template with equal video and audio lengths; calling the digital human model corresponding to the preset video type and using the digital human model to process the digital human material resources and the broadcast audio stream to obtain a digital human-driven video stream, wherein the digital human model is a model trained based on historical human videos; when the resource type of the digital human material resources is an external video type, performing image matting on the digital human material resources to obtain the foreground and background regions of the human figure; driving the foreground region of the human figure based on the broadcast audio stream to obtain the foreground video of the human figure; processing the foreground video of the human figure and the background region to obtain the digital human-driven video stream; when the resource type of the digital human material resources is an image type, extracting the human figure region from the digital human material resources; driving the human figure region based on the broadcast audio stream to obtain the digital human-driven video stream.

[0014] Furthermore, the steps for rendering all single-page images, all audio streams for broadcasting, and all digital human-driven video streams to generate the teaching broadcast video corresponding to the target file include: obtaining the subtitle file and, with background music enabled, obtaining the background music audio stream; performing audio ducking mixing on the background music audio stream and all broadcast audio streams to obtain multi-track audio; and, based on a preset rendering strategy, rendering all single-page images, all digital human-driven video streams, subtitle files, and multi-track audio to generate the teaching broadcast video. The preset rendering strategy includes at least: page dwell time and transition animation control, digital human position and size control, occlusion relationship control between the digital human and page materials control, subtitle safe area layout control, and background layer compositing logic control.

[0015] Furthermore, the steps of obtaining the subtitle file and, if background music is enabled, obtaining the background music audio stream include: performing timeline mapping processing on the broadcast audio stream and the single-page explanation text to generate the subtitle file; if background music is enabled, extracting features from the preset teaching broadcast video requirements to obtain music features; and obtaining the background music audio stream based on the music features.

[0016] According to another aspect of the present invention, a teaching broadcast video generation apparatus is also provided, comprising: a receiving unit for receiving an input target file, broadcast timbre parameters, and digital human material resources; a processing unit for parsing the target file, splitting it into multiple sets of single-page structured data, and processing each set of single-page structured data to generate multiple single-page explanatory texts, wherein the single-page structured data includes at least: single-page images; a generation unit for processing each single-page explanatory text based on the broadcast timbre parameters to generate a broadcast audio stream corresponding to the single-page explanatory text; a construction unit for constructing a digital human-driven video stream based on the broadcast audio stream and digital human material resources; and a rendering unit for rendering all single-page images, all broadcast audio streams, and all digital human-driven video streams to generate a teaching broadcast video corresponding to the target file.

[0017] Furthermore, the processing unit is also used to: determine the file format of the target file, and if the file format is not a preset format, convert the target file format to obtain a standardized file, wherein the format of the standardized file is a preset format; segment the standardized file to obtain multi-page sub-files, wherein each page sub-file corresponds to page number information; render each page sub-file to obtain multiple single-page images; extract text from each page sub-file to obtain multiple text data; extract the page structure from each page sub-file to obtain multiple page structure data; and for each page sub-file, aggregate the single-page images, text data, and page structure data of the sub-file to obtain the single-page structured data corresponding to the sub-file.

[0018] Furthermore, the processing unit is also used to: perform layout detection on the single-page images in the single-page structured data to obtain detection results; when the detection results indicate that the single-page images include non-text content, process the single-page images and text data in the single-page structured data based on contextual constraints to obtain a single-page explanatory text, wherein the contextual constraints include at least: duration constraints, terminology consistency constraints, style constraints, and coherence constraints; when the detection results indicate that the single-page images do not include non-text content, process the text data based on contextual constraints to obtain a single-page explanatory text.

[0019] Furthermore, the generation unit is also used to: process the single-page explanatory text to obtain standardized broadcast text; determine the timbre type of the broadcast timbre parameters; when the timbre type is a preset timbre type, obtain the preset text and preset voiceprint features corresponding to the preset timbre type from the preset timbre library, wherein the preset timbre library includes: multiple preset timbre types and preset text and preset voiceprint features corresponding to each preset timbre type; perform sound synthesis based on the preset text, preset voiceprint features, and standardized broadcast text to generate a broadcast audio stream; when the timbre type is not a preset timbre type, identify the broadcast timbre parameters to obtain reference text; and perform sound synthesis based on the reference text, broadcast timbre parameters, and standardized broadcast text to generate a broadcast audio stream.

[0020] Furthermore, the generation unit is also used to: pause the single-page explanatory text to obtain a first initial broadcast text; standardize the first initial broadcast text to obtain a second initial broadcast text; perform speech processing on the second initial broadcast text to obtain a third initial broadcast text; and perform audio continuity processing on the third initial broadcast text to obtain a standardized broadcast text.

[0021] Furthermore, the construction unit is also used to: determine the resource type of the digital human material resources, wherein the resource types include: preset video type, external video type, and image type; and adopt a driving strategy based on the resource type to process the broadcast audio stream and digital human material resources to construct a digital human-driven video stream.

[0022] Furthermore, the construction unit is also used for: when the resource type of the digital human material resource is a preset video type, obtaining a preset video template corresponding to the preset video type; aligning the video length of the preset video template with the audio length of the broadcast audio stream to obtain a target video template with the same video and audio lengths; calling the digital human model corresponding to the preset video type, and using the digital human model to process the digital human material resource and the broadcast audio stream to obtain a digital human-driven video stream, wherein the digital human model is a model trained based on historical human videos; when the resource type of the digital human material resource is an external video type, performing image matting on the digital human material resource to obtain the foreground and background regions of the person; driving the foreground region of the person based on the broadcast audio stream to obtain the foreground video of the person; processing the foreground video of the person and the background region to obtain a digital human-driven video stream; when the resource type of the digital human material resource is an image type, extracting the person region from the digital human material resource; driving the person region based on the broadcast audio stream to obtain a digital human-driven video stream.

[0023] Furthermore, the rendering unit is also used to: acquire subtitle files, and, if background music is enabled, acquire the background music audio stream; perform audio ducking mixing on the background music audio stream and all broadcast audio streams to obtain multi-track audio; and, based on a preset rendering strategy, render all single-page images, all digital human-driven video streams, subtitle files, and multi-track audio to generate a teaching broadcast video. The preset rendering strategy includes at least: page dwell time and transition animation control, digital human position and size control, occlusion relationship control between digital human and page materials control, subtitle safe area layout control, and background layer compositing logic control.

[0024] Furthermore, the rendering unit is also used to: perform timeline mapping processing on the broadcast audio stream and single-page explanatory text to generate subtitle files; extract features from the preset teaching broadcast video requirements to obtain music features when background music is enabled; and obtain the background music audio stream based on the music features.

[0025] According to another aspect of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the teaching broadcast video generation method described above.

[0026] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement any of the above-described teaching broadcast video generation methods.

[0027] In this invention, the system receives an input target file, broadcast tone parameters, and digital human material resources. The target file is parsed and processed to obtain multiple sets of single-page structured data. Each set of single-page structured data is processed to generate multiple single-page explanatory texts. For each single-page explanatory text, based on the broadcast tone parameters, the text is processed to generate a corresponding broadcast audio stream. Based on the broadcast audio stream and digital human material resources, a digital human-driven video stream is constructed. All single-page images, all broadcast audio streams, and all digital human-driven video streams are rendered to generate a teaching broadcast video corresponding to the target file. This solves the technical problem in related technologies where files cannot be converted into teaching videos broadcast by digital humans.

[0028] This invention employs an integrated collaborative processing flow encompassing document parsing, intelligent routing script generation, adaptive sound synthesis, multi-track digital human driving, and multi-track fusion rendering. By receiving target files, timbre parameters, and digital human material resources, the target file is parsed and split into single-page structured data containing multi-dimensional information such as single-page images. Based on the page content type, a language big model or a visual-language multimodal big model is intelligently invoked to generate a single-page narration script adapted for the spoken version. Combined with timbre feature extraction and text front-end optimization, highly natural-sounding audio is generated. Furthermore, a differentiated driving strategy is used for built-in or custom digital human resources to construct synchronized lip movements and facial expressions. The digital human video stream ultimately unifies the timing synchronization and layout rendering of page screens, digital human videos, broadcast audio, background sound, and subtitles, achieving the goal of fully automated closed-loop generation from multi-source courseware documents to standardized lecturer broadcast videos. This significantly reduces the costs of manual scriptwriting, dubbing, recording, and editing, and improves video production efficiency, content generation quality, and system engineering adaptability. It also solves problems such as long production cycles and high costs caused by reliance on manual processes in related technologies, as well as the lack of intelligent understanding and routing for complex document pages, the disconnect between timbre and image-driven approaches, and inconsistent fusion of multi-track materials in existing automation solutions. Attached Figure Description

[0029] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0030] Figure 1 This is a flowchart of an optional teaching broadcast video generation method according to an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of an optional document parsing and processing layer flow according to an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of an optional large-model intelligent routing layer process according to an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of an optional sound adaptive synthesis layer process according to an embodiment of the present invention;

[0034] Figure 5 This is a schematic diagram of an optional three-track digital human driving layer process according to an embodiment of the present invention;

[0035] Figure 6 This is a schematic diagram of an optional multi-track fusion rendering layer process according to an embodiment of the present invention;

[0036] Figure 7This is a schematic diagram of an optional intelligent audiovisual auxiliary generation layer process according to an embodiment of the present invention;

[0037] Figure 8 This is a schematic diagram of an optional teaching broadcast video generation process based on multimodal document understanding and digital human collaborative driving according to an embodiment of the present invention;

[0038] Figure 9 This is a schematic diagram of an optional teaching broadcast video generation device according to an embodiment of the present invention;

[0039] Figure 10 This is a hardware structure block diagram of an electronic device (or mobile device) for generating teaching broadcast videos according to an embodiment of the present invention. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0041] It should be noted that the terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0042] It should be noted that all related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) collected and involved in this invention are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data comply with the relevant laws, regulations, and standards of the relevant regions, and necessary measures have been taken to ensure compliance with public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse. For example, this system has an interface with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface, and the relevant information is obtained only after receiving consent from the aforementioned user or organization.

[0043] This invention proposes an integrated digital human broadcasting video generation solution for teaching / training documents. Through document parsing, intelligent routing of page-level large models, adaptive sound synthesis, dual-track digital human driving, and multi-track fusion rendering, it achieves automated generation from PPT / PDF / Word documents to complete lecturer broadcasting videos. This solves the problem of relying on manual page-by-page writing, dubbing, recording, and editing in existing document-to-lecture video conversion processes, enabling automatic conversion of corporate training documents into standardized lecturer broadcasting videos. By providing an intelligent routing mechanism based on page content type, it addresses the issue of poor unified processing when plain text pages and mixed text / image pages coexist, improving the quality and stability of page-by-page script generation. By proposing a dual-channel processing architecture that allows for parallel compatibility of built-in and custom resources, it solves the problem of process fragmentation caused by significant differences in audio and digital human image sources in existing systems. Furthermore, it resolves the fragmentation issues of subtitle timestamp generation, background music matching, page material synchronization, and multi-track audio / video synthesis, forming a complete and deployable automated video production chain.

[0044] The present invention will now be described in detail with reference to various embodiments.

[0045] Example 1

[0046] According to an embodiment of the present invention, an embodiment of a teaching broadcast video generation method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0047] Figure 1 This is a flowchart of an optional teaching broadcast video generation method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0048] Step S101: Receive the input target file, playback tone parameters, and digital human material resources.

[0049] In this embodiment of the invention, it can be applied to a teaching broadcast video generation system, which includes: a document parsing and processing layer, a large model intelligent routing layer, an adaptive sound synthesis layer, a three-track digital human driving layer, an intelligent audiovisual auxiliary generation layer, and a multi-track fusion rendering layer.

[0050] The document parsing and processing layer can receive teaching courseware documents (target files) such as PPT files, Word files, or PDF files input by the user, and also receive broadcast tone parameters and digital human material resources selected or input by the user. Here, broadcast tone parameters are used to specify the lecturer's voice characteristics in the final video, including system-built-in preset tones or custom tones determined by user-uploaded reference audio; digital human material resources refer to video templates or static images used to drive the virtual lecturer avatar.

[0051] Step S102: Parse the target file to obtain multiple sets of single-page structured data, and process each set of single-page structured data to generate multiple single-page explanatory texts. The single-page structured data includes at least one single-page image.

[0052] In this embodiment of the invention, the document parsing and processing layer can uniformly convert PPT and Word files into PDF intermediate format, use a PDF pagination engine to split the document page by page, perform image rendering on each page in parallel to obtain a single-page image that retains the layout, extract the underlying characters to obtain plain text, and extract metadata such as layout coordinates, and aggregate to form single-page structured data.

[0053] The large-scale intelligent routing layer can detect whether a page contains non-pure text elements such as charts. If it is a pure text page, it calls the language large model to polish and rewrite the underlying text in a conversational style to generate a single-page explanatory text. If it is a page with mixed text and images, it calls the visual language multimodal large model and combines it with the single-page images to perform content deduction and refinement to generate the corresponding single-page explanatory text.

[0054] Step S103: For each single-page explanation script, process the single-page explanation script based on the broadcast timbre parameters to generate the corresponding broadcast audio stream.

[0055] In this embodiment of the invention, the adaptive voice synthesis layer can generate a corresponding lecturer audio stream based on the lecturer's voice type selected by the user and the text of each page of the lecture script. If the voice parameter is a built-in voice, the preset reference text and voiceprint configuration are read directly; if it is a custom voice, the reference audio uploaded by the user is identified through the ASR (Automatic Speech Recognition) module to extract the text, and the original audio is combined with it as the voice condition. At the same time, front-end optimizations such as sentence segmentation, digital normalization, and speech rate control are performed on the single-page lecture script. Finally, the optimized text and voice conditions are input into the voice cloning synthesis model to generate a lecture audio stream consistent with the target voice.

[0056] Step S104: Based on the broadcast audio stream and digital human material resources, construct a digital human-driven video stream.

[0057] In this embodiment of the invention, the three-track digital human driving layer can select different digital human driving paths according to the type of digital human resource template, generating dynamic digital human video materials that can be superimposed on a specified area of ​​the teaching page. That is, the type and source of the digital human material resources can be detected first. If it is a system-built-in video template, a pre-trained high-fidelity digital human model is called, and a lip-synced video is generated by aligning the audio and video duration (loop expansion or truncation) and calling the cache mask. If it is a user-defined video template, real-time image matting is performed to separate the foreground of the person, and then the zero-shot generalization driving model is called to drive the lip movements and expressions according to the broadcast audio stream to generate the video foreground. If it is an image template, image matting is performed first, and then the photo digital human model is called to generate a speaking video containing head movements and blinking actions, which is then blended with the background.

[0058] Step S105: Render all single-page images, all broadcast audio streams, and all digital human-driven video streams to generate the teaching broadcast video corresponding to the target file.

[0059] In this embodiment of the invention, the multi-track fusion rendering layer can uniformly synthesize and render page image materials (single-page images), dynamic digital human video materials (digital human-driven video stream), subtitle timeline data (output by the intelligent audiovisual auxiliary generation layer), lecturer narration audio stream, and background music (output by the intelligent audiovisual auxiliary generation layer), outputting a complete lecturer narration teaching video file. Specifically, it can perform ducking mixing on the audio stream and background music to ensure clear speech, use a forced alignment algorithm to map the single-page explanation text and narration audio stream to generate timeline subtitles, and through the final rendering engine, according to preset layout control strategies (such as digital human position, subtitle safe area, page dwell time, etc.), synchronize and multi-track the single-page images, digital human video, audio, and subtitles in time, outputting the final video file.

[0060] In summary, an integrated collaborative processing workflow can be adopted, encompassing document parsing, intelligent routing script generation, adaptive sound synthesis, multi-track digital human driving, and multi-track fusion rendering. This workflow receives target files, timbre parameters, and digital human material resources. The target file is parsed and broken down into single-page structured data containing multi-dimensional information such as single-page images. Based on the page content type, a large language model or a visual-language multimodal model is intelligently invoked to generate a single-page narration script adapted for the spoken delivery. Combined with timbre feature extraction and text front-end optimization, highly natural-sounding audio is generated. Furthermore, a differentiated driving strategy is employed for built-in or custom digital human resources to construct synchronized lip movements and facial expressions. The digital human video stream ultimately unifies the timing synchronization and layout rendering of page images, digital human videos, broadcast audio, background sound, and subtitles, achieving the goal of fully automated closed-loop generation from multi-source courseware documents to standardized lecturer broadcast videos. This significantly reduces the costs of manual scriptwriting, voice-over, recording, and editing, while improving video production efficiency, content generation quality, and system engineering adaptability. It also solves problems such as long production cycles and high costs caused by reliance on manual processes in related technologies, as well as the lack of intelligent understanding and routing for complex document pages, the disconnect between timbre and image-driven approaches, and inconsistent fusion of multi-track materials in existing automation solutions.

[0061] To improve the accuracy of parsing and processing target files, the teaching broadcast video generation method provided in Embodiment 1 of this application determines the file format of the target file. If the file format is not a preset format, the target file format is converted to obtain a standardized file, wherein the standardized file format is a preset format. The standardized file is then segmented into multi-page sub-files, each sub-file corresponding to page number information. Each sub-file is rendered to obtain multiple single-page images. Text is extracted from each sub-file to obtain multiple text data. Page structure is extracted from each sub-file to obtain multiple page structure data. For each sub-file, the single-page images, text data, and page structure data are aggregated to obtain the corresponding single-page structured data.

[0062] In this embodiment of the invention, the file extension or header information of the input target file is first identified to determine its file format. If the file format is not PDF (i.e., not a preset format), a format conversion service is invoked, and a document processing library is used to convert the PPT file or Word file into PDF format, thereby obtaining a standardized file.

[0063] A standardized PDF pagination engine is used to read the document, dividing it into independent sub-files based on page boundaries, and adding page number index information to each sub-file. For each sub-file, an image rendering engine is used to render it as a bitmap or vector image to generate a single-page image. Simultaneously, a text parsing interface is called to extract the underlying character sequences from the sub-file as text data, and the coordinates, size, and hierarchical relationships of each element (such as images, text boxes, and charts) in the sub-file are parsed to generate page structure data. Finally, the single-page image, text data, page structure data, and corresponding page number information are encapsulated into a single data structure to form structured single-page data.

[0064] Here, the default format refers to the standard intermediate format that the system processes uniformly, namely PDF format, which is used to eliminate the parsing complexity caused by differences in the formats of different source files.

[0065] Figure 2 This is a schematic diagram of an optional document parsing and processing layer flow according to an embodiment of the present invention, such as... Figure 2 As shown, the document parsing and processing layer is used to uniformly process PPT, Word, and PDF files, and output single-page structured data that can be directly accessed by subsequent modules. The document parsing and processing layer includes: a multi-source file input layer, a format unification and preprocessing layer, a core pagination and parallel extraction layer, and a structured data encapsulation output layer. First, the multi-source file input layer receives PPT, Word, or PDF files input by the user. PPT and Word files are first converted to standard PDF documents by calling the format unification and preprocessing layer (e.g., PPT to PDF, Word to PDF). PDF files directly enter the subsequent process, thus forming a unified intermediate format: a standardized PDF document. Then, the core pagination and parallel extraction layer uses a PDF pagination engine to traverse the document, segment it by page, and record page number information. For each page, three processes are performed in parallel: image rendering, text extraction, and metadata extraction. Specifically, the image rendering component rasterizes the page into a single-page image to preserve the original layout and text / image content; the text extraction component parses underlying character and paragraph information to extract plain text content; and the metadata extraction component parses layout coordinates and source materials to characterize the page structure. Finally, a structured data encapsulation output layer aggregates extracted attributes such as page number, single-page image, underlying plain text, layout information, and source material index to output a single-page structured data package. This layer converts teaching documents from different sources into standardized page data, providing the foundational input for subsequent lecture script generation, digital human narration, and video rendering.

[0066] In this embodiment, the standardized conversion and structured decomposition of multi-source heterogeneous documents (PPT, Word, PDF) into a unified internal data format are realized, ensuring the compatibility and data integrity of the document parsing layer, avoiding parsing loss or errors caused by differences in file formats or complex page structures, thereby ensuring the reliability of the data foundation for subsequent lecture generation and video rendering processes.

[0067] To improve the accuracy of generating single-page explanatory texts, the teaching broadcast video generation method provided in Embodiment 1 of this application performs layout detection on the single-page images in the single-page structured data to obtain detection results. If the detection results indicate that the single-page images include non-text content, the single-page images and text data in the single-page structured data are processed based on contextual constraints to obtain the single-page explanatory text. These contextual constraints include at least: duration constraints, terminology consistency constraints, style constraints, and coherence constraints. If the detection results indicate that the single-page images do not include non-text content, the text data is processed based on contextual constraints to obtain the single-page explanatory text.

[0068] In this embodiment of the invention, a layout analysis model or rule engine is first invoked to scan and analyze the single-page images in the single-page structured data to identify whether there are non-pure text elements such as charts, flowcharts, complex formulas or images on the page, thereby outputting the detection results.

[0069] If the detection result indicates that the page contains non-text content, the single-page image and text data are input together into the visual language multimodal large model. At the same time, contextual constraints are injected as prompts or system instructions to guide the model to combine image visual information and text semantics to perform content inference, chart interpretation, and key point extraction, generating a single-page explanatory text. If the detection result indicates that the page does not contain non-text content, that is, it is determined to be a pure text page, the image understanding step is skipped, and the text data is directly input into the language large model. Similarly, the text is compressed, polished, and rewritten in a conversational style by combining contextual constraints to generate a single-page explanatory text.

[0070] Here, contextual constraints refer to the set of parameters used to limit and guide the generation behavior of large models. Specifically, these include: "duration length constraints," used to control the number of words in the generated text or the expected broadcast duration to match the video rhythm; "terminology consistency constraints," used to ensure the consistent expression of professional terms and proper nouns throughout the document; "style constraints," used to stipulate that the generated text must conform to the specific expression style of the lecturer's broadcast, such as conversational, friendly, or rigorous; and "coherence constraints," used to ensure that the lecture script generated on the current page is logically and semantically connected to the previous or next page.

[0071] Figure 3This is a schematic diagram of an optional large-model intelligent routing layer process according to an embodiment of the present invention, such as... Figure 3 As shown, the large-model intelligent routing layer is used to automatically select either a language large model or a visual-language multimodal large model to generate a single-page speech script suitable for video broadcasting, based on the content type of the single-page structured data. The input to this layer is single-page structured data, which includes: page image content and underlying plain text content.

[0072] The specific process is as follows: First, the image content is fed into the layout detection model to determine whether the current page contains non-pure text content such as images / charts / flowcharts / tables. If the detection result is negative, it means that the page is mainly text information, and there is no need to call the visual language multimodal large model for complex image understanding. At this time, the underlying pure text is directly input into the language large model LLM (Large Language Model) for underlying text compression, polishing and conversational rewriting, outputting a single-page speech script suitable for video broadcasting. This reduces the computational power consumption and latency overhead in the model inference process while meeting the text explanation requirements, thereby improving the overall generation efficiency. If the detection result is yes, it means that the current page contains images, charts, process relationships, or layout semantic information. Relying solely on the underlying plain text is insufficient to fully reconstruct the page's content. In this case, the input image content is fed into the Vision-Language Model (VLM). The VLM combines the image information to perform image deduction, chart interpretation, and key point extraction, outputting a single-page speech script suitable for video presentation. This enhances the system's ability to understand the semantics of complex pages and avoids the omission of key information due to relying solely on text extraction.

[0073] To ensure consistency and coherence of the presentation slides across pages during the broadcasting process, a global context constraint mechanism is introduced during the generation of the large models (Language Large Model LLM and Visual Language Multimodal Large Model VLM). This global context constraint mechanism includes at least the following: control constraints: control of broadcast duration and text length; lexicon constraints: consistency constraints on professional terminology; style constraints: constraints on colloquial language and speaker persona; and coherence constraints: constraints on the connection between the content of the previous page. Through these constraints, the generated slides from different pages maintain uniformity in length, terminology, expression style, and contextual coherence. Finally, the large model's intelligent routing layer outputs single-page presentation slides suitable for video broadcasting, providing text input for subsequent speech synthesis, digital human-driven processes, and subtitle generation. It also automatically matches different model paths based on the complexity of the page content, improving overall processing efficiency and content adaptability while ensuring the quality of the generated slides.

[0074] In this embodiment, the approach of uniformly calling the visual-language multimodal large model for all pages is not adopted. This is because directly using the visual-language multimodal large model for plain text pages would result in higher memory usage, computational overhead, and inference latency, leading to unnecessary resource waste. Furthermore, when processing simple text pages, the multimodal large model may over-rely on visual semantic inference, producing extended interpretations or illusory content inconsistent with the original text, affecting the accuracy of the presentation. Therefore, through page content detection and model routing mechanisms, a language large model is used for plain text pages, while a visual-language multimodal large model is used for mixed text and image pages. This achieves a balance between resource consumption, inference efficiency, and result reliability while ensuring content generation quality.

[0075] In this embodiment, by introducing an intelligent routing mechanism based on page content type and global context constraints, the problems of information omission, content illusion and style inconsistency that are prone to occur when related technologies process mixed documents are solved. This achieves a balance between the accuracy of lecture generation, generation efficiency and content coherence, and improves the professionalism and viewing experience of the final teaching video.

[0076] To improve the accuracy of generating the audio stream corresponding to the single-page explanation text, the teaching broadcast video generation method provided in Embodiment 1 of this application processes the single-page explanation text to obtain standardized broadcast text; determines the timbre type of the broadcast timbre parameters; if the timbre type is a preset timbre type, obtains the preset text and preset voiceprint features corresponding to the preset timbre type from the preset timbre library, wherein the preset timbre library includes: multiple preset timbre types and preset text and preset voiceprint features corresponding to each preset timbre type; performs sound synthesis based on the preset text, preset voiceprint features, and standardized broadcast text to generate a broadcast audio stream; if the timbre type is not a preset timbre type, identifies the broadcast timbre parameters to obtain reference text; and performs sound synthesis based on the reference text, broadcast timbre parameters, and standardized broadcast text to generate a broadcast audio stream.

[0077] In this embodiment of the invention, the received single-page explanatory text is first subjected to text front-end optimization processing, including sentence segmentation, pause marking, standardization of numbers and proper nouns, and speech rate and intonation marking, thereby generating standardized broadcast text that conforms to the input specifications of the speech synthesis engine. Here, standardized broadcast text refers to text data that has been adapted for processing by the speech synthesis engine after format standardization, punctuation optimization, and prosody marking.

[0078] The system reads the broadcast tone parameters to determine their tone type. If the result is a preset tone type, it queries the built-in preset tone library to retrieve the corresponding preset text and preset voiceprint features, using both as tone reference conditions. If the result is not a preset tone type (i.e., a user-defined tone), it calls the Automatic Speech Recognition (ASR) module to transcribe the reference audio contained in the broadcast tone parameters, extracts the reference text, and uses it along with the original broadcast tone parameters as tone reference conditions. Here, the timbre parameters refer to the input data used to define the characteristics of the output speech sound, which can be the system's built-in timbre identifiers or audio files uploaded by the user; timbre type refers to the classification of the sound source attributes, divided into system-preset timbre types and user-defined non-preset timbre types; the preset timbre library refers to the database stored internally by the system that contains various preset timbres and their corresponding acoustic model parameters; the preset text refers to the reference sentences pre-recorded and annotated for each preset timbre for extracting voiceprint features; the preset voiceprint features refer to the vector data extracted from the preset text that characterizes the physical attributes of a specific timbre; and the reference text refers to the corresponding text content identified from user-provided custom reference audio using ASR technology.

[0079] The obtained text data (preset text or reference text) and voiceprint features (preset voiceprint features or features extracted from the broadcast timbre parameters), along with the standardized broadcast text, are input into the sound cloning synthesis model to perform inference calculations and output the corresponding broadcast audio stream.

[0080] In this embodiment, a dual-channel processing architecture for adaptive sound synthesis is implemented, which not only ensures the fast recall efficiency of built-in timbres but also supports personalized cloning of user-defined timbres. Combined with text front-end optimization, it effectively solves the problems of poor audio continuity across pages, unnatural pronunciation, and inconsistent timbres, thereby improving the auditory experience of the final teaching video.

[0081] To improve the accuracy of generating standardized broadcast text, in the teaching broadcast video generation method provided in Embodiment 1 of this application, pause processing is applied to a single-page explanation script to obtain a first initial broadcast text; the first initial broadcast text is standardized to obtain a second initial broadcast text; the second initial broadcast text is processed for speech to obtain a third initial broadcast text; and the third initial broadcast text is processed for audio continuity to obtain a standardized broadcast text.

[0082] In this embodiment of the invention, a single-page explanatory text is first received. Natural language processing technology is used to identify syntactic structure and semantic boundaries, and pause markers (such as “…”) are inserted between sentences and phrases. <break>(Labels or specific time intervals) are used to generate the first initial broadcast text containing prosodic information.

[0083] Non-standard characters in the first initial broadcast text are cleaned and converted, for example, Arabic numerals are converted into Chinese pronunciation, English abbreviations are expanded into full names, and special symbols are converted into speech descriptions, to obtain a second initial broadcast text that conforms to spoken language habits.

[0084] The second initial broadcast text is subjected to phoneme-level pronunciation correction and stress marking. For polyphonic characters, the correct pronunciation is determined according to the context and intonation changes (such as rising and falling intonation) are marked to generate a third initial broadcast text containing detailed pronunciation instructions.

[0085] Analyze the acoustic features of the third initial broadcast text on the current page and the end of the broadcast audio on the previous page. By adjusting the speech rate and tone at the beginning of the current page or inserting brief silence intervals, eliminate the abruptness at the audio splicing points and generate the final standardized broadcast text for the synthesis engine.

[0086] Figure 4 This is a schematic diagram of an optional sound adaptive synthesis layer process according to an embodiment of the present invention, such as... Figure 4 As shown, the adaptive sound synthesis layer generates the corresponding lecturer audio stream based on the lecturer's voice type selected by the user and the text of each page of the lecture script. This layer mainly includes a voice feature extraction module, a lecture script text front-end processing module (TTS Frontend), and a core clone synthesis and output module.

[0087] First, in the timbre feature extraction module, the system receives the user-inputted broadcast reference audio and determines whether the selected timbre is a built-in system timbre. If it is a built-in timbre, the system directly reads and extracts the preset reference text and voiceprint configuration to obtain the preset text and voiceprint features, which serve as the timbre conditional input for subsequent speech synthesis. If it is a custom timbre, the system performs an ASR operation, which identifies the reference audio and extracts the audio reference text through the ASR module. This text is then combined with the original reference audio to form the conditional input for voice cloning, thereby achieving the preservation and reproduction of the user-defined lecturer timbre.

[0088] Simultaneously, the front-end processing module receives each page of the lecture script and performs text optimization processing on it. This optimization is achieved through acoustic and text control strategies, including: sentence segmentation and pause control, standardization of numbers and proper nouns, dynamic control of speech rate and intonation, and processing of audio continuity across pages. By standardizing the lecture text at the front end, the synthesized speech becomes more natural in terms of pauses, rhythm, pronunciation accuracy, and page transitions, improving the intelligibility and coherence of the final lecturer's audio.

[0089] In the core cloning synthesis and output module, preset text and voiceprint features, or recognized reference text and original reference audio, along with the front-end optimized broadcast text, are input into the voice cloning synthesis model to output a lecturer broadcast audio stream with the same timbre as the target timbre. This layer of processing supports both quick access to the system's built-in standard timbres and adaptive cloning of user-defined timbres. While ensuring broadcast efficiency, it also considers timbre consistency, natural pronunciation, and page continuity, providing stable audio input for subsequent digital human-driven and video rendering processes.

[0090] In this embodiment, a fine-grained text optimization link for the broadcast is constructed, which improves the naturalness, intelligibility, and smooth transition of audio across pages of the speech synthesis. This avoids the post-production modification costs caused by non-standard text format or abrupt audio transitions, and ensures the consistency of the final teaching video's auditory experience.

[0091] To improve the accuracy of constructing a digital human-driven video stream, the teaching broadcast video generation method provided in Embodiment 1 of this application determines the resource type of the digital human material resources, wherein the resource types include: preset video type, external video type, and image type; and adopts a driving strategy based on the resource type to process the broadcast audio stream and digital human material resources to construct a digital human-driven video stream.

[0092] In this embodiment of the invention, the file attributes or metadata of the digital human material resources are first read to identify their format characteristics and determine the resource type. If it is identified as a video file and its source is a system built-in library, it is determined to be a preset video type; if it is identified as a video file and its source is a user upload, it is determined to be an external video type; if it is identified as a static image file, it is determined to be an image type. Here, digital human material resources refer to the input source used to generate virtual lecturer images, which can be videos or images; the resource type is a classification of the source and form of digital human material, specifically including preset video types (high-quality video templates built into the system), external video types (video templates with backgrounds uploaded by users), and image types (static human images).

[0093] Based on the determined resource type, the corresponding driving strategy is executed: For preset video types, a pre-trained high-fidelity digital human model is invoked, and the video is directly generated by aligning audio and video durations (loop expansion or truncation) and invoking cached video masks; for external video types, real-time image matting is first performed to separate the foreground of the person, and then a zero-shot generalization driving model is invoked to generate the video foreground by driving lip movements and facial movements based on the broadcast audio stream; for image types, image matting is first performed to obtain the main subject of the person, and then a photo digital human model is invoked to generate a speaking video containing head movements and synchronized lip movements based on the broadcast audio stream, and then blended with the background. Finally, the dynamic human video material generated by the above processing is synthesized with the background to construct a digital human-driven video stream containing a digital human image. Here, the driving strategy refers to the specific combination of image processing and model inference algorithms used for different resource types; the high-fidelity digital human model refers to a dedicated model trained on specific built-in videos that can maintain high consistency of the person.

[0094] For example, through a three-track digital human driving layer, different digital human driving paths can be selected based on the type of digital human resource template to generate dynamic digital human video materials that can be overlaid on a specified area of ​​the teaching page. The input to this layer is a digital human resource template and synthesized audio. First, the input template medium is detected to determine whether it is a video template or an image template. If it is a video template, it further determines whether its source is a system-built-in template or a user-uploaded template. If it is an image template, it enters the image digital human driving path. Through the above medium recognition and source recognition mechanism, adaptive processing of different digital human resources is achieved.

[0095] In this embodiment, by constructing a three-track digital human driving mechanism, adaptive processing for three different resource types—built-in videos, custom videos, and images—is achieved. This not only utilizes a pre-trained model to ensure the high naturalness and efficiency of the built-in image, but also expands the compatibility with user-defined resources through zero-sample driving and photo-driven technologies, thereby improving the system's engineering practicality and scenario adaptability.

[0096] To achieve differentiated digital human-driven processing, in the teaching broadcast video generation method provided in Embodiment 1 of this application, when the resource type of the digital human material is a preset video type, a preset video template corresponding to the preset video type is obtained; the video length of the preset video template and the audio length of the broadcast audio stream are aligned to obtain a target video template with the same video and audio lengths; a digital human model corresponding to the preset video type is called, and the digital human material and broadcast audio stream are processed using the digital human model to obtain a digital human-driven video stream, wherein the digital human model is a model trained based on historical figure videos; when the resource type of the digital human material is an external video type, the digital human material is processed by image matting to obtain the foreground and background regions of the figure; the foreground region of the figure is driven based on the broadcast audio stream to obtain the foreground video of the figure; the foreground video of the figure and the background region are processed to obtain the digital human-driven video stream; when the resource type of the digital human material is an image type, the figure region is extracted from the digital human material; the figure region is driven based on the broadcast audio stream to obtain the digital human-driven video stream.

[0097] In this embodiment of the invention, when the resource type of the digital human material is determined to be a preset video type, the corresponding preset video template is retrieved from storage, and its video duration is checked against the audio duration of the broadcast audio stream. If the two are inconsistent, the preset video template is looped in either forward or reverse direction to extend its duration, or truncated to shorten its duration, until the video duration and audio duration are perfectly matched, generating a target video template. Then, a digital human model trained based on historical figures' videos is invoked, and a lip-synced digital human-driven video stream is generated by combining the target video template and the broadcast audio stream. Here, the preset video template refers to high-quality, pre-cut, and well-trained virtual lecturer video material pre-set by the system.

[0098] When the resource type is external video, the digital human material resource is first processed by image segmentation to separate the foreground and background regions of the human. Then, the audio stream is used to drive the foreground region of the human to generate a dynamic video containing lip movements and facial expressions, i.e., the foreground video of the human. Finally, the foreground video of the human is replaced or merged with the background region to construct the digital human-driven video stream. The target video template refers to a video segment that strictly corresponds to the audio duration after duration alignment processing. The digital human model refers to a dedicated driving network trained for a specific preset image that can maintain high consistency of the human.

[0099] When the resource type is an image, the subject is extracted from the static digital human material resource to obtain the human area, and the human area is directly driven by the broadcast audio stream to generate a dynamic video containing head movements, blinking and lip movements, which is directly output as a digital human driven video stream.

[0100] Figure 5 This is a schematic diagram of an optional three-track digital human driving layer process according to an embodiment of the present invention, as shown below. Figure 5 As shown, the three-track digital human driving layer is used to select different digital human driving paths based on the type of digital human resource template, generating dynamic digital human video materials that can be overlaid onto a specified area of ​​the teaching page. The three-track digital human driving layer includes: a resource input and modal distribution layer, a general video temporal preprocessing layer, a built-in high-fidelity video driving layer (track 1), an auto-generalization video driving layer (track 2), a photo digital human driving layer (track 3), and a rendering output layer. The resource input and modal distribution layer receives input: digital human resource templates and synthesized audio, and performs template media detection: video or image. If it is a video template, it enters the general video temporal preprocessing layer for audio-video length alignment and template source detection: built-in or custom. If it is a built-in image, it enters the built-in high-fidelity video driving layer (track 1), reads the cached video mask, and inputs it into the high-fidelity digital human model for processing. If it is a custom image, it enters the auto-generalization video driving layer (track 2), performs real-time video matting, and inputs it into the Zero-shot generalization driving model (zero-shot generalization driving model). If it's an image template, it enters the photo-digital human driving layer (track 3) for precise single-image cutout and background blending, and then is input into the photo-digital human model for processing. Finally, the dynamic digital human video footage is output through the rendering output layer.

[0101] For example, when the input is a system-built-in video template, the system enters the first digital human driving path. This path uses a high-fidelity digital human model pre-trained for the built-in video template. Each built-in image corresponds to an independent training model, which is trained based on approximately 5 minutes of high-quality human videos. Therefore, it performs well in terms of lip-sync, facial expression naturalness, and human consistency, and its inference speed is fast. Since the length of the built-in video template is fixed, while the length of the input synthesized audio is not fixed, the system performs audio-video duration alignment before driving: when the input audio duration exceeds the template duration, the template video is cyclically expanded using a combination of forward and reverse playback; when the input audio duration is shorter than the template duration, the template video is truncated to match the template length with the playback audio length. Furthermore, since the built-in templates have all been pre-cut and the video mask cached, this path does not require real-time cutting and can directly call the corresponding high-fidelity digital human model to generate dynamic digital human video materials, thus balancing generation quality and processing efficiency.

[0102] When the input is a user-uploaded video template, the system enters the second digital human driving path. Since user-uploaded videos typically have original backgrounds and cannot be directly overlaid onto the PPT page, the system first performs real-time background removal on the custom video to separate the foreground area of ​​the person. Then, it calls a zero-shot generalization driving model to generate lip movements and facial motions based on the input audio, resulting in a dynamic person video synchronized with the broadcast content. This zero-shot driving model can employ a digital human inference model based on DINet (Deep Inference Network), MuseTalk (a dialogue generation model based on the Muse (multimodal generative model) architecture), Wav2Lip (lip-syncing network), or improved versions thereof, to quickly drive the image of the uploaded video without requiring a dedicated model to be trained for each user. The generated foreground video can then be further pasted onto a specified background area of ​​the PPT page, thus meeting the needs for personalized lecturer image generation.

[0103] When the input is an image template, the system enters the third-track digital human driving path. The image template can be either a system-built-in image or a user-uploaded image. For user-uploaded images, the system first performs background removal to extract the main subject area. Then, the system calls the photo-based digital human model to generate a speaking video clip based on the input audio, including slight head movements, facial expressions, blinking, and synchronized lip movements. Since image-based digital humans only use a single image as input, unlike video-based digital humans which can extract the entire original video frame by frame, a "background removal first, background replacement, and video compositing" approach is adopted. This involves first obtaining the foreground of the subject, then blending it with the corresponding background area in the PPT to create a dynamic digital human video with a transparent foreground effect. Finally, this video is overlaid onto a designated area of ​​the PPT page. This path enables low-cost and rapid digital human presentation even in the absence of a live-action video template.

[0104] Through the aforementioned three-track digital human driving mechanism, differentiated processing flows are designed for built-in high-fidelity video avatars, user-defined video avatars, and image-based digital human avatars. This allows the system to achieve highly natural and efficient digital human broadcasting effects using pre-trained dedicated models, while also supporting the integration of personalized video and image templates in open scenarios. This enhances the adaptability, scalability, and engineering practicality of the digital human generation system. Finally, this layer uniformly outputs dynamic digital human video materials, providing video input for subsequent subtitle overlay, background music mixing, and full-page video rendering.

[0105] In this embodiment, a refined three-track digital human-driven process is implemented. Through time alignment, real-time image matting, and static image dynamization, the high-fidelity efficiency of the built-in image, the flexible adaptation of custom videos, and the low-cost generation of image resources are ensured, thereby improving the quality of digital human generation and the engineering implementation capability of the system in multiple scenarios.

[0106] To improve the accuracy of generating teaching broadcast videos corresponding to target files, the teaching broadcast video generation method provided in Embodiment 1 of this application obtains a subtitle file and, if background music is enabled, obtains a background music audio stream; performs audio ducking mixing on the background music audio stream and all broadcast audio streams to obtain multi-track audio; and renders all single-page images, all digital human-driven video streams, subtitle files, and multi-track audio based on a preset rendering strategy to generate a teaching broadcast video. The preset rendering strategy includes at least: page dwell time and transition animation control, digital human position and size control, occlusion relationship control between digital human and page materials, subtitle safe area layout control, and background layer compositing logic control.

[0107] In this embodiment of the invention, the multi-track fusion rendering layer can uniformly synthesize and render page image materials, dynamic digital human video materials, subtitle timeline data, lecturer broadcast audio stream and background music, and output a complete lecturer broadcast teaching video file.

[0108] Specifically, the subtitle file (such as SRT (SubRipText) format) output by the intelligent audiovisual auxiliary generation layer can be parsed first, and user configuration can be detected. If the background music function is detected to be enabled, the corresponding background music audio stream can be obtained from the music library or the generation module. Then, the audio processing stage is entered, where the background music audio stream and the broadcast audio streams of all pages are input into the audio mixing engine to perform audio ducking mixing processing. That is, when the presence of the broadcast audio stream is detected, the volume of the background music audio stream is automatically reduced, and the volume of the background music is restored during the broadcast intervals, thereby generating multi-track audio with distinct layers and clear hierarchy. Next, the processed multi-track audio, all single-page images output by the front-end module, all corresponding digital human-driven video streams, and the subtitle file are input into the rendering engine. According to the preset rendering strategy, spatiotemporal synchronization and visual compositing are performed to finally generate a complete teaching broadcast video.

[0109] Figure 6 This is a schematic diagram of an optional multi-track fusion rendering layer process according to an embodiment of the present invention, such as... Figure 6 As shown, the inputs received by the multi-track fusion rendering layer are stored in the multi-track material input pool (pre-module output), including: single-page image materials, dynamic digital human video materials, SRT / timeline caption data, lecturer narration audio stream, and background music (BGM). The lecturer narration audio stream and background music (BGM) are input to the audio mixing and ducking processor for processing to obtain the processed multi-track audio. Then, in the core multi-track compositing and rendering engine, based on the injected layout parameters, the single-page image materials, dynamic digital human video materials, SRT / timeline caption data, and multi-track audio are rendered by the final rendering engine to output a complete lecturer narration teaching video file. The injected layout parameters are obtained through a rendering layout control strategy, which includes: preset rendering templates and visual control constraints: page dwell time and transition animation control; digital human position, size, and material occlusion relationship; caption safe area layout; and background layer compositing logic.

[0110] Specifically, the multi-track fusion rendering layer first receives multi-track material input, and then inputs the lecturer's audio and background music into the audio mixing and ducking processing module. The background music is dynamically suppressed to ensure that the lecturer's voice is clearly audible. Subsequently, the system inputs the processed multi-track audio, page images, digital human video, and subtitle data into the final rendering engine, and combines preset rendering templates and visual control constraints to complete unified layout and timing compositing. Among them, the rendering layout control strategy includes at least page dwell time and transition animation control, digital human position and size control, occlusion relationship control between digital human and page materials, subtitle safe area layout control, and background layer compositing logic control. Based on the above constraints, the final rendering engine completes page-by-page content splicing, digital human overlay, subtitle overlay, audio synthesis, and overall timeline synchronization according to the set parameters, and outputs a complete lecturer's teaching video file. Through this layer's processing, the heterogeneous materials generated by the aforementioned modules can be uniformly merged, ensuring the consistency and integrity of the final video in terms of screen layout, audio expression, and subtitle display.

[0111] In this embodiment, an integrated multi-track fusion rendering mechanism was constructed. Through audio dynamic balancing and visual layout control, common problems such as audio interference, chaotic screen layout, and subtitle obstruction in the synthesis of multi-source heterogeneous materials were solved. This enabled efficient and high-quality automated generation from single-page materials to complete teaching videos, improving the standardization and aesthetics of the finished video.

[0112] To improve the accuracy of obtaining background music audio streams, in the teaching broadcast video generation method provided in Embodiment 1 of this application, the broadcast audio stream and the single-page explanatory text are processed by timeline mapping to generate subtitle files; when it is determined that background music is turned on, features are extracted from the preset teaching broadcast video requirements to obtain music features; based on the music features, the background music audio stream is obtained.

[0113] In this embodiment of the invention, the intelligent audiovisual auxiliary generation layer is used to generate background music audio streams and timeline subtitle data, providing auxiliary audiovisual materials for the final rendering of the film.

[0114] The system first receives the audio stream and the corresponding single-page explanatory text. Using forced alignment algorithms (such as CTC (Connectionist Temporal Classification) or Attention mechanisms), it calculates the correspondence between audio frames and text words, generating a timeline mapping containing precise start and end times. Based on this, it generates a subtitle file containing text and timestamps. Next, the system checks the user configuration. If background music is enabled, it extracts features such as scene style, course type, speech rate and rhythm, and emotional tone from the preset teaching video requirements, converting them into vector-like music features. Finally, the system inputs these music features into a background music retrieval or generation module, matching the most similar segments in the music library or synthesizing a background music audio stream that matches the features through a generative model.

[0115] Figure 7 This is a schematic diagram of an optional intelligent audiovisual auxiliary generation layer process according to an embodiment of the present invention, such as... Figure 7 As shown, the intelligent audiovisual auxiliary generation layer includes: audiovisual auxiliary data input, a background music (BGM) intelligent matching module, a subtitle forced alignment generation module, and auxiliary audiovisual resource output. The audiovisual auxiliary data input receives the user's BGM configuration, the text of each page of the lecture script, and the lecturer's audio stream. The background music (BGM) intelligent matching module can detect whether background music is enabled based on the user's BGM configuration. If enabled, it extracts semantic and style parameters based on each page of the lecture script, performing feature extraction: lecture script semantics / scene style / speech speed / course type, thereby retrieving matching BGM from the music library or generating an appropriate BGM segment using AI (Artificial Intelligence). If disabled, the BGM generation process is skipped, and no BGM is output. Simultaneously, the subtitle forced alignment generation module, based on each page of the lecture script and the lecturer's audio stream, calls a forced alignment algorithm to calculate the timeline mapping between audio and text, generating sentence-by-sentence, word-by-word, or character-by-character timestamps. The auxiliary audiovisual resource output is based on the background music (BGM) intelligent matching module, which outputs the BGM snippets and the adapted background music audio stream. Based on the data output by the subtitle forced alignment generation module, it outputs SRT / equivalent timeline subtitle data for accurate subtitle overlay in the final film.

[0116] Specifically, the intelligent audiovisual auxiliary generation layer inputs include at least the user's background music configuration, the text of each page of the lecture script, and the lecturer's audio stream. The background music generation module first detects whether the user has enabled background music. If enabled, it extracts features based on the lecture script's semantics, scene style, speech rate and rhythm, and course type, and retrieves matching background music from the music library, or generates suitable BGM fragments to output the corresponding background music audio stream. If not enabled, the background music generation process is skipped. The generated background music can be combined with the lecture audio for audio ducking mixing during the subsequent rendering stage to avoid interfering with the lecturer's speech. Simultaneously, the subtitle generation module receives the text of each page of the lecture script and the lecturer's audio stream, calls a forced alignment algorithm to calculate the timeline mapping relationship between the audio and text, generates sentence-by-sentence, word-by-word, or character-by-character timestamp data, and outputs SRT or equivalent subtitle files for accurate subtitle overlay in the final video stage. Through this layer, background music and accurate subtitle data can be added without affecting the main lecture content, improving the completeness, viewing experience, and expressive effect of the final teaching video.

[0117] In this embodiment, a fully automated process for subtitle generation and background music matching is realized. The accuracy of the subtitles is ensured by forced alignment, and the music style is intelligently adapted by feature extraction. This improves the professionalism and completeness of the teaching video in terms of audio-visual synchronization and atmosphere creation, and enhances the viewing experience of the final product.

[0118] The following describes in detail another optional implementation method.

[0119] This invention proposes a teaching broadcast video generation method based on multimodal document understanding and digital human collaborative driving, comprising six layers: a document parsing and processing layer, a large-model intelligent routing layer, a sound adaptive synthesis layer, a three-track digital human intelligent driving layer, an intelligent audiovisual auxiliary generation layer, and a multi-track fusion rendering layer. This method is applicable to various document types such as PPT, PDF, and Word, and can automatically complete page parsing, lecture script generation, speech synthesis, digital human driving, subtitle generation, and video rendering, forming a complete generation chain from courseware document input to lecturer broadcast video output. The document parsing and processing layer is responsible for splitting the input document into single-page images and simultaneously extracting the underlying text information of each page; the large-model intelligent routing layer calls the multimodal large model or language large model to generate a suitable script for the current page for oral broadcasting, depending on whether the page contains complex content such as images and charts; the sound adaptive synthesis layer supports two modes: built-in timbre and custom timbre, to realize the generation of lecturer broadcast audio; the three-track digital human intelligent driving layer supports three paths: built-in image and video, user-defined video, and custom image, to generate corresponding dynamic digital human video materials; the intelligent audiovisual auxiliary generation layer can generate or match background music as needed, and output accurate timestamp subtitle files using a forced alignment algorithm; the multi-track fusion rendering layer integrates the page screen, digital human video, broadcast audio, background music, and subtitles to output a complete teaching video.

[0120] Figure 8 This is a schematic diagram of an optional teaching broadcast video generation process based on multimodal document understanding and digital human collaborative driving according to an embodiment of the present invention, such as... Figure 8 As shown, the document parsing and processing layer segments the input PPT / Word / PDF file into single-page image sets and simultaneously extracts the underlying plain text of each page. The large-scale intelligent routing layer detects whether the single-page image set contains images / charts. If so, it combines the underlying plain text of each page and calls the multimodal large-scale model for inference to output the speech text for the current page. If not, it combines the underlying plain text of each page and calls the language large-scale model for polishing to output the speech text for the current page. Simultaneously, the adaptive voice synthesis layer receives the input broadcast reference audio, detects whether it is a built-in timbre. If so, it reads the preset reference text; if not, it uses ASR speech recognition to extract the text and combines it with the speech text for the current page to output the speaker's broadcast audio stream through the voice cloning synthesis model. The three-track intelligent digital human driving layer receives input digital human resource templates and performs template medium detection: video / image. If it's a video template, it performs general video preprocessing: length alignment and loop / truncation. It then detects the template source: built-in / custom. If it's a built-in video, it combines with the lecturer's audio stream and enters Track 1: a dedicated high-fidelity model driver to read the mask directly, outputting dynamic digital human video material. If it's a custom video, it combines with the lecturer's audio stream and enters Track 2: a generalized Zero-shot model driver for real-time image matting, outputting dynamic digital human video material. If it's an image template, it combines with the lecturer's audio stream and enters Track 3: a photo digital human model driving layer for precise image matting and background blending, outputting dynamic digital human video material. The intelligent audiovisual auxiliary generation layer determines whether background music is enabled. If so, it generates matching background music based on the text semantics; otherwise, it skips the background music. Simultaneously, it calls a forced alignment algorithm to process the lecturer's audio stream and the current page's speech text, outputting a precise timestamped subtitle file (SRT). The multi-track fusion rendering layer inputs a single-page image set, dynamic digital human video footage, lecturer broadcast audio stream mixed with audio ducking, and background music into the final rendering engine, and overlays subtitle layers to output a complete lecturer broadcast video.

[0121] In this invention embodiment, a complete processing chain is constructed, encompassing document parsing, script generation, speech synthesis, digital human driving, subtitle generation, and final rendering. This automatically converts PPT, PDF, and Word documents into lecture videos, reducing manual scriptwriting, recording, subtitle addition, and editing, thus improving video production efficiency. Through page content detection and model routing, a large language model is used for plain text pages, while a visual-language multimodal model is used for mixed text and image pages. This enhances the information comprehension capabilities of complex pages, avoiding information omissions or content distortion caused by relying on a single path, thereby improving the quality of script generation. Furthermore, for plain text pages, instead of directly calling the visual-language multimodal model, a large language model is used for text polishing and rewriting, reducing memory usage, computational overhead, and inference latency, avoiding unnecessary resource waste, and improving overall generation efficiency. In addition, it supports two speech synthesis methods: built-in voice and custom voice, and three driving paths: built-in video image, custom video image, and image-based digital human. This adapts to the personalized needs of different users for lecturer voices and images, improving the system's flexibility and applicability. Through processing mechanisms such as lecture context constraints, voice front-end optimization, forced subtitle alignment, intelligent background music matching, and audio ducking mixing, the generated video maintains good consistency between text expression, voice broadcasting, digit lip reading, and subtitle display, thereby improving the viewing experience of the final teaching video.

[0122] The embodiments of this invention have unified the design of multi-source document input, different digital human templates and multi-track material rendering process, which can stably output standardized teaching video results, and have good compatibility, scalability and engineering implementation capabilities, making it suitable for course production, corporate training, knowledge explanation and other scenarios.

[0123] The following is a detailed description with reference to another embodiment.

[0124] Example 2

[0125] The teaching broadcast video generation device provided in this embodiment includes multiple implementation units, each of which corresponds to a specific implementation step in Embodiment 1 above.

[0126] Figure 9 This is a schematic diagram of an optional teaching broadcast video generation device according to an embodiment of the present invention, such as... Figure 9 As shown, the teaching broadcast video generation device may include: a receiving unit 90, a processing unit 91, a generation unit 92, a construction unit 93, and a rendering unit 94.

[0127] The receiving unit 90 is used to receive the input target file, the broadcast tone parameters, and the digital human material resources;

[0128] Processing unit 91 is used to parse and process the target file, split it into multiple sets of single-page structured data, and process each set of single-page structured data to generate multiple single-page explanatory texts. The single-page structured data includes at least: single-page images.

[0129] The generation unit 92 is used to process each single-page explanation script based on the broadcast timbre parameters and generate the corresponding broadcast audio stream for each single-page explanation script.

[0130] Construction unit 93 is used to construct a digital human-driven video stream based on the broadcast audio stream and digital human material resources;

[0131] Rendering unit 94 is used to render all single-page images, all broadcast audio streams, and all digital human-driven video streams to generate the teaching broadcast video corresponding to the target file.

[0132] The aforementioned teaching broadcast video generation device can adopt an integrated collaborative processing flow of document parsing, intelligent routing script generation, adaptive sound synthesis, multi-track digital human driving, and multi-track fusion rendering. By receiving target files, broadcast timbre parameters, and digital human material resources, it parses and breaks down the target file into single-page structured data containing multi-dimensional information such as single-page images. Based on the page content type, it intelligently calls a language big model or a visual-language multimodal big model to generate a single-page explanation script adapted for the broadcast. Combining timbre feature extraction and text front-end optimization, it generates highly natural broadcast audio. Furthermore, it employs a differentiated driving strategy to construct synchronized audio for built-in or custom digital human resources. The digital human video stream, featuring shapes and expressions, ultimately unifies the timing and layout rendering of page images, digital human videos, broadcast audio, background sounds, and subtitles. This achieves the goal of fully automated closed-loop generation from multi-source courseware documents to standardized lecturer broadcast videos. As a result, it significantly reduces the costs of manual scriptwriting, voice-over, recording, and editing, and improves video production efficiency, content generation quality, and system engineering adaptability. This solves the problems in related technologies, such as the long production cycle and high cost caused by reliance on manual processes, as well as the lack of intelligent understanding and routing for complex document pages, the disconnect between timbre and image-driven approaches, and the inconsistency in the fusion of multi-track materials in existing automation solutions.

[0133] Optionally, the processing unit is further configured to: determine the file format of the target file, and if the file format is not a preset format, convert the target file format to obtain a standardized file, wherein the format of the standardized file is a preset format; segment the standardized file to obtain multi-page sub-files, wherein each page sub-file corresponds to page number information; render each page sub-file to obtain multiple single-page images; extract text from each page sub-file to obtain multiple text data; extract the page structure from each page sub-file to obtain multiple page structure data; and for each page sub-file, aggregate the single-page images, text data, and page structure data of the sub-file to obtain the single-page structured data corresponding to the sub-file.

[0134] Optionally, the processing unit is further configured to: perform layout detection on the single-page images in the single-page structured data to obtain detection results; if the detection results indicate that the single-page images include non-text content, process the single-page images and text data in the single-page structured data based on contextual constraints to obtain a single-page explanatory text, wherein the contextual constraints include at least: duration constraints, terminology consistency constraints, style constraints, and coherence constraints; if the detection results indicate that the single-page images do not include non-text content, process the text data based on contextual constraints to obtain a single-page explanatory text.

[0135] Optionally, the generation unit is also used to: process the single-page explanatory text to obtain standardized broadcast text; determine the timbre type of the broadcast timbre parameters; if the timbre type is a preset timbre type, obtain the preset text and preset voiceprint features corresponding to the preset timbre type from the preset timbre library, wherein the preset timbre library includes: multiple preset timbre types and preset text and preset voiceprint features corresponding to each preset timbre type; perform sound synthesis based on the preset text, preset voiceprint features, and standardized broadcast text to generate a broadcast audio stream; if the timbre type is not a preset timbre type, identify the broadcast timbre parameters to obtain reference text; perform sound synthesis based on the reference text, broadcast timbre parameters, and standardized broadcast text to generate a broadcast audio stream.

[0136] Optionally, the generation unit is also used to: pause the single-page explanatory text to obtain a first initial broadcast text; standardize the first initial broadcast text to obtain a second initial broadcast text; perform speech processing on the second initial broadcast text to obtain a third initial broadcast text; and perform audio continuity processing on the third initial broadcast text to obtain a standardized broadcast text.

[0137] Optionally, the construction unit is also used to: determine the resource type of the digital human material resources, wherein the resource types include: preset video type, external video type, and image type; and use a driving strategy based on the resource type to process the broadcast audio stream and the digital human material resources to construct a digital human-driven video stream.

[0138] Optionally, the construction unit is further configured to: when the resource type of the digital human material resource is a preset video type, obtain a preset video template corresponding to the preset video type; align the video length of the preset video template with the audio length of the broadcast audio stream to obtain a target video template with the same video length and audio length; call the digital human model corresponding to the preset video type, and use the digital human model to process the digital human material resource and the broadcast audio stream to obtain a digital human-driven video stream, wherein the digital human model is a model trained based on historical human videos; when the resource type of the digital human material resource is an external video type, perform image matting on the digital human material resource to obtain the foreground and background regions of the person; drive the foreground region of the person based on the broadcast audio stream to obtain the foreground video of the person; process the foreground video of the person and the background region to obtain a digital human-driven video stream; when the resource type of the digital human material resource is an image type, extract the person region from the digital human material resource; drive the person region based on the broadcast audio stream to obtain a digital human-driven video stream.

[0139] Optionally, the rendering unit is also used to: acquire subtitle files, and, if background music is enabled, acquire the background music audio stream; perform audio ducking mixing on the background music audio stream and all broadcast audio streams to obtain multi-track audio; and, based on a preset rendering strategy, render all single-page images, all digital human-driven video streams, subtitle files, and multi-track audio to generate a teaching broadcast video. The preset rendering strategy includes at least: page dwell time and transition animation control, digital human position and size control, occlusion relationship control between digital human and page materials control, subtitle safe area layout control, and background layer compositing logic control.

[0140] Optionally, the rendering unit is also used to: perform timeline mapping processing on the broadcast audio stream and the single-page explanatory text to generate subtitle files; extract features from the preset teaching broadcast video requirements to obtain music features when background music is enabled; and obtain the background music audio stream based on the music features.

[0141] The aforementioned teaching broadcast video generation device may also include a processor and a memory. The aforementioned receiving unit 90, processing unit 91, generation unit 92, construction unit 93, rendering unit 94, etc., are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0142] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and by adjusting kernel parameters, all single-page images, all audio streams, and all digital human-driven video streams can be rendered to generate the corresponding instructional video for the target file.

[0143] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0144] This invention also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: receiving an input target file, broadcast tone parameters, and digital human material resources; parsing and processing the target file to obtain multiple sets of single-page structured data; processing each set of single-page structured data to generate multiple single-page explanatory texts; for each single-page explanatory text, processing it based on the broadcast tone parameters to generate a broadcast audio stream corresponding to the single-page explanatory text; constructing a digital human-driven video stream based on the broadcast audio stream and digital human material resources; rendering all single-page images, all broadcast audio streams, and all digital human-driven video streams to generate a teaching broadcast video corresponding to the target file.

[0145] According to another aspect of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the teaching broadcast video generation method described above.

[0146] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the above-described teaching broadcast video generation method.

[0147] Figure 10 This is a hardware structure block diagram of an electronic device (or mobile device) for generating teaching broadcast videos according to an embodiment of the present invention. Figure 10 As shown, an electronic device may include one or more processors (e.g., Figure 10 The processors 1002a, 1002b, ..., 1002n, etc., may include, but are not limited to, processing devices such as microprocessors (MCUs) or programmable logic devices (FPGAs), and a memory 1004 for storing data. In addition, it may include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 10 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device may also include components that are more... Figure 10 The more or fewer components shown, or having the same Figure 10 The different configurations shown.

[0148] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0149] The embodiments or examples disclosed herein are not exhaustive, but merely illustrative of some embodiments or examples, and are not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment or example can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment or example can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment or example can be arbitrarily interchanged. Furthermore, optional methods or examples in a particular embodiment or example can be arbitrarily combined; moreover, embodiments or examples can be arbitrarily combined. For example, some or all steps of different embodiments or examples can be arbitrarily combined, and a particular embodiment or example can be arbitrarily combined with optional methods or examples of other embodiments or examples.

[0150] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0151] In the several embodiments provided by this invention, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection can be through some interfaces; the indirect coupling or communication connection of units or modules can be electrical or other forms.

[0152] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0153] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0154] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0155] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.< / break>

Claims

1. A method for generating teaching broadcast videos, characterized in that, include: Receives input target files, broadcast timbre parameters, and digital human material resources; The target file is parsed and split into multiple sets of single-page structured data. Each set of single-page structured data is then processed to generate multiple single-page explanatory texts. The single-page structured data includes at least one single-page image. For each of the single-page explanatory texts, the single-page explanatory texts are processed based on the broadcast timbre parameters to generate the corresponding broadcast audio stream; Based on the broadcast audio stream and the digital human material resources, a digital human-driven video stream is constructed; Render all the single-page images, all the broadcast audio streams, and all the digital human-driven video streams to generate the teaching broadcast video corresponding to the target file.

2. The teaching broadcast video generation method according to claim 1, characterized in that, The steps of parsing the target file and splitting it into multiple sets of single-page structured data include: The file format of the target file is determined, and if the file format is not a preset format, the format of the target file is converted to obtain a standardized file, wherein the format of the standardized file is the preset format; The standardized document is divided into multiple sub-files, each of which has a corresponding page number. Render each of the sub-files on each page to obtain multiple single-page images; Text extraction is performed on the sub-files on each page to obtain multiple text data; The page structure of each sub-file is extracted to obtain multiple page structure data. For each sub-file, the single-page image, text data, and page structure data of the sub-file are aggregated to obtain the single-page structured data corresponding to the sub-file.

3. The teaching broadcast video generation method according to claim 1, characterized in that, The steps for processing each set of single-page structured data to generate multiple single-page explanatory texts include: Layout detection is performed on the single-page image in the single-page structured data to obtain the detection result; When the detection result indicates that the single-page image includes non-text content, the single-page image and the text data in the single-page structured data are processed based on contextual constraints to obtain the single-page explanatory text. The contextual constraints include at least: duration constraints, terminology consistency constraints, style constraints, and coherence constraints. If the detection result indicates that the single-page image does not include non-text content, the text data is processed based on the contextual constraints to obtain the single-page explanatory text.

4. The teaching broadcast video generation method according to claim 1, characterized in that, The steps of processing the single-page explanation text based on the broadcast timbre parameters to generate the corresponding broadcast audio stream include: The single-page explanation text is processed to obtain a standardized broadcast text; Determine the timbre type of the broadcast timbre parameters; When the timbre type is a preset timbre type, the preset text and preset voiceprint features corresponding to the preset timbre type are obtained from the preset timbre library. The preset timbre library includes: multiple preset timbre types and the preset text and preset voiceprint features corresponding to each preset timbre type. Based on the preset text, the preset voiceprint features, and the standardized broadcast text, sound synthesis is performed to generate the broadcast audio stream; If the timbre type is not the preset timbre type, the broadcast timbre parameters are identified to obtain reference text; The broadcast audio stream is generated by synthesizing sound based on the reference text, the broadcast timbre parameters, and the standardized broadcast text.

5. The teaching broadcast video generation method according to claim 4, characterized in that, The steps for processing the single-page presentation text to obtain standardized broadcast text include: Pauses are applied to the single-page explanatory text to obtain the first initial broadcast text; The first initial broadcast text is normalized to obtain the second initial broadcast text; The second initial broadcast text is processed by speech to obtain the third initial broadcast text; The third initial broadcast text is subjected to audio continuity processing to obtain the standardized broadcast text.

6. The teaching broadcast video generation method according to claim 1, characterized in that, The steps for constructing a digital human-driven video stream based on the broadcast audio stream and the digital human material resources include: The resource type of the digital human material resources is determined, wherein the resource type includes: preset video type, external video type, and image type; The broadcast audio stream and the digital human material resources are processed using a driving strategy based on the resource type to construct the digital human-driven video stream.

7. The teaching broadcast video generation method according to claim 6, characterized in that, The steps of processing the broadcast audio stream and the digital human material resources using a driving strategy based on the resource type to construct the digital human-driven video stream include: If the resource type of the digital human material resource is the preset video type, obtain the preset video template corresponding to the preset video type; Align the video length of the preset video template with the audio length of the broadcast audio stream to obtain a target video template with the same video length and audio length. The digital human model corresponding to the preset video type is invoked, and the digital human material resources and the broadcast audio stream are processed using the digital human model to obtain the digital human-driven video stream. The digital human model is a model trained based on historical figures' videos. When the resource type of the digital human material is the external video type, the digital human material is subjected to image matting to obtain the foreground area and background area of ​​the person; Based on the broadcast audio stream, the foreground region of the person is driven to obtain the foreground video of the person; The foreground video of the person and the background area are processed to obtain the digital human-driven video stream; When the resource type of the digital human material resource is the image type, extract the human figure region from the digital human material resource; Based on the broadcast audio stream, the character area is driven to obtain the digital human driven video stream.

8. The teaching broadcast video generation method according to claim 1, characterized in that, The steps of rendering all the single-page images, all the broadcast audio streams, and all the digital human-driven video streams to generate the teaching broadcast video corresponding to the target file include: Obtain the subtitle file, and if background music is enabled, obtain the background music audio stream; The background music audio stream and all the broadcast audio streams are subjected to audio ducking mixing to obtain multi-track audio. Based on a preset rendering strategy, all the single-page images, all the digital human-driven video streams, the subtitle files, and the multi-track audio are rendered to generate the teaching broadcast video. The preset rendering strategy includes at least: page dwell time and transition animation control, digital human position and size control, occlusion relationship control between digital human and page materials control, subtitle safe area layout control, and background layer compositing logic control.

9. The teaching broadcast video generation method according to claim 8, characterized in that, The steps for obtaining the subtitle file and, assuming background music is enabled, obtaining the background music audio stream include: The audio stream and the single-page explanation text are time-axis mapped to generate the subtitle file. With background music enabled, feature extraction is performed on the preset teaching broadcast video requirements to obtain music features; Based on the music characteristics, the background music audio stream is obtained.

10. A teaching broadcast video generation device, characterized in that, include: The receiving unit is used to receive the input target file, the broadcast tone parameters, and the digital human material resources; The processing unit is used to parse the target file, split it into multiple sets of single-page structured data, and process each set of single-page structured data to generate multiple single-page explanatory texts, wherein the single-page structured data includes at least: single-page images; The generation unit is used to process each single-page explanation script based on the broadcast timbre parameters to generate a broadcast audio stream corresponding to the single-page explanation script. The construction unit is used to construct a digital human-driven video stream based on the broadcast audio stream and the digital human material resources; The rendering unit is used to render all the single-page images, all the broadcast audio streams, and all the digital human-driven video streams to generate the teaching broadcast video corresponding to the target file.

Images