Multi-modal content generation method and device, electronic equipment and storage medium

Abstract

The invention provides a multi-modal content generation method and device, electronic equipment and a storage medium. The method comprises the steps of receiving a document uploaded by a user and performing content extraction; performing block analysis and semantic recombination on the extracted content to generate a PPT text outline and a corresponding explanation script; according to the PPT text outline, matching or dynamically generating a visual material to obtain a first edition PPT; the initial edition PPT is subjected to iterative optimization, the optimized PPT and the explanation script are synthesized to obtain an explanation video, complex structured documents or unstructured content can be processed, it is ensured that the generated outline is clear in organization and accurate in content, and multi-modal content meeting visual aesthetics is output through iterative optimization.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal content generation method, device, electronic device, and storage medium. Background Art

[0002] People face a massive demand for information in their daily lives, work, and studies. Efficiently transforming complex text into clear, intuitive, and easy-to-understand multimodal presentations has become a pressing challenge. Traditional content generation methods often rely on manual labor, requiring significant time and effort. This is particularly true for information organization, outline construction, PowerPoint presentations, and video recording. This inefficient approach no longer meets the demands for rapid response and immediate dissemination. The need for efficient multimodal information generation is particularly acute in fields such as education, corporate training, marketing, and media content creation. For example, the online education industry needs to quickly transform textbook content into slides and video courses; businesses need to transform internal reports and product documentation into easily presented, illustrated content; and self-media content creators need to produce high-quality presentations and explanations with minimal time and effort. Related technologies use large language models to understand the content and generate outlines for articles, but these methods typically rely on single-shot reasoning and lack in-depth logical reasoning and semantic analysis. This one-shot reasoning often leads to "hallucinations," resulting in inaccurate outlines. This is particularly true for academic papers and technical documents with complex structures or strong logical relationships, making it difficult to ensure the accuracy and depth of the content. In addition, the existing PPT generation method relies on one-time generation of preset templates. If there are problems such as irrelevant images and texts or inconsistent layouts, they cannot be adjusted, and cannot meet users' high requirements for visual effects and content presentation. Summary of the Invention

[0003] The present invention provides a multimodal content generation method, device, electronic device and storage medium to address the defects of traditional content generation methods, such as the lack of in-depth logical reasoning and semantic analysis, the difficulty in ensuring the accuracy and depth of the content, and the inability to meet users' high requirements for visual effects and content presentation.

[0004] The present invention provides a multimodal content generation method, comprising: Receive documents uploaded by users and extract content; The extracted content is parsed and semantically reorganized to generate a PPT text outline and corresponding explanation script; Match or dynamically generate visual materials according to the PPT text outline to obtain a preliminary version of the PPT; The initial version of the PPT is iteratively optimized, and the optimized PPT is synthesized with the explanation script to obtain an explanation video.

[0005] According to the multimodal content generation method provided by the present invention, the extracted content is parsed and semantically reorganized to generate a PPT text outline, including: The extracted content is logically split into multiple content blocks according to the document and chapters; for each content block, it is asynchronously interacted with the large language model, extracting key information of the chapter based on the set prompt words, and returning the parsing results in a specified format; The parsing results are summarized, and semantic integration is performed on the summarized parsing results through a large language model to generate a logically coherent PPT text outline.

[0006] According to the multimodal content generation method provided by the present invention, the matching or dynamic generation of visual materials according to the PPT text outline to obtain the initial version of the PPT includes: Calculating the text similarity between the text content in the PPT text outline and the titles of the visual materials in the parsed chart library page by page, and if the calculated similarity is greater than a preset threshold, matching the corresponding visual materials in the chart library to the corresponding PPT pages; If all the calculated similarities are not greater than a preset threshold, an image generation model based on a diffusion model is called to generate a new chart related to the text content, and the new chart is inserted into the corresponding PPT page.

[0007] According to the multimodal content generation method provided by the present invention, the iterative optimization of the initial version of the PPT includes: A two-stage framework of pre-training evaluation and strategy collaborative optimization is constructed, and the initial version of the PPT is iteratively optimized through the two-stage framework, specifically including: In the first stage, a pre-trained PPT quality assessment model is used to perform fine-grained quality scoring on the PPT pages. The quality assessment model uses a multimodal joint encoder to jointly represent the visual and semantic features of page elements and output a hierarchical reward signal. In the second stage, the initial version of the PPT is iterated through the hierarchical reward signal and a discrete-continuous hybrid action strategy. The discrete-continuous hybrid action strategy includes a discrete action strategy network and a continuous parameter decision network, wherein: the discrete action strategy network is used to select high-order operation types from a preset templated adjustment action set; the continuous parameter decision network is used to dynamically generate continuous adjustment parameters of elements.

[0008] According to the multimodal content generation method provided by the present invention, the pre-training method of the PPT quality assessment model includes: We collected professionally designed PPT pages as positive samples and generated negative samples by randomly disrupting element layouts and destroying color schemes. We annotated each page with a visual aesthetic score and information density index. A parallel coding architecture is used to extract multimodal features from samples in positive and negative samples. The parallel coding architecture includes a visual encoder and a semantic encoder. The visual encoder uses an object detection network to extract the position and size of page elements, and then uses spatial pyramid pooling to obtain layout features based on the position and size of page elements. The semantic encoder is used to perform BERT encoding on text content to obtain text features, and the CLIP model is used to extract image-text correlation features from image text content. By fusing multimodal features through the attention mechanism and combining the visual aesthetic score and information density index of each page annotation, the PPT quality assessment model is trained with page quality regression prediction as the main task goal. The matching degree between PPT layout and graphic content is used as the auxiliary task goal, and a multi-dimensional quantitative score of the matching degree between PPT layout and graphic content is trained.

[0009] According to the multimodal content generation method provided by the present invention, the iterating of the initial version of the PPT through the layered reward signal and the discrete-continuous hybrid action strategy includes: Constructing a hybrid action space, the hybrid action space including a discrete action set and a continuous action set, the discrete action set including multiple fixed high-order operation types, and the continuous action set including a multi-dimensional adjustment continuity parameter for each high-order operation type; At each time step, the discrete action set outputs a discrete action type. If the action involves parameter adjustment, it triggers the continuous action set to generate specific indicator parameters to form a complete action pair. Design global rewards and local rewards. The global reward is the page score difference output by the quality assessment model, and the local rewards are the operation feasibility verification and historical action consistency rewards. The global reward is decomposed into each action step through linear weighting; Fixed quality assessment model parameters and reinforcement learning pre-training: discrete action strategies use prioritized experience replay to constrain the action selection space in the initial stage, while continuous action strategies use random noise exploration and adaptive training in a small parameter space. After the reinforcement learning pre-training is completed, multi-granularity optimization is performed on the initial version of the PPT, including: locating low-scoring pages through a quality assessment model, prioritizing pages with scores less than a predetermined threshold after sorting them from low to high; iteratively adjusting the target page to be optimized, rendering the adjusted PPT page after each step of the operation, and ending the iteration when the preset number of iterations or the scoring threshold is reached.

[0010] According to the multimodal content generation method provided by the present invention, synthesizing the optimized PPT with the explanation script to obtain the explanation video includes: Send the iteratively optimized PPT to the client, receive feedback from the client, and update the PPT content and explanation script based on the feedback; Converting the updated PPT content into video image frames, and generating corresponding audio from the updated presentation script using a text-to-speech engine; The audio and video image frames are combined to generate a complete video.

[0011] The present invention also provides a multimodal content generation device, comprising: The extraction module is used to receive documents uploaded by users and extract content; The first generation module is used to parse and semantically reorganize the extracted content into blocks to generate a PPT text outline and corresponding explanation script; The second generation module matches or dynamically generates visual materials according to the PPT text outline to obtain a preliminary version of the PPT; The optimization module is used to iteratively optimize the initial version of the PPT and synthesize the optimized PPT with the explanation script to obtain an explanation video.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multimodal content generation method as described in any one of the above items is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the multimodal content generation methods described above.

[0014] The multimodal content generation method, device, electronic device and storage medium provided by the present invention receive documents uploaded by users and extract content; perform block parsing and semantic reorganization on the extracted content to generate a PPT text outline and a corresponding explanation script; match or dynamically generate visual materials based on the PPT text outline to obtain a preliminary version of the PPT; iteratively optimize the preliminary version of the PPT, and synthesize the optimized PPT with the explanation script to obtain an explanation video. The method, device and storage medium can process complex structured documents or unstructured content, ensure that the generated outline is clear and the content is accurate, and output multimodal content that conforms to visual aesthetics through iterative optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 This is one of the flowcharts of the multimodal content generation method provided by an embodiment of the present invention; Figure 2 Schematic diagram of iterative optimization of an initial PPT based on reinforcement learning provided by an embodiment of the present invention; Figure 3 This is a second flow chart of the multimodal content generation method provided by an embodiment of the present invention; Figure 4 1 is a functional structure diagram of a multimodal content generation device provided by an embodiment of the present invention; Figure 5 It is a functional structure diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0018] Figure 1 A flowchart of a multimodal content generation method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the multimodal content generation method provided by the embodiment of the present invention includes: Step 101: Receive the document uploaded by the user and extract the content; Step 102: Parse the extracted content in blocks and reorganize its semantics to generate a PPT text outline and corresponding explanation script; Step 103: Match or dynamically generate visual materials according to the PPT text outline to obtain a preliminary version of the PPT; Step 104: iteratively optimize the initial version of the PPT, and synthesize the optimized PPT with the explanation script to obtain an explanation video.

[0019] Traditionally, large language models are used to understand the content and generate outlines for articles, but these methods typically rely on single-shot reasoning, lacking in-depth logical reasoning and semantic analysis. This one-shot reasoning often leads to "hallucinations," and the generated outlines can be inaccurate. This is especially true for academic papers and technical documents with complex structures or strong logical relationships, making it difficult to ensure the accuracy and depth of the content. Furthermore, existing PPT generation methods rely on pre-set templates for one-time generation. Issues such as inconsistent graphics and text or inconsistent layouts cannot be adjusted, failing to meet users' high expectations for visual quality and content presentation.

[0020] The multimodal content generation method provided by an embodiment of the present invention receives documents uploaded by users and extracts content; performs block parsing and semantic reorganization on the extracted content to generate a PPT text outline and a corresponding explanation script; matches or dynamically generates visual materials based on the PPT text outline to obtain a preliminary version of the PPT; iteratively optimizes the preliminary version of the PPT, and synthesizes the optimized PPT with the explanation script to obtain an explanation video. The method is capable of processing complex structured documents or unstructured content, ensuring that the generated outline is clear and the content is accurate, and outputs multimodal content that conforms to visual aesthetics through iterative optimization.

[0021] Based on any of the above embodiments, the step of parsing the extracted content in blocks and semantically reorganizing the extracted content to generate a PPT text outline includes: Step 201: The extracted content is logically split into multiple content blocks according to the document and chapters; each content block is asynchronously interacted with the large language model concurrently, key information of the chapter is extracted based on the set prompt words, and the parsing results are returned in a specified format; It should be noted that the present invention does not limit the large language model, and those skilled in the art can select a large language model with semantic analysis, reasoning and interaction capabilities as needed.

[0022] Step 202: Summarize the parsing results, and semantically integrate the summarized parsing results through a large language model to generate a logically coherent PPT text outline.

[0023] Existing technologies extract article text through preset copywriting or summary extraction models, but do not mention deep semantic analysis and logical reasoning. They are unable to effectively process complex structured documents or content with complex logical relationships. They perform poorly in the field of academic paper understanding and analysis, and it is difficult to accurately generate high-quality explanatory manuscripts and videos for academic papers or technical documents with complex logical relationships.

[0024] This embodiment of the present invention utilizes the latest Large Language Model (LLM) for deep semantic parsing and logical reorganization, enabling a more intelligent understanding of document content and logical relationships, going beyond extracting summaries or structured information. It supports logical reasoning and content reconstruction for complex documents, processing unstructured content (such as free text) and generating in-depth PPT content outlines and presentation copy, adapting to complex semantics, unstructured content, and cross-disciplinary academic documents. The LLM's powerful semantic understanding and reasoning capabilities ensure the coherence and accuracy of presentations.

[0025] Based on any of the above embodiments, matching or dynamically generating visual materials according to the PPT text outline to obtain a preliminary version of the PPT includes: Step 301: Calculate the text similarity between the text content in the PPT text outline and the titles of the visual materials in the parsed chart library page by page. If the calculated similarity is greater than a preset threshold, match the corresponding visual materials in the chart library to the corresponding PPT pages. Step 302: If all calculated similarities are not greater than a preset threshold, an image generation model based on a diffusion model is called to generate a new chart related to the text content, and the new chart is inserted into the corresponding PPT page.

[0026] Existing technologies use large language models to understand the content and generate outlines for articles, but they usually rely on single-shot reasoning and lack in-depth logical reasoning and semantic analysis. This one-time reasoning often leads to the phenomenon of "hallucination", and the generated outline may be inaccurate, especially when dealing with academic papers and technical documents with complex structures or strong logical relationships. It is difficult to ensure the accuracy and depth of the content. In addition, existing methods use a template library that has been created in advance to match the background images and layout of the article category. They rely too much on preset templates and lack dynamic adaptation capabilities, which can easily lead to the lack of personalization and flexibility in the generated PPT. The richness and scalability of the template library may become a bottleneck, especially when dealing with non-mainstream academic fields or complex content.

[0027] In an embodiment of the present invention, when parsing user documents, the multimodal images and table contents in the original text are extracted and retained. When making a PPT, similarity calculation is performed based on the text content in the outline and the pictures and tables in the document, and the visual content that best matches the semantics of the text content is retrieved and matched. When the visual content of the original text is insufficient, the latest diffusion model-based image generation technology is integrated to dynamically generate illustrations or background images based on the text content of the explanation, rather than relying solely on the template library or inherent images in the document, to ensure a high degree of consistency between the visual content and the theme, and solve the problem of monotonous content caused by missing pictures in the previous method, so as to better meet the high requirements of academic conferences, paper defenses and other scenarios for content consistency and academic rigor.

[0028] Based on any of the above embodiments, the iterative optimization of the initial version of the PPT includes: Step 401: Construct a two-stage framework of pre-training evaluation and strategy collaborative optimization, and iteratively optimize the initial version of the PPT through the two-stage framework, such as Figure 2 As shown, specifically including: Step 402: In the first stage, a pre-trained PPT quality assessment model is used to perform a fine-grained quality score on the PPT page. The quality assessment model uses a multimodal joint encoder to jointly represent the visual and semantic features of the page elements and output a hierarchical reward signal. Step 403. In the second stage, the initial version of the PPT is iterated through the hierarchical reward signal and the discrete-continuous hybrid action strategy, wherein the discrete-continuous hybrid action strategy includes a discrete action strategy network and a continuous parameter decision network, wherein: the discrete action strategy network is used to select a high-order operation type from a preset templated adjustment action set; the continuous parameter decision network is used to dynamically generate continuous adjustment parameters of elements.

[0029] The existing PPT generation method relies on one-time generation of preset templates. This single generation process will result in the PPT layout being less aesthetically pleasing, and is prone to problems such as irrelevant images and text or inconsistent layout, which cannot meet users' high requirements for visual effects and content presentation.

[0030] The present invention utilizes a hybrid strategy deep reinforcement learning-based intelligent PPT optimization method. By constructing a two-stage framework of "pre-training evaluation and strategy collaborative optimization," this method automatically improves the visual style and information density of PPTs. By designing a phased, collaborative deep reinforcement learning architecture, the first stage uses a static, pre-trained quality assessment model to achieve fine-grained page scoring. In the second stage, a strategy collaborative optimization algorithm in a discrete-continuous hybrid action space is used to achieve multi-dimensional layout adjustment. This effectively addresses the limited action space and rigid parameter adjustment issues inherent in traditional template-based tools for dynamic layout optimization.

[0031] Based on any of the above embodiments, the pre-training method of the PPT quality assessment model includes: Step 501: Collect professionally designed PPT pages as positive samples, and generate negative samples by randomly disrupting element layouts and destroying color schemes. Each page is annotated with a visual aesthetic score and information density index. Step 502: Perform multimodal feature extraction on samples from the positive and negative samples using a parallel coding architecture. The parallel coding architecture includes a visual encoder and a semantic encoder. The visual encoder uses an object detection network to extract page element positions and sizes, and obtains layout features based on the page element positions and sizes through spatial pyramid pooling. The semantic encoder performs BERT encoding on the text content to obtain text features, and uses the CLIP model to extract image-text correlation features from the image text content. Step 503: By fusing multimodal features through the attention mechanism and combining the visual aesthetic score and information density index of each page annotation, the PPT quality assessment model is trained with page quality regression prediction as the main task goal. The matching degree between PPT layout and graphic content is used as the auxiliary task goal, and a multi-dimensional quantitative score of the matching degree between PPT layout and graphic content is trained.

[0032] In an embodiment of the present invention, a PPT quality assessment model is based on deep feature fusion and adopts a multimodal joint encoder to jointly represent the visual features (layout rationality, color coordination, information density) and semantic features (content coherence, image-text relevance) of page elements (text, images, graphics). A page quality score regression model is established through pre-training to provide a differentiable reward signal for reinforcement learning.

[0033] Based on any of the above embodiments, the iterating of the initial version of the PPT using the hierarchical reward signal and the discrete-continuous hybrid action strategy includes: Step 601: construct a hybrid action space, wherein the hybrid action space includes a discrete action set and a continuous action set, wherein the discrete action set includes multiple fixed high-order operation types, and the continuous action set includes a multi-dimensional adjustment continuity parameter for each high-order operation type; Step 602: At each time step, the discrete action set outputs a discrete action type. If the action involves parameter adjustment, the continuous action set is triggered to generate specific indicator parameters to form a complete action pair. Step 603: Design global rewards and local rewards. The global reward is the page score difference output by the quality assessment model, and the local rewards are the operation feasibility verification and historical action consistency rewards. The global reward is decomposed into each action step through linear weighting. Step 604: Freeze the quality assessment model parameters and perform reinforcement learning pre-training: the discrete action strategy uses prioritized experience replay to constrain the action selection space in the initial stage, and the continuous action strategy uses random noise exploration to perform adaptive training in a small parameter space. Step 605: After the reinforcement learning pre-training is completed, multi-granularity optimization is performed on the initial version of the PPT, including: locating low-scoring pages through a quality assessment model, sorting them from low to high, and prioritizing pages with scores less than a predetermined threshold; iteratively adjusting the target page to be optimized, rendering the adjusted PPT page after each step, and ending the iteration when a preset number of iterations or a scoring threshold is reached.

[0034] In an embodiment of the present invention, a hybrid action space policy optimization framework is constructed, combining a deep Q-network (DQN) and a deterministic policy gradient (DDPG) algorithm in a cascaded fashion, known as the P-DQN algorithm framework. DQN, as a discrete action policy network, selects the optimal action type from a preset set of templated adjustment actions (including high-level operations such as element alignment, adding or removing components, and layout switching). DDPG, as a continuous parameter decision network, dynamically generates continuous adjustment parameters such as element size, position coordinates, and color values. The two achieve joint policy optimization through a sequential action coupling mechanism. This architecture utilizes a hierarchical reward distribution mechanism to decompose the global page score of the quality assessment model into immediate rewards associated with specific actions. This drives the intelligent agent to achieve a synergistic optimal balance between action selection and parameter adjustment, resulting in visually aesthetically pleasing presentations.

[0035] Based on any of the above embodiments, synthesizing the optimized PPT with the explanation script to obtain the explanation video includes: Step 701: Send the iteratively optimized PPT to the client, receive feedback from the client, and update the PPT content and explanation script based on the feedback; Step 702: Convert the updated PPT content into video image frames, and use a text-to-speech engine to generate corresponding audio from the updated explanation script; Step 703: Combine the audio and video image frames to generate a complete video.

[0036] Existing systems rely on fixed logic and templates, making them inflexible to user needs, such as adjusting presentation style, timbre, or visualization effects. This limits the user experience and makes personalized, customized presentations difficult. Traditional PPT generation methods lack flexible adjustments, preventing users from modifying content or layout to suit their needs, which in turn affects the final video presentation. This lack of intervention means the generated video may not perfectly meet the user's specific requirements, reducing both the user experience and the quality of the generated content.

[0037] The embodiment of the present invention provides a complete integrated process from user document extraction, outline and explanation text generation, PPT production, explanation audio synthesis to explanation video generation, which significantly reduces the complexity of manual operation. In each link, high-degree-of-freedom adjustment is supported, such as tone selection, template style customization, etc., which is more in line with the diverse personalized needs of users. After generating the PPT and explanation script, the user can choose to download and manually adjust and modify it to achieve higher content personalization and accuracy. After the user adjusts, the content can be re-uploaded to continue the subsequent video synthesis process, forming a two-way interaction of "user participation-system optimization". Users can choose to directly generate videos and enjoy an efficient automated experience, or they can fine-tune the content as needed to ensure that the generated results are more in line with expectations. The efficiency and flexibility of the present invention are improved by combining automation with user participation.

[0038] Furthermore, the text-to-speech engine utilizes TTS (Text To Speech) technology, enabling the system to generate audio for paper explanations that accurately match the presentation content, resulting in high-quality video explanations. TTS technology supports a variety of voice options, including male, female, child, and custom voices, allowing the generated explanation videos to be personalized based on user preferences. The system also supports multilingual speech generation, adapting to the needs of diverse language environments and making it particularly suitable for international or multilingual scenarios.

[0039] Based on any of the above embodiments, Figure 3 As shown, taking the generation of explanation videos of academic papers as an example, the multimodal content generation method specifically includes the following steps: Step 1: Document upload The user uploads the document to be parsed. The document can be an electronic document, such as a PDF document, a Word document, or a web page.

[0040] Step 2: The content extraction system uses the content extractor component to parse the document, extracting all content and structuring it into Markdown format. This includes extracting chapter titles, body text, and original graphics, while ensuring the order and consistency of the content. The extracted graphics are stored on disk, and the extracted text is passed to the subsequent processing module.

[0041] Step 3: Generate the first version of the PPT text outline and explanation copy. Based on the document content extracted in step 2, generate the first version of the PPT text outline and explanation copy. This step includes two sub-steps: block analysis and integration and reorganization: (1) Chunk parsing: The document content is logically divided into multiple content chunks (Chunks) according to the chapter. For each content chunk, the system asynchronously interacts with the large language model (LLM) to extract key information such as the main content of the chapter, experimental modules, theoretical methods, summary, etc. based on the set prompt words, and returns the processing results in the specified format.

[0042] (2) Integration and Reorganization: All parsing results are summarized and further integrated and polished through the large language model (LLM) to generate a preliminary PPT text outline and page-by-page matching explanation script. The integration process is based on preset semantic rules and document category features to ensure that the outline is clear and the content is accurate.

[0043] Step 4: Generate the first version of the PPT. Based on the first version of the PPT text outline obtained in step 3 and the chart materials extracted in step 2, this step uses an automated method to generate the first version of the PPT. In this step, users can freely select the PPT template style they want to use. The system generates the PPT page based on the predefined template page and PPT text outline content. To improve the presentation effect, the following two sub-steps are used to introduce visual chart materials: (1) The system will calculate the text similarity between the text content in the PPT outline and the title of the visual material in the chart library page by page, so as to accurately match the visual material in the chart library to the corresponding PPT page (2) If the PPT content page does not match the appropriate chart visual material, the image generation model based on the diffusion model is called to generate a new chart related to the text content.

[0044] (3) Finally, the system will insert the PPT text outline content and the matching / generated charts into the PPT to generate the first version of the PPT.

[0045] Step 5: Reasoning iteration and optimization Based on the initial version of the PPT generated in step 4, the system enters the reasoning and optimization stage. This step realizes the automatic improvement of the visual style and information density of the PPT by constructing a two-stage framework of "pre-training evaluation-strategy collaborative optimization". First, a pre-trained PPT quality assessment model is deployed to perform multi-dimensional quantitative scoring of the single-page layout structure, visual element organization, and image-text content matching to provide a differentiable reward signal for reinforcement learning; on this basis, a reinforcement learning optimizer for the hybrid action space is constructed, and the P-DQN (Parameterized Deep Q-Network) framework is used to achieve automatic style tuning. Specifically: (1) Quality assessment model pre-training stage Data Construction: We collected a large number of professionally designed PPT pages as positive samples, and generated negative samples by randomly disrupting element layouts and disrupting color schemes. Each page was annotated with a visual aesthetic score (annotated by a professional team) and an information density index (calculated based on factors such as the text word count / image area ratio).

[0046] Multimodal feature extraction: using parallel encoding architecture: Visual encoder: Uses the object detection network to extract the position / size of page elements and obtains layout features through spatial pyramid pooling.

[0047] Semantic Encoder: BERT is used to encode text content, and the CLIP model is used to extract image-text correlation features.

[0048] Joint training: Multimodal features are integrated through the attention mechanism, and a dual-task learning objective is designed. The main task is page quality regression prediction (mean square error loss). The PPT quality assessment model is trained to perform multi-dimensional quantitative scoring of the PPT layout and the matching degree of image and text content.

[0049] (2) Reinforcement learning strategy collaborative training stage Hybrid Action Space Construction: Discrete action set (Deep Q-Network, DQN): defines a variety of fixed high-level operation types, including templated operations such as "vertically center the title bar", "add data chart placeholder", and "switch to three-column layout".

[0050] Continuous action set (deep deterministic policy gradient, DDPG): defines the 6-dimensional adjustment continuity parameters of each element (x, y coordinate offset, width, height change, RGB color increment).

[0051] Sequential action coupling mechanism: At each time step t, DQN first outputs a discrete action type. If the action involves parameter adjustment (such as "enlarge the image"), it triggers DDPG to generate specific indicator parameters (such as "30%"), forming a complete action pair (such as "enlarge the image 30%").

[0052] Hierarchical reward design: Global reward: the difference in page scores output by the quality assessment model; local reward: feasibility verification of design operations (such as element overlap penalty), historical action consistency reward (such as the decay coefficient for continuous use of the same type of layout operations). The global reward is decomposed into each action step through linear weighting.

[0053] Training process: Freeze the quality assessment model parameters and perform reinforcement learning pre-training: The DQN part uses prioritized experience replay and constrains the action selection space in the initial stage (only basic layout-type actions are open). The DDPG part uses random noise exploration and first performs adaptive training in a small range of parameter space (such as ±5% size adjustment).

[0054] Workflow Optimization Execution Phase: After entering the initial PPT, the system performs multi-granular optimization. Using a quality assessment model, it identifies low-scoring pages, prioritizes them, and prioritizes pages with scores below a predetermined threshold. It then iteratively adjusts the target pages. After each step, the adjusted PPT page is rendered. Iterations terminate when a certain number of iterations or scoring threshold is reached. Ultimately, the visual quality of the PPT presentation is enhanced while maintaining content integrity.

[0055] Step 6: User Confirmation and Modification: The system will provide the generated PPT and presentation script for the user to download and modify. The user can manually adjust the content and upload the revised version again. The system will automatically update the PPT content and presentation script based on the changes to meet the user's personalized needs. If the user does not need to modify, they can proceed directly to the next step.

[0056] Step 7: The system generates the explanation video based on the user-approved PPT and explanation script. In the video generation module, the system converts the PPT into video image frames and, combined with the explanation script, generates the corresponding audio. During the audio generation phase, users can independently select timbre, explanation style, and accent configuration parameters. A text-to-speech (TTS) engine generates high-quality audio that matches the content. After combining the audio with the image frames, the system generates a complete video. Audio generation supports batch processing, and the system implements multi-tasking through asynchronous message queues, improving overall generation efficiency.

[0057] Through the above steps, the full-process automated conversion of documents to videos can be achieved, and the generated video clearly displays the core content of the document. For example, the video generated using this technology can help users quickly understand the content of the document, and is suitable for various scenarios such as academic research reports, internal corporate presentations, and teaching video production. Furthermore, if the user is interested in the content of the video, they can also conduct in-depth reading and analysis through the original document. The implementation of the present invention solves the technical difficulties in the prior art of quickly generating PPT presentations and explanation videos from academic papers or similar long documents. Through automated extraction, analysis, and generation processes, the document content can be efficiently converted into the PPT and explanation videos required for the report, thereby significantly reducing the user's time cost and meeting the needs of multiple scenarios such as academic research and corporate reports.

[0058] The multimodal content generation method provided by the embodiment of the present invention integrates a PDF extractor, an outline generator, a PPT generator, and an explanation video generator, etc., to meet the user's needs for rapid content generation. Starting from the PDF document uploaded by the user, a preliminary PPT outline and explanation copy are generated through block parsing and content reorganization. In the process of copy optimization and PPT production, a reinforcement learning algorithm is introduced to automatically adjust and optimize the copy content and PPT layout style through iterative evaluation and feedback of the generated results. Specifically, the system will adjust the organization of the outline content and the corresponding explanation copy in the PPT according to the effect in each step, and optimize the PPT template, layout, image synthesis and other links to achieve the best visual and information communication effect. Finally, the system combines TTS technology with the final draft of the explanation copy obtained by reasoning, converts the PPT into a video file, and realizes the automation of the entire process from text to visual explanation.

[0059] This embodiment of the present invention builds a fully automated system for the entire process from document upload to video generation, encompassing steps such as content extraction, outline generation, PPT production, and video synthesis. Through an efficient modular design, the system significantly reduces the complexity of manual operations, enabling users to quickly generate high-quality content. It introduces semantic parsing technology based on a large language model (LLM) to perform in-depth semantic analysis and logical reasoning on documents, enabling it to handle complex structured documents or unstructured content. Through block parsing and content reorganization, the generated outline is clear and accurate, making it suitable for cross-disciplinary academic documents and technical reports. Content generation fully integrates multiple data formats, such as text, images, and tables, and, combined with the latest image generation models, dynamically generates content-related charts or background images, ensuring high visual and semantic consistency. This system is particularly effective in the presentation of academic reports or technical documents, effectively enhancing the visualization of complex data and abstract concepts. Users can make personalized adjustments at every stage of content generation, including selecting PPT templates, customizing voice tones, and modifying presentation scripts. This application provides a two-way interaction mode between users and the system, allowing users to manually optimize and re-upload content, thereby balancing the efficiency of automated generation with the flexibility of personalized needs.

[0060] By establishing a closed-loop optimization chain of "perception-decision-evaluation," this embodiment of the present invention achieves multi-granularity PPT visual enhancement, from element-level fine-tuning to layout-level reconstruction. It also constructs a joint visual-semantic reasoning framework, providing pixel-level visual optimization guidance for reinforcement learning. Using a DQN-DDPG cascade architecture, it overcomes the limitations of traditional discrete / continuous action space separation, effectively addressing the core pain points of action space dimensionality collapse and rigid parameter adjustment in dynamic layout optimization, providing a scalable, intelligent solution for automated PPT design optimization. Incorporating TTS technology, it supports speech generation for multilingual explanatory texts and offers a variety of timbre options to accommodate diverse cultural contexts and application scenarios. For example, within multinational corporations or international conferences, the system can generate multilingual explanatory videos based on user needs, further expanding the technology's application scope. The combination of deep reasoning technology driven by a large language model and image generation technology ensures high standards of logic, rigor, and aesthetics in the generated content. In particular, in academic papers or corporate reports, the generated outlines, PPTs, and videos fully and accurately convey core information, meeting high standards of rigor. It makes full use of the latest developments in generative artificial intelligence technology, organically integrates large language models, image generation models and speech synthesis technology, overcomes the limitations of traditional content generation technology, and creates new solutions for multimodal information generation.

[0061] The multimodal content generation device provided by the present invention is described below. The multimodal content generation device described below and the multimodal content generation method described above can refer to each other.

[0062] Figure 4 A schematic diagram of the structure of a multimodal content generation device provided by an embodiment of the present invention, such as Figure 4 As shown, the multimodal content generation device provided by the embodiment of the present invention includes: Extraction module 401, for receiving documents uploaded by users and extracting content; The first generating module 402 is used to perform block parsing and semantic reorganization of the extracted content to generate a PPT text outline and a corresponding explanation script; The second generation module 403 matches or dynamically generates visual materials according to the PPT text outline to obtain a preliminary version of the PPT; The optimization module 404 is used to iteratively optimize the initial version of the PPT and synthesize the optimized PPT with the explanation script to obtain an explanation video.

[0063] The multimodal content generation device provided by an embodiment of the present invention receives documents uploaded by users and extracts content; performs block parsing and semantic reorganization on the extracted content to generate a PPT text outline and a corresponding explanation script; matches or dynamically generates visual materials based on the PPT text outline to obtain a preliminary version of the PPT; iteratively optimizes the preliminary version of the PPT, and synthesizes the optimized PPT with the explanation script to obtain an explanation video. The device is capable of processing complex structured documents or unstructured content, ensuring that the generated outline is clear and the content is accurate, and outputs multimodal content that conforms to visual aesthetics through iterative optimization.

[0064] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communications bus 540. The memory 530 includes a computer program, an operating system, and acquired data. The processor 510 may call the logic instructions in the memory 530 to execute a multimodal content generation method, which includes: receiving a document uploaded by a user and extracting the content; parsing and semantically reorganizing the extracted content to generate a PPT text outline and a corresponding explanation script; matching or dynamically generating visual materials based on the PPT text outline to obtain a preliminary PPT; iteratively optimizing the preliminary PPT, and synthesizing the optimized PPT with the explanation script to obtain an explanation video.

[0065] Furthermore, the logic instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the relevant art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0066] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multimodal content generation method provided by the above-mentioned methods, the method comprising: receiving documents uploaded by users and extracting content; performing block parsing and semantic reorganization on the extracted content to generate a PPT text outline and a corresponding explanation script; matching or dynamically generating visual materials according to the PPT text outline to obtain a preliminary version of the PPT; iteratively optimizing the preliminary version of the PPT, and synthesizing the optimized PPT with the explanation script to obtain an explanation video.

[0067] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0068] Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the relevant technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A multimodal content generation method, characterized in that: include: Receive documents uploaded by users and extract content; The extracted content is parsed and semantically reorganized to generate a PPT text outline and corresponding explanation script; Match or dynamically generate visual materials according to the PPT text outline to obtain a preliminary version of the PPT; The initial version of the PPT is iteratively optimized, and the optimized PPT is synthesized with the explanation script to obtain an explanation video.

2. The multimodal content generation method according to claim 1, wherein: The extracted content is parsed and semantically reorganized to generate a PPT text outline, including: The extracted content is logically split into multiple content blocks according to the document and chapters; for each content block, it is asynchronously interacted with the large language model, extracting key information of the chapter based on the set prompt words, and returning the parsing results in a specified format; The parsing results are summarized, and semantic integration is performed on the summarized parsing results through a large language model to generate a logically coherent PPT text outline.

3. The multimodal content generation method according to claim 1 or 2, characterized in that: The step of matching or dynamically generating visual materials according to the PPT text outline to obtain a preliminary version of the PPT includes: Calculating the text similarity between the text content in the PPT text outline and the titles of the visual materials in the parsed chart library page by page, and if the calculated similarity is greater than a preset threshold, matching the corresponding visual materials in the chart library to the corresponding PPT pages; If all the calculated similarities are not greater than a preset threshold, an image generation model based on a diffusion model is called to generate a new chart related to the text content, and the new chart is inserted into the corresponding PPT page.

4. The multimodal content generation method according to claim 1, wherein: The iterative optimization of the initial version of the PPT includes: A two-stage framework of pre-training evaluation and strategy collaborative optimization is constructed, and the initial version of the PPT is iteratively optimized through the two-stage framework, specifically including: In the first stage, a pre-trained PPT quality assessment model is used to perform fine-grained quality scoring on the PPT pages. The quality assessment model uses a multimodal joint encoder to jointly represent the visual and semantic features of page elements and output a hierarchical reward signal. In the second stage, the initial version of the PPT is iterated through the hierarchical reward signal and a discrete-continuous hybrid action strategy. The discrete-continuous hybrid action strategy includes a discrete action strategy network and a continuous parameter decision network, wherein: the discrete action strategy network is used to select high-order operation types from a preset templated adjustment action set; the continuous parameter decision network is used to dynamically generate continuous adjustment parameters of elements.

5. The multimodal content generation method according to claim 4, wherein: The pre-training method of the PPT quality assessment model includes: We collected professionally designed PPT pages as positive samples and generated negative samples by randomly disrupting element layouts and destroying color schemes. We annotated each page with a visual aesthetic score and information density index. A parallel coding architecture is used to extract multimodal features from samples in positive and negative samples. The parallel coding architecture includes a visual encoder and a semantic encoder. The visual encoder uses an object detection network to extract the position and size of page elements, and then uses spatial pyramid pooling to obtain layout features based on the position and size of page elements. The semantic encoder is used to perform BERT encoding on text content to obtain text features, and the CLIP model is used to extract image-text correlation features from image and text content. By fusing multimodal features through the attention mechanism and combining the visual aesthetic score and information density index of each page annotation, the PPT quality assessment model is trained with page quality regression prediction as the main task goal. The matching degree between PPT layout and graphic content is used as the auxiliary task goal, and a multi-dimensional quantitative score of the matching degree between PPT layout and graphic content is trained.

6. The multimodal content generation method according to claim 4, wherein: The iterating of the initial version of the PPT using the hierarchical reward signal and the discrete-continuous hybrid action strategy includes: Constructing a hybrid action space, the hybrid action space including a discrete action set and a continuous action set, the discrete action set including multiple fixed high-order operation types, and the continuous action set including a multi-dimensional adjustment continuity parameter for each high-order operation type; At each time step, the discrete action set outputs a discrete action type. If the action involves parameter adjustment, it triggers the continuous action set to generate specific indicator parameters to form a complete action pair. Design global rewards and local rewards. The global reward is the page score difference output by the quality assessment model, and the local rewards are the operation feasibility verification and historical action consistency rewards. The global reward is decomposed into each action step through linear weighting; Fixed quality assessment model parameters and reinforcement learning pre-training: discrete action strategies use prioritized experience replay to constrain the action selection space in the initial stage, while continuous action strategies use random noise exploration and adaptive training in a small parameter space. After the reinforcement learning pre-training is completed, multi-granularity optimization is performed on the initial version of the PPT, including: locating low-scoring pages through a quality assessment model, prioritizing pages with scores less than a predetermined threshold after sorting them from low to high; iteratively adjusting the target page to be optimized, rendering the adjusted PPT page after each step of the operation, and ending the iteration when the preset number of iterations or the scoring threshold is reached.

7. The multimodal content generation method according to claim 1, wherein: The synthesizing the optimized PPT and the explanation script to obtain the explanation video includes: Send the iteratively optimized PPT to the client, receive feedback from the client, and update the PPT content and explanation script based on the feedback; Converting the updated PPT content into image frames, and using a text-to-speech engine to generate corresponding audio from the updated explanation script; The audio and image frames are combined to generate a complete video.

8. A multimodal content generation device, characterized in that: include: The extraction module is used to receive documents uploaded by users and extract content; The first generation module is used to parse and semantically reorganize the extracted content into blocks to generate a PPT text outline and corresponding explanation script; The second generation module matches or dynamically generates visual materials according to the PPT text outline to obtain a preliminary version of the PPT; The optimization module is used to iteratively optimize the initial version of the PPT and synthesize the optimized PPT with the explanation script to obtain an explanation video.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the multimodal content generation method according to any one of claims 1 to 7 is implemented.

10. A non-transitory readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal content generation method according to any one of claims 1 to 7 is implemented.

Images