Speech content adaptive generation method based on multi-modal large model and real-time question and answer system

Abstract

This invention relates to the field of teaching aids technology, and more particularly to a method for adaptive generation of speech content based on a multimodal large model and a real-time question-answering system. The method includes the following steps: acquiring multimodal raw materials provided by the speaker and parsing them using a multimodal large model to extract the speech's logical structure, key entities, and content attribute tags; organizing the multimodal large model parsing results into a content tree and constructing a knowledge graph representing arguments, sub-arguments, cases, data, and conclusions, wherein the knowledge graph defines support, exemplification, and comparison relationships between nodes; learning from the speaker's historical audio / video and text data and outputting a speaker style vector; generating an initial speech suite and configuring adjustment strategy templates, priority definitions, and safety boundaries. In this invention, by organizing content into an operable knowledge graph / content tree and constructing a real-time closed loop of "perception—understanding—decision—generation," dynamic path selection and content generation based on real-time multimodal perception are achieved.

Description

Technical Field

[0001] This invention relates to a method for adaptive generation of speech content based on a multimodal large model and a real-time question-answering system, belonging to the field of teaching aid technology. Background Technology

[0002] Currently, the preparation and presentation process for speeches and classroom teaching is primarily based on linear documents or slides (PPT). Authors typically organize their presentation materials and slides in a fixed order beforehand, using tools such as text editors, slide generators, speech recognition / transcription, text summarization, and automatic image matching. Classroom interaction mainly relies on voting / answering systems, chat / question modules, or teaching assistant intervention to achieve limited real-time feedback collection. Some technical solutions can perform offline analysis of classroom recordings or audio (such as facial expression recognition, attention statistics, or voice emotion recognition) for post-class evaluation and improvement.

[0003] However, existing technologies mostly exist as point-like functions, failing to form a tightly coupled, holistic solution that integrates semantic and node-based representation of content with real-time multimodal perception, automatic decision-making, and content generation. Specifically, on the one hand, presentation content typically exists as linear, fragmented text or slides, lacking a unified structured representation (such as a content tree or knowledge graph), making it difficult to achieve dynamic jumps, deletions, and deep combinations at the node level. On the other hand, although there are methods for detecting or analyzing single modalities such as facial expressions and vocal emotions offline, the ability to uniformly collect, fuse, and semantically process multimodal real-time signals (including facial expressions, posture, gaze, ambient sound, vocal keywords, and speaker physiological / vocal characteristics) is insufficient, failing to drive immediate and interpretable content adjustment and presentation control. Furthermore, existing systems that automatically generate or supplement content typically do not model the speaker's personalized expression habits (such as speaking speed, pauses, and rhetorical preferences), resulting in generated content that does not match the speaker's style and has low acceptability. The retrieval and generation of visual / auditory materials are mostly manual or semi-automatic processes, making it difficult to ensure a high degree of consistency with the current narrative context. Therefore, there is an urgent need to improve the adaptive generation method of speech content based on multimodal large models and the real-time question-answering system to solve the above-mentioned problems. Summary of the Invention

[0004] The purpose of this invention is to provide an adaptive speech content generation method and a real-time question-answering system based on a multimodal large model, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: An adaptive speech content generation method and real-time question answering system based on a multimodal large model include the following steps: S1 Preparation and Training Phase: Receive multimodal raw materials provided by the speaker (e.g., course standards, syllabus, draft lecture notes, references, old version of presentation slides, reference videos, etc.) and use a multimodal large-scale model to analyze and extract the logical structure of the speech (e.g., chapters, arguments, evidence, transitions, and conclusions), key entities (e.g., core concepts, names, data, terminology, dates, etc.), and content attribute tags (e.g., "background introduction," "core technologies," "difficulty analysis," "success stories," "emotional appeal"). S2 Content Structuring: The parsing results of step S1 are organized into a content tree and a knowledge graph representing arguments, sub-arguments, cases, data and conclusions is constructed. The knowledge graph defines the support, example, comparison and other relationships between nodes, providing a logical basis for subsequent intelligent jumps, deletions and deepening. S3 Speaker Style Modeling: Based on the speaker's historical audio, video, and text learning, it outputs a speaker style vector (specifically: inputting the speaker's historical speech videos, audio, and text transcripts, analyzing and learning from the large model to quantify their: language style (vocabulary preference (formal / colloquial), sentence complexity, humor), expression patterns (common opening remarks, transition sentences, and concluding sentences), and prosodic features (average speaking speed, pause habits, and stress patterns), generating a "speaker style vector" for subsequent generation of adjusted content that matches their personal characteristics); S4 Policy Rule Base Construction: Based on steps S1~S3, generate an initial speech suite and configure and adjust policy templates, priority definitions, and security boundaries (including ethical boundaries). S5 Execution and Adaptation Phase: During the presentation, multimodal real-time perception, contextual understanding and state judgment, decision engine to generate adjustment instructions, real-time content adaptation and presentation, and recording and iteration are performed in a loop. S6: After the speech, perform offline retrospective analysis based on the recorded perception-decision-execution logs, automatically generate an evaluation report, and update the knowledge graph, strategy rule base, and speaker style vector accordingly to improve subsequent adaptive performance.

[0006] Furthermore, the multimodal parsing described in step S1 includes: natural language processing (sentence segmentation, syntactic analysis, semantic annotation), image / video visual analysis (keyframe extraction, scene and object recognition), and speech / audio processing (speech recognition, tone / emotion analysis), and cross-modal alignment of the results of each modality to generate a unified semantic representation.

[0007] Furthermore, the knowledge graph includes node type identifiers (including but not limited to "background introduction", "core technology", "difficulty analysis" and "successful cases") at the node level, and supports path search based on graph reasoning, similar node retrieval and evidence chain backtracking for real-time retrieval and generation.

[0008] Furthermore, the speaker style vector is composed of vocabulary / phrase preference distribution, sentence complexity statistics, and prosodic / rhythmic features (average speech rate, typical pause distribution, stress pattern), and is used in step S5 to constrain the text generation module so that the generated content conforms to the speaker's personalized expression.

[0009] Furthermore, the strategy rule base includes trigger conditions and response action pairs. The trigger conditions include at least the audience confusion threshold, the interaction rate threshold, and the time pressure threshold. The response actions include inserting metaphors, skipping extended content, triggering Q&A, or initiating a poll. When the trigger conditions are met in real time, the decision engine selects the corresponding response action according to priority.

[0010] Furthermore, the multimodal real-time perception in step S5 includes the acquisition of: audience modality (facial expressions, posture, gaze), environment modality (ambient sound intensity and real-time transcription), speaker modality (speech rate, volume, teleprompter deviation, physiological signals acquired by wearable devices), and progress modality (time elapsed, remaining time, current content node); the acquisition and use of the physiological signals follow the privacy protection and ethical boundaries defined in step S4.

[0011] Furthermore, the adjustment instructions generated by the decision engine include, but are not limited to, content flow instructions (deepen, skip, jump, insert case studies), visual assistance instructions (highlight, pop-up annotations, switch charts, play videos), interactive instructions (ask questions, initiate polls), and expression suggestion instructions (slow down speech, pause, gesture prompts); and the real-time generation module performs text rewriting, visual control, and interactive triggering, pushing the adjustment results to the teleprompter, AR device, or presentation screen for presentation by the speaker or audience.

[0012] Furthermore, it includes: a multimodal input module, a content parsing module, a content structuring and knowledge graph module, a multimodal material management module, a speaker style learning module, a strategy rule base module, a real-time perception module, a multimodal fusion and state judgment module, a decision engine module, a real-time generation and presentation module, and a data recording and iteration module; the modules work together to implement the steps of the method described in claim 1.

[0013] Furthermore, the system further includes a privacy and ethics boundary module and a question-answering enhancement module. The privacy and ethics boundary module is used to implement desensitization, minimize collection and access control during data collection, transmission, storage and generation, and trigger blocking or rollback when potentially illegal content is detected. The question-answering enhancement module performs retrieval enhancement generation (RAG) based on knowledge graphs and external controlled knowledge sources to provide real-time, auditable answers to audience questions during presentations and write the question-answer records back to the knowledge graph for subsequent optimization.

[0014] This invention has at least the following beneficial effects: This invention organizes content into an operable knowledge graph / content tree and constructs a real-time closed loop of "perception-understanding-decision-generation," enabling dynamic path selection and content generation based on multimodal real-time perception. It introduces speaker style vectors to ensure the adoptability of generated text and expression suggestions, automatically associates or generates visual / audio materials, and provides interaction (voting, Q&A) and quantitative evaluation, thereby improving the adaptability, participation, and information internalization efficiency of the classroom / lecture. At the same time, the design retains the teacher's ultimate control and incorporates privacy and ethical constraints, facilitating quantifiable effect verification and compliant deployment. Attached Figure Description

[0015] Figure 1 This is a flowchart of the speech content adaptive generation method based on a multimodal large model according to the present invention. Detailed Implementation

[0016] The following will describe in detail the implementation of this application with reference to the accompanying drawings and embodiments, so that the implementation process of how this application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly. Example

[0017] Teachers use the system to conduct an interactive, adaptive teaching presentation, specifically a 45-minute high school biology lesson on "Structure and Function of Cells"; Phase 1: Preparation and Training Phase—Building the Foundation of Teaching Content and Personalized Models; Step 1.1: Multimodal Input and Content Parsing: Input: The teacher provided the course standards and syllabus, lecture notes and materials, an older version of the PPT, and reference video materials, specifically: Curriculum Standards and Syllabus: Curriculum Standards, Chapter Teaching Objectives, and Individual Lesson Plans for "Structure and Function of Cells" Lecture notes and materials: lecture notes from previous years, electronic versions of corresponding chapters in textbooks, and three popular science articles on the latest cell imaging technologies; The old version of PowerPoint: a traditional linear presentation containing organelle images and basic functions; Reference videos: a 5-minute 3D animation demonstration of organelles, and a microscopic video showing white blood cells hunting bacteria; Processing: The multimodal large model analyzes these inputs and extracts: Logical structure: Identify the overall structure of "cell membrane → cytoplasm → cell nucleus" and the argumentation pattern of "structural description → functional explanation → real-life examples"; Key entities: Extract core concepts and terms such as "mitochondria", "chloroplasts", "ribosomes", "active transport", "ATP", and "protein synthesis".

[0018] Content attribute tags: Tag the content, such as "abstract concept: cell membrane fluidity", "microscopic structure: mitochondrial cristae", "difficulty: comparison of active and passive transport", "vivid example: amoeba engulfing food", "cutting-edge connection: CRISPR technology and cell nucleus".

[0019] Step 1.2: Deep Content Structuring and Knowledge Graph Construction The system will construct a "cell knowledge graph" from the parsed results.

[0020] Core nodes: "cell", "cell membrane", "mitochondria", "nucleus", etc. Relationship edge: "cell" includes "cell membrane", "cytoplasm" and "nucleus".

[0021] Mitochondria function as "energy factories," for example, "muscle cells need a large number of mitochondria after exercise."

[0022] "Active transport" requires the consumption of "ATP" compared to "passive transport".

[0023] Ribosomes are located on the endoplasmic reticulum and their function is to synthesize proteins.

[0024] Objective: This atlas visualizes the concept that "the cell is a system," providing a logical path for in-depth explanations of related knowledge points in the classroom based on student feedback (such as jumping from "protein synthesis" to "gene expression").

[0025] Step 1.3: Multimodal Material Association and Generation Association: The system automatically associates knowledge graph nodes with materials; Associate the “Cell Membrane Fluid Mosaic Model” node with a dynamic scintillation molecular animation.

[0026] The "chloroplast" node is associated with a video clip showing the flow of chloroplasts in a plant leaf under a microscope.

[0027] Generation: For the abstract process of "protein synthesis on ribosomes", the system generates a 30-second 3D simulation animation based on the description, showing the dynamic process of tRNA transporting amino acids.

[0028] Step 1.4: Speaker Style Learning and Modeling Input: Recordings of the teacher's classes from the past semester.

[0029] Analysis: The model learned Teacher Wang's style: colloquial language, fondness for "city metaphors" (such as "mitochondria are power plants" and "cell membranes are border control and customs"), habit of pausing after difficult points and asking "Did I explain it clearly?", and moderate average speaking speed.

[0030] Output: Generate a "teacher's teaching style vector" to ensure that the subsequently generated explanatory text and prompts conform to their style.

[0031] Step 1.5: Generate the initial teaching kit and policy rule base configuration Output: Generate a structured interactive lesson plan and a smart PPT presentation. The PPT is non-linear, and each knowledge node can be expanded.

[0032] Configure the strategy library: Core objective: To understand the correspondence between organelle structure and function, and to establish a biological perspective of "structure-function adaptation". Essential topics to cover: the basic functions of the cell membrane, mitochondria, and nucleus.

[0033] Expandable: fine structures of various organelles (such as mitochondrial cristae), special cases (plant cell walls).

[0034] This section may be edited out: some overly cutting-edge research introductions.

[0035] Adjust strategy template: If the status check is "Most students are confused" or the current node is "Abstract Concept", the instruction is: Insert a metaphorical example or switch to a more detailed animation.

[0036] If the status check indicates "high audience interest" or the current node has a "frontier connection" instruction: a further reading card will pop up, suggesting discussion after class.

[0037] If the status check indicates "Remaining time is 10% less than planned": skip one scalable case and proceed to the summary section.

[0038] If the voice recognition detects the keyword "cancer": when explaining "cell nucleus / DNA", provide a brief explanation of "cell carcinogenesis".

[0039] Phase Two: Execution and Adaptation Phase—Real-time Dynamic Perception and Adjustment (In Progress) Scene Segmentation: When explaining "Mitochondria—The Power Plant of the Cell" Loop step 2.1: Multimodal real-time sensing Audience modality: The camera revealed that over 60% of the students had slightly furrowed brows and leaned back; eye tracking showed that many students' gazes frequently shifted between the "ATP synthesis chemical formula" chart on the PPT and the teacher.

[0040] Environmental modality: The microphone picked up sporadic coughing and page-turning sounds, indicating a low level of environmental activity.

[0041] Speaker modality: The teacher speaks at a steady pace, but according to teleprompter tracking, he has spent more time on the current complex chart page than the average duration.

[0042] Progress mode: 30 minutes have been taken, slightly exceeding the original plan. Currently located at the "Mitochondrial-Energy Conversion" node in the knowledge graph.

[0043] Loop Step 2.2: Context Understanding and State Judgment The large model integrates all data to generate state judgments: "State Alert: Core Difficulty. Students are generally confused about the abstract concept of 'chemical energy conversion,' their classroom attention is declining, and the pace is slightly delayed. Immediate explanatory intervention is recommended." Step 2.3 of the loop: The decision engine generates adjustment instructions. Match strategy library rules (insert analogy cases if most are confusing or abstract concepts).

[0044] The knowledge graph retrieved a related analogy: "A real-life analogy: Mitochondria are like a car engine, and glucose is like gasoline. The energy released after combustion (oxidative decomposition) cannot be directly used to drive the car. It needs to be converted into electrical energy (like ATP) first. Only then can the battery (ATP) stably drive the motor (cellular life activities)." Generate instruction combinations: Content Stream Command: Insert a metaphorical example.

[0045] Visual aids: Highlight the "ATP" section in the PPT chart, switch views: switch from a chemical equation chart to an analogy diagram of "car engine-battery-car".

[0046] Instructions for expressing suggestions: It is suggested to pause here and ask, "Does this analogy help everyone understand the process of energy conversion?" Step 2.4: Real-time content adaptation and generation The text generation module transforms the metaphorical examples into conversational sentences that fit Teacher Wang's style, and pushes them to the teleprompter on the podium: "Classmates, we can imagine mitochondria as an engine..." The PPT automatically highlights key parts and switches to a more intuitive analogy.

[0047] The teleprompter clearly displays suggested questions.

[0048] Cyclic Step 2.5: Presentation, Delivery, and Human-Machine Collaboration For the teacher: The teleprompter highlights the metaphorical text and suggested questions. The teacher's bone conduction headphones whisper, "You could use an analogy." For students: The PPT on the screen transforms from complex chemical formulas into vivid car analogies.

[0049] Human-computer collaboration: When the teacher sees the prompt, they understand it and naturally use the analogy. They ask the students questions as suggested, and the students' expressions brighten up. They nod in agreement. The system senses this positive change and updates the status to "comprehension restored" in the next cycle. It continues to proceed along the original path, but may skip another complex extension diagram to ensure that the teaching is completed on time.

[0050] The core value of the system is reflected in this embodiment: From static lesson plans to dynamic learning paths: teaching is no longer about mechanically playing PPTs, but about dynamically selecting the most suitable explanation path based on students' real signals of "confusion" or "interest" (such as inserting metaphors or switching views). From listing knowledge points to a structured cognitive network: Teachers and students can see the "cell" as a knowledge graph of the system, understand the relationship between the parts, rather than memorizing isolated terms; From one-way instruction to two-way teaching resonance: students' micro-expressions and classroom atmosphere are quantified into effective feedback, directly prompting adjustments to teaching content and pace, making the classroom truly responsive to students' learning and achieving "teaching based on learning"; Through this embodiment, the system transforms a traditional biology classroom into a dynamic, interactive learning experience driven by data and AI, and centered on student understanding.

[0051] The foregoing description illustrates and describes several preferred embodiments of the present invention. However, as previously stated, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept described herein through the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A speech content adaptive generation method based on a multimodal large model, characterized in that, Includes the following steps: We acquire the multimodal raw materials provided by the speaker and use a large multimodal model to analyze and extract the logical structure, key entities, and content attribute tags of the speech. The results of the multimodal large model analysis are organized into a content tree and a knowledge graph representing arguments, sub-arguments, cases, data and conclusions is constructed. The knowledge graph defines the support, example and comparison relationships between nodes. Learn and output speaker style vectors based on the speaker's historical audio and video recordings and text. Generate an initial presentation suite and configure and adjust policy templates, priority definitions, and security boundaries; During the presentation, the system continuously executes multimodal real-time perception, contextual understanding and state judgment, decision engine to generate adjustment instructions, real-time content adaptation and presentation, and recording and iteration.

2. The speech content adaptive generation method based on a multimodal large model according to claim 1, characterized in that: It also includes the following steps: After the speech, an offline retrospective analysis is performed based on the recorded perception-decision-execution logs to automatically generate an evaluation report and update the knowledge graph, policy rule base, and speaker style vector accordingly to improve subsequent adaptive performance.

3. The method for adaptive generation of speech content based on a multimodal large model according to claim 1, characterized in that: The multimodal large model parsing includes natural language processing, image and video visual analysis, and speech and audio processing, and performs cross-modal alignment on the results of each modality to generate a unified semantic representation.

4. The speech content adaptive generation method based on a multimodal large model according to claim 1, characterized in that: The knowledge graph includes node type identifiers at the node level and supports path search, similar node retrieval, and evidence chain backtracking based on graph reasoning for real-time retrieval and generation.

5. The speech content adaptive generation method based on a multimodal large model according to claim 1, characterized in that: The speaker style vector consists of vocabulary and phrase preference distribution, sentence complexity statistics, and prosodic and rhythmic features.

6. The speech content adaptive generation method based on a multimodal large model according to claim 1, characterized in that: The strategy template includes a pair of trigger conditions and response actions. The trigger conditions include at least an audience confusion threshold, an interaction rate threshold, and a time pressure threshold. The response actions include inserting a metaphor, skipping extended content, triggering a Q&A session, or initiating a poll. When real-time perception reaches the triggering condition, the decision engine selects the corresponding response action according to priority.

7. The speech content adaptive generation method based on a multimodal large model according to claim 1, characterized in that: The multimodal real-time perception includes collecting audience modality, environment modality, speaker modality, and progress modality.

8. The method for adaptive generation of speech content based on a multimodal large model according to claim 1, characterized in that: The adjustment instructions generated by the decision engine include, but are not limited to, content flow instructions, visual assistance instructions, interactive instructions, and expression suggestion instructions; and the real-time generation module performs text rewriting, visual control, and interactive triggering, pushing the adjustment results to the teleprompter, AR device, or presentation screen for presentation by the speaker or audience.

9. A speech content adaptive real-time question answering system based on a multimodal large model as described in any one of claims 1-8, characterized in that: The system includes a multimodal input module, a content parsing module, a content structuring and knowledge graph module, a multimodal material management module, a speaker style learning module, a strategy rule base module, a real-time perception module, a multimodal fusion and state judgment module, a decision engine module, a real-time generation and presentation module, and a data recording and iteration module.

10. The speech content adaptive real-time question answering system based on a multimodal large model according to claim 9, characterized in that: The system further includes a privacy and ethics boundary module and a question-answering enhancement module. The privacy and ethics boundary module is used to implement desensitization, minimize collection and access control during data collection, transmission, storage and generation, and trigger blocking or rollback when potential illegal content is detected. The question-answering enhancement module performs retrieval enhancement generation based on knowledge graph and external controlled knowledge sources, and is used to provide real-time, auditable answers to audience questions during presentations and write the question-answering records back to the knowledge graph for subsequent optimization.

Images