Intelligent reporting method and system based on interaction of large language model and digital human

Abstract

The invention discloses an intelligent reporting method and system based on interaction of a large language model and a digital human. The intelligent reporting method comprises the following steps: performing streaming voice transcription and effective text screening through automatic voice recognition; performing user input intention analysis through structured cue word guidance and local lightweight large language model reasoning; performing multi-intention instruction branch scheduling according to a control class intention priority rule, and initializing a script index; voice broadcast is carried out through structured introduction script hierarchy analysis, meanwhile, demonstration control is carried out in parallel, and real-time interruption monitoring is kept. According to the invention, a double-engine collaborative architecture is combined with a dynamic content generation and process recovery technology of intention structured analysis, state precise management and context perception, so that the problem that structured introduction script broadcasting and high-degree-of-freedom voice interaction are difficult to consider at the same time is solved; and the problems of unsmooth recovery after interruption, stiff detailed adjustment response and lack of consistency in question answering and broadcasting flow splitting are solved.

Description

Technical Field

[0001] This invention relates to an intelligent reporting method based on a large language model and digital human interaction, and also to a corresponding intelligent reporting system, belonging to the field of human-computer interaction technology. Background Technology

[0002] Currently, there are three main technical solutions in the field of automated reporting, but all have significant functional shortcomings. First, fixed audio and video broadcasting systems rely on pre-recorded content, triggering playback with clicks, completely lacking interactivity. They cannot respond to real-time user questions or adjust the pace of the presentation according to needs, merely mechanically executing a pre-set process. Second, keyword-matching question-and-answer systems generate responses by recognizing keywords in user input and matching them with pre-set templates. While offering some interactive flexibility, their semantic understanding depth is limited, making it difficult to handle complex, ambiguous, or implicit questions. Third, knowledge-based large-model question-and-answer systems combine retrieval-enhanced generation technology with large language models, supporting natural language interaction, but lacking control over structured reporting processes. Once the presentation is interrupted by the user, it is difficult to restore the original context, leading to logical breaks.

[0003] None of the aforementioned technical solutions can simultaneously meet the dual requirements of structured script broadcasting and highly flexible voice interaction, resulting in a poor user experience in practical applications. When users interrupt the presentation, jump between modules, or adjust the level of detail, existing systems often respond abruptly, failing to achieve a natural flow of the presentation content and leading to a fragmented experience. Furthermore, the Q&A session is disconnected from the broadcast flow, easily deviating from the presentation topic and affecting the consistency and accuracy of information delivery. In addition, most existing systems only support voice broadcasting, making it difficult to coordinate with UI highlighting, page transitions, animation triggers, and other demonstration operations, limiting their application value in multi-dimensional presentation scenarios such as product demonstrations and company introductions.

[0004] Chinese patent application CN118886966A discloses a speech recognition-based advertising delivery solution that enables ad skipping by matching user voice with ad keywords. However, this solution still struggles to balance structured broadcasting with flexible interaction and has shortcomings in areas such as interruption recovery, content adjustment, and the integration of Q&A and broadcasting. Therefore, designing an intelligent reporting system that deeply integrates structured broadcasting and intelligent voice interaction, achieving smooth recovery after interruption, dynamic content adaptation, and seamless transition between Q&A and explanation, has become a pressing technical problem in this field. Summary of the Invention

[0005] The primary technical problem to be solved by this invention is to provide an intelligent reporting method based on a large language model and digital human interaction.

[0006] Another technical problem that this invention aims to solve is to provide an intelligent reporting system based on a large language model and digital human interaction.

[0007] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution: According to a first aspect of the present invention, an intelligent reporting method based on a large language model and digital human interaction is provided, comprising the following steps: S1: Continuously perform streaming speech transcription through a locally deployed automatic speech recognition module, and output valid text segments after sentence segmentation detection and silence filtering; S2: Input the valid text fragment into the intent recognition module. The intent recognition module constructs structured prompt words and calls a local lightweight large language model for inference, and outputs a structured intent instruction list. The intent types included in the structured intent instruction list are limited to start reporting, terminate reporting, skip the current module, briefly introduce the current module, introduce the current module in detail, and ask a question. S3: Traverse the structured intent instruction list and schedule its execution according to the rule that control intents take precedence over question-and-answer intents; S4: If the start report or continue report is executed, the broadcast text and associated control action of the current step are read from the structured introduction script according to the current module index and step index maintained by the state manager, and the voice broadcast and the control action are executed in parallel. During this process, the execution steps S1 and S2 are continuously returned to listen for the user's interrupt command.

[0008] Preferably, step S2 includes the following sub-steps: S21: Construct a structured prompt template; S22: Call the local lightweight large language model for inference; S23: Perform a comprehensive validation of the structured output generated by the large model; if the output is a valid JSON array and there are no conflicts, it is determined to be a valid intent instruction sequence; if the output is conflicting or the user's input intent is ambiguous, it is determined to be a conflict state and a targeted clarification request instruction is automatically generated. S24: Output structured intent instructions.

[0009] Preferably, the multi-intent instruction branch in step S3 includes: Branch A: The trigger condition is that the intent instruction list contains a startup intent; Branch B: The trigger condition is that the intent instruction list contains control intents; Branch C: The trigger condition is that the intent instruction list contains question-and-answer type intents and there are no control type intents.

[0010] Ideally, when branch C is triggered, the system will enter the intelligent question-and-answer mode and initiate the process of answering the user's question. The solution process includes the following steps: First, semantic analysis is performed on the extracted user questions to determine whether they fall within the coverage of the currently loaded structured introduction scripts; If the answer falls within the scope of the script, then the relevant answer information will be accurately extracted from the currently loaded script. If the scope of the script is exceeded, the search enhancement generation process will be initiated.

[0011] Preferably, the retrieval enhancement generation process is executed in three steps: S01: Using user questions as the core search keywords, perform similarity matching in a preset vector knowledge base, and select the Top-N knowledge fragments with the highest relevance; S02: Integrate two types of key contextual information with the retrieved knowledge fragments. The first type includes the current module topic and a summary of the core points of the previously broadcast content to ensure that the answer does not deviate from the current reporting scenario. The second type includes the user's previous questions and the system's responses during this report. S03: Input the integrated retrieval knowledge fragments, two types of context, and user questions into the large model. The large model generates answers according to preset rules, while controlling the length and conversational level of the answers to adapt to voice broadcasting scenarios.

[0012] Ideally, after generating the answer, a voice broadcast is performed. After the broadcast is completed, the system can be interrupted and resumed or wait for new instructions, depending on the user's needs.

[0013] Preferably, after step S4, the following is also included: S5: If a control intent is detected during the broadcast, the current broadcast is paused, and the operation corresponding to the control intent is executed; if the intent is a brief introduction or a detailed introduction, a summary or detailed explanation is dynamically generated and broadcast based on the original text, context fragments, and relevant knowledge retrieved from the vector knowledge base; if the intent is a question-and-answer session, the answer is obtained and broadcast by combining the current reporting context and the dialogue history of this interaction through a retrieval enhancement generation process. S6: After the broadcast in step S5 is completed, ask the user whether to continue the original reporting process; if the user confirms to continue, the state manager restores the module index and step index at the time of the interruption according to the breakpoint information it recorded, and returns to step S4 to continue execution; otherwise, generate a closing statement and return to the standby state.

[0014] Preferably, the structured introduction script is a hierarchical structure containing multiple modules; each module contains multiple steps that are executed sequentially; each step is associated with broadcast text and at least one control action.

[0015] According to a second aspect of the present invention, an intelligent reporting system based on a large language model and digital human interaction is provided, comprising a processor and a memory; wherein the memory is coupled to the processor and is used to store a computer program, which, when executed by the processor, enables the processor to implement the above-described intelligent reporting method.

[0016] Compared with existing technologies, this invention achieves a natural integration of structured presentation scripts and highly flexible voice interaction through a dual-engine collaborative architecture, structured intent parsing, and precise state management. The system can monitor user commands in real time, supporting interruptions, module jumps, or adjustments to content detail during presentations, and smoothly resumes playback from interruption points, maintaining contextual coherence. The Q&A session is deeply integrated with the presentation flow, dynamically generating answers based on the current presentation topic and historical interactions to ensure information consistency and accuracy. Simultaneously, it can trigger UI highlighting, page transitions, and animation activation, achieving synchronized output of voice explanations and visual demonstrations, thereby significantly improving the interactive experience, content adaptability, and multi-scenario application capabilities. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating an intelligent reporting method based on a large language model and digital human interaction, as shown in the first embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of an intelligent reporting system based on a large language model and digital human interaction, as shown in the second embodiment of the present invention. Detailed Implementation

[0018] The technical content of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0019] First Embodiment like Figure 1 As shown, the first embodiment of the present invention provides an intelligent reporting method based on a large language model and digital human interaction, which includes at least the following steps: S1: Continuously performs streaming speech transcription through a locally deployed automatic speech recognition module, and outputs valid text segments after sentence segmentation detection and silence filtering.

[0020] S2: Input the valid text fragment into the intent recognition module. The intent recognition module constructs structured prompt words and calls the local lightweight large language model for inference, and outputs a structured intent instruction list. The intent types included in the structured intent instruction list are limited to start reporting, terminate reporting, skip the current module, briefly introduce the current module, provide a detailed introduction to the current module, and ask a question.

[0021] S3: Traverse the structured intent instruction list and schedule its execution according to the rule that control intents take precedence over question-and-answer intents.

[0022] S4: If the start report or continue report is executed, the broadcast text and associated control action of the current step are read from the structured introduction script according to the current module index and step index maintained by the state manager, and the voice broadcast and control action are executed in parallel. During this process, the execution steps S1 and S2 are continuously returned to listen for the user's interrupt command.

[0023] S5: If a control intent is detected during the broadcast, the current broadcast is paused, and the operation corresponding to the control intent is executed. If the intent is a simple introduction or a detailed introduction, a summary or detailed explanation is dynamically generated based on the original text, context fragments, and relevant knowledge retrieved from the vector knowledge base for broadcast. If the intent is a question-and-answer session, the answer is obtained through a retrieval-enhanced generation process, based on the current reporting context and the dialogue history of this interaction, and then broadcast.

[0024] S6: After the broadcast in step S5 is completed, ask the user whether to continue the original reporting process; if the user confirms to continue, the state manager restores the module index and step index at the time of the interruption according to the breakpoint information recorded by the user, and returns to step S4 to continue execution; otherwise, generate a closing statement and return to the standby state.

[0025] The following is a detailed explanation of each step of the above method: In step S1, the core task is to achieve real-time capture and accurate preprocessing of user voice input, laying the foundation for subsequent intent parsing. Continuous acquisition of voice signals from the surrounding environment is achieved using a locally deployed automatic speech recognition module instead of a cloud-based one. The key advantage of this approach is avoiding data transmission delays and privacy risks, ensuring the immediacy and security of voice processing. This automatic speech recognition module has streaming capabilities, converting continuously acquired audio streams into continuous text segments in real time, breaking the efficiency limitations of traditional batch processing modes and enabling dynamic processing with simultaneous acquisition and conversion. The module incorporates an advanced sentence segmentation detection algorithm, accurately splitting complete semantic units by analyzing features such as pause duration and intonation changes in the voice signal, avoiding misjudgments of intent due to text fragmentation. It also features intelligent silence filtering, automatically filtering out environmental noise, meaningless pauses, and other invalid signals by recognizing parameters such as the amplitude and frequency of the audio signal, retaining only the voice content with actual semantic meaning. Strict effective text trigger conditions are set; the subsequent intent parsing process is only triggered when the text segment output by the automatic speech recognition module meets the semantic integrity requirements, preventing invalid signals from consuming computing resources. Throughout the process, continuous voice monitoring in standby mode ensures that users can initiate interactive commands at any time. This not only guarantees the convenience of user operation but also improves response efficiency and processing accuracy through local real-time processing and precise filtering mechanisms, providing solid support for the smooth execution of subsequent processes.

[0026] Step S2 includes the following sub-steps: S21: Build a structured prompt template.

[0027] S22: Call the local lightweight large language model for inference.

[0028] S23: Perform a comprehensive validation of the structured output generated by the large model.

[0029] S24: Output structured intent instructions.

[0030] Specifically: The core objective of step S21 is to provide clear constraints and a unified format for intent recognition, and to construct a structured prompt word template as follows: The module's role is clearly defined as an instruction parser for the intelligent reporting system, and the module is limited to recognizing only six types of intents: start reporting, stop reporting, skip the current module, briefly introduce the current module, provide a detailed introduction to the current module, and ask a question. Define conflict rules, which state that requiring both a brief introduction and a detailed introduction simultaneously is considered a conflict instruction; The output must be a JSON array, with each element containing the fields intent (intent type), target (target object, defaults to "current_module"), and focus (optional, the keyword for the issue focus). For ambiguous or conflicting scenarios, the response format should be clearly defined, and JSON data containing a conflict identifier and a clarification request should be returned. Reserved in the template<USER_INPUT> Placeholders are used to replace user text identified by ASR.

[0031] In step S22, the complete prompt word (including the replaced user text) constructed in step S21 is input into a locally deployed small-volume large language model (e.g., Qwen3-14B). The model's fast reasoning ability is used to perform semantic analysis on the user input and generate a structured output result that conforms to the prompt word constraints, thus avoiding network latency and dependency issues caused by cloud calls.

[0032] In the next step S23, a comprehensive verification of the structured output generated by the large model is performed: If the output is a valid JSON array and there are no logical conflicts such as the same target module containing both a brief introduction and a detailed introduction, receiving skip or terminate instructions in a non-reporting state, or a mix of start and Q&A instructions that cannot be separated, then it is determined to be a valid intent instruction sequence. If any of the above-mentioned conflict situations are detected, or if the user's input intent is ambiguous, it is determined to be a conflict state, and a targeted clarification request instruction will be automatically generated, such as "Do you want me to give a brief introduction or a detailed explanation? Please clarify." Finally (i.e., step S24), after verification, a list of intent commands in standard JSON format is output, as shown in the example: [ {"intent":"elaborate","target":"current_module","focus":"configuration method"}, {"intent":"question","query":"Will the resume page change after changing the type?"} ] This instruction list fully includes the type of user intent, the target object, and focus information (if any). It serves as the sole basis for subsequent process scheduling, ensuring unambiguous instruction transmission.

[0033] This step begins with the construction of structured prompts, creating a fixed template with strong constraints and guidance. The template clearly defines the system's role as an instruction parser for the intelligent reporting system, limiting the recognizable intent categories to initiating a report, terminating a report, skipping the current module, providing a brief introduction to the current module, providing a detailed introduction to the current module, and asking a question, thus preventing the model from recognizing irrelevant intents. The template strictly stipulates that the output format must be a JSON array, specifying the fields that each array element must contain. Intent type is a required field, the default target is the current module, and the question focus keyword is an optional field. It also defines conflict instruction judgment rules, defining a conflict as the simultaneous appearance of both a brief introduction and a detailed introduction. Vague inputs or conflicting instructions require a clarification request, ensuring a consistent output format and clear logic. Next, the constructed prompts are combined with the user's input text and input into a locally deployed lightweight large language model. The core reason for choosing a small-volume model like Qwen3-14B is its fast inference capability, enabling low-latency responses in a local environment and avoiding network dependencies associated with cloud calls. The model performs deep semantic analysis on the user input based on the prompt constraints, generating preliminary structured output results. Next, a conflict detection and clarification response process is executed. Built-in conflict detection rules comprehensively validate the model output, focusing on detecting three types of conflicts: simultaneous inclusion of both brief and detailed introductions of the same target module; receiving skip or terminate instructions in a non-reporting state; and a mixture of start and Q&A instructions that cannot be separated. If any of these conflicts are detected or the input intent is ambiguous, a targeted clarification request is automatically generated to guide the user to clarify their needs. If the model output is a valid JSON array without logical conflicts, it is confirmed as a valid intent instruction sequence. Finally, a standard JSON format intent instruction list is output, clearly presenting the type of each intent, its target audience, and optional key keywords to ensure complete and unambiguous instruction information. This list will serve as the sole basis for subsequent process scheduling. Through structured prompts, precise inference using a local lightweight large model, and rigorous conflict detection, deep analysis and standardized conversion of user input intent are achieved, effectively avoiding erroneous operations caused by ambiguous instructions.

[0034] The core logic of step S3 is to traverse the structured intent instruction list output by step S2, following the rule that "control intents have higher priority than question-and-answer intents," and trigger corresponding operation branches for different types of intents. The core technical advantage of this invention lies in constructing a dual-engine collaborative architecture of a reporting execution engine and a question-and-answer engine. The two achieve millisecond-level dynamic switching through a state manager and an intent recognition module, and strictly ensure contextual integrity and semantic coherence throughout the switching process. Its collaborative mechanism and state synchronization logic are as follows: First, in the dual-engine collaborative architecture provided by this invention, the functions of the two engines are clearly defined: the core responsibility of the reporting execution engine is to execute various tasks in an orderly manner according to a preset structured script. This includes converting the broadcast text into speech through text-to-speech (TTS) technology for broadcasting, as well as the parallel execution of related control actions (such as UI component highlighting, page navigation, animation triggering, etc.) that are associated with the content being explained. Throughout the broadcasting process, it continuously listens to the user's voice input, always maintaining the continuity of its execution logic and the stability of its running state, without interrupting the core task due to listening operations. The question-and-answer engine is built based on search-enhanced generation (RAG) technology and is specifically designed to respond to the user's question-and-answer intent. Its core objective is to generate accurate and context-appropriate answers to the user's questions while strictly adhering to the current reporting scenario and without deviating from the reporting topic.

[0035] Secondly, the dynamic switching trigger mechanism of the dual engines uses the structured instructions output by the intent recognition module as the sole signal: when the instruction belongs to the start category or the control category (specifically including termination, skip, jump, brief introduction, detailed introduction), the state manager will immediately trigger the switching process of "question-answering engine hibernation → reporting execution engine activation", and at the same time update the global system status to "reporting in progress" to ensure the priority execution of reporting-related tasks; when the instruction is a question-answering category and does not contain any control intent, the state manager will immediately trigger the switching operation of "reporting execution engine freeze → question-answering engine activation", and at the same time update the system status to "question-answering in progress". During the freezing process of the reporting execution engine, its current running status will be fully preserved, including key information such as module index, step index, and text-to-speech (TTS) playback progress, laying the foundation for the subsequent resumption of the reporting process.

[0036] Then, as the "central hub" of the dual-engine collaboration, the state manager constructs a complete state synchronization and context preservation logic by maintaining three types of core data in real time: First, the system's global state, clearly recording the current system's "standby / reporting / Q&A" mode, fundamentally avoiding the occurrence of logic conflicts when the two engines start or switch simultaneously; second, the script running state, accurately storing the breakpoint information of the reporting execution engine, specifically including the current module ID, step index, and already broadcast text fragments, ensuring that when switching back to the reporting execution engine, the broadcast process can be seamlessly resumed from the breakpoint; third, the interaction context data, caching the core summary of the broadcast content, the user's historical questions during this report, and the system's corresponding responses, ensuring that the Q&A engine always stays on the reporting topic when generating answers, without content gaps, and also ensuring that the reporting execution engine can avoid content duplication or omissions when it resumes operation.

[0037] Finally, to ensure semantic coherence during the dual-engine switching and interaction process, this invention designs a targeted semantic coherence guarantee mechanism: During engine switching, the state manager synchronizes three types of core data—system global state, script running state, and interaction context data—to the engine to be activated in real time. The question-and-answer engine, through the synchronized "script running state + interaction context data," can accurately grasp the current module's theme and the core of the already broadcast content, ensuring that the generated answers fit the reporting scenario and do not deviate from the core topic. The reporting execution engine, with the help of the synchronized "interaction context data," can dynamically adjust the focus of subsequent broadcasts. For example, if a user has asked a question about a certain functional detail and received an answer, subsequent broadcasts can appropriately simplify that part of the content to avoid redundant explanations. At the same time, after the question-and-answer session ends, the question-and-answer engine obtains user instructions through closed-loop guiding dialogue (such as "Do you need to continue the previous introduction?"). Based on the user feedback, the state manager decides whether to switch the question-and-answer engine to a dormant state and wake up the reporting execution engine to resume broadcasting from the previous breakpoint, ultimately forming a complete semantic closed loop of "broadcast-interaction-recovery," ensuring the coherence and smoothness of the entire reporting process.

[0038] In one embodiment of the present invention, the triggering conditions and execution flow of a specific branch are as follows: Branch A: The trigger condition is that the intent instruction list contains a startup intent.

[0039] Execution process: Based on the script identifier carried in the instruction, load the corresponding structured introduction script from the preset resource library; Initialize the script index parameters, setting the current module index and step index to 0, thus specifying the starting position of the script playback; After initialization, the structured presentation script will be played sequentially to initiate the formal reporting process.

[0040] For example: Branch B: The trigger condition is that the intent instruction list contains control-type intents.

[0041] The execution process is divided according to the specific intent type: If the intention is to terminate the report: Immediately interrupt the current audio playback process; Freeze the current script's position information (including the current module ID and step index) using the state manager to prevent index confusion; The large model is invoked to generate a concluding remark that fits the scenario (such as "The presentation has ended, thank you for listening"), and then broadcast it. Release the current script resources and return to standby state.

[0042] If the intention is to skip the current module: Ignore all unexecuted steps in the current module. Update the module index and step index to the first step of the next module; If no subsequent modules are detected, the reporting process is considered complete, an end prompt is generated, and the system returns to standby status.

[0043] If the intention is to jump to the target module: Based on module metadata (such as module title and keywords), fuzzy matching is performed on the target field in the instruction to locate the target module specified by the user. Reset the step pointer of the target module to its initial value to ensure that the broadcast starts from the beginning of the module; Once the location is determined, a structured script is presented sequentially, starting the presentation from the first step of the target module.

[0044] If the intention is to provide a brief introduction to the current module: Pause the currently running voice broadcast and script execution process; Dynamically generate and explain the "brief introduction" content.

[0045] If the intention is to provide a detailed introduction to the current module: Pause the currently running voice broadcast and script execution process; Dynamically generate "detailed introduction" content and explain the content.

[0046] Branch C: The trigger condition is that the intent instruction list contains question-and-answer type intents and there are no control type intents.

[0047] Execution process: Extract the content of the user's question from the intent command, as the user's core question; Pause the current script playback process (if it is in playback mode); Enter intelligent question-and-answer mode to initiate the process of answering user questions.

[0048] The solution process includes the following steps: First, semantic analysis is performed on the extracted user questions to determine whether they fall within the coverage of the currently loaded structured introductory script. This determination is based on the relevance of the question's keywords to the script's module theme, step-by-step announcement text, and control actions. If the question focuses on the explanation content and operational logic already included in the current script, it is determined to be "within the script's coverage"; if the question involves extended information or external related knowledge not mentioned in the script, it is determined to be "outside the script's coverage".

[0049] If the question is determined to fall within the script's coverage, relevant answer information is precisely extracted from the currently loaded script. The extraction logic is as follows: matching question keywords with the broadcast text and control action descriptions of each module, filtering out the content fragments with the highest semantic relevance, and then using a large model for language optimization to ensure that the answer is concise, accurate, and fits the current reporting rhythm, avoiding the stiff expression caused by directly copying the original script text.

[0050] If the problem is beyond the scope of the script, the search enhancement generation process will be initiated, which will be executed in three steps: S01: Using the user's question as the core keyword, perform similarity matching in the preset vector knowledge base, and select the Top-N knowledge fragments with the highest relevance (the number of fragments can be configured according to actual needs) to ensure that the answer is supported by reliable external knowledge.

[0051] S02: Integrate two types of key contextual information with the retrieved knowledge fragments. The first type is the "current reporting context", which includes the current module topic and a summary of the core points of the already broadcast content, to ensure that the answer does not deviate from the current reporting scenario. The second type is the "history of this interaction", which includes the user's previous questions and the system's responses in this report, to avoid the answer from repeating or conflicting with the historical interactions.

[0052] S03: Input the integrated "retrieval knowledge fragments + two types of context + user questions" into the large model. The large model generates answers according to the rules of "logical coherence, colloquial language, and accurate information", while controlling the length and conversational level of the answers to adapt to voice broadcasting scenarios.

[0053] After generating the answer, a voice broadcast is performed. Once the broadcast is complete, a closed-loop guidance process is triggered, asking the user, "Would you like to continue the presentation?" The user's response will determine the subsequent process: if the user explicitly indicates "continue," the state recovery process is triggered, and the report continues from the previous interruption point; if the user does not respond or indicates "do not continue," the current state is maintained, awaiting new user instructions (such as asking a new question, terminating the report, etc.).

[0054] Specific examples of the three branches mentioned above are as follows: Read the text content of the current step and send it to the speech synthesis module to start speech playback; execute the control operations associated with this step in parallel (such as calling the front-end scripting language to highlight the user interface component); during speech synthesis playback, continuously execute steps S1 to S3 and listen for user interruptions.

[0055] Upon receiving a summary instruction: pause the current speech synthesis playback; append context information: Current text: The concatenation of the original text from all steps in the current module; Context: The last paragraph of the previous module (truncated to ≤100 words); Following context: The first paragraph of the next module (truncated to ≤100 words); Constructing a summary and generating prompts: "You are generating a concise introduction for an intelligent reporting system..." The large model is invoked to generate a conversational summary of 80-120 words; this summary is used as a single step to replace the entire content of the current module; after the summary is read aloud, the system automatically moves to the next module and continues with step S4.

[0056] When a detailed instruction is received: pause speech synthesis; If a focus field exists, the search criteria are "current module topic + focus" to retrieve the top three most relevant knowledge fragments from the vector knowledge base; otherwise, only the module topic is used for retrieval; context is collected: Original content: The original introductory text for the current module; Knowledge retrieval: Retrieved knowledge fragments; User questions (if any): Original user question text; Detailed explanation of the generated prompts (including character settings, length limits, consistency constraints, etc.); The large model is called to generate 200-300 words of explanatory content, which is then divided into 1-3 segments according to semantics; The messages are sequentially fed into the speech synthesis system for broadcasting. After the broadcast ends, proactively ask, "Do you want to continue the previous introduction?", and wait for the user's confirmation before deciding whether to resume the original process.

[0057] When a question-and-answer intent is identified: determine if the question falls within the coverage of currently loaded scripts; if so, extract the answer from the scripts first; otherwise, initiate the enhanced search generation process: a) Using user questions as queries, retrieve relevant document fragments from the vector knowledge base; b) Injecting two types of contexts: Current reporting context (current module topic, summary of key points already broadcast); The dialogue history of this interaction; c) Combine the search results with the context and input them into the large model to generate a response; After the answer is read aloud, a closed-loop guidance is executed: "Do you want to continue the introduction?", allowing the user to decide on the subsequent process.

[0058] This step first iterates and parses the structured intent instruction list, following a priority rule that control intents take precedence over question-and-answer intents. The core basis for this rule is that control intents directly affect the overall flow of the reporting process; operations such as termination and redirection must be executed first to ensure user experience. Question-and-answer intents can respond after the control action is completed. If the instruction list contains a startup intent, the corresponding structured introduction script is loaded based on the script identifier in the instruction. This script is pre-parsed and stored by the introduction script manager and has a clear module and step hierarchy. Simultaneously, the module index and step index are initialized, both set to initial values, preparing for the sequential playback of the structured introduction script. Then, the script sequential playback steps are automatically initiated. If the instruction list contains control intents, the corresponding operation is executed according to the specific intent type: Upon receiving a termination intent, the current text-to-speech playback process is immediately interrupted, the current script's position information is frozen through the state manager to avoid index errors, and a context-appropriate closing statement is generated by the large model. After being played by the text-to-speech module, the current script resources are released, and the system returns to standby status. Upon receiving a skip current module intent, the module index and step index are directly updated, all remaining steps of the current module are skipped, and the system locates the first step of the next module. If no subsequent modules are detected, the reporting process is considered complete, a reporting end prompt is generated, and the system returns to standby status. Upon receiving a jump module intent, based on key information such as the title in the module metadata, fuzzy matching of the target field is performed to accurately locate the user-specified target module. At the same time, the step pointer of that module is reset to the initial position to ensure that the playback starts from the module's starting position. Upon receiving a brief introduction or detailed introduction intent, the current playback process is paused, and the corresponding dynamic summary generation or extended content generation steps are entered. If the instruction list contains question-and-answer intents, the query content in the instruction is extracted as the user's question, the current playback process is paused, and a context-aware intelligent question-and-answer process is initiated. Throughout the scheduling process, clear priority rules, precise index management, and targeted branch processing enable efficient responses to various user intents, ensuring clear operational logic and orderly execution, and fully meeting users' control and interaction needs for the reporting process.

[0059] The core technologies of step S4 lie in script-level parsing, parallel execution mechanisms, and continuous monitoring strategies. The structured introduction script is loaded and parsed by the introduction script manager. The script adopts a two-level hierarchical structure of modules and steps. Each module contains multiple logically related steps, and each step is associated with corresponding broadcast text and control actions. The broadcast text is the explanation content that the user needs to hear, and the control actions are demonstration operation instructions that accompany the explanation content. According to the initialized module and step indices, the broadcast and control operations are executed cyclically. First, the broadcast text of the current step is read and sent to the text-to-speech module. This module supports speech synthesis and real-time playback, and has playback interruption and resumption functions, allowing for flexible adjustment of the playback state based on subsequent user commands. Simultaneously with starting the text-to-speech broadcast, the control actions associated with the current step are executed in parallel through front-end calling technology. Examples include calling JavaScript to highlight UI components, trigger page jumps, and start demonstration animations. This achieves deep integration of voice explanation and visual demonstration, enhancing the intuitiveness and vividness of information delivery, and meeting the multi-dimensional display needs of product demonstrations, digital museums, and other scenarios. During text-to-speech playback, the voice monitoring and intent parsing process is uninterrupted. The initialization and voice monitoring, along with the structured parsing of user input intent, are continuously repeated. User voice is captured in real-time via microphone, converted into valid text by the automatic speech recognition module, and then parsed for intent. This ensures that users can interrupt the playback at any time via voice commands, such as asking questions, requesting adjustments to the level of detail, or jumping to a different module. An interruption response mechanism is implemented: once a valid control or question-and-answer intent from the user is detected, the current text-to-speech playback and control actions are immediately paused, prioritizing the user's command. The original playback flow is then resumed based on the user's choice after the command has been executed. This process, through hierarchical management of the structured presentation script, parallel execution of playback and control actions, and continuous monitoring during playback, ensures the structured and logical nature of the presentation content, achieves synchronized voice and visual demonstrations, and grants users full interactive control. It effectively solves the existing problems of disconnected playback and demonstration and the inability to interrupt in real-time.

[0060] In step S5, upon receiving a brief introduction intent, the text-to-speech playback pause mechanism is first triggered to avoid conflict between the current broadcast and the generated summary content. Then, context information is concatenated. The original text of all steps in the current module is extracted from the state manager and integrated to ensure the summary content covers the core information of the module. Simultaneously, the last paragraph of the previous module is extracted, with a length not exceeding 100 characters, and the first paragraph of the next module is extracted, also with a length not exceeding 100 characters. This supplementation of the context between the preceding and following modules ensures that the generated summary flows naturally with the overall reporting process, avoiding content fragmentation. Next, summary generation prompts are constructed. These prompts explicitly require the large model to generate a concise introduction of 80 to 120 characters in a conversational style, emphasizing that the content should highlight core points and be logically coherent. The prompts and the concatenated context information are input into the large model to generate a summary that meets the requirements. This summary is then used as a single step to replace the entire content of the current module and sent to the text-to-speech module for broadcast. After the broadcast, no further user intervention is required; the module index and step index are automatically updated, and the broadcast process for the next module begins, ensuring a smooth reporting rhythm. Upon receiving a detailed explanation intent, the text-to-speech playback is paused. If the intent includes a focus field, the current module's topic and the focus field are combined as search criteria. If no focus field is included, only the current module's topic is used as the search criteria. A search is performed in the vector knowledge base, selecting the top three most relevant knowledge fragments to provide rich information support for the detailed explanation. Subsequently, multi-dimensional contextual information is collected, including the original introductory text of the current module, the retrieved knowledge fragments, and the user's original question text (if any), ensuring that the generated detailed explanation content is both relevant to the module's topic and specifically addresses the user's potential needs. When constructing the detailed explanation generation prompts, the large model is explicitly designated as a professional presenter, with a content length limited to 200-300 words. Consistency with the current reporting context is emphasized to avoid information conflicts, while also requiring a clear and hierarchical content structure. The collected contextual information and prompts are input into the large model. After generating the detailed explanation content, it is semantically divided into 1-3 segments and sequentially fed into the text-to-speech module for playback, ensuring clear understanding for the user. After the broadcast ends, a prompt is made to confirm whether the user needs to continue the previous introduction. Based on the user's response, a decision is made on whether to resume the original reporting process or wait for further instructions. Through this series of technical means, the content of the presentation is dynamically adapted, which not only meets the different needs of users for information depth, but also ensures the continuity of the reporting process.

[0061] When a user asks a question, the system first assesses its scope, analyzing whether it falls within the coverage of currently loaded scripts. This assessment is based on the degree of matching between the question's keywords and the script module's theme, step-by-step broadcast text, and control action descriptions. If the matching degree reaches a preset threshold, the question is considered within the script's coverage, and key information is extracted from the script to generate the answer. The advantage of this strategy is that the script content is highly relevant to the current reporting topic, ensuring consistency between the answer and the reporting context and preventing information gaps. If the question is outside the script's coverage, a search-enhanced generation process is immediately initiated. Using the user's question as the core search keyword, a comprehensive search is conducted in the vector knowledge base. This knowledge base stores a vast amount of document resources related to the reporting topic. Vector similarity calculations are used to filter out the most relevant document fragments, providing sufficient knowledge support for the answer. Subsequently, two types of key contextual information are injected: one is the current reporting context, including the current module theme and a summary of the core points of already broadcast content, ensuring the answer fits the current reporting scenario; the other is the dialogue history of this interaction, avoiding duplicate answers or conflicts with previous interactions. The retrieved relevant fragments, injected contextual information, and user questions are integrated and pieced together to construct complete input content. This input is then fed into a large model to generate an answer. During the generation process, the large model adheres to the principles of logical coherence, information accuracy, and plain language to ensure user comprehension. After the answer is read aloud by the text-to-speech module, a closed-loop guidance operation is executed, proactively asking the user if they wish to continue the presentation. This guidance mechanism smoothly pulls the user back from the question-and-answer scenario to the reporting scenario, avoiding interruptions after the question-and-answer session ends. It empowers users to choose subsequent actions, improving the user experience while ensuring the integrity of the reporting process, effectively solving the problem of disconnect between existing question-and-answer and reporting mechanisms.

[0062] In the final step S6, after all interrupted operations, including Q&A interactions and control operations, are completed, the previous broadcast flow will not be automatically resumed. Instead, the current state will be maintained, awaiting further user instructions. This design fully respects the user's operational intentions and avoids the inconvenience caused by forced resumption of broadcasting. Throughout the entire operation, the state manager continuously maintains three types of core information: first, the system status, including the three states of standby reporting, Q&A, and so on, ensuring accurate identification of the current running stage; second, the breakpoint information of the current script, including the module ID and step index at the time of interruption, enabling precise flow recovery through accurate recording; and third, cached data of the already broadcast content, providing a basis for generating personalized closing remarks. If the user explicitly confirms to continue the previous introduction, the state manager immediately calls the stored breakpoint information, accurately restoring the module index and step index to the position before the interruption, and continuing to execute the structured introduction script's sequential broadcast steps at that position, ensuring a seamless reporting flow and avoiding content duplication or omissions. If no user response is received within a preset time, or if the user explicitly issues a termination instruction, the large model is invoked to generate a personalized closing remarks. The closing remarks combine the cached data of the already broadcast content for a brief summary, while also expressing gratitude, enhancing the user experience. After the closing remarks are broadcast, the currently loaded script resources are released, breakpoint information and cached data in the state manager are cleared, and the system returns to a standby state, awaiting the user's next startup command. This step effectively solves the problems of unrecoverable or chaotic recovery after interruption through precise information maintenance in the state manager, a user-confirmed recovery mechanism, and personalized termination handling. It achieves a complete closed loop in the reporting process, ensuring the continuity and flexibility of user interaction.

[0063] Second Embodiment Based on the above method, the second embodiment of the present invention provides an intelligent reporting system based on a large language model and digital human interaction. For example... Figure 2 As shown, the intelligent reporting system includes one or more processors and a memory. The memory is coupled to the processor and is used to store one or more programs. When the program is executed by the processor, the intelligent reporting method described in the above embodiments can be implemented.

[0064] The processor controls the overall operation of the intelligent reporting system to complete all or part of the steps described above. The processor can be a central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), digital signal processing (DSP) chip, etc. The memory stores various types of data to support the operation of the intelligent reporting system. This data may include, for example, instructions for any application or method operating on the intelligent reporting system, as well as application-related data. The memory can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, etc.

[0065] In one exemplary embodiment, the intelligent reporting system may be implemented by a computer chip or physical entity, or by a product with certain functions, to perform the methods described above and achieve the same technical effects. A typical embodiment is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interface device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0066] In another exemplary embodiment, the present invention also provides a computer-readable storage medium including program instructions that, when executed by a processor, implement the steps of the method in any of the above embodiments. For example, the computer-readable storage medium may be the memory including the program instructions described above, which can be executed by a processor to complete the above method and achieve the same technical effects as the above method.

[0067] It should be noted that the above embodiments are merely illustrative examples. The technical solutions of each embodiment can be combined, and all are within the protection scope of this invention.

[0068] The above provides a detailed description of the intelligent reporting method and system based on a large language model and digital human interaction provided by this invention. Any obvious modifications made by those skilled in the art without departing from the essence of this invention will constitute an infringement of the patent rights of this invention and will incur corresponding legal liability.

Claims

1. An intelligent reporting method based on a large language model and digital human interaction, characterized in that... Includes the following steps: S1: Continuously perform streaming speech transcription through a locally deployed automatic speech recognition module, and output valid text segments after sentence segmentation detection and silence filtering; S2: Input the valid text fragment into the intent recognition module. The intent recognition module constructs structured prompt words and calls a local lightweight large language model for inference, and outputs a structured intent instruction list. The intent types included in the structured intent instruction list are limited to start reporting, terminate reporting, skip the current module, briefly introduce the current module, introduce the current module in detail, and ask a question. S3: Traverse the structured intent instruction list and schedule its execution according to the rule that control intents take precedence over question-and-answer intents; S4: If the start report or continue report is executed, the broadcast text and associated control action of the current step are read from the structured introduction script according to the current module index and step index maintained by the state manager, and the voice broadcast and the control action are executed in parallel. During this process, the execution steps S1 and S2 are continuously returned to listen for the user's interrupt command.

2. The intelligent reporting method as described in claim 1, characterized in that... Step S2 includes the following sub-steps: S21: Construct a structured prompt template; S22: Call the local lightweight large language model for inference; S23: Perform a comprehensive validation of the structured output generated by the large model; if the output is a valid JSON array and there are no conflicts, it is determined to be a valid intent instruction sequence; if the output is conflicting or the user's input intent is ambiguous, it is determined to be a conflict state and a targeted clarification request instruction is automatically generated. S24: Output structured intent instructions.

3. The intelligent reporting method as described in claim 1, characterized in that... The multi-intent instruction branch in step S3 includes: Branch A: The trigger condition is that the intent instruction list contains a startup intent; Branch B: The trigger condition is that the intent instruction list contains control intents; Branch C: The trigger condition is that the intent instruction list contains question-and-answer type intents and there are no control type intents.

4. The intelligent reporting method as described in claim 3, characterized in that... When branch C is triggered, the intelligent question-and-answer mode will be entered, and the process of answering the user's question will be initiated. The solution process includes the following steps: First, semantic analysis is performed on the extracted user questions to determine whether they fall within the coverage of the currently loaded structured introduction scripts; If the answer falls within the scope of the script, then the relevant answer information will be accurately extracted from the currently loaded script. If the scope of the script is exceeded, the search enhancement generation process will be initiated.

5. The intelligent reporting method as described in claim 4, characterized in that... The search enhancement generation process is executed in three steps: S01: Using user questions as the core search keywords, perform similarity matching in a preset vector knowledge base, and select the Top-N knowledge fragments with the highest relevance; S02: Integrate two types of key contextual information with the retrieved knowledge fragments. The first type includes the current module topic and a summary of the core points of the previously broadcast content to ensure that the answer does not deviate from the current reporting scenario. The second type includes the user's previous questions and the system's responses during this report. S03: Input the integrated retrieval knowledge fragments, two types of context, and user questions into the large model. The large model generates answers according to preset rules, while controlling the length and conversational level of the answers to adapt to voice broadcasting scenarios.

6. The intelligent reporting method as described in claim 5, characterized in that... After generating an answer, the system will broadcast it via voice. Once the broadcast is complete, the system can resume operation or wait for new instructions, depending on the user's needs.

7. The intelligent reporting method as described in claim 1, characterized in that... The process after step S4 also includes: S5: If a control intent is detected during the broadcast, the current broadcast is paused, and the operation corresponding to the control intent is executed; if the intent is a brief introduction or a detailed introduction, a summary or detailed explanation is dynamically generated and broadcast based on the original text, context fragments, and relevant knowledge retrieved from the vector knowledge base; if the intent is a question-and-answer session, the answer is obtained and broadcast by combining the current reporting context and the dialogue history of this interaction through a retrieval enhancement generation process. S6: After the broadcast in step S5 is completed, ask the user whether to continue the original reporting process; if the user confirms to continue, the state manager restores the module index and step index at the time of the interruption according to the breakpoint information it recorded, and returns to step S4 to continue execution; otherwise, generate a closing statement and return to the standby state.

8. The intelligent reporting method as described in claim 1, characterized in that... The structured introduction script is a hierarchical structure containing multiple modules; each module contains multiple steps that are executed sequentially; each step is associated with a broadcast text and at least one control action.

9. An intelligent reporting system based on a large language model and digital human interaction, characterized in that... It includes a processor and a memory; wherein the memory is coupled to the processor and is used to store a computer program, which, when executed by the processor, causes the processor to implement the intelligent reporting method according to any one of claims 1 to 8.

Images