A multi-modal digital human interaction method and system based on a large language model

Abstract

The application discloses a kind of multi-modal digital human interaction method and system based on large language model, including the following steps: collecting voice, vision, text and business event data, generate multi-modal time axis cache;Multi-modal time axis cache is preprocessed, and dialogue state information is generated;Knowledge base and business system interface data retrieval enhancement are obtained, and evidence set is distributed with evidence identification;According to output strategy rule set and combined with evidence threshold, risk threshold and self-consistency score gate, if it is not satisfied, generate clarification interaction instruction and update dialogue state;Improved GPT-4o model is called, and structured interaction plan is output;According to plan, execute limited tool call and digital human driving presentation and record version.The application realizes multi-modal consistent understanding, evidence traceable and controllable digital human interaction, significantly improves reply accuracy, compliance and auditability, and is suitable for customer service, government affairs, enterprise knowledge assistant and other scenes.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and human-computer interaction technology, and in particular to a multimodal digital human interaction method and system based on a large language model. Background Technology

[0002] With the development of large language models and multimodal interaction technologies, digital human interaction solutions driven by multi-source information such as voice, text, and images are gradually increasing. Existing technologies have attempted to utilize large language models to process multimodal dialogue information. For example, patent application CN120296120A (publication date: July 8, 2025) discloses a multimodal dialogue processing method, device, electronic device, and storage medium. This method acquires multimodal dialogue information input by the user, combines it with role setting information and historical dialogue information to determine the dialogue state information, and then inputs it into the large language model to generate target dialogue response information, thereby enhancing the freedom and immersion of multimodal dialogue. Another example is patent CN119782470B (publication date: July 25, 2025), which discloses a retrieval-enhanced large language model generation method. This method improves the accuracy of the large language model in answering complex questions through similar document retrieval, knowledge graph conversion, sub-question decomposition, and prompt generation. For example, patent application CN121118962A (publication date: December 12, 2025) discloses a digital human interaction method and system based on a large model. It generates contextual information by acquiring multimodal instructions and combining them with the basic settings and historical information of the digital human. Then, it adjusts the initial text response according to the emotional state and personality traits to achieve continuous learning and growth of the digital human.

[0003] However, while the aforementioned existing solutions have made improvements from the perspectives of multimodal dialogue generation, retrieval enhancement, and digital human growth-oriented interaction, their technical focus is mostly concentrated on input understanding, knowledge supplementation, or stylistic response adjustment. They have not yet formed a complete solution for the reliable generation and secure execution of digital humans in real business interaction scenarios. Specifically, existing technologies typically do not establish a cross-modal association mechanism for voice data, visual data, text data, and business event data under a unified time benchmark. They mostly process multimodal input as a whole or combine historical information to generate context, making it difficult to ensure that different modal information stably belongs to the same interaction round. This can easily lead to temporal mismatches and semantic breaks when users speak while operating the page, uploading screenshots, or triggering business events.

[0004] Furthermore, existing technologies for controlling model output often remain at a single dimension. For example, retrieval enhancement solutions mainly address the questions of "whether relevant knowledge is retrieved" and "how to use retrieval results to improve the quality of answers," while digital human interaction solutions mainly address issues of emotional expression, personalized responses, or multimodal presentation. However, none of these solutions disclose a mechanism for jointly determining the sufficiency of evidence, the state of business risk, and the stability of model output. Therefore, in cases where evidence is insufficient but the response appears responsive, or where the risk is high but the response still follows a freely generated path, issues such as unfounded assertions, unauthorized promises, or unstable responses may still arise.

[0005] Furthermore, even when existing technologies incorporate external knowledge or retrieval results, they primarily provide relevant content as input prompts to the large language model. They lack the technical means to further transform evidence and risk states into constraint signals for the model's internal representation process. They also lack restricted structured output, tool call verification, and auditable recording mechanisms to complement digital human business execution. Consequently, model output remains primarily free text, making it difficult to balance interpretability of responses, verifiability of execution, and traceability of processes when dealing with business actions such as queries, processing, recommendations, and transfers.

[0006] In summary, existing technologies still fail to address the following technical issues: how to achieve unified timeline alignment for voice data, visual data, text data, and business event data during multimodal digital human interaction; how to implement joint gating of comprehensive evidence compliance status, risk compliance status, and output consistency before generation; and how to apply evidence constraints and risk constraints to the model generation process to form verifiable, executable, and auditable interaction results.

[0007] Therefore, how to provide a multimodal digital human interaction method and system based on a large language model is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0008] One objective of this invention is to propose a multimodal digital human interaction method and system based on a large language model. This invention fully utilizes multimodal timeline caching, intent classification and slot extraction, risk label classification, retrieval of enhanced evidence sets and self-consistency gating, and an improved GPT-4o restricted structured output and tool whitelist mechanism to complete the entire interaction process from multimodal acquisition and alignment, dialogue state generation, evidence retrieval to structured interaction plan generation and restricted tool call-driven presentation. It achieves traceable responses, controllable risks, and auditable tool calls, and has the advantages of high accuracy, strong compliance, and good stability.

[0009] A multimodal digital human interaction method based on a large language model according to an embodiment of the present invention includes the following steps: Step 1: Collect voice data, visual data, text data, and business event data, perform cross-modal timeline alignment, and generate a multimodal timeline cache; Step 2: Preprocess the multimodal timeline cache, obtain transcribed text based on speech segments, and perform intent classification, slot extraction, and risk label classification on the transcribed text and text data to generate dialogue state information; Step 3: Perform retrieval enhancement on the data returned by the knowledge base and business system interfaces based on the dialogue state information, obtain the evidence set, and assign evidence identifiers to the evidence; Step 4: Generate an output strategy based on the interaction output strategy rule set, and perform triple gating based on the evidence threshold, risk threshold and self-consistency score. If any gating is not satisfied, generate a clarification interaction instruction and update the dialogue state information. If the triple gating is satisfied, proceed to planning and generation. Step 5: Invoke the improved GPT-4o model, which introduces an evidence-risk condition modulation module and a restricted structured output module. The inputs are multimodal timeline cache summary, dialogue state information, evidence set, tool whitelist and compliance policy, and the output is a structured interaction plan. Step Six: Based on the structured interaction plan, execute restricted tool calls and digital human-driven presentation, generate the final response content, and form a version record of strategy and model configuration.

[0010] Optionally, step one specifically includes: Collect voice data, visual data, text data, and business event data; Perform Voice Activity Detection (VAD) on the speech data to detect speech activity intervals, generate speech segments using the silence intervals between adjacent speech activity intervals as segment boundaries, and record the start and end timestamps for each speech segment. Candidate frames are extracted from visual data at a preset sampling interval. The pixel difference between adjacent candidate frames is calculated and compared with the difference threshold. When the difference reaches the preset difference threshold, the scene change point is determined and the corresponding frame is recorded as a key frame to form a key frame sequence. A timestamp is recorded for each key frame and the frame interval between adjacent key frames is used as a visual segment. Business event data is encoded in the order of event occurrence, event types are mapped to event identifiers, and event parameters are serialized into parameter fields to form an event sequence. An event timestamp is recorded for each event. Based on a unified clock, timestamps are aligned for audio segments, visual segments, text data, and event sequences. Audio segment identifiers, keyframe identifiers, text segment identifiers, and event identifiers within the same time window are written into the same alignment record. The alignment records are stored in chronological order and indexed to generate a multimodal timeline cache.

[0011] Optionally, step two specifically includes: Preprocess the multimodal timeline cache, and read the voice segment, text data, and business event data aligned with the timestamp of the voice segment corresponding to the current interaction round; Automatic speech recognition (ASR) is performed on the speech segment to obtain the transcribed text, and the transcribed text is associated with and stored with the start and end timestamps of the speech segment; Perform rule-based text processing on transcribed text and text data; Intent categories are obtained by matching intents based on a pre-defined intent keyword dictionary; Based on the preset slot regularization rules, a set of key-value pairs consisting of slot names and slot values ​​is extracted, and a set of key slots is determined based on a pre-established mapping table between intent categories and required slot names. Risk tags are obtained by risk matching based on a pre-defined risk keyword dictionary; The intent category, key slot set, and risk label are combined to generate dialogue state information, which is then written to the multimodal timeline cache.

[0012] Optionally, step three specifically includes: Build search queries based on dialogue status information; Perform retrieval enhancements on the data returned by the interface between the pre-built knowledge base and the business system; Vector embedding retrieval and BM25 keyword retrieval are performed on the knowledge base. Vector embedding retrieval involves vectorizing the search query and retrieving similar entries from the vector index of the knowledge base. BM25 keyword retrieval involves segmenting the search query into keywords and retrieving matching entries from the inverted index of the knowledge base. The vector embedding retrieval results and the BM25 keyword retrieval results are deduplicated and merged, and a candidate evidence list is generated according to the fusion sorting rules. A preset number of items are selected from the candidate evidence list as an evidence set, and an evidence identifier is assigned to each item in the evidence set. The evidence identifier is associated with the evidence source identifier and the evidence content fragment and written into the evidence index.

[0013] Optionally, step four specifically includes: Output strategies are generated based on a pre-configured set of interactive output strategy rules. The evidence set is calculated to meet the standard status. The confidence markers of each piece of evidence in the evidence set are summarized and the number of evidence items that meet the evidence threshold is counted. When the number of evidence items that meet the evidence threshold reaches the preset number, the evidence is judged to meet the standard. Otherwise, the evidence is judged to be unqualified. Determine the risk compliance status for the risk labels in the dialogue status information, map the risk labels to risk level identifiers, and compare the risk level identifiers with risk thresholds. If the risk thresholds are met, the risk is deemed to be compliant; otherwise, the risk is deemed not compliant. The self-consistency score is calculated under the condition that the evidence meets the standard and the risk meets the standard. The self-consistency score is the proportion of key fields that are consistent in the structured candidate interaction plans generated for the same input under a preset number of sampling N. Gating is performed based on the evidence compliance status, risk compliance status, and self-consistency score. When any one of them fails to meet the gating condition corresponding to the output strategy, a clarification interaction instruction is generated and the dialogue status information is updated. When all gating conditions are met, the planning generation process begins.

[0014] Optionally, the improved GPT-4o model includes a multimodal input encoding module, a multimodal Transformer backbone inference module, an evidence-risk condition modulation module, and a restricted structured output module: The multimodal input encoding module concatenates the multimodal timeline cache summary, dialogue state information, evidence set, tool whitelist and compliance policy into fields and serializes them into an input sequence. The input sequence is segmented and word identifiers are generated. The word identifiers are mapped into vector representations and added element by element to the position encoding vector to obtain the input embedding sequence. The multimodal Transformer backbone inference module includes a self-attention layer and a feedforward layer. It calculates attention weights and context representations layer by layer for the input embedding sequence. The calculation of attention weights includes performing a dot product on the query vector and the key vector to obtain a relevance score, scaling the relevance score by a scaling factor and performing exponential operation and normalization to obtain weight coefficients, performing a weighted summation of the weight coefficients and the value vector to obtain the attention output, and adding the attention output to the representation of the previous layer and then normalizing it to obtain the backbone hidden state sequence. The evidence-risk condition modulation module constructs an evidence state vector based on the evidence set and a risk state vector based on the dialogue state information. The evidence state vector and the risk state vector are concatenated to obtain a condition vector. The condition vector is calculated by a multilayer perceptron to obtain a scaling vector and a bias vector. The module then performs element-wise multiplication and element-wise addition on the backbone hidden state sequence to complete the condition modulation and obtain the modulated hidden state sequence. The restricted structured output module generates a structured interaction plan by performing restricted decoding on the modulated hidden state sequence based on a preset set of structured fields. The restricted decoding includes generating output tokens field by field in order and calculating token score vectors at each decoding step. Suppression processing is performed on token scores that do not meet the preset field type constraints. Masking processing is performed on tool identifier token scores that do not belong to the tool whitelist. Tokens are generated based on the probability of the remaining tokens until the structured interaction plan is completed.

[0015] Optionally, step six specifically includes: The structured interaction plan was analyzed to obtain the output type identifier, tool call plan, digital human driving parameters, and the binding relationship between key conclusions and evidence identifiers; Restricted tool calls are executed according to the tool call plan. The restricted tool calls include matching and verifying the tool identifier in the tool call plan with the tool whitelist, verifying the consistency of the parameter fields in the tool call plan with the set of tool parameter fields corresponding to the tool whitelist, and calling the corresponding business system open interface to obtain the data returned by the business system interface and writing the data returned by the business system interface into the evidence set when the verification is successful. In restricted tool calls, when the tool identifier is not in the tool whitelist or the parameter fields do not meet the tool parameter field set, the output type identifier is set to clarification and a clarification interaction instruction is generated. When the business system's open interface returns failure, the output type identifier is set to refuse to answer or transfer to human agent and the corresponding interaction instruction is generated. The final response content is generated based on the binding relationship between the output type identifier and the key conclusions and evidence identifiers; Based on the digital human driving parameters, the digital human-driven presentation is executed. The audio stream output by the speech synthesis TTS and the facial expression and action parameters are synchronously driven to drive the digital human rendering engine. The version records of the structured interaction plan, the final response content, the tool call plan and the tool call result formation strategy and model configuration are also recorded.

[0016] A multimodal digital human interaction system based on a large language model according to an embodiment of the present invention includes the following modules: The timeline caching module is used to collect voice data, visual data, text data, and business event data, and perform cross-modal time correlation based on a unified time base to generate a multimodal timeline cache. The state generation module, connected to the timeline cache module, is used to extract the multimodal context information corresponding to the current interaction round based on the multimodal timeline cache and generate dialogue state information; The evidence retrieval module, connected to the state generation module, is used to perform evidence retrieval and association on knowledge base data and business system data based on dialogue state information, and obtain evidence sets and evidence identifiers. The strategy gating module is connected to the state generation module and the evidence retrieval module respectively. It is used to determine the evidence compliance status, risk compliance status and self-consistency score based on the dialogue state information and evidence set, execute triple gating, and output clarification instructions, refusal instructions or manual instructions when the gating is not satisfied. The plan generation module is connected to the evidence retrieval module and the policy gating module respectively. It is used to call the improved GPT-4o model when the gating is satisfied, and generate a structured interaction plan based on the multimodal timeline cache summary, dialogue state information, evidence set, tool constraint information and compliance policy. The execution and recording module, connected to the plan generation module, is used to generate the final response content and record version information of the interaction process based on the structured interaction plan execution restricted tool call and digital human-driven presentation.

[0017] The beneficial effects of this invention are: This invention addresses the technical problems in existing technologies, such as the difficulty in consistently attributing multimodal information to the same interaction round, semantic mismatch, and context breakage, by constructing a multimodal timeline cache and performing correlation processing on voice data, visual data, text data, and business event data under a unified time benchmark. As demonstrated in the embodiments, this invention improves the first-time resolution rate and reduces the average number of interaction rounds, thereby enhancing processing efficiency in multi-round interaction scenarios. Furthermore, by constructing an evidence set based on dialogue state information and assigning evidence identifiers to each evidence item, this invention solves the technical problems in existing technologies, such as the lack of external evidence binding for response content, untraceable generation process, and susceptibility to factual bias and unfounded assertions. As demonstrated in the embodiments, this invention reduces the rate of factual errors and unfounded assertions, thereby enhancing the factual constraint capability and evidence traceability of response content. Finally, by introducing a triple gating mechanism combining evidence compliance status, risk compliance status, and self-consistency score, this invention addresses the problem that existing technologies rely solely on a single rule or search result for control, which may still result in errors even when evidence is insufficient, risk is increased, or model output is unstable. Technical problems related to erroneous conclusions or unauthorized commitments; as shown in the embodiments, this invention can improve the accuracy of risk handling and maintain good autonomous processing capabilities while ensuring security; this invention generates structured interaction plans by using an improved GPT-4o model that includes an evidence-risk condition modulation module and a restricted structured output module, and executes restricted tool calls by combining a tool whitelist and parameter consistency verification, thus solving the technical problems in the prior art of unstable parsing of free text output, easy accidental triggering of tool calls, and inconsistency between response content and interface results. As shown in the embodiments, this invention can reduce the rate of accidental tool calls and improve user satisfaction, thereby improving the executability and call consistency of interaction plans; this invention solves the technical problems in the prior art of lacking versioned traces of the interaction process, difficulty in problem localization, and high subsequent optimization costs by recording structured interaction plans, final response content, tool call results, and strategy configuration and model configuration version information. As shown in the embodiments, this invention can still achieve a synergistic improvement in accuracy, security, and engineering maintainability with limited computational overhead.

[0018] In summary, this invention achieves comprehensive technical effects such as more accurate responses, more controllable risks, more verifiable results, and more traceable processes through the synergistic effects of multimodal timeline alignment, evidence binding, triple gating, conditional modulation, and restricted execution. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a multimodal digital human interaction method based on a large language model proposed in this invention; Figure 2 This is a schematic diagram of the improved GPT-4o model in a multimodal digital human interaction method based on a large language model proposed in this invention; Figure 3 This is a diagram illustrating the overall framework of a multimodal digital human interaction system based on a large language model proposed in this invention. Detailed Implementation

[0020] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0021] refer to Figures 1-2 A multimodal digital human interaction method based on a large language model includes the following steps: Step 1: Collect voice data, visual data, text data, and business event data, perform cross-modal timeline alignment, and generate a multimodal timeline cache; Step 2: Preprocess the multimodal timeline cache, obtain transcribed text based on speech segments, and perform intent classification, slot extraction, and risk label classification on the transcribed text and text data to generate dialogue state information; Step 3: Perform retrieval enhancement on the data returned by the knowledge base and business system interfaces based on the dialogue state information, obtain the evidence set, and assign evidence identifiers to the evidence; Step 4: Generate an output strategy based on the interaction output strategy rule set, and perform triple gating based on the evidence threshold, risk threshold and self-consistency score. If any gating is not satisfied, generate a clarification interaction instruction and update the dialogue state information. If the triple gating is satisfied, proceed to planning and generation. Step 5: Invoke the improved GPT-4o model, which introduces an evidence-risk condition modulation module and a restricted structured output module. The inputs are multimodal timeline cache summary, dialogue state information, evidence set, tool whitelist and compliance policy, and the output is a structured interaction plan. Step Six: Based on the structured interaction plan, execute restricted tool calls and digital human-driven presentation, generate the final response content, and form a version record of strategy and model configuration.

[0022] In this embodiment, step one specifically includes: Collect voice data, visual data, text data, and business event data. Voice data is audio signals collected by the microphone, visual data is image or video sequences, text data is text character sequences input by the user, and business event data is interactive event records generated by the front end of the business system or the open interface of the business system, including event type, event parameters, and event timestamp. Perform Voice Activity Detection (VAD) on the speech data to detect speech activity intervals, generate speech segments using the silence intervals between adjacent speech activity intervals as segment boundaries, and record the start and end timestamps for each speech segment. Candidate frames are extracted from visual data at a preset sampling interval. The pixel difference between adjacent candidate frames is calculated and compared with the difference threshold. When the difference reaches the preset difference threshold, the scene change point is determined and the corresponding frame is recorded as a key frame to form a key frame sequence. A timestamp is recorded for each key frame and the frame interval between adjacent key frames is used as a visual segment. Business event data is encoded in the order of event occurrence, event types are mapped to event identifiers, and event parameters are serialized into parameter fields to form an event sequence. An event timestamp is recorded for each event. Based on a unified clock, timestamps are aligned for audio segments, visual segments, text data, and event sequences. Audio segment identifiers, keyframe identifiers, text segment identifiers, and event identifiers within the same time window are written into the same alignment record. The alignment records are stored in chronological order and indexed to generate a multimodal timeline cache.

[0023] By writing voice data, visual data, text data, and business event data into time-aligned records according to a unified time base, the semantic mismatch and context breakage problems caused by the rough splicing of multimodal information in the order of reception in existing technologies can be avoided, thereby providing a stable and consistent temporal context for the generation of subsequent dialogue states.

[0024] In this embodiment, step two specifically includes: Preprocess the multimodal timeline cache, and read the voice segment, text data, and business event data aligned with the timestamp of the voice segment corresponding to the current interaction round; Automatic speech recognition (ASR) is performed on the speech segment to obtain the transcribed text, and the transcribed text is associated with and stored with the start and end timestamps of the speech segment; Perform rule-based text processing on transcribed text and text data; Intent categories are obtained by performing intent matching based on a preset intent keyword dictionary, where the intent keyword dictionary is a mapping table consisting of intent tags and keyword sets; Based on preset slot regular expression rules, a set of key-value pairs consisting of slot names and slot values ​​is extracted. The slot regular expression rules are a set of preset regular expressions for each slot name. The key slot set is determined based on a pre-established mapping table between intent categories and required slot names. The key slot set is a set of slot key-value pairs filtered from the key-value pair set. The filtering rules are as follows: add key-value pairs that match the required slot names corresponding to intent categories to the key slot set; add key-value pairs whose event parameter field names in the business event data are the same as the slot names and whose values ​​are not empty to the key slot set; and when there are slots with the same name, prioritize key-value pairs from the business event data. If there are no slots with the same name, select key-value pairs from the business event data. If there are no slots with the same name, select key-value pairs with the most recent timestamps to add to the key slot set. Risk tags are obtained by risk matching based on a preset risk keyword dictionary, where the risk keyword dictionary is a mapping table consisting of risk tags and keyword sets; The intent category, key slot set, and risk label are combined to generate dialogue state information, which is then written to the multimodal timeline cache.

[0025] In this embodiment, step three specifically includes: The retrieval query is constructed based on the dialogue state information. The retrieval query is composed of the intent category, the slot name and slot value in the key slot set, and the event type in the business event data. The system performs retrieval enhancement on the data returned by the pre-built knowledge base and the business system interface. The knowledge base is a data set that stores business rules, business processes and question and answer entries. The data returned by the business system interface is business status or business record data related to the current session obtained through the open interface of the business system. Vector embedding retrieval and BM25 keyword retrieval are performed on the knowledge base. Vector embedding retrieval vectorizes the search query and retrieves similar entries in the knowledge base vector index, while BM25 keyword retrieval performs keyword segmentation on the search query and retrieves matching entries in the knowledge base inverted index. The vector embedding search results and the BM25 keyword search results are deduplicated and merged, and a candidate evidence list is generated according to the fusion ranking rules. The fusion ranking rules are to take the ranking of the two types of searches for the same entry and calculate the fusion ranking. The candidate evidence list is obtained by sorting the fusion ranking from smallest to largest. A preset number of items are selected from the candidate evidence list to form an evidence set, and an evidence identifier is assigned to each item in the evidence set. The evidence identifier is associated with the evidence source identifier and the evidence content fragment and written into the evidence index.

[0026] In this embodiment, step four specifically includes: Output strategies are generated based on a pre-configured set of interactive output strategy rules. The set of interactive output strategy rules is a collection of rules consisting of strategy items. The strategy items include evidence threshold, risk threshold, clarification rules, refusal to answer rules, referral to human assistant rules, and tool call rules. Trigger conditions and corresponding output types are configured for each strategy item. The evidence set is calculated to meet the standard status. The confidence markers of each piece of evidence in the evidence set are summarized and the number of evidence items that meet the evidence threshold is counted. When the number of evidence items that meet the evidence threshold reaches the preset number, the evidence is judged to meet the standard. Otherwise, the evidence is judged to be unqualified. The confidence marker is the credibility level identifier of the evidence in the data returned by the knowledge base or business system interface. Determine the risk compliance status for the risk labels in the dialogue status information, map the risk labels to risk level identifiers, and compare the risk level identifiers with risk thresholds. If the risk thresholds are met, the risk is deemed to be compliant; otherwise, the risk is deemed not compliant. Under the conditions that the evidence meets the standard and the risk meets the standard, the self-consistency score is calculated. The self-consistency score is the proportion of key fields that are consistent in the structured candidate interaction plans generated for the same input under the preset number of sampling N. The preset number of sampling N is an integer greater than or equal to two. The structured candidate interaction plan is the interaction plan output according to the preset structured fields. The key fields include the output type identifier and the tool call identifier. Gating is performed based on the evidence compliance status, risk compliance status, and self-consistency score. When any of the gating conditions corresponding to the output strategy are not met, a clarification interaction instruction is generated and the dialogue status information is updated. The update includes writing the name of the slot to be supplemented into the dialogue status information and setting the output type identifier to clarification. When all the gating conditions are met, the planning generation process begins.

[0027] By combining evidence compliance status, risk compliance status, and self-consistency score for interactive decision-making, unlike existing technologies that rely solely on single risk interception or single evidence hit judgment, unreliable responses can be blocked in advance when evidence is insufficient, risk increases, or output is unstable. In this embodiment, the self-consistency score is used to measure the stability of the structured interaction plan output by the model under the same input conditions. Suppose N independent samplings are performed on the same input, resulting in N structured candidate interaction plans P1, P2, ..., PN. A set of key fields F is extracted from each candidate interaction plan, where the key fields include at least the output type identifier and the tool call identifier. For each key field fk, the proportion of the field value with the highest frequency in the N sampling results is calculated and denoted as uk. The self-consistency score Scons can then be expressed as: Scons = (1 / K)Σuk, where K is the total number of key fields. Regarding the value of N, this invention comprehensively considers output stability and online interaction latency: when N is too small, although the computational cost is low, it is more sensitive to accidental fluctuations and difficult to effectively identify whether there are real disagreements in the model; when N is too large, although the consistency judgment is more thorough, it will significantly increase the inference latency, which is not conducive to real-time interaction of the digital human. Therefore, in this invention, N is set to an integer greater than or equal to 2, and preferably 3 to 5. When N is 3, it can effectively distinguish between stable output and fluctuating output with lower computational overhead, which is more suitable for scenarios with high real-time requirements such as customer service and government affairs guidance. Therefore, in the preferred embodiment, N=3 is used, and the consistency ratio threshold is set to 0.67 to achieve a better balance between real-time performance and reliability.

[0028] In this embodiment, the improved GPT-4o model includes a multimodal input encoding module, a multimodal Transformer backbone inference module, an evidence-risk condition modulation module, and a restricted structured output module: The multimodal input encoding module concatenates the multimodal timeline cache summary, dialogue state information, evidence set, tool whitelist, and compliance policy into fields and serializes them sequentially into an input sequence. The input sequence is segmented and word identifiers are generated. The word identifiers are mapped to vector representations and added element-wise to the position encoding vector to obtain the input embedding sequence. The position encoding vector is a vector determined by the word position index. The multimodal timeline cache summary is a serialized summary of the current interaction round window data extracted from the multimodal timeline cache. The serialized summary includes the event sequence of transcribed text, text data, business event data, and keyframe identifiers of visual data. The tool whitelist is a list of identifiers of a pre-configured set of callable tools and is mapped to the tool parameter field set. The compliance policy is a pre-configured set of rules and includes sensitive content category identifiers, risk handling action identifiers, and output restriction field sets. The multimodal Transformer backbone inference module contains a self-attention layer and a feedforward layer. It calculates attention weights and context representations layer by layer for the input embedding sequence. The calculation of attention weights includes performing a dot product on the query vector and the key vector to obtain a relevance score, scaling the relevance score by a scaling factor and performing exponential and normalization operations to obtain weight coefficients, performing a weighted summation of the weight coefficients and the value vector to obtain the attention output, and adding the attention output to the representation of the previous layer and then normalizing it to obtain the backbone hidden state sequence. The evidence-risk condition modulation module constructs an evidence state vector based on the evidence set and a risk state vector based on the dialogue state information. The evidence state vector and the risk state vector are concatenated to obtain a condition vector. The condition vector is then processed by a multilayer perceptron to obtain a scaling vector and a bias vector. Element-wise multiplication and element-wise addition are performed on the backbone hidden state sequence to complete condition modulation. Element-wise multiplication involves scaling the backbone hidden state sequence channel by channel with the scaling vector, and element-wise addition involves superimposing the bias vector onto the scaled hidden state channel by channel to obtain the modulated hidden state sequence. The specific method for constructing the evidence state vector is as follows: Let the evidence set be E={e1,e2,...,em}, where each piece of evidence ei corresponds to at least evidence confidence ci, relevance score ri, timeliness score ti, and source credibility level si. Normalize the above items to the interval between 0 and 1, then the comprehensive evidence score qi of the i-th evidence can be expressed as: qi=λ1·ci+λ2·ri+λ3·ti+λ4·si, where λ1, λ2, λ3, and λ4 are weight coefficients, and satisfy λ1+λ2+λ3+λ4=1. Based on the comprehensive evidence score of each evidence item, construct the evidence state vector Ee=[mean(qi),max(qi),min(qi),count(qi≥θe) / m], where θe is the evidence qualification threshold, and m is the total number of evidence items.

[0029] The specific method for constructing the risk state vector is as follows: Let the risk label set be R={r1,r2,...,rn}. Each risk label rj corresponds to at least a risk level value gj, a sensitive field hit marker hj, and a treatment priority pj. After concatenating them and embedding them, a single risk label representation vj is obtained. Then, the risk state vector Er can be expressed as: Er=Pool(v1,v2,...,vn), where Pool is a pooling operation. Subsequently, the evidence state vector Ee and the risk state vector Er are concatenated to form a condition vector Ec=[Ee;Er], which is then input into a multilayer perceptron to obtain a scaling vector γ and a bias vector β, satisfying γ=Wγ·Ec+bγ and β=Wβ·Ec+bβ. These are then applied to the backbone hidden state H to obtain the modulated hidden state H′=γ⊙H+β, thus making the model generation process simultaneously constrained by the sufficiency of evidence and the risk state.

[0030] The restricted structured output module performs restricted decoding on the modulated hidden state sequence based on a preset set of structured fields to generate a structured interaction plan. Restricted decoding includes generating output tokens field by field in order and calculating token score vectors at each decoding step. Suppression processing is performed on token scores that do not meet the preset field type constraints. Tool identifier token scores that do not belong to the tool whitelist are masked. The masking process involves replacing the corresponding token score with a preset minimum value so that the probability after exponential operation and normalization is zero. Tokens are selected to be generated based on the remaining token probabilities until the structured interaction plan is completed. The structured interaction plan includes output type identifiers, response points, clarification items, tool call plan, digital human driving parameters, and the binding relationship between key conclusions and evidence identifiers.

[0031] The improved GPT-4o model is similar to the original GPT-4o model in that it includes multimodal input encoding and a multimodal Transformer backbone inference framework. Both models form input embedding sequences through word segmentation, embedding, and positional encoding, and obtain the backbone hidden state sequence through cross-modal fusion inference in self-attention and feedforward layers. The difference lies in the introduction of an evidence-risk condition modulation module and a restricted structured output module after the backbone: the former forms a condition vector based on the evidence set and dialogue state information, performing channel-by-channel scaling and bias superposition condition modulation on the backbone hidden state; the latter performs restricted decoding based on a preset structured field set and tool whitelist constraints, filtering out tool identifiers that do not meet the field type constraints or are not in the whitelist. By fusing the evidence state vector and the risk state vector to form a condition vector, and by conditionally modulating the hidden state of the model backbone, external evidence constraints and risk constraints can be pushed from the cue layer to the internal representation layer of the model, thus avoiding the problem in existing technologies where the model output only refers to evidence and is not subject to the joint evidence-risk constraints. Through improvements, the model can automatically reduce assertion strength and tend to output clarification or safety management plans when there is insufficient evidence or increased risk, reducing unfounded conclusions and unauthorized commitments. In tool invocation scenarios, the restricted structured output makes the output fields stable and parsable, and the tool identifiers and parameters are controlled by whitelists and constraints, reducing the risk of mis-invocation and "faked call success". At the same time, the structured interaction plan carries the binding relationship between key conclusions and evidence identifiers, which facilitates auditing and review, and improves the reliability of interaction and the maintainability of the project.

[0032] In this embodiment, step six specifically includes: The structured interaction plan was analyzed to obtain the output type identifier, tool call plan, digital human driving parameters, and the binding relationship between key conclusions and evidence identifiers; Restricted tool calls are executed according to the tool call plan. The restricted tool call includes matching and verifying the tool identifier in the tool call plan with the tool whitelist, verifying the consistency of the parameter fields in the tool call plan with the tool parameter field set corresponding to the tool whitelist, and calling the corresponding business system open interface to obtain the data returned by the business system interface and writing the data returned by the business system interface into the evidence set when the verification passes. In restricted tool calls, when the tool identifier is not in the tool whitelist or the parameter fields do not meet the tool parameter field set, the output type identifier is set to clarification and a clarification interaction instruction is generated. When the business system's open interface returns failure, the output type identifier is set to refuse to answer or transfer to human agent and the corresponding interaction instruction is generated. The final response content is generated based on the binding relationship between the output type identifier and the key conclusions and evidence identifiers. The final response content is rendered from the key points of the response according to the preset script template, and the evidence identifiers are embedded in the final response content in the form of reference identifiers. Based on the digital human driving parameters, the digital human-driven presentation is executed. The audio stream output by the speech synthesis TTS and the facial expression and action parameters are synchronously driven to drive the digital human rendering engine. The version records of the structured interaction plan, the final response content, the tool call plan and the tool call result formation strategy and model configuration are also recorded.

[0033] refer to Figure 3 A multimodal digital human interaction system based on a large language model includes the following modules: The timeline caching module is used to collect voice data, visual data, text data, and business event data, and perform cross-modal time correlation based on a unified time base to generate a multimodal timeline cache. The state generation module, connected to the timeline cache module, is used to extract the multimodal context information corresponding to the current interaction round based on the multimodal timeline cache and generate dialogue state information; The evidence retrieval module, connected to the state generation module, is used to perform evidence retrieval and association on knowledge base data and business system data based on dialogue state information, and obtain evidence sets and evidence identifiers. The strategy gating module is connected to the state generation module and the evidence retrieval module respectively. It is used to determine the evidence compliance status, risk compliance status and self-consistency score based on the dialogue state information and evidence set, execute triple gating, and output clarification instructions, refusal instructions or manual instructions when the gating is not satisfied. The plan generation module is connected to the evidence retrieval module and the policy gating module respectively. It is used to call the improved GPT-4o model when the gating is satisfied, and generate a structured interaction plan based on the multimodal timeline cache summary, dialogue state information, evidence set, tool constraint information and compliance policy. The execution and recording module, connected to the plan generation module, is used to generate the final response content and record version information of the interaction process based on the structured interaction plan execution restricted tool call and digital human-driven presentation.

[0034] Example 1: To verify the feasibility of this invention in practice, it was applied to a "multimodal digital human customer service" scenario on a large e-commerce platform. This customer service is deployed on both the APP and Web platforms. Users can input via voice and text, and can choose to upload order screenshots or share the page. The business side provides open interfaces for order inquiry, logistics inquiry, refund progress, and work order creation. The system collects voice data, visual data, text data, and business event data in real time on the front end. Voice data is segmented using VAD, with a silence interval threshold of 0.35s, a minimum segment duration of 0.6s, and a maximum duration of 12s. Visual keyframes are sampled every 0.5s, and pixel differences are calculated. A difference threshold of 0.18 is set; frames exceeding the threshold are recorded as keyframes. Business events are encoded into event sequences according to four categories: "page identifier, button trigger, form field change, and interface return," and timestamped. The multimodal timeline cache window takes the most recent 30 seconds and updates it continuously. Speech transcription is performed using ASR to obtain transcribed text, which is then associated with the start and end timestamps of the speech segments and written to the cache. Text normalization standardizes the format of numbers and dates and replaces spoken expressions with a thesaurus (approximately 1200 entries). A field mapping table (approximately 160 entries) maps event parameter field names to standard field names. Intent categories are matched using an intent keyword dictionary (18 intent tag categories), and slot extraction uses a set of regular expression rules (32 categories of slots, including order number, last four digits of mobile phone number, product name, refund amount, and time range). The set of key slots is determined based on the "Intent Category - Required Slot Name Mapping Table" (18 x a maximum of 5 required slots) and the rule that non-empty fields with the same name in the business event take priority.

[0035] In the evidence construction and retrieval enhancement phase, the knowledge base consists of business rules, process SOPs, FAQs, policy clauses, and template scripts, totaling approximately 42,000 entries. Data returned by the business system interface is obtained in real time through open interfaces, such as order status, logistics nodes, refund order numbers, and work order numbers. Retrieval enhancement employs a fusion of vector embedding retrieval and BM25 keyword retrieval, with the vector TopK set to 8 and the BM25 TopK set to 12. After fusion, the final TopK is set to 10 to form the evidence set, and each piece of evidence is assigned an evidence identifier. In the gating phase, an interaction output strategy rule set (including evidence thresholds, risk thresholds, clarification / refusal to answer / transfer to human / tool ​​invocation rules) is used. The evidence threshold is set to "at least 3 pieces of evidence with confidence markers reaching 'high' or 'medium-high'", and the risk threshold is set to "risk level not higher than level 2 (0-3)". The self-consistency score uses a preset sampling number N=3 to generate structured candidate interaction plans, with key fields being "output type identifier and tool invocation identifier", and the consistency ratio threshold is set to 0.67. When gating is not met, the system automatically generates clarification interaction instructions and writes the missing slots into the dialogue state information. For example, if an order number is missing, the user is guided to copy it from the page or upload a screenshot. When gating is met, the system proceeds to plan generation. Plan generation calls an improved GPT-4o model, taking into account a multimodal timeline cache summary, dialogue state information, evidence set, tool whitelist, and compliance policy, and outputs a structured interaction plan. The tool whitelist includes seven callable tools (order query, logistics query, refund progress, return address, discount verification, work order creation, and manual transfer), with a fixed set of parameter fields for each tool and validation of type and length. The compliance policy includes sensitive content categories (personal identification information, payment information, minor-related information, etc.) and risk handling actions (clarification, refusal to answer, transfer to manual, desensitized restatement) and a set of output restriction fields. Internally, the model uses evidence-risk condition modulation to scale and bias the hidden state of the backbone channel by channel, generating a more conservative plan when evidence is insufficient. The restricted structured output masks the scores of tool identifiers that do not meet field type constraints or are not in the whitelist, ensuring that the plan is parsable and executable. During the execution phase, restricted tools are invoked and digital humans are presented according to the structured interaction plan. If the tool parameter verification fails, it will be converted to clarification; if the interface returns a failure, it will be converted to manual transfer or refusal to answer, and reusable remedial instructions will be given. The final response is rendered according to the template based on the key points of the response, and the key conclusions and evidence identifiers are bound and written into the interaction log. The strategy and model configuration are recorded with version numbers for backtracking.

[0036] To verify the beneficial effects, a sample of real business traffic was selected over seven consecutive days, covering common intents such as "order inquiry / logistics inquiry / refund progress / complaint reassurance / rule consultation," totaling 3600 sessions, with 1200 sessions for each method. The comparison methods were set as follows: Method A was "direct concatenation of multimodal input + free text reply using a general large model (no evidence binding, no gating, tool invocation relies solely on prompt words)"; Method B was "RAG retrieval enhancement + free text reply (no self-consistency gating, no restricted structured output, tool invocation without whitelist verification)"; and Method C was the method of this invention. Evaluation indicators included first-time resolution rate, factual error rate, unfounded assertion rate, tool misuse rate, average number of rounds, average response latency, risk handling accuracy, and user satisfaction (5-point questionnaire). The statistical results are shown in Table 1 below: Table 1. Comparison of Key Indicators of Different Digital Human Interaction Methods

[0037] As shown in Table 1, Method C improved the first-time resolution rate by 18.4 percentage points compared to Method A and 7.8 percentage points compared to Method B, indicating that "multimodal timeline alignment + key slot identification + gating clarification" can reduce repeated confirmations and off-topic errors. The factual error rate decreased from 9.6% to 2.2%, and the unfounded assertion rate decreased from 14.8% to 1.9%, mainly due to the constraints of the evidence set and evidence identification mechanism, as well as the pre-interception of the triple gating of evidence threshold, risk threshold, and self-consistency score. Especially in rule consultation and refund progress issues, Method A often makes promises such as "giving the refund arrival time without querying the interface," while this invention will switch to clarification or manual intervention when the gating is not met or the tool returns a failure, and restrict the generation of promise statements in the structured plan. Therefore, the tool miscall rate decreased to 0.8%, and the risk handling accuracy rate increased to 95.4%. The average response latency was slightly higher than Method A, mainly due to the computational overhead brought by the introduction of retrieval fusion and self-consistent sampling N=3. However, the average number of rounds decreased from 3.7 to 2.8, resulting in a shorter overall dialogue time and an increase in user satisfaction to 4.46. The rate of switching to human agents decreased to 8.7%, indicating that the system can still handle more closed-loop issues while ensuring security.

[0038] To further verify the necessity of introducing an evidence-risk condition modulation module and a restricted structured output module after the multimodal Transformer backbone, a comparative test was conducted on the original GPT-4o model and the improved GPT-4o model under the same test samples, knowledge base, evidence set, tool whitelist, and compliance strategy as in Example 1. The original GPT-4o model retained only the multimodal input encoding module and the multimodal Transformer backbone inference module, outputting free text or a weakly constrained plan without field constraints; the improved GPT-4o model adopted the evidence-risk condition modulation module and the restricted structured output module of this invention. To ensure a fair comparison, both models used the same multimodal timeline cache summary, dialogue state information, and evidence set. The statistical results are shown in Table 2 below: Table 2 Performance Comparison of Original GPT-4o and Improved GPT-4o Models

[0039] As shown in Table 2, while the original GPT-4o possesses strong multimodal understanding capabilities, the lack of an evidence-risk condition modulation module means that the model may still output strong assertions along the free-generated path when evidence is insufficient, evidence conflicts occur, or risk increases, resulting in a still relatively high unfounded assertion rate. Furthermore, the absence of a restricted structured output module makes tool call identifiers and parameter fields more prone to unresolvable, unauthorized calls, or inconsistencies with the tool whitelist. In contrast, the improved GPT-4o transforms the evidence state vector and risk state vector into condition vectors and applies them to the hidden state of the backbone, making the model more inclined to generate clarification, rejection, or referral to human intervention plans under high-risk and low-evidence conditions. Combined with the restricted structured output mechanism, the structured interaction plan is more stable in terms of output type identifiers, tool call identifiers, and related fields, thus outperforming the original GPT-4o in terms of unfounded assertion rate, tool misuse rate, structured plan resolvability rate, risk handling accuracy rate, and key field consistency rate. Although the average response latency increases slightly, it is worthwhile to achieve higher interaction reliability and execution security, which demonstrates the necessity of introducing the evidence-risk condition modulation module and the restricted structured output module.

[0040] Based on the embodiments, the beneficial effects of the present invention are mainly reflected in three aspects: First, the multimodal timeline caching and alignment mechanism enables voice, keyframes, text, and business events to form a stable temporal context, reducing cross-modal mismatches; Second, enhanced retrieval forms an evidence set with evidence identification, and gating transforms uncertain scenarios into clarifications, significantly reducing unfounded assertions and factual errors; Third, the improved GPT-4o generates executable interaction plans with evidence-risk condition modulation and restricted structured output, and, in conjunction with tool whitelists and parameter verification, ensures consistency and auditability between tool calls and responses. At the same time, version records support rapid problem location and continuous optimization, thus achieving higher accuracy, higher security, and stronger engineering maintainability in real business customer service scenarios.

[0041] Example 2: To further verify the versatility of the technical solution of the present invention, the present invention was applied to a digital human-guided service scenario in a municipal government service hall. This scenario is deployed on self-service terminals and an online service platform. Users can interact with the digital human through voice, text input, uploading screenshots of certificates or materials, and clicking on service pages. The business side provides open interfaces for service inquiries, material verification, appointment scheduling, progress tracking, and manual transfer.

[0042] This embodiment maintains the same technical process as Embodiment 1, still using the front end to collect voice data, visual data, text data, and business event data in real time, and employing the same multimodal timeline caching, dialogue state generation, retrieval enhancement, gating strategy, improved GPT-4o model, and restricted tool invocation mechanism as Embodiment 1. Specifically, it still uses VAD segmentation, keyframe extraction, business event encoding, and timestamp alignment to form a multimodal timeline cache; it still generates an evidence set based on dialogue state information and assigns evidence identifiers to each piece of evidence; it still performs gating based on evidence thresholds, risk thresholds, and self-consistency scores, where the number of self-consistency samples is N=3, and the consistency ratio threshold is 0.67; when the gating is satisfied, it calls the improved GPT-4o model to output a structured interaction plan; when the gating is not satisfied, it generates clarification, rejection, or referral to human intervention results.

[0043] Unlike Example 1, the knowledge base content in this example is replaced by service guidelines, material lists, policy clauses, and form templates, instead of order rules, logistics rules, and refund rules. The business tools in the tool whitelist are replaced by government tools such as service inquiry, material verification, appointment scheduling, progress tracking, and manual transfer. Key slots are replaced by e-commerce fields such as order number and refund amount with government fields such as service name, applicant type, processing area, material name, and appointment time. Except for the different business semantic objects, the remaining technical processing logic remains consistent.

[0044] In practical applications, when a user says, "I want to cancel my sole proprietorship registration, what materials are still missing?" and simultaneously uploads a photo of their business license and clicks on the relevant page, the system first uses a multimodal timeline cache to uniformly associate the voice content, image content, page click events, and text content. It then generates dialogue status information, identifying the current intent, key slots, and risk tags. Subsequently, it retrieves evidence from the government knowledge base and business interfaces to form an evidence set. A structured interaction plan is generated when the evidence meets the standards, the risk meets the standards, and the self-consistency score meets the requirements. If the attribution of the matter is unclear, the materials are incomplete, or identity verification fails, the process is transferred to a clarification or manual transfer process, thus avoiding unfounded responses and misleading information.

[0045] This embodiment was verified using the same evaluation criteria as in Embodiment 1. The results show that the present invention can maintain a high first-time resolution rate and risk handling accuracy rate in government affairs guidance scenarios, and effectively reduce factual errors, unfounded assertions, and misuse of tools. This indicates that the present invention does not depend on specific e-commerce business scenarios, but can reuse the same technical framework in high compliance requirement scenarios such as government affairs guidance, and achieve traceable responses, controllable risks, verifiable results, and auditable processes.

[0046] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multimodal digital human interaction method based on a large language model, characterized in that, Includes the following steps: Step 1: Collect voice data, visual data, text data, and business event data, perform cross-modal timeline alignment, and generate a multimodal timeline cache; Step 2: Preprocess the multimodal timeline cache, obtain transcribed text based on speech segments, and perform intent classification, slot extraction, and risk label classification on the transcribed text and text data to generate dialogue state information; Step 3: Perform retrieval enhancement on the data returned by the knowledge base and business system interfaces based on the dialogue state information, obtain the evidence set, and assign evidence identifiers to the evidence; Step 4: Generate an output strategy based on the interaction output strategy rule set, and perform triple gating based on the evidence threshold, risk threshold and self-consistency score. If any gating is not satisfied, generate a clarification interaction instruction and update the dialogue state information. If the triple gating is satisfied, proceed to planning and generation. Step 5: Invoke the improved GPT-4o model, which introduces an evidence-risk condition modulation module and a restricted structured output module. The inputs are multimodal timeline cache summary, dialogue state information, evidence set, tool whitelist and compliance policy, and the output is a structured interaction plan. Step Six: Based on the structured interaction plan, execute the restricted tool calls and digital human-driven presentation, generate the final response content, and form a version record of the strategy and model configuration; The improved GPT-4o model includes a multimodal input encoding module, a multimodal Transformer backbone inference module, an evidence-risk condition modulation module, and a restricted structured output module. The multimodal input encoding module concatenates the multimodal timeline cache summary, dialogue state information, evidence set, tool whitelist and compliance policy into fields and serializes them into an input sequence. The input sequence is segmented and word identifiers are generated. The word identifiers are mapped into vector representations and added element by element to the position encoding vector to obtain the input embedding sequence. The multimodal Transformer backbone inference module includes a self-attention layer and a feedforward layer. It calculates attention weights and context representations layer by layer for the input embedding sequence. The calculation of attention weights includes performing a dot product on the query vector and the key vector to obtain a relevance score, scaling the relevance score by a scaling factor and performing exponential operation and normalization to obtain weight coefficients, performing a weighted summation of the weight coefficients and the value vector to obtain the attention output, and adding the attention output to the representation of the previous layer and then normalizing it to obtain the backbone hidden state sequence. The evidence-risk condition modulation module constructs an evidence state vector based on the evidence set and a risk state vector based on the dialogue state information. The evidence state vector and the risk state vector are concatenated to obtain a condition vector. The condition vector is calculated by a multilayer perceptron to obtain a scaling vector and a bias vector. The module then performs element-wise multiplication and element-wise addition on the backbone hidden state sequence to complete the condition modulation and obtain the modulated hidden state sequence. The restricted structured output module generates a structured interaction plan by performing restricted decoding on the modulated hidden state sequence based on a preset set of structured fields. The restricted decoding includes generating output tokens field by field in order and calculating token score vectors at each decoding step. Suppression processing is performed on token scores that do not meet the preset field type constraints. Masking processing is performed on tool identifier token scores that do not belong to the tool whitelist. Tokens are generated based on the probability of the remaining tokens until the structured interaction plan is completed.

2. The multimodal digital human interaction method based on a large language model according to claim 1, characterized in that, Step one specifically includes: Collect voice data, visual data, text data, and business event data; Perform Voice Activity Detection (VAD) on the speech data to detect speech activity intervals, generate speech segments using the silence intervals between adjacent speech activity intervals as segment boundaries, and record the start and end timestamps for each speech segment. Candidate frames are extracted from visual data at a preset sampling interval. The pixel difference between adjacent candidate frames is calculated and compared with the difference threshold. When the difference reaches the preset difference threshold, the scene change point is determined and the corresponding frame is recorded as a key frame to form a key frame sequence. A timestamp is recorded for each key frame and the frame interval between adjacent key frames is used as a visual segment. Business event data is encoded in the order of event occurrence, event types are mapped to event identifiers, and event parameters are serialized into parameter fields to form an event sequence. An event timestamp is recorded for each event. Based on a unified clock, timestamps are aligned for audio segments, visual segments, text data, and event sequences. Audio segment identifiers, keyframe identifiers, text segment identifiers, and event identifiers within the same time window are written into the same alignment record. The alignment records are stored in chronological order and indexed to generate a multimodal timeline cache.

3. The multimodal digital human interaction method based on a large language model according to claim 1, characterized in that, Step two specifically includes: Preprocess the multimodal timeline cache, and read the voice segment, text data, and business event data aligned with the timestamp of the voice segment corresponding to the current interaction round; Automatic speech recognition (ASR) is performed on the speech segment to obtain the transcribed text, and the transcribed text is associated with and stored with the start and end timestamps of the speech segment; Perform rule-based text processing on transcribed text and text data; Intent categories are obtained by matching intents based on a pre-defined intent keyword dictionary; Based on the preset slot regularization rules, a set of key-value pairs consisting of slot names and slot values ​​is extracted, and a set of key slots is determined based on a pre-established mapping table between intent categories and required slot names. Risk tags are obtained by risk matching based on a pre-defined risk keyword dictionary; The intent category, key slot set, and risk label are combined to generate dialogue state information, which is then written to the multimodal timeline cache.

4. The multimodal digital human interaction method based on a large language model according to claim 1, characterized in that, Step three specifically includes: Build search queries based on dialogue status information; Perform retrieval enhancements on the data returned by the interface between the pre-built knowledge base and the business system; Vector embedding retrieval and BM25 keyword retrieval are performed on the knowledge base. Vector embedding retrieval involves vectorizing the search query and retrieving similar entries from the vector index of the knowledge base. BM25 keyword retrieval involves segmenting the search query into keywords and retrieving matching entries from the inverted index of the knowledge base. The vector embedding retrieval results and the BM25 keyword retrieval results are deduplicated and merged, and a candidate evidence list is generated according to the fusion sorting rules. A preset number of items are selected from the candidate evidence list as an evidence set, and an evidence identifier is assigned to each item in the evidence set. The evidence identifier is associated with the evidence source identifier and the evidence content fragment and written into the evidence index.

5. The multimodal digital human interaction method based on a large language model according to claim 1, characterized in that, Step four specifically includes: Output strategies are generated based on a pre-configured set of interactive output strategy rules. The evidence set is calculated to meet the standard status. The confidence markers of each piece of evidence in the evidence set are summarized and the number of evidence items that meet the evidence threshold is counted. When the number of evidence items that meet the evidence threshold reaches the preset number, the evidence is judged to meet the standard. Otherwise, the evidence is judged to be unqualified. Determine the risk compliance status for the risk labels in the dialogue status information, map the risk labels to risk level identifiers, and compare the risk level identifiers with risk thresholds. If the risk thresholds are met, the risk is deemed to be compliant; otherwise, the risk is deemed not compliant. The self-consistency score is calculated under the condition that the evidence meets the standard and the risk meets the standard. The self-consistency score is the proportion of key fields that are consistent in the structured candidate interaction plans generated for the same input under a preset number of sampling N. Gating is performed based on the evidence compliance status, risk compliance status, and self-consistency score. When any one of them fails to meet the gating condition corresponding to the output strategy, a clarification interaction instruction is generated and the dialogue status information is updated. When all gating conditions are met, the planning generation process begins.

6. The multimodal digital human interaction method based on a large language model according to claim 1, characterized in that, Step six specifically includes: The structured interaction plan was analyzed to obtain the output type identifier, tool call plan, digital human driving parameters, and the binding relationship between key conclusions and evidence identifiers; Restricted tool calls are executed according to the tool call plan. The restricted tool calls include matching and verifying the tool identifier in the tool call plan with the tool whitelist, verifying the consistency of the parameter fields in the tool call plan with the set of tool parameter fields corresponding to the tool whitelist, and calling the corresponding business system open interface to obtain the data returned by the business system interface and writing the data returned by the business system interface into the evidence set when the verification is successful. In restricted tool calls, when the tool identifier is not in the tool whitelist or the parameter fields do not meet the tool parameter field set, the output type identifier is set to clarification and a clarification interaction instruction is generated. When the business system's open interface returns failure, the output type identifier is set to refuse to answer or transfer to human agent and the corresponding interaction instruction is generated. The final response content is generated based on the binding relationship between the output type identifier and the key conclusions and evidence identifiers; Based on the digital human driving parameters, the digital human-driven presentation is executed. The audio stream output by the speech synthesis TTS and the facial expression and action parameters are synchronously driven to drive the digital human rendering engine. The version records of the structured interaction plan, the final response content, the tool call plan and the tool call result formation strategy and model configuration are also recorded.

7. A multimodal digital human interaction system based on a large language model, applied to the multimodal digital human interaction method based on a large language model as described in any one of claims 1-6, characterized in that, Includes the following modules: The timeline caching module is used to collect voice data, visual data, text data, and business event data, and perform cross-modal time correlation based on a unified time base to generate a multimodal timeline cache. The state generation module, connected to the timeline cache module, is used to extract the multimodal context information corresponding to the current interaction round based on the multimodal timeline cache and generate dialogue state information; The evidence retrieval module, connected to the state generation module, is used to perform evidence retrieval and association on knowledge base data and business system data based on dialogue state information, and obtain evidence sets and evidence identifiers. The strategy gating module is connected to the state generation module and the evidence retrieval module respectively. It is used to determine the evidence compliance status, risk compliance status and self-consistency score based on the dialogue state information and evidence set, execute triple gating, and output clarification instructions, refusal instructions or manual instructions when the gating is not satisfied. The plan generation module is connected to the evidence retrieval module and the policy gating module respectively. It is used to call the improved GPT-4o model when the gating is satisfied, and generate a structured interaction plan based on the multimodal timeline cache summary, dialogue state information, evidence set, tool constraint information and compliance policy. The execution and recording module, connected to the plan generation module, is used to generate the final response content and record version information of the interaction process based on the structured interaction plan execution restricted tool call and digital human-driven presentation.

Images