A method and system for matching the dialogue intent of intelligent NPCs in multimodal interaction

Abstract

This invention relates to the field of natural language processing technology, specifically to a method and system for matching the intent of intelligent NPC dialogues in multimodal interaction. The method involves real-time acquisition of multimodal data, generating multimodal semantic features through submodal preprocessing; integrating the semantic features of each modality using a cross-modal fusion module based on semantic association, generating a candidate intent set based on semantic context representation and combined with a semantic parsing module and predefined intent templates; tracking changes in user intent in real time through a continuous semantic learning mechanism and an interactive memory module, combined with a Bayesian update method, dynamically adjusting the confidence level of each intent in the candidate intent set, and filtering the final intent; constructing an NPC semantic cognition model, and performing semantic consistency analysis to perform semantic checks on user input and NPC dialogue state; combining the final intent and a decision engine to generate a dialogue strategy and output synchronized response content; this invention improves the accuracy of dynamic matching of intelligent dialogue intents.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a method and system for matching the dialogue intent of intelligent NPCs in multimodal interaction. Background Technology

[0002] In the field of Natural Language Processing (NLP), building systems capable of fluent and in-depth dialogue with humans is one of its core goals. Current automated dialogue systems, especially in scenarios requiring continuous, context-aware interaction, suffer from limitations in their data processing methods across three levels:

[0003] 1) The singularity of the input data source limits the completeness of semantic understanding.

[0004] Traditional dialogue systems typically treat the user's language (whether speech or text) as the sole data source when processing input. For example, after speech input is converted into text by Automatic Speech Recognition (ASR), important features such as stress, emotion, and speech rate contained in the original acoustic information are discarded. This processing method reduces the rich and multidimensional communication signal to a single text data stream, directly leading to serious information loss in subsequent semantic processing and analysis stages.

[0005] 2) The semantic intent matching method lacks dynamic adaptability.

[0006] Current mainstream intent recognition technologies, as a key component of Natural Language Understanding (NLU), typically employ a static mapping mechanism. In the offline phase, the system establishes a mapping relationship from specific text patterns to predefined intent labels through supervised learning. However, in a real, continuous dialogue, a user's intent is fluid, potentially evolving, and may even generate new intents not seen during training. Existing data processing workflows cannot effectively handle this dynamic drift of intent and open-domain intents.

[0007] 3) The dialogue management and response generation strategies have relatively shallow context dependencies.

[0008] After determining the user's intent, the dialogue management module that generates responses often relies on simple state machines or shallow contextual information. This results in natural language responses that, while grammatically correct, may not reflect the deep dialogue history, the user's implicit emotional state, or long-term goals, leading to poor coherence and weak logic in the interaction. Existing methods fail to effectively integrate multi-turn historical interactions, user state changes reflected by multimodal information, and a long-term dialogue goal into the data processing flow for response generation, preventing the system from engaging in deep and coherent dialogue.

[0009] To address this, a method and system for matching the dialogue intent of intelligent NPCs with multimodal interaction is proposed. Summary of the Invention

[0010] The purpose of this invention is to provide a method and system for matching the dialogue intent of intelligent NPCs in multimodal interaction.

[0011] To achieve the above objectives, the present invention provides the following technical solution:

[0012] A method for matching the dialogue intent of intelligent NPCs in multimodal interaction includes:

[0013] Real-time acquisition of users' multimodal data, performing submodal preprocessing on each modal data, and generating multimodal semantic features;

[0014] By integrating the semantic features of each modality through a cross-modal fusion module based on semantic association, a unified semantic context representation is generated.

[0015] Based on semantic context representation, a set of intent candidates is obtained using a semantic parsing module and a predefined semantic intent template. Through a continuous semantic learning mechanism and an interactive memory module, combined with a Bayesian update method, changes in user intent are tracked in real time, and the confidence of each intent in the intent candidate set is dynamically adjusted. The final intent is selected based on a preset confidence threshold.

[0016] Construct an NPC semantic cognition model, detect potential conflicts between user input and the current semantic state of the NPC through semantic consistency analysis; when a semantic conflict is detected, trigger an adaptive adjustment mechanism for the dialogue strategy, generate a dialogue strategy by combining the final intent and the decision engine, and output synchronized natural language responses, facial animations and speech prosody.

[0017] Furthermore, the process of performing modal preprocessing on the data from each modality to generate multimodal semantic features includes:

[0018] The text input is segmented, part-of-speech tagging is performed, and syntactic analysis is conducted to form text semantic features;

[0019] Acoustic analysis is performed on the speech input to extract pitch, speech rate and emotional prosodic features, forming speech semantic features;

[0020] The system classifies facial expression input into emotions, identifies emotional states, and forms semantic features of facial expressions.

[0021] The gesture input is subjected to action pattern recognition, and the direction and intensity of the action are extracted to form gesture semantic features;

[0022] By performing semantic normalization, each semantic feature is mapped to a unified semantic representation space, generating multimodal semantic features.

[0023] Furthermore, the process of integrating semantic features from various modalities to generate a unified semantic context representation through a cross-modal fusion module based on semantic association includes:

[0024] By using a semantic attention allocation mechanism, the semantic relevance weights between semantic features of each modality are calculated, and semantic association vectors between modalities are generated.

[0025] Based on the semantic routing rules of cross-modal capsule networks, the semantic features of each modality are dynamically grouped and weighted according to the semantic association vector to form a multi-level semantic capsule representation;

[0026] Semantic capsule representations are semantically decoded and smoothed to generate a unified semantic context representation.

[0027] Furthermore, by employing a continuous semantic learning mechanism and an interactive memory module, combined with a Bayesian update method, the process of dynamically adjusting the confidence level of each intent in the intent candidate set by tracking changes in user intent in real time includes:

[0028] Through the interactive memory module, the semantic context representation of the current dialogue and the historical interactive semantic patterns are recorded to form a dynamic semantic interaction sequence; using a continuous semantic learning mechanism, the semantic patterns in the dynamic semantic interaction sequence are iteratively updated based on the dialogue interaction feedback to identify the trend of intent evolution.

[0029] Each intent in the intent candidate set, the evolution trend of the corresponding intent, and the semantic context representation are input into the likelihood estimation network to obtain the likelihood of each intent.

[0030] Based on the Bayesian probability update method, the confidence of each intent in the intent candidate set is updated by combining the prior probability and likelihood at the current time. The order and priority of each intent in the intent candidate set are dynamically adjusted by the confidence adjustment rule.

[0031] Furthermore, the process of detecting potential conflicts between user input and the current semantic state of the NPC through semantic consistency analysis includes:

[0032] The semantic context representation obtained by integrating user input is compared with the semantic state representation of the NPC.

[0033] By using semantic consistency verification rules, the matching degree between user semantic features and NPC semantic states is analyzed to detect potential semantic conflicts.

[0034] When a semantic conflict is detected, an adaptive dialogue adjustment mechanism is triggered to optimize the dialogue strategy.

[0035] Furthermore, the process of combining the final intent and the decision engine to generate a dialogue strategy and output synchronized natural language responses, facial animations, and speech prosody includes:

[0036] Based on the final intent, semantic context representation, and NPC semantic state, the dialogue policy rules are optimized through reinforcement learning to generate a dialogue update policy.

[0037] Generate natural language responses by utilizing natural language generation rules and based on dialogue update strategies and NPC character settings;

[0038] Based on the expression generation rules, matching facial expression animations are generated according to the final intention and emotional semantics.

[0039] The speech prosody is adjusted based on speech synthesis rules and output synchronously with natural language responses and facial animations.

[0040] A multimodal interactive intelligent NPC dialogue intent matching system, comprising:

[0041] The data acquisition and processing unit is used to acquire users' multimodal data in real time, perform submodal preprocessing on each modal data, and generate multimodal semantic features;

[0042] The semantic fusion unit is used to integrate the semantic features of various modalities through a cross-modal fusion module based on semantic association, and generate a unified semantic context representation.

[0043] The intent matching unit is used to obtain a set of intent candidates based on semantic context representation, using the semantic parsing module and predefined semantic intent templates; through a continuous semantic learning mechanism and interactive memory module, combined with the Bayesian update method, it tracks changes in user intent in real time and dynamically adjusts the confidence of each intent in the intent candidate set; and selects the final intent according to the preset confidence threshold.

[0044] The semantic detection and output unit is used to build an NPC semantic cognition model. It detects conflicts between user input and NPC semantic state through semantic consistency verification and triggers adaptive dialogue adjustment. It combines the final intent and decision engine to generate dialogue strategy and output synchronized natural language response, facial animation and speech prosody.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0046] 1. By calculating the semantic relevance weights between modalities through a semantic attention allocation mechanism, the contribution of important modalities is highlighted, thereby improving the accuracy of the semantic representation after fusion. Dynamic grouping and weighted integration are achieved through semantic routing rules based on cross-modal capsule networks, forming a multi-level semantic capsule representation. This preserves the complex semantic relationships between modalities and improves the richness and expressiveness of the semantic context representation. Through semantic decoding and smoothing, the generated unified semantic context representation is more coherent, reducing information loss in modal fusion and enhancing the reliability of subsequent intent recognition and matching.

[0047] 2. By combining the interactive memory module with the continuous semantic learning mechanism, the evolution trend of intent is identified, enabling NPCs to adapt to the dynamic changes of intent and improve the contextual relevance of the interaction. Through the likelihood estimation network and the Bayesian probability update method, the confidence of each intent in the intent candidate set is updated in real time and the sorting and priority of the intent candidate set are dynamically adjusted to ensure the accuracy of intent matching results.

[0048] 3. By comparing semantic similarity and verifying consistency rules, potential conflicts between user input and the current semantic state of the NPC are accurately detected, avoiding logical errors or semantic deviations in the dialogue; when a semantic conflict is detected, an adaptive dialogue adjustment mechanism is triggered to optimize the dialogue strategy, reduce misunderstandings and unnatural dialogue responses, and improve the coherence of the interaction and user satisfaction; based on reinforcement learning, the dialogue strategy rules are optimized, and the dialogue update strategy is generated by combining the final intent and semantic context representation to ensure that the NPC's response meets the user's intent and the needs of the scenario. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating a multimodal interactive intelligent NPC dialogue intent matching method according to the present invention.

[0050] Figure 2 A schematic diagram illustrating the process of generating a dialogue update strategy for this invention;

[0051] Figure 3 This is a schematic diagram of the structure of a multimodal interactive intelligent NPC dialogue intent matching system according to the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Please see Figures 1 to 3 This invention provides a method and system for matching the dialogue intent of intelligent NPCs in multimodal interaction, the technical solution of which is as follows:

[0054] Example 1:

[0055] To improve the accuracy of dynamic matching of dialogue intent in intelligent NPCs, a company used a multimodal interaction intelligent NPC dialogue intent matching method proposed in this invention. The flowchart of this method is as follows: Figure 1 As shown, it specifically includes:

[0056] Real-time acquisition of users' multimodal data, performing submodal preprocessing on each modal data, and generating multimodal semantic features;

[0057] Furthermore, the multimodal data includes: text data, voice data, facial expression data, and gesture data; text data and voice data are collected through text input devices (such as keyboards, touch screens, etc.) and microphones, respectively, while facial expression data and gesture data are collected through cameras;

[0058] Furthermore, the process of performing modal preprocessing on the data from each modality to generate multimodal semantic features includes:

[0059] The text input is segmented, part-of-speech tagging is performed, and syntactic analysis is conducted to form text semantic features;

[0060] Acoustic analysis is performed on the speech input to extract pitch, speech rate and emotional prosodic features, forming speech semantic features;

[0061] The system classifies facial expression input into emotions, identifies emotional states, and forms semantic features of facial expressions.

[0062] The gesture input is subjected to action pattern recognition, and the direction and intensity of the action are extracted to form gesture semantic features;

[0063] By performing semantic normalization, each semantic feature is mapped to a unified semantic representation space, generating multimodal semantic features.

[0064] Furthermore, the Stanford CoreNLP tool was used to segment and tag the text, and then the processing results were input into the pre-trained BERT-Large model for analysis and feature extraction to obtain the semantic features of the text. The model was trained on the BookCorpus and Wikipedia datasets with the following parameter settings: batch size of 16, AdamW optimizer, and learning rate of 2e-5.

[0065] Furthermore, the pre-trained Wav2Vec 2.0 model is directly used to perform feature analysis on the speech data, extracting pitch, speech rate and emotional prosodic features to generate speech semantic features; the speech data needs to be preprocessed before input to improve computational efficiency, including: downsampling to 16kHz, frame length of 25ms, and frame shift of 10ms;

[0066] Furthermore, the ResNet-50 backbone network of the pre-trained DeepFace model is used to identify the state in facial expression images, and the semantic features of facial expressions are extracted through fully connected layers. The DeepFace model is trained on the FER2013 and AffectNet datasets with a batch size of 32 and a learning rate of 3e-4.

[0067] Furthermore, independent fully connected layers are used to map the features of each modality to a unified 512-dimensional semantic space, with the activation function set to ReLU and layer normalization operations added.

[0068] By performing modal preprocessing on text, speech, facial expressions, and gestures, semantic features of each modality are extracted, ensuring the comprehensiveness and accuracy of subsequent semantic analysis. Through semantic normalization, the semantic features of different modalities are mapped to a unified semantic representation space, eliminating the heterogeneity between modalities and providing high-quality input for subsequent cross-modal fusion.

[0069] By integrating the semantic features of each modality through a cross-modal fusion module based on semantic association, a unified semantic context representation is generated.

[0070] Furthermore, the process of integrating semantic features from various modalities to generate a unified semantic context representation through a cross-modal fusion module based on semantic association includes:

[0071] By using a semantic attention allocation mechanism, the semantic relevance weights between semantic features of each modality are calculated, and semantic association vectors between modalities are generated.

[0072] Based on the semantic routing rules of cross-modal capsule networks, the semantic features of each modality are dynamically grouped and weighted according to the semantic association vector to form a multi-level semantic capsule representation;

[0073] Semantic capsule representations are semantically decoded and smoothed to generate a unified semantic context representation;

[0074] Furthermore, the semantic attention allocation mechanism is a multi-head attention mechanism, implemented through a Transformer layer. The generation process of the semantic association vector includes: concatenating the features of each modality to form a matrix; transforming the concatenated matrix into a query matrix, key matrix, and value matrix through linear transformation; obtaining the semantic association weight matrix using the attention weight calculation formula, which contains the semantic relevance weights between the semantic features of each modality; applying the semantic association weight matrix to the value matrix and transforming it into a semantic association vector; the settings in the multi-head attention mechanism are as follows: the number of attention heads is 8, each head processes 64-dimensional data, the total dimension is 512, and the Dropout rate to prevent overfitting is set to 0.1.

[0075] Furthermore, in order to improve the semantic discriminative ability of attention weights and enhance the semantic alignment between modalities, contrastive learning is introduced into the semantic attention allocation process to optimize the semantic relevance weights of the multi-head attention mechanism, making the semantic alignment between modalities more accurate. Modal features of the same dialogue and modal features of different dialogues are respectively used as positive and negative samples. The semantic distance of positive samples is minimized and the distance of negative samples is maximized by using InfoNCE loss.

[0076] Furthermore, the attention-weighted features, i.e., semantic association vectors, are input into a cross-modal capsule network. This network uses a transformation matrix to map the input features to a capsule space, dividing the multimodal features into 32 capsules, each containing a 16-dimensional vector. Then, a dynamic routing algorithm is used to iteratively optimize the routing coefficients to ensure the accuracy of semantic grouping. Next, the routing coefficients are used to weight the transformation matrix mapping results to generate semantic capsule representations. The number of routing iterations in the process is set to 3, and the Squash function is applied to the weighted results, so the semantic capsule representations are in the range of 0-1.

[0077] Furthermore, the 32 semantic capsule representations are concatenated into a 512-dimensional long vector. The long vector is then processed using a two-layer fully connected network to obtain a 512-dimensional semantic context representation. The first layer of the fully connected network reduces the 512-dimensional representation to 256-dimensional representation and removes negative values ​​using the ReLU activation function. Then, the second fully connected layer increases the dimensionality back to the original 512-dimensional representation.

[0078] The semantic attention allocation mechanism is used to calculate the semantic relevance weights between modalities, highlighting the contribution of important modalities and improving the accuracy of the semantic representation after fusion. Dynamic grouping and weighted integration are achieved through semantic routing rules based on cross-modal capsule networks to form multi-level semantic capsule representations, which improves the richness and expressiveness of semantic context representation. Semantic decoding and smoothing processes reduce information loss in modal fusion and enhance the reliability of subsequent intent recognition and matching.

[0079] Based on semantic context representation, a set of intent candidates is obtained using a semantic parsing module and a predefined semantic intent template. Through a continuous semantic learning mechanism and an interactive memory module, combined with a Bayesian update method, changes in user intent are tracked in real time, and the confidence of each intent in the intent candidate set is dynamically adjusted. The final intent is selected based on a preset confidence threshold.

[0080] Furthermore, the semantic context representation is processed using a pre-trained Seq2Seq model and converted into a structured semantic representation, generally in logical form. The predefined semantic intent template contains the intent type and related slots. Then, the logical form output by the semantic parsing module is matched with the predefined intent template. Next, slot values ​​are extracted from the logical form and filled into the matched intent template, which is used as the intent candidate set.

[0081] Furthermore, by employing a continuous semantic learning mechanism and an interactive memory module, combined with a Bayesian update method, the process of dynamically adjusting the confidence level of each intent in the intent candidate set by tracking changes in user intent in real time includes:

[0082] Through the interactive memory module, the semantic context representation of the current dialogue and the historical interactive semantic patterns are recorded to form a dynamic semantic interaction sequence; using a continuous semantic learning mechanism, the semantic patterns in the dynamic semantic interaction sequence are iteratively updated based on the dialogue interaction feedback to identify the trend of intent evolution.

[0083] Each intent in the intent candidate set, the evolution trend of the corresponding intent, and the semantic context representation are input into the likelihood estimation network to obtain the likelihood of each intent.

[0084] Based on the Bayesian probability update method, the confidence of each intent in the intent candidate set is updated by combining the prior probability and likelihood at the current time. The order and priority of each intent in the intent candidate set are dynamically adjusted by the confidence adjustment rule.

[0085] The following table shows the results of the intent confidence adjustment before and after for three different dialogues, denoted as Dialogue 1, Dialogue 2 and Dialogue 3 respectively. To facilitate the display of the adjustment results, one intent is selected from the intent candidate set of each dialogue for display. The corresponding intents are asking about the task, confirming the reward and expressing emotion, respectively. The results of the confidence adjustment before and after are shown in Table 1.

[0086] Table 1 Results before and after confidence level adjustment

[0087]

[0088] Furthermore, the interactive memory module employs a bidirectional LSTM network and is trained using the MultiWOZ dialogue dataset. During training, the initial learning rate is set to 1e-4, the dropout rate to 0.2, and the activation function used is the tanh function. The current semantic context representation and the historical semantic state vector are used as inputs to the interactive memory module, and the outputs are the current semantic state vector and the dynamic semantic interaction sequence. The historical semantic state vector is obtained by inputting the historical semantic context representation into the interactive memory module and is used to record historical semantic patterns. The dynamic semantic interaction sequence is a time series representing the changes in semantic patterns from the beginning of the dialogue to the present.

[0089] Furthermore, based on the user's dialogue and interaction feedback with the intelligent NPC, the cross-entropy loss is calculated, and the parameters of the bidirectional LSTM network are updated using an online gradient descent algorithm.

[0090] Furthermore, to reduce the number of interaction rounds required for new users and accelerate personalized semantic learning, online intent transfer learning is introduced during the continuous semantic learning process. This leverages cross-user dialogue data to share semantic patterns and accelerates the intent evolution trend learning of the bidirectional LSTM. Specifically, a cross-user dynamic semantic interaction sequence pool with a capacity of 10,000 sequences is maintained to store semantic interaction sequences from other users. Cosine similarity is used to match the current user's sequence with the sequences in the pool. The gradients of the matched sequences are then incorporated into the parameter updates of the bidirectional LSTM model in a weighted manner, thereby improving the efficiency of personalized adaptation.

[0091] Furthermore, the likelihood estimation network employs a three-layer fully connected network, taking each intent, semantic state vector, and semantic context representation in the intent candidate set as network input, and outputting the likelihood of each intent. Combining the prior probability and the likelihood, Bayes' theorem is applied to each intent to update its posterior probability, i.e., the confidence level. The prior probability is the posterior probability of the previous time step. If the intent is newly added or is the first interaction, an initial prior probability based on heuristic rules is assigned to it.

[0092] Furthermore, the updated confidence scores are normalized to generate priority weights, which are used to adjust the order and priority of each intent in the intent candidate set.

[0093] By combining the interactive memory module with a continuous semantic learning mechanism to identify intent evolution trends, NPCs can adapt to dynamic changes in intents and improve the contextual relevance of interactions. Through a likelihood estimation network and a Bayesian probability update method, the confidence of each intent in the intent candidate set is updated in real time, and the ranking and priority of the intent candidate set are dynamically adjusted to ensure the accuracy of intent matching results.

[0094] Construct an NPC semantic cognition model, detect potential conflicts between user input and the current semantic state of the NPC through semantic consistency analysis; when a semantic conflict is detected, trigger an adaptive adjustment mechanism for the dialogue strategy, generate a dialogue strategy by combining the final intent and the decision engine, and output synchronized natural language responses, facial animations and speech prosody.

[0095] Furthermore, the BDI model is used as the semantic cognition model of NPCs to model the cognitive state of NPCs, which includes beliefs, desires, and intentions. Beliefs are expressed through propositional logic expressions to show cognition about the world, including knowledge about the environment and one's own state. Desires are the long-term goals of NPCs, represented by utility functions. Intentions represent the actions that NPCs promise to perform or the sub-goals they promise to achieve.

[0096] Furthermore, the process of detecting potential conflicts between user input and the current semantic state of the NPC through semantic consistency analysis includes:

[0097] The semantic context representation obtained by integrating user input is compared with the semantic state representation of the NPC.

[0098] By using semantic consistency verification rules, the matching degree between user semantic features and NPC semantic states is analyzed to detect potential semantic conflicts.

[0099] When a semantic conflict is detected, an adaptive dialogue adjustment mechanism is triggered to optimize the dialogue strategy.

[0100] Furthermore, by combining the interactive memory module and semantic context representation, the user semantic state vector is obtained; then, the cosine similarity between the user semantic state vector and the current semantic state vector of the NPC is calculated, and a threshold comparison is performed on it, with the semantic similarity threshold set at 0.85.

[0101] Furthermore, the semantic consistency verification rules are as follows: if the semantic similarity is lower than the threshold, a semantic conflict is considered to exist; if the user's intent type does not match the NPC's intent type, a semantic conflict is considered to exist; if the slot information provided by the user contradicts the slot information collected by the NPC, a semantic conflict is considered to exist.

[0102] Furthermore, semantic consistency verification includes sentiment consistency and contextual coherence verification in addition to semantic similarity judgment. Sentiment consistency verification is achieved by extracting the user's sentiment vector from the semantic context representation and calculating its cosine similarity with the NPC's sentiment state. Contextual coherence verification is achieved by inputting the interaction memory sequence into a fully connected network for coherence prediction.

[0103] Furthermore, the adaptive dialogue adjustment mechanism employs a reinforcement learning strategy.

[0104] By comparing semantic similarity and using consistency verification rules, potential conflicts between user input and the current semantic state of the NPC can be accurately detected, avoiding logical errors or semantic deviations in the dialogue and providing a basis for subsequent dialogue strategy optimization.

[0105] Furthermore, the process of combining the final intent and the decision engine to generate a dialogue strategy and output synchronized natural language responses, facial animations, and speech prosody includes:

[0106] Based on the final intent, semantic context representation, and NPC semantic state, the dialogue policy rules are optimized through reinforcement learning to generate a dialogue update policy.

[0107] Generate natural language responses by utilizing natural language generation rules and based on dialogue update strategies and NPC character settings;

[0108] Based on the expression generation rules, matching facial expression animations are generated according to the final intention and emotional semantics.

[0109] The speech prosody is adjusted based on speech synthesis rules and output synchronously with natural language responses and facial animations;

[0110] Furthermore, the state space of reinforcement learning includes user intent, user semantic context, and the cognitive state of the NPC; the action space is a discrete set containing all combinations of dialogue strategies and accompanying behaviors; the reward function consists of user satisfaction (user feedback rating) and semantic consistency (semantic similarity).

[0111] Furthermore, the process for generating the dialogue update strategy can be referenced. Figure 2 Specifically, the process involves: constructing a deep neural network and defining a target network and an experience replay buffer; selecting actions using a greedy policy, executing the selected actions and observing the feedback from the environment, and storing the experience tuples (state, action, reward, next state, and whether the process has ended) in the experience replay buffer; training a deep Q-network using randomly sampled experience from the experience replay buffer, calculating the Q-value and loss function, and updating the parameters of the deep Q-network using gradient descent to minimize the loss; periodically copying the parameters of the deep Q-network to the target network until training is complete, selecting the action with the highest Q-value and mapping it to the dialogue policy rule to obtain the dialogue update policy;

[0112] Furthermore, the natural language responses are generated using a pre-trained T5-Large model, employing cue engineering techniques and additionally utilizing prior knowledge such as generation rules, dialogue update strategies, and NPC character settings to guide the model in generating expected language responses. Based on preset expression rules, the final intent and emotional semantics are mapped to expressions, and then animation blending techniques are used to create rich facial animations. A pre-trained WaveNet model is used to convert text into speech, and the prosody of the speech is adjusted according to sentiment labels and context labels.

[0113] By using reinforcement learning to optimize dialogue policy rules and combining the final intent and semantic context representation to generate dialogue update policies, we can ensure that the NPC's response conforms to the user's intent and the needs of the scenario, while reducing misunderstandings and unnatural dialogue responses, and improving the coherence of the interaction and user satisfaction.

[0114] This embodiment proposes a method for intelligent NPC dialogue intent matching in multimodal interaction. The method collects multimodal data in real time, generates multimodal semantic features through modal preprocessing, integrates the semantic features of each modality using a cross-modal fusion module based on semantic association, and generates an intent candidate set based on semantic context representation combined with a semantic parsing module and predefined intent templates. Through a continuous semantic learning mechanism and an interactive memory module, combined with a Bayesian update method, it tracks changes in user intent in real time, dynamically adjusts the confidence of each intent in the intent candidate set, and filters the final intent. An NPC semantic cognition model is constructed, and semantic consistency analysis is used to perform semantic checks on user input and NPC dialogue state. Finally, a dialogue strategy is generated by combining the final intent and a decision engine, and synchronous response content is output. This invention improves the accuracy of dynamic matching of intelligent dialogue intent.

[0115] Example 2:

[0116] This invention also proposes a multimodal interactive intelligent NPC dialogue intent matching system, the structure of which is as follows: Figure 3 As shown, it includes:

[0117] The data acquisition and processing unit is used to acquire users' multimodal data in real time, perform submodal preprocessing on each modal data, and generate multimodal semantic features;

[0118] The semantic fusion unit is used to integrate the semantic features of various modalities through a cross-modal fusion module based on semantic association, and generate a unified semantic context representation.

[0119] Furthermore, the process of integrating semantic features from various modalities to generate a unified semantic context representation through a cross-modal fusion module based on semantic association includes:

[0120] By using a semantic attention allocation mechanism, the semantic relevance weights between semantic features of each modality are calculated, and semantic association vectors between modalities are generated.

[0121] Based on the semantic routing rules of cross-modal capsule networks, the semantic features of each modality are dynamically grouped and weighted according to the semantic association vector to form a multi-level semantic capsule representation;

[0122] Semantic capsule representations are semantically decoded and smoothed to generate a unified semantic context representation.

[0123] The intent matching unit is used to obtain a set of intent candidates based on semantic context representation, using the semantic parsing module and predefined semantic intent templates; through a continuous semantic learning mechanism and interactive memory module, combined with the Bayesian update method, it tracks changes in user intent in real time and dynamically adjusts the confidence of each intent in the intent candidate set; and selects the final intent according to the preset confidence threshold.

[0124] Furthermore, by employing a continuous semantic learning mechanism and an interactive memory module, combined with a Bayesian update method, the process of dynamically adjusting the confidence level of each intent in the intent candidate set by tracking changes in user intent in real time includes:

[0125] Through the interactive memory module, the semantic context representation of the current dialogue and the historical interactive semantic patterns are recorded to form a dynamic semantic interaction sequence; using a continuous semantic learning mechanism, the semantic patterns in the dynamic semantic interaction sequence are iteratively updated based on the dialogue interaction feedback to identify the trend of intent evolution.

[0126] Each intent in the intent candidate set, the evolution trend of the corresponding intent, and the semantic context representation are input into the likelihood estimation network to obtain the likelihood of each intent.

[0127] Based on the Bayesian probability update method, the confidence of each intent in the intent candidate set is updated by combining the prior probability and likelihood at the current time. The order and priority of each intent in the intent candidate set are dynamically adjusted by the confidence adjustment rule.

[0128] The semantic detection and output unit is used to build an NPC semantic cognition model. It detects conflicts between user input and NPC semantic state through semantic consistency verification and triggers adaptive dialogue adjustment. It combines the final intent and decision engine to generate dialogue strategy and output synchronized natural language response, facial animation and speech prosody.

[0129] The performance comparison results of single-modal and multi-modal fusion and with and without dynamic adjustment of confidence are given below, as shown in Table 2. It can be seen that the solution of the present invention can improve the accuracy of intent matching and semantic consistency.

[0130] Table 2 Performance Comparison Results

[0131]

[0132] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for matching the dialogue intent of intelligent NPCs in multimodal interaction, characterized in that, include: Real-time acquisition of users' multimodal data, performing submodal preprocessing on each modal data, and generating multimodal semantic features; A cross-modal fusion module based on semantic association integrates semantic features from various modalities to generate a unified semantic context representation; the process includes: By using a semantic attention allocation mechanism, the semantic relevance weights between semantic features of each modality are calculated, and semantic association vectors between modalities are generated. Based on the semantic routing rules of cross-modal capsule networks, the semantic features of each modality are dynamically grouped and weighted according to the semantic association vector to form a multi-level semantic capsule representation; Semantic capsule representations are semantically decoded and smoothed to generate a unified semantic context representation; Based on semantic context representation, a candidate intent set is obtained using a semantic parsing module and a predefined semantic intent template. Through a continuous semantic learning mechanism and an interactive memory module, combined with a Bayesian update method, changes in user intent are tracked in real time, and the confidence level of each intent in the candidate intent set is dynamically adjusted. The final intent is then selected based on a preset confidence threshold. The process includes: Through the interactive memory module, the semantic context representation of the current dialogue and the historical interactive semantic patterns are recorded to form a dynamic semantic interaction sequence; using a continuous semantic learning mechanism, the semantic patterns in the dynamic semantic interaction sequence are iteratively updated based on the dialogue interaction feedback to identify the trend of intent evolution. Each intent in the intent candidate set, the evolution trend of the corresponding intent, and the semantic context representation are input into the likelihood estimation network to obtain the likelihood of each intent. Based on the Bayesian probability update method, the confidence of each intent in the intent candidate set is updated by combining the prior probability and likelihood at the current time. The order and priority of each intent in the intent candidate set are dynamically adjusted by the confidence adjustment rule. Construct an NPC semantic cognition model, detect potential conflicts between user input and the current semantic state of the NPC through semantic consistency analysis; when a semantic conflict is detected, trigger an adaptive adjustment mechanism for the dialogue strategy, generate a dialogue strategy by combining the final intent and the decision engine, and output synchronized natural language responses, facial animations and speech prosody.

2. The method for matching the dialogue intent of intelligent NPCs in multimodal interaction according to claim 1, characterized in that, The process of performing modal preprocessing on data from each modality to generate multimodal semantic features includes: The text input is segmented, part-of-speech tagging is performed, and syntactic analysis is conducted to form text semantic features; Acoustic analysis is performed on the speech input to extract pitch, speech rate and emotional prosodic features, forming speech semantic features; The system performs emotion classification on facial expression input, identifies emotional states, and forms semantic features of facial expressions. The gesture input is subjected to action pattern recognition, and the direction and intensity of the action are extracted to form gesture semantic features; By performing semantic normalization, each semantic feature is mapped to a unified semantic representation space, generating multimodal semantic features.

3. The method for matching the dialogue intent of intelligent NPCs in multimodal interaction according to claim 1, characterized in that, The process of detecting potential conflicts between user input and the current semantic state of an NPC through semantic consistency analysis includes: The semantic context representation obtained by integrating user input is compared with the semantic state representation of the NPC. By using semantic consistency verification rules, the matching degree between user semantic features and NPC semantic states is analyzed to detect potential semantic conflicts. When a semantic conflict is detected, an adaptive dialogue adjustment mechanism is triggered to optimize the dialogue strategy.

4. The method for matching the dialogue intent of intelligent NPCs in multimodal interaction according to claim 1, characterized in that, The process of combining the final intent and the decision engine to generate a dialogue strategy and output synchronized natural language responses, facial animations, and speech prosody includes: Based on the final intent, semantic context representation, and NPC semantic state, the dialogue policy rules are optimized through reinforcement learning to generate a dialogue update policy. Generate natural language responses by utilizing natural language generation rules and based on dialogue update strategies and NPC character settings; Based on the expression generation rules, matching facial expression animations are generated according to the final intention and emotional semantics. The speech prosody is adjusted based on speech synthesis rules and output synchronously with natural language responses and facial animations.

5. A multimodal interactive intelligent NPC dialogue intent matching system, characterized in that, include: The data acquisition and processing unit is used to acquire users' multimodal data in real time, perform submodal preprocessing on each modal data, and generate multimodal semantic features; The semantic fusion unit is used to integrate semantic features from various modalities through a cross-modal fusion module based on semantic association, generating a unified semantic context representation; the process includes: By using a semantic attention allocation mechanism, the semantic relevance weights between semantic features of each modality are calculated, and semantic association vectors between modalities are generated. Based on the semantic routing rules of cross-modal capsule networks, the semantic features of each modality are dynamically grouped and weighted according to the semantic association vector to form a multi-level semantic capsule representation; Semantic capsule representations are semantically decoded and smoothed to generate a unified semantic context representation; The intent matching unit, based on semantic context representation, utilizes a semantic parsing module and predefined semantic intent templates to obtain a candidate intent set. Through a continuous semantic learning mechanism and an interactive memory module, combined with a Bayesian update method, it tracks changes in user intent in real time and dynamically adjusts the confidence level of each intent in the candidate intent set. Based on a preset confidence threshold, it selects the final intent. The process includes: Through the interactive memory module, the semantic context representation of the current dialogue and the historical interactive semantic patterns are recorded to form a dynamic semantic interaction sequence; using a continuous semantic learning mechanism, the semantic patterns in the dynamic semantic interaction sequence are iteratively updated based on the dialogue interaction feedback to identify the trend of intent evolution. Each intent in the intent candidate set, the evolution trend of the corresponding intent, and the semantic context representation are input into the likelihood estimation network to obtain the likelihood of each intent. Based on the Bayesian probability update method, the confidence of each intent in the intent candidate set is updated by combining the prior probability and likelihood at the current time. The order and priority of each intent in the intent candidate set are dynamically adjusted by the confidence adjustment rule. The semantic detection and output unit is used to build an NPC semantic cognition model. It detects conflicts between user input and NPC semantic state through semantic consistency verification and triggers adaptive dialogue adjustment. It combines the final intent and decision engine to generate dialogue strategy and output synchronized natural language response, facial animation and speech prosody.

Images