Multi-modal interactive intelligent NPC dialogue intention matching method and system

Abstract

The invention relates to the technical field of natural language processing, in particular to a multi-modal interactive intelligent NPC dialogue intention matching method and system. The method comprises the following steps: collecting multi-modal data in real time, and generating multi-modal semantic features through sub-modal preprocessing; a cross-modal fusion module based on semantic association is adopted to integrate each modal semantic feature, and an intention candidate set is generated based on semantic context representation and in combination with a semantic analysis module and a predefined intention template; through a continuous semantic learning mechanism and an interactive memory module, in combination with a Bayesian updating method, tracking user intention change in real time, dynamically adjusting the confidence of each intention in the intention candidate set, and screening a final intention; constructing an NPC semantic cognition model, and performing semantic check on user input and an NPC dialogue state through semantic consistency analysis; generating a dialogue strategy in combination with the final intention and a decision engine, and outputting synchronous response content; according to the invention, the precision of intelligent dialogue intention dynamic matching is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, in particular to a multi-modal interactive intelligent NPC dialogue intent matching method and system. BACKGROUND

[0002] In the field of natural language processing (NLP), building a system that can have a smooth and deep dialogue with humans is one of its core goals; the current automated dialogue system, especially in the context of continuous, context-aware interaction, has the following three levels of limitations in its data processing method: 1) The singularity of the input data source limits the completeness of semantic understanding Traditional dialogue systems usually take the user's language (whether it is speech or text) as the only data source when processing input. For example, after speech input is converted into text through automatic speech recognition (ASR), important features such as intonation, emotion, and speech rate contained in the original acoustic information are discarded; this processing method reduces the rich and multi-dimensional communication signal to a single text data stream, directly leading to a serious loss of information in the subsequent semantic processing and analysis link.

[0003] 2) The semantic intent matching method lacks dynamic adaptability Current mainstream intent recognition technology, as a key link of natural language understanding (NLU), usually adopts a static mapping mechanism. The system establishes a mapping relationship from a specific text pattern to a predefined intent label through supervised learning in the offline stage; however, in a real continuous dialogue, the user's intent is fluid, may evolve, and even new intents that have not been seen during training may occur; the existing data processing flow cannot effectively handle the dynamic drift and out-of-domain intents of the intent.

[0004] 3) The context dependence of dialogue management and response generation strategy is shallow After determining the user's intent, the dialogue management module that generates responses often relies on a simple state machine or shallow context information, which leads to natural language replies that are grammatically correct but may not be consistent with deep dialogue history, user's implied emotional state or long-term goals, resulting in poor coherence and weak logic in the interaction; the existing method fails to effectively integrate the user's state changes reflected by multi-round historical interactions, multi-modal information, and a long-term dialogue goal into its response generation data processing flow, resulting in the system's inability to conduct deep and coherent dialogue.

[0005] Therefore, a multi-modal interactive intelligent NPC dialogue intent matching method and system are proposed. SUMMARY

[0006] The purpose of the present invention is to provide a multimodal interactive intelligent NPC dialogue intention matching method and system.

[0007] To achieve the above object, the present invention provides the following technical solutions: A multimodal interactive intelligent NPC dialogue intention matching method, comprising: Collect users' multimodal data in real time, perform modal preprocessing on each modal data, and generate multimodal semantic features; Through the cross-modal fusion module based on semantic association, the semantic features of each modality are integrated to generate a unified semantic context representation; Based on the semantic context representation, the semantic parsing module and predefined semantic intent templates are used to obtain a set of intent candidates. Through the continuous semantic learning mechanism and interactive memory module, combined with the Bayesian update method, the user's intent changes are tracked in real time, and the confidence level of each intent in the intent candidate set is dynamically adjusted. The final intent is screened based on the preset confidence threshold. Build an NPC semantic cognition model and detect potential conflicts between user input and the NPC's current semantic state through semantic consistency analysis. When a semantic conflict is detected, the adaptive adjustment mechanism of the dialogue strategy is triggered, combining the final intent and the decision engine to generate a dialogue strategy and output synchronized natural language responses, expression animations, and speech rhythm.

[0008] Furthermore, the process of performing modal preprocessing on each modal data and generating multimodal semantic features includes: Perform word segmentation, part-of-speech tagging, and syntactic analysis on text input to form text semantic features; Perform acoustic analysis on speech input to extract pitch, speech rate and emotional prosody features to form speech semantic features; Perform emotion classification on facial expression input, identify emotional state, and form expression semantic features; Perform action pattern recognition on gesture input, extract action direction and intensity, and form gesture semantic features; Through semantic normalization, each semantic feature is mapped to a unified semantic representation space to generate multimodal semantic features.

[0009] Furthermore, the semantic features of each modality are integrated through a cross-modal fusion module based on semantic association to generate a unified semantic context representation. The process includes: Through the semantic attention allocation mechanism, the semantic relevance weights between the semantic features of each modality are calculated to generate the semantic association vector between the modalities; Based on the semantic routing rules of the cross-modal capsule network, the semantic features of each modality are dynamically grouped and weighted according to the semantic association vector to form a multi-level semantic capsule representation; The semantic capsule representation is semantically decoded and smoothed to generate a unified semantic context representation.

[0010] Further, through the continuous semantic learning mechanism and the interactive memory module, and in combination with the Bayesian updating method, the process of tracking the user intention change in real time and dynamically adjusting the confidence of each intention in the intention candidate set includes: Through the interactive memory module, the semantic context representation of the current dialogue and the historical interactive semantic pattern are recorded to form a dynamic semantic interaction sequence; the continuous semantic learning mechanism is used to iteratively update the semantic pattern in the dynamic semantic interaction sequence according to the dialogue interaction feedback to identify the intention evolution trend; The likelihood estimation network is input with each intention in the intention candidate set, the evolution trend of the corresponding intention, and the semantic context representation to obtain the likelihood of each intention. Based on the Bayesian probability updating method, the confidence of each intention in the intention candidate set is updated in combination with the prior probability and the likelihood at the current time; and the ranking and priority of each intention in the intention candidate set are dynamically adjusted through the confidence adjustment rule.

[0011] Further, the process of detecting potential conflicts between the user input and the current semantic state of the NPC through semantic consistency analysis includes: The semantic context representation obtained by integrating the user input is compared with the NPC current semantic state representation in terms of semantic similarity; Through the semantic consistency verification rule, the matching degree of the user semantic features and the NPC semantic state is analyzed to detect potential semantic conflicts. When a semantic conflict is detected, the adaptive dialogue adjustment mechanism is triggered to optimize the dialogue strategy.

[0012] Further, the process of generating a dialogue strategy in combination with the final intention and the decision engine, and outputting a synchronous natural language reply, an expression animation, and a speech rhythm includes: Based on the final intention, the semantic context representation, and the NPC semantic state, the dialogue strategy rule is optimized through reinforcement learning to generate a dialogue update strategy; The natural language generation rule is used to generate a natural language reply according to the dialogue update strategy and the NPC role setting; An expression generation rule is used to generate a matching expression animation according to the final intention and emotional semantics; The speech rhythm is adjusted based on the speech synthesis rule and is output synchronously with the natural language reply and the expression animation.

[0013] An intelligent NPC dialogue intention matching system for multi-modal interaction includes: A data acquisition and processing unit is configured to acquire multi-modal data of a user in real time, pre-process the multi-modal data in a mode-specific manner, and generate multi-modal semantic features. a semantic fusion unit configured to integrate the semantic features of each modality by a cross-modal fusion module based on semantic association to generate a unified semantic context representation; an intention matching unit configured to obtain an intention candidate set by using a semantic parsing module and a predefined semantic intention template based on the semantic context representation, track changes in user intention in real time by a continuous semantic learning mechanism and an interactive memory module, dynamically adjust the confidence of each intention in the intention candidate set by a Bayesian updating method, and filter out a final intention according to a preset confidence threshold; a semantic detection and output unit configured to construct an NPC semantic cognition model, detect conflicts between user input and an NPC semantic state by semantic consistency verification, trigger adaptive dialogue adjustment, generate a dialogue strategy in combination with the final intention and a decision engine, and output a synchronous natural language reply, an expression animation, and a speech prosody.

[0014] Compared with the prior art, the present application has the following advantages: 1. The semantic relevance weight between modes is calculated by a semantic attention distribution mechanism to highlight the contribution of important modes and improve the accuracy of the semantic representation after fusion. Dynamic grouping and weighted integration are realized by a semantic routing rule based on a cross-modal capsule network to form a multi-level semantic capsule representation, which retains the complex semantic relationship between modes and improves the richness and expression ability of the semantic context representation. The unified semantic context representation generated by semantic decoding and smoothing processing is more coherent, reducing information loss in modal fusion and enhancing the reliability of subsequent intention recognition and matching.

[0015] 2. The interactive memory module is combined with the continuous semantic learning mechanism to identify the intention evolution trend, enabling the NPC to adapt to dynamic changes in intention and improve the context relevance of the interaction. The confidence of each intention in the intention candidate set is updated in real time and the ranking and priority of the intention candidate set are dynamically adjusted by a likelihood estimation network and a Bayesian probability updating method, ensuring the accuracy of the intention matching result.

[0016] 3. The potential conflict between user input and the current semantic state of the NPC is accurately detected by semantic similarity comparison and consistency verification rules to avoid logical errors or semantic deviations in the dialogue. The adaptive dialogue adjustment mechanism is triggered when a semantic conflict is detected to optimize the dialogue strategy, reduce misunderstandings and unnatural dialogue responses, and improve the coherence of the interaction and user satisfaction. The dialogue strategy rules are optimized based on reinforcement learning, and a dialogue update strategy is generated in combination with the final intention and the semantic context representation to ensure that the NPC response meets the user intention and scene requirements. BRIEF DESCRIPTION OF DRAWINGS

[0017] Fig. 1A flowchart of a multi-modal interactive intelligent NPC dialogue intention matching method of the present application is shown in the figure; Fig. 2 A flowchart of generating dialogue update strategies of the present application is shown in the figure; Fig. 3 A structural diagram of a multi-modal interactive intelligent NPC dialogue intention matching system of the present application is shown in the figure. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0019] Please refer to Figs. 1 to 3 The present application provides a multi-modal interactive intelligent NPC dialogue intention matching method and system, and the technical solutions are as follows: Embodiment one: In order to improve the accuracy of intelligent NPC dialogue intention dynamic matching, an enterprise uses a multi-modal interactive intelligent NPC dialogue intention matching method proposed by the present application. The flowchart of the method is shown in the figure Fig. 1 and specifically includes: Real-time collection of multi-modal data of users, modal pre-processing of each modal data, and generation of multi-modal semantic features; Further, the multi-modal data includes text data, voice data, facial expression data and gesture data. The text data and voice data are collected by text input devices (such as keyboard, touch screen, etc.) and microphones, respectively, and the facial expression data and gesture data are collected by cameras; Further, the process of modal pre-processing of each modal data to generate multi-modal semantic features includes: Text input is segmented, tagged with part of speech, and analyzed for syntax to form text semantic features; Acoustic analysis is performed on voice input to extract pitch, speech rate and emotional prosody features to form voice semantic features; Emotion classification is performed on facial expression input to identify emotional states to form expression semantic features; Action pattern recognition is performed on gesture input to extract action direction and intensity to form gesture semantic features; Through semantic normalization processing, each semantic feature is mapped to a unified semantic representation space to generate multi-modal semantic features; Further, the Stanford CoreNLP tool is used for word segmentation and part-of-speech tagging of the text, and then the processing results are input into the pre-trained BERT-Large model for analysis and feature extraction to obtain text semantic features; the model is trained on BookCorpus and Wikipedia datasets, and the parameter settings are as follows: batch size is 16, AdamW optimizer is used, and learning rate is set to 2e-5; Further, the pre-trained Wav2Vec 2.0 model is directly used for feature analysis of the speech data to extract pitch, speech rate and emotional prosody features to generate speech semantic features; the speech data needs to be preprocessed before input to improve computational efficiency, including: down-sampling to 16kHz, frame length of 25ms, and frame shift of 10ms; Further, the ResNet-50 backbone network of the pre-trained DeepFace model is used to identify the state in the expression image, and the expression semantic features are extracted through the fully connected layer; the DeepFace model is trained on the FER2013 and AffectNet datasets, the batch size is 32, and the learning rate is set to 3e-4; Further, an independent fully connected layer is used to map each modality feature to a unified 512-dimensional semantic space, and the activation function is set to ReLU and layer normalization operation is added.

[0020] Through the modality-specific preprocessing of text, speech, expression and gesture, the semantic features of each modality are extracted to ensure the comprehensiveness and accuracy of subsequent semantic analysis; through semantic normalization processing, the semantic features of different modalities are mapped to a unified semantic representation space, eliminating the heterogeneity between modalities and providing high-quality input for subsequent cross-modal fusion.

[0021] Through the cross-modal fusion module based on semantic association, the semantic features of each modality are integrated to generate a unified semantic context representation; Further, the process of generating a unified semantic context representation by integrating the semantic features of each modality through the cross-modal fusion module based on semantic association includes: Through the semantic attention allocation mechanism, the semantic correlation weights between the semantic features of each modality are calculated to generate semantic association vectors between modalities; Based on the semantic routing rules of the cross-modal capsule network, the semantic features of each modality are dynamically grouped and weighted integrated according to the semantic association vectors to form a multi-level semantic capsule representation; The semantic capsule representation is subjected to semantic decoding and smoothing processing to generate a unified semantic context representation; Further, the semantic attention allocation mechanism is a multi-head attention mechanism, realized through a Transformer layer; the generation process of the semantic correlation vector includes: splicing the features of each modality to form a matrix; converting the spliced matrix into a query matrix, a key matrix and a value matrix through linear transformation, obtaining a semantic correlation weight matrix using an attention weight calculation formula, which contains the semantic correlation weights between the semantic features of each modality; the semantic correlation weight matrix is applied to the value matrix and converted into a semantic correlation vector; the multi-head attention mechanism is set as follows: the number of attention heads is 8, each head processes 64-dimensional data, the total dimension is 512, and the Dropout rate for preventing overfitting is set to 0.1; Further, in order to improve the semantic differentiation ability of the attention weight and enhance the semantic alignment between the modalities, a contrastive learning is introduced in the semantic attention allocation process to optimize the semantic correlation weights of the multi-head attention mechanism, so that the semantic alignment between the modalities is more accurate; the modality features of the same dialogue and the modality features of different dialogues are taken as positive samples and negative samples respectively, and the InfoNCE loss is used to minimize the semantic distance of the positive samples and maximize the distance of the negative samples; Further, the features weighted by attention, i.e. the semantic correlation vector, are input into the cross-modal capsule network, which maps the input features to the capsule space using a transformation matrix, divides the multi-modal features into 32 capsules, and each capsule contains a 16-dimensional vector; then, the dynamic routing algorithm is used to iteratively optimize the routing coefficients to ensure the accuracy of the semantic grouping; then, the routing coefficients are used to weight the mapping results of the transformation matrix to generate semantic capsule representations; wherein the routing iteration number in the process is set to 3, and the Squash function is applied to the weighted results, so that the range of the semantic capsule representation is between 0 and 1; Further, the 32 semantic capsule representations are spliced into a 512-dimensional long vector, which is processed by a two-layer fully connected network to obtain a 512-dimensional semantic context representation; the first layer of the fully connected network reduces the dimension from 512 to 256, and uses the ReLU activation function to remove negative values, and then uses the second layer of the fully connected network to increase the dimension to the original 512.

[0022] The semantic correlation weights between the modalities are calculated by the semantic attention allocation mechanism to highlight the contribution of important modalities and improve the accuracy of the fused semantic representation; the dynamic grouping and weighted integration are realized by the semantic routing rules based on the cross-modal capsule network to form multi-level semantic capsule representations, which improves the richness and expression ability of the semantic context representation; through semantic decoding and smoothing processing, the information loss in modality fusion is reduced, and the reliability of subsequent intent recognition matching is enhanced.

[0023] Based on the semantic context representation, the semantic analysis module and the predefined semantic intent template are used to obtain an intent candidate set; through a continuous semantic learning mechanism and an interactive memory module, combined with a Bayesian updating method, the user's intent change is tracked in real time, and the confidence of each intent in the intent candidate set is dynamically adjusted; according to a preset confidence threshold, a final intent is screened out; Further, the semantic context representation is processed using a pre-trained Seq2Seq model to convert into a structured semantic representation, generally in a logical form; the predefined semantic intent template includes an intent type and related slots; then, the logical form output by the semantic analysis module is matched with the predefined intent template; next, the slot values are extracted from the logical form and filled into the matched intent template, which is taken as the intent candidate set; Further, the process of tracking the user's intent change in real time and dynamically adjusting the confidence of each intent in the intent candidate set through the continuous semantic learning mechanism and the interactive memory module, combined with the Bayesian updating method, includes: Through the interactive memory module, the semantic context representation of the current dialogue and the historical interactive semantic pattern are recorded to form a dynamic semantic interactive sequence; the semantic pattern in the dynamic semantic interactive sequence is iteratively updated according to the dialogue interaction feedback using the continuous semantic learning mechanism to identify the intent evolution trend; The likelihood estimation network is input with each intent in the intent candidate set, the corresponding intent evolution trend and the semantic context representation to obtain the likelihood of each intent; Based on the Bayesian probability updating method, the confidence of each intent in the intent candidate set is updated based on the prior probability and the likelihood at the current time; through the confidence adjustment rule, the ranking and priority of each intent in the intent candidate set are dynamically adjusted; The results before and after the intent confidence adjustment under three different dialogues are given below, which are denoted as dialogue one, dialogue two and dialogue three respectively; in order to facilitate the display of the adjustment results, one intent is selected from the intent candidate set in each dialogue for display, and the corresponding intents are inquiry task, confirmation reward and expression of emotion respectively; the results before and after the confidence adjustment are shown in Table 1.

[0024] Table 1 Results before and after confidence adjustment Further, the interactive memory module adopts a bidirectional LSTM network and is trained using the MultiWOZ dialogue dataset, with an initial learning rate of 1e-4, a Dropout rate of 0.2, and a tanh function used as the activation function; the current semantic context representation and the historical semantic state vector are input into the interactive memory module as inputs, and the current semantic state vector and the dynamic semantic interaction sequence are output; the historical semantic state vector is obtained by inputting the historical semantic context representation into the interactive memory module and is used to record the historical semantic pattern; the dynamic semantic interaction sequence is a time sequence representing the semantic pattern change of the dialogue from the beginning to the current time; Further, the cross-entropy loss is calculated according to the dialogue interaction feedback of the user and the intelligent NPC, and the online gradient descent algorithm is used to update the parameters of the bidirectional LSTM network. Further, in order to reduce the number of interaction rounds required by new users and accelerate personalized semantic learning, online intent transfer learning is introduced in the continuous semantic learning process, and the semantic patterns shared by cross-user dialogue data are used to accelerate the learning of the intent evolution trend of the bidirectional LSTM; specifically, a cross-user dynamic semantic interaction sequence pool with a capacity of 10,000 is maintained to store the semantic interaction sequences of other users; the cosine similarity is used to match the current user sequence with the sequences in the pool; the gradient of the matched sequence is integrated into the parameter update of the bidirectional LSTM model through a weighting method, thereby improving the efficiency of personalized adaptation; Further, the likelihood estimation network adopts a three-layer fully connected network, and each intent in the intent candidate set, the semantic state vector, and the semantic context representation are input into the network as inputs, and the likelihood of each intent is output; the prior probability and the likelihood are combined to apply the Bayes theorem to each intent to update its posterior probability, i.e., the confidence; the prior probability is the posterior probability at the previous time; if the intent is newly added or first interacted, an initial prior probability based on heuristic rules is assigned to it; Further, the updated confidence is normalized to generate a priority weight, which is used to adjust the ranking and priority of each intent in the intent candidate set.

[0025] By combining the interactive memory module with the continuous semantic learning mechanism, the intent evolution trend is recognized, enabling the NPC to adapt to the dynamic changes of the intent and improving the context relevance of the interaction; through the likelihood estimation network and the Bayes probability updating method, the confidence of each intent in the intent candidate set is updated in real time, and the ranking and priority of the intent candidate set are dynamically adjusted, ensuring the accuracy of the intent matching result.

[0026] The NPC semantic cognition model is constructed, potential conflicts between the user input and the current semantic state of the NPC are detected through semantic consistency analysis, when the semantic conflicts are detected, the adaptive adjustment mechanism of the dialogue strategy is triggered, the dialogue strategy is generated combining the final intention and the decision engine, and the synchronous natural language reply, expression animation and voice rhythm are output.

[0027] Further, the BDI model is used as the NPC semantic cognition model to model the cognition state of the NPC, including belief, desire and intention; the belief is expressed by the propositional logic expression to express the cognition of the world, including the knowledge about the environment and the state of itself; the desire is the long-term goal of the NPC, which is represented by the utility function; the intention represents the action or sub-goal to be achieved that the NPC is committed to perform; Further, the process of detecting potential conflicts between the user input and the current semantic state of the NPC through semantic consistency analysis includes: The semantic context representation obtained by integrating the user input is compared with the current semantic state representation of the NPC in terms of semantic similarity; The matching degree of the user semantic features and the NPC semantic state is analyzed through the semantic consistency checking rule, and potential semantic conflicts are detected; When the semantic conflicts are detected, the adaptive dialogue adjustment mechanism is triggered to optimize the dialogue strategy; Further, the user semantic state vector is obtained combining the interactive memory module and the semantic context representation; then, the cosine similarity of the user semantic state vector and the current semantic state vector of the NPC is calculated, and threshold comparison is performed, and the semantic similarity threshold is set to 0.85; Further, the semantic consistency checking rule is: if the semantic similarity is lower than the threshold, it is considered that there is a semantic conflict; if the intention type of the user does not match the intention type of the NPC, it is considered that there is a semantic conflict; if the slot information provided by the user is inconsistent with the slot information collected by the NPC, it is considered that there is a semantic conflict; Further, the semantic consistency checking includes not only the semantic similarity judgment but also the emotion consistency and context coherence checking; wherein, the emotion consistency checking is realized by extracting the user emotion vector from the semantic context representation and calculating the cosine similarity between the user emotion vector and the NPC emotion state; the context coherence checking is realized by inputting the interactive memory sequence into the fully connected network for coherence prediction; Further, the adaptive dialogue adjustment mechanism adopts the reinforcement learning strategy.

[0028] Through the semantic similarity comparison and the consistency checking rule, the potential conflicts between the user input and the current semantic state of the NPC are accurately detected, the logical errors or semantic deviations in the dialogue are avoided, and the judgment basis is provided for the subsequent dialogue strategy optimization.

[0029] Further, the process of generating a dialogue strategy in combination with the final intent and decision engine, and outputting a synchronized natural language reply, expression animation and voice prosody includes: Based on the final intent, semantic context representation and NPC semantic state, dialogue update strategy is generated by optimizing dialogue strategy rules through reinforcement learning; Natural language reply is generated by using natural language generation rules and according to dialogue update strategy and NPC role setting; Expression animation matching the final intent and emotional semantics is generated through expression generation rules; Voice prosody is adjusted based on speech synthesis rules, and is outputted synchronously with natural language reply and expression animation; Further, the state space of reinforcement learning contains user intent, user semantic context and NPC cognitive state; the action space is a discrete set containing all combinations of dialogue strategy and accompanying behavior; the reward function is composed of user satisfaction (user feedback score) and semantic consistency (semantic similarity); Further, the generation process of dialogue update strategy can refer to Fig. 2 , which is specifically: a deep neural network is constructed, and a target network and an experience replay buffer are defined; a greedy strategy is used to select actions, the selected actions are executed and the feedback of the environment is observed, and experience tuples (state, action, reward, next state and whether to end) are stored in the experience replay buffer; the deep Q network uses randomly sampled experiences in the experience replay buffer for training, calculates Q values and loss functions, and updates the parameters of the deep Q network using gradient descent to minimize the loss; periodically copy the parameters of the deep Q network to the target network until the training is completed, select the action with the highest Q value and map it to the dialogue strategy rule to obtain the dialogue update strategy; Further, natural language reply is generated by using a pre-trained T5-Large model, and prompt engineering technology is used to generate rules, dialogue update strategy and NPC role setting to guide the model to generate language replies that meet the expectations; based on preset expression rules, the final intent and emotional semantics are mapped to expressions, and then animation mixing technology is used to create rich expression animations; a pre-trained WaveNet model is used to convert text into speech, and the prosody of the speech is adjusted according to the emotional label and context label.

[0030] By using reinforcement learning to optimize dialogue strategy rules and generating dialogue update strategy in combination with final intent and semantic context representation, it is ensured that the response of NPC meets the user intent and scene requirements, while reducing misunderstandings and unnatural dialogue responses, and improving the coherence of interaction and user satisfaction.

[0031] The embodiment proposes a multi-modal interactive intelligent NPC dialogue intention matching method; the method collects multi-modal data in real time, generates multi-modal semantic features through modal pre-processing, integrates modal semantic features through a cross-modal fusion module based on semantic association, generates an intention candidate set based on semantic context representation and in combination with a semantic analysis module and a predefined intention template, tracks user intention changes in real time through a continuous semantic learning mechanism and an interactive memory module, dynamically adjusts the confidence of each intention in the intention candidate set in combination with a Bayesian updating method, and screens a final intention; an NPC semantic cognition model is constructed, semantic checking is performed on user input and NPC dialogue state through semantic consistency analysis; a dialogue strategy is generated in combination with the final intention and a decision engine, and synchronous response content is output; the precision of intelligent dialogue intention dynamic matching is improved.

[0032] Embodiment two The application also proposes a multi-modal interactive intelligent NPC dialogue intention matching system, the structure of the system is shown in Fig. 3 The application also proposes a multi-modal interactive intelligent NPC dialogue intention matching system, the structure of the system is shown in The data acquisition and processing unit is used for collecting multi-modal data of the user in real time, performing modal pre-processing on each modal data, and generating multi-modal semantic features; The semantic fusion unit is used for integrating each modal semantic feature through a cross-modal fusion module based on semantic association, and generating a unified semantic context representation; Further, the process of integrating each modal semantic feature through a cross-modal fusion module based on semantic association to generate a unified semantic context representation includes: The semantic correlation weight between each modal semantic feature is calculated through a semantic attention distribution mechanism, and a semantic association vector between modes is generated; The semantic routing rules based on the cross-modal capsule network are used to dynamically group and weight integrate each modal semantic feature according to the semantic association vector, and form a multi-level semantic capsule representation; The semantic capsule representation is subjected to semantic decoding and smoothing processing to generate a unified semantic context representation.

[0033] The intention matching unit is used for obtaining an intention candidate set based on the semantic context representation, using a semantic analysis module and a predefined semantic intention template, tracking user intention changes in real time through a continuous semantic learning mechanism and an interactive memory module, dynamically adjusting the confidence of each intention in the intention candidate set in combination with a Bayesian updating method, and screening a final intention according to a preset confidence threshold; Further, the process of tracking user intention changes in real time through a continuous semantic learning mechanism and an interactive memory module, and dynamically adjusting the confidence of each intention in the intention candidate set in combination with a Bayesian updating method includes: By the interactive memory module, the semantic context representation of the current dialogue and the historical interactive semantic pattern are recorded to form a dynamic semantic interactive sequence; the semantic pattern in the dynamic semantic interactive sequence is iteratively updated according to the dialogue interactive feedback by using a continuous semantic learning mechanism to identify the intention evolution trend; Each intention in the intention candidate set, the evolution trend of the corresponding intention and the semantic context representation are input into the likelihood estimation network to obtain the likelihood of each intention. Based on the Bayesian probability updating method, the confidence of each intention in the intention candidate set is updated in combination with the prior probability and the likelihood at the current time; the ranking and priority of each intention in the intention candidate set are dynamically adjusted by the confidence adjustment rule.

[0034] The semantic detection and output unit is used to construct an NPC semantic cognition model, detect the conflict between the user input and the NPC semantic state through semantic consistency verification, trigger adaptive dialogue adjustment, generate a dialogue strategy in combination with the final intention and the decision engine, and output a synchronous natural language reply, an expression animation and a voice prosody.

[0035] The performance comparison results under the conditions of single mode and multi-mode fusion and with or without confidence dynamic adjustment are given as follows: as shown in Table 2, it can be seen that the intention matching accuracy and the semantic consistency can be improved by using the scheme of the present application.

[0036] Table 2 Performance comparison results Although the embodiments of the present application have been shown and described, it can be understood by those of ordinary skill in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A multimodal interactive intelligent NPC dialogue intention matching method, characterized in that: include: Collect users' multimodal data in real time, perform modal preprocessing on each modal data, and generate multimodal semantic features; Through the cross-modal fusion module based on semantic association, the semantic features of each modality are integrated to generate a unified semantic context representation; Based on the semantic context representation, the semantic parsing module and predefined semantic intent templates are used to obtain a set of intent candidates. Through the continuous semantic learning mechanism and interactive memory module, combined with the Bayesian update method, the user's intent changes are tracked in real time, and the confidence level of each intent in the intent candidate set is dynamically adjusted. The final intent is screened based on the preset confidence threshold. Build an NPC semantic cognition model and detect potential conflicts between user input and the NPC's current semantic state through semantic consistency analysis; When a semantic conflict is detected, the adaptive adjustment mechanism of the dialogue strategy is triggered, combining the final intent and the decision engine to generate a dialogue strategy, and output synchronized natural language responses, expression animations, and voice rhythm.

2. A multimodal interactive intelligent NPC dialogue intention matching method according to claim 1, characterized in that: The process of performing modal preprocessing on each modal data and generating multimodal semantic features includes: Perform word segmentation, part-of-speech tagging, and syntactic analysis on text input to form text semantic features; Perform acoustic analysis on speech input to extract pitch, speech rate and emotional prosody features to form speech semantic features; Perform emotion classification on facial expression input, identify emotional state, and form expression semantic features; Perform action pattern recognition on gesture input, extract action direction and intensity, and form gesture semantic features; Through semantic normalization, each semantic feature is mapped to a unified semantic representation space to generate multimodal semantic features.

3. The method for matching multimodal interactive intelligent NPC conversation intentions according to claim 1, characterized in that: The process of integrating the semantic features of each modality through a cross-modal fusion module based on semantic association to generate a unified semantic context representation includes: Through the semantic attention allocation mechanism, the semantic relevance weights between the semantic features of each modality are calculated to generate the semantic association vector between the modalities; Based on the semantic routing rules of the cross-modal capsule network, the semantic features of each modality are dynamically grouped and weighted according to the semantic association vector to form a multi-level semantic capsule representation; Semantic capsule representations are semantically decoded and smoothed to generate a unified semantic context representation.

4. The method for matching multimodal interactive intelligent NPC conversation intentions according to claim 1, characterized in that: Through the continuous semantic learning mechanism and interactive memory module, combined with the Bayesian update method, the process of tracking user intent changes in real time and dynamically adjusting the confidence of each intent in the intent candidate set includes: The interactive memory module records the semantic context representation of the current conversation and the historical interaction semantic patterns to form a dynamic semantic interaction sequence. The continuous semantic learning mechanism iteratively updates the semantic patterns in the dynamic semantic interaction sequence based on the feedback from the conversation interaction to identify the evolution trend of intent. Each intent in the candidate set, its evolution trend, and its semantic context representation are input into the likelihood estimation network to obtain the likelihood of each intent. Based on the Bayesian probability update method, the confidence of each intent in the intent candidate set is updated in combination with the prior probability and likelihood at the current moment; through the confidence adjustment rules, the sorting and priority of each intent in the intent candidate set are dynamically adjusted.

5. The method for matching multimodal interactive intelligent NPC conversation intentions according to claim 1, characterized in that: The process of detecting potential conflicts between user input and the NPC's current semantic state through semantic consistency analysis includes: Compare the semantic similarity between the semantic context representation obtained by integrating the user input and the current semantic state representation of the NPC; Through semantic consistency verification rules, the matching degree between user semantic features and NPC semantic states is analyzed to detect potential semantic conflicts; When semantic conflicts are detected, the adaptive dialogue adjustment mechanism is triggered to optimize the dialogue strategy.

6. The method for matching multimodal interactive intelligent NPC conversation intentions according to claim 1, characterized in that: The process of combining the final intent with the decision engine to generate a dialogue strategy and output synchronized natural language responses, expression animations, and speech prosody includes: Based on the final intent, semantic context representation, and NPC semantic state, the dialogue policy rules are optimized through reinforcement learning to generate a dialogue update strategy. Generate natural language responses using natural language generation rules and based on dialogue update strategies and NPC role settings; Generate matching expression animations based on final intention and emotional semantics through expression generation rules; The speech rhythm is adjusted based on speech synthesis rules and output synchronously with natural language responses and expression animations.

7. A multimodal interactive intelligent NPC dialogue intention matching system, characterized by: include: The data acquisition and processing unit is used to collect the user's multimodal data in real time, perform modal preprocessing on each modal data, and generate multimodal semantic features; The semantic fusion unit is used to integrate the semantic features of each modality through a cross-modal fusion module based on semantic association to generate a unified semantic context representation; The intent matching unit is used to obtain a set of intent candidates based on the semantic context representation, using the semantic parsing module and predefined semantic intent templates; Through the continuous semantic learning mechanism and interactive memory module, combined with the Bayesian update method, it tracks the changes in user intent in real time and dynamically adjusts the confidence of each intent in the intent candidate set; and screens the final intent based on the preset confidence threshold. Semantic detection and output unit, used to build the NPC semantic cognition model, detect conflicts between user input and NPC semantic state through semantic consistency verification, and trigger adaptive dialogue adjustments; The final intent and decision engine are combined to generate a dialogue strategy, and output synchronized natural language responses, expression animations, and voice prosody.

Images