Intelligent agent intention understanding method based on multi-modal information fusion

Abstract

The invention discloses an agent intention understanding method based on multi-modal information fusion, relates to the technical field of natural language processing, and solves the technical problem that multi-agent intention understanding cannot be realized in a dynamic environment. Voice, text and visual features are mapped to the same hidden space through maximum mean value difference minimization, distribution difference between modes is eliminated, joint features in the hidden space can capture cross-mode complementary information at the same time, and the richness and anti-noise capability of feature expression are enhanced. CRF (Conditional Random Field) is combined with a special dictionary in the scheduling field to carry out candidate set expansion on fuzzy keywords, and specific semantic variants in the field are captured. The optimal grammar of the scheduling instruction is analyzed and decomposed into a feature vector causal relationship pair, and components of an intention and logic dependence of the intention are clarified. A dynamic knowledge graph is constructed based on the causal relationship, probability dependence among parameters is quantized by using a Bayesian network, the causal relationship strength is dynamically adjusted, and environmental changes are adapted.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing, specifically a method for understanding the intent of an intelligent agent based on multimodal information fusion. Background Technology

[0002] Agent intent understanding is a core problem in human-computer interaction, multi-agent systems (MAS), and autonomous decision-making within the field of artificial intelligence. Its goal is to infer the agent's intrinsic goals or plans by analyzing the agent's behavior, environmental context, and multimodal inputs, such as speech, text, and vision. Agent intent understanding infers the agent's current or future goals or plans by analyzing input data such as sensor data, communication signals, and historical behavior, combined with environmental context. The agent's behavior may be influenced by various environmental factors, and the same input may correspond to different intents. Therefore, an agent intent understanding method based on multimodal information fusion is needed.

[0003] Traditional agent intent understanding schemes directly map inputs to intent labels, but cannot explicitly explain the intent generation logic. Each agent independently stores historical experience, lacking unified memory management. At the same time, different agents may repeatedly explore the same task, traversing all historical memory nodes to match the current task, resulting in high computational overhead. The retrieval results may contain a large number of low-relevance memories, causing agent intent understanding to fall into the dilemma of "unable to achieve efficient and interpretable multi-agent intent understanding in dynamic environments". Summary of the Invention

[0004] This invention aims to at least solve one of the technical problems existing in the prior art; to this end, this invention proposes an agent intent understanding method based on multimodal information fusion to solve the technical problems of directly mapping input to intent labels but being unable to explicitly explain the intent generation logic, each agent independently storing historical experience, the same agent may repeatedly explore the same task, and the inability to achieve multi-agent intent understanding in dynamic environments.

[0005] To address the aforementioned problems, a first aspect of the present invention provides a method for understanding the intent of an intelligent agent based on multimodal information fusion, comprising the following steps:

[0006] For multimodal data collected by intelligent agents, modal differences are minimized by MMD (Maximum Mean Difference), and speech features, text features, and visual features are mapped to the same latent space. In the same latent space, CRF (Conditional Random Field) combined with a scheduling domain dictionary is used to expand the candidate set of fuzzy keywords and establish a fuzzy semantic alignment model.

[0007] Define a PCFG probabilistic context-free grammar for scheduling instructions, combine it with the establishment of a fuzzy semantic alignment model, generate possible syntax trees and calculate probabilities, use a Transformer decoder to generate candidate syntax trees, and select the optimal parser through the MDL minimum description length principle.

[0008] Based on the selected optimal parsing, the scheduling instructions are decomposed into causal relationships between feature vector causal pairs, and a Bayesian network is used to model the causal dependencies between parameter pairs to construct a dynamic knowledge graph.

[0009] Record the contextual associations during agent switching, construct a multi-agent global memory graph, use the GAT graph attention network to retrieve the Top-K related memory nodes of relevant historical memories, and record the feature vectors of the Top-K related memory nodes;

[0010] By combining the global memory map of multi-agents and the feature vectors of Top-K related memory nodes, the modal weights of multi-agent nodes are calculated. Based on the modal weights of multi-agent nodes and the global memory map of multi-agents, feature fusion is performed, an intent label prediction model is constructed, and the model prediction is performed to output the intent label and confidence score.

[0011] Optionally, in one example of the above aspects, for multimodal data collected by the agent, modal differences are minimized by the maximum mean difference of MMD, mapping speech features, text features, and visual features to the same latent space, in which the following steps are included:

[0012] The agent collects multimodal data from each dialogue, including: voice data, text data, and visual data;

[0013] Extracting speech feature vectors from speech data: Temporal acoustic feature vectors are processed by 1D-CNN, and LSTM captures fundamental frequency data as emotion feature vectors;

[0014] Extracting text feature vectors from text data: Generating context word vectors using BERT / RoBERTa, and then extracting syntactic feature vectors using BiLSTM;

[0015] Extracting visual feature vectors from visual data: Processing video streams using 3D-CNN to extract spatiotemporal feature vectors of facial movements;

[0016] For speech feature vectors, text feature vectors, and visual feature vectors, a joint embedding space is established to map speech features and text features to the same latent space. Based on the maximum mean difference, the modal difference is minimized, and the MMD maximum mean difference loss function is established to align feature biases.

[0017] Optionally, in one example of the above aspects, a fuzzy semantic alignment model is established by using a CRF (Conditional Random Field) combined with a scheduling domain dictionary to expand the candidate set of fuzzy keywords, including the following steps:

[0018] For the feature vectors aligned by feature bias, the feature vectors aligned by feature bias are arranged into an input sequence set according to the timestamp order of the samples corresponding to the feature vectors. The rationality of expanding fuzzy keywords is evaluated by using a CRF conditional random field combined with a scheduling domain dictionary.

[0019] By combining CRF (Conditional Random Field) with word vectors from the scheduling domain dictionary, word vectors from the scheduling domain dictionary whose rationality for expanding fuzzy keywords with feature vectors in the input sequence set is greater than a threshold are selected to form a candidate set of fuzzy keywords.

[0020] Based on the correspondence between feature vectors in the input sequence set and candidate sets of fuzzy keywords, and the established MMD maximum mean difference loss function, a fuzzy semantic alignment model is established.

[0021] Optionally, in one example of the above aspects, a PCFG probabilistic context-free grammar for scheduling instructions is defined. Combined with the establishment of a fuzzy semantic alignment model, possible syntax trees are generated and their probabilities are calculated. A Transformer decoder is used to generate candidate syntax trees, and the optimal parser is selected based on the minimum description length principle of MDL. This includes the following steps:

[0022] Define a PCFG probabilistic context-free grammar for scheduling instructions. Combined with a fuzzy semantic alignment model, for each dialogue, the input sequence set and candidate set of fuzzy keywords are used to filter the feature vectors of scheduling instructions and label them as parent nodes. A syntax tree of other feature vectors and parent nodes is built. Production rules are used to assign probability parameters to the feature vectors corresponding to the parent nodes to generate scheduling instructions.

[0023] The probability P(T) of generating scheduling instructions from the syntax tree corresponding to the parent node is sorted in descending order. The syntax trees corresponding to a predetermined proportion of P(T) before sorting are selected, and candidate syntax trees are generated using the Transformer decoder. The optimal parser is selected based on the minimum MDL description length principle.

[0024] in, Let len(T) be the optimal syntax tree selected by the MDL principle, err(T) be the description length of the selected syntax tree T, err(T) be the syntax error penalty term of the selected syntax tree T, and λ be the weight parameter.

[0025] Optionally, in one example of the above aspects, the scheduling instructions are decomposed into causal relationships between feature vector causal pairs based on the selected optimal parsing, and a dynamic knowledge graph is constructed by modeling the causal dependencies between parameter pairs using a Bayesian network, including the following steps:

[0026] The optimal syntax tree is selected based on the MDL principle corresponding to the selected optimal parser. The scheduling instruction feature vector of the parent node of the optimal syntax tree is decomposed into causal relationship pairs of feature vectors. A causal relationship graph between the scheduling instruction feature vector of the parent node and other feature vectors is established through a Bayesian network. Causal relationship edges are established between nodes with causal relationships. A dynamic knowledge graph is constructed.

[0027] Optionally, in one example of the above aspects, the following steps are included: recording the contextual associations during agent switching, constructing a multi-agent global memory graph, using a GAT graph attention network to retrieve the Top-K relevant memory nodes of related historical memories, and recording the feature vectors of the Top-K relevant memory nodes:

[0028] Record the contextual associations when switching agents. Based on the dialogue data in different agents during the switching process, construct a dynamic knowledge graph of the dialogue data in different agents. Label the time of agent switching and the type of agent in the constructed dynamic knowledge graph to form a multi-agent global memory graph.

[0029] GAT uses an attention mechanism to associate parent nodes in the dynamic knowledge graph of dialogue data from different agents in a multi-agent global memory graph according to the time sequence of agent switching. It aggregates the feature vectors of parent nodes with the feature vectors of neighboring parent nodes through the GAT graph attention mechanism and updates the feature vectors of the corresponding parent nodes.

[0030] Calculate the similarity score between the parent node and all other nodes in the dynamic knowledge graph of dialogue data from different agents.

[0031] Sort the nodes by similarity score from high to low, select the top few nodes as Top-K related memory nodes, and retain them in the latest dynamic knowledge graph.

[0032] Optionally, in one example of the above aspects, the modal weights of multi-agent nodes are calculated by combining the multi-agent global memory graph and the feature vectors of Top-K related memory nodes, including the following steps:

[0033] Calculate the cosine similarity between the feature vectors of nodes in each dynamic knowledge graph in the multi-agent global memory graph and the feature vectors of the Top-K related memory nodes, and use it as the modal relevance score;

[0034] The modal relevance score is converted into the modal weight of each node in the dynamic knowledge graph using the Softmax function: wsg = node indexed score / sum of the indexed scores of all Top-K nodes, where node indexed score = exp(sg / τ), τ is the temperature coefficient, sg is the modal relevance score of the node, and wsg is the modal weight of the node.

[0035] Optionally, in one example of the above aspects, feature fusion is performed based on the multi-agent node modality weights and the multi-agent global memory graph, and an intent label prediction model is constructed, including the following steps:

[0036] Based on the modal weights of multi-agent nodes, the feature vectors of multi-agent global memory graph nodes, and the feature vectors of the latest dynamic knowledge graph nodes, an overall fusion feature vector and a latest fusion feature vector are generated. Then, according to the intent feature logic rules defined by experts, an overall intent icon is added to the overall fusion feature vector of historical data, and a latest intent label is added to the latest fusion feature vector.

[0037] Building an intent label prediction model:

[0038] The classifier model is constructed by a fully connected layer and a softmax layer. The classifier model is trained by the overall fused feature vector and the latest fused feature vector labeled with labels. The overall intent label of the overall fused feature vector and the latest intent label of the latest fused feature vector are analyzed.

[0039] Define the confidence score calculation rules and calculate the confidence score of the output intent label.

[0040] Optionally, in one example of the above aspects, generating an overall fused feature vector and a latest fused feature vector, and adding an overall intent icon to the overall fused feature vector of historical data and adding a latest intent label to the latest fused feature vector by an intent feature logic rule defined by experts, includes the following steps:

[0041] The feature vectors of the nodes in the multi-agent global memory graph are sorted and spliced ​​together according to the modal weights of the nodes in each dynamic knowledge graph in the multi-agent global memory graph from high to low to generate the overall fused feature vector.

[0042] The feature vectors of the latest dynamic knowledge graph nodes are sorted and concatenated according to the modal weights of the nodes in the latest dynamic knowledge graph from high to low to generate the latest fused feature vector.

[0043] The overall intent label is set for the multi-agent dialogue record by the logical rules of intent features defined by experts, and the overall intent label is marked on the overall fusion feature vector corresponding to the dialogue record.

[0044] In multi-agent dialogue, the latest intent label is set for the latest dialogue record, and the latest fused feature vector corresponding to the dialogue record is labeled with the latest intent label.

[0045] Optionally, in one example of the above aspects, a confidence calculation rule is defined to calculate the confidence of the output intent label, including the following steps:

[0046] Filter historical data where the overall intent label of the overall fusion feature vector is the same as the overall intent label output by the current classifier model, calculate the maximum cosine similarity between the corresponding overall fusion feature vector and the current overall fusion feature vector, and use the maximum cosine similarity as the confidence score of the output intent label.

[0047] Select data from historical data whose latest intent label is the same as the latest intent label output by the current classifier model. Calculate the maximum cosine similarity between the latest fused feature vector and the current latest fused feature vector. Take a weighted average of the maximum cosine similarity and the confidence of the intent label of the corresponding overall fused feature vector. The result of the weighted average is used as the confidence of the output intent label.

[0048] Compared with the prior art, the beneficial effects of the present invention are:

[0049] This invention maps speech, text, and visual features to the same latent space by minimizing the maximum mean difference, eliminating distributional differences between modalities and solving the "modal gap" problem in traditional multimodal fusion. Joint features in the latent space can simultaneously capture complementary information across modalities, enhancing the richness of feature representation and its robustness to noise. Utilizing a Conditional Random Field (CRF) combined with a domain-specific dictionary for scheduling, the candidate set for fuzzy keywords is expanded through CRF + scheduling domain dictionary, capturing domain-specific semantic variations.

[0050] This invention transforms intent understanding into a structured parsing problem by defining a PCFG probabilistic context-free grammar for scheduling instructions, generating possible syntax trees, and calculating their probabilities. PCFG constrains the parsing space through grammatical rules, avoiding meaningless combinations and improving parsing rationality. When generating candidate syntax trees, the self-attention mechanism of Transformer is used to capture long-distance dependencies. The optimal parser is selected based on the minimum description length principle, achieving a balance between model complexity and data fitting capability.

[0051] This invention decomposes scheduling instructions into feature vector causal relationship pairs through optimal syntax parsing, clearly defining the components of intent and their logical dependencies. These causal relationship pairs intuitively demonstrate the intent generation logic, facilitating manual verification and debugging. By explicitly modeling causal dependencies, the model can distinguish core intent from interfering information, improving robustness in noisy environments. A dynamic knowledge graph is constructed based on these causal relationship pairs, using a Bayesian network to quantify the probabilistic dependencies between parameters. The Bayesian network supports online updates, dynamically adjusting the strength of causal relationships to adapt to environmental changes. Probabilistic modeling quantifies the uncertainty in intent understanding, providing confidence support for decision-making. A multi-agent global memory graph is constructed to record the context during agent switching, avoiding information silos.

[0052] This invention utilizes a GAT (Graph Attention Network) to retrieve the Top-K most relevant memory nodes from a global memory graph and extract their feature vectors. GAT focuses on key memories through attention weights, avoiding interference from irrelevant information. Compared to traversing all nodes, Top-K retrieval significantly reduces computational overhead and supports real-time decision-making. Combining the global memory graph and Top-K memory node features, multi-agent node modal weights are calculated, and the modality fusion ratio is dynamically adjusted. Modality weights are automatically assigned based on the task context. Low-relevance modalities are suppressed through weighting, improving feature representation efficiency. Based on the fused multi-modal features and modality weights, an intent label prediction model is constructed, outputting intent labels and confidence scores. By combining causal relationships, historical memory, and dynamic modality fusion, the model can capture complex intents. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0055] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Please see Figure 1 The first aspect of this invention provides a method for understanding the intent of an intelligent agent based on multimodal information fusion, comprising the following steps:

[0057] For the multimodal data collected by the agent, the modal difference is minimized by the maximum mean difference of MMD, and the speech features, text features and visual features are mapped to the same latent space. In the same latent space, the candidate set of fuzzy keywords is expanded by using a conditional random field (CRF) combined with a scheduling domain dictionary, and a fuzzy semantic alignment model is established. In this embodiment, the agent collects multimodal data authorized by the user, including speech data, text data and visual data.

[0058] Define a PCFG probabilistic context-free grammar for scheduling instructions, combine it with the establishment of a fuzzy semantic alignment model, generate possible syntax trees and calculate probabilities, use a Transformer decoder to generate candidate syntax trees, and select the optimal parser through the MDL minimum description length principle.

[0059] Based on the selected optimal parsing, the scheduling instructions are decomposed into causal relationships between feature vector causal pairs, and a Bayesian network is used to model the causal dependencies between parameter pairs to construct a dynamic knowledge graph.

[0060] Record the contextual associations during agent switching, construct a multi-agent global memory graph, use the GAT graph attention network to retrieve the Top-K related memory nodes of relevant historical memories, and record the feature vectors of the Top-K related memory nodes;

[0061] By combining the global memory map of multi-agents and the feature vectors of Top-K related memory nodes, the modal weights of multi-agent nodes are calculated. Based on the modal weights of multi-agent nodes and the global memory map of multi-agents, feature fusion is performed, an intent label prediction model is constructed, and the model prediction is performed to output the intent label and confidence score.

[0062] Specifically, in this embodiment, speech, text, and visual features are mapped to the same latent space by minimizing the maximum mean difference, eliminating distribution differences between modalities and solving the "modal gap" problem in traditional multimodal fusion. The joint features in the latent space can simultaneously capture complementary information across modalities, enhancing the richness of feature representation and noise resistance.

[0063] For example, speech recognition may make mistakes in noisy environments, but by collecting and analyzing visual and text features authorized by the user, supplementary information can be provided through latent space alignment, improving the accuracy of intent understanding.

[0064] By combining Conditional Random Fields (CRFs) with a domain-specific scheduling dictionary, the candidate set for fuzzy keywords is expanded through CRFs and the domain-specific scheduling dictionary, capturing domain-specific semantic variations. CRFs can model the contextual dependencies of keywords, avoiding the limitations of hard matching. The scheduling dictionary provides a domain-specific fuzzy lexicon, improving the model's generalization ability in vertical scenarios.

[0065] By defining a PCFG probabilistic context-free grammar for scheduling instructions, possible syntax trees are generated and probabilities are calculated, transforming intent understanding into a structured parsing problem. PCFG restricts the parsing space through grammatical rules, avoiding meaningless combinations and improving parsing rationality.

[0066] When generating candidate syntax trees, the self-attention mechanism of the Transformer is used to capture long-distance dependencies. The optimal parser is selected based on the minimum description length principle, striking a balance between model complexity and data fitting ability. MDL tends to choose "concise and accurate" parsers, avoiding overfitting (e.g., overly complex syntax trees) or underfitting (e.g., missing key information). MDL uses information theory criteria to quickly filter for the optimal solution, reducing computational costs.

[0067] The optimal syntax parsing of dispatch instructions is decomposed into feature vector causal relationship pairs, clearly defining the components of the intent and their logical dependencies. Causal relationship pairs visually demonstrate the generation logic of the intent, such as "the priority of the 'transport' action is increased due to the 'urgent' modifier," facilitating manual verification and debugging. By explicitly modeling causal dependencies, the model can distinguish core intent from interfering information, improving robustness in noisy environments.

[0068] For example, the instruction "Transport the goods from warehouse A to warehouse B by truck as soon as possible" can be broken down into causal pairs: [Transportation → Goods → Warehouse A], [Transportation → Tool → Truck], [Transportation → Time Modification → As soon as possible], thus clarifying the causal relationship between each element.

[0069] A dynamic knowledge graph is constructed based on causal relationships, and a Bayesian network is used to quantify the probabilistic dependencies between parameters. The Bayesian network supports online updates, dynamically adjusting the strength of causal relationships to adapt to environmental changes. Through probabilistic modeling, the uncertainty in intent understanding is quantified, providing confidence support for decision-making.

[0070] Construct a global memory graph of multiple agents to record the context when agents switch, thus avoiding information silos.

[0071] A global memory map enables different agents to share historical experiences, avoiding repeated trial and error. Through unified memory storage, it ensures that multiple agents have a consistent understanding of the same task, such as "all agents know that the goods should be transported to warehouse B first".

[0072] For example, in logistics scheduling, after agent A completes the transportation from "Warehouse A to Warehouse B", its memory will be recorded in the global graph, such as "road congestion", for agent B to refer to when planning subsequent routes.

[0073] The GAT (Graph Attention Network) is used to retrieve the Top-K most relevant memories from the global memory graph and extract their feature vectors. GAT focuses on key memories through attention weights, avoiding interference from irrelevant information. Compared to traversing all nodes, Top-K retrieval significantly reduces computational overhead and supports real-time decision-making.

[0074] Example: The current task is "transporting goods in the rain". GAT retrieves the top-3 memory nodes: [rainy day → transportation time + 2 hours], [rainy day → tools → anti-skid truck], [rainy day → route → avoid low-lying areas].

[0075] By combining global memory graphs and Top-K memory node features, multi-agent node modality weights are calculated, and the modality fusion ratio is dynamically adjusted. Modality weights are automatically assigned based on the task context. Low-relevance modalities are suppressed through weighting, thereby improving feature representation efficiency.

[0076] Based on the fused multimodal features and modality weights, an intent label prediction model is constructed, outputting intent labels and confidence scores. By combining causal relationships, historical memory, and dynamic modality fusion, the model can capture complex intents, such as "changing transportation routes due to weather." The confidence score output helps the decision-making system assess risks and avoid erroneous executions caused by low-confidence predictions.

[0077] In one embodiment of the present invention, for multimodal data collected by the intelligent agent, modal differences are minimized by the maximum mean difference of MMD, and speech features, text features, and visual features are mapped to the same latent space. In the same latent space, the following steps are included:

[0078] The agent collects multimodal data from each dialogue, including: voice data, text data, and visual data;

[0079] Extracting speech feature vectors from speech data: Temporal acoustic feature vectors are processed by 1D-CNN, and LSTM captures fundamental frequency data as emotion feature vectors;

[0080] Extracting text feature vectors from text data: Generating context word vectors using BERT / RoBERTa, and then extracting syntactic feature vectors using BiLSTM;

[0081] Extracting visual feature vectors from visual data: Processing video streams using 3D-CNN to extract spatiotemporal feature vectors of facial movements;

[0082] For speech feature vectors, text feature vectors, and visual feature vectors, a joint embedding space is established to map speech features and text features to the same latent space. Furthermore, based on minimizing modal differences using the maximum mean difference, the MMD maximum mean difference loss function is established for feature bias alignment.

[0083]

[0084] Where Lmmd is the maximum mean difference loss, used to measure the difference in the latent space distribution of speech features and text features. The smaller the value, the closer the distributions of the two modalities are. i represents the i-th sample, j represents the j-th sample, Zvsr is the embedding vector of the feature vector of the r-th sample in the visual feature vector in the latent space, Zvsr` is the embedding vector of the feature vector of the r`-th sample in the visual feature vector in the latent space; Zaui is the embedding vector of the feature vector of the i-th sample in the speech feature vector in the latent space, Zaui` is the embedding vector of the feature vector of the i`-th sample in the speech feature vector in the latent space, i and i` ∈ (1,2,...,n), n is the number of speech samples; Ztej is the embedding vector of the feature vector of the i-th sample in the text feature vector in the latent space, Ztej` is the embedding vector of the feature vector of the j`-th sample in the text feature vector in the latent space, j and j` ∈ (1,2,...,m), m is the number of text samples; k(,) represents the Gaussian kernel function.

[0085] This represents the average kernel function similarity between the latent vectors of all speech samples and the corresponding number of latent vectors of text samples. This is a correction term for the cross-modal similarity term among speech feature vectors, text feature vectors, and visual feature vectors;

[0086] In this embodiment, a microphone array is used to collect speech signals and extract acoustic features: MFCC and Mel spectrogram, as well as speech emotion features: fundamental frequency data;

[0087] Automatic speech recognition (ASR) is used to convert speech into text, and semantic features are extracted using NLP: word vectors and syntactic analysis.

[0088] The system captures facial expressions, body movements, and environmental scenes using a camera, and then extracts visual features using CNNs (such as ResNet).

[0089] In one embodiment of the present invention, a fuzzy semantic alignment model is established by using a Conditional Random Field (CRF) combined with a scheduling domain dictionary to expand the candidate set of fuzzy keywords, including the following steps:

[0090] For the feature vectors aligned by feature bias, the feature vectors are arranged into an input sequence set according to the timestamp order of the samples corresponding to the feature vectors. The rationality of expanding fuzzy keywords is evaluated using a CRF (Conditional Random Field) combined with a scheduling domain dictionary.

[0091]

[0092] Wherein, P(y|x) is the rationality coefficient of the extended fuzzy keywords, which represents the conditional probability of the output label sequence y after the candidate set of the extended fuzzy keywords when the input sequence of the feature vector in the input sequence set is x. It is used to measure the rationality of the extended fuzzy keywords.

[0093] Fz(x) is a normalization factor that ensures the sum of probabilities is 1;

[0094] The transition feature function depends on the current label yh, the previous label yh-1, and the input sequence x, and is used to capture the contextual dependencies of the label sequence.

[0095] The state feature function depends only on the current label yh and the input sequence x, and is used to capture the local matching relationship between a single label and the input; h∈(1,2,...,p), where p is the total number of feature vectors in the input sequence set;

[0096] λj and μi are feature weight parameters. λj corresponds to the transition feature weight parameter and μi corresponds to the state feature weight parameter. They are learned through training data. The larger the weight, the stronger the influence of the corresponding feature on the probability.

[0097] By combining CRF (Conditional Random Field) with word vectors from the scheduling domain dictionary, word vectors in the scheduling domain dictionary whose P(y|x) is greater than a threshold with the feature vectors in the input sequence set are selected to form a candidate set of fuzzy keywords;

[0098] Based on the correspondence between feature vectors in the input sequence set and candidate sets of fuzzy keywords, and the established MMD maximum mean difference loss function, a fuzzy semantic alignment model is established.

[0099] In one embodiment of the present invention, a PCFG probabilistic context-free grammar for scheduling instructions is defined. Combined with the establishment of a fuzzy semantic alignment model, possible syntax trees are generated and their probabilities are calculated. A Transformer decoder is used to generate candidate syntax trees, and the optimal parser is selected based on the minimum description length principle of MDL. The process includes the following steps:

[0100] We define a PCFG probabilistic context-free grammar for scheduling instructions. Combined with a fuzzy semantic alignment model, we filter scheduling instruction feature vectors from the input sequence set and candidate set of fuzzy keywords corresponding to each dialogue, labeling them as parent nodes. We then construct a syntax tree of other feature vectors and parent nodes. Through production rules, we assign probability parameters to the feature vectors corresponding to the parent nodes, indicating the likelihood of the syntax tree generating scheduling instructions.

[0101]

[0102] Where P(T) represents the probability that the syntax tree corresponding to the parent node will generate a scheduling instruction, Rdu is the du-th syntax production rule, Rdu∈(1,2,...,V), V is the total number of syntax production rules contained in the syntax tree T, parent(Rdu) is the condition given by the scheduling instruction feature vector of the parent node, and P(Rdu|parent(Rdu)) is the conditional probability of selecting syntax production rule Rdu under the given condition of the parent node, which is used to quantify the rationality of the syntax rule.

[0103] The probability P(T) of generating scheduling instructions from the syntax tree corresponding to the parent node is sorted in descending order. The syntax trees corresponding to a predetermined proportion of P(T) before sorting are selected, and candidate syntax trees are generated using the Transformer decoder. The optimal parser is selected based on the minimum MDL description length principle.

[0104] in, The optimal syntax tree selected through the MDL principle is the final output of the scheduling instruction syntax parsing result. len(T) is the description length of the selected syntax tree T, err(T) is the syntax error penalty term of the selected syntax tree T, which quantifies the number of syntax errors in the syntax tree T, such as the number of structures that do not conform to the scheduling domain syntax rules. λ is a weight parameter that balances the importance of description length and error penalty, and controls the contribution ratio of the two in the optimization objective. It is adjusted according to the usage effect, and the initial value is set to 0.3.

[0105] In one embodiment of the present invention, the scheduling instruction is decomposed into causal relationships of feature vector causal relationship pairs according to the selected optimal parsing, and a dynamic knowledge graph is constructed by modeling the causal dependencies between parameter pairs using a Bayesian network, including the following steps:

[0106] The optimal syntax tree is selected based on the MDL principle corresponding to the selected optimal parser. The scheduling instruction feature vector of the parent node of the optimal syntax tree is decomposed into causal relationship pairs of feature vectors. A causal relationship graph between the scheduling instruction feature vector of the parent node and other feature vectors is established through a Bayesian network. Causal relationship edges are established between nodes with causal relationships.

[0107] Construct a dynamic knowledge graph: G=(E,R,A), E={e1,e2,…}, R={rw};

[0108] Where E is the feature vector corresponding to the node, R is the correlation between the node and its parent node, A is the number of causal edges traversed by the shortest distance between the node and its parent node, ew is the feature vector corresponding to the event node, and rw is the correlation between the feature vectors of the node and its parent node. In this embodiment, the cosine similarity between the feature vectors is used to measure the correlation rw between the feature vectors of the node and its parent node.

[0109] In one embodiment of the present invention, the following steps are included: recording the contextual associations during agent switching, constructing a multi-agent global memory graph, using a GAT graph attention network to retrieve the Top-K related memory nodes of relevant historical memories, and recording the feature vectors of the Top-K related memory nodes:

[0110] Record the contextual associations when switching agents. Based on the dialogue data in different agents during the switching process, construct a dynamic knowledge graph of the dialogue data in different agents. Label the time of agent switching and the type of agent in the constructed dynamic knowledge graph to form a multi-agent global memory graph.

[0111] GAT uses an attention mechanism to associate parent nodes in the dynamic knowledge graph of dialogue data from different agents in a multi-agent global memory graph according to the time sequence of agent switching. It aggregates the feature vectors of parent nodes with the feature vectors of neighboring parent nodes through the GAT graph attention mechanism and updates the feature vectors of the corresponding parent nodes.

[0112] Calculate the similarity score between the parent node and all other nodes in the dynamic knowledge graph of dialogue data in different intelligent agents; in this embodiment, the similarity score is calculated by the cosine similarity between the feature vectors of the parent node and all other nodes.

[0113] Sort the nodes by similarity score from high to low, select the top few nodes as Top-K related memory nodes, and retain them in the latest dynamic knowledge graph.

[0114] In one embodiment of the present invention, the modal weights of multi-agent nodes are calculated by combining the multi-agent global memory graph and the feature vectors of Top-K related memory nodes, including the following steps:

[0115] Calculate the cosine similarity between the feature vectors of nodes in each dynamic knowledge graph in the multi-agent global memory graph and the feature vectors of the Top-K related memory nodes, and use it as the modal relevance score;

[0116] The modal relevance score is converted into the modal weight of each node in the dynamic knowledge graph using the Softmax function: wsg = node indexed score / sum of the indexed scores of all Top-K nodes, where node indexed score = exp(sg / τ), τ is the temperature coefficient, sg is the modal relevance score of the node, and wsg is the modal weight of the node.

[0117] In one embodiment of the present invention, feature fusion is performed based on the modal weights of multi-agent nodes and the global memory graph of multi-agents, and an intent label prediction model is constructed, including the following steps:

[0118] Based on the modal weights of multi-agent nodes, the feature vectors of multi-agent global memory graph nodes, and the feature vectors of the latest dynamic knowledge graph nodes, an overall fusion feature vector and a latest fusion feature vector are generated. Then, according to the intent feature logic rules defined by experts, an overall intent icon is added to the overall fusion feature vector of historical data, and a latest intent label is added to the latest fusion feature vector.

[0119] Building an intent label prediction model:

[0120] The classifier model is constructed by a fully connected layer and a softmax layer. The classifier model is trained by the overall fused feature vector and the latest fused feature vector labeled with labels. The overall intent label of the overall fused feature vector and the latest intent label of the latest fused feature vector are analyzed.

[0121] Define the confidence score calculation rules and calculate the confidence score of the output intent label.

[0122] In one embodiment of the present invention, an overall fused feature vector and a latest fused feature vector are generated. Based on expert-defined intent feature logic rules, an overall intent icon is added to the overall fused feature vector of historical data, and a latest intent label is added to the latest fused feature vector. This includes the following steps:

[0123] The feature vectors of the nodes in the multi-agent global memory graph are sorted and spliced ​​together according to the modal weights of the nodes in each dynamic knowledge graph in the multi-agent global memory graph from high to low to generate the overall fused feature vector.

[0124] The feature vectors of the latest dynamic knowledge graph nodes are sorted and concatenated according to the modal weights of the nodes in the latest dynamic knowledge graph from high to low to generate the latest fused feature vector.

[0125] The overall intent label is set for the multi-agent dialogue record by the logical rules of intent features defined by experts, and the overall intent label is marked on the overall fusion feature vector corresponding to the dialogue record.

[0126] In multi-agent dialogue, the latest intent label is set for the latest dialogue record, and the latest fused feature vector corresponding to the dialogue record is labeled with the latest intent label.

[0127] In one embodiment of the present invention, a confidence calculation rule is formulated to calculate the confidence of the output intent label, including the following steps:

[0128] Filter historical data where the overall intent label of the overall fusion feature vector is the same as the overall intent label output by the current classifier model, calculate the maximum cosine similarity between the corresponding overall fusion feature vector and the current overall fusion feature vector, and use the maximum cosine similarity as the confidence score of the output intent label.

[0129] Select data from historical data whose latest intent label is the same as the latest intent label output by the current classifier model. Calculate the maximum cosine similarity between the latest fused feature vector and the current latest fused feature vector. Take a weighted average of the maximum cosine similarity and the confidence of the intent label of the corresponding overall fused feature vector. The result of the weighted average is used as the confidence of the output intent label.

[0130] In this embodiment, the maximum cosine similarity between the latest fused feature vector and the current latest fused feature vector is calculated. The maximum cosine similarity is then weighted and averaged with the confidence of the intent label of the corresponding overall fused feature vector. The weight of the maximum cosine similarity is 0.75, and the weight of the confidence of the intent label of the corresponding overall fused feature vector is 0.25.

[0131] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A method for understanding the intent of an intelligent agent based on multimodal information fusion, characterized in that, Includes the following steps: For multimodal data collected by intelligent agents, modal differences are minimized by MMD (Maximum Mean Difference), and speech features, text features, and visual features are mapped to the same latent space. In the same latent space, CRF (Conditional Random Field) combined with a scheduling domain dictionary is used to expand the candidate set of fuzzy keywords and establish a fuzzy semantic alignment model. Define a PCFG probabilistic context-free grammar for scheduling instructions, combine it with the establishment of a fuzzy semantic alignment model, generate possible syntax trees and calculate probabilities, use a Transformer decoder to generate candidate syntax trees, and select the optimal parser through the MDL minimum description length principle. Based on the selected optimal parsing, the scheduling instructions are decomposed into causal relationships between feature vector causal pairs, and a Bayesian network is used to model the causal dependencies between parameter pairs to construct a dynamic knowledge graph. Record the contextual associations during agent switching, construct a multi-agent global memory graph, use the GAT graph attention network to retrieve the Top-K related memory nodes of relevant historical memories, and record the feature vectors of the Top-K related memory nodes; By combining the global memory map of multi-agents and the feature vectors of Top-K related memory nodes, the modal weights of multi-agent nodes are calculated. Based on the modal weights of multi-agent nodes and the global memory map of multi-agents, feature fusion is performed, an intent label prediction model is constructed, and the model prediction is performed to output the intent label and confidence score.

2. The method for understanding an agent's intent based on multimodal information fusion according to claim 1, characterized in that, For multimodal data collected by the agent, modal differences are minimized using the maximum mean difference (MMD) method, mapping speech features, text features, and visual features to the same latent space. Within this same latent space, the following steps are included: The agent collects multimodal data from each dialogue, including: voice data, text data, and visual data; Extracting speech feature vectors from speech data: Temporal acoustic feature vectors are processed by 1D-CNN, and LSTM captures fundamental frequency data as emotion feature vectors; Extracting text feature vectors from text data: Generating context word vectors using BERT / RoBERTa, and then extracting syntactic feature vectors using BiLSTM; Extracting visual feature vectors from visual data: Processing video streams using 3D-CNN to extract spatiotemporal feature vectors of facial movements; For speech feature vectors, text feature vectors, and visual feature vectors, a joint embedding space is established to map speech features and text features to the same latent space. Furthermore, based on minimizing modal differences using the maximum mean difference, the MMD maximum mean difference loss function is established for feature bias alignment. ； Where Lmmd is the maximum mean difference loss, i represents the i-th sample, j represents the j-th sample, Zvsr is the embedding vector of the r-th sample in the latent space of the visual feature vector, Zvsr` is the embedding vector of the r`-th sample in the latent space of the visual feature vector; Zaui is the embedding vector of the i-th sample in the latent space of the speech feature vector, Zaui` is the embedding vector of the i`-th sample in the latent space of the speech feature vector, i and i` ∈ (1,2,...,n), n is the number of speech samples; Ztej is the embedding vector of the i-th sample in the latent space of the text feature vector, Ztej` is the embedding vector of the j`-th sample in the latent space of the text feature vector, j and j` ∈ (1,2,...,m), m is the number of text samples; k(,) represents the Gaussian kernel function.

3. The method for understanding an agent's intent based on multimodal information fusion according to claim 2, characterized in that, Using a Conditional Random Field (CRF) combined with a scheduling domain dictionary, the candidate set of fuzzy keywords is expanded, and a fuzzy semantic alignment model is established, including the following steps: For the feature vectors aligned by feature bias, the feature vectors are arranged into an input sequence set according to the timestamp order of the samples corresponding to the feature vectors. The rationality of expanding fuzzy keywords is evaluated using a CRF (Conditional Random Field) combined with a scheduling domain dictionary. ； Where P(y|x) is the rationality coefficient of the extended fuzzy keywords, representing the conditional probability of the output label sequence y after the candidate set of the extended fuzzy keywords when the input sequence of the feature vector in the input sequence set is x; Fz(x) is a normalization factor that ensures the sum of probabilities is 1; The transition feature function depends on the current label yh, the previous label yh-1, and the input sequence x, and is used to capture the contextual dependencies of the label sequence. is the state feature function, which depends only on the current label yh and the input sequence x, and is used to capture the local matching relationship between a single label and the input; h∈(1,2,...,p), p is the total number of feature vectors in the input sequence set; λj and μi are feature weight parameters; By combining CRF (Conditional Random Field) with word vectors from the scheduling domain dictionary, word vectors in the scheduling domain dictionary whose P(y|x) is greater than a threshold with the feature vectors in the input sequence set are selected to form a candidate set of fuzzy keywords. Based on the correspondence between feature vectors in the input sequence set and candidate sets of fuzzy keywords, and the established MMD maximum mean difference loss function, a fuzzy semantic alignment model is established.

4. The method for understanding an agent's intent based on multimodal information fusion according to claim 1, characterized in that, Define a PCFG probabilistic context-free grammar for scheduling instructions, combine it with a fuzzy semantic alignment model, generate possible syntax trees and calculate probabilities, use a Transformer decoder to generate candidate syntax trees, and select the optimal parser based on the minimum description length principle of MDL, including the following steps: We define a PCFG probabilistic context-free grammar for scheduling instructions. Combined with a fuzzy semantic alignment model, we filter scheduling instruction feature vectors from the input sequence set and candidate set of fuzzy keywords corresponding to each dialogue, labeling them as parent nodes. We then construct a syntax tree of other feature vectors and parent nodes. Through production rules, we assign probability parameters to the feature vectors corresponding to the parent nodes, indicating the likelihood of the syntax tree generating scheduling instructions. ； Where P(T) represents the probability that the syntax tree corresponding to the parent node will generate a scheduling instruction, Rdu is the du-th syntax production rule, Rdu∈(1,2,...,V), V is the total number of syntax production rules contained in the syntax tree T, parent(Rdu) is the condition given by the scheduling instruction feature vector of the parent node, and P(Rdu|parent(Rdu)) is the conditional probability of selecting syntax production rule Rdu to be implemented under the given conditions of the parent node. The probability P(T) of generating scheduling instructions from the syntax tree corresponding to the parent node is sorted in descending order. The syntax trees corresponding to a predetermined proportion of P(T) before sorting are selected, and candidate syntax trees are generated using the Transformer decoder. The optimal parser is selected based on the minimum MDL description length principle. ; in, Let len(T) be the optimal syntax tree selected by the MDL principle, err(T) be the description length of the selected syntax tree T, err(T) be the syntax error penalty term of the selected syntax tree T, and λ be the weight parameter.

5. The method for understanding an agent's intent based on multimodal information fusion according to claim 1, characterized in that, Based on the selected optimal parsing, the scheduling instructions are decomposed into causal relationships between feature vector causal pairs, and a Bayesian network is used to model the causal dependencies between parameter pairs to construct a dynamic knowledge graph, including the following steps: The optimal syntax tree is selected based on the MDL principle corresponding to the selected optimal parser. The scheduling instruction feature vector of the parent node of the optimal syntax tree is decomposed into causal relationship pairs of feature vectors. A causal relationship graph between the scheduling instruction feature vector of the parent node and other feature vectors is established through a Bayesian network. Causal relationship edges are established between nodes with causal relationships. Construct a dynamic knowledge graph: G=(E,R,A), E={e1,e2,…}, R={rw}; Where E is the feature vector corresponding to the node, R is the correlation between the node and its parent node, A is the number of causal edges traversed by the shortest distance between the node and its parent node, ew is the feature vector corresponding to the event node, and rw is the correlation between the feature vectors of the node and its parent node.

6. The method for understanding an agent's intent based on multimodal information fusion according to claim 1, characterized in that, Record the contextual relationships during agent switching, construct a multi-agent global memory graph, use the GAT graph attention network to retrieve the Top-K relevant memory nodes of related historical memories, and record the feature vectors of the Top-K relevant memory nodes, including the following steps: Record the contextual associations when switching agents. Based on the dialogue data in different agents during the switching process, construct a dynamic knowledge graph of the dialogue data in different agents. Label the time of agent switching and the type of agent in the constructed dynamic knowledge graph to form a multi-agent global memory graph. GAT uses an attention mechanism to associate parent nodes in the dynamic knowledge graph of dialogue data from different agents in a multi-agent global memory graph according to the time sequence of agent switching. It aggregates the feature vectors of parent nodes with the feature vectors of neighboring parent nodes through the GAT graph attention mechanism and updates the feature vectors of the corresponding parent nodes. Calculate the similarity score between the parent node and all other nodes in the dynamic knowledge graph of dialogue data from different agents. Sort the nodes by similarity score from high to low, select the top few nodes as Top-K related memory nodes, and retain them in the latest dynamic knowledge graph.

7. The method for understanding an agent's intent based on multimodal information fusion according to claim 1, characterized in that, By combining the global memory graph of the multi-agent system and the feature vectors of the Top-K related memory nodes, the modal weights of the multi-agent nodes are calculated, including the following steps: Calculate the cosine similarity between the feature vectors of nodes in each dynamic knowledge graph in the multi-agent global memory graph and the feature vectors of the Top-K related memory nodes, and use it as the modal relevance score; The modal relevance score is converted into the modal weight of each node in the dynamic knowledge graph using the Softmax function: wsg = node indexed score / sum of the indexed scores of all Top-K nodes, where node indexed score = exp(sg / τ), τ is the temperature coefficient, sg is the modal relevance score of the node, and wsg is the modal weight of the node.

8. The method for understanding an agent's intent based on multimodal information fusion according to claim 1, characterized in that, Based on the modal weights of multi-agent nodes and the global memory graph of multi-agents, feature fusion is performed, and an intent label prediction model is constructed, including the following steps: Based on the modal weights of multi-agent nodes, the feature vectors of multi-agent global memory graph nodes, and the feature vectors of the latest dynamic knowledge graph nodes, an overall fusion feature vector and a latest fusion feature vector are generated. Then, according to the intent feature logic rules defined by experts, an overall intent icon is added to the overall fusion feature vector of historical data, and a latest intent label is added to the latest fusion feature vector. Building an intent label prediction model: The classifier model is constructed by a fully connected layer and a softmax layer. The classifier model is trained by the overall fused feature vector and the latest fused feature vector labeled with labels. The overall intent label of the overall fused feature vector and the latest intent label of the latest fused feature vector are analyzed. Define the confidence score calculation rules and calculate the confidence score of the output intent label.

9. The method for understanding an agent's intent based on multimodal information fusion according to claim 8, characterized in that, Generate an overall fused feature vector and a latest fused feature vector. Using expert-defined intent feature logic rules, add an overall intent icon to the overall fused feature vector of historical data and add a latest intent label to the latest fused feature vector. This includes the following steps: The feature vectors of the nodes in the multi-agent global memory graph are sorted and spliced ​​together according to the modal weights of the nodes in each dynamic knowledge graph in the multi-agent global memory graph from high to low to generate the overall fused feature vector. The feature vectors of the latest dynamic knowledge graph nodes are sorted and concatenated according to the modal weights of the nodes in the latest dynamic knowledge graph from high to low to generate the latest fused feature vector. The overall intent label is set for the multi-agent dialogue record by the logical rules of intent features defined by experts, and the overall intent label is marked on the overall fusion feature vector corresponding to the dialogue record. In multi-agent dialogue, the latest intent label is set for the latest dialogue record, and the latest fused feature vector corresponding to the dialogue record is labeled with the latest intent label.

10. The method for understanding an intelligent agent's intent based on multimodal information fusion according to claim 8, characterized in that, Define the confidence score calculation rules and calculate the confidence score of the output intent label, including the following steps: Filter historical data where the overall intent label of the overall fusion feature vector is the same as the overall intent label output by the current classifier model, calculate the maximum cosine similarity between the corresponding overall fusion feature vector and the current overall fusion feature vector, and use the maximum cosine similarity as the confidence score of the output intent label. Select data from historical data whose latest intent label is the same as the latest intent label output by the current classifier model. Calculate the maximum cosine similarity between the latest fused feature vector and the current latest fused feature vector. Take a weighted average of the maximum cosine similarity and the confidence of the intent label of the corresponding overall fused feature vector. The result of the weighted average is used as the confidence of the output intent label.

Images