Intelligent Interactive Question Answering Method and System Based on Multimodal Large Model Intent Recognition

Abstract

This invention provides an intelligent interactive question-answering method and system based on multimodal large model intent recognition, relating to the field of intelligent question-answering technology. It includes: through feature extraction, cross-modal semantic mapping, intent matching, key information extraction and response generation steps, it realizes unified processing of multimodal information, can accurately identify user intent, dynamically construct an association network to extract key information, improve the accuracy and semantic coherence of intelligent question answering, and enhance the human-computer interaction experience.

Description

Technical Field

[0001] This invention relates to the field of intelligent question answering technology, and in particular to an intelligent interactive question answering method and system based on multimodal large model intent recognition. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent question-answering systems have become an important means of human-computer interaction, widely used in customer service, education, healthcare, and other fields. Traditional question-answering systems mainly rely on single-modality information processing, such as text or voice, which is insufficient to meet the needs of modern complex interaction scenarios. In recent years, multimodal large model technology has improved the system's ability to understand user intent and the naturalness of interaction by integrating information from multiple modalities such as text, images, and voice.

[0003] However, existing large model technologies still have problems such as failing to identify and retain truly critical historical information, leading to semantic shifts or forgetting of key information in long dialogue scenarios, inability to accurately understand complex user intentions involving multiple modalities when there are inconsistencies or complementary relationships between modalities, and lack of sensitivity to intent transitions and intent evolution in multi-turn interactions, thus failing to provide a coherent and personalized interactive experience.

[0004] Therefore, a solution is urgently needed to address the problems existing in the current technology. Summary of the Invention

[0005] This invention provides an intelligent interactive question-answering method and system based on multimodal large model intent recognition, which can at least solve some of the problems existing in the prior art.

[0006] A first aspect of this invention provides an intelligent interactive question-answering method based on multimodal large model intent recognition, comprising:

[0007] Acquire multimodal interaction information from user input and extract features to obtain multimodal feature vectors;

[0008] Calculate the mutual information between feature vectors of different modalities, construct a cross-modal semantic mapping matrix based on the mutual information, map the feature vectors of different modalities to a unified semantic space using the cross-modal semantic mapping matrix, obtain unified semantic features, calculate the semantic similarity with prior knowledge in a preset knowledge base, determine supplementary information, and fuse the supplementary information with the unified semantic features to obtain fused features;

[0009] A user intent feature vector is constructed based on the fusion features. The user intent feature vector is then matched with a preset intent category library for similarity, and the category with the highest similarity is selected as the user intent category.

[0010] The importance distribution of key entities in historical dialogues is calculated based on word frequency and information gain. The co-occurrence frequency of entities and events is counted using a sliding window and an association network is constructed. The core semantic skeleton is obtained by pruning based on the association strength. Key paths are extracted and semantic coherence is calculated. The importance of historical dialogue information is ranked by combining path weight and coherence. Key information is determined based on the ranking results.

[0011] The response content is generated based on the key information and returned to the customer, and the associated network is updated based on the current dialogue content.

[0012] In one alternative implementation,

[0013] The mutual information between feature vectors of different modalities is calculated. Based on the mutual information, a cross-modal semantic mapping matrix is ​​constructed. The cross-modal semantic mapping matrix is ​​used to map the feature vectors of different modalities to a unified semantic space to obtain unified semantic features. The semantic similarity with prior knowledge in a preset knowledge base is calculated. Supplementary information is determined and fused with the unified semantic features to obtain fused features, including:

[0014] Obtain the multimodal feature vector to be processed, which includes text feature vector and image feature vector;

[0015] Calculate the joint probability distribution and edge probability distribution of the text feature vector and the image feature vector, and calculate the mutual information between the text feature vector and the image feature vector based on the joint probability distribution and the edge probability distribution;

[0016] A cross-modal semantic mapping matrix is ​​constructed based on the mutual information. The value of each element in the cross-modal semantic mapping matrix is ​​calculated by a normalized exponential function adjusted by a temperature parameter. The temperature parameter is used to control the distribution of semantic association strength in the cross-modal semantic mapping matrix. Based on the cross-modal semantic mapping matrix, the text feature vector and the image feature vector are mapped to a unified semantic space to obtain unified semantic features.

[0017] The similarity between the unified semantic feature and prior knowledge in the preset knowledge base is calculated. Prior knowledge with a similarity greater than a preset threshold is used as supplementary information. The unified semantic feature is used as the query vector, and the supplementary information is used as key-value pairs. Information fusion is performed through a multi-head attention mechanism to obtain the fused feature.

[0018] In one alternative implementation,

[0019] Calculate the similarity between the unified semantic features and prior knowledge in a preset knowledge base, and include prior knowledge with similarity greater than a preset threshold as supplementary information:

[0020] The unified semantic features are decomposed into a core semantic variable set and a context semantic variable set. A condition dependency graph is constructed based on the core semantic variable set and the context semantic variable set. According to the structure of the condition dependency graph, the initial probability distribution parameters are calculated using maximum likelihood estimation. The optimized probability distribution parameters are then iteratively optimized using the expectation-maximization algorithm.

[0021] The initial similarity value between each prior knowledge item in the preset knowledge base and the unified semantic feature is calculated based on the optimized probability distribution parameters, and the final similarity value is obtained by approximating the initial similarity value based on the Monte Carlo sampling method.

[0022] The similarity difference between different prior knowledge is calculated to obtain a similarity difference value. An adaptive similarity threshold is constructed based on the similarity difference value. The adaptive similarity threshold is dynamically adjusted according to the feedback score of the prior knowledge to obtain an adjusted similarity threshold. Prior knowledge with a final similarity value greater than the adjusted similarity threshold is identified as candidate supplementary information.

[0023] For each candidate supplementary information, a joint similarity value is calculated based on the final similarity value and the similarity difference value, and the candidate supplementary information with the largest joint similarity value is selected as the supplementary information output.

[0024] In one alternative implementation,

[0025] A user intent feature vector is constructed based on the fused features. The user intent feature vector is then matched with a preset intent category library for similarity, and the category with the highest similarity is selected as the user intent category, including:

[0026] The initial intent features are obtained by acquiring fusion features and mapping them to the semantic space through nonlinear transformation. The initial intent features are then subjected to dimensionality reduction to obtain the user intent feature vector.

[0027] The similarity between the user intent feature vector and each intent category in the preset intent category library is calculated to obtain a category similarity value. The category similarity value is normalized to obtain a probability distribution. The intent category with the highest probability is selected as the user intent category for output.

[0028] In one alternative implementation,

[0029] The importance distribution of key entities in historical dialogues is calculated based on word frequency and information gain. A sliding window is used to count the co-occurrence frequency of entities and events and construct an association network. The core semantic skeleton is obtained by pruning based on association strength. Key paths are extracted and semantic coherence is calculated. The importance of historical dialogue information is ranked by combining path weights and coherence. Based on the ranking results, key information includes:

[0030] Obtain historical dialogue content, extract entities from the historical dialogue content as candidate entities, calculate the word frequency score and information gain score of the candidate entities, obtain the entity importance distribution by weighted sum of word frequency score and information gain score, and determine key entities based on the entity importance distribution.

[0031] The sliding window size is determined based on the length of the historical dialogue content. The co-occurrence frequency of key entities and events is counted within the sliding window. The association strength is calculated based on the co-occurrence frequency, and an association network is constructed. The weights of the association network are set based on the association strength and key entities. The pruning threshold is determined based on the mean and standard deviation of the weights. The associations with weights greater than the pruning threshold in the association network are retained to obtain the core semantic skeleton.

[0032] The path length is determined based on the number of shortest paths between nodes in the core semantic skeleton. A decay factor is set based on the path length. The node centrality score is calculated using the path length and decay factor to extract the key path. The semantic coherence is obtained by calculating the cosine similarity of the semantic vectors of adjacent nodes in the key path.

[0033] Historical dialogue information is obtained based on the critical path. The centrality score of the critical path in the historical dialogue information is calculated to obtain the path weight. The importance ranking value is obtained by combining the path weight and the weighted sum of semantic coherence. The historical dialogue information is ranked according to the importance ranking value and the key information is determined.

[0034] In one alternative implementation,

[0035] Association strength is calculated based on co-occurrence frequency, and an association network is constructed. Weights of the association network are set based on association strength and key entities. A pruning threshold is determined based on the mean and standard deviation of the weights. Associations with weights greater than the pruning threshold are retained to obtain the core semantic skeleton, which includes:

[0036] Obtain the co-occurrence frequency of entities in historical dialogue content, divide the co-occurrence frequency by the square root of the product of the number of times each entity appears to obtain the association strength, and construct an association network based on the association strength;

[0037] Calculate the importance score of the key entity, and set the weight of the association network as the product of the association strength and the average of the importance scores of the key entity;

[0038] The community detection algorithm is used to divide the network into communities to obtain multiple communities. The number of edge sets and node sets within each community are counted. The connection density within the community is obtained by dividing the number of edge sets by half of the product of the number of node sets and the number of node sets minus one.

[0039] Calculate the mean and standard deviation of all weights in the network, and use the sum of the products of the mean and standard deviation of the weights and the first preset coefficient as the benchmark threshold. Then, use the sum of the products of the benchmark threshold and the internal connection density of the community and the second preset coefficient to obtain the adaptive pruning threshold for each community.

[0040] By comparing the adaptive pruning threshold of the community to which each edge belongs in the association network with the weight of the current edge, edges with weights greater than the adaptive pruning threshold of the corresponding community are retained, and edges with weights less than the adaptive pruning threshold of the corresponding community are removed to obtain the core semantic skeleton.

[0041] In one alternative implementation,

[0042] Generate response content based on the key information and return it to the customer; update the associated network based on the current dialogue content, including:

[0043] Based on the key information, generate response content that includes dialogue intent, user requests, and solutions, and return the response content to the customer;

[0044] The new entity is obtained by extracting entity, attribute, and relation triples from the current dialogue content. The co-occurrence frequency and semantic similarity between the new entities are calculated to obtain the entity weight. The interaction weight is obtained by calculating the association strength between the new entity and the original entity.

[0045] Add new entities as nodes, and entity weights and interaction weights as edges to the original association network. Recalculate the adaptive pruning threshold for the expanded association network based on the internal connection density of the community, and retain edges with weights greater than the corresponding community adaptive pruning threshold to obtain the updated association network.

[0046] A second aspect of this invention provides an intelligent interactive question-answering system based on multimodal large model intent recognition, comprising:

[0047] The first unit is used to acquire multimodal interaction information input by the user and extract features to obtain multimodal feature vectors;

[0048] The second unit is used to calculate the mutual information between feature vectors of different modalities, construct a cross-modal semantic mapping matrix based on the mutual information, map the feature vectors of different modalities to a unified semantic space using the cross-modal semantic mapping matrix, obtain unified semantic features, calculate the semantic similarity with prior knowledge in a preset knowledge base, determine supplementary information, and fuse the supplementary information with the unified semantic features to obtain fused features.

[0049] The third unit is used to construct a user intent feature vector based on the fusion features, perform similarity matching between the user intent feature vector and a preset intent category library, and select the category with the highest similarity as the user intent category.

[0050] The fourth unit is used to calculate the importance distribution of key entities in historical dialogues based on word frequency and information gain. It uses a sliding window to count the co-occurrence frequency of entities and events and constructs an association network. It prunes the core semantic skeleton based on the association strength, extracts key paths and calculates semantic coherence, and ranks the importance of historical dialogue information by combining path weights and coherence. It determines key information based on the ranking results.

[0051] The fifth unit is used to generate response content based on the key information and return it to the customer, and to update the associated network based on the current dialogue content.

[0052] A third aspect of the present invention provides an electronic device, comprising:

[0053] A processor and a memory for storing processor-executable instructions, wherein the processor is configured to invoke instructions stored in the memory to perform the aforementioned method.

[0054] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0055] This invention achieves unified representation and processing of information from different modalities through multimodal feature extraction and cross-modal semantic mapping, effectively solving the problems of limited information acquisition and incomplete expression in traditional single-modal interaction methods. It improves the accuracy and comprehensiveness of understanding user intent. Based on the cross-modal semantic mapping matrix and association network of mutual information, it dynamically captures the evolution of user intent and contextual relationships, accurately identifies user intent and extracts key information, enhances the coherence of interaction and contextual understanding, and reduces ambiguity and misunderstanding rate during interaction. By using a sliding window to count the co-occurrence frequency of entities and events and combining it with semantic coherence evaluation, it achieves intelligent filtering and importance ranking of historical dialogue information, reduces interference from irrelevant information, improves the relevance and quality of response content, and reduces computational resource consumption, thereby improving system response efficiency. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating the intelligent interactive question-answering method based on multimodal large model intent recognition, as described in an embodiment of the present invention.

[0057] Figure 2 This is a simulation diagram of the dynamic construction of the condition dependency graph according to an embodiment of the present invention.

[0058] Figure 3 This is a comparison chart of F1 score trends under different dialogue rounds in an embodiment of the present invention;

[0059] Figure 4 This is a flowchart illustrating the core semantic skeleton construction process of the intelligent interactive question-answering method based on multimodal large model intent recognition in an embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0062] Figure 1 This is a flowchart illustrating the intelligent interactive question-answering method based on multimodal large model intent recognition, as described in an embodiment of the present invention. Figure 1 As shown, the method includes:

[0063] Acquire multimodal interaction information from user input and extract features to obtain multimodal feature vectors;

[0064] Calculate the mutual information between feature vectors of different modalities, construct a cross-modal semantic mapping matrix based on the mutual information, map the feature vectors of different modalities to a unified semantic space using the cross-modal semantic mapping matrix, obtain unified semantic features, calculate the semantic similarity with prior knowledge in a preset knowledge base, determine supplementary information, and fuse the supplementary information with the unified semantic features to obtain fused features;

[0065] A user intent feature vector is constructed based on the fusion features. The user intent feature vector is then matched with a preset intent category library for similarity, and the category with the highest similarity is selected as the user intent category.

[0066] The importance distribution of key entities in historical dialogues is calculated based on word frequency and information gain. The co-occurrence frequency of entities and events is counted using a sliding window and an association network is constructed. The core semantic skeleton is obtained by pruning based on the association strength. Key paths are extracted and semantic coherence is calculated. The importance of historical dialogue information is ranked by combining path weight and coherence. Key information is determined based on the ranking results.

[0067] The response content is generated based on the key information and returned to the customer, and the associated network is updated based on the current dialogue content.

[0068] In one alternative implementation,

[0069] The mutual information between feature vectors of different modalities is calculated. Based on the mutual information, a cross-modal semantic mapping matrix is ​​constructed. The cross-modal semantic mapping matrix is ​​used to map the feature vectors of different modalities to a unified semantic space to obtain unified semantic features. The semantic similarity with prior knowledge in a preset knowledge base is calculated. Supplementary information is determined and fused with the unified semantic features to obtain fused features, including:

[0070] Obtain the multimodal feature vector to be processed, which includes text feature vector and image feature vector;

[0071] Calculate the joint probability distribution and edge probability distribution of the text feature vector and the image feature vector, and calculate the mutual information between the text feature vector and the image feature vector based on the joint probability distribution and the edge probability distribution;

[0072] A cross-modal semantic mapping matrix is ​​constructed based on the mutual information. The value of each element in the cross-modal semantic mapping matrix is ​​calculated by a normalized exponential function adjusted by a temperature parameter. The temperature parameter is used to control the distribution of semantic association strength in the cross-modal semantic mapping matrix. Based on the cross-modal semantic mapping matrix, the text feature vector and the image feature vector are mapped to a unified semantic space to obtain unified semantic features.

[0073] The similarity between the unified semantic feature and prior knowledge in the preset knowledge base is calculated. Prior knowledge with a similarity greater than a preset threshold is used as supplementary information. The unified semantic feature is used as the query vector, and the supplementary information is used as key-value pairs. Information fusion is performed through a multi-head attention mechanism to obtain the fused feature.

[0074] The process involves obtaining multimodal feature vectors to be processed, including text feature vectors and image feature vectors. Text feature vectors are extracted from user-input text using a pre-trained BERT model. For example, inputting the sentence "An orange cat is running on the grass" into the BERT model and performing multi-layer attention encoding yields a 768-dimensional text feature vector. Image feature vectors are extracted from the input image using a pre-trained ResNet50 network. For instance, a color image containing a cat, processed by a convolutional network, yields a 2048-dimensional image feature vector.

[0075] Multiple collected text-image pairs are used as training samples. For each pair, the dot product of the text feature vector and the image feature vector is calculated, and the softmax function is applied to obtain the association probability matrix between them. For example, assuming a set contains 100 text-image samples, the dot product of the text-image features for each pair is calculated, constructing a 100×100 association matrix, where each element represents the degree of association between the corresponding text-image pair. By normalizing the rows and columns respectively, the system obtains the joint probability distribution P(T, I), where T represents the text features and I represents the image features. The marginal probability distributions P(T) and P(I) are calculated, which are the cumulative values ​​of the joint probability in the image dimension and the text dimension, respectively. The mutual information MI(T;I) is calculated by dividing the joint probability distribution by the expected logarithm of each marginal probability distribution, reflecting the degree of interdependence between the two modalities.

[0076] Based on the calculated mutual information, a cross-modal semantic mapping matrix M is constructed. Each element Mij of the matrix is ​​calculated using a normalized exponential function adjusted by the temperature parameter τ: Mij equals exp(mutual information ij / τ) divided by the sum of all exp(mutual information ik / τ). Here, τ is the temperature parameter used to control the distribution of semantic association strength. When the value of τ is small (e.g., τ=0.1), the higher mutual information in the matrix is ​​further amplified, making the semantic mapping more focused on strongly correlated feature pairs; while when the value of τ is large (e.g., τ=10), the distribution of the mapping matrix becomes more uniform, allowing for broader cross-modal associations. In practical applications, the system determines the optimal τ value to be 0.5 through validation set experiments, at which point it can maintain the main semantic associations without excessively suppressing secondary associations.

[0077] Using a pre-constructed cross-modal semantic mapping matrix, text feature vectors and image feature vectors are mapped to a unified semantic space. The product of the text feature vector and the mapping matrix yields the mapped text features, and the product of the image feature vector and the transpose of the mapping matrix yields the mapped image features. These two mapped features are then weighted and fused, with the weights determined based on their respective feature confidence levels. Typically, the weights for text and image features are set to 0.6 and 0.4, respectively, resulting in a unified semantic feature vector that contains semantic information from both text and image. For example, the original 768-dimensional text features and 2048-dimensional image features are mapped to the same 512-dimensional unified semantic space.

[0078] The obtained unified semantic features are compared with prior knowledge in a pre-defined knowledge base for similarity calculation. This knowledge base contains knowledge entries from multiple domains, each stored as a vector. A cosine similarity calculation method is used to compare the unified semantic feature vector with each knowledge vector in the knowledge base. For example, the unified semantic features generated from an image and text description of a cat have a high similarity to knowledge entries in the knowledge base such as "feline characteristics" and "common pet habits." A preset threshold of 0.75 is set, and knowledge entries with similarity greater than this threshold are extracted as supplementary information.

[0079] Using unified semantic features as the query vector and supplementary information as key-value pairs, information fusion is achieved through a multi-head attention mechanism. An 8-head attention mechanism is employed, with each attention head having a dimension of 64. Eight different attention weights are computed in parallel to capture semantic associations from various aspects. For example, for the unified semantic features of cats, different attention heads may focus on supplementary knowledge about the cat's appearance, behavior, and living environment. The attention calculation results are concatenated and passed through a linear mapping layer to obtain a fused feature vector of dimension 512. This vector contains both the original text-image semantic information and relevant prior knowledge from the knowledge base.

[0080] In this embodiment, by calculating the mutual information between text feature vectors and image feature vectors, the semantic correlation between different modal data can be measured. From the perspective of information theory, the inherent correlation of cross-modal data is captured, avoiding the semantic information loss that may be caused by simple feature splicing. A normalized exponential function with temperature parameter adjustment is introduced to construct a cross-modal semantic mapping matrix, which can adaptively adjust the distribution of semantic correlation strength between different modal features. The unified semantic features are matched with the prior knowledge in the knowledge base for similarity, and information is fused through a multi-head attention mechanism, realizing knowledge-enhanced feature representation.

[0081] In one alternative implementation,

[0082] Calculate the similarity between the unified semantic features and prior knowledge in a preset knowledge base, and include prior knowledge with similarity greater than a preset threshold as supplementary information:

[0083] The unified semantic features are decomposed into a core semantic variable set and a context semantic variable set. A condition dependency graph is constructed based on the core semantic variable set and the context semantic variable set. According to the structure of the condition dependency graph, the initial probability distribution parameters are calculated using maximum likelihood estimation. The optimized probability distribution parameters are then iteratively optimized using the expectation-maximization algorithm.

[0084] The initial similarity value between each prior knowledge item in the preset knowledge base and the unified semantic feature is calculated based on the optimized probability distribution parameters, and the final similarity value is obtained by approximating the initial similarity value based on the Monte Carlo sampling method.

[0085] The similarity difference between different prior knowledge is calculated to obtain a similarity difference value. An adaptive similarity threshold is constructed based on the similarity difference value. The adaptive similarity threshold is dynamically adjusted according to the feedback score of the prior knowledge to obtain an adjusted similarity threshold. Prior knowledge with a final similarity value greater than the adjusted similarity threshold is identified as candidate supplementary information.

[0086] For each candidate supplementary information, a joint similarity value is calculated based on the final similarity value and the similarity difference value, and the candidate supplementary information with the largest joint similarity value is selected as the supplementary information output.

[0087] The unified semantic features are decomposed into a core semantic variable set and a contextual semantic variable set. The core semantic variable set contains keywords or phrases that express the main semantic content. For example, for the query "mobile phone lifespan", the core semantic variables might include "mobile phone" and "lifespan". The contextual semantic variable set contains peripheral information that helps understand the core semantics, such as related concepts like "electronic devices" and "durability".

[0088] Based on the decomposed set of semantic variables, a conditional dependency graph is constructed to explicitly represent the logical relationships between variables. For each pair of variables, the conditional mutual information value is calculated, and a connection is established in the graph when the mutual information value exceeds 0.6. For example, the mutual information value between "service life" and "durability" is 0.85, so a connection is established between these two nodes. After the graph is constructed, the initial probability distribution parameters are calculated using the maximum likelihood estimation method. By statistically analyzing the frequency and co-occurrence of each variable in the training data, the probability values ​​of each node and edge are obtained.

[0089] To further optimize the probability distribution parameters, the expectation-maximization algorithm is applied for iterative optimization. The initial values ​​of the iteration are set, and the parameters obtained from the maximum likelihood estimation are used as the starting point. The expectation step is executed to calculate the expected values ​​of the latent variables. The maximization step is executed to update the model parameters. The convergence condition is checked. If the parameter change is less than the preset value of 0.001 or the maximum number of iterations of 50 is reached, the iteration is stopped.

[0090] After obtaining the optimized probability distribution parameters, the initial similarity value between each piece of prior knowledge in the preset knowledge base and the unified semantic feature is calculated. During the calculation, the prior knowledge is also represented as a corresponding probability distribution, and the KL divergence or JS divergence between the two distributions is calculated. For example, for the prior knowledge "the average lifespan of electronic devices is 3-5 years", the system calculates its initial similarity value with the query "the lifespan of a mobile phone" to be 0.78.

[0091] Since precise similarity calculation is computationally complex in high-dimensional feature spaces, the system uses the Monte Carlo sampling method for approximate calculation. It randomly samples 1000 points from each of the two distributions and calculates the average distance between the sampled points as an estimate of the final similarity value.

[0092] To determine a suitable similarity threshold, the similarity difference between different prior knowledge sets is calculated. The system calculates the average similarity difference between the top 20% and bottom 20% of prior knowledge in the knowledge base, obtaining a similarity difference value. Based on this difference value, an adaptive similarity threshold is constructed, with an initial threshold set at 0.65.

[0093] This adaptive similarity threshold is dynamically adjusted based on feedback scores from prior knowledge. User feedback scores (ranging from 1 to 5) for each piece of supplementary information are recorded. When the average score is higher than 4, the threshold is decreased by 0.05; when the average score is lower than 3, the threshold is increased by 0.05. For example, if the current threshold is 0.65, and the average score of the most recent 10 pieces of supplementary information is 4.2, then the adjusted threshold becomes 0.60.

[0094] Prior knowledge with a final similarity value greater than the adjusted threshold is identified as candidate supplementary information. For each candidate supplementary information, the system further calculates a joint similarity value using the formula: Joint Similarity = 0.7 × Final Similarity Value + 0.3 × (1 - Relative Similarity Difference Value). Here, the relative similarity difference value represents the average similarity between this knowledge and other candidate knowledge, used to measure the uniqueness of the information.

[0095] For example, for candidate knowledge A, the final similarity value is 0.82, and the average similarity with other candidate knowledge is 0.45, then the joint similarity value is 0.7 × 0.82 + 0.3 × (1 - 0.45) = 0.739. For candidate knowledge B, the final similarity value is 0.78, and the average similarity with other candidate knowledge is 0.35, then the joint similarity value is 0.7 × 0.78 + 0.3 × (1 - 0.35) = 0.741. Although B's final similarity is slightly lower, its higher uniqueness results in a higher final joint similarity value.

[0096] The system selects the candidate supplementary information with the highest joint similarity value as the final output. In the example above, the system will select candidate knowledge B as supplementary information to provide to the user.

[0097] In this embodiment, the unified semantic features are decomposed into core semantic variables and contextual semantic variables. The structural relationship between variables is explicitly modeled by constructing a condition dependency graph. The probability distribution parameters are optimized by combining maximum likelihood estimation and expectation maximization algorithms. An adaptive threshold is constructed by analyzing the similarity differences between different prior knowledge and dynamically adjusted according to the actual feedback of knowledge. The optimal supplementary information is selected by jointly considering the final similarity value and the similarity difference value. This ensures both the relevance of the selected knowledge to the query features and the discriminability between knowledge.

[0098] In existing technologies, simple cosine similarity or Euclidean distance are usually used to calculate the similarity between features and prior knowledge. This ignores the semantic structure and conditional dependencies within the features. Furthermore, using fixed thresholds for knowledge filtering makes it difficult to adapt to the differences in the degree of knowledge association in different scenarios.

[0099] This embodiment uses a probabilistic graphical model to more accurately characterize the intrinsic semantic structure of features and improve the accuracy of similarity calculation. The Monte Carlo sampling method is used to approximate the initial similarity, which effectively reduces the computational complexity while maintaining the stability of similarity calculation. The multi-dimensional knowledge selection strategy significantly improves the accuracy and diversity of knowledge supplementation.

[0100] Figure 2This is a simulation diagram illustrating the dynamic construction of the conditional dependency graph in this invention. It demonstrates how the system constructs a dependency network containing core semantic variables and contextual semantic variables based on unified semantic feature decomposition. The central node "query topic" is tightly connected to six core semantic variables through conditional dependencies, with mutual information values ​​of 0.85, 0.89, 0.79, 0.82, 0.81, and 0.91, respectively. These high mutual information values ​​indicate a strong correlation between the core variables and the query topic. Simultaneously, the core semantic variables are connected to five contextual semantic variables through weaker dependencies (mutual information values ​​between 0.65 and 0.83).

[0101] Figure 2 This diagram reflects how the system identifies key concepts and establishes a dependency network when processing specific queries, laying the foundation for subsequent similarity calculations. The numerical values ​​marked on each connecting line in the diagram represent the strength of conditional mutual information, a key indicator for the system to quantify semantic associations. The entire dependency graph achieves a prediction capability of 93.5% with an error propagation rate of only 7.2%, fully demonstrating the superior performance of this technical solution in capturing complex semantic relationships.

[0102] Compared to traditional Bayesian networks, the dependency graph constructed in this approach can more accurately distinguish between core semantics and contextual semantics, and accurately quantify the strength of their dependencies through conditional mutual information values. This structured semantic representation can significantly improve the accuracy of subsequent knowledge matching, especially when handling complex queries.

[0103] In one alternative implementation,

[0104] A user intent feature vector is constructed based on the fused features. The user intent feature vector is then matched with a preset intent category library for similarity, and the category with the highest similarity is selected as the user intent category, including:

[0105] The initial intent features are obtained by acquiring fusion features and mapping them to the semantic space through nonlinear transformation. The initial intent features are then subjected to dimensionality reduction to obtain the user intent feature vector.

[0106] The similarity between the user intent feature vector and each intent category in the preset intent category library is calculated to obtain a category similarity value. The category similarity value is normalized to obtain a probability distribution. The intent category with the highest probability is selected as the user intent category for output.

[0107] The pre-trained BERT model processes the user input text "I want to check tomorrow's weather forecast," extracting a text embedding representation with a dimension of 768. An acoustic feature extraction module extracts fundamental frequency, energy, and MFCC features from the user's speech, forming a 128-dimensional acoustic feature vector. A feature fusion network then connects the text features and acoustic features to obtain an 896-dimensional fused feature.

[0108] After obtaining the fused features, a nonlinear transformation is used to map the fused features to the semantic space to obtain the initial intent features. This nonlinear mapping is implemented using a multilayer perceptron network, which includes three fully connected layers. Each layer uses the ReLU activation function, with a structure of 896-512-256-128. The 896-dimensional fused features are mapped to 512 dimensions through the first fully connected layer, then to 256 dimensions through the second layer, and finally to 128 dimensions through the third layer, resulting in the initial intent features. For the above input example, the obtained initial intent features are a 128-dimensional vector, represented as [0.32, 0.15, 0.78, ..., 0.41].

[0109] To improve the efficiency and accuracy of intent recognition, the initial intent features are dimensionality-reduced to obtain user intent feature vectors. Principal component analysis is used to reduce the 128-dimensional initial intent features to 64-dimensional features, retaining the most representative information in the original feature space. The covariance matrix of the initial intent features is calculated, and the eigenvalues ​​and eigenvectors of this matrix are solved. The eigenvectors corresponding to the 64 largest eigenvalues ​​are selected to construct a projection matrix. By multiplying the initial intent features by this projection matrix, a 64-dimensional user intent feature vector [0.25, 0.62, 0.18, ..., 0.37] is obtained.

[0110] The intent category matching stage performs similarity matching between the user's intent feature vector and a preset intent category library. The preset intent category library is a database containing multiple common intent categories, each with a corresponding feature vector representation. In this embodiment, the preset intent category library includes 20 common intent categories such as "weather query," "navigation request," "music playback," "alarm clock setting," and "message sending," each with a 64-dimensional feature vector representation.

[0111] The similarity between the user intent feature vector and each intent category in the preset intent category library is calculated to obtain the category similarity value. The cosine similarity metric is used to calculate the similarity, which is the cosine of the angle between the two vectors. For the user intent feature vector V and the intent category feature vector C, the dot product V·C is calculated, and then divided by the product of the magnitudes of the two vectors, |V|×|C|, to obtain the cosine similarity value. For example, the system calculates that the similarity between the user intent feature vector and the "weather query" category is 0.85, with the "navigation request" category it is 0.42, with the "music playback" category it is 0.38, with the "alarm clock setting" category it is 0.25, and with the "message sending" category it is 0.20.

[0112] The category similarity values ​​are normalized to obtain the probability distribution. The Softmax normalization method is used to convert each similarity value into a probability value, ensuring that the sum of all probability values ​​is 1. For category similarity values ​​[0.85, 0.42, 0.38, 0.25, 0.20, ...], the corresponding probability distribution [0.65, 0.15, 0.10, 0.05, 0.03, ...] is calculated.

[0113] The intent category with the highest probability is selected as the user intent category output. In this example, the probability of the "weather query" category is 0.65, which is significantly higher than the probabilities of other categories. Therefore, the system outputs "weather query" as the user's intent category. When the user inputs "I want to check tomorrow's weather forecast," the system correctly identifies the user's intent as querying weather information.

[0114] To improve the accuracy of intent recognition, a minimum confidence threshold of 0.5 is set. The recognition result is only confirmed when the highest probability value exceeds this threshold; otherwise, the system returns a "unclear intent" result and requests more information from the user. In this example, the probability of the "weather query" category (0.65) exceeds the threshold of 0.5, therefore the system confirms the user's intent as "weather query".

[0115] In this embodiment, the fused features are mapped to the semantic space through nonlinear transformation, which can better capture the complex semantic relationships between features and effectively extract deep semantic information. The intent recognition method based on similarity calculation and probability normalization is adopted to transform the intent recognition problem into a probability distribution prediction task. It not only outputs the final intent category, but also provides the confidence level of each intent category, providing richer basis for subsequent intent understanding and dialogue decision-making.

[0116] In one alternative implementation,

[0117] The importance distribution of key entities in historical dialogues is calculated based on word frequency and information gain. A sliding window is used to count the co-occurrence frequency of entities and events and construct an association network. The core semantic skeleton is obtained by pruning based on association strength. Key paths are extracted and semantic coherence is calculated. The importance of historical dialogue information is ranked by combining path weights and coherence. Based on the ranking results, key information includes:

[0118] Obtain historical dialogue content, extract entities from the historical dialogue content as candidate entities, calculate the word frequency score and information gain score of the candidate entities, obtain the entity importance distribution by weighted sum of word frequency score and information gain score, and determine key entities based on the entity importance distribution.

[0119] The sliding window size is determined based on the length of the historical dialogue content. The co-occurrence frequency of key entities and events is counted within the sliding window. The association strength is calculated based on the co-occurrence frequency, and an association network is constructed. The weights of the association network are set based on the association strength and key entities. The pruning threshold is determined based on the mean and standard deviation of the weights. The associations with weights greater than the pruning threshold in the association network are retained to obtain the core semantic skeleton.

[0120] The path length is determined based on the number of shortest paths between nodes in the core semantic skeleton. A decay factor is set based on the path length. The node centrality score is calculated using the path length and decay factor to extract the key path. The semantic coherence is obtained by calculating the cosine similarity of the semantic vectors of adjacent nodes in the key path.

[0121] Historical dialogue information is obtained based on the critical path. The centrality score of the critical path in the historical dialogue information is calculated to obtain the path weight. The importance ranking value is obtained by combining the path weight and the weighted sum of semantic coherence. The historical dialogue information is ranked according to the importance ranking value and the key information is determined.

[0122] The term "entity" refers to a specific object within the historical dialogue content, including identifiable, independent individuals such as people, locations, organizations, product names, and times. For example, in a customer service dialogue scenario, an entity could be a specific product model, user account, order number, or shipping address. An "event" refers to a specific action, state change, or scene description that occurs during the dialogue, including the action subject, the action itself, and related elements. For example, order placement, refund requests, logistics and delivery, and product usage in a customer service dialogue all fall under the category of events. Events often consist of interactions between multiple entities, reflecting dynamic information within the dialogue content.

[0123] Candidate entities are extracted from historical dialogue content, including named entities such as names of people, places, organizations, times, and quantities. For each candidate entity, its word frequency score is calculated, which is the number of times the entity appears in the historical dialogue divided by the total number of times all entities appear in the historical dialogue. For example, in a historical dialogue about "product sales," "mobile phone" appears 15 times, "computer" appears 8 times, and "tablet" appears 5 times, with a total of 50 appearances. Therefore, the word frequency score for "mobile phone" is 0.3, the word frequency score for "computer" is 0.16, and the word frequency score for "tablet" is 0.1.

[0124] For each candidate entity, an information gain score is calculated to measure its ability to distinguish the dialogue topic. The calculation method involves dividing the historical dialogue content into a subset containing the entity and a subset not containing it, calculating the information entropy of each subset separately, and subtracting the weighted average information entropy of the two subsets from the original information entropy. For example, if the above dialogue is divided into two subsets based on whether it contains "phone," the original information entropy is 1.5. After the division, the information entropy of the subset containing "phone" is 0.8 with a weight of 0.4, and the information entropy of the subset not containing "phone" is 0.6 with a weight of 0.6. Therefore, the information gain score for "phone" is 1.5 - (0.8 × 0.4 + 0.6 × 0.6) = 0.82.

[0125] The word frequency score and information gain score are weighted and fused to obtain the entity importance distribution. The weighting coefficients can be adjusted according to the specific application scenario. For example, if the word frequency weight is 0.4 and the information gain weight is 0.6, then the entity importance of "mobile phone" is 0.4 × 0.3 + 0.6 × 0.82 = 0.612. Based on the entity importance distribution, the top N entities with the highest scores are selected as key entities. In this embodiment, if N = 2, then "mobile phone" and "computer" are selected as key entities.

[0126] The size of the sliding window is determined based on the length of the historical dialogue content, generally set to 1 / 5 to 1 / 3 of the number of dialogue rounds. For example, for a historical content containing 30 rounds of dialogue, the sliding window size would be 10 rounds. Within the sliding window, the co-occurrence frequency of key entities and events is counted. An event consists of a verb and its related components. For example, in a certain sliding window, "mobile phone" and "sales" co-occurred 8 times, and "computer" and "sales" co-occurred 5 times.

[0127] Association strength is calculated based on co-occurrence frequency. Association strength equals the co-occurrence frequency divided by the total number of words in the window, multiplied by a certain adjustment coefficient. For example, if the total number of words in the window is 200 and the adjustment coefficient is 100, then the association strength of "mobile phone-sales" is 8 ÷ 200 × 100 = 4, and the association strength of "computer-sales" is 5 ÷ 200 × 100 = 2.5. When constructing the association network, key entities and events are used as nodes, and association strength is used as the weight of the edges. The pruning threshold is determined based on the mean and standard deviation of the weights of all edges in the association network, usually set to the mean minus 0.5 times the standard deviation. For example, if the mean weight of the edges in the association network is 3 and the standard deviation is 1, then the pruning threshold is 3 - 0.5 × 1 = 2.5. Associations with weights greater than the pruning threshold are retained, forming the core semantic skeleton. In this embodiment, the "mobile phone-sales" association is retained, while the "computer-sales" association is located at the threshold edge.

[0128] The path length is determined based on the number of shortest paths between nodes in the core semantic skeleton. For example, the path from "user" to "phone" to "sales" to "delivery" constitutes a path of length 3. A decay factor is set based on the path length. The decay factor is typically a value between 0.8 and 0.9 to reduce the importance of longer paths. The node centrality score is calculated using the path length and the decay factor. The centrality score is equal to the number of times the node appears in all paths multiplied by the sum of the decay factors of the corresponding paths. For example, if the "phone" node appears in 3 paths of different lengths, with decay factors of 0.9, 0.81, and 0.729 respectively (assuming a decay factor of 0.9), then the node centrality score of "phone" is 0.9 + 0.81 + 0.729 = 2.439.

[0129] When extracting the critical path, the node with the highest node centrality score is selected as the starting point. Connecting nodes are then selected according to their association strength from high to low to construct the critical path. The cosine similarity of the semantic vectors of adjacent nodes in the critical path is calculated to obtain semantic coherence. For example, in the path "user-mobile phone-sales-delivery", the cosine similarities of adjacent node pairs are 0.75, 0.82, and 0.68, respectively, and the semantic coherence of this path is the average of these values, 0.75.

[0130] Historical dialogue information is obtained based on the critical path. The centrality score of the critical path in the historical dialogue information is calculated to obtain the path weight. The importance ranking value is obtained by combining the weighted sum of the path weight and semantic coherence. For example, if the path weight is 0.7 and the semantic coherence weight is 0.3, then for a path with a centrality score of 2.439 and semantic coherence of 0.75, the importance ranking value is 0.7 × 2.439 + 0.3 × 0.75 = 1.9323. The historical dialogue information is then sorted according to the importance ranking value, and the top M pieces of information are selected as key information. For example, if M = 5, then the top 5 dialogue pieces with the highest importance ranking values ​​are selected as key information, including: "User inquires about phone model," "Salesperson introduces phone features," "User decides to buy phone," "Salesperson explains delivery method," and "User confirms delivery time," etc.

[0131] In this embodiment, a dual evaluation mechanism of word frequency score and information gain score can comprehensively measure the importance of entities in the dialogue content. It considers both the frequency of entity occurrence and the entity's contribution to information differentiation, making the extraction of key entities more accurate and reasonable. A co-occurrence statistical method based on sliding window is adopted, and an association network is constructed by combining association strength and entity importance. The core semantic skeleton is obtained through adaptive pruning, which effectively preserves the core semantic structure in the dialogue content and avoids interference from irrelevant information. Path length and decay factor are introduced to calculate node centrality, and the quality of key paths is evaluated in combination with semantic coherence. This ensures that the extracted key paths have both strong structural importance and maintain semantic coherence, thereby improving the accuracy of key information extraction.

[0132] Figure 3 This chart compares the F1 score trends across different dialogue rounds in this invention. As the number of dialogue rounds increases, the F1 score of this technical solution (sliding window + association strength, circular markers) significantly improves, increasing from 0.50 at 10 rounds to 0.89 at 60 rounds, showing a stable upward trend. In contrast, the PMI co-occurrence statistical method (square markers) only achieves an F1 score of 0.71 at 60 rounds, while the WordNet semantic network method (triangle markers) is even lower, at only 0.55. Data shows that when the number of dialogue rounds exceeds 20, the advantage of this technical solution begins to expand significantly. At 50 rounds, the F1 score of this technical solution (0.85) is 27% higher than the PMI co-occurrence statistical method (0.67) and 70% higher than the WordNet semantic network method (0.50).

[0133] The performance difference primarily stems from the fact that the sliding window technique used in this solution better captures dynamic changes in the dialogue, while the association strength calculation method accurately assesses the semantic association between entities and events. Traditional PMI co-occurrence statistics methods, although capable of calculating word pairs...

[0134] In one alternative implementation,

[0135] Association strength is calculated based on co-occurrence frequency, and an association network is constructed. Weights of the association network are set based on association strength and key entities. A pruning threshold is determined based on the mean and standard deviation of the weights. Associations with weights greater than the pruning threshold are retained to obtain the core semantic skeleton, which includes:

[0136] Obtain the co-occurrence frequency of entities in historical dialogue content, divide the co-occurrence frequency by the square root of the product of the number of times each entity appears to obtain the association strength, and construct an association network based on the association strength;

[0137] Calculate the importance score of the key entity, and set the weight of the association network as the product of the association strength and the average of the importance scores of the key entity;

[0138] The community detection algorithm is used to divide the network into communities to obtain multiple communities. The number of edge sets and node sets within each community are counted. The connection density within the community is obtained by dividing the number of edge sets by half of the product of the number of node sets and the number of node sets minus one.

[0139] Calculate the mean and standard deviation of all weights in the network, and use the sum of the products of the mean and standard deviation of the weights and the first preset coefficient as the benchmark threshold. Then, use the sum of the products of the benchmark threshold and the internal connection density of the community and the second preset coefficient to obtain the adaptive pruning threshold for each community.

[0140] By comparing the adaptive pruning threshold of the community to which each edge belongs in the association network with the weight of the current edge, edges with weights greater than the adaptive pruning threshold of the corresponding community are retained, and edges with weights less than the adaptive pruning threshold of the corresponding community are removed to obtain the core semantic skeleton.

[0141] The system extracts entities from historical dialogues and uses named entity recognition technology to identify individuals, locations, organizations, and times from the historical dialogue text. For example, in a historical dialogue discussing technology products, entities such as "smartphone," "tablet," "laptop," and "smart home" can be identified.

[0142] After acquiring the entities, calculate the co-occurrence frequency between them. Two entities appearing in the same dialog window constitute one co-occurrence. The dialog window can be a sentence, a round, or a fixed range of characters. For example, if the dialog window is set to a single round, the number of times "smartphone" and "tablet" co-occur in the same round is counted as 15, the total number of times "smartphone" appears alone is 30, and the total number of times "tablet" appears alone is 25.

[0143] When calculating association strength, the co-occurrence frequency is divided by the square root of the product of the frequency of each entity. For example, the association strength between "smartphone" and "tablet" is 15 divided by the square root of (30 multiplied by 25), which is approximately 0.55. The association strength of all entity pairs is calculated, and an association network is constructed, where nodes are entities, edges are associations between entities, and the initial weight of the edges is the association strength.

[0144] Calculate the importance score of key entities. This can be done using degree centrality, which calculates the number of connections each entity has with other entities. For example, "smartphone" is associated with 15 other entities, so its degree centrality is 15. Algorithms such as PageRank can also be used to calculate entity importance. Using "smartphone" as an example, let's assume its importance score is 0.85.

[0145] Adjust the weights of the association network by multiplying the association strength by the average importance score of the two connected key entities, and use this as the new weight for the edge of the association network. Taking "smartphone" and "tablet" as an example, assuming the importance score of "tablet" is 0.75, the weight of the edge between these two entities is adjusted to 0.55 multiplied by (0.85 plus 0.75) divided by 2, that is, 0.55 multiplied by 0.8, which equals 0.44.

[0146] Community detection algorithms are used to partition the network into communities, such as the Louvain algorithm or label propagation algorithm. The partitioning result divides the network into multiple communities, with closer connections between nodes within each community. Assume the algorithm divides the network into three communities, each containing entities related to "communication equipment," "home appliances," and "office equipment," respectively.

[0147] For each community, count the number of internal edge sets and node sets. Taking the "Communication Equipment" community as an example, it contains 10 nodes (entities) and 30 edges (associations). Calculate the internal connection density of the community, which is the number of edge sets divided by half the product of the number of node sets and the number of node sets minus one. For the "Communication Equipment" community, the connection density is 30 divided by half of (10 multiplied by 9), which is 30 divided by 45, equal to 0.67.

[0148] Calculate the mean and standard deviation of all weights in the network. Assume the mean weight of all edges in the network is 0.3 and the standard deviation is 0.15. Set the sum of the products of the mean and standard deviation of the weights and a first preset coefficient as the baseline threshold. Assuming the first preset coefficient is set to 1.0, the baseline threshold is 0.3 plus 0.15 multiplied by 1.0, which equals 0.45.

[0149] An adaptive pruning threshold is set for each community. The adaptive pruning threshold for that community is the sum of the product of the baseline threshold, the community's internal connection density, and a second preset coefficient. Assuming the second preset coefficient is set to -0.2, the adaptive pruning threshold for the "Communication Equipment" community is 0.45 plus 0.67 multiplied by -0.2, approximately equal to 0.32. For communities with a connection density higher than 0.8, the adaptive pruning threshold is lowered accordingly, retaining more edges; while for communities with a connection density lower than 0.2, the adaptive pruning threshold is increased, retaining only edges with higher weights.

[0150] Compare the adaptive pruning threshold of the community to which each edge belongs in the network with the weight of the current edge. If the weight of the edge is greater than the adaptive pruning threshold of its community, the edge is retained; otherwise, the edge is removed. For example, the edge weight between "smartphone" and "tablet" is 0.44, which is greater than the adaptive pruning threshold of 0.32 for the "communication device" community, so the edge is retained.

[0151] In this embodiment, the association strength is calculated by combining co-occurrence frequency with normalization, which can effectively eliminate the influence of differences in entity occurrence frequency. By standardizing the calculation by dividing the co-occurrence frequency by the square root of the product of the occurrence frequency of each entity, the association strength calculation becomes more objective, avoiding the dominance of high-frequency entities in the association strength calculation. The differentiated processing method based on community structure can selectively remove weak connections between communities while maintaining high-density connections within the community, making the network simplification more reasonable. The adaptive threshold setting method can dynamically adjust the pruning criteria according to the characteristics of different communities, avoiding the problems of over-pruning or redundant retention that may be caused by a single fixed threshold.

[0152] Figure 4 This is a flowchart illustrating the core semantic skeleton construction process of the intelligent interactive question-answering method based on multimodal large model intent recognition in an embodiment of the present invention.

[0153] In one alternative implementation,

[0154] Generate response content based on the key information and return it to the customer; update the associated network based on the current dialogue content, including:

[0155] Based on the key information, generate response content that includes dialogue intent, user requests, and solutions, and return the response content to the customer;

[0156] The new entity is obtained by extracting entity, attribute, and relation triples from the current dialogue content. The co-occurrence frequency and semantic similarity between the new entities are calculated to obtain the entity weight. The interaction weight is obtained by calculating the association strength between the new entity and the original entity.

[0157] Add new entities as nodes, and entity weights and interaction weights as edges to the original association network. Recalculate the adaptive pruning threshold for the expanded association network based on the internal connection density of the community, and retain edges with weights greater than the corresponding community adaptive pruning threshold to obtain the updated association network.

[0158] Based on the acquired key information, analyze the user's dialogue intent, identify the user's specific needs, and combine this with problem-solving experience from historical dialogues to generate a complete response that includes intent understanding, needs analysis, and solutions. The generated response must be logically coherent and semantically complete to ensure it accurately conveys its message to the customer and resolves their problem.

[0159] From the newly added dialogue content, entity information and its corresponding attribute features are extracted through semantic analysis. Simultaneously, various relationships between entities are identified, and this information is organized into entity-attribute-relationship triples. For each newly extracted entity, the number of times it co-occurs within the dialogue window is counted, and the semantic similarity between entities is calculated. The weighted result of these two indicators is used as the association weight between entities. Furthermore, the association strength between the new entity and existing entities in the association network is calculated as the interaction weight between entities.

[0160] The newly extracted entities are added as new nodes to the existing association network, and the calculated inter-entity weights and interaction weights are added as connecting edges. The expanded association network is then re-divided into communities, and the connection density within each community is calculated. Based on the updated community structure and weight distribution characteristics, the adaptive pruning threshold for each community is recalculated. Finally, the network is optimized according to the new pruning threshold, retaining important connections with larger weights, resulting in an updated association network that reflects the current dialogue state.

[0161] For example, in a smart customer service scenario, a user reports: "My phone screen has been flickering for a week now, and I want to apply for after-sales repair. This phone is a model A purchased offline last year and is still under warranty."

[0162] Based on key information analysis, we know the user's intent is to request mobile phone repair, and their core request is to resolve the screen flickering issue. Therefore, we generated the following response: "We understand that your Model A phone is experiencing a screen flickering problem. Since this model is still under warranty, we recommend that you have it repaired through official after-sales channels. The nearest official after-sales service center is located at..."

[0163] New entities are extracted from the current dialogue, including "Model A", "Screen", "Flickering", "Offline", and "Warranty Period", as well as entity attributes such as "Purchase Time: Last Year" and "Fault Duration: One Week". Co-occurrence relationships and semantic similarity between the new entities are calculated to obtain entity weights; for example, "Model A - Screen" and "Screen - Flickering" have higher weights. Simultaneously, the association strength between the new entities and existing entities (such as "After-sales Service" and "Official Channels") is calculated.

[0164] These new nodes and edges are added to the existing network, and the communities are re-divided into groups such as "Product Information Community," "Fault Description Community," and "After-Sales Service Community." The internal connection density of each community is calculated, and the pruning threshold is updated accordingly. Important connections are retained to obtain a new relational network that can better represent the core semantic structure of the current dialogue.

[0165] In this embodiment, a complete response content containing dialogue intent, user demands, and solutions is generated by parsing key information. The multi-dimensional information organization method ensures the comprehensiveness of the response content. The structured response generation method can not only accurately understand and respond to user needs, but also provide targeted solutions, thereby improving the service quality of the dialogue system. By simultaneously considering co-occurrence frequency and semantic similarity to calculate the weights between entities, it reflects both the statistical correlation of entities in the dialogue and the degree of semantic correlation between entities, making the characterization of entity relationships more comprehensive and accurate. The dynamic knowledge integration method enables the association network to continuously accumulate and update dialogue information, improving the timeliness and completeness of knowledge representation.

[0166] A second aspect of this invention provides an intelligent interactive question-answering system based on multimodal large model intent recognition, comprising:

[0167] The first unit is used to acquire multimodal interaction information input by the user and extract features to obtain multimodal feature vectors;

[0168] The second unit is used to calculate the mutual information between feature vectors of different modalities, construct a cross-modal semantic mapping matrix based on the mutual information, map the feature vectors of different modalities to a unified semantic space using the cross-modal semantic mapping matrix, obtain unified semantic features, calculate the semantic similarity with prior knowledge in a preset knowledge base, determine supplementary information, and fuse the supplementary information with the unified semantic features to obtain fused features.

[0169] The third unit is used to construct a user intent feature vector based on the fusion features, perform similarity matching between the user intent feature vector and a preset intent category library, and select the category with the highest similarity as the user intent category.

[0170] The fourth unit is used to calculate the importance distribution of key entities in historical dialogues based on word frequency and information gain. It uses a sliding window to count the co-occurrence frequency of entities and events and constructs an association network. It prunes the core semantic skeleton based on the association strength, extracts key paths and calculates semantic coherence, and ranks the importance of historical dialogue information by combining path weights and coherence. It determines key information based on the ranking results.

[0171] The fifth unit is used to generate response content based on the key information and return it to the customer, and to update the associated network based on the current dialogue content.

[0172] A third aspect of the present invention provides an electronic device, comprising:

[0173] A processor and a memory for storing processor-executable instructions, wherein the processor is configured to invoke instructions stored in the memory to perform the aforementioned method.

[0174] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0175] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent interactive question-answering method based on multimodal large model intent recognition, characterized in that, include: Acquire multimodal interaction information from user input and extract features to obtain multimodal feature vectors; Calculate the mutual information between feature vectors of different modalities, construct a cross-modal semantic mapping matrix based on the mutual information, map the feature vectors of different modalities to a unified semantic space using the cross-modal semantic mapping matrix, obtain unified semantic features, calculate the semantic similarity with prior knowledge in a preset knowledge base, determine supplementary information, and fuse the supplementary information with the unified semantic features to obtain fused features; A user intent feature vector is constructed based on the fusion features. The user intent feature vector is then matched with a preset intent category library for similarity, and the category with the highest similarity is selected as the user intent category. The importance distribution of key entities in historical dialogues is calculated based on word frequency and information gain. The co-occurrence frequency of entities and events is counted using a sliding window and an association network is constructed. The core semantic skeleton is obtained by pruning based on the association strength. Key paths are extracted and semantic coherence is calculated. The importance of historical dialogue information is ranked by combining path weight and coherence. Key information is determined based on the ranking results. The response content is generated based on the key information and returned to the customer, and the associated network is updated based on the current dialogue content.

2. The method according to claim 1, characterized in that, The mutual information between feature vectors of different modalities is calculated. Based on the mutual information, a cross-modal semantic mapping matrix is ​​constructed. The cross-modal semantic mapping matrix is ​​used to map the feature vectors of different modalities to a unified semantic space to obtain unified semantic features. The semantic similarity with prior knowledge in a preset knowledge base is calculated. Supplementary information is determined and fused with the unified semantic features to obtain fused features, including: Obtain the multimodal feature vector to be processed, which includes text feature vector and image feature vector; Calculate the joint probability distribution and edge probability distribution of the text feature vector and the image feature vector, and calculate the mutual information between the text feature vector and the image feature vector based on the joint probability distribution and the edge probability distribution; A cross-modal semantic mapping matrix is ​​constructed based on the mutual information. The value of each element in the cross-modal semantic mapping matrix is ​​calculated by a normalized exponential function adjusted by a temperature parameter. The temperature parameter is used to control the distribution of semantic association strength in the cross-modal semantic mapping matrix. Based on the cross-modal semantic mapping matrix, the text feature vector and the image feature vector are mapped to a unified semantic space to obtain unified semantic features. The similarity between the unified semantic feature and prior knowledge in the preset knowledge base is calculated. Prior knowledge with a similarity greater than a preset threshold is used as supplementary information. The unified semantic feature is used as the query vector, and the supplementary information is used as key-value pairs. Information fusion is performed through a multi-head attention mechanism to obtain the fused feature.

3. The method according to claim 2, characterized in that, Calculate the similarity between the unified semantic features and prior knowledge in a preset knowledge base, and include prior knowledge with similarity greater than a preset threshold as supplementary information: The unified semantic features are decomposed into a core semantic variable set and a context semantic variable set. A condition dependency graph is constructed based on the core semantic variable set and the context semantic variable set. According to the structure of the condition dependency graph, the initial probability distribution parameters are calculated using maximum likelihood estimation. The optimized probability distribution parameters are then iteratively optimized using the expectation-maximization algorithm. The initial similarity value between each prior knowledge item in the preset knowledge base and the unified semantic feature is calculated based on the optimized probability distribution parameters, and the final similarity value is obtained by approximating the initial similarity value based on the Monte Carlo sampling method. The similarity difference between different prior knowledge is calculated to obtain a similarity difference value. An adaptive similarity threshold is constructed based on the similarity difference value. The adaptive similarity threshold is dynamically adjusted according to the feedback score of the prior knowledge to obtain an adjusted similarity threshold. Prior knowledge with a final similarity value greater than the adjusted similarity threshold is identified as candidate supplementary information. For each candidate supplementary information, a joint similarity value is calculated based on the final similarity value and the similarity difference value, and the candidate supplementary information with the largest joint similarity value is selected as the supplementary information output.

4. The method according to claim 1, characterized in that, A user intent feature vector is constructed based on the fused features. The user intent feature vector is then matched with a preset intent category library for similarity, and the category with the highest similarity is selected as the user intent category, including: The initial intent features are obtained by acquiring fusion features and mapping them to the semantic space through nonlinear transformation. The initial intent features are then subjected to dimensionality reduction to obtain the user intent feature vector. The similarity between the user intent feature vector and each intent category in the preset intent category library is calculated to obtain a category similarity value. The category similarity value is normalized to obtain a probability distribution. The intent category with the highest probability is selected as the user intent category for output.

5. The method according to claim 1, characterized in that, The importance distribution of key entities in historical dialogues is calculated based on word frequency and information gain. A sliding window is used to count the co-occurrence frequency of entities and events and construct an association network. The core semantic skeleton is obtained by pruning based on association strength. Key paths are extracted and semantic coherence is calculated. The importance of historical dialogue information is ranked by combining path weights and coherence. Based on the ranking results, key information includes: Obtain historical dialogue content, extract entities from the historical dialogue content as candidate entities, calculate the word frequency score and information gain score of the candidate entities, obtain the entity importance distribution by weighted sum of word frequency score and information gain score, and determine key entities based on the entity importance distribution. The sliding window size is determined based on the length of the historical dialogue content. The co-occurrence frequency of key entities and events is counted within the sliding window. The association strength is calculated based on the co-occurrence frequency, and an association network is constructed. The weights of the association network are set based on the association strength and key entities. The pruning threshold is determined based on the mean and standard deviation of the weights. The associations with weights greater than the pruning threshold in the association network are retained to obtain the core semantic skeleton. The path length is determined based on the number of shortest paths between nodes in the core semantic skeleton. A decay factor is set based on the path length. The node centrality score is calculated using the path length and decay factor to extract the key path. The semantic coherence is obtained by calculating the cosine similarity of the semantic vectors of adjacent nodes in the key path. Historical dialogue information is obtained based on the critical path. The centrality score of the critical path in the historical dialogue information is calculated to obtain the path weight. The importance ranking value is obtained by combining the path weight and the weighted sum of semantic coherence. The historical dialogue information is ranked according to the importance ranking value and the key information is determined.

6. The method according to claim 5, characterized in that, Association strength is calculated based on co-occurrence frequency, and an association network is constructed. Weights of the association network are set based on association strength and key entities. A pruning threshold is determined based on the mean and standard deviation of the weights. Associations with weights greater than the pruning threshold are retained to obtain the core semantic skeleton, which includes: Obtain the co-occurrence frequency of entities in historical dialogue content, divide the co-occurrence frequency by the square root of the product of the number of times each entity appears to obtain the association strength, and construct an association network based on the association strength; Calculate the importance score of the key entity, and set the weight of the association network as the product of the association strength and the average of the importance scores of the key entity; The community detection algorithm is used to divide the network into communities to obtain multiple communities. The number of edge sets and node sets within each community are counted. The connection density within the community is obtained by dividing the number of edge sets by half of the product of the number of node sets and the number of node sets minus one. Calculate the mean and standard deviation of all weights in the network, and use the sum of the products of the mean and standard deviation of the weights and the first preset coefficient as the benchmark threshold. Then, use the sum of the products of the benchmark threshold and the internal connection density of the community and the second preset coefficient to obtain the adaptive pruning threshold for each community. By comparing the adaptive pruning threshold of the community to which each edge belongs in the association network with the weight of the current edge, edges with weights greater than the adaptive pruning threshold of the corresponding community are retained, and edges with weights less than the adaptive pruning threshold of the corresponding community are removed to obtain the core semantic skeleton.

7. The method according to claim 1, characterized in that, Generate response content based on the key information and return it to the customer; update the associated network based on the current dialogue content, including: Based on the key information, generate response content that includes dialogue intent, user requests, and solutions, and return the response content to the customer; The new entity is obtained by extracting entity, attribute, and relation triples from the current dialogue content. The co-occurrence frequency and semantic similarity between the new entities are calculated to obtain the entity weight. The interaction weight is obtained by calculating the association strength between the new entity and the original entity. Add new entities as nodes, and entity weights and interaction weights as edges to the original association network. Recalculate the adaptive pruning threshold for the expanded association network based on the internal connection density of the community, and retain edges with weights greater than the corresponding community adaptive pruning threshold to obtain the updated association network.

8. An intelligent interactive question-answering system based on multimodal large model intent recognition, used to implement the method of any one of claims 1-7, characterized in that, include: The first unit is used to acquire multimodal interaction information input by the user and extract features to obtain multimodal feature vectors; The second unit is used to calculate the mutual information between feature vectors of different modalities, construct a cross-modal semantic mapping matrix based on the mutual information, map the feature vectors of different modalities to a unified semantic space using the cross-modal semantic mapping matrix, obtain unified semantic features, calculate the semantic similarity with prior knowledge in a preset knowledge base, determine supplementary information, and fuse the supplementary information with the unified semantic features to obtain fused features. The third unit is used to construct a user intent feature vector based on the fusion features, perform similarity matching between the user intent feature vector and a preset intent category library, and select the category with the highest similarity as the user intent category. The fourth unit is used to calculate the importance distribution of key entities in historical dialogues based on word frequency and information gain. It uses a sliding window to count the co-occurrence frequency of entities and events and constructs an association network. It prunes the core semantic skeleton based on the association strength, extracts key paths and calculates semantic coherence, and ranks the importance of historical dialogue information by combining path weights and coherence. It determines key information based on the ranking results. The fifth unit is used to generate response content based on the key information and return it to the customer, and to update the associated network based on the current dialogue content.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images