Intelligent customer service interaction system and method, and storage medium
By leveraging the multimodal perception, intent recognition, and dynamic knowledge base management of the intelligent customer service interaction system, precise multimodal combination response strategies are generated, solving the problem that traditional customer service interaction systems cannot effectively integrate multi-dimensional inputs and improving user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 湖北消费金融股份有限公司
- Filing Date
- 2025-06-18
- Publication Date
- 2026-07-31
AI Technical Summary
Traditional customer service interaction systems often rely on single-modal interaction, which cannot effectively integrate multi-dimensional user input, resulting in a poor user experience.
An intelligent customer service interaction system was designed, including a multimodal perception module, an intent recognition module, a dynamic knowledge base management module, and an interaction decision module. By collecting multimodal data streams of user input in real time, feature extraction and intent recognition are performed, semantic relevance scores are calculated, and multimodal combined response strategies are generated.
It enables accurate acquisition of multimodal combined interactive response strategies, thereby improving the user experience.
Smart Images
Figure CN120892610B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of the intersection of artificial intelligence and customer service, and in particular to an intelligent customer service interaction system, method and storage medium. Background Technology
[0002] With the development of artificial intelligence technology, intelligent customer service interaction systems are evolving from single-modal interaction to multimodal integration. Traditional customer service interaction systems mostly rely on single-modal interaction (such as text or voice), which can only process structured text and has a low accuracy rate in understanding spoken expressions and complex contexts. They also cannot effectively integrate multi-dimensional user input (such as voice emotion, image scenes, and operational behaviors), resulting in significant response biases in the generated single-modal interaction response strategies and a poor user experience. Therefore, how to accurately acquire multimodal combined interaction response strategies has become an urgent problem to be solved.
[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention
[0004] The main objective of this invention is to provide an intelligent customer service interaction system, method, and storage medium, addressing the technical problem of how to accurately acquire multimodal combined interactive response strategies.
[0005] To achieve the above objectives, the present invention provides an intelligent customer service interaction system, which includes a multimodal perception module, an intent recognition module, a dynamic knowledge base management module, and an interaction decision module.
[0006] The multimodal perception module is used to collect the raw interactive data stream input by the user in real time, and to extract features from the raw interactive data stream to obtain multimodal fusion features.
[0007] The intent recognition module is used to perform intent recognition on the multimodal fusion features to obtain intent recognition results;
[0008] The dynamic knowledge base management module is used to calculate the semantic relevance score between the intent recognition result and each entity and relation in the dynamic knowledge base.
[0009] The dynamic knowledge base management module is also used to select a set of candidate knowledge nodes from the dynamic knowledge base based on the semantic relevance score, and determine the associated path information corresponding to the set of candidate knowledge nodes.
[0010] The interactive decision module is used to generate a multimodal combined response strategy based on the user terminal performance evaluation results, the candidate knowledge node set, and the associated path information.
[0011] Optionally, the multimodal perception module further includes a data processing module and a feature fusion submodule;
[0012] The data processing module is used to determine the data type corresponding to the original interactive data stream;
[0013] The data processing module is also used to perform split processing on the original interactive data stream based on the data type to obtain split data corresponding to each data type;
[0014] The data processing module is also used to extract corresponding modal features from each of the split data;
[0015] The feature fusion submodule is used to perform spatial alignment and noise suppression processing on each modal feature, and to fuse the processed modal features through a cross-modal fusion algorithm to obtain multimodal fused features.
[0016] Optionally, calculating the semantic relevance score between the intent recognition result and each entity and relation in the dynamic knowledge base includes:
[0017] Based on the entities and relations in the dynamic knowledge base, the TransE algorithm is used to generate embedding vectors for each entity and each relation.
[0018] The intent recognition result is mapped to the knowledge graph embedding space of the dynamic knowledge base to obtain a comprehensive intent vector;
[0019] Calculate the cosine similarity, Jaccard coefficient, and path hop weight between the integrated intent vector and each entity embedding vector, and calculate the similarity between the integrated intent vector and each relation embedding vector.
[0020] The semantic association score is calculated using the cosine similarity, the Jaccard coefficient, the path hop count weight, and the similarity score through a semantic association scoring formula.
[0021] Optionally, the semantic association score formula is:
[0022]
[0023] In the formula, To synthesize the intent vector, For the i-th entity vector, For the j-th relation vector, To calculate the semantic relevance score between the intent vector, the i-th entity vector, and the j-th relation vector, To synthesize the cosine similarity between the intent vector and the i-th entity vector, Let Jaccard coefficient be the coefficient between the i-th entity vector and the j-th entity vector. Let be the path hop weight between the i-th entity vector and the j-th entity vector. To synthesize the cosine similarity between the intent vector and the j-th relation vector, , , , This refers to the correlation weight coefficient.
[0024] Optionally, selecting a set of candidate knowledge nodes from the dynamic knowledge base based on the semantic relevance score includes:
[0025] The knowledge nodes in the dynamic knowledge base are sorted based on the semantic relevance score.
[0026] Based on the knowledge node sorting results, multiple candidate knowledge nodes are selected from the dynamic knowledge base according to preset selection rules, and a candidate knowledge node set is constructed based on the multiple candidate knowledge nodes.
[0027] Optionally, generating a multimodal combined response strategy based on the user terminal performance evaluation results, the candidate knowledge node set, and the associated path information includes:
[0028] Determine the attribute information of each candidate knowledge node in the candidate knowledge node set, and generate a combination of multimodal output formats based on the attribute information;
[0029] Based on the combination of the multimodal output formats, a multimodal combined response strategy is generated according to the user terminal performance evaluation results, the candidate knowledge node set, and the associated path information.
[0030] Optionally, the step of generating a multimodal combined response strategy based on the multimodal output format combination according to the user terminal performance evaluation result, the candidate knowledge node set, and the associated path information includes:
[0031] Based on the intent recognition results, determine the explicit intent classification code and the implicit emotion feature vector;
[0032] Determine the confidence scores corresponding to the explicit intention classification code and the implicit sentiment feature vector, respectively;
[0033] Calculate the confidence difference value based on the confidence score;
[0034] Determine whether the confidence difference value is greater than a preset threshold;
[0035] If the confidence difference value is less than or equal to the preset threshold, the combination of multimodal output formats is adjusted according to the user terminal performance evaluation results.
[0036] Based on the adjusted multimodal output format combination, a multimodal combined response strategy is generated according to the candidate knowledge node set and the associated path information.
[0037] Optionally, the intelligent customer service interaction system further includes a feedback optimization module:
[0038] The feedback optimization module is used to obtain the user's interaction feedback information and update the association weight coefficient of the dynamic knowledge base based on the interaction feedback information.
[0039] Furthermore, to achieve the above objectives, the present invention also proposes an intelligent customer service interaction method, the method comprising the following steps:
[0040] The system collects raw interactive data streams from user input in real time and extracts features from these raw interactive data streams to obtain multimodal fusion features.
[0041] Intent recognition is performed on the multimodal fusion features to obtain intent recognition results;
[0042] Calculate the semantic relevance score between the intent recognition result and each entity and relation in the dynamic knowledge base;
[0043] Based on the semantic relevance score, a set of candidate knowledge nodes is selected from the dynamic knowledge base, and the associated path information corresponding to the set of candidate knowledge nodes is determined.
[0044] A multimodal combined response strategy is generated based on the user terminal performance evaluation results, the candidate knowledge node set, and the associated path information. Intelligent customer service interaction is realized based on the multimodal combined response strategy.
[0045] Furthermore, to achieve the above objectives, the present invention also proposes an intelligent customer service interaction device, the device comprising: a memory, a processor, and an intelligent customer service interaction program stored in the memory and executable on the processor, the intelligent customer service interaction program being configured to implement the steps of the intelligent customer service interaction method described above.
[0046] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing an intelligent customer service interaction program, wherein the intelligent customer service interaction program, when executed by a processor, implements the steps of the intelligent customer service interaction method as described above.
[0047] This invention designs an intelligent customer service interaction system, which includes a multimodal perception module, an intent recognition module, a dynamic knowledge base management module, and an interaction decision module. The multimodal perception module collects the raw interactive data stream input by the user in real time, extracts features from the raw interactive data stream, and obtains multimodal fusion features. The intent recognition module performs intent recognition on the multimodal fusion features to obtain intent recognition results. The dynamic knowledge base management module calculates the semantic relevance score between the intent recognition result and each entity and relation in the dynamic knowledge base. Based on the semantic relevance score, the dynamic knowledge base management module selects a set of candidate knowledge nodes from the dynamic knowledge base and determines the corresponding association path information. The interaction decision module generates a multimodal combined response strategy based on the user terminal performance evaluation results, the candidate knowledge node set, and the association path information. This invention, through the cooperation of the multimodal perception module, intent recognition module, dynamic knowledge base management module, and interaction decision module, generates accurate multimodal combined response strategies based on the raw interactive data stream input by the user, thereby achieving intelligent customer service interaction and improving user experience. Attached Figure Description
[0048] Figure 1 This is a structural block diagram of the first embodiment of the intelligent customer service interaction system of the present invention;
[0049] Figure 2 This is a flowchart illustrating the first embodiment of the intelligent customer service interaction method of the present invention.
[0050] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0051] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.
[0052] Reference Figure 1 , Figure 1 This is a structural block diagram of the first embodiment of the intelligent customer service interaction system of the present invention.
[0053] It should also be noted that the memory, as a storage medium, may include intelligent customer service interaction programs.
[0054] like Figure 1 As shown, the intelligent customer service interaction system proposed in this embodiment of the invention includes a multimodal perception module 1001, an intent recognition module 1002, a dynamic knowledge base management module 1003, and an interaction decision module 1004.
[0055] The multimodal perception module 1001 is used to collect the original interactive data stream input by the user in real time, and to extract features from the original interactive data stream to obtain multimodal fusion features.
[0056] Furthermore, the multimodal perception module also includes a data processing module and a feature fusion submodule. The data processing module further includes a text processing submodule, a speech processing submodule, an image processing submodule, and a behavior analysis submodule.
[0057] The data processing module determines the data type corresponding to the original interactive data stream. If the data type is text, image, voice, or operation behavior, the original interactive data stream is split based on the data type to obtain the split data corresponding to each data type. The split data of the text type is transmitted to the text submodule, which extracts features from the split data to obtain text modal features. The split data of the voice type is transmitted to the voice submodule, which extracts features from the split data to obtain voice modal features. The split data of the image type is transmitted to the image submodule, which extracts features from the split data to obtain image modal features. The split data of the operation behavior type is transmitted to the behavior analysis submodule, which extracts features from the split data to obtain behavior modal features.
[0058] It should also be noted that the data stream corresponding to the text type is the text information entered by the user, the data stream corresponding to the image type is the image selected or entered by the user, the data stream corresponding to the voice type is the voice input by the user, and the data stream corresponding to the operation behavior type is the user's operation information on the interface, such as page switching, click coordinates, and dwell time.
[0059] The text processing submodule uses a pre-trained Bidirectional Encoder Representations from Transformers (BERT) model or a variant thereof to process the input text data and output text modal features. The text modal features are semantic embedding vectors, the identified entity set and its type, and the feature representation of the text sentiment polarity score.
[0060] The speech processing submodule combines Mel frequency cepstral coefficients or similar acoustic feature extraction techniques with an end-to-end automatic speech recognition model to process the input speech data and output speech modal features. The speech modal features are the transcribed text content (with optional confidence level), speech emotion feature vectors (such as dimensions of pleasantness, excitement, calmness, etc.), speech rate and intonation features.
[0061] Image processing submodule: Applying an object detection model (YOLOv7 model) or similar object detection and scene understanding algorithms to process the input image and / or video frame data, and outputting image modal features. The image modal features are the feature representations of the detected object list (including bounding boxes, categories, and confidence scores), scene classification labels, and feature vectors of key regions.
[0062] Behavior Analysis Submodule: Models user operation behavior sequences (such as page switching sequences, click coordinate sequences, and dwell time sequences) using Long Short-Term Memory Network (LSTM) or Temporal Convolutional Network (TCN), and outputs behavioral modal features. Behavior modal features are user operation pattern encoding vectors, intention prediction probability distributions (preliminary), and feature representations of abnormal behavior indicators (such as frequent exits and long periods of stagnation).
[0063] Furthermore, the feature fusion submodule performs spatial alignment and noise suppression on each modal feature, and fuses the processed modal features through a cross-modal fusion algorithm to obtain multimodal fused features.
[0064] In the specific implementation, the Align alignment function can be used to align features of different modalities to the same space, and the Denoise denoising function can be used to denoise the aligned features. Then, the Fusion fusion function can be used to fuse the denoised features to obtain multimodal fused features, which are represented in the form of vectors.
[0065] The intent recognition module 1002 is used to perform intent recognition on the multimodal fusion features to obtain intent recognition results.
[0066] The intent recognition results include explicit intent classification encoding and implicit emotion feature vectors.
[0067] It should be noted that the intent recognition module specifically adopts a multi-task learning framework to build a deep learning model. The input is multimodal fusion features. The model structure is as follows: a shared bottom encoder (such as the Transformer layer) extracts common features, and the upper layer connects two independent task-specific output layers, which output explicit intent classification code, implicit sentiment feature vector and confidence score.
[0068] Task 1: Explicit Intent Classification: The first output layer performs a multi-category classification task, outputting the explicit intent classification codes and corresponding confidence scores for user intents belonging to 8 predefined major categories (such as consultation, processing, complaint, inquiry, etc.) and their 32 subcategories (such as inquiring about phone bills, complaining about logistics delays, etc.).
[0069] Task 2: Implicit Sentiment Polarity Analysis: The second output layer performs regression or classification tasks, outputting implicit sentiment feature vectors and corresponding confidence scores for users' sentiment states belonging to 5 levels (e.g., very negative, negative, neutral, positive, very positive).
[0070] It should also be noted that joint training: deep learning models are trained end-to-end by minimizing the weighted loss functions of the two tasks (such as cross-entropy loss for intent classification, mean squared error or classification cross-entropy for sentiment analysis).
[0071] The dynamic knowledge base management module 1003 is used to calculate the semantic relevance score between the intent recognition result and each entity and relation in the dynamic knowledge base.
[0072] It should also be noted that the dynamic knowledge base management module uses knowledge graphs as the core storage structure, organizes domain knowledge in the form of entity-relationship-attribute triples, and introduces graph neural networks to realize dynamic optimization of knowledge nodes.
[0073] In the knowledge graph initialization phase, core entity types and association rules are defined through domain ontology modeling, and a basic graph is constructed by combining industry documents and historical interaction data. The real-time update mechanism designs an incremental learning framework that automatically triggers the knowledge node reconstruction process when the user's error correction frequency for a specific issue reaches a preset threshold or when policy and regulation changes are detected.
[0074] The system employs an embedding representation learning method to generate vectorized representations of entities and relationships, constructing a dynamically evolving association network. To address the issue of knowledge timeliness, a timeliness decay factor is introduced, automatically reducing the retrieval priority of expired clause-type knowledge. A version control mechanism is also established to track the change history of knowledge nodes. In the knowledge retrieval stage, the system adopts a hierarchical index architecture, preloading high-frequency access paths to edge computing nodes and combining reinforcement learning algorithms to optimize the retrieval path selection strategy, significantly improving the response efficiency of complex queries.
[0075] Furthermore, the processing method for calculating the semantic relevance score between the intent recognition result and each entity and relation in the dynamic knowledge base is as follows: Based on each entity and relation in the dynamic knowledge base, an entity embedding vector and a relation embedding vector are generated using the TransE algorithm; the intent recognition result is mapped to the knowledge graph embedding space of the dynamic knowledge base to obtain a comprehensive intent vector; the cosine similarity, Jaccard coefficient, and path hop weight between the comprehensive intent vector and each entity embedding vector are calculated respectively, and the similarity between the comprehensive intent vector and each relation embedding vector is calculated; the semantic relevance score is calculated using the semantic relevance scoring formula based on the cosine similarity, Jaccard coefficient, path hop weight, and similarity.
[0076] In this embodiment, it is assumed that the explicit intent classification is encoded as follows: The latent sentiment feature vector is categorize and encode explicit intentions as And the latent sentiment feature vector is By mapping the embedding function to the knowledge graph embedding space of the dynamic knowledge base, the embedding vector of explicit intent classification encoding is obtained. Embedded vectors of latent sentiment feature vectors Afterwards, by and Weighted concatenation is performed to generate a comprehensive intent vector. .
[0077]
[0078] Furthermore, the cosine similarity, Jaccard coefficient, and path hop weight between the integrated intent vector and each entity embedding vector are calculated respectively. Based on the cosine similarity, Jaccard coefficient, path hop weight, and similarity, the semantic association score is calculated using the semantic association scoring formula.
[0079]
[0080] In the formula, For entities and Path length between To synthesize the intent vector, For the i-th entity vector, For the j-th relation vector, To calculate the semantic relevance score between the intent vector, the i-th entity vector, and the j-th relation vector, To synthesize the cosine similarity between the intent vector and the i-th entity vector, Let Jaccard coefficient be the coefficient between the i-th entity vector and the j-th entity vector. Let be the path hop weight between the i-th entity vector and the j-th entity vector. To synthesize the cosine similarity between the intent vector and the j-th relation vector, , , , This refers to the correlation weight coefficient.
[0081] The dynamic knowledge base management module 1003 is further configured to select a set of candidate knowledge nodes from the dynamic knowledge base based on the semantic relevance score, and determine the associated path information corresponding to the set of candidate knowledge nodes.
[0082] Furthermore, the process of selecting a set of candidate knowledge nodes from the dynamic knowledge base based on the semantic relevance score is as follows: sorting each knowledge node in the dynamic knowledge base according to the score from high to low based on the semantic relevance score; selecting multiple candidate knowledge nodes from the dynamic knowledge base from high to low according to the sorting results of the knowledge nodes and constructing a set of candidate knowledge nodes based on the multiple candidate knowledge nodes.
[0083] It should also be noted that candidate knowledge nodes can be entity nodes or relation nodes.
[0084] The preset selection rule is a user-defined rule that selects a preset number of candidate knowledge nodes. The association path information can be understood as the connection relationship between each candidate knowledge node.
[0085] The interactive decision module 1004 is used to generate a multimodal combined response strategy based on the user terminal performance evaluation results, the candidate knowledge node set, and the associated path information.
[0086] Furthermore, the processing method for generating a multimodal combined response strategy based on the user terminal performance evaluation results, candidate knowledge node set, and associated path information is as follows: determine the attribute information of each candidate knowledge node in the candidate knowledge node set, and generate a combination of multimodal output forms based on the attribute information; generate a multimodal combined response strategy based on the combination of multimodal output forms according to the user terminal performance evaluation results, candidate knowledge node set, and associated path information.
[0087] The attribute information can be visualization resources, audio resources, and text summary complexity. Visual resources: whether the node contains information that can be visualized, such as images, charts, videos, etc.; Audio resources: whether the node contains audio content, such as voice explanations, audio clips, etc.; Text summary complexity: whether the text content of the node has been simplified or summarized, and the complexity of the summary (e.g., whether it is suitable for quick reading or requires more time to understand).
[0088] Multimodal output format combination can be understood as an information output method that combines multiple modalities (such as text, images, audio, video, etc.). For example, for a knowledge node, text description, related images, and audio explanation can be provided simultaneously.
[0089] In the specific implementation, it is also necessary to determine the confidence scores corresponding to the explicit intent classification code and the implicit sentiment feature vector respectively; calculate the confidence difference value based on the confidence score; determine whether the confidence difference value is greater than a preset threshold; if the confidence difference value is less than or equal to the preset threshold, adjust the multimodal output format combination according to the user terminal performance evaluation results; generate a multimodal combined response strategy based on the adjusted multimodal output format combination according to the candidate knowledge node set and associated path information. If the confidence difference value is greater than the preset threshold, initiate the manual review process.
[0090] It should also be noted that if the confidence difference value is less than or equal to the preset threshold, it is necessary to evaluate the output format that the user terminal can support based on its performance. For example, low-performance devices may not be able to play high-definition videos smoothly and are more suitable for text and low-resolution image output. Therefore, it is necessary to adjust the combination of multimodal output formats and select the output method that is most suitable for the performance of the user terminal while meeting the user's needs, so as to provide the best user experience.
[0091] Furthermore, the intelligent customer service interaction system also includes a feedback optimization module; the feedback optimization module acquires user interaction feedback information and updates the association weight coefficients of the dynamic knowledge base based on the interaction feedback information.
[0092] In this embodiment, the intelligent customer service interaction system includes a multimodal perception module, an intent recognition module, a dynamic knowledge base management module, and an interaction decision module. The multimodal perception module collects the raw interaction data stream input by the user in real time, extracts features from the raw interaction data stream, and obtains multimodal fusion features. The intent recognition module performs intent recognition on the multimodal fusion features to obtain intent recognition results. The dynamic knowledge base management module calculates the semantic relevance score between the intent recognition result and each entity and relation in the dynamic knowledge base. Based on the semantic relevance score, the dynamic knowledge base management module selects a set of candidate knowledge nodes from the dynamic knowledge base and determines the corresponding association path information. The interaction decision module generates a multimodal combined response strategy based on the user terminal performance evaluation results, the candidate knowledge node set, and the association path information. This embodiment, through the cooperation of the multimodal perception module, intent recognition module, dynamic knowledge base management module, and interaction decision module, generates an accurate multimodal combined response strategy based on the raw interaction data stream input by the user, thereby achieving intelligent customer service interaction and improving user experience.
[0093] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the intelligent customer service interaction method of the present invention.
[0094] like Figure 2 As shown, the intelligent customer service interaction method proposed in this embodiment of the invention includes:
[0095] Step S10: Collect the raw interactive data stream input by the user in real time, and extract features from the raw interactive data stream to obtain multimodal fusion features;
[0096] Step S20: Perform intent recognition on the multimodal fusion features to obtain the intent recognition result;
[0097] Step S30: Calculate the semantic relevance score between the intent recognition result and each entity and relation in the dynamic knowledge base;
[0098] Step S40: Select a set of candidate knowledge nodes from the dynamic knowledge base based on the semantic relevance score, and determine the association path information corresponding to the set of candidate knowledge nodes;
[0099] Step S50: Generate a multimodal combined response strategy based on the user terminal performance evaluation results, the candidate knowledge node set, and the associated path information, and realize intelligent customer service interaction based on the multimodal combined response strategy.
[0100] Other embodiments or specific implementations of the intelligent customer service interaction system of the present invention can be referred to the above-described method embodiments, and will not be repeated here.
[0101] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0102] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0104] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.
Claims
1. An intelligent customer service interaction system, characterized in that, The intelligent customer service interaction system includes a multimodal perception module, an intent recognition module, a dynamic knowledge base management module, and an interaction decision module. The multimodal perception module is used to collect the raw interactive data stream input by the user in real time and determine the data type corresponding to the raw interactive data stream. The data type includes text type, voice type, image type and operation behavior type. The original interactive data stream is split based on the data type to obtain split data corresponding to each data type; and the corresponding modal features are extracted from each split data. Spatial alignment and noise suppression are performed on each modal feature, and the processed modal features are fused using a cross-modal fusion algorithm to obtain multimodal fused features; The intent recognition module is used to perform intent recognition on the multimodal fusion features to obtain intent recognition results, which include explicit intent classification codes and implicit emotion feature vectors. The dynamic knowledge base management module is used to calculate the semantic relevance score between the intent recognition result and each entity and relation in the dynamic knowledge base. The dynamic knowledge base management module is also used to select a set of candidate knowledge nodes from the dynamic knowledge base based on the semantic relevance score, and determine the associated path information corresponding to the set of candidate knowledge nodes. The interactive decision module is used to determine an explicit intent classification code and an implicit emotion feature vector based on the intent recognition result, determine the confidence score corresponding to the explicit intent classification code and the implicit emotion feature vector respectively, calculate the confidence difference value based on the confidence score, and determine whether the confidence difference value is greater than a preset threshold. If the confidence difference value is less than or equal to the preset threshold, the combination of multimodal output formats is adjusted according to the user terminal performance evaluation results. Based on the adjusted multimodal output format combination, a multimodal combined response strategy is generated according to the candidate knowledge node set and the associated path information.
2. The system as described in claim 1, characterized in that, The calculation of the semantic relevance score between the intent recognition result and each entity and relation in the dynamic knowledge base includes: Based on the entities and relations in the dynamic knowledge base, the TransE algorithm is used to generate embedding vectors for each entity and each relation. The intent recognition result is mapped to the knowledge graph embedding space of the dynamic knowledge base to obtain a comprehensive intent vector; Calculate the cosine similarity, Jaccard coefficient, and path hop weight between the integrated intent vector and each entity embedding vector, and calculate the similarity between the integrated intent vector and each relation embedding vector. The semantic association score is calculated using the cosine similarity, the Jaccard coefficient, the path hop count weight, and the similarity score through a semantic association scoring formula.
3. The system as described in claim 2, characterized in that, The semantic association scoring formula is as follows: In the formula, To synthesize the intent vector, For the i-th entity vector, For the j-th relation vector, To calculate the semantic relevance score between the intent vector, the i-th entity vector, and the j-th relation vector, To synthesize the cosine similarity between the intent vector and the i-th entity vector, Let Jaccard coefficient be the coefficient between the i-th entity vector and the j-th entity vector. Let be the path hop weight between the i-th entity vector and the j-th entity vector. To synthesize the cosine similarity between the intent vector and the j-th relation vector, , , , This refers to the correlation weight coefficient.
4. The system as described in claim 3, characterized in that, The step of selecting a set of candidate knowledge nodes from the dynamic knowledge base based on the semantic relevance score includes: The knowledge nodes in the dynamic knowledge base are sorted based on the semantic relevance score. Based on the knowledge node sorting results, multiple candidate knowledge nodes are selected from the dynamic knowledge base according to preset selection rules, and a candidate knowledge node set is constructed based on the multiple candidate knowledge nodes.
5. The system as described in claim 1, characterized in that, The intelligent customer service interaction system also includes a feedback optimization module: The feedback optimization module is used to obtain the user's interaction feedback information and update the association weight coefficient of the dynamic knowledge base based on the interaction feedback information.
6. An intelligent customer service interaction method, applied to the intelligent customer service interaction system of claim 1, characterized in that, The method includes the following steps: The system collects raw interactive data streams from user input in real time and extracts features from these raw interactive data streams to obtain multimodal fusion features. Intent recognition is performed on the multimodal fusion features to obtain intent recognition results; Calculate the semantic relevance score between the intent recognition result and each entity and relation in the dynamic knowledge base; Based on the semantic relevance score, a set of candidate knowledge nodes is selected from the dynamic knowledge base, and the associated path information corresponding to the set of candidate knowledge nodes is determined. A multimodal combined response strategy is generated based on the user terminal performance evaluation results, the candidate knowledge node set, and the associated path information. Intelligent customer service interaction is realized based on the multimodal combined response strategy.
7. A storage medium, characterized in that, The storage medium stores an intelligent customer service interaction program, which, when executed by a processor, implements the steps of the intelligent customer service interaction method as described in claim 6.