An intelligent customer service dialogue generation method based on multi-modal interaction and dynamic knowledge enhancement

Abstract

The application discloses a kind of intelligent customer service dialogue generation methods based on multimodal interaction and dynamic knowledge enhancement, specific steps are as follows: S1: intention recognition: according to the content input by user, the intention of user is identified;S2, knowledge retrieval: based on user intention and historical interaction record, generates dynamic query vector, and carries out weighted retrieval in a double-source knowledge base combined with knowledge graph and document paragraph, wherein, the weight of knowledge is dynamically updated according to user feedback;S3: reply generation: input user intention and the knowledge segment related to intention into large language model, obtain several candidate replies;S4, compliance verification;S5, response output.The application significantly improves the interaction accuracy, knowledge timeliness and service personalization level of intelligent customer service through multimodal fusion understanding, dynamic knowledge enhancement and controllable content generation.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing (NLP), and in particular to an intelligent customer service dialogue generation method based on multimodal interaction and dynamic knowledge enhancement. Background Technology

[0002] With the popularization of e-commerce and online services, intelligent customer service systems have become a key hub for communication between enterprises and users. Their technological development has roughly gone through the following stages:

[0003] The first stage is customer service systems based on rule and keyword matching. These systems rely on manually pre-set rule bases and process templates, triggering fixed responses by matching keywords in user input. Their advantages are simplicity and precise, controllable answers, but their disadvantages are also significant: poor generalization ability, inability to understand the diverse expressions of user intent, and inability to respond if the user's question changes slightly or exceeds the rule base, resulting in a very rigid interactive experience.

[0004] The second stage involved customer service systems that incorporated traditional Natural Language Processing (NLP) and machine learning models. Through techniques such as intent recognition and entity extraction, the system's ability to understand user input was improved to some extent. However, these shallow models still heavily rely on large amounts of manually labeled domain data and struggle to capture deep semantic relationships and contextual information, remaining inadequate for handling complex or long-distance dialogues.

[0005] In recent years, the emergence of pre-trained models, represented by Large Language Models (LLMs), has brought breakthroughs to the field of intelligent customer service. Their powerful language understanding and generation capabilities have greatly improved the fluency and naturalness of conversations. However, customer service systems based on large models have also exposed new technical shortcomings in practical applications:

[0006] The static and outdated nature of knowledge: The core knowledge of large models originates from the massive datasets prior to their training deadline, forming a static knowledge snapshot. Large models cannot acquire dynamic knowledge such as frequently updated product information, promotional activities, and business policies in enterprise customer service scenarios in real time. This not only prevents them from answering the latest questions but also makes them prone to "hallucination" when facing knowledge blind spots, leading to the fabrication of inaccurate information and posing risks to the enterprise.

[0007] Limitations of Contextual Understanding and Personalized Services: Although large models possess a certain capacity for contextual understanding, crucial historical information will gradually be lost during continuous multi-turn dialogues without a dedicated dialogue state tracking mechanism. This makes it difficult for the system to truly understand users' long-term preferences and personalized needs, resulting in homogenized responses and an inability to provide truly personalized services.

[0008] The limited diversity of interaction modalities: Current mainstream intelligent customer service interactions are still primarily text-based. However, in real-world scenarios, users often inquire by sending images (such as screenshots of product malfunctions or order details), voice messages, or a combination of text and images. Existing technologies generally lack the ability to effectively integrate and understand such multimodal inputs, failing to extract key intents from non-textual information, which greatly limits the scope and efficiency of solving complex problems. Summary of the Invention

[0009] This invention primarily addresses the technical problems of existing technologies, such as poor generalization ability, insufficient domain adaptability, and weak personalization capabilities, by providing an automated, efficient, and accurate intelligent customer service dialogue generation method based on multimodal interaction and dynamic knowledge enhancement.

[0010] The present invention addresses the aforementioned technical problems primarily through the following technical solution: a method for generating intelligent customer service dialogues based on multimodal interaction and dynamic knowledge enhancement, comprising the following steps:

[0011] S1: Intent Recognition: Recognize the user's intent based on the user's input;

[0012] S2: Knowledge Retrieval: Based on the user's intent, retrieve knowledge fragments related to the intent from knowledge graph nodes and document paragraphs;

[0013] S3: Response Generation: Input the user's intent and related knowledge fragments into the large language model to obtain several candidate responses;

[0014] S4: Compliance Validation: Filter and select candidate responses based on the compliance rule base and user history, and adjust the output style to obtain compliant responses;

[0015] S5: Response Output: Output the compliant response to the user.

[0016] Preferably, step S1 specifically includes:

[0017] S11: Extract text embedding features h using a pre-trained language model. t The pre-trained language model can be BERT or other text feature extraction models;

[0018] S12: Extract speech features h using a speech feature extraction model. a The speech feature extraction model can be the Wav2Vec model.

[0019] S13: Extract image features h using an image feature extraction model. v The image feature extraction model can be either ResNet or CLIP.

[0020] S14: Employ a cross-modal attention mechanism to align multimodal features and generate a joint representation h. fuse :

[0021] h fuse =CrossAttn(h t ,h a ,h v CrossAttn(Q,K,V)=softmax(QK) T / d k 0.5 V;

[0022] S15: Generate the intent probability distribution using softmax:

[0023] p(y i |h fuse =softmax(W) i ×h fuse +b i );

[0024] In the formula, y i It is the i-th intention category, W i It is the weight matrix for the i-th intention category, b i It is the bias vector of the i-th intention category, p(y i |h fuse ) is the posterior probability of the i-th intent category in the final output; the intent categories have a definite list, such as "check the weather", "check orders", "chat", etc., and the model learns to input these predefined categories through training; the weight matrix and bias vector of the intent categories are also determined through training.

[0025] S16: Annotate slots in text using a BiLSTM-CRF model: s=CRF(BiLSTM(h t ),s is a structured sequence of entity slot labels.

[0026] Preferably, step S2 specifically includes:

[0027] S21: Joint representation h fuse The vector q is obtained by concatenating the entity slot label series s with the vector s and inputting it into the multilayer perceptron.

[0028] S22: Modify the original query vector q to obtain the modified query vector q. dynamic :

[0029] q dynamic =q+λ•Σ(w k_i •K i ),ki ∈Ф hist ;

[0030] In the formula, λ is the feedback intensity coefficient, which defaults to 0.3 and controls the influence of historical knowledge on the current query; k i For the i-th knowledge fragment involved in the user's historical interactions, Ф hist For knowledge fragment k i The set, K i For knowledge fragment k i The embedded representation, w k_i It is a knowledge fragment k i The weights;

[0031] S23: Calculate the query vector q dynamic The node similarity score (n) with each node in the knowledge graph j )=q dynamic T •GNN(n j );n j It is the j-th node in the knowledge graph, GNN(n j ) is n j Vector representation of;

[0032] S24: Incorporate dynamic weights into node similarity to obtain the final node similarity:

[0033] final_score(n j )=score(n j )•w n_j ;

[0034] In the formula, w n_j Let be the weight of the j-th node;

[0035] S25: Calculate the query vector q using the BM25 algorithm. dynamic With the m-th document paragraph d m Document relevance BM25(d) m ,q dynamic Then, d is obtained by adjusting the weights of document paragraphs. m The final document relevance is calculated using the following formula:

[0036] final_BM25(d m ,q dynamic )= BM25(d m ,q dynamic )•(1+log(1+w d_m ));

[0037] w d_m It is the m-th document paragraph d m Feedback weights;

[0038] S26: Sort nodes according to the final node similarity, and sort document paragraphs according to the final document relevance;

[0039] S27: From the two sorted lists, select several nodes whose final node similarity is higher than the node similarity threshold, and select several document paragraphs whose final document relevance is higher than the document relevance threshold. Extract the knowledge fragments contained in the selected nodes and document paragraphs; these are the final retrieved knowledge fragments relevant to the intent. Both the node similarity threshold and the document relevance threshold are empirical parameters. The number of nodes and document paragraphs selected is also determined based on the scenario, but is generally set to 50 each.

[0040] The database stores knowledge graph nodes and document paragraphs. Knowledge graph nodes are structured entities / relationships, such as <product, attribute, price> triples, atomic units of fact with fine granularity. Document paragraphs are excerpts from technical documents. Both knowledge graph nodes and document paragraphs contain knowledge fragments. The main purpose of step S2 is to retrieve the most relevant knowledge fragment from the database based on the user's input dialogue intent and the knowledge fragments involved in historical interactions to generate a response.

[0041] As a preferred option, knowledge fragment k i weight w k_i Determined in the following manner:

[0042] After each session, the weights of the knowledge fragments are recalculated. After the t-th session, the weight of the i-th knowledge fragment used in this session is... i weight w k_i (t) Calculated using the following formula:

[0043] w k_i (t) =α·w k_i (t-1) +(1-α)·(β·Explicit(k i )+(1-β)·Implicit(k i ));

[0044] In the formula, α∈[0,1] is the historical weight decay coefficient optimized through cross-validation, with a default value of 0.7; β∈[0,1] is the explicit / implicit feedback balance factor adjusted according to business needs, with a default value of 0.4; Explicit(k i Implicit(k) represents normalized rating feedback, such as user rating / 5; i The implicit feedback composite score is determined using the following formula:

[0045] Implicit(k i =η·Number of follow-up questions / max(number of follow-up questions) + (1-η)·Dwell time / Total conversation duration;

[0046] η∈[0,1] represents the weights of follow-up questions and dwell time, with a default value of 0.3. The number of follow-up questions is the number of subsequent questions under the same intent, and the dwell time is the user response delay time after a reply is generated. The knowledge fragments whose weights are recalculated only refer to those used in the current session; unused knowledge fragments are not updated in weight. Low-weight knowledge fragments are downgraded to reduce their probability of appearing in search results.

[0047] The training process for each parameter is as follows:

[0048] Weight decay coefficient α: Minimizes the weight update error Δw by training the LSTM prediction model with real-time dialogue data. t =γ·(dL / dw+α·sign(w t-1 ));

[0049] In the formula, Δw t Let be the weight update amount in the t-th iteration; γ be the learning rate; L be the loss function; α be the weight decay coefficient, used to control the penalty intensity of historical weights; and the sign function is applied to the weight w at the previous time step. t-1 Generate symbolic gradient correction terms;

[0050] The feedback balance factor β and the weights η for follow-up questions and dwell time are optimized using a reinforcement learning algorithm, and the reward function is the probability that the retrieval result is adopted.

[0051] Local update: After every 100 dialogues, fine-tune the MLP and GNN parameters using data within a sliding window.

[0052] Global Updates: Monthly full updates to the knowledge graph embedding and retrieval models.

[0053] As a preferred option, the j-th node n in the knowledge graph j weight w n_j Determined through the following steps:

[0054] A11: For each knowledge fragment k i Perform entity recognition and extract entity set E i ={e i1 ,e i2 …};

[0055] A12: Link entities to knowledge graph nodes. This step involves determining whether a node is mentioned by a certain knowledge fragment.

[0056] A13: If the j-th node n in the knowledge graph jLinked to knowledge fragment k i The t-th entity e it Then the knowledge fragment k i Partial weights are integrated into node n j The formula for the weights is:

[0057] w n_j '=Σ[w n_j (i) •tf-idf(e it ,k i )] / Σ[tf-idf(e it ,k i )];

[0058] w n_j (i) = w k_i •sim(e it ,n j );

[0059] Where tf-idf is the term frequency-inverse frequency function, and sim is the entity-node link confidence function, which is usually provided by entity linking tools;

[0060] A14: Obtain the node weights by diffusing weights through edge relationships in the knowledge graph:

[0061] w n_j =δ•w n_j '+(1-δ)•Σ[w n_p / out-degree(n p )];

[0062] In the formula, δ is the damping coefficient obtained through training, and n p For n j The p-th neighbor node, w n_p For n p The current weight, out-degree(n) p ) is n p The degree of departure.

[0063] As a preferred option, the m-th document paragraph d m weight w d_m Determined through the following steps:

[0064] A21: If knowledge fragment k i and the m-th document paragraph d m If there is a relationship of affiliation, then establish a mapping;

[0065] A22: By aggregating document paragraphs d m The weight of the knowledge fragments contained in the document paragraph d is obtained. m initial weight wd_m ':

[0066] w d_m '=Σ[w k_i •conf(k i )] / Σ[conf(k i )];

[0067] conf(k i ) is the confidence score for fragment extraction, representing the knowledge fragment k. i It represents the confidence level that the knowledge fragment is real and valid, and that the extraction process is error-free. The higher the value, the more reliable the knowledge fragment is in the knowledge fragment extraction model's opinion. It is obtained directly from the knowledge fragment extraction model.

[0068] A23: Update document feedback weights based on user behavior logs. d_m :

[0069] w d_m =ε•w d_m '+(1-ε)•(ζ•CRT m +(1-ζ)•human_score m );

[0070] CTR m It is document paragraph d m Click-through rate when used as an answer source;

[0071] human_score m It is the operations staff's handling of document paragraph d m The quality score is indicated (e.g., 1-5 points);

[0072] ε and ζ are trainable hyperparameters.

[0073] Preferably, step S3 specifically includes:

[0074] S31: Encode user intent and intent-related knowledge fragments into a structured prompt;

[0075] S32: Calculate the prefix vector P=MLP([h fuse ;mean-pool(Ф)]), where mean-pool is the average pooling operation, and Ф is a knowledge fragment related to the intent;

[0076] S33: Inject the prefix vector P into the large language model through Perfix-tuning;

[0077] S34: Input the structured prompt into the large language model that has been injected with the prefix vector P. The large language model uses probability p(y'|x, Ф)=Π t [p(wdt |wd<t,h fuse , Ф) generates candidate responses;

[0078] p(y’|x, Ф) represents the probability of generating candidate response y’, where y’ is the text of the candidate response, x is the user's complete original input, and Ф is the knowledge segment related to the intent. wd t is the current word to be generated at time step t, and wd<t represents all the previously generated words, i.e., wd1, wd2, … wd t-1 , p(wd t |wd<t,h fuse , Ф) represents the single-step generation probability.

[0079] The compliance check in step S4 is a regular filtering, screening, and adjustment process, including filtering / replacing sensitive words and adjusting the reply to meet the requirements in terms of formality, friendliness, and conciseness. Sensitive word filtering is a mandatory item, and style adjustment is an optional item.

[0080] For the response output in step S5, if the user needs a voice reply, the VITS model is used to generate voice output; if the compliant reply involves data or charts, D3.js is called to render and generate dynamic charts; then the output layout is optimized according to the output device, and finally the output is made.

[0081] The substantial effects brought by this invention are:

[0082] 1. It significantly improves the accuracy and comprehensiveness of user intent understanding and realizes true multi-modal interaction. Most of the existing technologies are limited to processing pure text inputs and cannot handle complex scenarios where users send mixed information such as pictures and voices. By introducing a cross-modal attention mechanism, this invention deeply fuses the features of text, voice, and image modalities to generate a unified joint representation vector. This enables the system to comprehensively analyze all input information. For example, it can understand complex intents containing cross-modal references such as "This (pointing to a certain component in the picture) is broken. How to repair it?" sent by the user. Compared with the existing technologies, this invention breaks through the limitation of a single modality and elevates the accuracy of intent recognition and the breadth of application scenarios to a new level.

[0083] 2. This invention achieves dynamic adaptive enhancement of the knowledge base, effectively solving the problems of knowledge lag and "illusion" in large models. Traditional intelligent customer service, especially systems based on static large models, cannot acquire real-time updated domain knowledge. This invention innovatively designs a dynamic knowledge weight feedback mechanism. By capturing users' explicit (ratings) and implicit (follow-up questions, dwell time) feedback in real time, it can automatically adjust the weights of knowledge graph nodes and document paragraphs. The weights of high-quality, user-popular knowledge will dynamically increase, while the weights of outdated or inaccurate knowledge will decrease. This "living" knowledge base can self-evolve and iteratively optimize, ensuring that the knowledge sources provided to the large model for generating responses are always the latest and most accurate, thereby fundamentally suppressing the "illusion" phenomenon of the model and guaranteeing the authenticity and timeliness of customer service responses.

[0084] 3. Significantly enhances personalized service capabilities and the coherence of multi-turn dialogues. Existing technologies often neglect context and user history when processing long dialogues. This invention cleverly incorporates the user's historical interaction records when constructing the query vector for knowledge retrieval, generating a dynamically corrected query vector q. dynamic This means that each knowledge retrieval by the system takes into account the user's historical preferences and the current dialogue context. Simultaneously, the dynamic weighting mechanism itself is a form of personalization, with frequently accessed knowledge points naturally receiving higher weights. Therefore, this invention can provide personalized knowledge recommendations and response generation for different users, moving from a "one-size-fits-all" approach to a "personalized experience," significantly improving user satisfaction and the consistency of the dialogue experience.

[0085] 4. Improved controllability and domain adaptability of response generation, achieving deep integration of large-scale model capabilities and domain knowledge. Directly using general-purpose large-scale models to answer professional questions is not only costly but also results in uncontrollable content. This invention achieves accurate and low-cost "management" of large-scale models through a "retrieval-generation" enhancement (RAG) paradigm combined with efficient prefix-tuning technology. This invention first retrieves the most relevant and verified knowledge fragments from a dynamic knowledge base, then encodes them as prefix vectors and injects them into the large-scale model to guide the generation of responses. This approach leverages the superior language organization and generation capabilities of large-scale models while strictly constraining the generated content within the accurate domain knowledge scope, greatly improving the reliability of responses and enabling them to quickly adapt to any specific customer service domain. Attached Figure Description

[0086] Figure 1 This is a flowchart of an intelligent customer service dialogue generation method based on multimodal interaction and dynamic knowledge enhancement according to the present invention. Detailed Implementation

[0087] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0088] Example: A method for generating intelligent customer service dialogues based on multimodal interaction and dynamic knowledge enhancement, such as... Figure 1 As shown, it includes the following steps:

[0089] S1: Intent Recognition: Recognize the user's intent based on the user's input;

[0090] S2: Knowledge Retrieval: Based on the user's intent, retrieve knowledge fragments related to the intent from knowledge graph nodes and document paragraphs;

[0091] S3: Response Generation: Input the user's intent and related knowledge fragments into the large language model to obtain several candidate responses;

[0092] S4: Compliance Validation: Filter and select candidate responses based on the compliance rule base and user history, and adjust the output style to obtain compliant responses;

[0093] S5: Response Output: Output the compliant response to the user.

[0094] Step S1 specifically involves:

[0095] S11: Extract text embedding features h using a pre-trained language model. t The pre-trained language model can be BERT or other text feature extraction models;

[0096] S12: Extract speech features h using a speech feature extraction model. a The speech feature extraction model can be the Wav2Vec model.

[0097] S13: Extract image features h using an image feature extraction model. v The image feature extraction model can be either ResNet or CLIP.

[0098] S14: Employ a cross-modal attention mechanism to align multimodal features and generate a joint representation h. fuse :

[0099] h fuse =CrossAttn(h t ,h a ,h v CrossAttn(Q,K,V)=softmax(QK) T / d k 0.5 V;

[0100] S15: Generate the intent probability distribution using softmax:

[0101] p(y i |h fuse =softmax(W) i ×h fuse +b i );

[0102] In the formula, y i It is the i-th intention category, W i It is the weight matrix for the i-th intention category, b i It is the bias vector of the i-th intention category, p(y i |h fuse ) is the posterior probability of the i-th intent category in the final output; the intent categories have a definite list, such as "check the weather", "check orders", "chat", etc., and the model learns to input these predefined categories through training; the weight matrix and bias vector of the intent categories are also determined through training.

[0103] S16: Annotate slots in text using a BiLSTM-CRF model: s=CRF(BiLSTM(h t ),s is a structured sequence of entity slot labels.

[0104] Step S2 specifically involves:

[0105] S21: Joint representation h fuse The vector q is obtained by concatenating the entity slot label series s with the vector s and inputting it into the multilayer perceptron.

[0106] S22: Modify the original query vector q to obtain the modified query vector q. dynamic :

[0107] q dynamic =q+λ•Σ(w k_i •K i ),k i ∈Ф hist ;

[0108] In the formula, λ is the feedback intensity coefficient, which defaults to 0.3 and controls the influence of historical knowledge on the current query; k i For the i-th knowledge fragment involved in the user's historical interactions, Ф hist For knowledge fragment k i The set, K i For knowledge fragment k i The embedded representation, w k_i It is a knowledge fragment k i The weights;

[0109] S23: Calculate the query vector q dynamic The node similarity score (n) with each node in the knowledge graph j )=q dynamic T •GNN(n j );n j It is the j-th node in the knowledge graph, GNN(n j ) is n j Vector representation of;

[0110] S24: Incorporate dynamic weights into node similarity to obtain the final node similarity:

[0111] final_score(n j )=score(n j )•w n_j ;

[0112] In the formula, w n_j Let be the weight of the j-th node;

[0113] S25: Calculate the query vector q using the BM25 algorithm. dynamic With the m-th document paragraph d m Document relevance BM25(d) m ,q dynamic Then, d is obtained by adjusting the weights of document paragraphs. m The final document relevance is calculated using the following formula:

[0114] final_BM25(d m ,q dynamic )= BM25(d m ,q dynamic )•(1+log(1+w d_m ));

[0115] w d_m It is the m-th document paragraph d m Feedback weights;

[0116] S26: Sort nodes according to the final node similarity, and sort document paragraphs according to the final document relevance;

[0117] S27: From the two sorted lists, select several nodes whose final node similarity is higher than the node similarity threshold, and select several document paragraphs whose final document relevance is higher than the document relevance threshold. Extract the knowledge fragments contained in the selected nodes and document paragraphs; these are the final retrieved knowledge fragments relevant to the intent. Both the node similarity threshold and the document relevance threshold are empirical parameters. The number of nodes and document paragraphs selected is also determined based on the scenario, but is generally set to 50 each.

[0118] The database stores knowledge graph nodes and document paragraphs. Knowledge graph nodes are structured entities / relationships, such as <product, attribute, price> triples, atomic units of fact with fine granularity. Document paragraphs are excerpts from technical documents. Both knowledge graph nodes and document paragraphs contain knowledge fragments. The main purpose of step S2 is to retrieve the most relevant knowledge fragment from the database based on the user's input dialogue intent and the knowledge fragments involved in historical interactions to generate a response.

[0119] Knowledge Fragment k i weight w k_i Determined in the following manner:

[0120] After each session, the weights of the knowledge fragments are recalculated. After the t-th session, the weight of the i-th knowledge fragment used in this session is... i weight w k_i (t) Calculated using the following formula:

[0121] w k_i (t) =α·w k_i (t-1) +(1-α)·(β·Explicit(k i )+(1-β)·Implicit(k i ));

[0122] In the formula, α∈[0,1] is the historical weight decay coefficient optimized through cross-validation, with a default value of 0.7; β∈[0,1] is the explicit / implicit feedback balance factor adjusted according to business needs, with a default value of 0.4; Explicit(k i Implicit(k) represents normalized rating feedback, such as user rating / 5; i The implicit feedback composite score is determined using the following formula:

[0123] Implicit(k i =η·Number of follow-up questions / max(number of follow-up questions) + (1-η)·Dwell time / Total conversation duration;

[0124] η∈[0,1] represents the weights for follow-up questions and dwell time, with a default value of 0.3. The number of follow-up questions is the number of subsequent questions under the same intent, and the dwell time is the user response delay after a reply is generated. Knowledge fragments refer to knowledge fragments contained in knowledge graph nodes and document paragraphs, i.e., knowledge fragments already in the database. Unused knowledge fragments do not have their weights updated.

[0125] The training process for each parameter is as follows:

[0126] Weight decay coefficient α: Minimizes the weight update error Δw by training the LSTM prediction model with real-time dialogue data. t =γ·(dL / dw+α·sign(w t-1 ));

[0127] In the formula, Δw t Let be the weight update amount in the t-th iteration; γ be the learning rate; L be the loss function; α be the weight decay coefficient, used to control the penalty intensity of historical weights; and the sign function is applied to the weight w at the previous time step. t-1 Generate symbolic gradient correction terms;

[0128] The feedback balance factor β and the weights η for follow-up questions and dwell time are optimized using a reinforcement learning algorithm, and the reward function is the probability that the retrieval result is adopted.

[0129] The j-th node n in the knowledge graph j weight w n_j Determined through the following steps:

[0130] A11: For each knowledge fragment k i Perform entity recognition and extract entity set E i ={e i1 ,e i2 …};

[0131] A12: Link entities to knowledge graph nodes. This step involves determining whether a node is mentioned by a certain knowledge fragment.

[0132] A13: If the j-th node n in the knowledge graph j Linked to knowledge fragment k i The t-th entity e it Then the knowledge fragment k i Partial weights are integrated into node n j The formula for the weights is:

[0133] w n_j '=Σ[w n_j (i) •tf-idf(e it ,k i)] / Σ[tf-idf(e it ,k i )];

[0134] w n_j (i) = w k_i •sim(e it ,n j );

[0135] Where tf-idf is the term frequency-inverse frequency function, and sim is the entity-node link confidence function, which is usually provided by entity linking tools;

[0136] A14: Obtain the node weights by diffusing weights through edge relationships in the knowledge graph:

[0137] w n_j =δ•w n_j '+(1-δ)•Σ[w n_p / out-degree(n p )];

[0138] In the formula, δ is the damping coefficient obtained through training, and n p For n j The p-th neighbor node, w n_p For n p The current weight, out-degree(n) p ) is n p The degree of departure.

[0139] The m-th document paragraph d m weight w d_m Determined through the following steps:

[0140] A21: If knowledge fragment k i and the m-th document paragraph d m If there is a relationship of affiliation, then establish a mapping;

[0141] A22: By aggregating document paragraphs d m The weight of the knowledge fragments contained in the document paragraph d is obtained. m initial weight w d_m ':

[0142] w d_m '=Σ[w k_i •conf(k i )] / Σ[conf(k i )];

[0143] conf(k i ) is the confidence score for fragment extraction, representing the knowledge fragment k. iThis represents the confidence level of the knowledge fragment being genuine and valid, and the extraction process being error-free. A higher value indicates that the knowledge fragment extraction model considers the knowledge fragment more reliable, and this value is directly derived from the knowledge fragment extraction model. The knowledge extraction model can employ a sequence labeling model based on the BERT-BiLSTM-CRF architecture. In this architecture, the model labels each character or word in the input document to identify the boundaries of knowledge fragments, while the CRF layer calculates the probability of the entire optimal labeled sequence during decoding. This sequence probability, after normalization, can then be used as the knowledge fragment k. i Confidence level conf(k) i) ;

[0144] A23: Update document feedback weights based on user behavior logs. d_m :

[0145] w d_m =ε•w d_m '+(1-ε)•(ζ•CRT m +(1-ζ)•human_score m );

[0146] CTR m It is document paragraph d m Click-through rate when used as an answer source;

[0147] human_score m It is the operations staff's handling of document paragraph d m The quality score is indicated (e.g., 1-5 points);

[0148] ε and ζ are trainable hyperparameters.

[0149] Step S3 specifically involves:

[0150] S31: Encode user intent and intent-related knowledge fragments into a structured prompt;

[0151] S32: Calculate the prefix vector P=MLP([h fuse ;mean-pool(Ф)]), where mean-pool is the average pooling operation, and Ф is a knowledge fragment related to the intent;

[0152] S33: Inject the prefix vector P into the large language model through Perfix-tuning;

[0153] S34: Input the structured prompt into the large language model that has been injected with the prefix vector P. The large language model uses probability p(y'|x, Ф)=Π t [p(wd t |wd <t,h fuse Generate candidate responses;

[0154] p(y’|x, Ф) represents the probability of generating a candidate response y’, where y’ is the text of the candidate response, x is the user's complete original input, and Ф is the knowledge fragment related to the intent, wd t is the current word to be generated at time step t, and wd<t represents all the previously generated words, i.e., wd1, wd2, … wd t-1 , p(wd t |wd<t, h fuse , Ф) represents the single-step generation probability.

[0155] The compliance check in step S4 is a conventional filtering, screening, and adjustment process, including filtering / replacing sensitive words and adjusting the reply to meet the requirements in terms of formality, friendliness, and conciseness. Sensitive word filtering is a required item, and style adjustment is an optional item.

[0156] For the response output in step S5, if the user needs a voice reply, the voice output is generated through the VITS model; if the compliant reply involves data or charts, D3.js is called to render and generate dynamic charts; then the output layout is optimized according to the output device, and finally the output is made.

[0157] The model architecture used in this embodiment is as follows:

[0158] (1) Base model: A pre-trained model with the LLaMA-3 or GPT-4 architecture.

[0159] (2) Extended modules: ① Knowledge retriever: Dense vector retrieval based on Faiss. ② Dialogue state tracker: Context encoder of LSTM_Attention.

[0160] The training method adopts two-stage training: pre-training with general corpus + fine-tuning with domain data (customer service logs). The loss function combines the generation loss and the knowledge retrieval correlation score.

[0161] The optimization method during deployment is as follows: (1) Use model quantization (Quantization) and distillation techniques to reduce the inference latency. (2) Support API interfaces for integration with enterprise CRM systems.

[0162] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Those skilled in the art of the present invention can make various modifications or supplements to the described specific embodiments or use similar methods for substitution, but will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0163] Although this paper makes extensive use of terms such as knowledge retrieval, response generation, and weighting, the possibility of using other terms is not excluded. The use of these terms is merely for the convenience of describing and explaining the essence of this invention; interpreting them as any additional limitation would contradict the spirit of this invention.

Claims

1. A method for generating intelligent customer service dialogues based on multimodal interaction and dynamic knowledge enhancement, characterized in that, Includes the following steps: S1: Intent Recognition: Recognize the user's intent based on the user's input; S2: Knowledge Retrieval: Retrieve knowledge fragments related to the user's intent; S3: Response Generation: Input the user's intent and related knowledge fragments into the large language model to obtain several candidate responses; S4: Compliance Validation: Filter and select candidate responses based on the compliance rule base and user history, and adjust the output style to obtain compliant responses; S5: Response Output: Output a compliant response to the user; Step S1 specifically involves: S11: Extract text embedding features h using a pre-trained language model. t ; S12: Extract speech features h using a speech feature extraction model. a ; S13: Extract image features h using an image feature extraction model. v ; S14: Employ a cross-modal attention mechanism to align multimodal features and generate a joint representation h. fuse : h fuse =CrossAttn(h t ,h a ,h v ),CrossAttn(Q,K,V)=softmax(QK T / d k 0.5 )V; S15: Generate the intent probability distribution using softmax: p(y i |h fuse )=softmax(W i × h fuse +b i ): In the formula, y i It is the i-th intention category, W i It is the weight matrix for the i-th intention category, b i It is the bias vector of the i-th intention category, p(y i |h fuse ) is the posterior probability of the i-th intent category in the final output; S16: Annotate slots in text using a BiLSTM-CRF model: s=CRF(BiLSTM(h t ),s is a structured sequence of entity slot labels; Step S2 specifically involves: S21: Joint representation h fuse The vector q is obtained by concatenating the entity slot label series s with the vector s and inputting it into the multilayer perceptron. S22: Modify the original query vector q to obtain the modified query vector q. dynamic : q dynamic =q+λ•Σ(w k_i •K i ),k i ∈Ф hist ; In the formula, λ is the feedback intensity coefficient, which is 0.3 by default, and k i For the i-th knowledge fragment involved in the user's historical interactions, Ф hist For knowledge fragment k i The set, K i For knowledge fragment k i The embedded representation, w k_i It is a knowledge fragment k i The weights; S23: Calculate the query vector q dynamic The node similarity score (n) with each node in the knowledge graph j )=q dynamic T •GNN(n j );n j It is the j-th node in the knowledge graph, GNN(n j ) is n j Vector representation of; S24: Incorporate dynamic weights into node similarity to obtain the final node similarity: final_score(n j )=score(n j )•w n_j ; In the formula, w n_j Let be the weight of the j-th node; S25: Calculate the query vector q using the BM25 algorithm. dynamic With the m-th document paragraph d m Document relevance BM25(d) m ,q dynamic Then, d is obtained by adjusting the feedback weights of document paragraphs. m The final document relevance is calculated using the following formula: final_BM25(d m ,q dynamic )= BM25(d m ,q dynamic )•(1+log(1+w d_m )); w d_m It is the m-th document paragraph d m Feedback weights; S26: Sort nodes according to the final node similarity, and sort document paragraphs according to the final document relevance; S27: In the two sorted lists, select several nodes whose final node similarity is higher than the node similarity threshold, and select several document paragraphs whose final document relevance is higher than the document relevance threshold. Extract the knowledge fragments contained in the selected nodes and the knowledge fragments contained in the selected document paragraphs. These are the final retrieved knowledge fragments related to the intent.

2. The intelligent customer service dialogue generation method based on multimodal interaction and dynamic knowledge enhancement according to claim 1, characterized in that, Knowledge Fragment k i weight w k_i Determined in the following manner: After each session, the weights of the knowledge fragments are recalculated. After the t-th session, the weight of the i-th knowledge fragment used in this session is... i weight w k_i (t) Calculated using the following formula: w k_i (t) =α·w k_i (t-1) +(1-α)·(β·Explicit(k i )+(1-β)·Implicit(k i )); In the formula, α∈[0,1] is the historical weight decay coefficient optimized through cross-validation; β∈[0,1] is the explicit / implicit feedback balance factor adjusted according to business needs; Explicit(k i Implicit(k) represents normalized rating feedback; i The implicit feedback composite score is determined using the following formula: Implicit(k i =η·Number of follow-up questions / max(number of follow-up questions) + (1-η)·Dwell time / Total conversation duration; η∈[0,1] represents the weights of follow-up questions and dwell time. The number of follow-up questions is the number of subsequent questions under the same intent, and the dwell time is the user response delay time after the reply is generated.

3. The intelligent customer service dialogue generation method based on multimodal interaction and dynamic knowledge enhancement according to claim 2, characterized in that, The j-th node n in the knowledge graph j weight w n_j Determined through the following steps: A11: For each knowledge fragment k i Perform entity recognition and extract entity set E i ={e i1 ,e i2 …}; A12: Link entities to knowledge graph nodes; A13: If the j-th node n in the knowledge graph j Linked to knowledge fragment k i The t-th entity e it Then the knowledge fragment k i Partial weights are integrated into node n j The formula for the weights is: w n_j ’=Σ[w n_j (i) ·tf-idf(e it ,k i )] / Σ[tf-idf(e it ,k i )]; w n_j (i) = w k_i ·sim(e it ,n j ); Where tf-idf is the term frequency-inverse frequency function, and sim is the entity-node link confidence function; A14: Obtain the node weights by diffusing weights through edge relationships in the knowledge graph: w n_j =δ•w n_j ’+(1-δ)•Σ[w n_p / out-degree(n p )]; In the formula, δ is the damping coefficient obtained through training, and n p For n j The p-th neighbor node, w n_p For n p The current weight, out-degree(n) p ) is n p The degree of departure.

4. The intelligent customer service dialogue generation method based on multimodal interaction and dynamic knowledge enhancement according to claim 2, characterized in that, The m-th document paragraph d m weight w d_m Determined through the following steps: A21: If knowledge fragment k i and the m-th document paragraph d m If there is a relationship of affiliation, then establish a mapping; A22: By aggregating document paragraphs d m The weight of the knowledge fragments contained in the document paragraph d is obtained. m initial weight w d_m ': w d_m '=Σ[w k_i •conf(k i )] / Σ[conf(k i )]; conf(k i () represents the confidence level of fragment extraction; A23: Update document feedback weights based on user behavior logs. d_m : w d_m =e•w d_m '+(1-e)•(ζ•CRT m +(1-ζ)•human_score m ); CTR m It is document paragraph d m Click-through rate when used as an answer source; human_score m It is the operations staff's handling of document paragraph d m The quality score is marked; ε and ζ are trainable hyperparameters.

5. The intelligent customer service dialogue generation method based on multimodal interaction and dynamic knowledge enhancement according to claim 2, wherein step S3 specifically comprises: S31: Encode user intent and intent-related knowledge fragments into a structured prompt; S32: Calculate the prefix vector P=MLP([h fuse ;mean-pool(Ф)]), where mean-pool is the average pooling operation, and Ф is a knowledge fragment related to the intent; S33: Inject the prefix vector P into the large language model through Perfix-tuning; S34: Input the structured prompt into the large language model that has been injected with the prefix vector P. The large language model uses probability p(y'|x, Ф)=Π t [p(wd t |wd <t,h fuse Generate candidate responses; p(y’|x, Ф) represents the probability of generating a candidate response y’, where y’ is the text of the candidate response, x is the user's complete original input, and Ф is the knowledge fragment related to the intent, wd t is the current word to be generated at time step t, and wd<t represents all the previously generated words, i.e., wd1, wd2, … wd t-1 , p(wd t |wd<t, h fuse , Ф) represents the single-step generation probability.

Images