Digital intelligent customer service system based on multiple modes

Abstract

The invention discloses a multi-modal-based digital intelligent customer service system. Data compensation is carried out on initial multi-modal problem data by adopting a dynamic adaptive compensation mechanism through a multi-modal input module. And carrying out dynamic weight distribution on the target multi-modal problem data by adopting a cross-modal attention mechanism through a multi-modal comparison learning module. And performing intention recognition on the target multi-modal semantic vector through an intention recognition module according to the historical interaction data and a preset dynamic knowledge graph. And inputting the user intention data into the target reinforcement learning reply model through a reply generation module to construct reply data. And performing potential problem prediction by adopting the target multi-modal semantic vector, the historical interaction sequence corresponding to the user and the time data through a potential problem prediction module. And by utilizing the dynamic updating capability of the knowledge graph, the context sensitive identification of the user problem is realized, and the identification accuracy is improved. And the potential problem prediction module turns from passive response to active service, so that the interaction efficiency and the user satisfaction are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of digital and intelligent customer service technology, and in particular to a multimodal digital and intelligent customer service system. Background Art

[0002] Digital customer service systems aim to automate responses to user questions, improve service efficiency, and reduce labor costs. Existing digital customer service systems primarily rely on fixed knowledge bases or traditional machine learning models to semantically parse user questions and generate responses. However, faced with increasingly complex user interaction scenarios, these traditional systems are experiencing significant technical bottlenecks.

[0003] Traditional systems are often trained based on static knowledge bases or single-modal data, making it difficult to effectively process the multimodal input generated by users in actual interactions, resulting in deviations in the semantic parsing of complex questions. Furthermore, existing models lack the ability to deeply mine historical user interaction data and are unable to dynamically optimize response strategies based on real-time conversations. When user questions involve multi-round conversational logic, domain-specific terminology, or personalized context, the system often generates responses that deviate from the user's actual needs or even irrelevantly address the question due to insufficient training data coverage. This results in low response accuracy during customer service and fails to meet the precision, real-time, and personalization requirements of intelligent services. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal digital customer service system to solve the technical problems that the existing digital customer service system has limited dynamic learning ability for user problems, has difficulty in accurately understanding user problems, and leads to poor user experience.

[0005] The technical solution of the present invention is achieved as follows:

[0006] The present invention provides a multimodal digital customer service system, comprising:

[0007] A multimodal input module is used to obtain initial multimodal question data input by the user and use a dynamic adaptive compensation mechanism to perform data compensation on the initial multimodal question data to generate target multimodal question data;

[0008] A multimodal contrastive learning module is used to dynamically assign weights to the target multimodal question data using a cross-modal attention mechanism to generate a target multimodal semantic vector;

[0009] An intent recognition module is used to perform intent recognition on the target multimodal semantic vector based on historical interaction data and a preset dynamic knowledge graph to generate user intent data;

[0010] a response generation module, configured to input the user intention data into a target reinforcement learning response model to construct response data and generate target response data;

[0011] The potential problem prediction module is used to use the target multimodal semantic vector, the historical interaction sequence and time data corresponding to the user to predict potential problems, and generate a potential problem probability distribution list and multimodal response data.

[0012] Optionally, the multimodal input module includes:

[0013] A data acquisition module is used to acquire initial multimodal question data input by a user, and perform data preprocessing on the initial multimodal question data to generate intermediate multimodal question data;

[0014] A data detection module, configured to identify the modality types missing from the intermediate multimodal problem data and generate a modality missing type;

[0015] A compensation data generation module is configured to input the known modality specific data corresponding to the intermediate multimodal problem data and the modality missing type into an adversarial network trained by historical interaction data, and dynamically generate modality compensation data matching the modality missing type;

[0016] A data fusion module is used to fuse the modal compensation data with the intermediate multimodal problem data to generate target multimodal problem data.

[0017] Optionally, the multimodal contrastive learning module includes:

[0018] An inter-modal attention weight matrix construction module is used to construct a matrix based on the user historical interaction preference factor corresponding to the user and the target multimodal question data to generate an inter-modal attention weight matrix;

[0019] An initial multimodal semantic vector generation module is used to fuse the modal feature matrix corresponding to the target multimodal question data using a multi-head attention mechanism according to the inter-modal attention weight matrix to generate an initial multimodal semantic vector;

[0020] The target multimodal semantic vector generation module is used to optimize the cross-sample semantic discrimination of the initial multimodal semantic vector to generate a target multimodal semantic vector.

[0021] Optionally, the inter-modal attention weight matrix construction module executes the following steps:

[0022] Calculate the recognition accuracy of each modality in the historical interaction data set corresponding to the user, and generate a modality accuracy vector;

[0023] Substituting the modal accuracy vector into the flexibility maximum function with a temperature parameter, and calculating the modal preference factor vector;

[0024] Performing a projection transformation on each modal feature vector corresponding to the target multimodal question data to generate a query vector and a key vector;

[0025] The query vector, the key vector and the modality preference factor vector are used to construct a matrix to generate an inter-modality attention weight matrix.

[0026] Optionally, the initial multimodal semantic vector generation module performs the following steps:

[0027] Calculating the characteristic entropy value of the modal characteristic matrix corresponding to the target multimodal problem data using a preset hybrid norm to obtain a modal entropy value vector;

[0028] The modal entropy value vector is used to construct a diagonal matrix to generate an entropy value diagonal matrix;

[0029] Adjusting the multi-head attention weights of the inter-modal attention weight matrix using the entropy diagonal matrix and the multi-head attention parameters corresponding to the target multimodal problem data to generate a multi-head attention weight matrix set;

[0030] An initial multimodal semantic vector is constructed using the multi-head attention weight matrix set and the value matrix corresponding to the multi-head attention weight matrix set.

[0031] Optionally, the target multimodal semantic vector generation module executes the following steps:

[0032] Calculating a diversity index using the initial multimodal semantic vector and a current batch of samples corresponding to the initial multimodal semantic vector to obtain a data diversity index;

[0033] Dynamically adjust and compare learning temperature using the data diversity index to generate dynamic temperature parameters;

[0034] Performing cross-sample comparative learning optimization on the dynamic temperature parameter, the initial multimodal semantic vector, and the sample data corresponding to the initial multimodal semantic vector to generate a multimodal feature;

[0035] The multimodal features are fused with global semantic information through bilinear transformation to calculate the target multimodal semantic vector.

[0036] Optionally, the intention recognition module includes:

[0037] A target dynamic knowledge graph generation module is used to update the node weights and edge connection strengths of a preset dynamic knowledge graph online using the entity relationship change data corresponding to the target multimodal question data to generate a target dynamic knowledge graph;

[0038] An intent probability distribution generation module is used to use a graph attention network to perform node embedding alignment on the target multimodal semantic vector and the target dynamic knowledge graph to generate an intent probability distribution;

[0039] The intent generation module is used to perform attention-based weighted fusion of the intent probability distribution and the user behavior patterns corresponding to the historical interaction data to construct user intent data.

[0040] Optionally, the intention recognition module further includes:

[0041] A multi-emotion feature extraction module is used to extract emotional features from the target multimodal question data to generate physiological signal features and text emotional polarity features, wherein the physiological signal features include voice fundamental frequency jitter rate and facial expression action unit intensity;

[0042] An emotion weight coefficient generation module is used to input the physiological signal features, the text emotion features and the user behavior pattern in the historical interaction data into a gated recurrent network to generate an emotion weight coefficient;

[0043] The intention enhancement module is used to dynamically adjust the user intention data by using the emotion weight coefficient to generate emotion-enhanced user intention data.

[0044] Optionally, the reply generation module executes the following steps:

[0045] The initial reinforcement learning response model is trained using a preset hierarchical reward mechanism and a proximal policy optimization algorithm to generate an intermediate reinforcement learning response model.

[0046] Adopting an adversarial training mechanism to enhance the response robustness of the intermediate reinforcement learning response model and generate a target reinforcement learning response model;

[0047] The user intention data is input into the target reinforcement learning response model to construct response data and generate target response data.

[0048] Optionally, the execution steps of the potential problem prediction module are:

[0049] A multi-head attention mechanism is used to perform temporal modeling on the historical interaction sequence corresponding to the user to generate an enhanced semantic vector with contextual dependencies;

[0050] Perform multi-scale feature extraction on the time data corresponding to the user to generate a time-sensitive feature vector;

[0051] Dynamically weighting the target multimodal semantic vector, the enhanced semantic vector, and the time-sensitive feature vector through a learnable gated fusion network to construct a fusion prediction feature matrix;

[0052] Using a soft maximization classifier with a temperature adjustment parameter to perform a linear transformation on the fusion prediction feature matrix to generate a probability distribution list of potential problems;

[0053] Selecting a preset number of candidate questions with the highest confidence in the potential question probability distribution list to generate multiple target candidate questions;

[0054] The target candidate question is input into the target reinforcement learning response model to construct response data and generate multimodal response data.

[0055] The beneficial effects of the present invention are:

[0056] The present invention adopts a dynamic adaptive compensation mechanism through the multimodal input module to pre-process the multimodal data such as text, voice, and image input by the user, which can effectively fill the information gaps in cross-modal data and generate more complete and accurate target multimodal question data. The multimodal comparative learning module adopts a cross-modal attention mechanism to break the limitations of traditional single-modal independent processing, and realizes the deep fusion of different modal information by dynamically allocating weights of modalities such as text, voice, and image. It accurately captures the implicit intentions in user questions, solves the shallow semantic analysis defects of "literal understanding" of traditional systems, and significantly improves the accuracy of intent recognition. The intent recognition module combines historical interaction data with a preset dynamic knowledge graph. On the one hand, it builds a personalized understanding model through user historical behavior, and on the other hand, it uses the dynamic update capability of the knowledge graph to achieve context-sensitive recognition of user questions. The reply generation module uses a target reinforcement learning model and user feedback as a reward signal to dynamically optimize the reply strategy, generate replies that are more in line with user needs, and avoid the mechanical feel of traditional templated replies. The potential question prediction module combines multimodal semantic vectors, historical interaction sequences, and time data to predict potential user questions in advance, generating a probabilistic distribution list of potential questions and multimodal response data. This reduces repetitive user input and waiting time, and enables "predictive service" by proactively pushing relevant information, shifting from passive response to proactive service, significantly improving interaction efficiency and user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a schematic diagram of the structure of a multimodal digital customer service system of the present invention;

[0058] Figure 2 This is a schematic diagram of the structure of a multimodal input module in a multimodal digital customer service system according to the present invention;

[0059] Figure 3 Schematic diagram of the structure of a multimodal comparative learning module in a multimodal digital customer service system of the present invention;

[0060] Figure 4 This is a structural diagram of an intention recognition module in a multimodal digital customer service system of the present invention. DETAILED DESCRIPTION

[0061] In order to better understand the technical content of the present invention, specific embodiments are provided below, and the present invention is further described in conjunction with the accompanying drawings.

[0062] See also Figures 1 to 4 , a multimodal digital customer service system, including:

[0063] The multimodal input module is used to obtain the initial multimodal question data input by the user and use a dynamic adaptive compensation mechanism to compensate the initial multimodal question data to generate target multimodal question data;

[0064] A multimodal contrastive learning module, which uses a cross-modal attention mechanism to dynamically assign weights to target multimodal question data and generate a target multimodal semantic vector.

[0065] The intent recognition module is used to identify the intent of the target multimodal semantic vector based on historical interaction data and the preset dynamic knowledge graph to generate user intent data;

[0066] A response generation module is used to input user intention data into the target reinforcement learning response model to construct response data and generate target response data;

[0067] The potential problem prediction module is used to predict potential problems using the target multimodal semantic vector, the user's corresponding historical interaction sequence and time data, and generate a potential problem probability distribution list and multimodal response data.

[0068] In the embodiment of the present invention, Figure 1As shown, the multimodal input module receives initial multimodal question data (text, voice, image, etc.) input by the user and performs preprocessing such as cleaning and normalization to generate intermediate multimodal question data. The missing modal types in the intermediate multimodal question data are then identified. Specifically, a generative adversarial network (GAN) trained using historical interaction data (a generative adversarial network is a generative model based on deep learning. Its core concept is derived from the "adversarial game" in game theory. It achieves high-quality data generation by having two neural networks (generator and discriminator) compete with each other and optimize collaboratively) is used to dynamically generate compensatory data for the missing modalities based on known modal features (such as generating image features that match the text description). This compensatory data is then fused with the original data to form the complete target multimodal question data.

[0069] The multimodal contrastive learning module constructs a cross-modal attention weight matrix based on user historical interaction preference factors (e.g., higher weight is assigned to the voice modality if a user's historical voice input accuracy is high) and target multimodal question data. By dynamically adjusting the multi-head attention weights based on modal feature entropy (reflecting data uncertainty), a multi-head attention mechanism is integrated to generate an initial multimodal semantic vector. The target multimodal semantic vector is generated by optimizing the cross-sample semantic discriminability of the initial multimodal semantic vector.

[0070] The intent recognition module uses entity relationship change data corresponding to the target multimodal question data to online update the node weights and edge connection strengths of the preset dynamic knowledge graph to generate the target dynamic knowledge graph. A graph attention network is used to align the node embeddings of the target multimodal semantic vector and the target dynamic knowledge graph to generate an intent probability distribution. The module also integrates sentiment weight coefficients (voice pitch jitter, text sentiment polarity) with historical interaction data to dynamically adjust intent weights and generate sentiment-enhanced user intent data.

[0071] The response generation module uses a pre-defined hierarchical reward mechanism (basic semantic reward, diversity reward, and timeliness reward) and a Proximal Policy Optimization (PPO) algorithm to train the model and generate an intermediate reinforcement learning response model. Adversarial training improves the model's robustness, enabling the generated responses to handle noisy inputs (such as typos and semantic ambiguity), and generates a target reinforcement learning response model. Based on user intent data, target response data containing text, links, and charts is generated (for example, a text explanation for the "order anomaly" intent is generated, along with a logistics query link and a distribution map of abnormal orders).

[0072] The potential problem prediction module uses multi-head attention to model the contextual dependencies of historical interaction sequences and generates enhanced semantic vectors with contextual dependencies. Then, time-sensitive features are extracted in combination with time data (such as interaction period and interval) to generate time-sensitive feature vectors. The multimodal semantic vector, historical enhanced semantic vector, and time features are fused to generate a probability distribution list of potential problems (such as the confidence level of 85% in predicting that the user may ask "how to get a refund" next). For high-confidence potential problems, namely target candidate problems, text, links, and graphic replies are generated simultaneously, and relevant solutions are actively pushed to obtain multimodal reply data. From the data layer (dynamic compensation of the multimodal input module), the model layer (dynamic weighting of the contrastive learning module), to the knowledge layer (dynamic knowledge update of the intent recognition module), a three-layer dynamic adaptation mechanism is formed. Through multimodal fusion, dynamic knowledge representation, reinforcement learning-driven reply strategy, and forward-looking problem prediction, the present invention constructs a digital customer service technology system of "precise understanding-intelligent response-active optimization", effectively breaking through the bottlenecks of traditional systems in dynamic learning, semantic understanding, and user experience, providing efficient and personalized solutions for customer service in complex scenarios and improving user experience. It solves the technical problem that the existing digital customer service system has limited dynamic learning ability for user problems, making it difficult to accurately understand user problems, resulting in poor user experience.

[0073] Furthermore, the multimodal input module includes:

[0074] A data acquisition module is used to obtain the initial multimodal question data input by the user, and perform data preprocessing on the initial multimodal question data to generate intermediate multimodal question data;

[0075] The data detection module is used to identify the missing modality types in the intermediate multimodal problem data and generate the modality missing types;

[0076] The compensation data generation module is used to input the known modal specific data and modal missing type corresponding to the intermediate multimodal problem data into the adversarial network trained by historical interaction data, and dynamically generate modal compensation data that matches the modal missing type;

[0077] The data fusion module is used to fuse the modal compensation data with the intermediate multimodal problem data to generate target multimodal problem data.

[0078] In an embodiment of the present invention, the data acquisition module collects the initial multimodal question data (including text, voice, image, video, etc.) input by the user in real time, and performs preprocessing to generate intermediate data. The preprocessing steps include: text segmentation and embedding (such as BERT word vector), speech signal Mel spectrum conversion, image pixel normalization, etc., and outputs intermediate multimodal question data of unified dimension (such as text embedding dimension 768, speech feature dimension 80, image feature dimension 2048). The data detection module is used to identify the modality type missing in the intermediate data (such as determining whether the voice or image modality is missing) and generate a modality missing type label (such as "voice missing" and "image missing"). The data detection module uses a modality existence detection algorithm to identify the missing modality type. The specific identification process is to judge the missing modality through the modality type mask matrix of the input data (such as a three-dimensional vector [1, 0, 1] indicating the presence of text and image, and the absence of voice). The data detection module supports multimodal combination detection (such as 12 combination scenarios such as pure text input, text + image input, etc.). The compensation data generation module is based on a GAN trained with historical interaction data. It dynamically generates modality compensation data that matches the missing modality type based on known modality data and missing modality type. The adversarial network used in the compensation data generation module includes a generator and a discriminator. The generator is a 4-layer fully connected network with an activation function of LeakyReLU (α=0.2) and a Tanh function in the output layer. The discriminator is a 3-layer convolutional network with a step size of 2 and a Sigmoid function in the last layer. The generator is used to input known modality features (such as text embedding) and missing modality type encoding (such as a one-hot vector [0, 1, 0] representing missing speech) and output compensation modality features (such as speech Mel spectrum features). The discriminator is used to determine the distribution consistency between the generated compensation data and the real modality data. The loss function is:

[0079]

[0080] Among them, L WGAN-GP is the loss function of the Wasserstein Generative Adversarial Network with Gradient Penalty (WGAN-GP); E[·] is the mathematical expectation, which is the average operation of the expression in the brackets under the corresponding distribution; x~p real To indicate that the sample x is from the real modal data distribution p real The sampled one, p realIt describes the distribution of real modal data (such as real images, speech, and other modal data) in the data space. D(x) is the output of the discriminator D for the input sample x. In the framework of the Wasserstein generative adversarial network (WGAN), this output has a distance metric meaning, which is used to measure the degree to which the input data, i.e., the generated compensated data, approaches the real modal data. is the mathematical expectation of the output of the discriminator D on the generated data; z~p z To represent the latent variable z as drawn from the distribution p z G(z) is the sample data generated by the generator G based on the input z (such as generated image, speech, and other modal data); D(G(z)) is the output of the discriminator D on the sample G(z) output by the generator, which is used to measure the degree to which the generated data approaches the real data; λ is the gradient penalty coefficient, usually set to 10, which is used to control the strength of the gradient penalty term, balance the impact of the original WGAN loss and the gradient penalty term, ensure the stability of the discriminator training, avoid gradient vanishing or exploding, and enable the generator to more accurately learn the real data distribution; For the interpolation distribution p interp Next, right Find the mathematical expectation; To represent the sample is the interpolated distribution p interp The sampled one, p interp is the true data distribution p real and generate data distribution p gen The interpolation distribution is used to constrain the discriminator to also meet specific conditions in the transition region between real data and generated data; The input of the discriminator D The gradient of the discriminator output to the input rate of change; is the discriminator D The L2 norm of the gradient is used to measure the magnitude of the gradient; This is to square the deviation of the gradient norm from 1. By penalizing this deviation, the gradient of the discriminator D is forced to satisfy the Lipschitz constraint (the gradient norm is close to 1), avoiding gradient vanishing or exploding, and ensuring the stability of the discriminator training.

[0081] The model is pre-trained using complete multimodal samples from historical interaction datasets (e.g., 100,000+ text-speech-image alignment data) so that the cosine similarity between the generated compensated data and the true modality is ≥ 0.85 (e.g., the generated speech features are consistent with the emotional tendency of the true speech).

[0082] The data fusion module is used to fuse the compensation data with the intermediate data to generate the complete target multimodal question data. The fusion strategy adopted is as follows: (1) When the modality exists: directly retain the original features (such as the image features provided by the user); (2) When the modality is missing: replace it with the generated compensation features (such as the speech features generated by the adversarial network); (3) Multimodal feature splicing: use concatenation (Concat) or attention weighting (such as text feature weight 0.6, compensation speech feature weight 0.4), and output the target multimodal question data with a unified dimension (such as dimension 1024).

[0083] Furthermore, the multimodal contrastive learning module includes:

[0084] The inter-modal attention weight matrix construction module is used to construct a matrix based on the user's corresponding historical interaction preference factor and the target multimodal question data to generate the inter-modal attention weight matrix;

[0085] The initial multimodal semantic vector generation module is used to fuse the modal feature matrix corresponding to the target multimodal question data using a multi-head attention mechanism according to the inter-modal attention weight matrix to generate an initial multimodal semantic vector;

[0086] The target multimodal semantic vector generation module is used to optimize the cross-sample semantic discrimination of the initial multimodal semantic vector and generate the target multimodal semantic vector.

[0087] In an embodiment of the present invention, an inter-modal attention weight matrix construction module is used to construct an inter-modal attention weight matrix based on the user's historical interaction preference factor (such as the user's frequency and accuracy of use of voice, text, and image modalities) and the characteristic distribution of the target multimodal question data, and dynamically allocate the importance of each modality in the semantic representation. The preference factor is calculated through the user's historical interaction data (such as if a user's voice input accounts for 70% and the accuracy is 85%, then the initial weight of the voice modality is higher than other modalities), breaking the traditional fixed weight model and making the weight distribution more in line with user habits. The preference factor is corrected using the characteristic variance of the target multimodal question data (reflecting the richness of each modal information) (such as if the image modality feature variance is large at a certain moment, it means that the image contains key information, and the weight is dynamically adjusted upward) to form the final inter-modal attention weight matrix.

[0088] The initial multimodal semantic vector generation module uses an h-head attention mechanism (e.g., h = 8). Each head focuses on modal interaction features in a different subspace (e.g., the first head captures the sentiment association between text and speech, while the second head focuses on the entity association between image and text). It outputs multiple attention features, which are then concatenated to form a multi-head attention. The dynamic weights in the inter-modal attention weight matrix are then incorporated into the multi-head attention through matrix multiplication to produce the initial multimodal semantic vector.

[0089] The target multimodal semantic vector generation module is used to optimize the cross-sample semantic discrimination of the initial multimodal semantic vector. By contrastive learning, it maximizes the semantic similarity of similar samples (such as the intention of "return") and minimizes the similarity of heterogeneous samples (such as "return" and "exchange") to generate the target multimodal semantic vector.

[0090] Furthermore, the execution steps of the inter-modal attention weight matrix construction module are:

[0091] Calculate the recognition accuracy of each modality in the user's corresponding historical interaction dataset and generate a modality accuracy vector;

[0092] Substitute the modal accuracy vector into the flexibility maximum function with temperature parameters to calculate the modal preference factor vector;

[0093] Project the feature vectors of each modality corresponding to the target multimodal question data to generate a query vector and a key vector;

[0094] The query vector, key vector and modality preference factor vector are used to construct a matrix to generate the inter-modality attention weight matrix.

[0095] In this embodiment of the present invention, the inter-modal attention weight matrix construction module calculates the recognition accuracy of each modality such as text, voice, and image in the historical interaction data set corresponding to the user (such as the proportion of the text modality that correctly recognizes the user's intention in the historical interaction), and generates a modality accuracy vector A = [a text , a voice , a image ], where a text is the text recognition accuracy; a voice is the speech recognition accuracy; a image is the image recognition accuracy. Substitute the modal accuracy vector A into the softmax function with temperature parameter τ, that is Among them, x i is the i-th element in the modal accuracy vector A; τ is the temperature parameter, which is used to control the concentration of the output distribution of the softmax function; exp(x i / τ) is the value of x i / τ performs exponential operation to amplify the difference between the accuracy rates of different modes, making the high-accuracy mode more advantageous in the calculation; ∑ j exp(x j / τ) is the value of all elements x in the modal accuracy vector j After performing exponential operation, the sum is taken as the denominator to realize normalization, ensuring that the sum of each element of the preference factor vector F is 1, and the modal preference factor vector F = [f text , f voice, f image ], where f text is the text preference factor; f voice is the voice preference factor; f image is the image preference factor.

[0096] The modal feature vectors (such as text embedding vectors, speech feature vectors, and image feature vectors) corresponding to the target multimodal question data are transformed through the linear projection matrix W Q and W K Perform projection transformation to generate query vector Q = [q text ,q voice ,q image ](where q text is the text query vector; q voice is the voice query vector; q image is the image query vector) and the key vector K=[k text , k voice , k image ](where k voice is the speech key vector; k image is the image key vector; k image is the image key vector). The projection matrix dimension of the query vector and key vector is d k =64, the dimension of the value vector is d v =64.

[0097] The query vector Q, key vector K and modality preference factor vector F are used to construct the matrix, for example, by the formula (W attn is the inter-modal attention weight matrix, which is used to characterize the attention weights between different modalities in multimodal data and determine the importance of each modality in fusion; Q is the query vector; K is the key vector; QK T The query vector is multiplied by the key vector transpose; d k is the dimension of the key vector K; ⊙ is the matrix element multiplication; Diag(F) converts the modality preference factor vector F into a diagonal matrix, incorporates the user modality preference, and adjusts the weights of each modality to generate the inter-modality attention weight matrix W attn .

[0098] Furthermore, the execution steps of the initial multimodal semantic vector generation module are:

[0099] The preset mixed norm is used to calculate the characteristic entropy value of the modal characteristic matrix corresponding to the target multimodal problem data to obtain the modal entropy value vector;

[0100] The modal entropy value vector is used to construct a diagonal matrix to generate an entropy value diagonal matrix;

[0101] The multi-head attention weights of the inter-modal attention weight matrix are adjusted using the entropy diagonal matrix and the multi-head attention parameters corresponding to the target multimodal problem data to generate a set of multi-head attention weight matrices.

[0102] The initial multimodal semantic vector is constructed using the multi-head attention weight matrix set and the value matrix corresponding to the multi-head attention weight matrix set.

[0103] In the embodiment of the present invention, a preset hybrid norm (such as combining L1 and L2 norms) is used to calculate the modal feature matrix X of the target multimodal problem data. modal The characteristic entropy value of . The entropy value formula is: H(a)=-∑ i p(a i )logp(a i ), where p(a i ) is the eigenvalue a i The probability distribution of . Calculate each modality (text, speech, image, etc.) separately to generate the modal entropy value vector H = [h text , h voise , h image ].

[0104] Construct the modal entropy vector H as the entropy diagonal matrix D H =Diag(H), the diagonal elements of the matrix are the entropy values ​​of each mode, and the rest are 0. Using the entropy value diagonal matrix D H and preset multi-head attention parameters (such as the number of heads h, the projection matrix of each head Adjust the inter-modal attention weight matrix W attn For each attention head i, calculate (Example operation), generate a multi-head attention weight matrix set {W 1,hesd , W 2,hesd ,…,W h,head}. For the multi-head attention weight matrix set {W i,head} and the corresponding value matrix V = [v text , v voice , v image ] (generated by modal feature projection), through Construct the initial multimodal semantic vector Z init .

[0105] Furthermore, the target multimodal semantic vector generation module executes the following steps:

[0106] The initial multimodal semantic vector and the current batch sample set corresponding to the initial multimodal semantic vector are used to calculate the diversity index to obtain the data diversity index;

[0107] Use data diversity indicators to dynamically adjust and compare learning temperature to generate dynamic temperature parameters;

[0108] Perform cross-sample comparative learning optimization on the dynamic temperature parameters, the initial multimodal semantic vector, and the sample data corresponding to the initial multimodal semantic vector to generate multimodal features;

[0109] The multimodal features are fused with the global semantic information through bilinear transformation, and the target multimodal semantic vector is calculated.

[0110] In the embodiment of the present invention, the target multimodal semantic vector generation module uses the initial multimodal semantic vector Z init The data diversity index δ is generated by calculating the cosine similarity matrix between the samples and the corresponding current batch sample set, and then using the Shannon entropy formula for diversity index.

[0111] According to the data diversity index δ, the contrast learning temperature is dynamically adjusted using the formula (τ0 is the initial temperature, τ0 = 0.1, δ0 is the normalization constant, δ0 = 1.) When δ is high (sample diversity), τ increases, making the probability distribution of contrastive learning more uniform; when δ is low (sample similarity), τ decreases, enhancing the aggregation of similar samples and the discrimination of heterogeneous samples, thus obtaining a dynamic temperature parameter.

[0112] The dynamic temperature parameter τ, the initial multimodal semantic vector Z init Compare the learning framework with the sample data S input and use the loss function (Among them, L contrast is the contrastive learning loss function; z i is the feature vector of the current sample, representing the semantic representation of a sample after being processed by the model; j For z i The positive sample feature vector of z i Sample features belonging to the same category (same intention); sim(z i ,z j ) is used to calculate z i With z j The similarity between the two, commonly used cosine similarity, measures the closeness of the two in the feature space; τ is the dynamic temperature parameter; N is the total number of samples in the current batch, covering all samples participating in comparative learning; z k is the feature vector of the kth sample in the current batch (k = 1, 2, ..., N), which is used to calculate the normalization term of the denominator to ensure the probability normalization of the loss function; j For z i positive samples) to generate more discriminative multimodal features F multi .

[0113] By bilinear transformation (W b is a bilinear transformation matrix, G is a global semantic information matrix, such as the global features extracted by the pre-trained language model), and the multimodal features F multi Fusion with global semantic information to calculate the target multimodal semantic vector Z target ; b is the bias term; W b is the bilinear transformation matrix.

[0114] Furthermore, the intent recognition module includes:

[0115] The target dynamic knowledge graph generation module is used to update the node weights and edge connection strengths of the preset dynamic knowledge graph online using the entity relationship change data corresponding to the target multimodal question data to generate the target dynamic knowledge graph;

[0116] The intent probability distribution generation module uses a graph attention network to align the node embeddings of the target multimodal semantic vector and the target dynamic knowledge graph to generate the intent probability distribution;

[0117] The intent generation module is used to perform attention-based weighted fusion of the intent probability distribution and the user behavior patterns corresponding to historical interaction data to construct user intent data.

[0118] In an embodiment of the present invention, the target dynamic knowledge graph generation module uses the entity relationship change data corresponding to the target multimodal question data (such as the user mentions the new product attributes) to update the node weights (such as the importance of the product node) and the edge connection strength (such as the strength of the relationship between "mobile phone" and "wireless charging") of the preset dynamic knowledge graph online to generate the target dynamic knowledge graph K targte This module captures changes in entity relationships in multimodal data in real time (such as text mentioning new features or images showcasing new products) and uses an incremental learning algorithm to update node weights and edge connection strengths. The specific incremental learning steps are: Using the online stochastic gradient descent (SGD) algorithm, the knowledge graph node weights and edge connection strengths are adjusted in real time based on changes in entity relationships in the target multimodal problem data.

[0119] The intention probability distribution generation module uses the graph attention network (GAT) to analyze the target multimodal semantic vector Z target and the target dynamic knowledge graph K target Perform node embedding alignment and calculate the probability distribution P of each intent node intent =[p1,p2,…,p N ], where p1 is the probability of the first intention category; p2 is the probability of the second intention category; p Nis the probability of the Nth intent category; N is the number of intent categories. GAT uses an attention mechanism to weightedly aggregate neighbor node features, dynamically aligning multimodal semantic vectors with knowledge graph node embeddings. It leverages the updated dynamic knowledge graph structure (such as enhanced edge connections) to guide intent reasoning, making the probability distribution more consistent with real-time semantics.

[0120] The intent generation module dynamically assigns weights through the attention mechanism (e.g., a weight of 0.7 for high-frequency user behavior patterns and 0.3 for low-frequency ones), making intent generation more aligned with user habits. For example, if a user frequently asks directly, "How do I return a product?", when they currently input "order return," the behavior pattern has a high weight and is accurately identified as a "return intention." The intent generation module combines historical behavior patterns (long-term memory) with current intent probabilities (short-term semantics) using the attention mechanism to balance the contributions of both.

[0121] Furthermore, the intent recognition module also includes:

[0122] The multi-emotion feature extraction module is used to extract emotional features from the target multimodal question data and generate physiological signal features and text emotional polarity features. The physiological signal features include the voice fundamental frequency jitter rate and the intensity of facial expression action units.

[0123] The emotion weight coefficient generation module is used to input physiological signal features, text emotion features and user behavior patterns in historical interaction data into the gated recurrent network to generate emotion weight coefficients;

[0124] The intention enhancement module is used to dynamically adjust the user intention data using the emotion weight coefficient to generate emotion-enhanced user intention data.

[0125] In an embodiment of the present invention, a multi-emotion feature extraction module is used to extract emotional features from target multimodal question data, and generate physiological signal features (speech fundamental frequency jitter rate, facial expression action unit intensity) and text emotion polarity features. Breaking through the traditional single text emotion analysis, it simultaneously captures the physiological signal features of speech (fundamental frequency jitter rate reflects emotional fluctuations) and facial expressions (action unit intensity quantifies expression intensity), and combines the text emotion polarity (positive, negative, neutral) to form a comprehensive emotion feature set. The speech fundamental frequency jitter rate directly reflects the degree of emotional excitement (such as the jitter rate increases significantly when angry), and the facial expression action unit intensity quantifies the emotion intensity through facial muscle activity (such as frowning corresponds to the intensity of dissatisfaction), making the emotion feature more measurable and objective, and providing a solid data foundation for the subsequent emotion weight calculation.

[0126] The sentiment weight coefficient generation module inputs physiological signal features, text sentiment features, and user behavior patterns from historical interaction data into a gated recurrent neural network (GRU) to generate sentiment weight coefficients ω = [ω1, ω2, ω3] (ω1, ω2, ω3 are the weights of physiological signals, text sentiment, and historical behavior patterns, respectively). GRUs excel at processing sequential data and can effectively capture the temporal dependencies between sentiment features and historical behavior patterns. By integrating physiological signals, text sentiment, and historical behavior patterns, the GRU uses gating mechanisms (update gate and reset gate) to selectively forget and remember information.

[0127] The intention enhancement module uses the emotion weight coefficient ω to analyze the user intention data I user Perform dynamic weighted adjustment, i.e. I enhanced =ω1·I physio +ω2·I text +ω3·I history (Among them, I physio , I text , I history are physiological signals, text emotions, and user intention data of historical behavior patterns), generating emotion-enhanced user intention data I enhanced Dynamic weighting based on the emotion weight coefficient ensures that intent data is more aligned with emotional intensity and tendency. Combined with the historical behavior pattern weight ω3, personalized emotion enhancement is achieved. Frequent users often interact with strong emotions, and by increasing their emotion weight, intent recognition is more consistent with their habits. For infrequent users, flexible adjustments are made based on their current emotional characteristics, improving the personalization and accuracy of overall intent recognition.

[0128] Furthermore, the execution steps of the reply generation module are:

[0129] The initial reinforcement learning response model is trained using a preset hierarchical reward mechanism and a proximal policy optimization algorithm to generate an intermediate reinforcement learning response model.

[0130] Adopting adversarial training mechanism to enhance the response robustness of intermediate reinforcement learning response model and generate target reinforcement learning response model;

[0131] The user intention data is input into the target reinforcement learning response model to construct the response data and generate the target response data.

[0132] In an embodiment of the present invention, the preset hierarchical reward mechanism includes accuracy reward, emotion matching reward, and information richness reward. The accuracy reward is used to evaluate the degree of match between the reply and the user's intention (e.g., if the intention is "return", the reply must include the return process). The emotion matching reward is used to measure the consistency between the emotional polarity of the reply and the user's emotion (e.g., when the user is angry, the reply must reflect empathy). The information richness reward is used to encourage the provision of a complete solution (e.g., not only informing the return address, but also providing logistics suggestions). Compared with the traditional single reward function, this mechanism enables the model to learn a more comprehensive reply strategy. The proximal policy optimization (PPO) algorithm avoids performance fluctuations caused by large changes in the strategy by limiting the strategy update step size. The initial reinforcement learning reply model is trained by adopting the preset hierarchical reward mechanism (such as accuracy reward, emotion matching reward, information richness reward) and the proximal policy optimization (PPO) algorithm to generate an intermediate reinforcement learning reply model.

[0133] The adversarial training mechanism includes a generator and a discriminator. The generator is an intermediate reinforcement learning response model that generates candidate responses. The discriminator is used to determine whether the generated response is vulnerable to attack (such as containing sensitive words or logical contradictions). Through adversarial games, the generator learns to generate more robust responses, and the discriminator learns to identify potential attack points. During the training process, adversarial samples (such as inputs containing spelling errors and offensive words) are dynamically injected to force the model to learn a robust response strategy. For example, when the input contains "junk products", the model can still generate professional and soothing responses. Therefore, an adversarial training mechanism (generator-discriminator architecture) is used to enhance the response robustness of the intermediate reinforcement learning response model and generate the target reinforcement learning response model.

[0134] User intent data (especially emotion-enhanced user intent data) is fed into the target reinforcement learning response model to construct response data and generate target response data. The model adjusts its response strategy based on the intensity of the user's emotion. For example, when the user is angry, it prioritizes expressing apologies and providing quick solutions (e.g., "We apologize for the inconvenience. We will immediately process your return."). When the user is satisfied, it provides value-added information (e.g., "Thank you for your approval! If you have any other needs, please feel free to contact us."). Furthermore, personalized responses are generated based on behavioral patterns in historical user interaction data (e.g., high-frequency users prefer concise responses, while low-frequency users require detailed explanations), thereby improving user satisfaction scores.

[0135] Furthermore, the execution steps of the potential problem prediction module are:

[0136] A multi-head attention mechanism is used to perform temporal modeling on the user's corresponding historical interaction sequence to generate an enhanced semantic vector with contextual dependencies;

[0137] Perform multi-scale feature extraction on the user's corresponding time data to generate a time-sensitive feature vector;

[0138] The target multimodal semantic vector, enhanced semantic vector, and time-sensitive feature vector are dynamically weighted through a learnable gated fusion network to construct a fusion prediction feature matrix.

[0139] A soft maximization classifier with temperature adjustment parameters is used to perform linear transformation on the fusion prediction feature matrix to generate a list of potential problem probability distributions;

[0140] Select the candidate questions with the highest confidence in the potential question probability distribution list according to a preset number to generate multiple target candidate questions;

[0141] The target candidate question is input into the target reinforcement learning response model to construct the response data and generate multimodal response data.

[0142] In an embodiment of the present invention, the multi-head attention mechanism simultaneously focuses on short-term (recent conversations) and long-term dependencies (early key issues) in historical interactions by calculating multiple attention heads in parallel. For example, the user first inquires about "product functions" and then repeatedly provides feedback on "usage failures". The model can capture the evolution from functional understanding to quality issues. Through the self-attention mechanism, each historical interaction vector interacts with other vectors to enhance semantic expression. For example, the association between the historical "return policy" and the current "product damage" is strengthened, and the generated enhanced semantic vector more comprehensively reflects the context of the user's problem. The multi-head attention mechanism is used to perform time series modeling on the historical interaction sequence corresponding to the user (such as the past 10 conversations) to generate an enhanced semantic vector with contextual dependencies.

[0143] Multi-scale feature extraction is performed on the user's corresponding time data (such as interaction timestamps and historical session intervals) to generate a time-sensitive feature vector. The specific extraction process is: simultaneously extracting micro (specific time, such as 9:00 a.m.), meso (weekday / weekend), and macro (quarter / season) time features. For example, users frequently inquire on weekday evenings and may potentially inquire about "home repairs" on weekend mornings. Multi-scale features can capture such periodic patterns, and the richness of temporal features is three times higher than that of single-scale methods. Feature extraction of multi-scale temporal data is performed using a temporal convolutional network (TCN), automatically learning the association between temporal patterns and user needs. For example, if it is found that users frequently inquire about billing issues at the beginning of each month, the model enhances its sensitivity to the time point at the beginning of the month, improves the accuracy of temporal pattern recognition, and obtains a time-sensitive feature vector with high accuracy.

[0144] The target multimodal semantic vector, enhanced semantic vector, and time-sensitive feature vector are dynamically weighted using a learnable gated fusion network to construct a fused prediction feature matrix. The gated fusion network dynamically generates weights based on the inputs. For example, in emergency scenarios, the current semantic vector has a high weight (e.g., when the user is emotionally agitated). In long-term demand prediction, the historical enhanced semantic vector has a high weight (e.g., when the user has long focused on a certain type of problem). A gating mechanism selectively fuses different features to avoid interference from redundant information. For example, when temporal features conflict with semantic features (e.g., a user has historically inquired about weekend issues but currently has an urgent problem), the gated network automatically balances the weights of the two.

[0145] A softmax classifier with a temperature adjustment parameter is used to perform a linear transformation on the fused prediction feature matrix to generate a list of potential problem probability distributions. The temperature parameter is dynamically adjusted based on the input data. For example, in scenarios with high uncertainty (e.g., ambiguous user questions), increasing the temperature parameter smoothes the probability distribution and explores more possible problems. In scenarios with high certainty (e.g., clear user requirements), decreasing the temperature parameter focuses on high-probability problems. This temperature adjustment avoids the overconfidence problem of traditional softmax classifiers (e.g., completely ignoring low-probability problems) at low temperatures, resulting in a more reasonable probability distribution of potential problems.

[0146] The preset number is a critical value set based on actual needs, such as 3. According to the preset number, the candidate questions with the highest confidence in the probability distribution list of potential problems are selected to generate multiple target candidate questions. This step adopts a multi-candidate question strategy, retaining multiple high-confidence questions at the same time, rather than only selecting the highest probability questions, to deal with question ambiguity. For example, the user question "How long is the product warranty?" may correspond to multiple potential questions such as "warranty policy", "extended warranty", and "warranty coverage". The confidence threshold is automatically adjusted according to the temperature parameter. For example, when the temperature parameter is large, the threshold is relaxed to ensure sufficient candidate questions; when the temperature parameter is small, the threshold is tightened to ensure candidate quality. The dynamic threshold improves the balance between the recall and precision of candidate questions.

[0147] The target candidate question is input into the target reinforcement learning response model to construct response data and generate multimodal response data (such as text, voice, and picture instructions). This step uses a multimodal adaptive generation method to automatically select the optimal response modality combination based on the candidate question type. For example: Complex operation questions: Generate a combined text and picture response. Urgent questions: Prioritize voice broadcast for quick solutions. Combined with the user's historical behavior patterns (such as high-frequency users prefer concise responses) and emotional state (such as angry users need soothing language), the output of the reinforcement learning response model is secondary optimized.

[0148] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal digital customer service system, characterized by: include: A multimodal input module is used to obtain initial multimodal question data input by the user and use a dynamic adaptive compensation mechanism to perform data compensation on the initial multimodal question data to generate target multimodal question data; A multimodal contrastive learning module is used to dynamically assign weights to the target multimodal question data using a cross-modal attention mechanism to generate a target multimodal semantic vector; An intent recognition module is used to perform intent recognition on the target multimodal semantic vector based on historical interaction data and a preset dynamic knowledge graph to generate user intent data; a response generation module, configured to input the user intention data into a target reinforcement learning response model to construct response data and generate target response data; The potential problem prediction module is used to use the target multimodal semantic vector, the historical interaction sequence and time data corresponding to the user to predict potential problems, and generate a potential problem probability distribution list and multimodal response data.

2. The system according to claim 1, wherein: The multimodal input module includes: A data acquisition module is used to acquire initial multimodal question data input by a user, and perform data preprocessing on the initial multimodal question data to generate intermediate multimodal question data; A data detection module, configured to identify the modality types missing from the intermediate multimodal problem data and generate a modality missing type; A compensation data generation module is configured to input the known modality specific data corresponding to the intermediate multimodal problem data and the modality missing type into an adversarial network trained by historical interaction data, and dynamically generate modality compensation data matching the modality missing type; A data fusion module is used to fuse the modal compensation data with the intermediate multimodal problem data to generate target multimodal problem data.

3. The system according to claim 1, wherein: The multimodal contrastive learning module includes: An inter-modal attention weight matrix construction module is used to construct a matrix based on the user historical interaction preference factor corresponding to the user and the target multimodal question data to generate an inter-modal attention weight matrix; An initial multimodal semantic vector generation module is used to fuse the modal feature matrix corresponding to the target multimodal question data using a multi-head attention mechanism according to the inter-modal attention weight matrix to generate an initial multimodal semantic vector; The target multimodal semantic vector generation module is used to optimize the cross-sample semantic discrimination of the initial multimodal semantic vector to generate a target multimodal semantic vector.

4. The system according to claim 3, characterized in that The execution steps of the inter-modal attention weight matrix construction module are: Calculate the recognition accuracy of each modality in the historical interaction data set corresponding to the user, and generate a modality accuracy vector; Substituting the modal accuracy vector into the flexibility maximum function with a temperature parameter, and calculating the modal preference factor vector; Performing a projection transformation on each modal feature vector corresponding to the target multimodal question data to generate a query vector and a key vector; The query vector, the key vector and the modality preference factor vector are used to construct a matrix to generate an inter-modality attention weight matrix.

5. The system according to claim 3 or 4, characterized in that The execution steps of the initial multimodal semantic vector generation module are: Calculating the characteristic entropy value of the modal characteristic matrix corresponding to the target multimodal problem data using a preset hybrid norm to obtain a modal entropy value vector; The modal entropy value vector is used to construct a diagonal matrix to generate an entropy value diagonal matrix; Adjusting the multi-head attention weights of the inter-modal attention weight matrix using the entropy diagonal matrix and the multi-head attention parameters corresponding to the target multimodal problem data to generate a multi-head attention weight matrix set; An initial multimodal semantic vector is constructed using the multi-head attention weight matrix set and the value matrix corresponding to the multi-head attention weight matrix set.

6. The system according to claim 3, wherein: The target multimodal semantic vector generation module executes the following steps: Calculating a diversity index using the initial multimodal semantic vector and a current batch of samples corresponding to the initial multimodal semantic vector to obtain a data diversity index; Dynamically adjust and compare learning temperature using the data diversity index to generate dynamic temperature parameters; Performing cross-sample comparative learning optimization on the dynamic temperature parameter, the initial multimodal semantic vector, and the sample data corresponding to the initial multimodal semantic vector to generate a multimodal feature; The multimodal features are fused with global semantic information through bilinear transformation to calculate the target multimodal semantic vector.

7. The system according to claim 1, wherein: The intention recognition module includes: A target dynamic knowledge graph generation module is used to update the node weights and edge connection strengths of a preset dynamic knowledge graph online using the entity relationship change data corresponding to the target multimodal question data to generate a target dynamic knowledge graph; An intent probability distribution generation module is used to use a graph attention network to perform node embedding alignment on the target multimodal semantic vector and the target dynamic knowledge graph to generate an intent probability distribution; The intent generation module is used to perform attention-based weighted fusion of the intent probability distribution and the user behavior patterns corresponding to the historical interaction data to construct user intent data.

8. The system according to claim 7, characterized in that The intention recognition module also includes: A multi-emotion feature extraction module is used to extract emotional features from the target multimodal question data to generate physiological signal features and text emotional polarity features, wherein the physiological signal features include voice fundamental frequency jitter rate and facial expression action unit intensity; An emotion weight coefficient generation module is used to input the physiological signal features, the text emotion features and the user behavior pattern in the historical interaction data into a gated recurrent network to generate an emotion weight coefficient; The intention enhancement module is used to dynamically adjust the user intention data by using the emotion weight coefficient to generate emotion-enhanced user intention data.

9. The system according to claim 1, wherein: The execution steps of the reply generation module are: The initial reinforcement learning response model is trained using a preset hierarchical reward mechanism and a proximal policy optimization algorithm to generate an intermediate reinforcement learning response model. Adopting an adversarial training mechanism to enhance the response robustness of the intermediate reinforcement learning response model and generate a target reinforcement learning response model; The user intention data is input into the target reinforcement learning response model to construct response data and generate target response data.

10. The system according to claim 1, wherein: The execution steps of the potential problem prediction module are: A multi-head attention mechanism is used to perform temporal modeling on the historical interaction sequence corresponding to the user to generate an enhanced semantic vector with contextual dependencies; Perform multi-scale feature extraction on the time data corresponding to the user to generate a time-sensitive feature vector; Dynamically weighting the target multimodal semantic vector, the enhanced semantic vector, and the time-sensitive feature vector through a learnable gated fusion network to construct a fusion prediction feature matrix; Using a soft maximization classifier with a temperature adjustment parameter to perform a linear transformation on the fusion prediction feature matrix to generate a probability distribution list of potential problems; Selecting a preset number of candidate questions with the highest confidence in the potential question probability distribution list to generate multiple target candidate questions; The target candidate question is input into the target reinforcement learning response model to construct response data and generate multimodal response data.

Images