Intelligent customer service method and system based on multi-modal dynamic fusion

Abstract

The invention discloses an intelligent customer service method and system based on multi-modal dynamic fusion, and the method comprises the steps: obtaining a user service demand and a behavior log on a service terminal; performing feature extraction and dynamic fusion on the user service demand and the behavior log; inputting the fusion features into a multi-task learning model to obtain intention categories and emotion scores; and generating a service action through reinforcement learning based on the intention category and the emotion score. Through multi-modal dynamic fusion and an active service mechanism, the intention of the user can be more accurately understood, and the corresponding service action is provided at the first time, so that the dialogue turns are reduced, and the service efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and customer service technology, and more particularly to an intelligent customer service method and system based on multimodal dynamic fusion. Background Art

[0002] With the rapid development of internet technology and the increasing diversity of financial services, users have placed higher demands on the convenience and efficiency of banking services. As a key interactive interface between banks and users, the intelligence level of intelligent customer service systems directly affects user experience and satisfaction.

[0003] However, traditional bank intelligent customer service systems mainly rely on voice or text or a combination of the two to process user requests. This approach has many limitations and is difficult to meet users' growing service needs and cannot provide effective services; for example, reference information is one-sided and recognition accuracy is low.

[0004] Furthermore, traditional intelligent customer service systems typically employ a passive response model, generating responses based solely on the user's current input. This ignores the user's actions on the service terminal and fails to proactively intervene and guide based on their actions. This results in inefficient service and an inability to promptly respond to potential user needs. Furthermore, due to the limitations of a single modality and passive response model, users often need to repeat the same question or intent multiple times to receive a satisfactory response. This not only increases conversational turns and reduces service efficiency, but also severely impacts the user experience.

[0005] Therefore, a smarter and more efficient customer service approach is needed to provide services and improve user experience and satisfaction. Summary of the Invention

[0006] In view of this, the present invention provides an intelligent customer service method and system based on multimodal dynamic fusion, which aims to achieve accurate intention recognition and proactive service by combining voice, text and user behavior data.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present application provides an intelligent customer service method based on multimodal dynamic fusion, the steps comprising:

[0009] Obtain user service requirements and behavior logs on service terminals;

[0010] Extract features and dynamically fuse user service needs and behavior logs;

[0011] Input the fused features into the multi-task learning model to obtain intent categories and sentiment scores;

[0012] Generate service actions via reinforcement learning based on intent categories and sentiment scores.

[0013] In an optional embodiment, the user service demand includes voice signals and text data.

[0014] In an optional embodiment, feature extraction of user behavior logs includes: modeling user behavior log sequences through an LSTM network, and identifying abnormal operations using an attention mechanism; the attention calculation formula is:

[0015] α t =softmax(W·[h t :p t :r t ]+b)

[0016] Where, α t represents the attention weight at time step t, h t represents the hidden state at time step t, p t represents the pause time of time step t, r t represents the number of repetitions of time step t, W represents the weight matrix, and b represents the bias term.

[0017] Introducing pause time and repetition count as additional features significantly improves the sensitivity of anomaly detection.

[0018] In an optional embodiment, feature fusion includes:

[0019] The FastDTW algorithm is used to align multimodal feature sequences;

[0020] The aligned features are input into the gated fusion network to obtain the initial weights of the multimodal features;

[0021] Dynamically modify the initial weights;

[0022] The multimodal features are weightedly fused using the corrected weights.

[0023] In an optional embodiment, dynamically modifying the initial weights includes:

[0024] When the voice emotion is triggered, the initial weight is dynamically modified according to the following formula;

[0025]

[0026] When abnormal behavior is triggered, the initial weight is dynamically modified according to the following formula;

[0027]

[0028] Where w语音 、w 文本 and w 行为 Respectively represent the dynamically modified voice weight, text weight and behavior weight, and They represent the initial weight of speech, initial weight of text and initial weight of behavior respectively.

[0029] In an optional embodiment, generating a service action based on the intent category and the sentiment score through reinforcement learning includes:

[0030] Define the state space, action space and reward function, train the reinforcement learning network; and use the trained reinforcement learning network to generate service actions; wherein,

[0031] The state space includes intent categories and sentiment scores,

[0032] The action space includes pop-up guidance, manual transfer, product recommendation and silent waiting.

[0033] Determine the reward function as:

[0034] R t =0.6·Operation success rate+0.3·User satisfaction-0.1·Conversation rounds.

[0035] In an optional embodiment, the reinforcement learning network includes:

[0036] Input layer: receives a 7-dimensional state vector as input, which consists of a 6-dimensional intent category and a 1-dimensional sentiment score;

[0037] Hidden layer: contains at least two layers of fully connected neural networks, with 64 neurons in the first layer and 32 neurons in the second layer. Each layer uses the ReLU activation function.

[0038] Output layer: takes a 4-dimensional Q-value vector as output, which corresponds to all possible actions in the action space.

[0039] In a second aspect, the present application further provides an intelligent customer service system based on multimodal dynamic fusion, which applies the intelligent customer service method based on multimodal dynamic fusion as described above, and the system includes:

[0040] Interaction data acquisition unit, used to obtain user service requirements and behavior logs on the service terminal;

[0041] The data processing unit is used to extract features and dynamically fuse user service requirements and behavior logs;

[0042] Intent sentiment analysis unit, used to input fusion features into the multi-task learning model to obtain intent categories and sentiment scores;

[0043] The service action generation unit is used to generate service actions through reinforcement learning based on intent categories and sentiment scores.

[0044] Compared with the prior art, the intelligent customer service method and system based on multimodal dynamic fusion disclosed in the present invention have the following effects:

[0045] 1. By combining voice signals, text data, and user behavior logs, we can more comprehensively understand user intent, thereby overcoming the one-sidedness of single-modal information and significantly improving the accuracy of intent recognition;

[0046] 2. By introducing user behavior logs and using LSTM networks and attention mechanisms to model user behavior, it can proactively identify abnormal user operations and potential needs. Furthermore, based on reinforcement learning, service actions can be generated for proactive intervention and guidance, thereby better meeting users' diverse needs and improving service efficiency and user experience.

[0047] 3. Introducing sentiment scoring to dynamically modify the weights of multimodal features enables the system to better understand the user's emotional state and adjust service strategies based on the user's emotional state when processing user requests, further improving user satisfaction.

[0048] Through multimodal dynamic fusion and proactive service mechanism, the present invention can understand user intentions more accurately and provide corresponding service actions in the first time, thereby reducing conversation rounds and improving service efficiency. This application significantly improves the intelligence level and service efficiency of the intelligent customer service system, solves many limitations of traditional customer service systems, and provides users with a more convenient, efficient and personalized service experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0050] Figure 1 This is a flow chart of the intelligent customer service method based on multimodal dynamic fusion of the present invention;

[0051] Figure 2 This is a flow chart of the dynamic fusion of multimodal features of the present invention;

[0052] Figure 3 This is a schematic diagram of the decision-making layer execution process of the present invention. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0054] The following description sets forth many specific details to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0055] The embodiment of the present invention discloses an intelligent customer service method based on multimodal dynamic fusion. Figure 1 ; Mainly includes the following steps:

[0056] First, obtain user service needs and behavior logs on the service terminal and pre-process them first;

[0057] In this embodiment, the service demand includes voice signals and text data; wherein,

[0058] Voice signals are collected through a microphone or call recordings, supporting real-time streaming processing. Voice preprocessing uses acoustic feature extraction tools (such as Librosa) to obtain low-level features such as MFCC (Mel-Frequency Cepstral Coefficients) and pitch.

[0059] Text data: Text from online chats and ASR-translated text. Text preprocessing includes word segmentation, stop word removal, and conversion into word vectors or word frequency features.

[0060] User behavior logs on service terminals: Records operational events within terminals (e.g., apps) (e.g., page jumps, button clicks, dwell time), with millisecond-level timestamp accuracy. Parses JSON-formatted logs to extract structured data such as event type and timestamp.

[0061] Second, extract features and dynamically integrate user service needs and behavior logs;

[0062] For feature extraction, the feature extraction layer performs advanced modeling and deep feature extraction on the pre-processed features. In an optional embodiment,

[0063] Speech module: uses the Wave2Vec 2.0 model to extract features such as pitch, speaking rate, and silent segments;

[0064] Text module: Encodes semantic information through the RoBERTa model and extracts key entities (such as "transfer failed") through dependency parsing.

[0065] Behavior module: Models user operation sequences through the LSTM network and combines the attention mechanism to identify abnormal operations.

[0066] The behavior module is a core component of feature extraction, used to analyze user behavior data within applications to capture potential needs and abnormal operation patterns. This module receives user behavior logs, including structured data such as page jump paths, button click events, dwell time, operation intervals, and repetition times. It then uses a two-layer LSTM (Long Short-Term Memory) network to perform time series modeling of user operation sequences.

[0067] Furthermore, combined with the attention mechanism, the weight of each time step is dynamically calculated, focusing on abnormal events (such as clicking the same button three times within 5 seconds or staying on a single page for more than 60 seconds), and outputting a hidden state sequence.

[0068] In this embodiment, the attention formula is:

[0069] α t =softmax(W·[h t :p t :r t ]+b)

[0070] Where h t is the LSTM hidden state, p t is the pause time, r t is the number of repetitions.

[0071] Introducing pause time and repetition count as additional features can significantly improve the sensitivity of anomaly detection.

[0072] It should be noted that Transformer can also be used instead of LSTM to perform time series modeling of user operation sequences. However, due to the risk of overfitting for short sequence data, LSTM is preferred.

[0073] Dynamic feature fusion is a key step in achieving collaborative analysis of multimodal data. Its core task is to align speech, text, and behavioral features in the time dimension and dynamically assign weights. In an optional embodiment, the fusion process refers to Figure 2 ,

[0074] First, we used the Fast DTW (Dynamic Time Warping) algorithm to align the time series of the multimodal data, using a 5-second window size and a 1-second step size. This ensured that the "unable to transfer money" event in the voice message and the "failed click" event in the behavior log were accurately aligned on the timeline.

[0075] Secondly, the aligned feature matrix is input into the gated fusion network, and the initial weights of each modality are calculated through the fully connected layer;

[0076] In this embodiment, the initial weight calculation formula is as follows:

[0077]

[0078] The initial weights satisfy:

[0079] Then, the initial weight is dynamically modified. In this embodiment, the initial weight is modified according to the preset conditions. The modification rules are as follows:

[0080] (1) Voice emotion intensity triggering conditions

[0081] Trigger threshold: speech emotion score S 语音 >First preset threshold, in this embodiment, is 3 (e.g., high anxiety scenario);

[0082] Adjustment logic:

[0083]

[0084] Normalization: Ensure that the sum of weights is still 1, that is, w 语音 +w 文本 +w 行为 =1.

[0085] (2) Triggering conditions for behavioral abnormality

[0086] Trigger threshold: behavioral emotion score S 行为 >The second preset threshold, in this embodiment, is 7 (such as repeated clicks).

[0087] Adjustment logic:

[0088]

[0089] At this time, the behavioral modality weight is fixed at 0.8, and the remaining 0.2 weight is distributed between voice and text according to the initial ratio.

[0090] In the above formula, w 语音 、w 文本 and w 行为 Respectively represent the dynamically modified voice weight, text weight and behavior weight, and They represent the initial weight of speech, initial weight of text and initial weight of behavior respectively.

[0091] This application dynamically adjusts weights based on real-time sentiment scores (such as voice emotion intensity or behavioral abnormality) to make the fusion strategy more tailored to the needs of the current scenario. For example, it can increase the weight of voice modality in high-anxiety scenarios, or prioritize behavioral modality when behavioral abnormalities occur, thereby increasing the system's sensitivity to key information.

[0092] It should be noted that the speech emotion score and behavior emotion score of the present application are obtained by mapping the feature sequences output by the speech module and the behavior module respectively by Sofmax.

[0093] The weight formula of this application is dynamically adjusted based on the voice emotion score, text confidence and behavioral abnormality. For example, when the voice emotion level exceeds 3 points (high anxiety), the voice weight is increased by 0.2; if the behavioral abnormality exceeds 0.7 (such as repeated clicks), the behavioral weight is directly increased to 0.8.

[0094] Finally, the multimodal features are weightedly fused using the corrected weights; the fusion formula can be expressed as:

[0095] f 融合 =w 语音 ·f 语音 +w 文本 ·f 文本 +w 行为 ·f 行为

[0096] For the best selection, the weighted multimodal features are reduced to a 256-dimensional multimodal feature vector through PCA and input into the decision layer.

[0097] Third, the execution process of the decision-making level of this application is as follows Figure 3 Shown, including:

[0098] Input the fused features into the multi-task learning model to obtain intent categories and sentiment scores;

[0099] and generating service actions through reinforcement learning based on intent categories and sentiment scores.

[0100] As the core control unit of customer service, the decision output layer comprehensively analyzes data and generates the optimal service strategy through the collaborative work of multi-task learning models and reinforcement learning technology.

[0101] In an optional embodiment, the multi-task learning model adopts a shared bottom layer + task branch structure, extracts high-level semantic features through the shared layer, and compresses features through a 256→128→64-dimensional fully connected layer to reduce computational redundancy. Furthermore, the model input is a dynamically fused multimodal feature (256 dimensions), and the output is an intent category and sentiment score. As a preferred implementation scheme, the intent category branch outputs six common business labels (such as "transfer failure" and "financial consulting") through the Softmax function, and the sentiment score branch regresses and analyzes the user's emotional level (0-5 points).

[0102] In an optional embodiment, generating a service action based on the intent category and the sentiment score through a reinforcement learning engine includes the following steps:

[0103] 1. Define the state space

[0104] Intent category: intent classification results output by the multi-task model (including six categories of labels: transfer failure, account inquiry, password reset, transaction consultation, card loss report, and bill dispute);

[0105] Sentiment score: the sentiment intensity value (0-5 points) output by the multi-task model;

[0106] State vector construction: Concatenate the intent category (one-hot encoding) and the sentiment score into a state vector s t .

[0107] 2. Action space definition. This application supports four types of service actions:

[0108] Pop-up guide: Display step-by-step operation tutorial;

[0109] Transfer to manual customer service: transfer the conversation to manual customer service;

[0110] Product recommendation: recommend relevant financial products;

[0111] Silently wait: Maintain the current state and wait for further user input.

[0112] Action code: a t ∈{0,1,2,3}

[0113] 3. Reward Function Design

[0114] In order to quantify service effects and balance efficiency and user experience, the reward function of this application is designed as follows:

[0115] R t = 0.6·Operation success rate + 0.3·User satisfaction - 0.1·Conversation rounds

[0116] Among them, the operation success rate (0-1) indicates whether the user completes the task in the subsequent operation (such as successful transfer);

[0117] User satisfaction (0-5 means calculated by user rating after the conversation or implicit behavior (such as stay time);

[0118] The conversation turn represents the total number of interactions in the current conversation (normalized to 0-1).

[0119] In an exemplary embodiment, the DQN network structure includes:

[0120] Input layer: 7-dimensional state vector (6-dimensional intent + 1-dimensional sentiment score);

[0121] Hidden layer: two-layer fully connected network (64→32 neurons), ReLU activation;

[0122] Output layer: 4-dimensional Q-value vector, corresponding to the expected cumulative reward of each action.

[0123] The training process includes:

[0124] Experience playback: store historical data (s t , a t , r t , s t-1 ), buffer capacity 10,000, batch size 64;

[0125] ε-greedy exploration: the initial exploration rate is ε = 0.5, and it decays by 0.05 every 1000 steps;

[0126] Target network update: copy the main network parameters to the target network every 1000 steps to stabilize training;

[0127] Loss function: Mean squared error (MSE) calculates the Q value prediction error.

[0128] An example of reinforcement learning is:

[0129] Scenario: User intent is "transfer failed" (label 4), with a sentiment score of 4.2.

[0130] State vector: intent one-hot encoding (6 dimensions) + sentiment score (1 dimension) → 7-dimensional vector.

[0131] Q-value calculation: DQN outputs the Q-value of each action (such as 1.5, 0.4, 0.3, 0.1).

[0132] Action selection: select the action with the highest Q value (pop-up guidance);

[0133] Reward calculation: The user successfully completes the transfer (success rate 1), the satisfaction score is 4, and the conversation rounds are 3 times. The reward at this time is: R t =0.6×1+0.3×0.8-0.1×0.3=0.87.

[0134] The system dynamically optimizes strategies through a reward function, where operation success rate (60% weight) and user satisfaction (30% weight) serve as positive incentives, and conversation rounds (10% weight) serve as an efficiency penalty, ensuring that service actions respond quickly while taking into account user experience.

[0135] In another embodiment, the present application further provides an intelligent customer service system based on multimodal dynamic fusion, which applies the intelligent customer service method based on multimodal dynamic fusion as described above, and the system includes:

[0136] Interaction data acquisition unit, used to obtain user service requirements and behavior logs on the service terminal;

[0137] The data processing unit is used to extract features and dynamically fuse user service requirements and behavior logs;

[0138] Intent sentiment analysis unit, used to input fusion features into the multi-task learning model to obtain intent categories and sentiment scores;

[0139] The service action generation unit is used to generate service actions through reinforcement learning based on intent categories and sentiment scores.

[0140] In this embodiment, the system implementation details are consistent with the aforementioned intelligent customer service method, so they will not be repeated here. The detailed process refers to the above description.

[0141] This application significantly improves the accuracy of customer intent recognition by combining multimodal data and performing dynamic weight allocation; at the same time, by triggering intervention through page dwell time and abnormal operations, it breaks through the traditional passive response mode and can effectively improve service efficiency and user experience.

[0142] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0143] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An intelligent customer service method based on multimodal dynamic fusion, characterized in that: Obtain user service requirements and behavior logs on service terminals; Extract features and dynamically fuse user service needs and behavior logs; Input the fused features into the multi-task learning model to obtain intent categories and sentiment scores; Generate service actions via reinforcement learning based on intent categories and sentiment scores.

2. The intelligent customer service method according to claim 1, characterized in that: The user service demand includes voice signals and text data.

3. The intelligent customer service method according to claim 1, characterized in that: Feature extraction of user behavior logs includes: modeling user behavior log sequences through LSTM networks and identifying abnormal operations using the attention mechanism; the attention calculation formula is: a t =softmax(W·[h t :p t :r t ]+b) Where, α t represents the attention weight at time step t, h t represents the hidden state at time step t, p t represents the pause time of time step t, r t represents the number of repetitions of time step t, W represents the weight matrix, and b represents the bias term.

4. The intelligent customer service method according to claim 2, characterized in that: Feature fusion includes: The FastDTW algorithm is used to align multimodal feature sequences; The aligned features are input into the gated fusion network to obtain the initial weights of the multimodal features; Dynamically modify the initial weights; The multimodal features are weightedly fused using the corrected weights.

5. The intelligent customer service method according to claim 4, characterized in that: Dynamically modify the initial weights, including: When the voice emotion is triggered, the initial weight is dynamically modified according to the following formula; When abnormal behavior is triggered, the initial weight is dynamically modified according to the following formula; Where w 语音 、w 文本 and w 行为 Respectively represent the dynamically modified voice weight, text weight and behavior weight, and They represent the initial weight of speech, initial weight of text and initial weight of behavior respectively.

6. The intelligent customer service method according to claim 1, characterized in that: Generate service actions based on intent categories and sentiment scores through reinforcement learning, including: Define the state space, action space and reward function, train the reinforcement learning network; and use the trained reinforcement learning network to generate service actions; wherein, The state space includes intent categories and sentiment scores, The action space includes pop-up guidance, manual transfer, product recommendation and silent waiting. Determine the reward function as: R t =0.6·Operation success rate+0.3·User satisfaction-0.1·Conversation rounds.

7. The intelligent customer service method according to claim 6, characterized in that: The reinforcement learning network consists of: Input layer: receives a 7-dimensional state vector as input, which consists of a 6-dimensional intent category and a 1-dimensional sentiment score; Hidden layer: contains at least two layers of fully connected neural network, each layer uses ReLU activation function; Output layer: takes a 4-dimensional Q-value vector as output, which corresponds to all possible actions in the action space.

8. An intelligent customer service system based on multimodal dynamic fusion, characterized by: The intelligent customer service method based on multimodal dynamic fusion according to any one of claims 1 to 7 is applied, and the system includes: Interaction data acquisition unit, used to obtain user service requirements and behavior logs on the service terminal; The data processing unit is used to extract features and dynamically fuse user service requirements and behavior logs; Intent sentiment analysis unit, used to input fusion features into the multi-task learning model to obtain intent categories and sentiment scores; The service action generation unit is used to generate service actions through reinforcement learning based on intent categories and sentiment scores.

9. The intelligent customer service system according to claim 8, characterized in that: The data processing unit includes a feature extraction module and a feature fusion module. The feature fusion module performs the following steps to achieve feature fusion: The FastDTW algorithm is used to align multimodal feature sequences; The aligned features are input into the gated fusion network to obtain the initial weights of the multimodal features; Dynamically modify the initial weights; The multimodal features are weightedly fused using the corrected weights.

Images