An intelligent customer service method that uses reinforcement learning to adaptively adjust interaction strategies

Abstract

The present invention discloses an intelligent customer service method for adaptively adjusting interaction strategies using reinforcement learning, comprising the following steps: S1, obtaining natural language text input by a user, generating a multidimensional state vector; S2, applying a real-time dynamic clustering algorithm to group users, and updating clustering results; S3, generating an action space of a specific state, and combining a rule-based strategy with a learning-based strategy to enrich the action space; S4, aggregating neighbor node features through a graph convolutional network, and extracting local features; S5, applying a graph attention network to weight neighbor node features, and dynamically assigning attention weights; S6, using a convoluted ladder-type deep exploration network to optimize interaction strategies and optimize action selection; S7, combining online learning with an experience playback mechanism, continuously updating interaction strategies, and responding quickly to dynamically changing user needs. The present invention combines reinforcement learning, graph neural networks, and sentiment analysis technology to achieve adaptive adjustment and optimization of intelligent customer service interaction strategies.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent customer service, and in particular to an intelligent customer service method for adaptively adjusting interaction strategies using reinforcement learning. Background Art

[0002] In the current intelligent customer service system, more and more companies are using artificial intelligence technology to improve the user interaction experience. These systems use natural language processing (NLP) technology to parse user input, identify their needs, and provide corresponding feedback based on preset rules or models. However, most traditional intelligent customer service systems rely on static predefined rules and models trained based on historical data, which makes it difficult to cope with dynamically changing user needs, especially when dealing with diverse scenarios with different user emotional states and behavior patterns. Their responses often lack flexibility and personalization. This limitation causes the system to easily fall into a fixed mode when facing complex interaction scenarios, and it is unable to continuously optimize the user experience, thereby reducing user satisfaction.

[0003] Most existing intelligent customer service systems manage conversations based on a single user behavior or input data, for example, parsing and responding to user questions based on keyword matching, intent recognition, or simple context processing. Although this method can effectively cope with simple interaction scenarios, when the user's emotional state, behavior pattern, and conversation background are more complex, traditional static models are difficult to dynamically adjust strategies. For example, when the user's emotions fluctuate greatly, the system still mechanically answers according to the preset rules, and cannot effectively capture the user's emotional changes and make corresponding adjustments, which may cause user dissatisfaction. In addition, many existing systems perform poorly in the context management of multi-round conversations, and are prone to dialogue state breaks. Especially during long-term interactions, the system cannot reasonably maintain the consistency of the context, resulting in a poor user experience.

[0004] Although reinforcement learning has been gradually applied to the field of intelligent customer service in recent years, optimizing the decision-making process through feedback-based learning mechanisms, existing reinforcement learning models are often based only on immediate feedback, ignoring the long-term experience and emotional fluctuations of users. When users interact with customer service systems, their emotional state and needs may change as the conversation progresses. Therefore, reinforcement learning models that rely solely on immediate feedback are difficult to accurately judge the emotional changes of users in long-term interactions. Furthermore, existing reinforcement learning algorithms often only select in a smaller fixed strategy space and cannot dynamically expand or generate new strategies, resulting in insufficient strategy richness and flexibility when facing personalized user needs. Traditional reinforcement learning strategies focus more on short-term benefits and ignore the improvement of user satisfaction over a long period of time.

[0005] In addition, existing intelligent customer service systems lack dynamic adaptability when dealing with multi-user scenarios. Generally speaking, traditional systems preset several fixed strategies for different user groups, but this approach is difficult to adapt to the rapid changes and personalized characteristics of user needs. In particular, when user behavior and emotional state change, the system cannot adjust strategies in real time and cannot meet the immediate needs of users. For example, when the same user interacts with the system in different emotional states, the traditional system may still provide the same response and cannot provide emotionally sensitive personalized services. This leads to the rigidity of the system and cannot effectively improve user satisfaction and loyalty.

[0006] With the development of technology, although advanced algorithms such as Graph Neural Networks (GNNs) have been introduced to improve the interaction modeling capabilities between users and systems, most applications only stay at the static modeling of users and customer service systems, lacking the capture and utilization of dynamic interaction features. Existing Graph Convolutional Networks (GCNs) can effectively extract the relationship features between users and systems, but in practical applications, user status and system status are constantly changing, and existing models cannot dynamically track these changes and respond quickly. In addition, most GCNs models use a uniform neighbor aggregation method and cannot perform weighted processing on specific neighbor nodes under different user states, which limits the flexibility and adaptability of the model in multi-level feature aggregation.

[0007] In order to meet these challenges, some graph neural networks based on attention mechanisms (such as graph attention networks, GATs) have emerged in recent years. They improve the model's feature extraction capabilities by assigning different weights to different neighbor nodes, especially in complex user states and behavior patterns. However, when faced with dynamic interaction scenarios, existing graph attention networks still have the problem of insufficient adaptation to user emotional changes and behavioral fluctuations, and are unable to respond to emotional changes in a timely manner. In addition, these models lack deep integration with reinforcement learning and cannot achieve long-term dynamic optimization of user feedback. This results in the system still having shortcomings in improving user satisfaction, and it is difficult to maintain a high level of interaction quality in multiple rounds of dialogue.

[0008] Therefore, how to provide an intelligent customer service method that uses reinforcement learning to adaptively adjust interaction strategies is an urgent problem that technicians in this field need to solve. Summary of the invention

[0009] One purpose of the present invention is to propose an intelligent customer service method that uses reinforcement learning to adaptively adjust the interaction strategy. The present invention combines advanced technologies such as reinforcement learning, graph neural network (GCN) and sentiment analysis, and accurately captures user status, emotional fluctuations and system interaction relationships through real-time dynamic clustering, graph convolutional network and graph attention network, and dynamically optimizes the interaction strategy of intelligent customer service. The system performs multi-level strategy optimization through a convoluted ladder deep exploration network, and combines immediate and delayed reward mechanisms to balance short-term and long-term benefits, ensuring that the system provides flexible and personalized responses in complex scenarios, significantly improving user satisfaction and interactive experience, and has high adaptability, flexibility and continuous optimization capabilities.

[0010] According to an embodiment of the present invention, an intelligent customer service method for adaptively adjusting an interaction strategy using reinforcement learning includes the following steps:

[0011] S1. Obtain the natural language text input by the user and perform preprocessing to generate a structured multi-dimensional state vector, including the user's text information, sentiment analysis results, historical conversation context, and user attributes;

[0012] S2. Based on the generated multi-dimensional state vector, a real-time dynamic clustering algorithm is applied to group users. Users of the same type share effective strategies, and the clustering results are updated as user behaviors and emotions change.

[0013] S3. Generate an action space for a specific state based on the user's dynamic clustering results. The action space includes information provision, proactive questioning, emotional response, and transfer to human customer service. Combine rule-based strategies with learning-based strategies to enrich the action space.

[0014] S4, aggregate neighbor node features on the dynamic graph constructed by user status, customer service actions and system status through graph convolutional network, and extract local features;

[0015] S5. Based on the local features extracted by the graph convolutional network, the graph attention network is used to weight the features of neighboring nodes, and the attention weights are dynamically allocated according to the correlation between the user state and the action selection;

[0016] S6, using a convoluted ladder deep exploration network to optimize interaction strategies, combining user feedback information, including immediate rewards and delayed rewards, to optimize action selection;

[0017] S7. Combining online learning with the experience replay mechanism, the system continuously updates strategies in real-time interactions, stores high-quality interaction samples in the experience replay pool for replay training, and responds quickly to dynamically changing user needs.

[0018] Optionally, the S2 specifically includes:

[0019] S21. Generate a user feature vector set based on the natural language text input by the user and the multi-dimensional state vector generated by preprocessing:

[0020] ;

[0021] in, represents the user feature vector at the tth conversation time, represents the i-th dimension in the user feature, and n represents the number of feature dimensions;

[0022] S22. Based on the generated user feature vectors, the system uses an improved density clustering algorithm to dynamically group users in real time. The improved density clustering algorithm uses a hybrid distance metric to measure the similarity between users. The hybrid distance is composed of a weighted sum of the Euclidean distance and the Mahalanobis distance:

[0023] ;

[0024] in, and They represent the feature vectors of the i-th user and the j-th user respectively, S represents the covariance matrix of the feature vector, and They represent the weighting coefficients of Euclidean distance and Mahalanobis distance respectively, m represents the dimension of the feature vector, represents a hybrid similarity measure between two users;

[0025] S23, the system is based on a specific neighborhood radius And the minimum number of neighborhood points MinPts to cluster users. If a user There are at least MinPts users in the domain, and the mixed distances with MinPts users all satisfy , then the user as the core point, and all users in the neighborhood are classified into the same cluster;

[0026] S24. The system introduces user sentiment analysis results to cluster radius Perform adaptive adjustments, detect the user's emotional state in real time based on the sentiment analysis model, and calculate the sentiment score:

[0027] ;

[0028] in, Indicates user The sentiment score, represents the output of the sentiment analysis model, represents the emotion change amplification coefficient, ReLU represents the activation function, and k represents the number of emotion dimensions. Indicates user and the weight coefficient of sentiment dimension j, represents the emotional temperature parameter, Represents the attention mechanism for emotional features Weighting of:

[0029] ;

[0030] in, represents a specific feature sentiment representation vector, represents the attention weight matrix;

[0031] S25. The system dynamically adjusts the clustering radius according to the user's emotional fluctuations and time dynamics. The adjusted clustering radius for:

[0032] ;

[0033] in, represents the initial clustering radius, Indicates the influence coefficient of emotional fluctuation on radius adjustment, Indicates user The emotional change value, represents the time attenuation coefficient, represents the adjustment parameter, N represents the total number of users, Indicates user The weight in the group, Indicates user Time of emotional change, Indicates user the time interval between emotional changes;

[0034] S26. When the user's emotions fluctuate Exceeding the global dynamic emotion change threshold , the system triggers re-clustering and re-evaluates user groups:

[0035] ;

[0036] in, represents the attenuation coefficient of emotional change;

[0037] S27. Each clustering result is linked with the strategy sharing system. Users in the same cluster share the interaction strategy generated based on the group. The interaction strategy generation process combines the behavior patterns and emotional changes of the current user group, and periodically recalculates the feature vectors of all users.

[0038] Optionally, the S3 specifically includes:

[0039] S31. Based on the dynamic clustering results of users, the system generates an action space of a specific state for each user. The action space includes information provision, active questioning, emotional response, and transfer to manual customer service. The generation of the action space is carried out through dynamic embedding and multi-dimensional combination:

[0040] ;

[0041] in, Indicates user The dynamic space, represents the nth possible action, Indicates user The cluster embedding vector of Indicates user A multidimensional vector of emotional states, Indicates user contextual features of the current conversation, Represents the combination of multi-dimensional embeddings to generate personalized action spaces;

[0042] S32. Based on the user's current status , the system generates a dynamic priority for each action in a specific state and calculates the dynamic priority:

[0043] ;

[0044] in, Indicates action The priority in the user's current state, Indicates action The weight matrix of Indicates action The bias term, represents the weight of the interaction factor, Indicates action Influence factors obtained through historical interactions during interaction with users:

[0045] ;

[0046] in, Indicates the user at the tth interaction emotional changes, Indicates action At the tth interaction with the user The correlation of, T represents the total number of interactions;

[0047] S33. The system optimizes the final selection of actions by combining the rule-based strategy and the learning-based strategy:

[0048] ;

[0049] in, represents the action selection probability under the rule-based strategy, represents the weight of the rule-based strategy, Indicates user Rules-based scoring of actions, represents the correction coefficient based on sentiment analysis, It indicates the degree of deviation between the action and the user's current state, and Z indicates the normalization coefficient;

[0050] S34, the learning base strategy is optimized through the reinforcement learning model, taking into account the user's long-term feedback and immediate feedback:

[0051] ;

[0052] in, represents the probability of action selection under the learning base strategy, , and represents the weight coefficients of different learning strategies, Indicates immediate feedback of the current action, represents the long-term return value, Represents the weight of sentiment drive:

[0053] ;

[0054] in, Indicates user The sentiment score of is based on the output of the current sentiment analysis model. Indicates action A history of emotional feedback, and Represents the weight parameter for regulating emotion and action feedback;

[0055] S35. The system combines the rule-based strategy and the learning-based strategy to generate the final action selection strategy. The hybrid strategy is generated by the weighted formula:

[0056] ;

[0057] in, represents the final action selection strategy, , and represents the weighting parameter, Represents the priority difference between the rule-based policy and the learning-based policy.

[0058] Optionally, the S4 specifically includes:

[0059] S41. The system processes the dynamic graph constructed by the user status, customer service action and system status through the graph convolution network. The nodes of the dynamic graph represent the user status, customer service action or system status, and the edges represent the relationship between them. The constructed dynamic graph is defined as , where V represents the node set and E represents the edge set;

[0060] S42. On the constructed dynamic graph, the system performs feature aggregation on each node through the graph convolutional network, and aggregates the feature information of neighboring nodes. The calculation formula for feature aggregation is:

[0061] ;

[0062] in, Indicates that node i is in The feature vector of the layer, N(i) represents the set of neighbor nodes of node i, represents the normalization factor between node i and its neighbor node j, Indicates The weight matrix of the layer, Indicates that node i is in The feature vector of the layer, represents a nonlinear activation function, Indicates The bias term of the layer;

[0063] S43, when the system performs multi-layer graph convolution operations, it extracts local features of the dynamic graph by aggregating features of neighboring nodes multiple times, and extracts higher-level graph structure information layer by layer;

[0064] S44. Finally, local feature information is extracted to generate complex interaction features between user status, customer service actions and system status.

[0065] Optionally, the S5 specifically includes:

[0066] S51. Based on the local features extracted by the graph convolutional network, the system uses the graph attention network to dynamically weight the features of each node. The system calculates the attention coefficient of the neighboring nodes based on the complex relationship between the user state and the action node:

[0067] ;

[0068] in, Represents the node in the kth attention head For neighbor nodes The attention weight, exp represents the exponential function, represents the nonlinear activation function in the kth attention head, represents the weight matrix in the kth attention head, represents the weight matrix of the kth attention head, N(i) represents the node The set of neighbor nodes of Representation Node The characteristic vector of Representation Node The characteristic vector of Representation Node The characteristic vector of Representation Node and Feature concatenation operation;

[0069] S52, the system calculates the attention weight , perform multi-head feature weighted aggregation on each node to generate the aggregated node feature representation:

[0070] ;

[0071] in, Represents the node in the kth attention head The aggregated feature vector, Representation Node and nodes The interaction dependency factor in the kth attention head:

[0072] ;

[0073] in, Indicates user and customer actions The emotional fluctuation value in the tth interaction, Representation Node and nodes The strength of the relationship in the tth interaction, T represents the total number of interactions, represents the smoothing factor in the kth attention head;

[0074] S53. After all attention heads are aggregated, the system generates the final node feature representation by fusing the features of multiple attention heads:

[0075] ;

[0076] in, Representation Node The final feature representation of , K represents the number of attention heads in the multi-head attention mechanism, represents the feature fusion weight of the kth attention head, represents the global nonlinear activation function;

[0077] S54. Based on the final feature representation of each node, the system calculates the selection priority of each action:

[0078] ;

[0079] in, Representation Node Features Select actions based on The priority of A learnable weight vector representing action selection, Indicates action The bias term.

[0080] Optionally, the S6 specifically includes:

[0081] S61, the system adopts a spiral staircase deep exploration network to optimize the interaction strategy. The spiral staircase deep exploration network combines user feedback information to select and optimize actions. The system is based on the current state Generate Action Expected Return ;

[0082] S62. The system dynamically adjusts its strategy through the "deep exploration" mechanism, adopts a variety of exploration strategies under different conditions, and introduces the exploration factor ,pass - Greedy strategy for action selection:

[0083] ;

[0084] in, Indicates in status Next select action The probability of Indicates in status Choose the action with the largest expected reward;

[0085] S63. During the exploration process, the system generates a multi-level strategy optimization path through a spiral ladder structure. Each layer corresponds to a different depth of exploration range, gradually approaching the optimal solution. The strategy optimization update formula of the kth layer is:

[0086] ;

[0087] in, represents the strategic return value of the kth layer, represents the balance coefficient between levels, represents the strategic return value of the k-1th layer, Indicates immediate reward, Indicates in status The maximum benefit among the optional actions is represents the discount factor;

[0088] S64. The system combines the delayed reward mechanism to optimize strategy selection in the long-term interaction process and calculate the delayed reward value:

[0089] ;

[0090] in, represents the delayed reward value, represents the time decay factor, Time represents the calculation period of delayed reward, represents the long-term reward weight;

[0091] S65. The system introduces a weighted feedback mechanism during the optimization process, which guides the strategy update through the user's immediate feedback and delayed feedback:

[0092] ;

[0093] in, Indicates immediate reward, and Represent the weighted coefficients of immediate reward and delayed reward respectively, satisfying ;

[0094] S66. The system gradually optimizes strategy selection through a deep exploration mechanism of a multi-level spiral ladder structure, combined with immediate feedback and delayed feedback.

[0095] The beneficial effects of the present invention are:

[0096] First of all, the present invention can dynamically respond to the complex emotional states and behavior patterns of users, and realize personalized interactive response. By extracting the text information, sentiment analysis results and historical conversation context input by users through natural language processing technology, the system can generate a structured multidimensional state vector and perform real-time dynamic clustering based on this information. In this way, the system can not only group users, but also automatically adjust the grouping strategy based on real-time emotional and behavioral changes, ensuring that the personalized needs of users can be met in a timely manner. This combination of real-time dynamic clustering and sentiment analysis greatly improves the system's performance in user emotional sensitivity, allowing the system to respond more appropriately when the user's emotions fluctuate greatly, effectively improving user satisfaction and interactive experience.

[0097] Secondly, the present invention greatly enhances the system's feature extraction and complex relationship modeling capabilities by combining graph convolutional networks (GCN) and graph attention networks (GAT). By modeling user status, customer service actions, and system status in a dynamic graph, the system can effectively aggregate neighbor node features in the graph convolutional network, capture the complex interactive relationship between the user and the system, and dynamically weight the relationship features between different nodes through the graph attention network. Unlike existing static models, the graph attention network can dynamically assign attention weights based on the correlation between user status and action selection, thereby ensuring more accurate feature extraction in complex interactive scenarios. This flexible feature aggregation mechanism allows the system to make dynamic weightings based on different interactive relationships, ensuring that key features receive higher weights, greatly improving the accuracy of strategy selection.

[0098] Furthermore, the present invention effectively solves the limitation problem of the existing system in the strategy space by introducing the Spiral Ladder Deep Exploration Network. This network structure enables the system to continuously expand the exploration scope in multi-level strategy optimization, and gradually approach the optimal strategy through the combination of deep exploration and multi-level optimization paths. This design breaks through the limitation that traditional reinforcement learning can only select actions in a fixed strategy space, ensuring that the system can flexibly generate new interaction strategies when facing complex and changeable user needs, greatly improving the system's response flexibility and adaptability. At the same time, the system combines instant feedback with delayed reward mechanisms to balance short-term and long-term benefits, so that the system can not only improve user satisfaction in the present, but also improve the overall user experience through long-term optimization strategies.

[0099] In addition, the system ensures continuous optimization of strategies in real-time interactions through online learning and experience replay mechanisms. The experience replay pool stores high-quality interaction samples, and the system can regularly use these samples for replay training to learn from historical data. This mechanism effectively avoids the occurrence of system forgetting, ensuring that the system can continue to learn from past successes or failures and optimize strategy selection. At the same time, the introduction of the online learning mechanism gives the system the ability to quickly adapt to new data and new behavior patterns, allowing the system to quickly respond to changes in user behavior, further improving its flexibility and real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0101] Figure 1 This is a flow chart of an intelligent customer service method for adaptively adjusting interaction strategies using reinforcement learning proposed by the present invention;

[0102] Figure 2 This is a real-time dynamic clustering flow chart of an intelligent customer service method that utilizes reinforcement learning to adaptively adjust interaction strategies proposed by the present invention. DETAILED DESCRIPTION

[0103] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0104] refer to Figure 1 and Figure 2 , an intelligent customer service method for adaptively adjusting interaction strategies using reinforcement learning, comprising the following steps:

[0105] S1. Obtain the natural language text input by the user and perform preprocessing to generate a structured multi-dimensional state vector, including the user's text information, sentiment analysis results, historical conversation context, and user attributes;

[0106] S2. Based on the generated multi-dimensional state vector, a real-time dynamic clustering algorithm is applied to group users. Users of the same type share effective strategies, and the clustering results are updated as user behaviors and emotions change.

[0107] S3. Generate an action space for a specific state based on the user's dynamic clustering results. The action space includes information provision, proactive questioning, emotional response, and transfer to human customer service. Combine rule-based strategies with learning-based strategies to enrich the action space.

[0108] S4, aggregate neighbor node features on the dynamic graph constructed by user status, customer service actions and system status through graph convolutional network, and extract local features;

[0109] S5. Based on the local features extracted by the graph convolutional network, the graph attention network is used to weight the features of neighboring nodes, and the attention weights are dynamically allocated according to the correlation between the user state and the action selection;

[0110] S6, using a convoluted ladder deep exploration network to optimize interaction strategies, combining user feedback information, including immediate rewards and delayed rewards, to optimize action selection;

[0111] S7. Combining online learning with the experience replay mechanism, the system continuously updates strategies in real-time interactions, stores high-quality interaction samples in the experience replay pool for replay training, and responds quickly to dynamically changing user needs.

[0112] In this implementation, S2 specifically includes:

[0113] S21. Generate a user feature vector set based on the natural language text input by the user and the multi-dimensional state vector generated by preprocessing:

[0114] ;

[0115] in, represents the user feature vector at the tth conversation time, represents the i-th dimension in the user feature, and n represents the number of feature dimensions;

[0116] S22. Based on the generated user feature vectors, the system uses an improved density clustering algorithm to dynamically group users in real time. The improved density clustering algorithm uses a hybrid distance metric to measure the similarity between users. The hybrid distance is composed of a weighted sum of the Euclidean distance and the Mahalanobis distance:

[0117] ;

[0118] in, and They represent the feature vectors of the i-th user and the j-th user respectively, S represents the covariance matrix of the feature vector, and They represent the weighting coefficients of Euclidean distance and Mahalanobis distance respectively, m represents the dimension of the feature vector, represents a hybrid similarity measure between two users;

[0119] S23, the system is based on a specific neighborhood radius And the minimum number of neighborhood points MinPts to cluster users. If a user There are at least MinPts users in the domain, and the mixed distances with MinPts users all satisfy , then the user as the core point, and all users in the neighborhood are classified into the same cluster;

[0120] S24. The system introduces user sentiment analysis results to cluster radius Perform adaptive adjustments, detect the user's emotional state in real time based on the sentiment analysis model, and calculate the sentiment score:

[0121] ;

[0122] in, Indicates user The sentiment score, represents the output of the sentiment analysis model, represents the emotion change amplification coefficient, ReLU represents the activation function, and k represents the number of emotion dimensions. Indicates user and the weight coefficient of sentiment dimension j, represents the emotional temperature parameter, Represents the attention mechanism for emotional features Weighting of:

[0123] ;

[0124] in, represents a specific feature sentiment representation vector, represents the attention weight matrix;

[0125] S25. The system dynamically adjusts the clustering radius according to the user's emotional fluctuations and time dynamics. The adjusted clustering radius for:

[0126] ;

[0127] in, represents the initial clustering radius, Indicates the influence coefficient of emotional fluctuation on radius adjustment, Indicates user The emotional change value, represents the time attenuation coefficient, represents the adjustment parameter, N represents the total number of users, Indicates user The weight in the group, Indicates user Time of emotional change, Indicates user the time interval between emotional changes;

[0128] S26. When the user's emotions fluctuate Exceeding the global dynamic emotion change threshold , the system triggers re-clustering and re-evaluates user groups:

[0129] ;

[0130] in, represents the attenuation coefficient of emotional change;

[0131] S27. Each clustering result is linked with the strategy sharing system. Users in the same cluster share the interaction strategy generated based on the group. The interaction strategy generation process combines the behavior patterns and emotional changes of the current user group, and periodically recalculates the feature vectors of all users.

[0132] In this implementation, S3 specifically includes:

[0133] S31. Based on the dynamic clustering results of users, the system generates an action space of a specific state for each user. The action space includes information provision, active questioning, emotional response, and transfer to manual customer service. The generation of the action space is carried out through dynamic embedding and multi-dimensional combination:

[0134] ;

[0135] in, Indicates user The dynamic space, represents the nth possible action, Indicates user The cluster embedding vector of Indicates user A multidimensional vector of emotional states, Indicates user contextual features of the current conversation, Represents the combination of multi-dimensional embeddings to generate personalized action spaces;

[0136] S32. Based on the user's current status , the system generates a dynamic priority for each action in a specific state and calculates the dynamic priority:

[0137] ;

[0138] in, Indicates action The priority in the user's current state, Indicates action The weight matrix of Indicates action The bias term, represents the weight of the interaction factor, Indicates action Influence factors obtained through historical interactions during interaction with users:

[0139] ;

[0140] in, Indicates the user at the tth interaction emotional changes, Indicates action At the tth interaction with the user The correlation of, T represents the total number of interactions;

[0141] S33. The system optimizes the final selection of actions by combining the rule-based strategy and the learning-based strategy:

[0142] ;

[0143] in, represents the action selection probability under the rule-based strategy, represents the weight of the rule-based strategy, Indicates user Rules-based scoring of actions, represents the correction coefficient based on sentiment analysis, It indicates the degree of deviation between the action and the user's current state, and Z indicates the normalization coefficient;

[0144] S34, the learning base strategy is optimized through the reinforcement learning model, taking into account the user's long-term feedback and immediate feedback:

[0145] ;

[0146] in, represents the probability of action selection under the learning base strategy, , and represents the weight coefficients of different learning strategies, Indicates immediate feedback of the current action, represents the long-term return value, Represents the weight of sentiment drive:

[0147] ;

[0148] in, Indicates user The sentiment score of is based on the output of the current sentiment analysis model. Indicates action A history of emotional feedback, and Represents the weight parameter for regulating emotion and action feedback;

[0149] S35. The system combines the rule-based strategy and the learning-based strategy to generate the final action selection strategy. The hybrid strategy is generated by the weighted formula:

[0150] ;

[0151] in, represents the final action selection strategy, , and represents the weighting parameter, Represents the priority difference between the rule-based policy and the learning-based policy.

[0152] In this implementation manner, the S4 specifically includes:

[0153] S41. The system processes the dynamic graph constructed by the user status, customer service action and system status through the graph convolution network. The nodes of the dynamic graph represent the user status, customer service action or system status, and the edges represent the relationship between them. The constructed dynamic graph is defined as , where V represents the node set and E represents the edge set;

[0154] S42. On the constructed dynamic graph, the system performs feature aggregation on each node through the graph convolutional network, and aggregates the feature information of neighboring nodes. The calculation formula for feature aggregation is:

[0155] ;

[0156] in, Indicates that node i is in The feature vector of the layer, N(i) represents the set of neighbor nodes of node i, represents the normalization factor between node i and its neighbor node j, Indicates The weight matrix of the layer, Indicates that node i is in The feature vector of the layer, represents a nonlinear activation function, Indicates The bias term of the layer;

[0157] S43, when the system performs multi-layer graph convolution operations, it extracts local features of the dynamic graph by aggregating features of neighboring nodes multiple times, and extracts higher-level graph structure information layer by layer;

[0158] S44. Finally, local feature information is extracted to generate complex interaction features between user status, customer service actions and system status.

[0159] In this implementation manner, S5 specifically includes:

[0160] S51. Based on the local features extracted by the graph convolutional network, the system uses the graph attention network to dynamically weight the features of each node. The system calculates the attention coefficient of the neighboring nodes based on the complex relationship between the user state and the action node:

[0161] ;

[0162] in, Represents the node in the kth attention head For neighbor nodes The attention weight, exp represents the exponential function, represents the nonlinear activation function in the kth attention head, represents the weight matrix in the kth attention head, represents the weight matrix of the kth attention head, N(i) represents the node The set of neighbor nodes of Representation Node The characteristic vector of Representation Node The characteristic vector of Representation Node The characteristic vector of Representation Node and Feature concatenation operation;

[0163] S52, the system calculates the attention weight , perform multi-head feature weighted aggregation on each node to generate the aggregated node feature representation:

[0164] ;

[0165] in, Represents the node in the kth attention head The aggregated feature vector, Representation Node and nodes The interaction dependency factor in the kth attention head:

[0166] ;

[0167] in, Indicates user and customer actions The emotional fluctuation value in the tth interaction, Representation Node and nodes The strength of the relationship in the tth interaction, T represents the total number of interactions, represents the smoothing factor in the kth attention head;

[0168] S53. After all attention heads are aggregated, the system generates the final node feature representation by fusing the features of multiple attention heads:

[0169] ;

[0170] in, Representation Node The final feature representation of , K represents the number of attention heads in the multi-head attention mechanism, represents the feature fusion weight of the kth attention head, represents the global nonlinear activation function;

[0171] S54. Based on the final feature representation of each node, the system calculates the selection priority of each action:

[0172] ;

[0173] in, Representation Node Features Select actions based on The priority of A learnable weight vector representing action selection, Indicates action The bias term.

[0174] In this implementation manner, S6 specifically includes:

[0175] S61, the system adopts a spiral staircase deep exploration network to optimize the interaction strategy. The spiral staircase deep exploration network combines user feedback information to select and optimize actions. The system is based on the current state Generate Action Expected Return ;

[0176] S62. The system dynamically adjusts its strategy through the "deep exploration" mechanism, adopts a variety of exploration strategies under different conditions, and introduces the exploration factor ,pass - Greedy strategy for action selection:

[0177] ;

[0178] in, Indicates in status Next select action The probability of Indicates in status Choose the action with the largest expected reward;

[0179] S63. During the exploration process, the system generates a multi-level strategy optimization path through a spiral ladder structure. Each layer corresponds to a different depth of exploration range, gradually approaching the optimal solution. The strategy optimization update formula of the kth layer is:

[0180] ;

[0181] in, represents the strategic return value of the kth layer, represents the balance coefficient between levels, represents the strategic return value of the k-1th layer, Indicates immediate reward, Indicates in status The maximum benefit among the optional actions is represents the discount factor;

[0182] S64. The system combines the delayed reward mechanism to optimize strategy selection in the long-term interaction process and calculate the delayed reward value:

[0183] ;

[0184] in, represents the delayed reward value, represents the time decay factor, Time represents the calculation period of delayed reward, represents the long-term reward weight;

[0185] S65. The system introduces a weighted feedback mechanism during the optimization process, which guides the strategy update through the user's immediate feedback and delayed feedback:

[0186] ;

[0187] in, Indicates immediate reward, and Represent the weighted coefficients of immediate reward and delayed reward respectively, satisfying ;

[0188] S66. The system gradually optimizes strategy selection through a deep exploration mechanism of a multi-level spiral ladder structure, combined with immediate feedback and delayed feedback.

[0189] Embodiment 1:

[0190] In order to verify the feasibility of the present invention in implementation, the present invention is applied to the intelligent customer service system of a large e-commerce enterprise. The system needs to process tens of thousands of customer requests every day, covering various service types such as pre-sales consultation, after-sales service, order tracking, returns and exchanges, complaints, etc. Due to the large user group and complex and diverse user needs, the customer service system faces the following problems in actual operation: First, users have large emotional fluctuations. Some users are emotionally excited when encountering problems, and their emotional states change rapidly, while the existing system cannot effectively capture these emotional fluctuations and provide targeted services. Secondly, facing users with the same problems, the traditional customer service system lacks effective dynamic grouping and personalized responses, resulting in mechanical and repetitive problems in the service, reducing user satisfaction. Finally, due to the differences in the long-term behavior patterns of different users, it is difficult for the customer service system to optimize strategies in real time to cope with these diverse needs.

[0191] In order to improve this situation, the e-commerce company decided to adopt the intelligent customer service method based on reinforcement learning to adaptively adjust the interaction strategy proposed in this invention. Through the technical means of combining sentiment analysis, dynamic clustering and reinforcement learning, it can accurately capture user emotions, behaviors and historical interaction data, and dynamically adjust customer service strategies to improve user experience. The application scenarios are as follows:

[0192] At the beginning of the conversation between the user and the customer service system, the system first obtains the text information entered by the user, and pre-processes the text through natural language processing technology to extract the user's emotional state, conversation context, and historical interaction records. For example, when a user inquires about the return progress, the sentiment analysis model identifies the user's emotion as "anxiety", and words such as "urgent" and "when can I get my money back" frequently appear in his language expression. The system combines historical conversation data to generate a multi-dimensional state vector of the user, including text information, emotional state (anxiety), conversation context (inquiring about the return progress), and user attributes (VIP customer, two return records in the past three months).

[0193] Based on the generated multi-dimensional state vector, the system further uses an improved real-time dynamic clustering algorithm to group the user with other users with similar emotions and behavior patterns. This clustering process not only takes into account the user's text content and emotions, but also combines the user's historical behavior patterns and emotional fluctuations in the current conversation. For example, the system groups the user with other users who have recently returned goods and are currently anxious. Unlike traditional static grouping, the present invention uses real-time dynamic clustering to ensure that the system can adjust the grouping strategy as user behavior and emotions change. After clustering is completed, the system shares the best interaction strategy between these users to avoid repeated training and decision-making, thereby improving response speed.

[0194] After clustering is completed, the system generates a specific action space for the user, including "providing return progress information", "actively calming the user's emotions", "recommending to wait for processing results" or "transferring to manual customer service". The system assigns priorities to these actions based on the user's current emotional state and behavior pattern, and dynamically adjusts the weights of each action in combination with the graph convolutional network and the graph attention network. For this anxious user, the system's weighted priorities are: "actively calming emotions" (priority 0.8), "providing return progress" (priority 0.7), and "transferring to manual customer service" (priority 0.5). The system selects the action with the highest priority, that is, calming the user's emotions first, and the system automatically generates a reply: "Hello, we have received your return application and are currently processing it. Don't worry, we will process it as soon as possible and notify you." This reply can both relieve the user's anxiety and maintain the user's trust in the system.

[0195] After the system provides a response, it will immediately adjust and optimize the strategy based on the user's feedback. If the user replies "OK, thank you" and does not express further dissatisfaction, the system will record this feedback and add it to the reinforcement learning model as positive feedback to optimize subsequent decisions; but if the user replies "Why is it so slow?" and expresses further anxiety, the system will treat this feedback as negative feedback, and combine the user's new emotional state and conversation context to readjust the action priority, and may eventually choose to "transfer to manual customer service" to meet the user's immediate needs.

[0196] Table 1 Comparison of optimization effects of intelligent customer service system based on reinforcement learning

[0197] ;

[0198] Table 1 shows the changes in key indicators of the intelligent customer service system based on reinforcement learning three months before and after its application, and details the significant effects brought by the present invention. First, the user satisfaction score increased from 4.23 points before application to 4.57 points, which shows that the system has made great progress in optimizing emotional response and personalized services. Through dynamic clustering and real-time adjustment strategies, the system can better capture user emotional fluctuations and provide targeted services, significantly improving the overall user experience.

[0199] In terms of user complaint rate, the complaint rate per thousand requests after application dropped from 119.8 to 97.6, which shows that the system effectively reduced user complaints and dissatisfaction. By introducing the spiral ladder deep exploration network, the system can gradually optimize the strategy, thereby improving the long-term satisfaction of users and reducing complaints caused by improper response.

[0200] The transfer rate of manual customer service dropped from 34.7% before the application to 27.8%, which means that the intelligent customer service system has become more autonomous and efficient in solving user problems, reducing dependence on manual customer service. By combining immediate feedback with delayed feedback, the system can better handle complex user requests, avoid transferring users to manual customer service too early, and improve overall efficiency.

[0201] In terms of service efficiency improvement, the system has achieved a 12.3% improvement. This is due to the combination of reinforcement learning and experience replay mechanism. The system can continuously optimize strategies through historical data and share effective strategies based on the behavior patterns of different user groups, reducing the time cost of repeated learning and response.

[0202] The average response time was also shortened from 2.48 seconds to 1.54 seconds, further demonstrating the system's optimized response speed. Through the combination of sentiment analysis, graph convolutional networks, and graph attention networks, the system can understand and process user input more quickly, and dynamically adjust strategies based on real-time feedback, making responses more timely and accurate.

[0203] Finally, the daily request processing volume increased from approximately 49,560 to 55,870, indicating a significant improvement in the system's overall workload processing capacity. This data reflects the system's significant progress in improving processing efficiency and reducing user waiting time.

[0204] This embodiment demonstrates the application of the present invention in an intelligent customer service system. Through the combination of reinforcement learning, graph convolutional networks, graph attention networks, sentiment analysis and other technologies, the system can adaptively and dynamically adjust the interaction strategy, improve the response speed and accuracy, and significantly improve the user satisfaction and service experience.

[0205] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An intelligent customer service method for adaptively adjusting interaction strategies using reinforcement learning, characterized in that: The steps include: S1. Obtain the natural language text input by the user and perform preprocessing to generate a structured multi-dimensional state vector, including the user's text information, sentiment analysis results, historical conversation context, and user attributes; S2. Based on the generated multi-dimensional state vector, a real-time dynamic clustering algorithm is applied to group users. Users of the same type share effective strategies, and the clustering results are updated as user behaviors and emotions change. S3. Generate an action space for a specific state based on the user's dynamic clustering results. The action space includes information provision, proactive questioning, emotional response, and transfer to human customer service. Combine rule-based strategies with learning-based strategies to enrich the action space. S4, aggregate neighbor node features on the dynamic graph constructed by user status, customer service actions and system status through graph convolutional network, and extract local features; S5. Based on the local features extracted by the graph convolutional network, the graph attention network is used to weight the features of neighboring nodes, and the attention weights are dynamically allocated according to the correlation between the user state and the action selection; S6, using a convoluted ladder deep exploration network to optimize interaction strategies, combining user feedback information, including immediate rewards and delayed rewards, to optimize action selection; S7. Combining online learning and experience replay mechanism, the system continuously updates strategies in real-time interaction, stores high-quality interaction samples in the experience replay pool for replay training, and responds quickly to dynamically changing user needs; The S6 specifically includes: S61, the system adopts a spiral staircase deep exploration network to optimize the interaction strategy. The spiral staircase deep exploration network combines user feedback information to select and optimize actions. The system is based on the current state Generate Action Expected Return ; S62. The system dynamically adjusts its strategy through the "deep exploration" mechanism, adopts a variety of exploration strategies under different conditions, and introduces the exploration factor ,pass - Greedy strategy for action selection: ; in, Indicates in status Next select action The probability of Indicates in status Next, choose the action with the largest expected reward; S63. During the exploration process, the system generates a multi-level strategy optimization path through a spiral ladder structure. Each layer corresponds to a different depth of exploration range, gradually approaching the optimal solution. The strategy optimization update formula of the kth layer is: ; in, represents the strategic return value of the kth layer, represents the balance coefficient between levels, represents the strategic return value of the k-1th layer, Indicates immediate reward, Indicates in status The maximum benefit among the optional actions is represents the discount factor; S64. The system combines the delayed reward mechanism to optimize strategy selection in the long-term interaction process and calculate the delayed reward value: ； in, represents the delayed reward value, represents the time decay factor, Time represents the calculation period of delayed reward, represents the long-term reward weight; S65. The system introduces a weighted feedback mechanism during the optimization process, which guides the strategy update through the user's immediate feedback and delayed feedback: ； in, Indicates immediate reward, and Represent the weighted coefficients of immediate reward and delayed reward respectively, satisfying ; S66. The system gradually optimizes strategy selection through a deep exploration mechanism of a multi-level spiral ladder structure, combined with immediate feedback and delayed feedback.

2. According to claim 1, the intelligent customer service method for adaptively adjusting interaction strategies using reinforcement learning is characterized in that: The S2 specifically includes: S21. Generate a user feature vector set based on the natural language text input by the user and the multi-dimensional state vector generated by preprocessing: ; in, represents the user feature vector at the tth conversation time, represents the i-th dimension in the user feature, and n represents the number of feature dimensions; S22. Based on the generated user feature vectors, the system uses an improved density clustering algorithm to dynamically group users in real time. The improved density clustering algorithm uses a hybrid distance metric to measure the similarity between users. The hybrid distance is composed of a weighted sum of the Euclidean distance and the Mahalanobis distance: ; in, and They represent the feature vectors of the i-th user and the j-th user respectively, S represents the covariance matrix of the feature vector, and They represent the weighting coefficients of Euclidean distance and Mahalanobis distance respectively, m represents the dimension of the feature vector, represents a hybrid similarity measure between two users; S23, the system is based on a specific neighborhood radius And the minimum number of neighborhood points MinPts to cluster users. If a user There are at least MinPts users in the domain, and the mixed distances with MinPts users all satisfy , then the user as the core point, and all users in the neighborhood are classified into the same cluster; S24. The system introduces user sentiment analysis results to cluster radius Perform adaptive adjustments, detect the user's emotional state in real time based on the sentiment analysis model, and calculate the sentiment score: ; in, Indicates user The sentiment score, represents the output of the sentiment analysis model, represents the emotion change amplification coefficient, ReLU represents the activation function, and k represents the number of emotion dimensions. Indicates user and the weight coefficient of sentiment dimension j, represents the emotional temperature parameter, Represents the attention mechanism for emotional features Weighting of: ; in, represents a specific feature sentiment representation vector, represents the attention weight matrix; S25. The system dynamically adjusts the clustering radius according to the user's emotional fluctuations and time dynamics. The adjusted clustering radius for: ; in, represents the initial clustering radius, Indicates the influence coefficient of emotional fluctuation on radius adjustment, Indicates user The emotional change value, represents the time attenuation coefficient, represents the adjustment parameter, N represents the total number of users, Indicates user The weight in the group, Indicates user Time of emotional change, Indicates user the time interval between emotional changes; S26. When the user's emotions fluctuate Exceeding the global dynamic emotion change threshold , the system triggers re-clustering and re-evaluates user groups: ; in, represents the attenuation coefficient of emotional change; S27. Each clustering result is linked with the strategy sharing system. Users in the same cluster share the interaction strategy generated based on the group. The interaction strategy generation process combines the behavior patterns and emotional changes of the current user group, and periodically recalculates the feature vectors of all users.

3. According to claim 1, the intelligent customer service method for adaptively adjusting interaction strategies using reinforcement learning is characterized in that: The S3 specifically includes: S31. Based on the dynamic clustering results of users, the system generates an action space of a specific state for each user. The action space includes information provision, active questioning, emotional response, and transfer to manual customer service. The generation of the action space is carried out through dynamic embedding and multi-dimensional combination: ; in, Indicates user The dynamic space, represents the nth possible action, Indicates user The cluster embedding vector of Indicates user A multidimensional vector of emotional states, Indicates user contextual features of the current conversation, Represents the combination of multi-dimensional embeddings to generate personalized action spaces; S32. Based on the user's current status , the system generates a dynamic priority for each action in a specific state and calculates the dynamic priority: ; in, Indicates action The priority in the user's current state, Indicates action The weight matrix of Indicates action The bias term, represents the weight of the interaction factor, Indicates action Influence factors obtained through historical interactions during interaction with users: ； in, Indicates the user at the tth interaction emotional changes, Indicates action At the tth interaction with the user The correlation of, T represents the total number of interactions; S33. The system optimizes the final selection of actions by combining the rule-based strategy and the learning-based strategy: ; in, represents the action selection probability under the rule-based strategy, represents the weight of the rule-based strategy, Indicates user Action scoring based on rules, represents the correction coefficient based on sentiment analysis, It indicates the degree of deviation between the action and the user's current state, and Z indicates the normalization coefficient; S34, the learning base strategy is optimized through the reinforcement learning model, taking into account the user's long-term feedback and immediate feedback: ; in, represents the probability of action selection under the learning base strategy, , and represents the weight coefficient of different learning strategies, Indicates immediate feedback of the current action, represents the long-term return value, Represents the weight of sentiment drive: ; in, Indicates user The sentiment score of is based on the output of the current sentiment analysis model. Indicates action A history of emotional feedback, and Represents the weight parameter for regulating emotion and action feedback; S35. The system combines the rule-based strategy and the learning-based strategy to generate the final action selection strategy. The hybrid strategy is generated by the weighted formula: ; in, represents the final action selection strategy, , and represents the weighting parameter, Represents the priority difference between the rule-based policy and the learning-based policy.

4. According to claim 1, the intelligent customer service method for adaptively adjusting interaction strategies using reinforcement learning is characterized in that: The S4 specifically includes: S41. The system processes the dynamic graph constructed by the user status, customer service action and system status through the graph convolution network. The nodes of the dynamic graph represent the user status, customer service action or system status, and the edges represent the relationship between them. The constructed dynamic graph is defined as , where V represents the node set and E represents the edge set; S42. On the constructed dynamic graph, the system performs feature aggregation on each node through the graph convolutional network, and aggregates the feature information of neighboring nodes. The calculation formula for feature aggregation is: ; in, Indicates that node i is in The feature vector of the layer, N(i) represents the set of neighbor nodes of node i, represents the normalization factor between node i and its neighbor node j, Indicates The weight matrix of the layer, Indicates that node i is in The feature vector of the layer, represents a nonlinear activation function, Indicates The bias term of the layer; S43, when the system performs multi-layer graph convolution operations, it extracts local features of the dynamic graph by aggregating features of neighboring nodes multiple times, and extracts higher-level graph structure information layer by layer; S44. Finally, local feature information is extracted to generate complex interaction features between user status, customer service actions and system status.

5. According to claim 1, the intelligent customer service method for adaptively adjusting interaction strategies using reinforcement learning is characterized in that: The S5 specifically includes: S51. Based on the local features extracted by the graph convolutional network, the system uses the graph attention network to dynamically weight the features of each node. The system calculates the attention coefficient of the neighboring nodes based on the complex relationship between the user state and the action node: ; in, Represents the node in the kth attention head For neighbor nodes The attention weight, exp represents the exponential function, represents the nonlinear activation function in the kth attention head, represents the weight matrix in the kth attention head, represents the weight matrix of the kth attention head, N(i) represents the node The set of neighbor nodes of Representation Node The characteristic vector of Representation Node The characteristic vector of Representation Node The characteristic vector of Representation Node and Feature concatenation operation; S52, the system calculates the attention weight , perform multi-head feature weighted aggregation on each node to generate the aggregated node feature representation: ; in, Represents the node in the kth attention head The aggregated feature vector, Representation Node and nodes The interaction dependency factor in the kth attention head: ; in, Indicates user and customer actions The emotional fluctuation value in the tth interaction, Representation Node and nodes The strength of the relationship in the tth interaction, T represents the total number of interactions, represents the smoothing factor in the kth attention head; S53. After all attention heads are aggregated, the system generates the final node feature representation by fusing the features of multiple attention heads: ; in, Representation Node The final feature representation of , K represents the number of attention heads in the multi-head attention mechanism, represents the feature fusion weight of the kth attention head, represents the global nonlinear activation function; S54. Based on the final feature representation of each node, the system calculates the selection priority of each action: ; in, Representation Node Features Select actions based on The priority of A learnable weight vector representing action selection, Indicates action The bias term.

Images