Intelligent customer service optimization method and system based on reinforcement learning

Abstract

The invention relates to the technical field of natural language processing, and discloses an intelligent customer service optimization method and system based on reinforcement learning, and the method comprises the steps: obtaining a current dialogue between a user and an intelligent customer service and user experience, evaluating the problem difficulty degree of the current dialogue, and obtaining the problem difficulty degree of the current dialogue according to the problem difficulty degree of the current dialogue and a preset difficulty degree threshold value; dividing the current dialogue into simple questions and difficult questions, and constructing a reward mechanism based on user experience; for a simple problem, a service track is generated through intelligent customer service autonomous decision processing, and a decision model is updated by adopting a reinforcement learning strategy according to the service track and a reward mechanism; and processing the difficult problem by using the artificial customer service, recording the decision making process of the artificial customer service as an expert demonstration track, and updating the decision making model by adopting a mixed strategy combining imitation learning and reinforcement learning according to the expert demonstration track and a reward mechanism. A solution of manual customer service intervention is used as a demonstration track, so that the intelligent customer service performs imitation learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, in particular to an intelligent customer service optimization method and system based on reinforcement learning. BACKGROUND

[0002] With the rapid development of artificial intelligence technology in recent years, intelligent customer service systems have been widely used in e-commerce, finance, telecommunications and many other industries; traditional intelligent customer service mostly relies on predefined rule base or retrieval-based question and answer system, the interaction mode is relatively fixed and the service experience is relatively rigid, although such systems can efficiently handle a large number of repetitive simple consultations, but when facing user diversification, colloquial expression and problems that need to be solved through multiple interactions, it is often difficult to provide satisfactory and personalized services; in order to break through the limitations of traditional intelligent customer service, some researches introduce deep learning and reinforcement learning technology into the construction of dialogue system; reinforcement learning enables the agent to learn the optimal strategy according to the rewards or punishments obtained in the interaction with the environment, so that the intelligent customer service has the ability to optimize the long-term cumulative rewards through trial and error, thereby being able to handle more complex and task-oriented dialogue scenarios.

[0003] However, directly applying standard reinforcement learning algorithms to real intelligent customer service scenarios faces serious challenges; in real customer service dialogue, a successful interaction often needs to go through multiple rounds of communication, and whether the problem is solved can only be judged at the end of the dialogue, which means that the system cannot obtain a clear reward signal at most intermediate steps, resulting in extremely low learning efficiency of the model and making it difficult to learn effectively; imitation learning can quickly build a model with basic capabilities by directly learning dialogue data of human experts, this method based on supervised learning avoids the exploration risk of reinforcement learning, but the upper limit of the model is limited by the level of human experts in the data set, and the model cannot discover better strategies through self-learning, and what the model learns is often the imitation of surface behavior rather than deep decision logic, resulting in insufficient generalization ability.

[0004] In recent years, a hybrid method combining imitation learning and reinforcement learning has also been actively explored, but how to effectively and stably integrate the prior knowledge of human experts and the autonomous exploration ability of reinforcement learning is still a technical problem; simply putting expert data into the experience pool as high-quality experience may be overwhelmed by the vast amount of low-quality data generated by the model during exploration; and excessive reliance on expert data for pre-training may cause the model to fall into a local optimum and make it difficult to effectively explore in the subsequent reinforcement learning phase, how to design a mechanism that can efficiently absorb the valuable experience of human experts to solve problems while retaining the ability of autonomous learning and generalization is a key problem in the field of intelligent customer service technology. SUMMARY

[0005] In view of the above problems, the present application is proposed.

[0006] To solve the above technical problems, the present application provides the following technical solutions: an intelligent customer service optimization method based on reinforcement learning, comprising:

[0007] Obtaining the current dialogue and user experience of the user and the intelligent customer service, evaluating the difficulty of the current dialogue, and dividing the current dialogue into simple questions and difficult questions according to the difficulty of the current dialogue and a preset difficulty threshold, and constructing a reward mechanism based on the user experience;

[0008] For the simple questions, using the intelligent customer service to autonomously decide and process to generate a service track, and updating the decision model according to the service track and the reward mechanism by using a reinforcement learning strategy;

[0009] Using the artificial customer service to process the difficult questions and recording the decision process of the artificial customer service as an expert demonstration track, and updating the decision model according to the expert demonstration track and the reward mechanism by using a hybrid strategy combining imitation learning and reinforcement learning.

[0010] As a preferred scheme of the intelligent customer service optimization method based on reinforcement learning, wherein: the evaluation of the difficulty of the current dialogue comprises obtaining a complete state sequence from the start of the dialogue to the current time in the current dialogue;

[0011] Processing the complete state sequence through a difficulty detection module, and outputting a continuous difficulty score representing the complexity of the current dialogue as the difficulty of the current dialogue.

[0012] As a preferred scheme of the intelligent customer service optimization method based on reinforcement learning, wherein: dividing the current dialogue into simple questions and difficult questions according to the difficulty of the current dialogue and a preset difficulty threshold comprises comparing the continuous difficulty score of the current dialogue with the preset difficulty threshold, and when the continuous difficulty score is less than or equal to the preset difficulty threshold, marking the current dialogue as a simple question processed by the intelligent customer service autonomously and assigning it to an autonomous decision path;

[0013] When the continuous difficulty score is greater than the preset difficulty threshold, marking the current dialogue as a difficult question transferred to the artificial customer service for processing and assigning it to an expert decision path.

[0014] As a preferred scheme of the intelligent customer service optimization method based on reinforcement learning, wherein: constructing a reward mechanism based on the user experience comprises obtaining a direct score of the user for the dialogue service as an explicit feedback score;

[0015] Perform sentiment analysis on user-input text to obtain a sentiment score;

[0016] A satisfaction score is constructed by fusing the explicit feedback score and the sentiment analysis score.

[0017] Calculate the trend of the satisfaction rating during the dialogue process to obtain the change in satisfaction level.

[0018] The satisfaction score and the change in satisfaction are weighted and fused to generate a real-time reward function.

[0019] As a preferred embodiment of the intelligent customer service optimization method based on reinforcement learning described in this invention, the method of updating the decision model using a reinforcement learning strategy according to the service trajectory and the reward mechanism includes calculating the advantage function value of each state-action pair based on the state-action pairs in the service trajectory and the reward mechanism.

[0020] Construct the policy gradient objective function using the aforementioned advantage function value;

[0021] The policy network parameters of the decision model are updated by optimizing the policy gradient objective function.

[0022] As a preferred embodiment of the intelligent customer service optimization method based on reinforcement learning described in this invention, the following is an embodiment: updating the decision model using a hybrid strategy combining imitation learning and reinforcement learning based on the expert demonstration trajectory and the reward mechanism includes calculating the behavioral cloning loss between the model strategy and the expert decision based on the expert demonstration trajectory.

[0023] The reinforcement learning loss is calculated based on the state-action pairs in the expert demonstration trajectory and the reward mechanism.

[0024] The behavior cloning loss and the reinforcement learning loss are weighted and combined to construct a hybrid loss function, and the decision model is updated based on the hybrid loss function.

[0025] As a preferred embodiment of the intelligent customer service optimization method based on reinforcement learning described in this invention, the behavioral cloning loss between the expert demonstration trajectory calculation model strategy and expert decision includes extracting the decision action sequence executed by the human customer service representative from the expert demonstration trajectory.

[0026] Calculate the cross-entropy loss between the policy output of the decision model in the corresponding state and the decision action sequence;

[0027] A regularization term is introduced to prevent the decision model from overfitting to the expert demonstration trajectory. The regularization term is obtained by calculating the distribution distance between the current model policy and the initial pre-trained policy.

[0028] An intelligent customer service optimization system based on reinforcement learning, wherein:

[0029] A dialogue evaluation module obtains a current dialogue of a user with an intelligent customer service and a user experience, evaluates a difficulty of a question in the current dialogue, and divides the current dialogue into a simple question and a difficult question according to the difficulty of the question in the current dialogue and a preset difficulty threshold, and constructs a reward mechanism based on the user experience;

[0030] An intelligent processing module generates a service track by using the intelligent customer service to autonomously decide processing for the simple question, and updates a decision model according to the service track and the reward mechanism by using a reinforcement learning strategy;

[0031] An artificial processing module processes the difficult question by using an artificial customer service and records a decision process of the artificial customer service as an expert demonstration track, and updates the decision model according to the expert demonstration track and the reward mechanism by using a hybrid strategy combining imitation learning and reinforcement learning.

[0032] A computer device, comprising a memory and a processor; the memory stores a computer program, wherein the processor implements the steps of the method of any one of the present application when executing the computer program.

[0033] A computer readable storage medium, having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method of any one of the present application.

[0034] The present application has the following advantages: the intelligent customer service optimization method based on reinforcement learning provided by the present application solves the problem that the experience of an artificial customer service is difficult to apply to an intelligent customer service by using a solution of the artificial customer service intervention as a demonstration track to enable the intelligent customer service to learn from the track of human intervention, thereby improving the generalization ability of the model and the ability to quickly adapt to new problems and new solutions. The difficulty detection module actively predicts the difficulty of dialogue processing, thereby avoiding the problem of a decline in user experience caused by passive response of a traditional intelligent customer service, and achieving accurate shunting and differentiated processing of questions. The double-path processing mechanism allocates simple questions to an autonomous decision path for efficient processing and transfers difficult questions to an expert decision path for professional processing, thereby ensuring processing efficiency and service quality. The hybrid strategy combining imitation learning and reinforcement learning ensures that the model can quickly absorb expert experience, retains the ability of autonomous exploration and strategy optimization, and prevents overfitting of the expert track by introducing a regularization constraint term, thereby maintaining the generalization ability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0036] Fig. 1 The overall flowchart of the intelligent customer service optimization method based on reinforcement learning provided for the first embodiment of the present application.

[0037] Fig. 2 The difficulty detection flowchart provided for the first embodiment of the present application.

[0038] Fig. 3 The double-path model updating flowchart provided for the first embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0040] Embodiment 1, refer to Figs. 1-3 For an embodiment of the present application, an intelligent customer service optimization method based on reinforcement learning is provided, comprising:

[0041] S1: obtaining the current conversation of the user and the intelligent customer service and the user experience, evaluating the difficulty of the current conversation, and dividing the current conversation into simple questions and difficult questions through the difficulty threshold of the current conversation, and constructing a reward mechanism based on the user experience.

[0042] The traditional intelligent customer service is usually passive when facing user problems, and relies on dialogue failure or user negative feedback to update the model. This processing mechanism seriously affects the user experience and causes adverse effects on the company. Therefore, this embodiment abandons the idea that only problems are difficult problems, and introduces a real-time difficulty detection module D φ The difficulty detection module is located at the front end of the dialogue process, which is used to quantitatively evaluate the expected difficulty of the current conversation, and obtain the difficulty of the current conversation. As shown in Fig. 2 , first, the difficulty detection module D φ performs real-time difficulty detection at the beginning of the dialogue, and if the difficulty detection module D φIf the output is less than or equal to the preset difficulty threshold H, then autonomous decision-making is performed; if it is greater than the preset difficulty threshold H, then expert decision-making is performed. In both autonomous and expert decision-making, all decision trajectories and execution actions need to be retained.

[0043] Specifically, assessing the difficulty of the current dialogue problem includes obtaining the complete state sequence (s0, s1, ..., s2) from the start of the dialogue to the current moment. t The complete state sequence is processed by the difficulty detection module, which outputs a continuous difficulty score p representing the complexity of the current dialogue. difficult The difficulty level of the current dialogue issue is used as the basis for the evaluation process, which is carried out by the difficulty detection module D. φ Execution is performed by executing the complete state sequence (s0, s1, ..., s) from the start of the dialogue to the current moment. t Input difficulty detection module D φ Then, the continuous difficulty score p is output. difficult .

[0044] Continuous difficulty score p difficult The calculation formula is as follows:

[0045] p difficult =D φ (s0,s1,...,s t );

[0046] Where, p difficult The continuous difficulty score is used to represent the complexity of the current dialogue; D φ This is a deep learning-based difficulty detection module, where the difficulty score p is continuous. difficult The difficulty detection module D in the calculation formula φ Through training on a large amount of relevant data, representations of various latent features such as problem complexity were learned; (s0,s1,...,s t A sequence of dialogue states is present in the context of a single conversation, that is, a complete sequence of states from the start of the conversation to the current time t, used to capture the changing trends of the conversation.

[0047] It should be noted that when the user's current conversation with the intelligent customer service passes the aforementioned difficulty detection module D... φ Then, by comparing the obtained continuous difficulty scores with the preset difficulty threshold, the current dialogue continues to be executed along the following autonomous decision-making path or expert decision-making path. The trajectories completed by different paths are optimized in different ways in the subsequent reinforcement learning stage, so that the model learns a better solution.

[0048] In this embodiment, the deep learning-based difficulty detection module D φThe sequence modeling method based on deep neural network is adopted, and hierarchical feature extraction is performed through a multi-layer neural network architecture. First, text semantic understanding and feature encoding are performed at the bottom layer to extract basic features such as keywords, syntax structure, and sentiment tendency in the dialogue. Then, time sequence modeling is performed at the middle layer to capture the time evolution law and context dependency of the dialogue state sequence. Finally, complexity comprehensive evaluation is performed at the top layer to fuse multi-dimensional feature information and output a continuous difficulty score p difficult , which quantitatively represents the complexity of the current dialogue. A value close to 0 indicates that the question is relatively simple, and a value close to 1 indicates that the question is relatively complex. Through the hierarchical feature learning and evaluation mechanism, the difficulty detection module can accurately quantify and evaluate the expected processing difficulty of the current dialogue at the front end of the dialogue process.

[0049] In an optional embodiment, the difficulty detection module based on deep learning can also adopt the following implementation manners, such as adopting a sequence encoding network based on a Transformer architecture to capture long-distance dependency in the dialogue state sequence through a self-attention mechanism, thereby enhancing the judgment accuracy of dialogue complexity; adopting a time sequence modeling method based on a recurrent neural network to model the time evolution of the dialogue state through an LSTM network, or adopting a local feature extraction method based on a convolutional neural network to capture key information patterns in the dialogue through multi-scale convolution kernels, thereby realizing active prediction and accurate evaluation of the dialogue processing difficulty.

[0050] Further, the current dialogue is divided into simple questions and difficult questions by comparing the continuous difficulty score of the current dialogue with the preset difficulty threshold value. When the continuous difficulty score is less than or equal to the preset difficulty threshold value, the current dialogue is marked as a simple question handled by the intelligent customer service and is assigned to the autonomous decision path. When the continuous difficulty score is greater than the preset difficulty threshold value, the current dialogue is marked as a difficult question transferred to the human customer service for handling and is assigned to the expert decision path, thereby ensuring that difficult questions can be handled by experts in a timely manner and maximizing the autonomous processing range of AI.

[0051] Specifically, the preset difficulty threshold value refers to a preset difficulty threshold value H obtained by the embodiment after reviewing historical dialogue data. The setting of the difficulty threshold value H balances the demand for automation processing efficiency and service quality assurance.

[0052] The autonomous decision path refers to a system running the autonomous decision path when the continuous difficulty score p difficult ≤ the preset difficulty threshold value H; and the expert decision path refers to switching to the expert decision path when the continuous difficulty score p difficult > the preset difficulty threshold value H.

[0053] In order to solve the problem of user experience decline caused by the passive response mode of traditional intelligent customer service, a real-time difficulty detection module is introduced to quantitatively evaluate the expected processing difficulty of the current dialogue in the front end of the dialogue process. The difficulty detection module based on deep learning technology takes the complete state sequence from the beginning of the dialogue to the current time as input and outputs a continuous difficulty score, realizing an active prediction mechanism for dialogue processing difficulty. Compared with the traditional method, the present application avoids the lag processing mode that relies on dialogue failure or negative feedback and can identify potential processing difficulties before the problem worsens, thereby realizing preventive intervention. By comparing the continuous difficulty score with the preset threshold, the corresponding processing path allocation is automatically triggered to ensure that difficult problems are timely handled by experts, while maximizing the autonomous processing range of artificial intelligence, realizing the optimization of resource allocation and the continuous guarantee of user experience.

[0054] Further, in order to give a reasonable reward to the content output by the model, the embodiment uses a multi-directional fusion continuous reward mechanism, takes the user's satisfaction as the core optimization target to construct the reward mechanism, and rewards or punishes each step to make the model give the user the best satisfaction under the correct decision path.

[0055] Specifically, constructing the reward mechanism based on the user experience includes obtaining the direct score of the user for the dialogue service as an explicit feedback score S exp , wherein the explicit feedback score S exp is obtained through the direct score given by the user after the end of the dialogue or during the dialogue process, and the direct score is normalized to the [0, 1] interval, wherein 0 represents complete dissatisfaction and 1 represents complete satisfaction, thereby directly reflecting the subjective feelings and immediate evaluation of the user on the service quality.

[0056] For the user input text, the BERT sentiment classifier is used to identify the sentiment tendency of the user's text input during the dialogue process to obtain a sentiment analysis score S sent , thereby accurately capturing the subtle emotional changes in the user's text and providing an objective emotional state evaluation for the reward mechanism. The specific formula of the sentiment analysis score S sent is as follows:

[0057] S sent = SoftMax(BERT(u t ));

[0058] wherein u t is the user input text; SoftMax(·) is a normalization function used to convert the output of the BERT sentiment classifier into a probability distribution form to ensure that the sentiment analysis score is within a reasonable range.

[0059] After the sentiment analysis score calculation, the explicit feedback score S exp is fused with the sentiment analysis score S sent to obtain the satisfaction score, and the specific formula is as follows:

[0060] S t = w1·S exp + w2·S sent ;

[0061] wherein S t is the satisfaction score at the t time; w1 and w2 are weight coefficients of the explicit feedback score S exp and the sentiment analysis score S sent . In order to make the reinforcement learning optimize the intelligent customer service to be targeted at user experience and problem solving, w1 is taken as 0.75, so that the model pays more attention to the direct display score and emotion of the user, w2 is taken as 0.25, so as to encourage the model to pay attention to the improvement trend of user satisfaction, and prevent the model from only paying attention to the absolute value and ignoring the change trajectory of user experience.

[0062] Based on the user satisfaction score at the continuous time step, the change amount ΔS t of the satisfaction is obtained by calculating the change trend of the user satisfaction in the conversation process, and the specific formula is as follows:

[0063] ΔS t = S t - S t-1 ;

[0064] wherein ΔS t is the satisfaction change amount at the t time; S t-1 is the satisfaction score at the t-1 time.

[0065] Further, in order to reduce the noise influence and improve the signal stability, on the basis of calculating the basic satisfaction change amount, a time decay smoothing processing mechanism is adopted to optimize the satisfaction change amount, so as to effectively reduce the influence of single point fluctuation on the overall evaluation, and the specific formula of the optimized satisfaction change amount is as follows:

[0066]

[0067] wherein S is the optimized satisfaction change amount at the t time; λ is the decay factor, and the value range is [0.7, 0.8].

[0068] According to the multi-dimensional characteristics of user experience, the satisfaction score and the satisfaction change amount obtained above are weighted and fused to generate a real-time reward function, so that the model can pay attention to both the absolute level and the improvement trend of user satisfaction, and comprehensively reflect the quality of user experience in the dialogue process. The specific formula of the real-time reward function is as follows:

[0069]

[0070] wherein R t is the real-time reward value at the t time; a is the weight coefficient of the satisfaction score; S t is the satisfaction score at the t time; b is the weight coefficient of the satisfaction change amount, and the weight coefficients a and b are used to balance the absolute level and the improvement trend of user satisfaction. By analyzing a large amount of historical dialogue data, the influence degree of the absolute value and the change trend of user satisfaction on the final user experience evaluation is counted, and then the weight proportion is determined. is the satisfaction change amount after optimization at the t time.

[0071] In view of the problem that the real demand of users cannot be accurately captured due to the single-dimensional evaluation of the traditional reward mechanism, a multi-dimensional reward mechanism integrating explicit feedback score, implicit sentiment score and satisfaction change amount is constructed to realize comprehensive quantitative evaluation of user experience. Among them, by combining user direct score and text sentiment analysis based on BERT sentiment classifier, both user subjective evaluation and objective sentiment state recognition are obtained to supplement the evaluation dimension, avoiding the deviation and incompleteness that may exist in simply relying on user subjective score. At the same time, by introducing the satisfaction change amount and adopting time decay smoothing processing, not only the absolute level of user satisfaction is concerned, but also the improvement trend of user experience is valued, preventing the model from only pursuing instantaneous high score and ignoring the continuous improvement of user experience. A more accurate and stable guidance signal is provided for subsequent reinforcement learning, so that the model can learn the decision-making strategy that is really oriented to user experience.

[0072] S2: For the simple question, the intelligent customer service autonomously decides to process to generate a service track, and according to the service track and the reward mechanism, a reinforcement learning strategy is used to update the decision-making model.

[0073] As Fig. 3 shown is a double-path model update flowchart, including an autonomous decision path and an expert decision path. After the path category judgment in step S1, the proximal policy optimization (PPO-Clip) algorithm is used in the autonomous decision path to update the subsequent decision-making model. In the expert decision path, the expert track of the cloned artificial customer service is calculated to obtain the corresponding loss, so that the decision-making model can learn from the expert track, and then the update of the decision-making model is realized.

[0074] Further, after the current dialogue is evaluated by the difficulty detection module, for simple questions assigned to the autonomous decision path, the intelligent customer service autonomous decision processing mode is adopted. Since traditional intelligent customer service often lacks effective self-learning and optimization mechanism, the processing strategy update lags behind, and cannot adapt to the changes in user demand in a timely manner. Therefore, the embodiment adopts a reinforcement learning strategy, so that the model can continuously learn and optimize the decision strategy through the interaction process with the user.

[0075] For the simple question, generating a service track by intelligent customer service autonomous decision processing means that when the current dialogue is marked as a simple question processed by intelligent customer service autonomous decision and assigned to the autonomous decision path, the policy model π θ is executed under the autonomous decision path. t Output action a t with probability distribution: a t ~ π θ (a t |s t ); the policy model π θ continuously executes the step of outputting action a t with probability distribution according to the current state s t until the dialogue naturally ends, and generates a complete service track τ A = (s0, a0, s1, a1,..., s T , a T ).

[0076] For the problem of strategy update lag caused by the lack of effective self-learning mechanism of traditional intelligent customer service, by letting the policy model be the only decision maker under the autonomous decision path, outputting action probability distribution according to the current state and continuously executing until the dialogue ends, and generating a complete service track, the intelligent customer service has the ability to independently process simple questions, avoids resource consumption of manual intervention, and provides rich training data for the reinforcement learning algorithm. Not only does it improve the response speed and processing efficiency of customer service, but also gradually improves the decision-making ability of the model through continuous self-training, and realizes the improvement of the quality of intelligent customer service.

[0077] In order to solve the problem of high learning variance and jitter, the embodiment adopts an advantage function calculation method based on Critic network, and for each state-action pair (s t , a t ) in the service track τ A , combines the real-time reward value R t obtained by the reward mechanism constructed in step S1 to calculate the corresponding advantage function value, and the specific formula is as follows:

[0078] A(s t , at ) = R t + γV(s t+1 ) - V(s t );

[0079] where A(s t , a t ) is the advantage function value of state-action pair (s t , a t ); R t is the real-time reward value at the t-th moment; γ is the discount factor; V(s t ) is the state value function learned by the Critic network, representing the value estimate of state s t .

[0080] Based on the obtained advantage function value, a proximal policy optimization (PPO-Clip) algorithm is used to construct a policy gradient objective function to achieve stable policy update. After performing advantage function calculation, an importance sampling ratio mechanism is introduced to construct a clipping type policy gradient objective function, and the specific formula is as follows:

[0081]

[0082] where L CLIP (θ) is the policy gradient objective function of the proximal policy optimization algorithm; r t (θ) is the importance sampling ratio; ∈ is the clipping parameter; is the estimated advantage function value; clip is a clipping function used to limit the input value within a specified range, preventing the policy update from being too aggressive and ensuring the stability of the training process; π θ (a t | s t ) is the probability of outputting action a t from the current policy for state s t . is the probability of outputting action a t from the old policy for state s t .

[0083] According to the constructed policy gradient objective function, the gradient ascent method is used to update the parameters of the decision model. In the proximal policy optimization algorithm, an adaptive KL constraint mechanism is introduced to adjust the KL divergence target to ensure the stability of the policy update, where is the KL divergence between the old policy and the new policy; δ target is a preset KL divergence threshold used to control the magnitude of policy update. When the KL divergence exceeds the threshold, the learning rate is automatically adjusted or the update of the current round is stopped to prevent the policy from deviating too far and causing performance degradation.

[0084] To solve the problems of high learning variance and unstable training of traditional reinforcement learning algorithms in customer service scenarios, an advantage function calculation method based on a Critic network and a PPO-Clip algorithm are used to construct a policy gradient objective function, so that stable and reliable model parameter updating is realized. By introducing an importance sampling ratio and a clipping mechanism, the amplitude of policy updating is effectively controlled, and performance shocks or declines caused by excessively aggressive policy changes are prevented. In addition, through an adaptive KL constraint mechanism, when the policy deviation exceeds a preset threshold, the learning rate is automatically adjusted or the updating is stopped, ensuring the stability and convergence of the training process, so that the model can continuously optimize on the basis of maintaining existing performance, avoiding the performance fluctuations and training divergence problems commonly seen in traditional methods, and providing a reliable guarantee for the long-term stable operation of intelligent customer service.

[0085] S3: processing the difficult problem by using the artificial customer service and recording the decision-making process of the artificial customer service as an expert demonstration trajectory, updating the decision-making model by using a hybrid strategy combining imitation learning and reinforcement learning according to the expert demonstration trajectory and the reward mechanism.

[0086] Further, processing the difficult problem by using the artificial customer service and recording the decision-making process of the artificial customer service as an expert demonstration trajectory means that when the current conversation is marked as a difficult problem transferred to the artificial customer service for processing and assigned to an expert decision path, the current entire context, i.e. all conversations up to time step k, is seamlessly pushed to the artificial customer service workstation, and the subsequent interaction is taken over by a human expert. The processing process of the expert is recorded as an expert demonstration trajectory where s k is the conversation state at the expert intervention moment, is the decision action performed by the expert in state s k The expert demonstration trajectory τ B is regarded as a high-value standard example for solving similar problems, providing high-quality training samples for subsequent imitation learning.

[0087] To solve the problem of insufficient ability of traditional intelligent customer service in handling difficult problems, the complete decision-making process of the artificial customer service in handling difficult problems is recorded as an expert demonstration trajectory, realizing accurate capture and structured storage of human expert experience. The complete state-action pair of the expert from the intervention moment is recorded, forming a standard example for solving a specific type of difficult problem, providing high-quality training samples for model learning. Compared with traditional rule bases or simple imitation learning, the present application not only saves the specific operation steps of the expert, but also implicitly records the decision logic and problem solving ideas of the expert. By using these high-value expert trajectories as the target of imitation learning, the model can learn the handling strategies and decision patterns of human experts when facing difficult problems, thereby having stronger handling ability when encountering similar difficult problems.

[0088] It should be noted that although the traditional imitation learning method can quickly build a model with basic ability, the upper limit of the model is limited by the level of the human expert in the data set, and the model cannot discover better strategies through self-learning, and the model learned is often the imitation of surface behavior rather than deep decision logic, resulting in insufficient generalization ability; therefore, the embodiment adopts a hybrid strategy combining imitation learning and reinforcement learning, which can not only enable the model to efficiently absorb the valuable experience of human experts to solve problems, but also retain the ability of autonomous learning and generalization, so that the decision of the model is more in line with human preferences.

[0089] According to the expert demonstration trajectory and the reward mechanism, the hybrid strategy combining imitation learning and reinforcement learning is used to update the decision model, which includes extracting the part of the trajectory performed by the expert from the expert demonstration trajectory τ B , and constructing an expert decision action sequence:

[0090] τ exper t=(s k ,a k ,...);

[0091] Wherein, τ expert is the expert decision action sequence; (s k ,a k ) is the state-action pair of the expert at the k-th time step.

[0092] Based on the extracted expert decision action sequence, the Behavioral Cloning Loss of the model imitation output strategy is calculated, and then the difference between the model strategy and the expert decision is effectively measured, wherein since in the intelligent customer service scene in the vertical field, the decision (action) of the model is mainly discrete, that is, some answerable templates are selected to answer, so the cross-entropy loss is mainly used for calculation in this embodiment:

[0093]

[0094] Wherein, L BC (θ) is the Behavioral Cloning Loss; |τ expert | is the length of the expert decision action sequence; |A| is the size of the action space; is an indicator function, which takes the value of 1 when the expert action a k is equal to action j, otherwise 0; π θ (a j |s k ) is the probability of the model selecting action a k under state s j .

[0095] To prevent the model from overfitting the artificial customer service decision path for high rewards, a regularization term is added to the behavior cloning loss to maintain the generalization ability of the model, wherein the regularization term is obtained by calculating the distribution distance between the current model strategy and the initial pre-training strategy, and the specific formula is as follows:

[0096]

[0097] wherein, is the behavior cloning loss containing the regularization term; λ reg is the regularization coefficient; π pretrained is the pre-training strategy.

[0098] For the state-action pairs in the expert demonstration trajectory, the reinforcement learning loss based on the expert trajectory is calculated in combination with the reward mechanism constructed in step S1, wherein the advantage function based on the expert reward is first calculated:

[0099]

[0100] wherein, is the advantage function calculated based on the expert reward; R expert_bonus is the reward bonus obtained by the expert trajectory; γ is the discount factor; V(s t ) is the state value function.

[0101] After the calculation of the advantage function based on the expert reward is performed, the PPO-Clip reinforcement learning loss calculation method is adopted to construct the reinforcement learning loss based on the expert trajectory, and the specific formula of the reinforcement learning loss based on the expert trajectory is as follows:

[0102]

[0103] wherein, L RL (θ) is the reinforcement learning loss based on the expert trajectory; r t (θ) is the importance sampling ratio.

[0104] Finally, the behavior cloning loss containing the regularization term and the reinforcement learning loss based on the expert trajectory L RL (θ) are combined by weighting to construct a complete hybrid loss function:

[0105]

[0106] wherein, L hybrid (θ) is the hybrid loss function; α BC and α RL are the weight coefficients of the behavior cloning loss and the reinforcement learning loss, and the weight proportion is determined by analyzing the historical dialogue data.

[0107] By performing model parameter updating based on a hybrid loss function L hybrid (θ) can achieve effective integration of imitation learning and reinforcement learning, ensuring that the model can directly learn the best practices of humans and understand why the best practices can bring high rewards, learning the underlying strategy logic, thereby improving the model's ability to handle various problems and generalization performance.

[0108] To address the performance upper limit and insufficient generalization ability of pure imitation learning, and the low exploration efficiency of pure reinforcement learning in difficult scenarios, a hybrid strategy combining behavior cloning loss and reinforcement learning loss is constructed to combine expert experience learning and autonomous exploration ability. The hybrid strategy directly learns the decision-making behavior of experts through behavior cloning loss to quickly acquire basic problem-solving ability, and makes the model understand the reward logic behind the expert behavior through reinforcement learning loss to have adaptation and generalization ability in new scenarios. In addition, a regularization term is introduced to prevent overfitting to expert trajectories, maintaining the model's generalization ability and innovation potential. This avoids the rigid imitation that may result from pure imitation learning and the low-efficiency exploration in difficult scenarios that may result from pure reinforcement learning, enabling the model to fully absorb the valuable experience of human experts while retaining the ability for autonomous learning and strategy optimization.

[0109] On the other hand, the embodiment also provides an intelligent customer service optimization system based on reinforcement learning, which comprises:

[0110] A dialogue evaluation module obtains the current dialogue between the user and the intelligent customer service and the user experience, evaluates the difficulty of the current dialogue, divides the current dialogue into simple questions and difficult questions based on the difficulty of the current dialogue and a preset difficulty threshold, and constructs a reward mechanism based on the user experience.

[0111] An intelligent processing module generates a service trajectory by using the intelligent customer service to autonomously decide and process the simple questions, and updates a decision-making model using a reinforcement learning strategy based on the service trajectory and the reward mechanism.

[0112] An artificial processing module processes the difficult questions by using an artificial customer service and records the decision-making process of the artificial customer service as an expert demonstration trajectory, and updates the decision-making model using a hybrid strategy combining imitation learning and reinforcement learning based on the expert demonstration trajectory and the reward mechanism.

[0113] If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or parts of the present application that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0114] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, which can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instruction execution systems, apparatus or devices. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in connection with an instruction execution system, apparatus or device, or in conjunction with these instruction execution systems, apparatus or devices.

[0115] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or other suitable medium on which the program can be printed, because the program can be obtained electronically, for example, by optical scanning of the paper or other medium, followed by editing, interpreting or otherwise processing, if necessary, in other suitable ways to obtain the program electronically, and then storing it in a computer memory.

[0116] It should be understood that portions of the present application can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, implementation can be with any or a combination of the following technologies, which are all well known in the art: a discrete logic circuit having logic gates for implementing logic functions upon an application of data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0117] It should be noted that the above examples are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, which should be covered in the scope of claims of the present application.

Claims

1. A method for optimizing intelligent customer service based on reinforcement learning, characterized in that, include: The system acquires the current dialogue and user experience between the user and the intelligent customer service, assesses the difficulty of the questions in the current dialogue, and divides the current dialogue into simple and difficult questions based on the difficulty of the questions in the current dialogue and a preset difficulty threshold. A reward mechanism is then constructed based on the user experience. To address the aforementioned simple problem, a service trajectory is generated using intelligent customer service autonomous decision-making. Based on the service trajectory and the reward mechanism, a reinforcement learning strategy is employed to update the decision-making model. The difficult problem is handled by human customer service representatives, and the decision-making process of the human customer service representatives is recorded as an expert demonstration trajectory. Based on the expert demonstration trajectory and the reward mechanism, the decision-making model is updated using a hybrid strategy that combines imitation learning and reinforcement learning.

2. The intelligent customer service optimization method based on reinforcement learning as described in claim 1, characterized in that: The assessment of the difficulty of the current dialogue includes obtaining the complete state sequence of the current dialogue from the start of the dialogue to the current moment. The complete state sequence is processed by the difficulty detection module, and a continuous difficulty score representing the complexity of the current dialogue is output as the problem difficulty of the current dialogue.

3. The intelligent customer service optimization method based on reinforcement learning as described in claim 2, characterized in that: The current dialogue is divided into simple and difficult questions by comparing the difficulty level of the current question with a preset difficulty threshold. This includes comparing the continuous difficulty score of the current dialogue with the preset difficulty threshold. When the continuous difficulty score is less than or equal to the preset difficulty threshold, the current dialogue is marked as a simple question to be handled autonomously by the intelligent customer service and assigned to the autonomous decision-making path. When the continuous difficulty score is greater than the preset difficulty threshold, the current dialogue is marked as a difficult problem to be transferred to human customer service and assigned to the expert decision-making path.

4. The intelligent customer service optimization method based on reinforcement learning as described in claim 3, characterized in that: Building a reward mechanism based on the user experience includes obtaining users' direct ratings of the dialogue service as explicit feedback ratings; Perform sentiment analysis on user-input text to obtain a sentiment score; A satisfaction score is constructed by fusing the explicit feedback score and the sentiment analysis score. Calculate the trend of the satisfaction rating during the dialogue process to obtain the change in satisfaction level. The satisfaction score and the change in satisfaction are weighted and fused to generate a real-time reward function.

5. The intelligent customer service optimization method based on reinforcement learning as described in claim 4, characterized in that: Updating the decision model using a reinforcement learning strategy based on the service trajectory and the reward mechanism includes calculating the advantage function value of each state-action pair based on the state-action pairs in the service trajectory and the reward mechanism. Construct the policy gradient objective function using the aforementioned advantage function value; The policy network parameters of the decision model are updated by optimizing the policy gradient objective function.

6. The intelligent customer service optimization method based on reinforcement learning as described in claim 5, characterized in that: The process of updating the decision model using a hybrid strategy combining imitation learning and reinforcement learning, based on the expert demonstration trajectory and the reward mechanism, includes calculating the behavioral cloning loss between the model strategy and the expert decision based on the expert demonstration trajectory. The reinforcement learning loss is calculated based on the state-action pairs in the expert demonstration trajectory and the reward mechanism. The behavior cloning loss and the reinforcement learning loss are weighted and combined to construct a hybrid loss function, and the decision model is updated based on the hybrid loss function.

7. The intelligent customer service optimization method based on reinforcement learning as described in claim 6, characterized in that: The behavioral cloning loss between the expert demonstration trajectory calculation model strategy and expert decision-making includes extracting the sequence of decision-making actions performed by human customer service representatives from the expert demonstration trajectory. Calculate the cross-entropy loss between the policy output of the decision model in the corresponding state and the decision action sequence; A regularization term is introduced to prevent the decision model from overfitting to the expert demonstration trajectory. The regularization term is obtained by calculating the distribution distance between the current model policy and the initial pre-trained policy.

8. A reinforcement learning-based intelligent customer service optimization system employing the method described in any one of claims 1-7, characterized in that: The dialogue evaluation module acquires the current dialogue and user experience between the user and the intelligent customer service, evaluates the difficulty of the questions in the current dialogue, and divides the current dialogue into simple questions and difficult questions by comparing the difficulty of the questions in the current dialogue with a preset difficulty threshold, and constructs a reward mechanism based on the user experience. The intelligent processing module, for the aforementioned simple problem, utilizes intelligent customer service autonomous decision-making to generate a service trajectory, and updates the decision-making model using a reinforcement learning strategy based on the service trajectory and the reward mechanism; The manual processing module utilizes human customer service representatives to handle the difficult problems and records the decision-making process of the human customer service representatives as an expert demonstration trajectory. Based on the expert demonstration trajectory and the reward mechanism, the decision-making model is updated using a hybrid strategy that combines imitation learning and reinforcement learning.

9. A computer device, comprising: A memory and a processor; the memory stores a computer program, characterized in that: when the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.

Images