Intelligent revisit method, device, equipment and storage medium based on reinforcement learning

Abstract

The present invention discloses an intelligent return visit method based on reinforcement learning, comprising: obtaining a customer portrait, wherein the customer portrait includes a plurality of characteristic tags for identifying customers; determining a target customer to be returned based on the selected characteristic tags of the customer portrait; reading relevant information of the target customer and establishing a call connection with the target customer; inputting relevant information of the target customer into a preset return visit model; generating text conversation content for a conversation with the target customer through the return visit model, and outputting the text conversation content after speech synthesis. The present invention also discloses an intelligent return visit device, equipment, and computer-readable storage medium based on reinforcement learning. The present invention realizes dialogue communication with the returned customers, and the answers are accurate and timely, effectively improving the return visit effect, and also reducing the customer churn rate.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an intelligent revisit method, device, equipment and storage medium based on reinforcement learning. Background Art

[0002] Typically, to improve service quality, companies conduct follow-up visits with customers via phone to understand customer feedback and improve their services. Traditional follow-up visit systems rely primarily on manual labor. In the new economic environment, labor costs are rising, and the cost of customer service for companies is also rising. Furthermore, with the development of internet technology, customer service personnel face a more fragmented and diversified customer service experience. In addition to answering and making phone calls, they also need to respond to customer inquiries from online customer service or WeChat official accounts. This process is cumbersome and difficult to guarantee timely feedback, which in turn makes it difficult for companies to guarantee the effectiveness of customer follow-up visits. Summary of the Invention

[0003] The main purpose of the present invention is to provide an intelligent revisit method, device, equipment and storage medium based on reinforcement learning, aiming to solve the technical problem of how to realize intelligent revisit through machine means.

[0004] To achieve the above object, the present invention provides an intelligent revisit method based on reinforcement learning, which comprises the following steps:

[0005] Obtaining a customer profile, wherein the customer profile includes multiple characteristic tags for identifying the customer;

[0006] Determine the target customers to be revisited based on the selected characteristic tags of the customer portrait;

[0007] Read relevant information of the target customer and establish a call connection with the target customer;

[0008] Inputting relevant information of the target customers into a preset return visit model;

[0009] The return visit model is used to generate text conversation content with the target customer, and the text conversation content is synthesized into speech and then output.

[0010] Optionally, before the step of obtaining the customer portrait, the method further includes:

[0011] Read customer information data with a unified field format;

[0012] Concatenate the information in each field in sequence to form a customer information chain, wherein the customer unique identity field is used as the primary key of the customer information chain;

[0013] Based on the characteristic dimensions corresponding to the preset return visit scenarios, the field information in the customer information chain is labeled and classified according to different characteristic dimensions, and the label set corresponding to each characteristic dimension is output;

[0014] Based on the preset label quantization mapping relationship, all labels are quantized and assigned values to obtain the feature set corresponding to each feature dimension;

[0015] Based on the feature set corresponding to each feature dimension, a feature label portrait corresponding to each feature dimension is drawn and output as a customer portrait.

[0016] Optionally, before the step of obtaining the customer portrait, the method further includes:

[0017] Obtain historical dialogue data corresponding to multiple manual return visits, where the historical dialogue data is marked with a state set S, an action set A, and a reward value set R;

[0018] Take any state s in the state set S and any action a in the action set A as input samples, and the corresponding next state as output samples, and construct the state model P(s, a) through machine learning;

[0019] Take any state s in the state set S and any action a in the action set A as input samples, and the reward value corresponding to action a in the reward value set R as output samples, and construct the reward model R(s, a) through machine learning;

[0020] The historical dialogue data, state model P(s, a) and reward model R(s, a) are used as training samples, and the Dyna-Q algorithm based on the value function is used to construct a revisit model.

[0021] Optionally, generating text conversation content with the target customer through the return visit model, and outputting the text conversation content after speech synthesis includes:

[0022] Generate corresponding greetings through the return visit model according to the relevant information of the target customer;

[0023] When the call with the target customer is successfully connected, the greeting is synthesized and output;

[0024] The target customer's reply voice is input into the return visit model after voice recognition, so as to generate text conversation content with the target customer through the return visit model, and the text conversation content is output after voice synthesis.

[0025] Optionally, after the step of generating text conversation content with the target customer through the return visit model and performing speech synthesis on the text conversation content and then outputting it, the method further includes:

[0026] If the current return visit model is unable to answer the target customer, the current target customer's question content and historical conversation records are sent to the customer service end for display and manual processing by the customer service;

[0027] When the manual processing is manually taken over, the current call process with the target customer is transferred to the customer service end;

[0028] When the manual processing is a manual reply, the customer service reply content returned by the customer service end is received and output after voice synthesis.

[0029] Furthermore, to achieve the above-mentioned purpose, the present invention also provides an intelligent revisiting device based on reinforcement learning, the intelligent revisiting device based on reinforcement learning comprising:

[0030] An acquisition module is used to acquire a customer profile, wherein the customer profile includes a plurality of characteristic tags for identifying the customer;

[0031] A determination module is used to determine the target customers to be revisited based on the selected feature tags of the customer portrait;

[0032] A reading module, configured to read relevant information of the target customer and establish a call connection with the target customer;

[0033] An input module, used for inputting relevant information of the target customers into a preset return visit model;

[0034] The dialogue reply module is used to generate text dialogue content for the dialogue with the target customer through the return visit model, and output the text dialogue content after speech synthesis.

[0035] Optionally, the intelligent revisit device based on reinforcement learning further includes:

[0036] The customer information processing module is used to read customer information data with a unified field format; concatenate the information of each field in sequence to form a customer information chain, wherein the customer unique identity field is used as the primary key of the customer information chain;

[0037] The customer portrait generation module is used to label and classify the field information in the customer information chain according to different feature dimensions based on the feature dimensions corresponding to the preset return visit scenarios, and output the label set corresponding to each feature dimension; based on the preset label quantization mapping relationship, all labels are quantified and assigned values respectively to obtain the feature set corresponding to each feature dimension; based on the feature set corresponding to each feature dimension, the feature label portrait corresponding to each feature dimension is drawn and output as the customer portrait.

[0038] Optionally, the intelligent revisit device based on reinforcement learning further includes:

[0039] Corpus acquisition module, used to obtain historical dialogue corpus corresponding to multiple manual return visits, wherein the historical dialogue corpus is marked with a state set S, an action set A, and a reward value set R;

[0040] The state model construction module is used to construct the state model P(s, a) through machine learning, taking any state s in the state set S and any action a in the action set A as input samples and the corresponding next state as output samples;

[0041] The reward model construction module is used to take any state s in the state set S and any action a in the action set A as input samples, and the reward value corresponding to action a in the reward value set R as output samples, and construct the reward model R(s, a) through machine learning;

[0042] The revisit model construction module is used to use the historical dialogue data, state model P(s, a), and reward model R(s, a) as training samples and adopt the Dyna-Q algorithm based on the value function to construct a revisit model.

[0043] Optionally, the dialogue reply module includes:

[0044] A greeting generation unit, configured to generate a corresponding greeting based on the relevant information of the target customer and the return visit model;

[0045] A greeting synthesis unit, configured to synthesize the greeting into speech and output it when a call with the target customer is successfully connected;

[0046] The dialogue reply unit is used to perform speech recognition on the target customer's reply speech and input it into the return visit model, so as to generate text dialogue content with the target customer through the return visit model, and output the text dialogue content after speech synthesis.

[0047] Optionally, the intelligent revisit device based on reinforcement learning further includes:

[0048] The callback transfer module is used to send the current target customer's question content and historical conversation records to the customer service end for display if the current callback model cannot answer the target customer, so as to provide manual processing for the customer service; when the manual processing is manual takeover, the current call process with the target customer is transferred to the customer service end; when the manual processing is manual reply, the customer service reply content returned by the customer service end is received and output after voice synthesis.

[0049] Furthermore, to achieve the above-mentioned purpose, the present invention also provides an intelligent revisit device based on reinforcement learning, wherein the intelligent revisit device based on reinforcement learning includes a memory, a processor, and an intelligent revisit program based on reinforcement learning stored in the memory and executable on the processor. When the intelligent revisit program based on reinforcement learning is executed by the processor, the steps of the intelligent revisit method based on reinforcement learning as described in any one of the above-mentioned items are implemented.

[0050] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which an intelligent revisit program based on reinforcement learning is stored. When the intelligent revisit program based on reinforcement learning is executed by a processor, the steps of the intelligent revisit method based on reinforcement learning as described in any one of the above-mentioned items are implemented.

[0051] The present invention uses customer profiling to analyze and identify customers, making it easier to quickly identify target customer groups in actual return visit business scenarios. At the same time, the present invention generates a return visit model through reinforcement learning training. Then, based on the return visit model, the present invention uses TTS to synthesize speech, converting computer-generated or externally input text information into fluent, understandable Chinese spoken output, thereby enabling conversations with return customers. The present invention uses the return visit model as the core of the intelligent return visit robot and applies it to return visit business scenarios. This not only saves manual seats, but also significantly improves the quality and efficiency of return visits, providing accurate and timely responses, effectively improving the effectiveness of return visits and reducing customer churn. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a schematic diagram of the structure of the operating environment of the intelligent revisit device based on reinforcement learning involved in the embodiment of the present application;

[0053] Figure 2 This is a flow chart of the first embodiment of the intelligent revisit method based on reinforcement learning of the present invention;

[0054] Figure 3 This is a flow chart of a second embodiment of the intelligent revisit method based on reinforcement learning of the present invention;

[0055] Figure 4 Schematic diagram of the flow of the third embodiment of the intelligent revisit method based on reinforcement learning of the present invention;

[0056] Figure 5 for Figure 2 A detailed flow chart of step S150 in an embodiment;

[0057] Figure 6 2 is a flow chart of the fourth embodiment of the intelligent revisit method based on reinforcement learning of the present invention.

[0058] Figure 7 This is a functional module diagram of the first embodiment of the intelligent revisit device based on reinforcement learning of the present invention;

[0059] Figure 8 This is a functional module diagram of a second embodiment of an intelligent revisit device based on reinforcement learning according to the present invention;

[0060] Figure 9 This is a functional module diagram of the third embodiment of the intelligent revisit device based on reinforcement learning of the present invention.

[0061] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0062] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0063] The present invention provides an intelligent revisit device based on reinforcement learning.

[0064] Reference Figure 1 , Figure 1 This is a structural diagram of the operating environment of the intelligent revisit device based on reinforcement learning involved in the embodiment of the present application.

[0065] like Figure 1 As shown, the intelligent revisit device based on reinforcement learning includes: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the network interface 1004 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0066] Those skilled in the art will understand that Figure 1 The hardware structure of the intelligent revisit device based on reinforcement learning shown in the figure does not constitute a limitation of the intelligent revisit device based on reinforcement learning, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0067] like Figure 1As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a computer program. The operating system is a program that manages and controls the reinforcement learning-based intelligent revisiting device and software resources, and supports the operation of the reinforcement learning-based intelligent revisiting program and other software and / or programs.

[0068] exist Figure 1 In the hardware structure of the reinforcement learning-based intelligent revisiting device shown, network interface 1004 is primarily used for accessing the network; user interface 1003 is primarily used for detecting confirmation instructions and editing instructions, etc. Processor 1001 can be used to invoke the reinforcement learning-based intelligent revisiting program stored in memory 1005 and perform the operations of the various embodiments of the reinforcement learning-based intelligent revisiting method described below.

[0069] Based on the above-mentioned hardware structure of the intelligent revisit device based on reinforcement learning, various embodiments of the intelligent revisit method based on reinforcement learning of the present application are proposed.

[0070] Reference Figure 2 , Figure 2 1 is a flow chart of the first embodiment of the intelligent revisit method based on reinforcement learning of the present invention. In this embodiment, the intelligent revisit method based on reinforcement learning includes the following steps:

[0071] Step S110: obtaining a customer profile, wherein the customer profile includes a plurality of characteristic tags for identifying the customer;

[0072] In this embodiment, to improve the effectiveness of customer revisits, target customers for revisits are identified through customer profiling. Customer profiling primarily utilizes massive amounts of data stored in server logs and databases for analysis and mining, thereby assigning "labels" to users. Labels are identifiers that represent a specific characteristic of a customer and can be used for business operations and data analysis.

[0073] Step S120, determining target customers to be revisited based on the selected feature tags of the customer portrait;

[0074] In this embodiment, target customers can be quickly and accurately identified through customer profiling. For example, customers can be identified by age, job, type of product purchased, new or old customers, and target customers corresponding to that age group can be identified through feature tags corresponding to age.

[0075] It should be further explained that the target customers in this embodiment may be all customers selected according to feature tags, or one or more customers may be randomly selected from all selected customers as target customers.

[0076] For different businesses, for example, if there are a large number of dissatisfied users recently, then these dissatisfied users are the target group for our return visits. Through customer portrait analysis, we found that young people aged 18-30 are more dissatisfied, so we further narrow the user group for return visits, for example, by randomly selecting only a portion of customers for in-depth return visits.

[0077] Step S130, reading relevant information of the target customer and establishing a call connection with the target customer;

[0078] In this embodiment, after determining the target customer to be revisited, the relevant information of the target customer is further read, such as basic information, recorded business transaction information, etc., and then the phone number is extracted from the basic information, and the Internet phone program interface is called to establish a call connection with the target customer.

[0079] It should be noted that, in this embodiment, when there are multiple customers to be revisited, the revisits can be made by online calls with multiple customers at the same time, or by telephone calls one after another.

[0080] Step S140, inputting the relevant information of the target customer into a preset return visit model;

[0081] In this embodiment, to achieve automated customer follow-up visits, a follow-up model for automated telephone follow-up visits is prioritized. This model is preferably generated through reinforcement learning training. Generally speaking, follow-up visits primarily focus on user satisfaction, product deficiencies, and user suggestions. However, in reality, user responses vary widely, are not answered in order, and often contain a large amount of useless information. Therefore, a specialized intelligent question-answering robot (the core of which is the follow-up model) can be trained through reinforcement learning based on historical data from manual follow-up visits.

[0082] Reinforcement learning (RL) is a paradigm and methodology in machine learning that describes and solves the problem of how intelligent agents learn strategies to maximize rewards or achieve specific goals during their interactions with their environment. In actual customer follow-up scenarios, achieving a single round of dialogue is the preferred goal for reinforcement learning. The follow-up model is then adjusted through a reward and penalty mechanism. Once the learning goal is achieved, the resulting follow-up model can be deployed in the customer follow-up business scenario, enabling intelligent customer follow-up.

[0083] Step S150: Generate text conversation content with the target customer through the return visit model, and perform speech synthesis on the text conversation content and output it.

[0084] In this embodiment, after the target customer's information is input into the return visit model, the return visit model can realize automatic intelligent dialogue with the target customer. Through TTS speech synthesis technology, the text information generated by the computer itself or externally input is converted into speech that the user can understand and then output, so that formal communication with the return customer can be carried out.

[0085] This embodiment uses customer profiling to analyze and identify customers, facilitating the rapid identification of target customer groups in actual return visit scenarios. Furthermore, a return visit model is generated through reinforcement learning training. Then, based on the return visit model, text-to-speech (TTS) synthesized speech is used to convert computer-generated or externally input text information into fluent, human-understandable spoken Chinese, thereby enabling conversations with return customers. This embodiment utilizes the return visit model as the core of an intelligent return visit robot and applies it to return visit scenarios. This not only saves on manual agent seats, but also significantly improves the quality and efficiency of return visits. Answers are accurate and timely, effectively enhancing return visit effectiveness and reducing customer churn.

[0086] Reference Figure 3 , Figure 3 This is a flow chart of a second embodiment of the intelligent revisit method based on reinforcement learning of the present invention. In this embodiment, before the above step S110, it also includes:

[0087] Step S210, reading customer information data with a unified field format;

[0088] In this embodiment, when a customer registers an account, the customer's relevant information is entered into the database in advance according to a unified field format, such as: gender, age, place of origin, highest education level, registration time, telephone number, etc.; at the same time, in subsequent business transactions, the customer's transaction information is also recorded in the database and associated with the account.

[0089] In this embodiment, to facilitate the generation of customer profiles, customer information data must first be read from the database, preferably in a pre-specified unified field format. For example, customer information data can be read based on fields such as name, gender, age, occupation, transaction ID, transaction content, and transaction amount to facilitate relevant business scenario analysis, such as customer satisfaction analysis, customer complaint analysis, and customer purchasing interest analysis.

[0090] Step S220: Concatenate the information in each field in sequence to form a customer information chain, wherein the customer unique identity field is used as the primary key of the customer information chain;

[0091] In this embodiment, to facilitate labeling, the information data of the same customer is concatenated sequentially by field, thus forming a customer information chain. Each user corresponds to a unique information chain, and each information chain is distinguished by a primary key, which is composed of the corresponding value of the customer's unique identification field, such as ID number or mobile phone number. In addition, the primary key ensures that data in the information chain can be subsequently removed or corrected.

[0092] Optionally, to ensure the accuracy of the customer portrait, further format verification is performed on all fields in the information chain to ensure that the formats of each field are the same.

[0093] Step S230: Based on the feature dimensions corresponding to the preset revisit scenarios, the field information in the customer information chain is labeled and classified according to different feature dimensions, and a label set corresponding to each feature dimension is output;

[0094] In this embodiment, different analysis dimensions are set according to the needs of the preset follow-up scenarios (such as identity characteristics, resume characteristics, user complaint characteristics, customer purchase interest characteristics, etc.), and the field information of the above steps is labeled and classified according to different dimensional attributes (such as identity characteristics include gender, age, place of origin, permanent residence, etc.), and finally the label set corresponding to each feature dimension is output.

[0095] In this embodiment, in order to ensure the integrity of the label, each label needs to be verified. For example, the primary key values and information chains of n customers are randomly selected first, and then the relevant data labels in the label set are matched according to the primary key values to determine whether the label is complete. If there is an incomplete label, an alarm will be prompted to the display interface, and the user can choose to eliminate the sample data with problems.

[0096] Step S240: Based on the preset label quantization mapping relationship, all labels are quantized and assigned values to obtain a feature set corresponding to each feature dimension;

[0097] In this embodiment, for ease of calculation, each feature label needs to be quantized. Specifically, the feature label quantization processing is implemented according to a pre-set label quantization mapping relationship. For example, the labels are: Male = 1, Female = 2, Age [20, 25] = 3; Age (25, 30] = 4; Age (30, 100) = 5; Ethnicity (Han) = 6; Ethnicity (Non-Han) = 7; Highest Education Level (Bachelor's Degree) = 8; Highest Education Level (Bachelor's Degree or Above) = 9, and so on, until the last label n).

[0098] According to the label quantization mapping relationship, all labels can be quantitatively assigned to obtain the feature set corresponding to each feature dimension. For example, the feature set corresponding to the identity feature is: [1...k], and the feature set corresponding to the user complaint feature is: [3...m].

[0099] Step S250: Based on the feature set corresponding to each feature dimension, draw the feature label portrait corresponding to each feature dimension and output it as the customer portrait.

[0100] In this embodiment, after obtaining the feature set corresponding to each feature dimension, a feature label portrait corresponding to each feature dimension of each customer information can be drawn.

[0101] For example, feature dimension A: {1, 4, 6, 8, 13, 15, 17, 18, 20…k}, including users a, b, c; feature dimension B: {2, 5, 7, 9, 11, 12, 17, 19, 21…j}, including users m, p, q.

[0102] In this embodiment, in actual return visit application scenarios, the machine can quickly and accurately lock in target customers through customer portraits and thereby improve customer return visit effects. It is easy to operate, simple to implement, and reduces deployment difficulty.

[0103] Reference Figure 4 , Figure 4 This is a flow chart of a third embodiment of the intelligent revisit method based on reinforcement learning of the present invention. In this embodiment, before the above step S110, it also includes:

[0104] Step S310: Obtain historical dialogue data corresponding to multiple manual return visits, wherein the historical dialogue data is marked with a state set S, an action set A, and a reward value set R;

[0105] Step S320: Using any state s in the state set S and any action a in the action set A as input samples and the corresponding next state as output samples, a state model P(s, a) is constructed through machine learning.

[0106] Step S330: Using any state s in the state set S and any action a in the action set A as input samples, and the reward value corresponding to action a in the reward value set R as output samples, a reward model R(s, a) is constructed through machine learning.

[0107] In step S340 , the historical dialogue data, the state model P(s, a), and the reward model R(s, a) are used as training samples, and a Dyna-Q algorithm based on a value function is used to construct a revisit model.

[0108] It should be noted that, in this embodiment, steps S310-S340 may be performed before or after the above steps S210-S250.

[0109] In this embodiment, the historical dialogue data is marked with:

[0110] State set S: contains T states s in a certain order. A state refers to the state of a round of dialogue, such as the start state, the ongoing state, and the end state.

[0111] Action set A: contains T actions a corresponding to state s. Actions specifically refer to the actions taken by the agent in the current state, such as the selected answer or question content;

[0112] Reward value set R: contains T reward values r corresponding to action a. The reward value specifically refers to the reward score for the current action, such as the accuracy of the action.

[0113] Assume that the conversation data in the training sample is represented by the following sets of experience:

[0114] S1,A1,R2;S2,A2,R2;...,ST

[0115] For each of the above experiences, it can be converted into (T-1) sets of training samples, namely:

[0116] S1,A1→S2;S1,A1→R2

[0117] S2,A2→S3; S2,A2→R3 ......

[0119] S(T-1),A(T-1)→ST; S(T-1),A(T-1)→RT

[0120] The training samples on the left together form a classification model or density estimation model, with input: state and action, and output: next state. The training samples on the right together form a regression model training set, with input: state and action, and output: action reward value.

[0121] This embodiment first trains the above grouped training samples to construct the state model P(s, a) and the reward model R(s, a), respectively. Then, the historical dialogue data, the state model P(s, a), and the reward model R(s, a) are used as input parameters to construct the revisit model using the Dyna-Q algorithm based on the value function. The specific implementation logic is as follows:

[0122] (1) Initialize any state s, the state value Q(s,a) corresponding to any action a, and initialize the reward model R(s,a) and state model P(s,a);

[0123] (2) Loop through the following processes:

[0124] for i=1 to maximum number of iterations T:

[0125] a)s←current state;

[0126] b) a←∈-greedy(S,Q);

[0127] c) Execute action a and get the new state s′ and reward value r

[0128] d) Update the value function using Q-Learning:

[0129] Q(s,a)=Q(s,a)+α[r+γmaxaQ(s′,a)-Q(s,a)]

[0130] e) Use s,a,s′ to update the state model P(s,a), and use s,a,r to update the state model R(s,a);

[0131] f) for j = 1 to the maximum number of times n:

[0132] i) Randomly select a state s that has appeared before, and then randomly select an action a from the actions that have appeared in state s;

[0133] ii) Obtain s′ based on the state model P(S,A) and r based on the reward model R(s,a);

[0134] iii) Update the value function using Q-Learning:

[0135] Q(s,a)=Q(s,a)+α[r+γmaxaQ(s′,a)-Q(s,a)].

[0136] In this embodiment, the Dyna framework first interacts with the environment and updates the value function in each iteration. It then performs n model predictions, also updating the value function. The model construction approach in this embodiment leverages both experience with the environment and the model's predictions, combining model-based and non-model-based reinforcement learning. Learning from both the model and experience with the environment is used to update the value function.

[0137] In this embodiment, the interaction between the agent and the environment corresponding to reinforcement learning, through state, action, and reward, is as follows: after the agent performs a certain action, the environment will transition to a new state, for which the environment will provide a reward signal (positive or negative reward). The agent then executes new actions according to a specific strategy based on the new state and the reward feedback from the environment. This approach enables the construction of a revisit model. Deploying this revisit model in a revisit application scenario can enable intelligent question-and-answer sessions with customers, thereby achieving automated and intelligent revisits.

[0138] Reference Figure 5 , Figure 5 for Figure 2 Based on the first embodiment of the method of the present invention, in this embodiment, the above step S150 further includes:

[0139] Step S1501, generating a corresponding greeting using the return visit model based on the relevant information of the target customer;

[0140] In this embodiment, to improve the success rate of customer return visits, a greeting is further generated for use in the return visit opening. For example, "Hello Mr. XX, I'm XX. I'd like to conduct a return survey with you today. Is it convenient for you now?" The greeting has a fixed template and only requires the relevant personal information of the target customer, such as name, gender, occupation, and historical transaction records, to be filled in.

[0141] In this embodiment, the relevant information of the target customer includes the basic information of the customer, such as name, age, occupation, etc.

[0142] Step S1502: When the call with the target customer is successfully connected, the greeting is synthesized and output;

[0143] Step S1503: Perform speech recognition on the target customer's reply speech and input it into the return visit model, so as to generate text conversation content with the target customer through the return visit model, and perform speech synthesis on the text conversation content and output it.

[0144] In this embodiment, after establishing a call connection with the customer, the corresponding greeting is converted into real-person voice output through speech synthesis technology, and the customer's reply voice is simultaneously recognized. After obtaining the text reply content, it is input into the follow-up model for processing, thereby realizing voice interaction between the intelligent follow-up robot (with a built-in follow-up model) and the customer, thereby realizing automated follow-up.

[0145] Reference Figure 6 , Figure 6 This is a flow chart of a fourth embodiment of the intelligent revisit method based on reinforcement learning of the present invention. In this embodiment, after the above step S150, the following steps are further included:

[0146] Step S410: If the current callback model cannot answer the target customer, the current target customer's question content and historical conversation records are sent to the customer service end for display and manual processing by the customer service;

[0147] Step S420: When the manual processing is manual takeover, the current call process with the target customer is transferred to the customer service end;

[0148] Step S430: When the manual processing is a manual reply, the customer service reply content returned by the customer service end is received and output after voice synthesis.

[0149] In order to further improve the effect of follow-up visits and ensure that customers have a good experience with intelligent follow-up visits, for questions that the intelligent follow-up robot cannot handle (that is, the follow-up model cannot give an answer), this embodiment specifically sends the current target customer's question content and historical conversation records to the customer service end for display, so that the customer service can manually handle them.

[0150] In this embodiment, the following two methods are preferably used for manual processing:

[0151] A. Replace manual labor with intelligent return visit robots

[0152] In this case, the current call with the target customer is transferred to the customer service end, and then the customer service staff of the customer service end directly communicates with the returning customer by phone;

[0153] B. Manual reply

[0154] In this case, the customer service staff will manually input the reply content into the intelligent return call robot, and then the customer service reply content will be output after voice synthesis.

[0155] In this embodiment, if there are sufficient samples of historical conversation data, the accuracy and intelligence of the callback model can be further improved. If there are questions that cannot be answered, they can be transferred to the front-end customer service staff for manual processing, thereby ensuring a good customer experience with the intelligent callback and improving the callback effect.

[0156] The present invention also provides an intelligent revisit device based on reinforcement learning.

[0157] Reference Figure 7 , Figure 7 This is a functional module diagram of the first embodiment of the intelligent revisit device based on reinforcement learning of the present invention. In this embodiment, the intelligent revisit device based on reinforcement learning includes:

[0158] An acquisition module 10 is used to acquire a customer profile, wherein the customer profile includes a plurality of characteristic tags for identifying the customer;

[0159] A determination module 20 is used to determine target customers to be revisited based on the selected feature tags of the customer portrait;

[0160] A reading module 30 is used to read the relevant information of the target customer and establish a call connection with the target customer;

[0161] An input module 40 is used to input the relevant information of the target customer into a preset return visit model;

[0162] The dialogue reply module 50 is used to generate text dialogue content for the dialogue with the target customer through the return visit model, and output the text dialogue content after speech synthesis.

[0163] Since the description of the embodiment is the same as that of the intelligent revisit method based on reinforcement learning of the present invention, this embodiment does not elaborate on the embodiment of the intelligent revisit device based on reinforcement learning.

[0164] This embodiment uses customer profiling to analyze and identify customers, facilitating the rapid identification of target customer groups in actual return visit scenarios. Furthermore, a return visit model is generated through reinforcement learning training. Then, based on the return visit model, text-to-speech (TTS) synthesized speech is used to convert computer-generated or externally input text information into fluent, human-understandable spoken Chinese, thereby enabling conversations with return customers. This embodiment utilizes the return visit model as the core of an intelligent return visit robot and applies it to return visit scenarios. This not only saves on manual agent seats, but also significantly improves the quality and efficiency of return visits. Answers are accurate and timely, effectively enhancing return visit effectiveness and reducing customer churn.

[0165] Reference Figure 8 , Figure 8 This is a functional module diagram of the second embodiment of the intelligent revisit device based on reinforcement learning of the present invention. In this embodiment, the intelligent revisit device based on reinforcement learning also includes:

[0166] The customer information processing module 60 is used to read customer information data with a unified field format; concatenate the information of each field in sequence to form a customer information chain, wherein the customer unique identity field is used as the primary key of the customer information chain;

[0167] The customer portrait generation module 70 is used to label and classify the field information in the customer information chain according to different feature dimensions based on the feature dimensions corresponding to the preset return visit scenarios, and output the label set corresponding to each feature dimension; based on the preset label quantization mapping relationship, all labels are quantified and assigned values respectively to obtain the feature set corresponding to each feature dimension; based on the feature set corresponding to each feature dimension, the feature label portrait corresponding to each feature dimension is drawn and output as the customer portrait.

[0168] Since the description of the embodiment is the same as that of the intelligent revisit method based on reinforcement learning of the present invention, this embodiment does not elaborate on the embodiment of the intelligent revisit device based on reinforcement learning.

[0169] Reference Figure 9 , Figure 9This is a functional module diagram of the third embodiment of the intelligent revisit device based on reinforcement learning of the present invention. In this embodiment, the intelligent revisit device based on reinforcement learning includes:

[0170] Corpus acquisition module 80, for acquiring historical dialogue corpus corresponding to multiple manual return visits, wherein the historical dialogue corpus is marked with a state set S, an action set A, and a reward value set R;

[0171] A state model building module 90 is configured to construct a state model P(s, a) by machine learning using any state s in the state set S and any action a in the action set A as input samples and the corresponding next state as output samples;

[0172] The reward model construction module 100 is used to construct a reward model R(s, a) through machine learning, using any state s in the state set S and any action a in the action set A as input samples and the reward value corresponding to action a in the reward value set R as output samples;

[0173] The revisit model construction module 110 is used to construct a revisit model using the historical dialogue data, the state model P(s, a), and the reward model R(s, a) as training samples and adopting the Dyna-Q algorithm based on the value function.

[0174] Since the description of the embodiment is the same as that of the intelligent revisit method based on reinforcement learning of the present invention, this embodiment does not elaborate on the embodiment of the intelligent revisit device based on reinforcement learning.

[0175] Optionally, in a specific embodiment, the dialogue reply module includes:

[0176] A greeting generation unit, configured to generate a corresponding greeting based on the relevant information of the target customer and the return visit model;

[0177] A greeting synthesis unit, configured to synthesize the greeting into speech and output it when a call with the target customer is successfully connected;

[0178] The dialogue reply unit is used to perform speech recognition on the target customer's reply speech and input it into the return visit model, so as to generate text dialogue content with the target customer through the return visit model, and output the text dialogue content after speech synthesis.

[0179] Optionally, in a specific embodiment, the intelligent revisit device based on reinforcement learning further includes:

[0180] The callback transfer module is used to send the current target customer's question content and historical conversation records to the customer service end for display if the current callback model cannot answer the target customer, so as to provide manual processing for the customer service; when the manual processing is manual takeover, the current call process with the target customer is transferred to the customer service end; when the manual processing is manual reply, the customer service reply content returned by the customer service end is received and output after voice synthesis.

[0181] The present invention also provides a non-volatile computer-readable storage medium.

[0182] In this embodiment, the computer-readable storage medium stores a reinforcement learning-based intelligent revisit program. When executed by a processor, the reinforcement learning-based intelligent revisit program implements the steps of the reinforcement learning-based intelligent revisit method described in any of the above-mentioned embodiments. The method implemented when the reinforcement learning-based intelligent revisit program is executed by the processor can be referenced to the various embodiments of the reinforcement learning-based intelligent revisit method of the present invention, and thus will not be described in detail.

[0183] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, or of course by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM) and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0184] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of the present invention, or directly or indirectly used in other related technical fields, all fall within the protection of the present invention.

Claims

1. An intelligent revisit method based on reinforcement learning, characterized in that: The intelligent revisit method based on reinforcement learning includes the following steps: Obtain historical dialogue corpus corresponding to multiple manual return visits, wherein the historical dialogue corpus is marked with a state set S, an action set A, and a reward value set R; Take any state s in the state set S and any action a in the action set A as input samples, and the corresponding next state as output samples, and construct the state model P(s, a) through machine learning; Take any state s in the state set S and any action a in the action set A as input samples, and the reward value corresponding to action a in the reward value set R as output samples, and construct the reward model R(s,a) through machine learning; The historical dialogue data, state model P(s, a), and reward model R(s, a) are used as training samples, and the Dyna-Q algorithm based on the value function is used to build a revisit model; Obtaining a customer profile, wherein the customer profile includes multiple characteristic tags for identifying the customer; Determine target customers to be revisited based on the selected characteristic tags of the customer portrait; Read relevant information of the target customer and establish a call connection with the target customer; Inputting relevant information of the target customer into the return visit model; The return visit model is used to generate text conversation content with the target customer, and the text conversation content is synthesized into speech and then output.

2. The intelligent revisit method based on reinforcement learning according to claim 1, characterized in that: Before the step of obtaining the customer portrait, the following steps are also included: Read customer information data with a unified field format; Concatenate the information in each field in sequence to form a customer information chain, wherein the customer unique identity field is used as the primary key of the customer information chain; Based on the feature dimensions corresponding to the preset return visit scenarios, the field information in the customer information chain is labeled and classified according to different feature dimensions, and the label set corresponding to each feature dimension is output; Based on the preset label quantization mapping relationship, all labels are quantized and assigned values to obtain the feature set corresponding to each feature dimension; Based on the feature set corresponding to each feature dimension, the feature label portrait corresponding to each feature dimension is drawn and output as the customer portrait.

3. The intelligent revisit method based on reinforcement learning according to claim 1, characterized in that: Generating text conversation content with the target customer through the return visit model, and outputting the text conversation content after speech synthesis includes: Generate corresponding greetings through the return visit model according to the relevant information of the target customer; When the call with the target customer is successfully connected, the greeting is synthesized and output; The target customer's reply voice is input into the return visit model after voice recognition, so as to generate text conversation content with the target customer through the return visit model, and the text conversation content is output after voice synthesis.

4. The intelligent revisit method based on reinforcement learning according to any one of claims 1 to 3, characterized in that: After the steps of generating text conversation content with the target customer through the return visit model and outputting the text conversation content after speech synthesis, the method further includes: If the current return visit model is unable to answer the target customer, the current target customer's question content and historical conversation records are sent to the customer service end for display and manual processing by the customer service; When the manual processing is manually taken over, the current call process with the target customer is transferred to the customer service end; When the manual processing is a manual reply, the customer service reply content returned by the customer service end is received and output after voice synthesis.

5. An intelligent revisit device based on reinforcement learning, characterized in that: The intelligent revisit device based on reinforcement learning includes: Corpus acquisition module, used to obtain historical dialogue corpus corresponding to multiple manual return visits, wherein the historical dialogue corpus is marked with a state set S, an action set A, and a reward value set R; The state model construction module is used to construct the state model P(s, a) through machine learning, taking any state s in the state set S and any action a in the action set A as input samples and the corresponding next state as output samples; The reward model construction module is used to take any state s in the state set S and any action a in the action set A as input samples, and the reward value corresponding to action a in the reward value set R as output samples, and construct the reward model R(s, a) through machine learning; A revisit model construction module is used to use the historical dialogue data, the state model P(s, a), and the reward model R(s, a) as training samples and adopt the Dyna-Q algorithm based on the value function to construct a revisit model; An acquisition module is used to acquire a customer profile, wherein the customer profile includes a plurality of characteristic tags for identifying the customer; A determination module is used to determine the target customers to be revisited based on the selected feature tags of the customer portrait; A reading module, configured to read relevant information of the target customer and establish a call connection with the target customer; An input module, used for inputting relevant information of the target customers into the return visit model; The dialogue reply module is used to generate text dialogue content for the dialogue with the target customer through the return visit model, and output the text dialogue content after speech synthesis.

6. The intelligent revisit device based on reinforcement learning according to claim 5, characterized in that: The dialogue reply module includes: A greeting generation unit, configured to generate a corresponding greeting based on the relevant information of the target customer and the return visit model; A greeting synthesis unit, configured to synthesize the greeting into speech and output it when a call with the target customer is successfully connected; The dialogue reply unit is used to perform speech recognition on the target customer's reply speech and input it into the return visit model, so as to generate text dialogue content with the target customer through the return visit model, and output the text dialogue content after speech synthesis.

7. The intelligent revisit device based on reinforcement learning according to claim 5, characterized in that: The intelligent revisit device based on reinforcement learning also includes: The customer information processing module is used to read customer information data with a unified field format; concatenate the information of each field in sequence to form a customer information chain, wherein the customer unique identity field is used as the primary key of the customer information chain; The customer portrait generation module is used to label and classify the field information in the customer information chain according to different feature dimensions based on the feature dimensions corresponding to the preset return visit scenarios, and output the label set corresponding to each feature dimension; based on the preset label quantization mapping relationship, all labels are quantified and assigned values respectively to obtain the feature set corresponding to each feature dimension; based on the feature set corresponding to each feature dimension, the feature label portrait corresponding to each feature dimension is drawn and output as the customer portrait.

8. The intelligent revisit device based on reinforcement learning according to any one of claims 5 to 7, characterized in that: The intelligent return visit device based on reinforcement learning also includes: a return visit transfer module, which is used to send the current target customer's question content and historical conversation records to the customer service end for display if the current return visit model cannot answer the target customer, so as to provide manual processing for the customer service; when the manual processing is manual takeover, the current call process with the target customer is transferred to the customer service end; when the manual processing is manual reply, the customer service reply content returned by the customer service end is received and output after voice synthesis.

9. An intelligent revisit device based on reinforcement learning, characterized in that: The reinforcement learning-based intelligent revisit device includes a memory, a processor, and a reinforcement learning-based intelligent revisit program stored in the memory and executable on the processor. When the reinforcement learning-based intelligent revisit program is executed by the processor, the steps of the reinforcement learning-based intelligent revisit method according to any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores an intelligent revisit program based on reinforcement learning. When the intelligent revisit program based on reinforcement learning is executed by a processor, the steps of the intelligent revisit method based on reinforcement learning as described in any one of claims 1 to 4 are implemented.

Images