Customer service risk assessment and early warning method and system based on historical data
Through a two-layer Markov decision-making process model based on historical data and a deep Q network algorithm, dynamic risk assessment and real-time early warning are realized, solving the problem of lack of dynamicity and flexibility of risk assessment in the existing technology, and improving risk management The level of intelligence and automation has improved customer service quality and data security.
Patent Information
- Application Number
- CN202410500222.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-03-21
- Filing Date
- 2024-04-24
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-04-24
AI Technical Summary
The existing risk assessment methods lack dynamicity and flexibility, and cannot effectively deal with rapidly changing market conditions and customer behavior, and traditional data risk control methods have problems such as lagging in predefined rules and incomplete risk coverage.
Using a two-layer Markov decision-making process (HMDP) model based on historical data, combined with deep Q network (DQN) algorithm, a risk assessment algorithm is built, scenario analysis is performed through top-level decision-making models and detailed emergency response plans are formulated to achieve dynamic Risk assessment and real-time early warning.
It provides personalized risk management services, improves the intelligence and automation level of risk management, enhances enterprise data security protection and customer service quality, and supports enterprise intelligent transformation and sustainable development.
Smart Images

Figure CN119089235A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of information security technology, and in particular to a customer service risk assessment and early warning method and system based on historical data. Background Art
[0002] In the modern business environment, the operation of customer service systems is accompanied by various uncertainties, which may affect the operation of enterprises and customer satisfaction. Therefore, effective risk assessment and early warning systems are essential to improve service quality, prevent potential problems and maintain customer relationships. Most current risk assessment methods are based on static data analysis and determine risk levels based on historical experience. However, these methods are often insufficient to cope with rapidly changing market conditions and customer behaviors, and lack dynamism.
[0003] In terms of enterprise data management, risk assessment and early warning also play an important role. Enterprises need to promptly discover and respond to various data security risks, such as data leakage, illegal access, malicious tampering, etc., to ensure the security of data assets and business continuity. Traditional data risk control methods mainly rely on rule engines and expert experience, and have limitations such as lags in predefined rules and incomplete risk coverage. In addition, in terms of archive management during enterprise operations, most existing archive management systems use shallow analysis methods such as keyword retrieval and word frequency statistics, which cannot fully explore the value of archives. Summary of the invention
[0004] In view of this, the present application provides a customer service risk assessment and early warning method and system based on historical data, which realizes dynamic risk assessment and real-time early warning feedback in the process of enterprise data and archive management by establishing a risk assessment model based on historical data and combining a two-layer decision-making framework. It solves the problems of the lack of comprehensiveness, dynamism and flexibility of risk assessment in the prior art, and the inability to provide personalized and multi-level services.
[0005] This application provides a customer service risk assessment and early warning method based on historical data, including:
[0006] Collect historical data on customer service;
[0007] Construct a two-level Markov decision process HMDP model;
[0008] Based on the HMDP model, construct a risk assessment algorithm;
[0009] The risk assessment algorithm is used to analyze the historical data so as to assess and warn of the risks of customer service.
[0010] Optionally, construct an HMDP model, including:
[0011] Constructing a top-level decision model of HMDP, wherein the top-level decision model is used to identify and classify different customer service scenarios;
[0012] The underlying decision model of HMDP is constructed, and the underlying decision model is used to construct specific risk assessment and response plans.
[0013] Optionally, construct a top-level decision model of HMDP, including:
[0014] Categorize historical service scenarios;
[0015] According to the classified historical service scenarios, the corresponding risk factors are integrated to construct the environment state, state transition probability matrix and reward function;
[0016] A greedy approach is used to construct the decision strategy.
[0017] Optionally, construct an underlying decision model of HMDP, including:
[0018] Define fine-grained actions for response plans;
[0019] Design reward and penalty criteria for each of the fine-grained actions.
[0020] Optionally, the risk assessment algorithm is a deep Q network DQN method, then based on the HMDP model, a risk assessment algorithm is constructed, including:
[0021] Obtaining the environment state S, the fine-grained action set A of the response plan, and the reward function R defined by the HMDP model;
[0022] Initialize the Q value table.
[0023] Optionally, analyzing the historical data using the risk assessment algorithm includes:
[0024] Classify the historical data into service scenarios, and obtain the environment state St of the first scenario at time t;
[0025] Based on the Q value at time t, select an action that can maximize the expected reward from the fine-grained action set A of the response plan;
[0026] Execute the action that can maximize the expected reward, obtain the reward function Rt, and the state St+1 at time t+1;
[0027] According to the reward function Rt and the state St+1 at time t+1, the Bellman equation is used to update the Q value.
[0028] Optionally, before analyzing the historical data using the risk assessment algorithm, the method further includes:
[0029] Obtain a historical data training set, and use the historical data training set to perform model training on the DQN;
[0030] Perform parameter tuning and DQN model validation, using grid search to find the best parameter combination;
[0031] The DQN model is optimized.
[0032] Optionally, after analyzing the historical data using the risk assessment algorithm, the method further includes:
[0033] The accuracy of risk prediction is improved through stepwise annealing and random sampling algorithms.
[0034] Optionally, the accuracy of risk prediction is improved through stepwise annealing and random sampling algorithms, including:
[0035] Using simulated withdrawal technology to optimize the decision-making process of the risk assessment algorithm;
[0036] Monte Carlo simulation method is used to test the robustness of the risk assessment algorithm in diverse environments;
[0037] Based on the test results, the parameters of the risk assessment algorithm are adjusted.
[0038] The embodiment of the present invention also provides a customer service risk assessment and early warning system based on historical data, including:
[0039] Collection module, used to collect historical data of customer service;
[0040] A construction module is used to construct a two-level Markov decision process HMDP model; based on the HMDP model, a risk assessment algorithm is constructed;
[0041] The analysis module is used to analyze the historical data using the risk assessment algorithm so as to assess and warn of the risks of customer service.
[0042] This application provides a customer service risk assessment and early warning method and system based on historical data. By establishing a risk assessment model based on historical data and combining it with a two-layer decision-making framework, dynamic risk assessment and real-time early warning feedback are achieved. Among them, the top-level model is responsible for scenario analysis and macro-decision-making of risks, while the lower-level model formulates detailed emergency response plans for specific situations. It can not only provide personalized risk management services, but also improve the intelligence and automation level of risk management. In terms of enterprise data and file management, it can strengthen enterprise data security protection and file management level internally, and improve customer service quality and user experience externally, providing strong support for enterprise intelligent transformation and sustainable development, and ultimately providing enterprises and customers with a more multi-level, reliable and efficient customer service experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the disclosed embodiments in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.
[0044] Figure 1 A flowchart of a customer service risk assessment and early warning method based on historical data provided in an embodiment of the present application;
[0045] Figure 2 Schematic diagram of the structure of a customer service risk assessment and early warning system based on historical data provided in one embodiment. DETAILED DESCRIPTION
[0046] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.
[0047] It should be clear that the following embodiments of the present disclosure are described by specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.
[0048] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.
[0049] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0050] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.
[0051] Embodiment 1
[0052] The embodiment of the present application provides a customer service risk assessment and early warning system based on historical data, which is installed in a cloud service cluster. The system improves the awareness and response capabilities of potential service risks by implementing a two-level Markov decision process (HMDP). It uses a large amount of historical interaction data, machine learning and statistical modeling techniques to predict and respond to possible service risks, and formulate effective emergency plans.
[0053] The embodiments of the present application are applicable to customer services such as e-commerce and finance, and can also be applied to fields such as medical care, transportation, and smart cities to achieve dynamic risk assessment and real-time warning. These fields also involve a large number of customer services and decision-making processes.
[0054] Figure 1 A flowchart of a customer service risk assessment and early warning method based on historical data is provided in an embodiment of the present disclosure. The customer service risk assessment and early warning method based on historical data includes steps S1 to S4.
[0055] S1. Collect historical data on customer service;
[0056] In order to build a reliable risk assessment and early warning system, it is first necessary to collect and pre-process historical customer service data to establish a good data foundation.
[0057] Specifically, first, extract customer interaction history data from the service platform database. The service platform database can be an e-commerce data platform, a financial data platform, or other data platform related to customer service. Secondly, access the service platform database and select the corresponding data export mode to ensure the integrity and consistency of the data. Finally, extract the customer's interaction records, query logs, and service results, including timestamps, frequencies, durations, customer feedback, etc. In addition, for various types of data, the correlation of various types of data can be analyzed in advance to identify the data types and features that are most valuable for risk assessment. In this way, other irrelevant types can be selected.
[0058] After collecting historical data, data cleaning techniques can also be used to remove noise and handle missing values. For example, data cleaning procedures can be applied to remove duplicate records, erroneous data entries, and irrelevant data to increase data availability; missing data can be handled, such as using interpolation algorithms to fill in gaps or removing records with more missing data; data normalization can be performed, such as normalizing numerical features and encoding categorical features to reduce the impact of data dimensions; and finally, features can be optimized through data transformation to adapt to the needs of the Markov decision process. For example, dimensionality reduction techniques, such as principal component analysis (PCA), can be used to extract the most informative features.
[0059] S1 is the foundation for subsequent risk assessment and early warning system establishment. High-quality data collection and preprocessing directly affect the accuracy and efficiency of the model. Through thorough data preprocessing, it is ensured that the Markov decision process can run on a clean and rich data set, which will provide reliable input for the final risk assessment and enable the early warning system to respond quickly and accurately to potential service risks.
[0060] S2, construct a two-level Markov decision process HMDP model;
[0061] The core of this step is to establish a hierarchical decision-making model, namely the Hierarchical Markov Decision Process (HMDP), for complex risk assessment and emergency planning.
[0062] The HMDP model is a hierarchical decision model, which includes a top layer and a bottom layer. The top layer is responsible for macro risk assessment and strategy guidance, while the bottom layer focuses on specific operations and response measures.
[0063] The HMDP model decomposes the problem into two levels: top level and bottom level:
[0064] Top-level model: Responsible for macro-evaluation of different customer service scenarios and situations, and determining the direction of further detailed analysis or macro-strategies to be implemented. For example, the top level may determine whether a certain customer group has a higher service risk.
[0065] Bottom-level model: Under the direction or strategy specified by the top-level model, the bottom-level model enters into specific operations and makes specific service decisions and response measures. This part usually requires more detailed risk assessment and more specific response actions. For example, after identifying a high-risk customer group, the bottom-level model works on how to reduce this risk, which may be through actions such as sending coupons and providing additional services.
[0066] The construction of the HMDP model described in S2 includes the following steps S21-S22:
[0067] S21. Construct a top-level decision model of HMDP, wherein the top-level decision model is used to identify and classify different customer service scenarios;
[0068] Specifically, the top-level decision model of HMDP is constructed in S21, including steps S211-S213:
[0069] S211. Classify historical service scenarios;
[0070] Specifically, clustering algorithms such as K-Means, hierarchical clustering, or DBSCAN are used to process historical service data and automatically divide the data into several categories, each of which represents a service scenario.
[0071] Evaluate and adjust clustering results based on business logic to ensure that each classification is consistent with the actual service scenario.
[0072] Taking K-Means as an example, the process of using the K-Means clustering algorithm to classify historical service scenarios includes:
[0073] A1. Data Preprocessing
[0074] A11. Clean the data and remove outliers and erroneous fields.
[0075] A12. Standardize numerical data to avoid certain features having too much influence on distance calculation due to dimensionality.
[0076] A13. Encode non-numeric data, such as using One-Hot Encoding.
[0077] A2. Determine the K value
[0078] A21. Use the Elbow Method to determine the optimal number of clusters K. This method calculates the sum of squared errors (SSE) corresponding to different K values and selects the point where the SSE decreases rapidly as the K value.
[0079] A22. The Silhouette Method can be used to assist in verifying the rationality of the K value.
[0080] A3. Perform K-Means Clustering
[0081] A31. Randomly select K cluster centers.
[0082] A32. Assign each data point to the nearest cluster center to form K clusters.
[0083] A33. Calculate the mean of each cluster and use it as the new cluster center.
[0084] A34. Repeat steps A32-A33 until the cluster center is stable or the maximum number of iterations is reached.
[0085] A4. Clustering result evaluation and optimization
[0086] A41. Evaluate clustering effect: Use indicators such as Silhouette Coefficient or Cohesion / Separation to evaluate.
[0087] A42.Business logic verification: Work with the business team to confirm whether the clustering results are consistent with business understanding.
[0088] A43. Fine-tune model parameters or redefine features based on feedback from the business team to achieve the best business application results.
[0089] S212. According to the classified different historical service scenarios, the corresponding risk factors are integrated to construct the environment state, state transition probability matrix and reward function;
[0090] Specifically, S212 includes:
[0091] Determine the state space: Convert the clustering results into the state space of the Markov decision process. Each service scenario corresponds to one or more states. Assume that the customer service scenario is divided into four environmental states: initial consultation, further consultation, targeted answer, and service termination. Each state represents a different stage of the customer in the service process.
[0092] Construct a transition probability matrix: Count the frequency of transitions between different scenarios and use this as the transition probability. Specifically, count the number of times a customer transitions from one service state to another in historical data, and use the transition frequency to estimate the probability. For example, the transition probability from initial consultation to further consultation is P(C2|C1). In this way, the transition probabilities of different states constitute a transition probability matrix.
[0093] Reward function definition: Determine the reward value based on the business indicators affected by each service scenario, such as customer satisfaction, processing time, etc. For example, determine the reward value based on business key performance indicators (KPIs), such as customer satisfaction, fast response time, etc. Set a positive reward for each state terminal transition (such as C4), and set a smaller negative reward or zero reward for the intermediate state (such as C1, C2) to encourage the system to move to the terminal state.
[0094] The following is a description of a specific example:
[0095] The state space consists of:
[0096] Status "C1": The customer submitted a service request but has not received a response yet.
[0097] Status "C2": The customer receives an initial response but requires more information.
[0098] Status "C3": The customer has received targeted answers and the problem has been partially resolved.
[0099] Status "C4": The customer's problem is completely resolved and the service is completed.
[0100] The state transition probability matrix is shown in Table 1, and the sum of each row is 1:
[0101] Table 1
[0102]
[0103] For example, P(C2|C1)=0.7 means that the probability of transitioning from C1 to C2 is 70%.
[0104] Reward values include:
[0105] C4 status (Problem Solved) provides a positive reward of +100.
[0106] The C3 state receives a small positive reward of +10 (because the problem is moving towards being solved).
[0107] The C2 state has no reward and no penalty of 0 (remains neutral, encourages information collection).
[0108] In the C1 state, a small penalty of -10 is given due to delayed service (to encourage quick response).
[0109] S213. Use the greedy method to construct a decision-making strategy.
[0110] Using the greedy strategy in reinforcement learning, each time select the action that maximizes the reward (or long-term return) in the current state as part of the strategy.
[0111] This strategy may perform well in the early stages of modeling, but needs to be optimized over time to obtain the global optimal strategy.
[0112] The decision-making process using the Greedy Policy of reinforcement learning in S213 mainly includes the following steps:
[0113] B1. Initialization state and behavior value function
[0114] Assume that there is a behavior value function Q table, which records the expected returns of taking different actions in each state.
[0115] B2. Choose an action strategy
[0116] At each decision point (current state s), select the action a that maximizes Q(s,a) as the current optimal action strategy.
[0117] That is, in state s, find the Q values corresponding to all possible actions and select the action with the highest Q value to execute.
[0118] B3. Update Q value table
[0119] Perform actions and observe the results to see if the expected reward was achieved.
[0120] Update the Q value corresponding to the current state and action based on the immediate reward obtained and the maximum Q value of the next state to improve the strategy.
[0121] Suppose in a certain customer service situation, we need to choose the best response action to improve customer satisfaction:
[0122] Current state: The customer submitted a request to return an item.
[0123] Possible actions: A1 provides a return label, A2 provides instant online customer service chat, and A3 provides an in-store return option.
[0124] The behavior value function Q table is initialized as shown in Table 2:
[0125] Table 2
[0126]
[0127] In this state s, action A2 has the largest Q value, that is, Q(s,A2) = 8, so according to the greedy strategy, action A2 (providing instant online customer service chat) is selected as the optimal strategy.
[0128] After executing the action, suppose the new immediate reward is +10 and the customer satisfaction score is also improved. Then the Q value of action A2 in state s can be updated. If the Bellman equation is used:
[0129] Q(s,A2) = original Q value + learning rate × (immediate reward + discount rate × maximum next Q value - original Q value)
[0130] Assuming a learning rate of 0.5, a discount rate of 0.8, and a maximum Q-value of 12 for the next state, the update is calculated as:
[0131] Q(s,A2)=8+0.5×(10+0.8×12-8)=15
[0132] Finally, the Q table updated according to the greedy strategy is shown in Table 3:
[0133] Table 3
[0134]
[0135] In this way, the greedy strategy will continue to drive the strategy to evolve in the direction of generating higher immediate rewards and long-term returns, ultimately forming an efficient decision-making strategy.
[0136] S22. Construct an underlying decision model of HMDP, where the underlying decision model is used to construct a specific risk assessment and response plan.
[0137] In step S22, the underlying decision model will be implemented, involving the definition of actions for specific risk assessment and the design of reward and penalty criteria for these actions. This is the underlying part of the HMDP model, and its purpose is to make more detailed operational decisions based on the top-level strategy guidance.
[0138] Among them, constructing the underlying decision model of HMDP in S22 includes steps S211-S212:
[0139] S221. Define fine-grained actions for response plans;
[0140] Assume that in the customer service scenario, the following fine-grained actions are defined:
[0141] D1: Provide instant online answers.
[0142] D2: Send detailed instructions document.
[0143] D3: Arrange call back service.
[0144] D4: Submit to the technical support team.
[0145] First, analyze historical service records to identify common micro-steps in the problem-solving process. Second, communicate with the customer service team to understand which micro-steps can have a positive impact on customer satisfaction. Finally, determine the action set and list all possible service response plan actions to ensure that each action can point to a different service goal or problem-solving method.
[0146] S222. Design reward and punishment criteria for each of the fine-grained actions.
[0147] For example, the reward and punishment standards are formulated accordingly:
[0148] D1: If the problem is solved immediately, the reward is +10; if the response is overdue, the penalty is -5.
[0149] D2: If customer satisfaction improves, reward +8; if the document does not help solve the problem, penalty -3.
[0150] D3: Timely service will earn you a +15 reward; failure to contact the customer within the agreed time will earn you a -10 penalty.
[0151] D4: The technical team will receive +20 points for solving the problem; if there is no feedback after the scheduled time, the team will be penalized -15 points.
[0152] Specifically, S222 includes:
[0153] Determine the reward and punishment mechanism and set corresponding rewards based on the contribution of each micro action to the overall goal.
[0154] Monitor the implementation effect in real time and monitor and evaluate the results after the action is executed.
[0155] Customer feedback analysis, customer satisfaction surveys or feedback analysis provide a measure of the effectiveness of planned actions.
[0156] The reward and punishment standards are adjusted dynamically and adjusted in a timely manner based on real-time data and customer feedback.
[0157] In summary, the constructed HMDP model can effectively grasp and guide risk decisions in the customer service process, thereby realizing dynamic assessment and early warning of customer service risks. The top-level model provides macro control of risks in different service scenarios, while the low-level model can formulate refined response strategies for each specific situation.
[0158] S3. Constructing a risk assessment algorithm based on the HMDP model;
[0159] The risk assessment calculation algorithm works under the guidance of this HMDP model. It selects the best action by calculating the expected reward based on the current state and the action options provided by HMDP, and then updates the model (learning) based on the results of the action, in order to make better risk assessments and warnings when encountering similar situations in the future.
[0160] Among them, the risk assessment calculation algorithm is a specific algorithm used to perform risk assessment and early warning based on the established two-level Markov decision process (HMDP) model. That is, the HMDP model provides a framework for risk assessment, and the risk assessment calculation algorithm is a method for operating under this framework.
[0161] In the embodiment of the present invention, exemplarily, the risk assessment algorithm is a deep Q network DQN method, then in S3, based on the HMDP model, a risk assessment algorithm is constructed, specifically including:
[0162] S31. Obtain the environment state S, the fine-grained action set A of the response plan and the reward function R defined by the HMDP model;
[0163] For example:
[0164] Environmental state S: We assume that there are four states: new customer consultation (S1), order processing (S2), customer service question (S3) and after-sales service evaluation (S4).
[0165] Fine-grained action set A of the response plan: The action set includes: providing product information (A1), confirming order details (A2), answering service questions (A3) and collecting customer feedback (A4).
[0166] Reward function R: The reward for each action is determined based on user feedback and business goals. For example, if the state is changed from S2 to user satisfaction through action A2, there is a positive reward R_positive; if the service speed is delayed, there is a penalty R_negative.
[0167] S31 specifically includes:
[0168] Define the environment state S, covering all key customer service nodes.
[0169] List all possible actions A, which represent all possible service response measures.
[0170] Construct a reward function R, combine business indicators and customer feedback, and determine the reward or penalty value for each action.
[0171] S32. Initialize the Q value table.
[0172] Create a Q table to store and update the values of state-action pairs. Each record in this table represents the expected reward of an action in a given state.
[0173] Initialize all Q values to 0 or a small random number to provide a starting point for the learning process.
[0174] Make sure the update mechanism is in place so that after each interaction, the Q table is updated with the values corresponding to the current state and action.
[0175] Q values are shown in Table 4:
[0176] Table 4
[0177]
[0178] Through this initialization process, the DQN algorithm will have a starting point to explore the effects of various state and action combinations, learn and optimize the target strategy.
[0179] S4. Analyze the historical data using the risk assessment algorithm to assess and warn of risks in customer service.
[0180] S4 specifically includes steps S41-S44:
[0181] S41. Classify the service scenarios of the historical data and obtain the environmental state S of the first scenario at time t t ;
[0182] Use a classification algorithm (such as K-Means clustering, see S211) to automatically classify service scenarios; assign a state label to each type of service scenario, so that each scenario corresponds to a state in the HMDP model; at time t, based on the current customer interaction or service request, determine the state S of the current service scenario t For example, the service interactions in the historical data are divided into multiple scenarios, such as product consultation, order inquiry, complaint handling, etc. At time t, the environment state may be that the customer is making an order inquiry.
[0183] S42. Based on the Q value at time t, select an action that can maximize the expected reward from the fine-grained action set A of the response plan;
[0184] Access the Q value table at time t and check the current state S t The Q value of each action is given. Action A with the highest Q value is selected t As the best response strategy. In state S t Next, select action A with the highest expected reward according to the Q value table t .
[0185] S43. Execute the action that can maximize the expected reward and obtain the reward function R t , and the state S at time t+1 t+1 ;
[0186] Actual execution of action A t , and monitor the customer's reaction or interaction results. Determine the immediate reward R obtained after performing the action t . Record the change in customer service status after the action is executed and identify it as the new status S t+1 . Specifically,
[0187] Execute action A t , such as responding to customers regarding order inquiries and providing requested information.
[0188] According to the execution results, you will get a reward of R t , for example, positive rewards are obtained when customer satisfaction increases.
[0189] The new state observed is S t+1 , such as transferring from order inquiry to confirmation of receipt status.
[0190] S44. Use the Bellman equation to update the Q value based on the reward function Rt and the state St+1 at time t+1.
[0191] Update the Q(St,At) value according to the Bellman equation, the learning rate (α), and the discount rate (γ). The new Q value is based on the old Q value, combined with the actual reward Rt and the next state S t+1 The expected Q-value of the best action.
[0192] In addition, optionally, for different application scenarios, the embodiment of the present application may also use other reinforcement learning methods such as Proximal Policy Optimization (PPO) and Asynchronous Advantage Actor-Critic (A3C) to perform risk assessment, for example:
[0193] In the inventory and order flow management during the promotion period, in order to cope with the high volatility of demand, the original DQN algorithm can be replaced by PPO:
[0194] PPO is used for dynamic inventory management. It improves the sample efficiency and stability of the algorithm by resampling the previous behavior (importance sampling), reducing the change of learning rate (step size) and running multiple data optimizations in one update phase.
[0195] When the order volume surges, the PPO algorithm not only predicts the risk of insufficient inventory, but also evaluates the long-term benefits under various promotional strategies, thereby selecting the optimal replenishment plan and pricing strategy to balance immediate sales and inventory costs.
[0196] Compared with DQN, the PPO algorithm can more effectively learn in the multi-action space of inventory management, quickly adapt to dynamic changes during promotions, and iterate and optimize strategies before the next promotion cycle.
[0197] In monitoring and warning of abnormal purchasing behavior, A3C can process and respond faster:
[0198] The A3C algorithm uses multiple working copies to explore and learn in parallel, quickly collecting enough diverse experiences to train the model, which helps speed up the training process of the abnormal behavior detection model.
[0199] When an abnormal transaction is detected, the "Actor" component of A3C can give decision recommendations, such as temporarily freezing the order or conducting further identity verification; while the "Critic" component evaluates the quality of the decision and quickly updates the strategy.
[0200] Compared with DQN, A3C can adapt to emerging fraud methods more quickly and continuously improve the efficiency of identifying and preventing fraud in the process of continuous transactions.
[0201] In addition, before analyzing the historical data using the risk assessment algorithm, the method further includes:
[0202] Obtain a historical data training set, and use the historical data training set to perform model training on the DQN;
[0203] For example, we use customer service interaction data from the past year, including the initial state of each service, the actions taken, the rewards obtained, and the final state of the service. We clean and preprocess the data to ensure data quality. We use this dataset to train the DQN model. We overcome the correlation problem between consecutive samples by replaying historical data (Experience Replay) and improve training efficiency by updating in mini-batch.
[0204] Perform parameter tuning and DQN model validation, using grid search to find the best parameter combination;
[0205] The standard training, validation, and test dataset splitting methods are used. Cross-validation and grid search techniques are used to optimize the hyperparameters during DQN training to determine the best parameter combination. Performance evaluation on the validation set helps select the most appropriate model and use it on the test set for final evaluation.
[0206] The DQN model is optimized.
[0207] Use technical means such as early stopping and regularization to prevent overfitting. Adjust the exploration-exploitation strategy, such as using the epsilon-greedy strategy, to ensure that the exploration and exploitation of the model are balanced. Monitor indicators such as average reward and sliding average, and adjust and optimize them as training iterates. Based on the performance of the model, continue to iterate the training to achieve the required performance indicators.
[0208] In addition, after analyzing the historical data using the risk assessment algorithm, the method further includes:
[0209] The accuracy of risk prediction is improved through stepwise annealing and random sampling algorithms, including:
[0210] Using simulated withdrawal technology to optimize the decision-making process of the risk assessment algorithm;
[0211] Simulated annealing is a heuristic search algorithm that solves optimization problems by imitating the annealing process in the physical process. In the optimization of the DQN model, it can help jump out of the local optimum and find a better solution in the global search.
[0212] Detailed steps:
[0213] Initialization temperature and decay rate: Set a starting high temperature and determine a decay rate to gradually reduce the temperature as the iterations proceed.
[0214] Select an initial solution: Randomly or strategically select an initial set of DQN parameters as the current solution.
[0215] Iterative optimization process: In each iteration, small random changes are made to the current solution and the cost function of the new solution (such as the increase in expected reward) is calculated.
[0216] Accept the new solution according to probability: if the new solution is better, accept the new solution; if the cost function of the new solution is higher, still accept the new solution with a certain probability, and the probability is related to the temperature and cost difference.
[0217] Lower the temperature: As the iteration proceeds, the temperature is gradually lowered to reduce the probability of accepting a bad solution, thereby converging to the global optimal solution or approximate solution.
[0218] Monte Carlo simulation method is used to test the robustness of the risk assessment algorithm in diverse environments;
[0219] Monte Carlo simulation is a numerical calculation method based on probability statistics. In risk assessment, this method can be used to evaluate the performance of algorithms in different scenarios.
[0220] The detailed steps include:
[0221] Simulation environment construction: Create a variety of simulated customer service environments to reflect the diverse scenarios that may be encountered in reality.
[0222] Run simulations: Run multiple model tests based on probability distributions and estimate the performance of the algorithm through these independent simulation experiments.
[0223] Collect data: Summarize the results of experiments, especially focusing on the performance of the algorithm under extreme conditions.
[0224] Robustness evaluation: Evaluate the average performance and robustness of the DQN strategy based on data, that is, the consistency of performance in various environments.
[0225] Based on the test results, the parameters of the risk assessment algorithm are adjusted.
[0226] Analyze simulated data: Identify environmental factors that make a big difference in algorithm performance.
[0227] Parameter fine-tuning: Adjust the parameters of DQN, such as exploration rate, discount factor, learning rate, etc., according to the simulation results.
[0228] Repeat simulation: The adjusted model is subjected to Monte Carlo simulation again to verify the effect of the adjusted parameters.
[0229] Continue to optimize: If the performance of the adjusted model is significantly improved, fix these parameters. If the effect is not obvious or the model performance decreases, consider other parameter combinations or optimization strategies.
[0230] In summary, the embodiments of the present invention can be applied to various industries, taking the e-commerce platform as an example:
[0231] 1. Monitoring and early warning of abnormal purchasing behavior
[0232] Customer A makes a large number of high-value purchases in a short period of time, which is very different from his past purchase history. The system immediately identifies the abnormal pattern and triggers the risk assessment algorithm.
[0233] In this case, the system obtains the optimal strategy through DQN, which may be to temporarily freeze the order and send a verification request to customer A to confirm the authenticity of the order, or to notify the risk management team for manual review.
[0234] 2. Inventory and order flow management during promotion period
[0235] Before a large-scale promotion, e-commerce platform B systematically evaluates the incoming traffic and possible order volume.
[0236] The underlying decision model calculates the possible risk of insufficient inventory and automatically adjusts the predicted inventory threshold. If the risk assessment algorithm predicts that inventory loss is too fast, the system automatically places an order for replenishment or adjusts the promotion strategy to prevent out-of-stock.
[0237] 3. Monitoring and handling of negative feedback in customer service interactions
[0238] Customer C expressed dissatisfaction with product quality while chatting with customer service. The underlying decision model captured this risk signal and performed a detailed sentiment analysis on it.
[0239] The system determines the best response strategy based on the DQN algorithm evaluation: it may be to provide an immediate price reduction to customer C to compensate for his dissatisfaction, or to report the situation to the senior customer service team to provide a more personalized service solution.
[0240] 4. Personalized recommendations and risk prevention
[0241] Through customer D’s historical purchase data, the system learns the customer’s preference for a certain type of goods and predicts future purchase trends based on the HMDP model.
[0242] If the early warning system identifies a trend that may lead to inventory backlogs, it will notify the supply chain in advance to adjust procurement plans, or push promotional activities to these customers in advance through personalized recommendation algorithms to balance supply and demand and reduce risks.
[0243] This application provides a customer service risk assessment and early warning method based on historical data. By establishing a risk assessment model based on historical data and combining it with a two-layer decision-making framework, dynamic risk assessment and real-time early warning feedback are achieved. Among them, the top-level model is responsible for scenario analysis and macro-decision-making of risks, while the lower-level model formulates detailed emergency response plans for specific situations. It can not only provide personalized risk management services, but also improve the intelligence and automation level of risk management. In terms of enterprise data and archive management, it can strengthen enterprise data security protection and archive management level internally, and improve customer service quality and user experience externally, providing strong support for enterprise intelligent transformation and sustainable development, and ultimately providing enterprises and customers with a more multi-level, reliable and efficient customer service experience.
[0244] Embodiment 2
[0245] Figure 2 A schematic diagram of a customer service risk assessment and early warning system based on historical data provided in an embodiment of the present disclosure, the customer service risk assessment and early warning system based on historical data 200 comprises:
[0246] Collection module 201, used to collect historical data of customer service;
[0247] A construction module 202 is used to construct a two-level Markov decision process HMDP model; based on the HMDP model, a risk assessment algorithm is constructed;
[0248] The analysis module 203 is used to analyze the historical data using the risk assessment algorithm so as to assess and warn the risks of customer service.
[0249] The construction module 201 is used to construct the HMDP model, which specifically includes:
[0250] Constructing a top-level decision model of HMDP, wherein the top-level decision model is used to identify and classify different customer service scenarios;
[0251] The underlying decision model of HMDP is constructed, and the underlying decision model is used to construct specific risk assessment and response plans.
[0252] The construction module 201 is used to construct the top-level decision model of HMDP, including:
[0253] Categorize historical service scenarios;
[0254] According to the classified historical service scenarios, the corresponding risk factors are integrated to construct the environment state, state transition probability matrix and reward function;
[0255] A greedy approach is used to construct the decision strategy.
[0256] The construction module 201 is used to construct the underlying decision model of HMDP, including:
[0257] Define fine-grained actions for response plans;
[0258] Design reward and penalty criteria for each of the fine-grained actions.
[0259] Optionally, the risk assessment algorithm is a deep Q network DQN method, then based on the HMDP model, a risk assessment algorithm is constructed, including:
[0260] Obtaining the environment state S, the fine-grained action set A of the response plan, and the reward function R defined by the HMDP model;
[0261] Initialize the Q value table.
[0262] The analysis module 203 is used to analyze the historical data using the risk assessment algorithm, including:
[0263] Classify the historical data into service scenarios, and obtain the environment state St of the first scenario at time t;
[0264] Based on the Q value at time t, select an action that can maximize the expected reward from the fine-grained action set A of the response plan;
[0265] Execute the action that can maximize the expected reward, obtain the reward function Rt, and the state St+1 at time t+1;
[0266] According to the reward function Rt and the state St+1 at time t+1, the Bellman equation is used to update the Q value.
[0267] In addition, the system includes:
[0268] A training module, used to obtain a historical data training set, and use the historical data training set to perform model training on the DQN;
[0269] The validation module is used to perform parameter adjustment and DQN model validation, using grid search to find the best parameter combination;
[0270] The optimization module is used to optimize the DQN model.
[0271] Optionally, the system further comprises:
[0272] The prediction module is used to improve the accuracy of risk prediction through stepwise annealing and random sampling algorithms.
[0273] Optionally, the accuracy of risk prediction is improved through stepwise annealing and random sampling algorithms, including:
[0274] Using simulated withdrawal technology to optimize the decision-making process of the risk assessment algorithm;
[0275] Monte Carlo simulation method is used to test the robustness of the risk assessment algorithm in diverse environments;
[0276] Based on the test results, the parameters of the risk assessment algorithm are adjusted.
[0277] This application provides a customer service risk assessment and early warning system based on historical data. By establishing a risk assessment model based on historical data and combining it with a two-layer decision-making framework, dynamic risk assessment and real-time early warning feedback are achieved. Among them, the top-level model is responsible for scenario analysis and macro-decision-making of risks, while the lower-level model formulates detailed emergency response plans for specific situations. It can not only provide personalized risk management services, but also improve the intelligence and automation level of risk management, and ultimately provide enterprises and customers with a more multi-level, reliable and efficient customer service experience.
[0278] The system of the embodiments of the present disclosure can execute the method provided by the embodiments of the present disclosure, and the implementation principles are similar. The actions performed by each module in the system of each embodiment of the present disclosure correspond to the steps in the method of each embodiment of the present disclosure. For the detailed functional description of each module of the system, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.
[0279] The above is only an optional implementation method for some implementation scenarios of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the scheme of the present disclosure, other similar implementation methods based on the technical ideas of the present disclosure are also within the protection scope of the embodiments of the present disclosure.
Claims
1. A customer service risk assessment and early warning method based on historical data, characterized in that: The method comprises: Collect historical data on customer service; Construct a two-level Markov decision process HMDP model; Based on the HMDP model, construct a risk assessment algorithm; The risk assessment algorithm is used to analyze the historical data so as to assess and warn of the risks of customer service.
2. The method according to claim 1, characterized in that Construct the HMDP model, including: Constructing a top-level decision model of HMDP, wherein the top-level decision model is used to identify and classify different customer service scenarios; The underlying decision model of HMDP is constructed, and the underlying decision model is used to construct specific risk assessment and response plans.
3. The method according to claim 2, characterized in that Construct the top-level decision model of HMDP, including: Categorize historical service scenarios; According to the classified historical service scenarios, the corresponding risk factors are integrated to construct the environment state, state transition probability matrix and reward function; A greedy approach is used to construct the decision strategy.
4. The method according to claim 2, characterized in that: Construct the underlying decision model of HMDP, including: Define fine-grained actions for response plans; Design reward and penalty criteria for each of the fine-grained actions.
5. The method according to claim 1, characterized in that: The risk assessment algorithm is a deep Q network DQN method, and based on the HMDP model, a risk assessment algorithm is constructed, including: Obtaining the environment state S, the fine-grained action set A of the response plan, and the reward function R defined by the HMDP model; Initialize the Q value table.
6. The method according to claim 5, characterized in that Analyzing the historical data using the risk assessment algorithm includes: The historical data is classified into service scenarios, and the environment state S of the first scenario at time t is obtained. t; Based on the Q value at time t, select an action that can maximize the expected reward from the fine-grained action set A of the response plan; Execute the action that can maximize the expected reward and obtain the reward function R t , and the state S at time t+1 t+1 ; According to the reward function R t and the state S at time t+1 t+1 , use the Bellman equation to update the Q value.
7. The method according to claim 5, characterized in that Before analyzing the historical data using the risk assessment algorithm, the method further includes: Obtain a historical data training set, and use the historical data training set to perform model training on the DQN; Perform parameter tuning and DQN model validation, using grid search to find the best parameter combination; The DQN model is optimized.
8. The method according to claim 1, characterized in that After analyzing the historical data using the risk assessment algorithm, the method further includes: The accuracy of risk prediction is improved through stepwise annealing and random sampling algorithms.
9. The method according to claim 8, characterized in that The accuracy of risk prediction is improved through stepwise annealing and random sampling algorithms, including: Using simulated withdrawal technology to optimize the decision-making process of the risk assessment algorithm; Monte Carlo simulation method is used to test the robustness of the risk assessment algorithm in diverse environments; Based on the test results, the parameters of the risk assessment algorithm are adjusted.
10. A customer service risk assessment and early warning system based on historical data, characterized in that: include: Collection module, used to collect historical data of customer service; Building module, used to build a two-level Markov decision process HMDP model; Based on the HMDP model, construct a risk assessment algorithm; The analysis module is used to analyze the historical data using the risk assessment algorithm so as to assess and warn of the risks of customer service.
Citation Information
Patent Citations
Incoming call service prediction method and system based on service map and Markov model
CN112508295A
Product recommendation method and device, computer equipment and storage medium
CN113256390A
Loan management strategy determination method and system, and electronic equipment
CN113888304A
Electric vehicle charging guide strategy method based on hierarchical deep reinforcement learning
CN114117910A
Power distribution network risk dynamic early warning method, system and device and storage medium
CN114219045A
