Power supply customer service hierarchical classification decision model construction method and system based on reinforcement learning

Abstract

The application discloses a power supply customer service grading classification decision model construction method and system based on reinforcement learning, and relates to the intelligent power customer service technical field; the method comprises the following steps: collecting historical excellent handling cases, extracting keywords by using a TF-IDF algorithm, and combining expert annotation to construct a grading classification expert strategy library; defining a four-dimensional state space, which comprises BERT semantic features, LDA deep appeal labels, AHP risk levels and processing links; defining an action space, which comprises recommended handling actions and reply speech templates; designing a reward function which comprehensively considers the artificial adoption rate, customer satisfaction, complaint escalation and handling timeliness; using a PPO algorithm to perform strategy iteration optimization, and realizing the fusion of the expert strategy and the reinforcement learning strategy through a three-stage progressive fusion mechanism; and outputting a complete processing scheme comprising grading results, classification results, recommended handling actions and reply speech templates after the strategy converges. The application realizes the standardization, intelligentization and continuous optimization of power supply customer service work order handling.

Description

Technical Field

[0001] This invention relates to the field of intelligent power customer service technology, and in particular to a method and system for constructing a hierarchical and classification decision-making model for power customer service based on reinforcement learning, which is used for intelligent hierarchical, classification and handling decisions of power customer service work orders. Background Technology

[0002] The power supply customer service sector handles a large volume of customer requests, including power outage repairs, electricity bill inquiries, fault complaints, and new energy business inquiries. This involves a large volume of work orders, complex service types, and diverse processing scenarios. Currently, power supply companies rely heavily on human experience for decision-making in handling customer service work orders, which presents the following technical challenges:

[0003] (i) Expert tacit knowledge is difficult to quantify and reuse. Senior customer service personnel have accumulated a lot of excellent experience in classifying and handling work orders over a long period of time. However, this experience mostly exists in the form of tacit knowledge in their personal minds. There is a lack of a systematic extraction, quantification and reuse mechanism, which makes it difficult to promote excellent handling experience on a large scale. The handling level of different customer service personnel varies greatly, and the quality of work order handling is uneven.

[0004] (ii) Lack of quantitative standards for work order classification. The existing work order classification relies heavily on subjective human judgment and lacks a unified risk quantification indicator system and classification threshold standards. This can easily lead to high-risk work orders being delayed in processing due to failure to be identified in a timely manner, while low-risk work orders occupy valuable resources, affecting overall service efficiency.

[0005] (iii) Lack of dynamic optimization mechanism for handling decisions. Existing decision-making methods are mostly static rules, which cannot dynamically adjust handling strategies based on real-time signals such as customer feedback and handling effects, making it difficult to continuously improve customer satisfaction.

[0006] (iv) Existing intelligent solutions have limitations. Some power supply companies have attempted to introduce intelligent decision-making models, but they mostly use traditional machine learning algorithms such as decision trees and logistic regression, which can only achieve simple work order classification or priority ranking, and cannot output complete handling decision-making schemes that include hierarchical classification, handling actions, and script templates. At the same time, existing solutions have not solved the technical problems of low decision reliability and high exploration risk in the cold start phase of reinforcement learning in actual implementation. In addition, existing technologies lack effective preprocessing methods for industry-specific noise such as colloquial expressions, typos, and professional abbreviations in work order texts, which affects the accuracy of the model's understanding of work order semantics.

[0007] In summary, there is an urgent need for a method to construct a hierarchical and classified decision-making model for power supply customer service that can transform tacit expert experience into reusable strategies, accurately quantify work order risks, support continuous dynamic optimization of handling decisions, and solve the reliability problem of cold start in reinforcement learning. Summary of the Invention

[0008] In view of the problems that the above or existing expert experience is difficult to reuse, the lack of quantitative standards for work order classification, the inability to dynamically optimize handling decisions, and the low reliability of decisions in the cold start phase of reinforcement learning, this invention proposes a method and system for constructing a power supply customer service classification decision model based on reinforcement learning.

[0009] To address the aforementioned technical problems, this invention provides the following technical solution: a method and system for constructing a hierarchical and classification decision-making model for power supply customer service based on reinforcement learning, comprising the following steps:

[0010] Step S1: Collect historical outstanding handling cases in the power supply customer service field, extract keywords using the TF-IDF algorithm, and extract expert experience rules by combining expert annotations to construct a hierarchical and classified expert strategy library P. The expert experience rules are in the form of "IF condition THEN hierarchical result, classification result, handling action".

[0011] Step S2: Define the reinforcement learning state space S;

[0012] The state space S consists of four dimensions: work order semantic features X1, deep-level request tags X2, risk level X3, and processing steps X4, i.e., S = [X1, X2, X3, X4]. T ;

[0013] The semantic feature X1 of the work order is extracted by encoding the work order text using a BERT pre-trained model;

[0014] The deep-level appeal tag X2 was obtained using the LDA topic model.

[0015] The risk level X3 is calculated using the Analytic Hierarchy Process (AHP) and mapped to three levels: high, medium, and low.

[0016] The processing stage X4 represents the code of the current processing stage of the work order;

[0017] Step S3: Define a reinforcement learning action space A, which includes a set of recommended actions A1 and a set of response script templates A2, i.e., A = A1∪A2;

[0018] The recommended action set A1 includes four core power supply customer service actions: escalation of supervision, on-site verification, collaborative handling, and routine handling.

[0019] The reply script template set A2 contains standardized reply script templates designed for different business types and demand scenarios;

[0020] Step S4: Design a reinforcement learning reward function R(s,a,s′), which comprehensively considers four dimensions: manual adoption rate, customer satisfaction, whether the complaint is escalated, and processing time.

[0021] The expression is R(s,a,s′)=α·r1+β·r2+γ·r3+δ·r4, where α+β+γ+δ=1, r1 is the reward for manual adoption, r2 is the reward for customer satisfaction, r3 is the reward for escalation of complaints, and r4 is the reward for timely handling.

[0022] Step S5: Iteratively optimize the policy using the Proximal Policy Optimization (PPO) algorithm, with the expert policy library P as the initial policy. A three-stage progressive integration mechanism is defined, comprising a cold start stage, a transition stage, and a stabilization stage.

[0023] During the cold start phase, decision-making relies entirely on the expert strategy library;

[0024] During the transition phase, the expert strategy and the reinforcement learning strategy are weighted and fused according to dynamic weights.

[0025] During the stable phase, reinforcement learning strategies dominate decision-making, while expert strategies serve as a safety net.

[0026] Step S6: Iterate training until the policy converges, output the optimal policy π(a|s). For the input power customer service work order to be processed, extract its state features s, select the optimal action a, and output a complete processing solution including work order classification results, work order categorization results, recommended processing actions and reply script templates.

[0027] Furthermore:

[0028] In step S2, the semantic feature X1 of the work order is extracted using the BERT-base-chinese pre-trained model. The first 6 layers of the Transformer are frozen, the last 6 layers and the output layer are fine-tuned, and the 768-dimensional vector corresponding to the [CLS] label of the last hidden layer of BERT is taken as the semantic feature vector of the work order text.

[0029] Furthermore:

[0030] In step S2, the method for extracting the deep appeal tag X2 is as follows: set the number of topics in the LDA topic model K=8, the hyperparameters α=0.1 and β=0.01, the number of iterations is 1000, and the topic distribution θ is output for each work order text. d =(θ1,θ2,…,θ8), represented using one-hot encoding, with a dimension of 8;

[0031] The risk level X3 is calculated as follows: The Analytic Hierarchy Process (AHP) is used to determine the weights of each risk indicator, and the risk score Sr is calculated. It is then normalized to the [0,1] interval and mapped to three levels: high risk (Sr≥0.70), medium risk (0.30≤Sr<0.70), and low risk (Sr<0.30).

[0032] The processing stage X4 uses integer coding: 1 = acceptance, 2 = triage, 3 = handling, and 4 = follow-up visit.

[0033] 4. The method according to claim 3,

[0034] Its features are,

[0035] The risk assessment indicator system of the AHP method includes 5 primary indicators: scope of impact, urgency, public opinion risk, economic loss, and social impact, as well as 12 secondary indicators.

[0036] The AHP judgment matrix is ​​constructed using the Saaty 1-9 scaling method, the multi-expert judgment matrix is ​​aggregated using the geometric mean method, the weight calculation method uses the eigenvalue method, and the consistency ratio CR = CI / RI.

[0037] in, RI is a consistency index, which stands for random consistency index.

[0038] When CR < 0.10, the judgment matrix is ​​considered to have satisfactory consistency, and the weighting results are acceptable.

[0039] Furthermore:

[0040] In step S4, the reward function The four components are defined as follows:

[0041] r1 is the reward for human adoption. If the action recommended by the agent is adopted by humans, then r1=1; otherwise, r1=0.

[0042] r2 is the customer satisfaction reward, r2=score / 100, where score is the customer satisfaction rating (0-100 points).

[0043] r3 represents the penalty reward for escalating complaints. If no escalation occurs after action a is performed on the work order, r3 = 1; if an escalation occurs, r3 = -1.

[0044] r4 is a timeliness bonus, if the work order processing time t 处置 If ≤ standard time limit T, then Otherwise, r4 = -0.5;

[0045] The reward weights satisfy α=0.25, β=0.35, γ=0.3, δ=0.1, the discount factor γ0=0.99, and the cumulative reward... .

[0046] Furthermore:

[0047] In step S5, the three-stage progressive fusion mechanism is specifically implemented as follows:

[0048] The cold start phase: expert strategy weights =1.0, reinforcement learning strategy weight =0.0, decision-making relies entirely on the expert strategy library;

[0049] The transition phase: It decreases linearly from 1.0 to 0.3. The weights increase linearly from 0.0 to 0.7, and the formula for the weight change is: =min(0.7,(n−500) / 2500×0.7), where n is the number of processed work orders, and the final action selection probability is... , where δ is the indicator function;

[0050] When reinforcement learning recommends actions that seriously conflict with expert policies, the expert policy actions must be used.

[0051] The stable phase: reinforcement learning policy weights ∈(0.8,0.9), expert strategy weights ∈(0.1,0.2), reinforcement learning strategy dominates decision-making, while expert strategy serves as a safety net;

[0052] When the confidence level of the reinforcement learning strategy falls below a preset threshold, or when the probability of the reinforcement learning output action leading to an escalation of a complaint exceeds 10% in the past, the strategy reverts to the expert strategy.

[0053] Furthermore:

[0054] In step S5, the specific steps for iterative optimization of the policy using the near-end policy optimization (PPO) algorithm are as follows:

[0055] Step S5.1: Initialize the policy network parameters θ0 and the value network parameters φ0. The policy network is a three-layer fully connected network with an input dimension of 778 and an output dimension of 44. The value network is a three-layer fully connected network with an input dimension of 778 and an output dimension of 1.

[0056] Step S5.2: The agent, according to the current policy... Interact with the environment to collect samples Where N is the number of samples in each round, N=1000;

[0057] Step S5.3: Calculate the advantage function using Generalized Advantage Estimation (GAE):

[0058] ,

[0059] in, , The state value output by the value network. For GAE parameters, λ=0.95;

[0060] Step S5.4: Update the policy network parameters θ by minimizing the PPO-Clip objective function, where the objective function is... ,in, ε is the cutting factor, ε=0.2;

[0061] Step S5.5: Minimize the mean square error loss Update the value network parameter φ;

[0062] Step S5.6: Repeat steps S5.2 to S5.5 until the cumulative reward Gt stabilizes and the fluctuation range is ≤5%, and the strategy converges.

[0063] Furthermore:

[0064] In step S1, when collecting historical excellent handling cases, cases with empty work order text, missing handling results, or missing satisfaction scores are directly removed.

[0065] Outliers whose processing time exceeds the standard time limit by more than 10 times will be removed; outliers whose satisfaction scores exceed the range of 0-100 or are non-numerical will be removed.

[0066] The power work order text is standardized, including: converting colloquial expressions into standardized terms based on a standardized mapping table of power customer service work order terminology; correcting typos based on a power industry-specific error correction dictionary; unifying the expansion of abbreviations based on a table of commonly used abbreviations and full names for power supply customer service; and constructing an industry-specific stop word list by adding functional words for power customer service communication to the general Chinese stop word list.

[0067] Furthermore:

[0068] In step S6, the complete processing solution output adopts a structured JSON format and includes at least the following fields: work order classification result, including risk level and risk score;

[0069] The work order classification results include primary classification, secondary classification, and classification confidence level;

[0070] Recommended actions include action type, action code, priority, execution criteria, recommended responsible role, initial response time limit, total work order processing time limit, reasons for recommendation, and a list of alternative actions;

[0071] The response script template includes a template identifier, template text, a list of replaceable fields, and the source of the field values. The replaceable fields in the response script template are marked with double curly braces {{field name}} as placeholders, and each replaceable field includes the current value and the data source path.

[0072] A system for constructing a hierarchical classification decision-making model for power supply customer service based on reinforcement learning, comprising:

[0073] The data acquisition and preprocessing module is used to collect historical outstanding handling cases and real-time customer service work order data in the power supply customer service field, and to perform missing value processing, outlier processing and text normalization processing.

[0074] The expert strategy library construction module is used to extract keywords from excellent cases using the TF-IDF algorithm, combine expert annotations to extract expert experience rules, and construct a hierarchical and classified expert strategy library P.

[0075] The state space definition module is used to define the reinforcement learning state space S. The state space S consists of four dimensions: work order semantic features X1 extracted by the BERT model, deep appeal labels X2 mined by the LDA model, risk level X3 calculated by the Analytic Hierarchy Process (AHP), and processing link encoding X4.

[0076] The action space definition module is used to define the reinforcement learning action space A, which includes a set of recommended disposal actions A1 and a set of response script templates A2.

[0077] The reward function design module is used to design the reinforcement learning reward function R(s,a,s′). The reward function takes the human adoption rate, customer satisfaction, whether the complaint is escalated, and the timeliness of handling as the core reward signals and is implemented in a weighted summation form.

[0078] The policy iteration optimization module is used to perform policy iteration optimization using the proximal policy optimization PPO algorithm, and uses the expert policy library P as the initial policy. It achieves the fusion of expert policy and reinforcement learning policy through a three-stage progressive fusion mechanism.

[0079] The solution output module is used to output a complete processing solution based on the optimal strategy π*(a|s) obtained from training, including the work order classification result, recommended handling action and reply script template for the input power supply customer service work order to be handled.

[0080] The beneficial effects of this invention are as follows:

[0081] This invention combines TF-IDF keyword extraction with expert annotation to transform the implicit handling experience of senior experts in the power supply customer service field into explicit rules in the form of "IF condition THEN graded result, classification result, and handling action". It constructs a graded and classified expert strategy library, which effectively solves the technical problem that expert experience is difficult to systematically reuse and pass on in the existing technology, and enables excellent handling experience to be promoted on a large scale.

[0082] This invention employs the Analytic Hierarchy Process (AHP) to construct a risk assessment index system specifically for power supply customer service, comprising 5 primary indicators and 12 secondary indicators. A judgment matrix is ​​constructed using the Saaty 1-9 scaling method, and a consistency test (CR<0.10) is combined to ensure the weights are scientifically reasonable. After normalization, the risk scores are mapped to three levels: high (≥0.70), medium (0.30~0.70), and low (<0.30). This overcomes the technical shortcomings of traditional subjective judgment, such as inconsistent work order grading standards and untimely handling of high-risk work orders.

[0083] This invention constructs a reward function based on four dimensions: manual adoption rate, customer satisfaction, complaint escalation, and handling timeliness. It uses the PPO algorithm combined with GAE advantage estimation for strategy iteration, enabling the model to continuously optimize decision-making strategies based on real-time feedback signals. This results in an 8.6 percentage point increase in customer satisfaction (from 85.6% to 94.2%) and a 1.4 percentage point decrease in complaint escalation rate, overcoming the technical shortcomings of existing technologies where decision-making rules are fixed and cannot be dynamically adjusted based on customer feedback.

[0084] This invention reduces the time spent on manual decision-making from an average of 8.5 minutes per order to 2.1 minutes per order (a reduction of 6.4 minutes per order) through intelligent hierarchical and classification decision-making and standardized response template recommendations. It also increases the timeliness of handling work orders from 78.3% to 92.5% (an increase of 14.2 percentage points), thus achieving standardization and efficiency in handling power supply customer service work orders.

[0085] The "cold start → transition → stability" three-stage progressive fusion mechanism designed in this invention relies entirely on expert strategies to ensure decision reliability during the cold start stage (n<500). During the transition stage (500≤n<2500), smooth transition is achieved through linear weight decay. During the stability stage (n≥2500), reinforcement learning dominates decision-making while retaining expert strategies as a safety net (backoff when RL confidence is low or the probability of complaint escalation is >10%). This not only ensures the safety of decision-making in the early stages of model training but also enables continuous optimization of the strategy, solving the technical problems of difficult cold start and high exploration risk in the application of reinforcement learning in existing technologies.

[0086] This invention adopts a "feature-splitting input" strategy—BERT semantic vectors (768-dimensional) and LDA topic vectors (8-dimensional) are directly concatenated as input to the policy network, while TF-IDF keyword vectors (10,000-dimensional sparse) are not directly used in network training but are used to trigger expert rule matching. This effectively avoids the curse of dimensionality and semantic information dilution caused by directly concatenating high-dimensional sparse features with dense semantic features, thereby improving model training efficiency and decision accuracy. Attached Figure Description

[0087] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0088] Figure 1 This is a flowchart of the overall technical solution of the present invention;

[0089] Figure 2 This is a block diagram of the overall system architecture of the present invention;

[0090] Figure 3 This is a diagram illustrating the interaction framework of the reinforcement learning state space-action space-reward function of this invention.

[0091] Figure 4 This is a hierarchical structure diagram of the AHP risk assessment index system of this invention;

[0092] Figure 5 This is a graph showing the weight change of the three-stage progressive fusion mechanism of this invention.

[0093] Figure 6 This is a flowchart of the data preprocessing and text normalization process of this invention;

[0094] Figure 7 This is a schematic diagram of the JSON data structure of the output scheme of this invention. Detailed Implementation

[0095] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0096] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0097] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0098] Example 1

[0099] like Figure 1 As shown, this embodiment provides a method for constructing a hierarchical classification decision-making model for power supply customer service based on reinforcement learning, including the following steps:

[0100] First, we collected historical best practices in power supply customer service, extracted keywords using the TF-IDF algorithm, and combined these with expert annotations to extract expert experience rules, thus constructing a hierarchical and categorized expert strategy library P.

[0101] Specifically, customer service work order data from a municipal power supply company over the past three years were collected, and 10,000 outstanding cases with customer satisfaction ≥95%, no escalation of complaints, and timely handling were selected to form a collection of outstanding cases. , where n=10000.

[0102] Keywords for each case were extracted using the TF-IDF algorithm, and combined with annotations from five senior customer service experts, 800 experience rules were extracted and categorized into four types based on business type: power outage repair, electricity bill inquiry, fault complaint, and new energy business inquiry, thus constructing an expert strategy database. The rule format is "IF condition THEN graded result, classification result, and action".

[0103] Subsequently, the reinforcement learning state space is defined. X1 is the semantic feature of the work order. The work order text is encoded using the BERT-base-chinese pre-trained model. The first 6 Transformer layers are frozen, and the last 6 layers and the output layer are fine-tuned. The 768-dimensional vector corresponding to the [CLS] label of the last hidden layer of BERT is taken as the semantic feature vector.

[0104] X2 represents the deep-level appeal label, which is mined using the LDA topic model. The number of topics is set to K=8, hyperparameters α=0.1 and β=0.01, and the number of iterations is 1000. The output topic distribution θ is then determined. d =(θ1,θ2,…,θ8), represented using one-hot encoding, with a dimension of 8;

[0105] X3 represents the risk level, calculated using the Analytic Hierarchy Process (AHP) and normalized to the [0,1] interval, mapping it to high risk (S_r≥0.70), medium risk (0.30≤S_r<0.70), and low risk (S_r<0.30). X4 represents the processing stage, encoded using integers: 1=Acceptance, 2=Triage, 3=Disposal, 4=Follow-up. The total dimensions of the state space are 768+8+2=778.

[0106] Define the action space A = A1 ∪ A2. A1 is the set of recommended actions, containing a 11 Upgraded supervision, a 12 On-site verification, a 13 Collaborative processing, a 14 The standard procedure involves four actions; A2 is a collection of response templates, with 40 standardized templates designed for four different business scenarios, covering various needs and situations. The total output dimension of the action space is 4 + 40 = 44.

[0107] Design the reward function R(s,a,s′)=α·r1+β·r2+γ·r3+δ·r 4， Where α=0.25, β=0.35, γ=0.3, δ=0.1, and discount factor γ0=0.99. r1 is the reward for manual adoption; if the agent's recommended action is adopted, r1=1, otherwise r1=0.

[0108] r2 is the customer satisfaction reward, r2 = score / 100, where score is the customer satisfaction rating (0-100 points); r3 is the complaint escalation penalty reward, r3 = 1 if no complaint escalation occurs, and r3 = -1 if a complaint escalates; r4 is the processing time reward, if t 处置 ≤T Otherwise, r4 = −0.5. The standard time limit T is set according to the business type (power outage repair T = 2 hours, electricity bill consultation T = 30 minutes, fault complaint T = 1 hour, new energy business consultation T = 1 hour).

[0109] The PPO algorithm is used for iterative policy optimization. The expert policy library P is used as the initial policy. Initialize the policy network parameters θ0 and the value network parameters φ0.

[0110] The policy network is a three-layer fully connected network with 778 input dimensions and 44 output dimensions; the value network is also a three-layer fully connected network with 778 input dimensions and 1 output dimension. A three-stage progressive fusion mechanism is defined: a cold start stage (n<500). =1.0, =0.0, completely dependent on expert strategy; transition phase (500≤n<2500). =min(0.7,(n−500) / 2500×0.7), =1.0−α_rl, weighted fusion decision; stable phase (n≥2500). ∈(0.8,0.9), ∈(0.1,0.2), RL strategy is dominant, and expert strategy is the backup.

[0111] The agent, according to the current policy Interact with the environment to collect samples Each round of sampling N=1000.

[0112] Calculating the dominance function using GAE ,

[0113] in, , λ=0.95.

[0114] By minimizing the PPO-Clip objective function:

[0115] ;

[0116] Update the policy network, where ε = 0.2;

[0117] By minimizing the mean square error loss Update the value network.

[0118] Iterate through the training for 50 rounds until the cumulative reward G_t stabilizes (fluctuation range ≤ 5%), the policy converges, and the optimal policy π*(a|s) is output.

[0119] For each incoming power supply customer service work order to be processed, extract its status feature s, select the optimal action a*, and output a complete processing solution in structured JSON format: work order classification result (risk level and risk score, including the top 3 risk factors), work order classification result (primary classification + secondary classification + classification confidence), recommended processing action (action type, code, priority, execution standard, responsible role, response time limit, SLA, recommendation reason, alternative action list), and reply script template (template identifier, template text, list of replaceable fields including field values ​​and source paths).

[0120] Example 2

[0121] like Figure 2 As shown in the figure, this embodiment provides a power supply customer service hierarchical classification decision-making model construction system based on reinforcement learning. The system includes a data acquisition and preprocessing module, an expert strategy library construction module, a state space definition module, an action space definition module, a reward function design module, a strategy iteration optimization module, and a scheme output module.

[0122] The data acquisition and preprocessing module is responsible for collecting historical excellent handling case data, real-time customer service work order data, expert-annotated data, and parameter configuration data from the power supply company's customer service system and historical database. It then performs missing value handling (empty text, missing handling results, and missing satisfaction scores are directly removed), outlier handling (handling time > 10T, and satisfaction scores exceeding the 0-100 range are removed), and text normalization. The selection criteria for historical excellent handling cases are customer satisfaction ≥ 95%, no escalation of complaints, and meeting handling timeliness standards.

[0123] The expert strategy library construction module uses the TF-IDF algorithm to extract keywords from excellent cases. The vocabulary size is limited to 10,000 words, and the n-gram range is (1,2). The top 30 words with TF-IDF scores in a single document are taken as keywords. Combined with expert annotations, rules in the form of "IF condition THEN graded result, classification result, and action" are extracted and stored according to business type to construct a graded and classified expert strategy library P.

[0124] The state space definition module defines the state space S=[x1,x2,x3,x4]ᵀ∈R. 778 The semantic features of the work order x1 are extracted by the BERT-base-chinese model, outputting a 768-dimensional semantic vector; the deep appeal label x2 is mined by the LDA model (K=8, α=0.1, β=0.01, 1000 iterations), with 8-dimensional one-hot encoding; the risk level x3 is calculated by the AHP method to normalize the risk score and map it to three levels: high / medium / low; the processing stage x4 is an integer encoding of the four stages of acceptance / diversion / disposal / follow-up.

[0125] The action space definition module defines the action space A=A1∪A2. A1 contains four recommended actions: escalation of supervision, on-site verification, collaborative processing, and routine handling. A2 contains 40 standardized response script templates.

[0126] The reward function design module designs the reward function:

[0127] R(s,a,s′)=α·r1+β·r2+γ·r3+δ·r4, discount factor γ0=0.99.

[0128] r1 is the reward for manual adoption (adoption = 1, non-adoption = 0);

[0129] r2 = score / 100 (score is 0-100 points);

[0130] r3 is the penalty for escalating a complaint (no escalation = 1, escalation = −1).

[0131] r4 is a reward for timely handling (t) 处置 When ≤T= Otherwise = −0.5).

[0132] The strategy iteration optimization module adopts the PPO algorithm, with the expert policy library P as the initial policy, calculates the advantage function through GAE (λ=0.95), updates the policy network through PPO-Clip (ε=0.2), and updates the value network through mean squared error loss.

[0133] Simultaneously, a three-stage progressive integration mechanism is implemented (cold start with 100% experts → transitional linear decay → stable RL-led + expert backup).

[0134] The solution output module, based on the optimal strategy π*(a|s), outputs a complete processing solution in structured JSON format for the input power supply customer service work orders to be processed.

[0135] The modules work together through data flow and control flow to form a complete power supply customer service hierarchical classification decision-making model construction system.

[0136] Example 3

[0137] like Figure 3 As shown, this embodiment details the interaction framework of the reinforcement learning state space, action space, and reward function.

[0138] At each time step t, the agent obtains the current state s_t∈S of the power supply customer service work order from the environment. The state st is composed of four dimensions: the work order semantic feature x1 is a 768-dimensional dense vector, obtained by the [CLS] label output of the BERT-base-chinese model, capturing the deep semantic information of the work order text; the deep appeal label x2 is an 8-dimensional one-hot vector, output by the LDA topic model, corresponding to 8 categories of deep appeal topics (fault repair, fee inquiries, business processing, complaint and rights protection, new installation and capacity expansion, power quality, safety hazards, opinions and suggestions);

[0139] Risk level X3 is a continuous value in the interval [0,1], calculated by weighted average using the AHP method: high risk ≥ 0.70, medium risk 0.30~0.70, and low risk < 0.30. Processing stage X4 is an integer code of {1,2,3,4}, corresponding to the four stages of acceptance, triage, handling, and follow-up. The total dimension of the state space is 778.

[0140] The agent selects action at from action space A based on the current policy π(a|s). Action space A contains a joint decision involving two branches: the recommended action branch selects one from four options A1 = {escalation of supervision, on-site verification, collaborative handling, and routine handling}; the response template branch selects one from 40 standardized templates in A2. The total action output dimension is 44 dimensions, with the recommended action outputting a 4-dimensional probability distribution and the response template outputting a 40-dimensional probability distribution.

[0141] After the environment performs the action at, the comprehensive reward signal R(st,at,s{t+1}) is calculated and fed back to the agent. The reward signal is obtained by weighted summation of four components: human adoption reward r1 (1 for adoption, 0 for non-adoption), reflecting the customer service personnel's degree of acceptance of the model's recommendation;

[0142] The customer satisfaction reward r2 = score / 100 (score∈[0,100]) reflects the customer's satisfaction with the handling result; the complaint escalation penalty reward r3 (+1 for no escalation, −1 for escalation) reflects whether the handling effectively controls the risk; the handling timeliness reward r4 (1−t_handling / T for meeting the target, −0.5 for exceeding the time limit) reflects the handling efficiency. The four components are weighted and summed according to the weights α=0.25, β=0.35, γ=0.3, and δ=0.1 to obtain the comprehensive reward R.

[0143] Simultaneously, the environment transitions to the next state s{t+1}, forming a complete interactive loop of "state → action → reward → next state"; the agent maximizes the cumulative discounted reward.

[0144] With γ0=0.99 as the target, the strategy is continuously optimized using the PPO algorithm.

[0145] Example 4

[0146] like Figure 4 As shown in the figure, this embodiment details the hierarchical structure and weight determination method of the AHP risk assessment index system.

[0147] The target layer of the risk assessment indicator system is the "risk score S_r of power supply customer service work orders".

[0148] The criteria layer comprises five primary indicators: C1 Scope of Impact (weight 0.25), C2 Urgency Level (weight 0.30), C3 Public Opinion Risk (weight 0.20), C4 Economic Loss (weight 0.15), and C5 Social Impact (weight 0.10). Each primary indicator has 12 secondary indicators.

[0149] The scope of influence C1 includes:

[0150] C11 affects the number of users (1 point ≤ 10 users, 3 points 11-50 users, 5 points 51-200 users, 7 points 201-1000 users, 9 points 1001-10000 users, 10 points > 10000 users), C12 affects the geographical area, and C13 involves the line / equipment level.

[0151] The emergency level C2 is further divided into: C21 emergency level of the fault (involving personal safety or large-scale power outages, etc.) and C22 timeliness requirements.

[0152] Public opinion risk C3 includes: C31 Complaint tendency (whether there is a tendency to escalate complaints or media exposure) and C32 Number of repeat calls.

[0153] Economic losses C4 are further divided into: C41 direct economic losses (estimated power loss or equipment damage caused by power outages) and C42 indirect economic impacts (estimated impacts on the production and operation of commercial users).

[0154] The social impact C5 is further divided into: C51 Impact on important users (whether it involves important users such as government agencies, hospitals, and schools), C52 Sensitive time period factors (whether it occurs during sensitive periods such as major events, holidays, and extreme weather), and C53 Public safety impact (whether there are public safety hazards such as electric shock and fire).

[0155] Each secondary indicator is scored using a tiered quantitative standard of 1-10 points, forming the "Power Supply Customer Service Work Order Risk Assessment Indicator Quantitative Standard Manual".

[0156] The weighting method is as follows: Five senior experts in the power supply customer service field were invited to use the Saaty 1-9 scale to compare the importance of indicators at the same level pairwise, constructing a judgment matrix. 1 = equally important, 3 = slightly important, 5 = significantly important, 7 = strongly important, 9 = extremely important, and 2 / 4 / 6 / 8 are median values.

[0157] The judgment matrix from multiple experts is aggregated using the geometric mean method:

[0158] , where m is the number of experts.

[0159] The weights of the indicators are calculated using the eigenvalue method: the maximum eigenvalue λmax of the judgment matrix A and its corresponding eigenvector w are calculated, and the weights of each indicator are obtained after normalizing the eigenvector.

[0160] Calculate the consistency ratio CR = CI / RI, where, RI is the random consistency index (RI=1.12 when n=5). When CR<0.10, the judgment matrix has satisfactory consistency.

[0161] If CR ≥ 0.10, review the comparison items in the judgment matrix that are clearly inconsistent, organize experts to re-discuss and revise, and recalculate CR until the standard is met; if the standard is still not met after multiple revisions, exclude the judgment matrix of that expert.

[0162] Ultimately, the work order risk score Sr = Where q is the total number of risk indicators, wi is the weight of the i-th risk indicator, and si is the score of the i-th risk indicator. After the scores are normalized to the interval [0,1], they are mapped to three levels: high risk (Sr≥0.70), medium risk (0.30≤Sr<0.70), and low risk (Sr<0.30).

[0163] Example 5

[0164] like Figure 3 and combined Figure 4 As shown, this embodiment details the specific calculation process of the reward function.

[0165] For any time step t, after the agent selects action a_t in state s_t, the environment calculates the reward function R(s_t, a_t, s_{t+1}) based on the actual handling result of the work order. Specifically, taking a power outage work order as an example:

[0166] Suppose the agent recommends the action "on-site verification" (a 12 If the recommendation is adopted by customer service personnel, then r1=1. If the customer follow-up score after the work order is processed is 92 points, then r2=92 / 100=0.92.

[0167] If no complaint escalation occurs after the work order is executed, then r3 = 1. If the actual processing time t for the work order... 处置 =1.5 hours, while the standard time limit for power outage repair is T=2 hours, so r4=1−1.5 / 2=0.25.

[0168] Overall Rewards:

[0169] R=0.25×1+0.35×0.92+0.3×1+0.1×0.25=0.25+0.322+0.3+0.025=0.897.

[0170] If the recommended action for the same work order is "routine processing" (a 14 If the suggestion is not adopted (the customer service representative changes it to "on-site verification"), then r1=0; if the customer satisfaction score is 70 points, then r2=0.7.

[0171] If no complaint escalation occurs, r3=1; if the processing time is 1.5 hours ≤ 2 hours, r4=0.25. Then R=0.25×0+0.35×0.7+0.3×1+0.1×0.25=0+0.245+0.3+0.025=0.570.

[0172] Lower reward values ​​guide the agent to learn to prioritize the "on-site verification" action, which has a higher probability of being adopted, in similar situations.

[0173] If the action leads to an escalation of the complaint, then r3 = −1, and R will significantly decrease or even become negative, thus penalizing the agent. If the handling time exceeds the standard time limit (e.g., t_handling = 3 hours, T = 2 hours), then r4 = −0.5, and the same penalty is imposed.

[0174] The agent's goal is to maximize the cumulative discount reward:

[0175] , where γ0=0.99, makes the agent not only focus on the current reward when making decisions, but also pay more attention to the long-term cumulative effect.

[0176] Example 6

[0177] like Figure 5 As shown, this embodiment details the implementation process of the three-stage progressive fusion mechanism.

[0178] In this embodiment, the expert strategy weight is defined as follows: The weights of the reinforcement learning strategy are Both satisfy + =1. The three-stage division is based on the number of processed work orders, n, as the horizontal axis:

[0179] Cold start phase (0≤n<500): =1.0, =0.0. The agent makes no autonomous decisions, relying 100% on rule actions matched from the expert policy library P as recommended outputs. This stage ensures that the model can still output reliable action suggestions even in the early stages of training when there is no accumulation of high-quality data, avoiding high-risk decision-making errors caused by random exploration.

[0180] Transition phase (500≤n<2500): =min(0.7,(n−500) / 2500×0.7), =1.0−α_rl.

[0181] The weight of the expert strategy decreases linearly from 1.0 to 0.3, while the weight of the reinforcement learning strategy increases linearly from 0.0 to 0.7.

[0182] The probability of choosing the final action is:

[0183] , where δ is the indicator function, a e xpert is the recommended action determined by the expert strategy.

[0184] When the action recommended by RL seriously conflicts with the expert strategy (e.g., RL recommends "routine handling" while the expert determines "escalation of supervision"), the expert strategy action is enforced to ensure safety. During this phase, the RL strategy gradually accumulates experience, while the expert strategy continuously provides safety constraints.

[0185] Stable phase (n≥2500): ∈(0.8,0.9), ∈(0.1,0.2). The reinforcement learning strategy dominates the decision-making process, while the expert strategy serves as a safety net. When the confidence of the RL strategy falls below a preset threshold (e.g., the probability of the highest action <0.6), or when the probability of the RL output action leading to escalation of complaints exceeds 10% historically, the decision reverts to the expert strategy. During the stable phase, a small weight of the expert strategy is retained to ensure that expert experience always participates in the decision-making process and prevents the RL strategy from "strategy drift" due to environmental changes.

[0186] Example 7

[0187] like Figure 3 and Figure 1 As shown in the figure, this embodiment details the complete process of strategy iterative optimization using the PPO algorithm.

[0188] Initialization Phase: The expert policy base P is used as the initial policy π0(a|s), and the policy network parameters θ0 and value network parameters φ0 are initialized. The policy network is a three-layer fully connected feedforward neural network: Input layer Flatten (778 dimensions) → Hidden layer 1 Dense (256 dimensions, ReLU activation, Dropout rate 0.2) → Hidden layer 2 Dense (128 dimensions, ReLU activation, Dropout rate 0.2) → Output layer, with two branches—a 4-dimensional recommended action output (Softmax activation) and a 40-dimensional response template output (Softmax activation). The value network is also a three-layer fully connected feedforward neural network: Input layer Flatten (778 dimensions) → Hidden layer 1 Dense (256 dimensions, ReLU activation, Dropout rate 0.2) → Hidden layer 2 Dense (128 dimensions, ReLU activation, Dropout rate 0.2) → Output layer Dense (1 dimension, Linear, no activation). The policy network has a learning rate of 3e-4 (linearly decaying to 1e-5), the value network has a learning rate of 1e-3 (linearly decaying), and both optimizers are Adam (β1=0.9, β2=0.999, ε=1e-8). The maximum norm of gradient clipping is 0.5.

[0189] Interactive sampling phase: The agent samples state s in state space S according to the current policy πθ(a|s), selects action a, executes the action and obtains reward R(s,a,s′), then transitions to the next state s′ and collects samples. Each round of sampling N=1000.

[0190] Advantage function calculation stage:

[0191] Calculating the dominance function using GAE ,in, γ0=0.99, λ=0.95, and V_φ(s) is the state value output by the value network.

[0192] Policy update phase: Update the policy network parameters θ by minimizing the PPO-Clip objective function.

[0193] The objective function is

[0194] ,

[0195] in, The probability ratio is ε = 0.2.

[0196] The policy network is updated 10 times after each round of sampling. At the same time, a KL divergence penalty coefficient β=0.01 is introduced. When the actual KL divergence exceeds 1.5 times the target value of 0.01, β is doubled, and when it is less than 1 / 1.5 of the target value, β is halved.

[0197] Value network update phase:

[0198] By minimizing the mean square error loss Update the value network parameter φ.

[0199] Iteration Termination Determination: Repeat the above steps until the cumulative reward Gt tends to stabilize and the fluctuation range is ≤5%, the policy converges, and the optimal policy π*(a|s) is output. In this embodiment, the policy converges after 50 rounds of iterative training.

[0200] Example 8

[0201] like Figure 6 As shown, this embodiment details the data preprocessing and text normalization process.

[0202] After the original work order data is input, a missing value check is performed first: If the work order text is empty, it is directly removed—the work order text is the only input source for BERT semantic feature extraction; empty text cannot be encoded or extracted for keywords, and the work order has no modeling value; if the handling result is missing, it is directly removed—the handling result is the core basis for expert strategy extraction; without the handling result, the optimal action cannot be determined; if the satisfaction score is missing, it is directly removed—satisfaction is one of the core conditions for selecting excellent cases (≥95%); if it is missing, it cannot be determined whether it is an excellent case. If any of the above missing conditions are met, it is marked as invalid data and removed from the set of excellent cases.

[0203] Data that passes the missing value test is then used for outlier detection: If the processing time exceeds 10 times the standard time T for the corresponding business type (i.e., actual processing time > 10T), it is marked as an outlier and removed. This situation is judged as a data recording error or an extreme anomaly; if the satisfaction score exceeds the range of 0-100 or is not a numerical value, it is marked as an outlier and removed.

[0204] The cleaned data then undergoes text standardization processing. First, based on a standardized mapping table of terminology used in power customer service work orders, colloquial expressions are converted into standardized terminology. For example, "the light is not on," "there's no electricity," and "the circuit breaker tripped" are uniformly converted to "power outage," and "how is the electricity bill calculated?" and "why is the electricity bill so high?" are uniformly converted to "electricity bill consultation." Second, typos are automatically corrected using a dedicated power industry error correction dictionary, combined with contextual matching. Then, based on a table of commonly used abbreviations and full names in power customer service, abbreviations are uniformly expanded to their full names. For example, "CT" is expanded to "current transformer," "PT" to "voltage transformer," and "AVC" to "automatic voltage control." Finally, stop word filtering is performed: based on a general Chinese stop word list, functional words related to power customer service communication are added to construct a dedicated stop word list for power customer service. New words include "hello," "please ask," "trouble," "thank you," "help," "take a look," "that," and "this," etc. These words frequently appear in the work order text but contribute very little to semantic understanding and are therefore filtered out.

[0205] The normalized text data processed as described above is used for subsequent BERT encoding, TF-IDF keyword extraction, and LDA topic modeling. In the missing value and outlier detection branches, the data removed is not included in the training set or the best case study set.

[0206] Example 9

[0207] like Figure 7 As shown, this embodiment details the structured JSON format of the final output scheme.

[0208] For any pending power supply customer service work order, after the optimal strategy π*(a|s) decision, the output is a JSON object containing five first-level fields: work_order_id is the work order number;

[0209] `classification_result` is the work order classification result; `category_result` is the work order classification result; `recommended_action` is the recommended action; `reply_template` is the reply template.

[0210] The work order classification result includes: the risk_level field, which takes the value of "high", "medium" or "low"; the risk_score field, which takes the risk score value in the range of [0,1]; and the top_risk_factors field, which is a list of the top 3 main risk factors in descending order of contribution, each of which includes factor_name and contribution.

[0211] The work order classification result (category_result) includes: the primary_category field, which takes the value of one of "Power Outage Repair", "Electricity Bill Inquiry", "Fault Complaint", or "New Energy Business Inquiry"; the secondary_category field is the corresponding second-level sub-category (e.g., "Power Outage Repair" is divided into low-voltage line fault, high-voltage line fault, meter fault, transformer fault, equipment burnout, switch tripping, voltage abnormality, etc.); and the confidence field is the classification confidence level (0-1). When the highest confidence level is <0.6, the system marks it as "low confidence classification" and outputs the candidates field as a list of candidate classifications for quick manual correction.

[0212] The recommended action includes: action_type (Chinese name of the action); action_code (internal action code); priority (urgency level); execution_standard (description of specific execution standards); responsible_role (recommended responsibility role); deadline_minutes (first response time limit in minutes); sla_hours (total work order processing time in hours); action_reason (reason for recommendation); and alternative_actions (list of alternative actions, each containing action (action name), code (action code), and confidence).

[0213] The reply template (reply_template) includes: template_id (template identifier); template_type (template type); applicable_scenario (applicable scenario description); template_text (template text), marked with double curly braces {{field_name}} as placeholders for replaceable fields; and fields (a detailed list of replaceable fields, each containing name, value, and source (data source path, using dot-separated path notation, such as "user_info.name"). For fields that are not yet available, the value is filled with "N / A" and the status is marked as "pending," to be automatically filled in once the data is complete.

[0214] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0215] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0216] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0217] A computer-readable storage medium in which the methods described above according to embodiments of the present invention can be implemented in hardware, firmware, or implemented as recordable on a storage medium, or implemented as computer code originally stored on a remote storage medium or a non-transitory machine-readable storage medium and to be stored on a local storage medium after being downloaded over a network, so that the methods described herein can be processed by such software stored on a storage medium using a general-purpose computer, a special-purpose processor, or programmable or special-purpose hardware.

[0218] The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; furthermore, the storage medium can also include combinations of the above types of memory. It is understood that a computer, processor, microprocessor controller, or programmable hardware includes storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0219] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0220] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0221] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

Claims

1. A method for constructing a hierarchical classification decision-making model for power supply customer service based on reinforcement learning. Its features are, Includes the following steps: Step S1: Collect historical outstanding handling cases in the power supply customer service field, extract keywords using the TF-IDF algorithm, and extract expert experience rules by combining expert annotations to construct a hierarchical and classified expert strategy library P. The expert experience rules are in the form of "IF condition THEN hierarchical result, classification result, handling action". Step S2: Define the reinforcement learning state space S; The state space S consists of four dimensions: work order semantic features X1, deep-level request tags X2, risk level X3, and processing steps X4, i.e., S = [X1, X2, X3, X4]. T ; The semantic feature X1 of the work order is extracted by encoding the work order text using a BERT pre-trained model; The deep-level appeal tag X2 was obtained using the LDA topic model. The risk level X3 is calculated using the Analytic Hierarchy Process (AHP) and mapped to three levels: high, medium, and low. The processing stage X4 represents the code of the current processing stage of the work order; Step S3: Define a reinforcement learning action space A, which includes a set of recommended actions A1 and a set of response script templates A2, i.e., A = A1∪A2; The recommended action set A1 includes four core power supply customer service actions: escalation of supervision, on-site verification, collaborative handling, and routine handling. The reply script template set A2 contains standardized reply script templates designed for different business types and demand scenarios; Step S4: Design a reinforcement learning reward function R(s,a,s′), which comprehensively considers four dimensions: manual adoption rate, customer satisfaction, whether the complaint is escalated, and processing time. The expression is R(s,a,s′)=α·r1+β·r2+γ·r3+δ·r4, where α+β+γ+δ=1, r1 is the reward for manual adoption, r2 is the reward for customer satisfaction, r3 is the reward for escalation of complaints, and r4 is the reward for timely handling. Step S5: Iteratively optimize the policy using the Proximal Policy Optimization (PPO) algorithm, with the expert policy library P as the initial policy. A three-stage progressive integration mechanism is defined, comprising a cold start stage, a transition stage, and a stabilization stage. During the cold start phase, decision-making relies entirely on the expert strategy library; During the transition phase, the expert strategy and the reinforcement learning strategy are weighted and fused according to dynamic weights. During the stable phase, reinforcement learning strategies dominate decision-making, while expert strategies serve as a safety net. Step S6: Iterate training until the policy converges, output the optimal policy π(a|s). For the input power customer service work order to be processed, extract its state features s, select the optimal action a, and output a complete processing solution including work order classification results, work order categorization results, recommended processing actions and reply script templates.

2. The method according to claim 1, Its features are, In step S2, the semantic feature X1 of the work order is extracted using the BERT-base-chinese pre-trained model. The first 6 layers of the Transformer are frozen, the last 6 layers and the output layer are fine-tuned, and the 768-dimensional vector corresponding to the [CLS] label of the last hidden layer of BERT is taken as the semantic feature vector of the work order text.

3. The method according to claim 1, Its features are, In step S2, the method for extracting the deep appeal tag X2 is as follows: set the number of topics in the LDA topic model K=8, the hyperparameters α=0.1 and β=0.01, the number of iterations is 1000, and the topic distribution θ is output for each work order text. d =(θ1,θ2,…,θ8), represented using one-hot encoding, with a dimension of 8; The risk level X3 is calculated as follows: the AHP method is used to determine the weights of each risk indicator, and the risk score Sr is calculated. It is then normalized to the [0,1] interval and mapped to three levels: high risk (Sr≥0.70), medium risk (0.30≤Sr<0.70), and low risk (Sr<0.30). The processing stage X4 uses integer coding: 1 = acceptance, 2 = triage, 3 = handling, and 4 = follow-up visit.

4. The method according to claim 3, Its features are, The risk assessment indicator system of the AHP method includes 5 primary indicators: scope of impact, urgency, public opinion risk, economic loss, and social impact, as well as 12 secondary indicators. The AHP judgment matrix is ​​constructed using the Saaty 1-9 scaling method, the multi-expert judgment matrix is ​​aggregated using the geometric mean method, the weight calculation method uses the eigenvalue method, and the consistency ratio CR = CI / RI. in, RI is a consistency index, which stands for random consistency index. When CR < 0.10, the judgment matrix is ​​considered to have satisfactory consistency, and the weighting results are acceptable.

5. The method according to claim 1, Its features are, In step S4, the reward function The four components are defined as follows: r1 is the reward for human adoption. If the action recommended by the agent is adopted by humans, then r1=1; otherwise, r1=0. r2 is the customer satisfaction reward, r2=score / 100, where score is the customer satisfaction rating (0-100 points). r3 represents the penalty reward for escalating complaints. If no escalation occurs after action a is performed on the work order, r3 = 1; if an escalation occurs, r3 = -1. r4 is a timeliness bonus, if the work order processing time t 处置 If ≤ standard time limit T, then Otherwise, r4 = -0.5; The reward weights satisfy α=0.25, β=0.35, γ=0.3, δ=0.1, the discount factor γ0=0.99, and the cumulative reward... .

6. The method according to claim 1, Its features are, In step S5, the three-stage progressive fusion mechanism is specifically implemented as follows: The cold start phase: expert strategy weights =1.0, reinforcement learning strategy weight =0.0, decision-making relies entirely on the expert strategy library; The transition phase: It decreases linearly from 1.0 to 0.

3. The weights increase linearly from 0.0 to 0.7, and the formula for the weight change is: =min(0.7,(n−500) / 2500×0.7), where n is the number of processed work orders, and the final action selection probability is... , where δ is the indicator function; When reinforcement learning recommends actions that seriously conflict with expert policies, the expert policy actions must be used. The stable phase: reinforcement learning policy weights ∈(0.8,0.9), expert strategy weights ∈(0.1,0.2), reinforcement learning strategy dominates decision-making, while expert strategy serves as a safety net; When the confidence level of the reinforcement learning strategy falls below a preset threshold, or when the probability of the reinforcement learning output action leading to an escalation of a complaint exceeds 10% in the past, the strategy reverts to the expert strategy.

7. The method according to claim 1, Its features are, In step S5, the specific steps for iterative optimization of the policy using the near-end policy optimization (PPO) algorithm are as follows: Step S5.1: Initialize the policy network parameters θ0 and the value network parameters φ0. The policy network is a three-layer fully connected network with an input dimension of 778 and an output dimension of 44. The value network is a three-layer fully connected network with an input dimension of 778 and an output dimension of 1. Step S5.2: The agent, according to the current policy... Interact with the environment to collect samples Where N is the number of samples in each round, N=1000; Step S5.3: Calculate the advantage function using Generalized Advantage Estimation (GAE): ， in, , The state value output by the value network. For GAE parameters, λ=0.95; Step S5.4: Update the policy network parameters θ by minimizing the PPO-Clip objective function, where the objective function is... ,in, ε is the cutting factor, ε=0.2; Step S5.5: Minimize the mean square error loss Update the value network parameter φ; Step S5.6: Repeat steps S5.2 to S5.5 until the cumulative reward Gt tends to stabilize and the fluctuation range is ≤5%, and the strategy converges.

8. The method according to claim 1, Its features are, In step S1, when collecting historical excellent handling cases, cases with empty work order text, missing handling results, or missing satisfaction scores are directly removed. Outliers whose processing time exceeds 10 times the standard time limit will be removed. Outliers with satisfaction scores outside the 0-100 range or that are not numerical values ​​will be removed. The power work order text is standardized, including: converting colloquial expressions into standardized terms based on a standardized mapping table of power customer service work order terminology; correcting typos based on a power industry-specific error correction dictionary; unifying the expansion of abbreviations based on a table of commonly used abbreviations and full names for power supply customer service; and constructing an industry-specific stop word list by adding functional words for power customer service communication to the general Chinese stop word list.

9. The method according to claim 1, Its features are, In step S6, the output complete processing solution adopts a structured JSON format and includes at least the following fields: work order classification result, including risk level and risk score; The work order classification results include primary classification, secondary classification, and classification confidence level; Recommended actions include action type, action code, priority, execution criteria, recommended responsible role, initial response time limit, total work order processing time limit, reasons for recommendation, and a list of alternative actions; The response script template includes a template identifier, template text, a list of replaceable fields, and the source of the field values. The replaceable fields in the response script template are marked with double curly braces {{field name}} as placeholders, and each replaceable field includes the current value and the data source path.

10. A system for constructing a hierarchical classification decision-making model for power supply customer service based on reinforcement learning. Its features are, include: The data acquisition and preprocessing module is used to collect historical outstanding handling cases and real-time customer service work order data in the power supply customer service field, and to perform missing value processing, outlier processing and text normalization processing. The expert strategy library construction module is used to extract keywords from excellent cases using the TF-IDF algorithm, combine expert annotations to extract expert experience rules, and construct a hierarchical and classified expert strategy library P. The state space definition module is used to define the reinforcement learning state space S. The state space S consists of four dimensions: work order semantic features X1 extracted by the BERT model, deep appeal labels X2 mined by the LDA model, risk level X3 calculated by the Analytic Hierarchy Process (AHP), and processing link encoding X4. The action space definition module is used to define the reinforcement learning action space A, which includes a set of recommended disposal actions A1 and a set of response script templates A2. The reward function design module is used to design the reinforcement learning reward function R(s,a,s′). The reward function takes the human adoption rate, customer satisfaction, whether the complaint is escalated, and the timeliness of handling as the core reward signals and is implemented in a weighted summation form. The policy iteration optimization module is used to perform policy iteration optimization using the proximal policy optimization PPO algorithm, and uses the expert policy library P as the initial policy. It achieves the fusion of expert policy and reinforcement learning policy through a three-stage progressive fusion mechanism. The solution output module is used to output a complete processing solution based on the optimal strategy π*(a|s) obtained from training, including the work order classification result, recommended handling action and reply script template for the input power supply customer service work order to be handled.

Images