Iterative optimization method and device for intelligent customer service model

Abstract

The invention provides an iterative optimization method and device for an intelligent customer service model. The method comprises the steps that session data is acquired and pre-marked for quality inspection, the session data after quality inspection is acquired, the session data comprises error data, and the error data comprises a problem marked with an error tag, a reply of a corresponding error and a marked problem scene tag; counting the occurrence rate of error data in each problem scene, screening session data corresponding to the problem scene with the occurrence rate exceeding a threshold value, and marking the session data as bidding data; acquiring annotation data made by an annotation expert; inputting the annotation data into the large model for distillation, and generating chain thinking process data; and combining the annotation data and the chain-type thinking process data as training samples, and carrying out model iterative optimization on the intelligent customer service model. According to the method, the intelligent customer service model is iteratively optimized through the automatic process, the efficiency of solving online problems is improved, the occurrence rate of the online problems is stably reduced, and meanwhile complete online quality data is provided.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to an iterative optimization method and apparatus for an intelligent customer service model. Background Technology

[0002] With the continuous development of artificial intelligence technology, large-scale models are being applied more and more widely in the field of intelligent customer service. For example, in the used car sales scenario, by introducing large-scale models as car sales consultants, they can answer customers' questions related to car purchases and improve the efficiency of handling customer issues. However, existing intelligent customer service models have many problems in the training and optimization process, such as the reliance on manual inspection, which is inefficient and prone to missed checks and misjudgments; the complexity of the used car sales scenario, with problems varying greatly from person to person, making it difficult to effectively attribute causes; and the lagging labeling process, making it difficult to promptly submit real online problems to experts for labeling and training. These problems seriously affect the efficiency and quality of intelligent customer service. Summary of the Invention

[0003] This disclosure provides an iterative optimization method and apparatus for an intelligent customer service model.

[0004] Firstly, this disclosure provides an iterative optimization method for an intelligent customer service model, including:

[0005] Acquire conversation data between the intelligent customer service model and the user, the conversation data including multiple conversations, the conversations including questions raised by the user and responses provided by the intelligent customer service model;

[0006] The session data is pre-labeled using a pre-labeling model. The pre-labeled labels include session ratings. The pre-labeled session data is then used for quality inspection.

[0007] Acquire the session data after quality inspection. The session data after quality inspection includes error data, which includes questions marked with error labels and their corresponding incorrect answers, as well as question scenario labels. The question scenario labels are the labels of the question scenarios to which the question is clustered.

[0008] Statistically analyze the occurrence rate of error data in each problem scenario, filter out problem scenarios with an occurrence rate exceeding a threshold, and identify them as high-frequency problem scenarios;

[0009] Acquire session data corresponding to high-frequency problem scenarios and mark them as bidding data;

[0010] Obtain annotation data from annotation experts on the tender data, including standard responses to issues with incorrectly labeled tags;

[0011] The labeled data is input into a large model for distillation to generate chain-like thinking process data.

[0012] The labeled data and the chain-thinking process data are combined as training samples to iteratively optimize the intelligent customer service model.

[0013] In some alternative implementations, the pre-labeled tags further include at least one of the following: whether the user has negative emotions, whether the user suspects it is a robot, whether the user requests a real person's reply, whether there is an obvious irrelevant answer, whether the message will be sent repeatedly, and whether there is a rhetorical question.

[0014] In some optional implementations, after the session data is pre-labeled using a pre-labeling model, the method further includes storing sessions rated as needing improvement in a review pool space.

[0015] In some optional implementations, before calculating the occurrence rate of error data in each problem scenario, the method further includes: obtaining error data from online users in real time.

[0016] In some optional implementations, the step of calculating the occurrence rate of error data under each problem scenario includes: recording the error data in a message table, and calculating the occurrence rate of error data under each problem scenario based on the message table.

[0017] In some alternative implementations, the annotation data further includes the path to the tool that was called for the issue of the incorrectly labeled problem.

[0018] In some optional implementations, the method further includes: obtaining constructed verification test cases, adding the verification test cases to a regression verification set based on the corresponding problem scenarios, and using the regression verification set to verify the iteratively optimized intelligent customer service model.

[0019] Secondly, this disclosure provides an iterative optimization device for an intelligent customer service model, comprising:

[0020] The conversation data acquisition module is configured to acquire conversation data between the intelligent customer service model and the user. The conversation data includes multiple conversations, and the conversations include questions raised by the user and responses provided by the intelligent customer service model.

[0021] The pre-labeling module is configured to pre-label the session data using a pre-labeling model. The pre-labeled labels include session ratings, and the pre-labeled session data is used for quality inspection.

[0022] The error data acquisition module is configured to acquire session data after quality inspection. The session data after quality inspection includes error data, which includes questions marked with error tags and their corresponding incorrect answers, as well as question scenario tags. The question scenario tags are the tags of the question scenario to which the question is clustered.

[0023] The occurrence rate statistics module is configured to count the occurrence rate of error data in each problem scenario, filter problem scenarios with an occurrence rate exceeding a threshold, and identify them as high-frequency problem scenarios.

[0024] The bidding data marking module is configured to acquire session data corresponding to high-frequency problem scenarios and mark it as bidding data;

[0025] The annotation data acquisition module is configured to acquire annotation data made by annotation experts for the bidding data, the annotation data including standard answers to user questions about incorrectly labeled tags;

[0026] The chain thinking process data acquisition module is configured to input the labeled data into a large model for distillation to generate chain thinking process data;

[0027] The iterative optimization module is configured to combine the labeled data and the chained thinking process data as training samples to perform iterative optimization of the intelligent customer service model.

[0028] In some alternative implementations, the pre-labeled tags further include at least one of the following: whether the user has negative emotions, whether the user suspects it is a robot, whether the user requests a real person's reply, whether there is an obvious irrelevant answer, whether the message will be sent repeatedly, and whether there is a rhetorical question.

[0029] In some optional implementations, the pre-labeling module is further configured to store sessions rated as needing improvement into the review pool space.

[0030] In some optional implementations, the error data acquisition module is further configured to acquire error data reported in real time by online users.

[0031] In some optional implementations, the occurrence rate statistics module is further configured to: record error data in a message table, and statistically analyze the occurrence rate of error data in each problem scenario based on the message table.

[0032] In some alternative implementations, the annotation data further includes the path to the tool that was called for the issue of the incorrectly labeled problem.

[0033] In some optional implementations, the method further includes: obtaining constructed verification test cases, adding the verification test cases to a regression verification set based on the corresponding problem scenarios, and using the regression verification set to verify the iteratively optimized intelligent customer service model.

[0034] Thirdly, this disclosure provides a computer device, including: one or more processors; and a storage device having one or more programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform the method as described in the first aspect.

[0035] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by one or more processors, implements the method described in the first aspect.

[0036] To address the technical issues of inefficiency and poor quality in intelligent customer service models, this disclosure proposes an iterative optimization method and apparatus for intelligent customer service models. This disclosure extracts problem conversations through pre-labeling and quality inspection, focusing on them during subsequent labeling to improve the accuracy and effectiveness of online quality inspection. This disclosure designs a solution based on the scenario dimension; specifically, it locates user problems to problem scenarios through clustering, determines the data that needs to be trained based on the problem scenarios, improves the difficulty of effectively attributing complex business scenarios, and enhances the overall versatility of the solution. During labeling, this disclosure uses expert annotation of actual standard answers, and distills the thought process data from a large model, thereby improving labeling efficiency. Simultaneously, the distilled thought process format conforms to a unified standard, and these are combined to obtain training samples, improving the optimization effect of iterative training.

[0037] In summary, this disclosure continuously iterates and optimizes the intelligent customer service model through a highly automated process of problem discovery, analysis, correction, and training and repair, thereby improving the efficiency of resolving online related issues, steadily reducing the online problem rate of the intelligent customer service model, and providing complete online quality data and online model quality statistics. Attached Figure Description

[0038] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings:

[0039] Figure 1 This is a schematic diagram of an exemplary system architecture of at least one embodiment of this disclosure;

[0040] Figure 2 This is a timing diagram of an iterative optimization method for an intelligent customer service model according to an embodiment of the present disclosure;

[0041] Figure 3 This is a flowchart illustrating an iterative optimization method for an intelligent customer service model according to an embodiment of the present disclosure;

[0042] Figure 4This is a schematic diagram of the structure of an iterative optimization device for an intelligent customer service model according to an embodiment of the present disclosure;

[0043] Figure 5 This is a schematic diagram of the hardware configuration of a computer device according to an embodiment of the present disclosure. Detailed Implementation

[0044] To gain a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this disclosure.

[0045] Figure 1 An exemplary system architecture 100 is shown, representing an embodiment of the iterative optimization method and apparatus for the intelligent customer service model disclosed herein.

[0046] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0047] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, specifically, shopping applications suitable for used cars, etc.

[0048] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with information input devices (e.g., keyboard, mouse, touchscreen, microphone, camera, etc.) and information output devices (e.g., display screen, speakers, etc.), including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc. When terminal devices 101, 102, and 103 are software, they can be installed on the terminal devices listed above. They can be implemented as multiple software programs or software modules, or as a single software program or software module. No specific limitations are made here.

[0049] In some cases, the iterative optimization method of the intelligent customer service model provided in this disclosure can be executed by the server 105, and correspondingly, the iterative optimization device of the intelligent customer service model can also be set in the server 105.

[0050] It should be noted that server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0051] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0052] refer to Figure 2 , Figure 2 This is a timing diagram of the iterative optimization method for the intelligent customer service model proposed in this publication.

[0053] The iterative optimization method and apparatus for intelligent customer service models disclosed herein provide a reasonable processing scheme for the training and optimization of online intelligent customer service models, and are used to solve many problems existing in the training and optimization process of existing online intelligent customer service models, such as the following:

[0054] 1. The quality inspection process currently relies heavily on manual labor, which is inefficient and prone to omissions and misjudgments. Furthermore, the intelligent customer service response process requires the combination of multiple modules, and the accuracy of judgment needs to be improved.

[0055] 2. The business scenarios are complex, and the problems are diverse. Each person may ask different questions about a particular type of problem, leading to different results. There is no standard for problem clustering and attribution, making it impossible to cluster problems to the scenario dimension.

[0056] 3. In the annotation process, the submitted data usually lags behind the actual online data, making it difficult to promptly hand over the real online issues to experts for annotation and training solutions.

[0057] like Figure 1 As shown, the iterative optimization method of the intelligent customer service model disclosed herein includes the following processing stages and steps.

[0058] I. Online Data Extraction Phase

[0059] 1. Extracting conversation data: Obtaining conversation data between the intelligent customer service model and the user. The conversation data includes multiple conversations, which include questions raised by the user and responses provided by the intelligent customer service model.

[0060] Here, the intelligent customer service model can be deployed on a server. Users can communicate with the server through terminal devices and ask questions online. The intelligent customer service model can provide answers to the user's questions, just like a human customer service representative. The conversation data between the intelligent customer service model and the user can be stored in real-time on the server's local storage device or an external storage device connected to the server. The server can retrieve the required conversation data from the server's local storage device or the external storage device connected to the server in real-time. For example, the server can extract conversation data periodically, such as extracting the previous day's conversation data every day, or extracting the conversation data of the previous N days every N days, where N is a positive integer.

[0061] For example, user questions might include: "What is the price of a certain car model?" or "What are the monthly payments if I buy it with a loan and pay in installments?" The intelligent customer service model's response could include text information such as the specific price amount, and could further include links to online car loan calculators or other tools.

[0062] 2. Pre-labeling: The acquired session data is pre-labeled using a pre-labeling model. The pre-labeled labels include session ratings, and the pre-labeled session data is used for quality inspection.

[0063] Here, a large-scale pre-labeling model can be used to preprocess the acquired session data. Preprocessing includes: pre-labeling the session data using an automated rule engine via the pre-labeling model, resulting in pre-labeled tags on the session data. These pre-labeled tags include at least a session rating, which is the rating given by the pre-labeling model for the current session. For example, ratings from highest to lowest could include: Excellent, Good, Satisfactory, Needs Improvement, etc.

[0064] Here, the pre-labeled tags may further include at least one of the following: whether the user has negative emotions, whether the user suspects that it is a robot (AI), whether the user requests a real person's reply, whether the model obviously gives irrelevant answers, whether the model repeatedly sends messages, and whether the model asks questions that it already knows the answer to.

[0065] Here, the pre-labeled model can be deployed on the same server as the intelligent customer service model, or they can be deployed separately on different servers.

[0066] 3. Provide pre-labeled session data.

[0067] Here, the pre-labeled session data will be used for quality inspection in subsequent steps. For example, pre-labeled session data can be provided to a quality inspection platform for review. By using pre-labeled session data instead of the original session data, quality inspectors can improve the efficiency of the quality inspection process. For instance, if a session receives a rating of "needs improvement," it indicates that the intelligent customer service model's response in that session is ineffective and requires close attention from the quality inspector.

[0068] II. Online Data Quality Inspection Phase

[0069] 1. Design a quality inspection system in advance, identify potential problem scenarios in clustering, define scenario specifications, and establish a problem scenario tag library so that user problems can be classified into the identified problem scenarios during subsequent quality inspections.

[0070] Here, problem scenarios can be hierarchically set, with each level of problem scenario including multiple levels of problem scenarios. For example, one can define a problem scenario—Violation of the knowledge base—and further define multiple levels of problem scenarios under "Violation of the knowledge base," such as: down payment ratio / installment period; calculating annual interest rate independently; incorrect interest calculation.

[0071] 2. The pre-labeled session data enters the quality inspection platform, where quality inspectors conduct specialized quality checks and mark incorrect tags and scene tags.

[0072] During quality inspection, problem analysis and clustering labeling can be performed by combining the model's thought process and context. Problem analysis may include analyzing whether the intelligent customer service model's responses to user questions are incorrect, identifying incorrect responses and their corresponding problems, and labeling them with error tags. Clustering labeling may include mapping the problem corresponding to a particular incorrect response to a specific problem scenario category based on the numerous problem scenarios determined in the previous design, and labeling it with a problem scenario tag. Optionally, clustering labeling may also include a final conversation rating made by the quality inspector for the conversation, which may be the same as or different from the pre-labeled conversation rating. Furthermore, error data is generated, which may include the problems marked with error tags and their corresponding incorrect responses, as well as the labeled problem scenario tags. The problem scenario tags are the labels indicating the problem scenario to which the question is clustered.

[0073] For example, during quality inspection, sessions with poor ratings or negative feedback can receive increased attention. For example, sessions with a final rating of "needs improvement" can be moved to a review pool for back-to-back reviews.

[0074] III. Data Statistical Analysis Stage

[0075] 1. Obtain the quality-checked session data, which includes the error data (badcases) mentioned above. Optionally, error data (badcases) from real-time feedback from online users can also be obtained simultaneously. Perform data statistics; for example, this data can be automatically aggregated into a message table at time T+1 (the next cycle), and the message table can be used to record and analyze this data. This way, the occurrence rate of error data in a single problem scenario can be statistically analyzed in real time based on the tags (e.g., error labels) on the message table. Optionally, the message table supports quickly retrieving data based on tags.

[0076] 2. For clustering-related problem scenarios, analyze and determine the repair plan and timeliness based on the clustering problem scenario to identify the solution path and solution. For example, analyze and determine whether the corresponding erroneous data needs to be addressed through training and mark it accordingly. Among these, problem scenarios with an erroneous data occurrence rate exceeding a set threshold can be filtered out. For example, problem scenarios with an occurrence rate exceeding 5% can be identified as high-frequency problem scenarios and determined to require training for resolution.

[0077] 3. For session data corresponding to high-frequency problem scenarios that require training to resolve, mark them as bidding data. Then, for example, all bidding data can be directly obtained from the aggregated message table through the training tags to quickly train online problem data and ensure that the training data is consistent with the online data trend.

[0078] IV. Online Issue Labeling Phase

[0079] 1. For the session data that needs to be labeled and resolved through training, i.e. bidding data, it can be extracted in batches and sent to the labeling platform for labeling.

[0080] a. Expert Annotation: This involves defining standard responses to user questions and / or developing custom tools to be used in this response. Specifically, based on unified annotation standards, a semi-automated annotation tool can be used, with annotation experts manually annotating the bidding data. The annotation data can include expert responses to incorrectly labeled issues, and may further include the paths to the calling tools.

[0081] b. Large-scale model distillation: After annotation, the labeled data can be input into a large model based on user questions and standard annotation responses. The large model then distills the chain-thinking process data. The labeled data and the chain-thinking process data are then used as training samples.

[0082] V. Model Update and Iteration Phase

[0083] 1. Training Model: Using the training samples obtained in the previous step, the intelligent customer service model is iteratively optimized and trained in real time, enabling the model to learn relevant knowledge from the latest online conversations. For example, the conversation data from the first day can be obtained on the second day, and after generating training samples through the previous four steps, iterative optimization training can be completed on the same day, resulting in an updated and optimized model on the third day.

[0084] 2. The new version of the intelligent customer service model, after iteration and optimization, has been launched.

[0085] 3. Optionally, regression testing can be used to confirm the effectiveness of the iteratively optimized intelligent customer service model.

[0086] Specifically, for issues requiring testing and repair, a batch of verification cases can be quickly constructed by combining issue clustering scenarios and repair solutions. Verification cases can be clustered into a regression verification set based on the corresponding issue scenarios. Subsequently, regression verification can be performed on the iteratively optimized intelligent customer service model based on the regression verification set.

[0087] 4. Optionally, for some typical online issues, key nodes can be extracted from the aggregated data and added to the test set for replay, increasing the consistency between the test set and the actual online scenario.

[0088] The above describes the iterative optimization method for the intelligent customer service model disclosed herein. Key technical points of this disclosed solution include, but are not limited to:

[0089] 1. Online data extraction and quality inspection stages:

[0090] Utilize automated rule engines and pre-labeling models to pre-label online session data;

[0091] Establish unified standards for problem classification and clustering, and group problems into problem scenarios for unified analysis and resolution.

[0092] 2. Data Statistical Analysis Phase: Determine bidding data based on problem scenario analysis;

[0093] 3. Online issue labeling stage:

[0094] Establish unified annotation standards and use semi-automated annotation tools for annotation;

[0095] For the labeled questions and standard answers, large models are used to distill the thought process data, improving labeling efficiency and the consistency of the thought process format;

[0096] 4. Model update and iteration phase:

[0097] After the annotation is completed, the intelligent customer service model is trained in near real-time through an automated training data stream, enabling it to learn relevant knowledge in real time through the latest online conversations.

[0098] 5. Problem Regression Verification Phase:

[0099] We constructed a multi-dimensional verification indicator system, and used methods such as online typical problem replay and problem scenario clustering to verify the repair effect. We also precipitated general scenarios such as domain general knowledge into a general test set for intelligent customer service.

[0100] The technical effects achieved by the disclosed solution include, but are not limited to:

[0101] 1. By pre-annotating, problematic sessions can be extracted so that they can be focused on by designated personnel during subsequent annotation. In addition, sessions that fail the first round of annotation can be set to enter back-to-back checks, improving the accuracy and effectiveness of online quality inspection.

[0102] 2. During annotation, annotation experts only annotate the response process and standard response. The thought process is distilled from the large model, which improves annotation efficiency. At the same time, the format of the distilled thought process data conforms to the unified standard.

[0103] 3. During quality inspection and data verification, online issues are clustered into various scenarios through a unified issue clustering scenario standard, which increases the focus of analysis and repair. At the same time, the test sets accumulated for similar issue scenarios can be reused in the industry's general test sets.

[0104] refer to Figure 3 , Figure 3 This is a flowchart of an iterative optimization method for an intelligent customer service model according to an embodiment of the present disclosure. The iterative optimization method for the intelligent customer service model of the present disclosure can be achieved by, for example... Figure 1 The system architecture shown is implemented in server 102.

[0105] like Figure 3 As shown, the iterative optimization method for the intelligent customer service model disclosed herein may include the following steps:

[0106] Step 301: Obtain the conversation data between the intelligent customer service model and the user. The conversation data includes multiple conversations, which include questions raised by the user and responses provided by the intelligent customer service model.

[0107] Step 302: Pre-label the session data using a pre-labeling model. The pre-labeled labels include session ratings. The pre-labeled session data is then used for quality inspection.

[0108] Step 303: Obtain the session data after quality inspection. The session data after quality inspection includes error data, which includes questions marked with error labels and their corresponding incorrect answers, as well as question scenario labels. The question scenario label is the label of the question scenario to which the question is clustered.

[0109] Step 304: Calculate the occurrence rate of error data in each problem scenario, filter out problem scenarios with an occurrence rate exceeding the threshold, and identify them as high-frequency problem scenarios;

[0110] Step 305: Obtain the session data corresponding to high-frequency problem scenarios and mark it as bidding data;

[0111] Step 306: Obtain the annotation data made by the annotation experts for the bidding data. The annotation data includes the standard answers given for the problems of incorrectly labeled data.

[0112] Step 307: Input the labeled data into the large model for distillation to generate chain-thinking process data;

[0113] Step 308: Combine labeled data and chained thinking process data as training samples to iteratively optimize the intelligent customer service model.

[0114] In some alternative implementations, the pre-labeled tags further include at least one of the following: whether the user has negative emotions, whether the user suspects it is a robot, whether the user requests a real person's reply, whether there is an obvious irrelevant answer, whether the message will be sent repeatedly, and whether there is a rhetorical question.

[0115] In some alternative implementations, after step 302, the method further includes storing sessions rated as needing improvement into the review pool space.

[0116] In some alternative implementations, prior to step 304, the method further includes: acquiring error data from real-time feedback from online users.

[0117] In some optional implementations, step 304 includes: recording error data in a message table, and statistically analyzing the occurrence rate of error data in each problem scenario based on the message table.

[0118] In some alternative implementations, the annotation data further includes the path to the tool that was called for the issue that was incorrectly labeled.

[0119] In some optional implementations, the method further includes: obtaining constructed verification test cases, adding the verification test cases to a regression verification set based on the corresponding problem scenarios, and using the regression verification set to verify the iteratively optimized intelligent customer service model.

[0120] It should be noted that the implementation of each step in the method of this embodiment can be referred to the description of other embodiments in this disclosure, and will not be repeated here.

[0121] refer to Figure 4 , Figure 4 This is a schematic diagram of an iterative optimization apparatus for an intelligent customer service model according to an embodiment of the present disclosure. This apparatus embodiment is similar to... Figure 3 Corresponding to the illustrated method embodiments, it can be applied to, for example... Figure 1 The system architecture shown is for server 102.

[0122] like Figure 4 As shown, the iterative optimization device 400 for the intelligent customer service model disclosed herein may include:

[0123] The conversation data acquisition module 401 is configured to acquire conversation data between the intelligent customer service model and the user. The conversation data includes multiple conversations, which include questions raised by the user and responses provided by the intelligent customer service model.

[0124] The pre-labeling module 402 is configured to pre-label session data using a pre-labeling model. The pre-labeled labels include session ratings, and the pre-labeled session data is used for quality inspection.

[0125] The error data acquisition module 403 is configured to acquire session data after quality inspection. The session data after quality inspection includes error data, which includes questions marked with error tags and their corresponding incorrect responses, as well as question scenario tags. The question scenario tags are the tags of the question scenario to which the question is clustered.

[0126] The occurrence rate statistics module 404 is configured to count the occurrence rate of error data in each problem scenario, filter problem scenarios with an occurrence rate exceeding a threshold, and identify them as high-frequency problem scenarios.

[0127] The bid data marking module 405 is configured to acquire session data corresponding to high-frequency problem scenarios and mark it as bid data.

[0128] The annotation data acquisition module 406 is configured to acquire annotation data made by annotation experts for the bidding data. The annotation data includes standard answers to user questions about incorrectly labeled tags.

[0129] The chain thinking process data acquisition module 407 is configured to input labeled data into a large model for distillation to generate chain thinking process data.

[0130] The iterative optimization module 408 is configured to combine labeled data and chained thinking process data as training samples to perform iterative optimization of the intelligent customer service model.

[0131] In some alternative implementations, the pre-labeled tags further include at least one of the following: whether the user has negative emotions, whether the user suspects it is a robot, whether the user requests a real person's reply, whether there is an obvious irrelevant answer, whether the message will be sent repeatedly, and whether there is a rhetorical question.

[0132] In some alternative implementations, the pre-labeling module 402 is further configured to store sessions rated as needing improvement into the review pool space.

[0133] In some optional implementations, the error data acquisition module 403 is further configured to acquire error data reported by online users in real time.

[0134] In some optional implementations, the occurrence rate statistics module 404 is further configured to: record error data in a message table, and statistically analyze the occurrence rate of error data in each problem scenario based on the message table.

[0135] In some alternative implementations, the annotation data further includes the path to the tool that was called for the issue that was incorrectly labeled.

[0136] In some optional implementations, the apparatus further includes: a regression verification module (not shown in the figure), configured to acquire constructed verification cases, add the verification cases to a regression verification set based on the corresponding problem scenarios, and use the regression verification set to verify the iteratively optimized intelligent customer service model.

[0137] It should be noted that the implementation details and technical effects of each module in the device of this embodiment can be found in the descriptions of other embodiments in this disclosure, and will not be repeated here. The implementation schemes of each module in this device are diverse, as long as the purpose of the module can be achieved; actual deployment is not limited to specific implementation schemes.

[0138] refer to Figure 5 , Figure 5 This is a schematic diagram of the structure of a computer device for implementing a server according to an embodiment of the present disclosure. Figure 5 As shown, the computer device 500 of this disclosure may include:

[0139] One or more processors 501;

[0140] Memory 502, on which one or more programs 503 are stored;

[0141] Components such as processor 501 and memory 502 can be coupled together via bus system 504; bus system 504 is used to realize the connection and communication between these components.

[0142] When one or more programs 503 are executed by one or more processors 501, the one or more processors 501 implement the iterative optimization method of the intelligent customer service model as disclosed in the above method embodiments.

[0143] The bus system 504 may include a data bus, a power bus, a control bus, and a status signal bus. The memory 502 may be volatile or non-volatile, or both. The processor 501 may be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0144] This disclosure also provides a computer-readable storage medium storing a computer program thereon, which, when executed by one or more processors, implements the iterative optimization method for the intelligent customer service model disclosed in the above method embodiments.

[0145] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0146] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0147] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0148] It should be understood that the terms "system" and "network" are often used interchangeably in this disclosure. The term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Furthermore, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0149] The above description is merely a preferred embodiment of this disclosure and is not intended to limit the scope of protection of this disclosure. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An iterative optimization method for an intelligent customer service model, characterized in that, include: Acquire conversation data between the intelligent customer service model and the user, the conversation data including multiple conversations, the conversations including questions raised by the user and responses provided by the intelligent customer service model; The session data is pre-labeled using a pre-labeling model. The pre-labeled labels include session ratings. The pre-labeled session data is then used for quality inspection. Acquire the session data after quality inspection. The session data after quality inspection includes error data, which includes questions marked with error labels and their corresponding incorrect answers, as well as question scenario labels. The question scenario labels are the labels of the question scenarios to which the question is clustered. Statistically analyze the occurrence rate of error data in each problem scenario, filter out problem scenarios with an occurrence rate exceeding a threshold, and identify them as high-frequency problem scenarios; Acquire session data corresponding to high-frequency problem scenarios and mark them as bidding data; Obtain annotation data from annotation experts on the tender data, including standard responses to issues with incorrectly labeled tags; The labeled data is input into a large model for distillation to generate chain-like thinking process data. The labeled data and the chain-thinking process data are combined as training samples to iteratively optimize the intelligent customer service model.

2. The method according to claim 1, characterized in that, The pre-labeled tags further include at least one of the following: whether the user has negative emotions, whether the user suspects it is a robot, whether the user requests a real person's reply, whether there is an obvious irrelevant answer, whether the message will be sent repeatedly, and whether there is a rhetorical question.

3. The method according to claim 1, characterized in that, After pre-labeling the session data using a pre-labeling model, the method further includes: Sessions rated as needing improvement are stored in the review pool space.

4. The method according to claim 1, characterized in that, Before calculating the occurrence rate of error data in each problem scenario, the method further includes: Obtain error data from online users in real time.

5. The method according to claim 1, characterized in that, The statistics on the occurrence rate of erroneous data in each problem scenario include: Error data is recorded in a message table, and the occurrence rate of error data in each problem scenario is statistically analyzed based on the message table.

6. The method according to claim 1, characterized in that, The annotation data further includes the tool invocation path for issues with incorrectly labeled tags.

7. The method according to claim 1, characterized in that, The method further includes: Obtain the constructed verification test cases, and add the verification test cases to the regression verification set based on the corresponding problem scenarios; The iteratively optimized intelligent customer service model was validated using the regression validation set.

8. An iterative optimization device for an intelligent customer service model, characterized in that, include: The conversation data acquisition module is configured to acquire conversation data between the intelligent customer service model and the user. The conversation data includes multiple conversations, and the conversations include questions raised by the user and responses provided by the intelligent customer service model. The pre-labeling module is configured to pre-label the session data using a pre-labeling model. The pre-labeled labels include session ratings, and the pre-labeled session data is used for quality inspection. The error data acquisition module is configured to acquire session data after quality inspection. The session data after quality inspection includes error data, which includes questions marked with error tags and their corresponding incorrect answers, as well as question scenario tags. The question scenario tags are the tags of the question scenario to which the question is clustered. The occurrence rate statistics module is configured to count the occurrence rate of error data in each problem scenario, filter problem scenarios with an occurrence rate exceeding a threshold, and identify them as high-frequency problem scenarios. The bidding data marking module is configured to acquire session data corresponding to high-frequency problem scenarios and mark it as bidding data; The annotation data acquisition module is configured to acquire annotation data made by annotation experts for the bidding data, the annotation data including standard answers to user questions about incorrectly labeled tags; The chain thinking process data acquisition module is configured to input the labeled data into a large model for distillation to generate chain thinking process data; The iterative optimization module is configured to combine the labeled data and the chained thinking process data as training samples to perform iterative optimization of the intelligent customer service model.

9. A computer device, characterized in that, include: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by one or more processors, implements the method as described in any one of claims 1-7.

Images