Customer service conversation quality inspection method and device

Abstract

The present invention discloses a customer service conversation quality inspection method and device, and relates to the field of computer technology. A specific implementation of the method includes: obtaining a customer service conversation log, performing summary identification on the user conversation and the customer service response in the customer service conversation log according to summary parameters and a summary identification model, and respectively determining the user summary label corresponding to the user conversation and the customer service summary label corresponding to the customer service response; determining the user number corresponding to the user conversation and the customer service number corresponding to the customer service response, and grouping the user summary label and the customer service summary label according to the correspondence between the user number and the customer service number in the customer service conversation log; matching the user summary label and the customer service summary label in the same group according to the summary matching rule, and determining the quality inspection result according to the matching result. This implementation expands the applicable scenarios of the quality inspection method, improves the accuracy of the quality inspection results, and improves the quality inspection efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a customer service conversation quality inspection method and device. Background Art

[0002] In the service industry such as e-commerce, quality inspection systems have emerged to evaluate and improve the quality of customer service. In customer service response services, the single sentence technique, emotions, and context of the response are all important indicators for evaluating service quality.

[0003] For contextual quality inspection, the existing method mainly involves the data team screening and extracting data instances, which are then manually annotated by a professional annotation team; the algorithm team debugs the model based on the data instances and annotation results, and finally uses the debugged model to perform quality inspection on the actual customer service response conversations.

[0004] There are at least the following problems in the prior art:

[0005] The existing quality inspection methods are difficult to meet the different needs of users. When user needs change, the entire quality inspection process needs to be readjusted, resulting in a narrow range of applicable scenarios for the existing quality inspection methods and difficulty in maintaining quality inspection plans. At the same time, there are problems such as low quality inspection accuracy and time-consuming model debugging. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a customer service session quality inspection method and device, which can complete the quality inspection of customer service sessions in combination with summary tags and summary matching rules, which helps to expand the applicable scenarios of the quality inspection method and reduce the maintenance coefficient of the quality inspection solution. At the same time, it also improves the accuracy of the quality inspection results and improves the quality inspection efficiency.

[0007] To achieve the above object, according to one aspect of an embodiment of the present invention, a customer service conversation quality inspection method is provided, comprising:

[0008] Obtain the customer service conversation log, perform summary recognition on the user conversation and the customer service response in the customer service conversation log according to the summary parameters and the summary recognition model, and respectively determine the user summary label corresponding to the user conversation and the customer service summary label corresponding to the customer service response;

[0009] Determine the user ID corresponding to the user session and the customer service ID corresponding to the customer service response, and group the user summary tags and customer service summary tags according to the correspondence between the user ID and the customer service ID in the customer service session log;

[0010] According to the profile matching rules, the user profile tags and customer service profile tags in the same group are matched, and the quality inspection results are determined based on the matching results.

[0011] Furthermore, before the step of performing summary recognition on the user session and the customer service response in the customer service session log according to the summary parameters and the summary recognition model, the method further includes a model training step:

[0012] Obtain historical customer service session logs, perform summary recognition on the historical customer service session logs according to summary parameters and the initial summary recognition model, and determine historical user summary tags and historical customer service summary tags;

[0013] Group historical user summary tags and historical customer service summary tags according to the corresponding relationship;

[0014] Match historical user summary tags and historical customer service summary tags in the same group according to summary matching rules, and determine the matching accuracy rate;

[0015] Determine whether the matching accuracy is greater than the accuracy threshold. If so, determine that the initial summary recognition model is the summary recognition model; if not, adjust the model parameters and execute the model training steps in a loop.

[0016] Further, the user summary tags and the customer service summary tags are grouped according to the correspondence between the user ID and the customer service ID in the customer service session log, further comprising:

[0017] Determine the timestamps corresponding to the user session and the customer service response in the customer service session log;

[0018] The user summary tags and customer service summary tags are grouped according to the corresponding relationship, timestamp and grouping time interval.

[0019] Furthermore, it also includes:

[0020] receiving updated summary matching rules;

[0021] The user profile tags and customer service profile tags in the same group are matched according to the updated profile matching rules.

[0022] Furthermore, it also includes:

[0023] The adjusted grouping time interval is received, and the grouping result is updated according to the adjusted grouping time interval.

[0024] Furthermore, the quality inspection result is determined according to the matching result, including:

[0025] If at least one of the matching results is successful, the corresponding quality inspection result is passed;

[0026] If all matches in the matching results fail, the corresponding quality inspection result is quality inspection failure.

[0027] According to another aspect of an embodiment of the present invention, a customer service conversation quality inspection device is provided, comprising:

[0028] The log acquisition module is used to: acquire the customer service session log, perform summary recognition on the user session and the customer service response in the customer service session log according to the summary parameters and the summary recognition model, and respectively determine the user summary label corresponding to the user session and the customer service summary label corresponding to the customer service response;

[0029] A grouping module is used to: determine the user number corresponding to the user session and the customer service number corresponding to the customer service answer, and group the user summary label and the customer service summary label according to the correspondence between the user number and the customer service number in the customer service session log;

[0030] The quality inspection module is used to match the user summary tags and customer service summary tags in the same group according to the summary matching rules, and determine the quality inspection results according to the matching results.

[0031] Furthermore, before the step of performing summary recognition on the user session and the customer service response in the customer service session log according to the summary parameters and the summary recognition model, the device further includes a model training module for:

[0032] Obtain historical customer service session logs, perform summary recognition on the historical customer service session logs according to summary parameters and the initial summary recognition model, and determine historical user summary tags and historical customer service summary tags;

[0033] Group historical user summary tags and historical customer service summary tags according to the corresponding relationship;

[0034] Match historical user summary tags and historical customer service summary tags in the same group according to summary matching rules, and determine the matching accuracy rate;

[0035] Determine whether the matching accuracy is greater than the accuracy threshold. If so, determine that the initial summary recognition model is the summary recognition model; if not, adjust the model parameters and execute the model training steps in a loop.

[0036] According to another aspect of an embodiment of the present invention, there is provided an electronic device for customer service conversation quality inspection, including:

[0037] one or more processors;

[0038] a storage device for storing one or more programs,

[0039] When one or more programs are executed by one or more processors, the one or more processors implement any of the customer service session quality inspection methods described above.

[0040] According to another aspect of the embodiments of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, any of the above methods is implemented.

[0041] An embodiment of the above invention has the following advantages or beneficial effects: because the customer service session log is obtained, the user session and the customer service response in the customer service session log are summarized and identified according to the summary parameters and the summary recognition model, and the user summary tag corresponding to the user session and the customer service summary tag corresponding to the customer service response are determined respectively; the user number corresponding to the user session and the customer service number corresponding to the customer service response are determined, and the user summary tag and the customer service summary tag are grouped according to the correspondence between the user number and the customer service number in the customer service session log; the user summary tag and the customer service summary tag in the same group are matched according to the summary matching rule, and the technical means of determining the quality inspection result according to the matching result is overcome. Therefore, when the user demand changes, the entire quality inspection process needs to be readjusted in the existing quality inspection method, resulting in the existing quality inspection method having a narrow application scenario and difficulty in maintaining the quality inspection scheme; at the same time, there are still technical problems such as low quality inspection accuracy and long model debugging time, so as to achieve the ability to complete the quality inspection of the customer service session in combination with the summary tag and the summary matching rule, and achieve the technical effect of expanding the applicable scenario of the quality inspection method, reducing the maintenance coefficient of the quality inspection scheme, improving the accuracy of the quality inspection result, and improving the quality inspection efficiency.

[0042] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0044] Figure 1 It is a schematic diagram of the main process of a customer service conversation quality inspection method provided according to an embodiment of the present invention;

[0045] Figure 2 is a schematic diagram of the main process of a customer service conversation quality inspection method provided according to another embodiment of the present invention;

[0046] Figure 3 is a schematic diagram of main modules of a customer service conversation quality inspection device provided according to an embodiment of the present invention;

[0047] Figure 4 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0048] Figure 5 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0050] Figure 1 is a schematic diagram of the main process of a customer service conversation quality inspection method provided according to an embodiment of the present invention; Figure 1 As shown, the customer service conversation quality inspection method provided by the embodiment of the present invention mainly includes:

[0051] Step S101, obtain a customer service conversation log, perform summary recognition on the user conversation and the customer service response in the customer service conversation log according to summary parameters and a summary recognition model, and respectively determine a user summary tag corresponding to the user conversation and a customer service summary tag corresponding to the customer service response.

[0052] Specifically, according to an embodiment of the present invention, unlike the existing method of manually annotating data instances (conversation logs) with a unified standard by an annotation team, the present invention extracts summaries from user conversations and customer service responses through summary parameters and summary recognition models, so as to facilitate subsequent quality inspection operations of customer service conversations based on summary tags. Since the accuracy of the summary recognition model is relatively high, it helps to improve the accuracy of subsequent quality inspection results; on the other hand, by only identifying the summary information of a single-sentence conversation, it is equivalent to only labeling each conversation with a summary tag, which significantly improves the efficiency of customer service conversation quality inspection.

[0053] According to a specific implementation of an embodiment of the present invention, the summary parameters may be adjusted according to actual conditions to further improve the accuracy of the determined summary labels, and at the same time, help expand the applicable scenarios of the quality inspection method for customer service conversations.

[0054] Further, according to an embodiment of the present invention, before the step of performing summary recognition on the user session and the customer service response in the customer service session log according to the summary parameters and the summary recognition model, the method further includes a model training step:

[0055] Obtain historical customer service session logs, perform summary recognition on the historical customer service session logs according to summary parameters and the initial summary recognition model, and determine historical user summary tags and historical customer service summary tags;

[0056] Group historical user summary tags and historical customer service summary tags according to the corresponding relationship;

[0057] Match historical user summary tags and historical customer service summary tags in the same group according to summary matching rules, and determine the matching accuracy rate;

[0058] Determine whether the matching accuracy is greater than the accuracy threshold. If so, determine that the initial summary recognition model is the summary recognition model; if not, adjust the model parameters and execute the model training steps in a loop.

[0059] Specifically, according to an embodiment of the present invention, the above-mentioned initial summary recognition model can adopt the existing summary recognition model testCNN, and the initial summary model can be trained in combination with summary parameters and historical customer session logs. Among them, the summary parameters for user sessions can be set to the type of "Ask XX / Consult XX", such as "Ask about orders / Consult about orders", "Ask about invoices / Consult about invoices", etc.; the summary parameters for customer responses can be the type of "Answer XX / Reply", such as "Reply to order details" and "Reply to invoice information". That is, it is only necessary to perform summary recognition on the context of the customer's response and determine the summary label to perform subsequent quality inspection operations. Through the above settings, the quality inspection process of customer service responses is simplified and the quality inspection efficiency is improved.

[0060] Step S102, determining the user number corresponding to the user session and the customer service number corresponding to the customer service response, and grouping the user summary tags and the customer service summary tags according to the correspondence between the user number and the customer service number in the customer service session log.

[0061] Through the above settings, user summary tags and customer service summary tags are grouped according to the correspondence between user numbers and customer service numbers, which helps to subsequently complete the matching of user summary tags and customer service summary tags in the same group, and then determine the quality inspection results, thereby significantly improving the accuracy of the quality inspection results.

[0062] Further, according to an embodiment of the present invention, the user summary tags and the customer service summary tags are grouped according to the correspondence between the user number and the customer service number in the customer service session log, and the method further includes:

[0063] Determine the timestamps corresponding to the user session and the customer service response in the customer service session log;

[0064] The user summary tags and customer service summary tags are grouped according to the corresponding relationship, timestamp and grouping time interval.

[0065] According to a specific implementation of an embodiment of the present invention, based on the user session timestamp in the customer service session log, the customer service response corresponding to a period of time before and after the timestamp (i.e., the grouping time interval) is determined. The corresponding user session (corresponding to the user summary tag) and the customer service response (corresponding to the customer service summary tag) are grouped into the same group, so as to complete the quality inspection of the customer service session in a subsequent manner of intra-group matching.

[0066] Specifically, since a user may ask the same question several times in a row, the customer service may only answer the first question. In order to avoid quality inspection failure due to unanswered questions after the first time, the previous answers are added when determining the grouping time interval, that is, the grouping time interval includes a period of time before and after the user session timestamp.

[0067] Preferably, according to an embodiment of the present invention, the method further includes:

[0068] The adjusted grouping time interval is received, and the grouping result is updated according to the adjusted grouping time interval.

[0069] Specifically, based on the user's quality inspection requirements for customer service responses and the quality inspection accuracy requirements, the grouping time interval can be adjusted. The smaller the grouping time interval, the higher the quality inspection requirements for customer service responses and the quality inspection accuracy.

[0070] Step S103, matching the user summary tags and customer service summary tags in the same group according to the summary matching rule, and determining the quality inspection result according to the matching result.

[0071] Preferably, according to an embodiment of the present invention, the method further includes:

[0072] receiving updated summary matching rules;

[0073] The user profile tags and customer service profile tags in the same group are matched according to the updated profile matching rules.

[0074] According to an embodiment of the present invention, the quality inspection method of customer service response provided by the present invention supports fast and dynamic configuration of matching rules to adapt to different quality inspection requirements in order to further expand the applicable scenarios. For example, for the first time, the summary matching rule can be configured, and the customer service response summary that matches the user session summary "ask order type" is "answer order type". Based on the above summary matching rule, in the grouped message, if the user session summary "ask order type" is identified, then the customer service response summary in the same group must have "answer order type" to pass the quality inspection, otherwise it will not pass. If it is necessary to relax the requirements for customer service responses, the above summary matching rule can be adjusted, such as setting the customer service response summary that matches the user session summary "ask order type" to "answer order status". Then, after receiving the updated summary matching rule, the quality inspection operation of the customer service response can be performed according to the updated summary matching rule without the need for program modification or online.

[0075] Specifically, according to an embodiment of the present invention, the step of determining the quality inspection result according to the matching result includes:

[0076] If at least one of the matching results is successful, the corresponding quality inspection result is passed;

[0077] If all matches in the matching results fail, the corresponding quality inspection result is quality inspection failure.

[0078] According to an embodiment of the present invention, since a customer may ask the same question several times in a row and the customer service may only answer the first question, the quality inspection can be determined to have passed as long as there is a successful match between a user session summary and a customer service answer summary in the same group.

[0079] According to the technical solution of the embodiment of the present invention, the customer service session log is obtained, the user session and the customer service answer in the customer service session log are summarized and identified according to the summary parameters and the summary identification model, and the user summary tag corresponding to the user session and the customer service summary tag corresponding to the customer service answer are determined respectively; the user number corresponding to the user session and the customer service number corresponding to the customer service answer are determined, and the user summary tag and the customer service summary tag are grouped according to the correspondence between the user number and the customer service number in the customer service session log; the user summary tag and the customer service summary tag in the same group are matched according to the summary matching rule, and the technical means of determining the quality inspection result according to the matching result is overcome. Therefore, when the user demand changes, the entire quality inspection process needs to be readjusted in the existing quality inspection method, resulting in the existing quality inspection method having a narrow application scenario and difficult maintenance of the quality inspection scheme; at the same time, there are still technical problems such as low quality inspection accuracy and long model debugging time, so as to achieve the ability to complete the quality inspection of the customer service session in combination with the summary tag and the summary matching rule, and achieve the technical effect of expanding the applicable scenario of the quality inspection method, reducing the maintenance coefficient of the quality inspection scheme, improving the accuracy of the quality inspection result, and improving the quality inspection efficiency.

[0080] Figure 2 is a schematic diagram of the main process of a customer service conversation quality inspection method provided according to another embodiment of the present invention; Figure 2 As shown, the customer service conversation quality inspection method provided by the embodiment of the present invention mainly includes:

[0081] Step S201 - Step S205: training step of the summary recognition model.

[0082] Step S201, obtaining historical customer service session logs, performing summary recognition on the historical customer service session logs according to summary parameters and an initial summary recognition model, and determining historical user summary tags and historical customer service summary tags.

[0083] Specifically, according to an embodiment of the present invention, the above-mentioned initial summary recognition model can adopt the existing summary recognition model testCNN (the testCNN model is currently the most commonly used model for question classification, which can be used for both short texts and long texts. Since the present invention is mainly aimed at customer service questions and answers, it belongs to short texts. Furthermore, for customer service responses in the e-commerce industry, some specific parameters can be filled in and the model can be trained by debugging parameters). Combined with the summary parameters and historical customer session logs, the summaries of user sessions and customer service responses in the logs are respectively identified, and historical user summary labels and historical customer service summary labels are determined, so as to facilitate the subsequent determination of the matching accuracy of the summary labels and achieve optimized training of the summary recognition model.

[0084] Specifically, the summary parameters for user sessions can be set to, for example, "ask about orders / inquire about orders", "ask about invoices / inquire about invoices", etc.; the summary parameters for customer responses can be "reply to order details", "reply to invoice information", etc. In other words, it is only necessary to perform summary recognition on the context of customer responses and determine summary labels, and then it is possible to determine whether the summary recognition model meets the actual quality inspection requirements based on the matching accuracy of the user session summary and the customer service response probability.

[0085] Step S202, grouping historical user summary tags and historical customer service summary tags according to the corresponding relationship; matching historical user summary tags and historical customer service summary tags in the same group according to the summary matching rule, and determining the matching accuracy.

[0086] Step S203, determine whether the matching accuracy is greater than the accuracy threshold. If so, that is, the matching accuracy is greater than the accuracy threshold, execute step S204; if not, that is, the matching accuracy is less than or equal to the accuracy threshold, execute step S205.

[0087] Step S204: determine that the initial summary recognition model is a summary recognition model.

[0088] Step S205, adjust the model parameters, and go to step S201 to restart the model training step.

[0089] Specifically, according to the embodiment of the present invention, the data used in the model training process is all the content sent between the user and the customer service during the response process. Therefore, the model parameters adjusted are mainly for the classification of different questions consulted by users. In addition, if different identification is required for users and customer service, different parameters can be added for training; if separate identification is not required, the same parameters can also be used for training.

[0090] Through the above settings, the summary recognition model that meets the matching accuracy can be trained by continuously adjusting the model parameters.

[0091] Step S206-Step S208: Quality inspection steps of actual customer service conversations.

[0092] Step S206, obtaining the customer service conversation log, performing summary recognition on the user conversation and the customer service response in the customer service conversation log according to the summary parameters and the summary recognition model, and respectively determining the user summary tag corresponding to the user conversation and the customer service summary tag corresponding to the customer service response.

[0093] According to a specific implementation of an embodiment of the present invention, the summary parameters may be adjusted according to actual conditions to further improve the accuracy of the determined summary labels, and at the same time, help expand the applicable scenarios of the quality inspection method for customer service conversations.

[0094] Step S207, determine the user number corresponding to the user session and the customer service number corresponding to the customer service response, determine the timestamps corresponding to the user session and the customer service response in the customer service session log; group the user summary tags and customer service summary tags according to the corresponding relationship, timestamp and grouping time interval.

[0095] Through the above settings, user summary tags and customer service summary tags are grouped according to the correspondence between user numbers and customer service numbers, which helps to subsequently complete the matching of user summary tags and customer service summary tags in the same group, and then determine the quality inspection results, thereby significantly improving the accuracy of the quality inspection results.

[0096] Specifically, since a user may ask the same question several times in a row, the customer service may only answer the first question. In order to avoid quality inspection failure due to unanswered questions after the first time, the previous answers are added when determining the grouping time interval, that is, the grouping time interval includes a period of time before and after the user session timestamp.

[0097] Preferably, according to an embodiment of the present invention, the method further includes:

[0098] The adjusted grouping time interval is received, and the grouping result is updated according to the adjusted grouping time interval.

[0099] Specifically, based on the user's quality inspection requirements for customer service responses and the quality inspection accuracy requirements, the grouping time interval can be adjusted. The smaller the grouping time interval, the higher the quality inspection requirements for customer service responses and the quality inspection accuracy.

[0100] Step S208, matching the user summary tags and customer service summary tags in the same group according to the summary matching rules, if at least one of the matching results is successful, the corresponding quality inspection result is quality inspection passed; if all the matching results fail, the corresponding quality inspection result is quality inspection failed.

[0101] Through the above settings, the present invention provides a method of separate identification (respectively performing summary identification on user sessions and customer service responses to determine summary labels) and combined quality inspection (matching user summary labels and customer service summary labels in the same group), which simplifies the customer service response quality inspection process, improves the accuracy of quality inspection results, and expands the applicable scenarios of customer service response quality inspection.

[0102] Furthermore, according to an embodiment of the present invention, the method further includes:

[0103] receiving updated summary matching rules;

[0104] The user profile tags and customer service profile tags in the same group are matched according to the updated profile matching rules.

[0105] According to an embodiment of the present invention, the quality inspection method of customer service response provided by the present invention supports fast and dynamic configuration of matching rules to adapt to different quality inspection requirements in order to further expand the applicable scenarios. For example, for the first time, the summary matching rule can be configured, and the customer service response summary that matches the user session summary "ask order type" is "answer order type". Based on the above summary matching rule, in the grouped message, if the user session summary "ask order type" is identified, then the customer service response summary in the same group must have "answer order type" to pass the quality inspection, otherwise it will not pass. If it is necessary to relax the requirements for customer service responses, the above summary matching rule can be adjusted, such as setting the customer service response summary that matches the user session summary "ask order type" to "answer order status". Then, after receiving the updated summary matching rule, the quality inspection operation of the customer service response can be performed according to the updated summary matching rule without the need for program modification or online.

[0106] According to the technical solution of the embodiment of the present invention, the customer service session log is obtained, the user session and the customer service answer in the customer service session log are summarized and identified according to the summary parameters and the summary identification model, and the user summary tag corresponding to the user session and the customer service summary tag corresponding to the customer service answer are determined respectively; the user number corresponding to the user session and the customer service number corresponding to the customer service answer are determined, and the user summary tag and the customer service summary tag are grouped according to the correspondence between the user number and the customer service number in the customer service session log; the user summary tag and the customer service summary tag in the same group are matched according to the summary matching rule, and the technical means of determining the quality inspection result according to the matching result is overcome. Therefore, when the user demand changes, the entire quality inspection process needs to be readjusted in the existing quality inspection method, resulting in the existing quality inspection method having a narrow application scenario and difficult maintenance of the quality inspection scheme; at the same time, there are still technical problems such as low quality inspection accuracy and long model debugging time, so as to achieve the ability to complete the quality inspection of the customer service session in combination with the summary tag and the summary matching rule, and achieve the technical effect of expanding the applicable scenario of the quality inspection method, reducing the maintenance coefficient of the quality inspection scheme, improving the accuracy of the quality inspection result, and improving the quality inspection efficiency.

[0107] Figure 3 Schematic diagram of the main modules of the customer service conversation quality inspection device provided according to an embodiment of the present invention; Figure 3 As shown, the customer service conversation quality inspection device 300 provided in the embodiment of the present invention mainly includes:

[0108] The log acquisition module 301 is used to: acquire the customer service conversation log, perform summary recognition on the user conversation and the customer service response in the customer service conversation log according to the summary parameters and the summary recognition model, and respectively determine the user summary tag corresponding to the user conversation and the customer service summary tag corresponding to the customer service response.

[0109] Specifically, according to an embodiment of the present invention, unlike the existing method of manually annotating data instances (conversation logs) with a unified standard by an annotation team, the present invention extracts summaries from user conversations and customer service responses through summary parameters and summary recognition models, so as to facilitate subsequent quality inspection operations of customer service conversations based on summary tags. Since the accuracy of the summary recognition model is relatively high, it helps to improve the accuracy of subsequent quality inspection results; on the other hand, by only identifying the summary information of a single-sentence conversation, it is equivalent to only labeling each conversation with a summary tag, which significantly improves the efficiency of customer service conversation quality inspection.

[0110] According to a specific implementation of an embodiment of the present invention, the summary parameters may be adjusted according to actual conditions to further improve the accuracy of the determined summary labels, and at the same time, help expand the applicable scenarios of the quality inspection method for customer service conversations.

[0111] Further, according to an embodiment of the present invention, the customer service conversation quality inspection device 300 further includes a model training module, which is used to:

[0112] Obtain historical customer service session logs, perform summary recognition on the historical customer service session logs according to summary parameters and the initial summary recognition model, and determine historical user summary tags and historical customer service summary tags;

[0113] Group historical user summary tags and historical customer service summary tags according to the corresponding relationship;

[0114] Match historical user summary tags and historical customer service summary tags in the same group according to summary matching rules, and determine the matching accuracy rate;

[0115] Determine whether the matching accuracy is greater than the accuracy threshold. If so, determine that the initial summary recognition model is the summary recognition model; if not, adjust the model parameters and execute the model training steps in a loop.

[0116] Specifically, according to an embodiment of the present invention, the above-mentioned initial summary recognition model can adopt the existing summary recognition model testCNN, and the initial summary model can be trained in combination with summary parameters and historical customer session logs. Among them, the summary parameters for user sessions can be set to the type of "Ask XX / Consult XX", such as "Ask about orders / Consult about orders", "Ask about invoices / Consult about invoices", etc.; the summary parameters for customer responses can be the type of "Answer XX / Reply", such as "Reply to order details" and "Reply to invoice information". That is, it is only necessary to perform summary recognition on the context of the customer's response and determine the summary label to perform subsequent quality inspection operations. Through the above settings, the quality inspection process of customer service responses is simplified and the quality inspection efficiency is improved.

[0117] The grouping module 302 is used to determine the user number corresponding to the user session and the customer service number corresponding to the customer service response, and group the user summary tags and the customer service summary tags according to the correspondence between the user number and the customer service number in the customer service session log.

[0118] Through the above settings, user summary tags and customer service summary tags are grouped according to the correspondence between user numbers and customer service numbers, which helps to subsequently complete the matching of user summary tags and customer service summary tags in the same group, and then determine the quality inspection results, thereby significantly improving the accuracy of the quality inspection results.

[0119] Further, according to an embodiment of the present invention, the grouping module 302 is further configured to:

[0120] Determine the timestamps corresponding to the user session and the customer service response in the customer service session log;

[0121] The user summary tags and customer service summary tags are grouped according to the corresponding relationship, timestamp and grouping time interval.

[0122] According to a specific implementation of an embodiment of the present invention, based on the user session timestamp in the customer service session log, the customer service response corresponding to a period of time before and after the timestamp (i.e., the grouping time interval) is determined. The corresponding user session (corresponding to the user summary tag) and the customer service response (corresponding to the customer service summary tag) are grouped into the same group, so as to complete the quality inspection of the customer service session in a subsequent manner of intra-group matching.

[0123] Specifically, since a user may ask the same question several times in a row, the customer service may only answer the first question. In order to avoid quality inspection failure due to unanswered questions after the first time, the previous answers are added when determining the grouping time interval, that is, the grouping time interval includes a period of time before and after the user session timestamp.

[0124] Preferably, according to an embodiment of the present invention, the customer service conversation quality inspection device 300 further includes a grouping time interval updating module, which is used to:

[0125] The adjusted grouping time interval is received, and the grouping result is updated according to the adjusted grouping time interval.

[0126] Specifically, based on the user's quality inspection requirements for customer service responses and the requirements for quality inspection accuracy, the grouping time interval can be adjusted. The smaller the grouping time interval, the higher the quality inspection requirements for customer service responses and the quality inspection accuracy (at this time, the main reference factor for the quality inspection of customer service responses is the customer service response efficiency). If the quality inspection requirements for customer service responses change, the grouping time interval can be adjusted based on the actual situation.

[0127] The quality inspection module 303 is used to match the user summary tags and customer service summary tags in the same group according to the summary matching rule, and determine the quality inspection result according to the matching result.

[0128] Preferably, according to an embodiment of the present invention, the customer service conversation quality inspection device 300 further includes a summary matching rule updating module, which is used to:

[0129] receiving updated summary matching rules;

[0130] The user profile tags and customer service profile tags in the same group are matched according to the updated profile matching rules.

[0131] According to an embodiment of the present invention, the quality inspection method for customer service responses provided by the present invention further expands the applicable scenarios and supports fast and dynamic configuration of matching rules to adapt to different quality inspection requirements. For example, for the first time, a summary matching rule can be configured, and the customer service response summary that matches the user session summary "ask for order type" is "answer for order type". Based on the above summary matching rule, in the grouped message, if the user session summary "ask for order type" is identified, then the customer service response summary in the same group must have "answer for order type" in order to pass the quality inspection, otherwise it will not pass. If the requirements for customer service responses need to be relaxed, the above summary matching rule can be adjusted, such as setting the customer service response summary that matches the user session summary "ask for order type" to "answer for order status" and / or "answer for order type". Then, upon receiving the updated summary matching rule, the quality inspection operation of the customer service response can be performed according to the updated summary matching rule without the need for program modification or online.

[0132] Specifically, according to an embodiment of the present invention, the quality inspection module 303 is also used for:

[0133] If at least one of the matching results is successful, the corresponding quality inspection result is passed;

[0134] If all matches in the matching results fail, the corresponding quality inspection result is quality inspection failure.

[0135] According to an embodiment of the present invention, since a customer may ask the same question several times in a row and the customer service may only answer the first question, the quality inspection can be determined to have passed as long as there is a successful match between a user session summary and a customer service answer summary in the same group.

[0136] According to the technical solution of the embodiment of the present invention, the customer service session log is obtained, the user session and the customer service answer in the customer service session log are summarized and identified according to the summary parameters and the summary identification model, and the user summary tag corresponding to the user session and the customer service summary tag corresponding to the customer service answer are determined respectively; the user number corresponding to the user session and the customer service number corresponding to the customer service answer are determined, and the user summary tag and the customer service summary tag are grouped according to the correspondence between the user number and the customer service number in the customer service session log; the user summary tag and the customer service summary tag in the same group are matched according to the summary matching rule, and the technical means of determining the quality inspection result according to the matching result is overcome. Therefore, when the user demand changes, the entire quality inspection process needs to be readjusted in the existing quality inspection method, resulting in the existing quality inspection method having a narrow application scenario and difficult maintenance of the quality inspection scheme; at the same time, there are still technical problems such as low quality inspection accuracy and long model debugging time, so as to achieve the ability to complete the quality inspection of the customer service session in combination with the summary tag and the summary matching rule, and achieve the technical effect of expanding the applicable scenario of the quality inspection method, reducing the maintenance coefficient of the quality inspection scheme, improving the accuracy of the quality inspection result, and improving the quality inspection efficiency.

[0137] Figure 4 An exemplary system architecture 400 is shown to which the customer service session quality inspection method or the customer service session quality inspection device according to the embodiment of the present invention can be applied.

[0138] like Figure 4 As shown, the system architecture 400 may include terminal devices 401, 402, 403, a network 404 and a server 405 (this architecture is only an example, and the components included in the specific architecture may be adjusted according to the specific application). The network 404 is used to provide a medium for a communication link between the terminal devices 401, 402, 403 and the server 405. The network 404 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0139] Users can use terminal devices 401, 402, 403 to interact with server 405 through network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, 403, such as quality inspection applications, web browser applications, search applications, instant messaging tools, email clients, data processing software, etc. (only for example).

[0140] The terminal devices 401 , 402 , and 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0141] Server 405 may be a server that provides various services, such as a server that performs quality inspection / data processing (for example only) on the user's terminal devices 401, 402, and 403. The server may analyze and process the received customer service conversation log and other data, and feed back the processing results (such as matching results - for example only) to the terminal device.

[0142] It should be noted that the customer service conversation quality inspection method provided in the embodiment of the present invention is generally executed by the server 405 , and accordingly, the customer service conversation quality inspection device is generally arranged in the server 405 .

[0143] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0144] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system 500 of a terminal device or a server suitable for implementing an embodiment of the present invention. Figure 5 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0145] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0146] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0147] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present invention are executed.

[0148] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0149] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor, for example, may be described as: a processor including a log acquisition module, a grouping module and a quality inspection module. The names of these modules do not, in certain cases, constitute limitations on the modules themselves, for example, the log acquisition module may also be described as "a module for: obtaining customer service session logs, performing summary identification of user sessions and customer service responses in customer service session logs according to summary parameters and summary identification models, and respectively determining user summary labels corresponding to user sessions and customer service summary labels corresponding to customer service responses".

[0151] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The computer-readable medium carries one or more programs. When the one or more programs are executed by a device, the device includes: obtaining a customer service session log, performing summary identification on the user session and the customer service response in the customer service session log according to the summary parameters and the summary identification model, and respectively determining the user summary tag corresponding to the user session and the customer service summary tag corresponding to the customer service response; determining the user number corresponding to the user session and the customer service number corresponding to the customer service response, and grouping the user summary tag and the customer service summary tag according to the correspondence between the user number and the customer service number in the customer service session log; matching the user summary tag and the customer service summary tag in the same group according to the summary matching rule, and determining the quality inspection result according to the matching result. The technical means overcomes the existing quality inspection method. When the user demand changes, the entire quality inspection process needs to be readjusted, resulting in the existing quality inspection method having a narrow application scenario and difficulty in maintaining the quality inspection plan; at the same time, there are still technical problems such as low quality inspection accuracy and long model debugging time, thereby achieving the ability to complete the quality inspection of the customer service session in combination with the summary tag and the summary matching rule, and achieving the technical effect of expanding the applicable scenarios of the quality inspection method, reducing the maintenance coefficient of the quality inspection plan, improving the accuracy of the quality inspection result, and improving the quality inspection efficiency.

[0152] According to the technical solution of the embodiment of the present invention, the customer service session log is obtained, the user session and the customer service answer in the customer service session log are summarized and identified according to the summary parameters and the summary identification model, and the user summary tag corresponding to the user session and the customer service summary tag corresponding to the customer service answer are determined respectively; the user number corresponding to the user session and the customer service number corresponding to the customer service answer are determined, and the user summary tag and the customer service summary tag are grouped according to the correspondence between the user number and the customer service number in the customer service session log; the user summary tag and the customer service summary tag in the same group are matched according to the summary matching rule, and the technical means of determining the quality inspection result according to the matching result is overcome. Therefore, when the user demand changes, the entire quality inspection process needs to be readjusted in the existing quality inspection method, resulting in the existing quality inspection method having a narrow application scenario and difficult maintenance of the quality inspection scheme; at the same time, there are still technical problems such as low quality inspection accuracy and long model debugging time, so as to achieve the ability to complete the quality inspection of the customer service session in combination with the summary tag and the summary matching rule, and achieve the technical effect of expanding the applicable scenario of the quality inspection method, reducing the maintenance coefficient of the quality inspection scheme, improving the accuracy of the quality inspection result, and improving the quality inspection efficiency.

[0153] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A customer service conversation quality inspection method, characterized in that: include: Obtain a customer service conversation log, perform summary recognition on the user conversation and the customer service response in the customer service conversation log according to the summary parameters and the summary recognition model, and respectively determine a user summary tag corresponding to the user conversation and a customer service summary tag corresponding to the customer service response; Determine a user number corresponding to the user session and a customer service number corresponding to the customer service response, and group the user summary tag and the customer service summary tag according to the correspondence between the user number and the customer service number in the customer service session log; Match the user summary tag and the customer service summary tag in the same group according to the summary matching rule, and determine the quality inspection result according to the matching result; The training steps of the outline recognition model include: Obtain historical customer service session logs, perform summary recognition on the historical customer service session logs according to summary parameters and an initial summary recognition model, and determine historical user summary tags and historical customer service summary tags; grouping the historical user summary tags and the historical customer service summary tags according to the correspondence between the user ID and the customer ID in the historical customer service session log; Matching the historical user summary tags and the historical customer service summary tags in the same group according to the summary matching rule, and determining the matching accuracy rate; Determine whether the matching accuracy is greater than an accuracy threshold, and if so, determine that the initial summary recognition model is the summary recognition model; if not, adjust the model parameters and execute the model training step in a loop.

2. The customer service conversation quality inspection method according to claim 1, characterized in that: The grouping of the user summary tags and the customer service summary tags according to the correspondence between the user numbers and the customer service numbers in the customer service session logs also includes: Determine the timestamps corresponding to the user session and the customer service response in the customer service session log; The user summary tag and the customer service summary tag are grouped according to the corresponding relationship, the timestamp and the grouping time interval.

3. The customer service conversation quality inspection method according to claim 1 or 2, characterized in that: Also includes: receiving updated summary matching rules; The user summary tags and the customer service summary tags in the same group are matched according to the updated summary matching rule.

4. The customer service conversation quality inspection method according to claim 2, characterized in that: Also includes: The adjusted grouping time interval is received, and the grouping result is updated according to the adjusted grouping time interval.

5. The customer service conversation quality inspection method according to claim 1, characterized in that: Determining the quality inspection result according to the matching result includes: If at least one of the matching results is successfully matched, the corresponding quality inspection result is quality inspection passed; If all the matching results fail, the corresponding quality inspection result is quality inspection failure.

6. A customer service conversation quality inspection device, characterized in that: include: The log acquisition module is used to: acquire the customer service conversation log, perform summary recognition on the user conversation and the customer service response in the customer service conversation log according to the summary parameters and the summary recognition model, and respectively determine the user summary label corresponding to the user conversation and the customer service summary label corresponding to the customer service response; A grouping module, used to: determine the user number corresponding to the user session and the customer service number corresponding to the customer service response, and group the user summary tag and the customer service summary tag according to the correspondence between the user number and the customer service number in the customer service session log; A quality inspection module, used to: match the user summary tag and the customer service summary tag in the same group according to a summary matching rule, and determine a quality inspection result according to the matching result; The device also includes a model training module, which is used to: Obtain historical customer service session logs, perform summary recognition on the historical customer service session logs according to summary parameters and an initial summary recognition model, and determine historical user summary tags and historical customer service summary tags; grouping the historical user summary tags and the historical customer service summary tags according to the correspondence between the user ID and the customer ID in the historical customer service session log; Matching the historical user summary tags and the historical customer service summary tags in the same group according to the summary matching rule, and determining the matching accuracy rate; Determine whether the matching accuracy is greater than an accuracy threshold, and if so, determine that the initial summary recognition model is the summary recognition model; if not, adjust the model parameters and execute the model training step in a loop.

7. An electronic device for customer service conversation quality inspection, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images