A method and apparatus for processing conversation information

Abstract

This invention discloses a method and apparatus for processing conversation information, relating to the field of customer service technology. The method includes: extracting individual sentences from conversational statements generated by customer service; classifying the individual sentences using preset positive factor models and negative factor models; wherein the positive factor model is obtained by training a classification model through multiple positive clusters generated from positive key sentence clustering, and the negative factor model is obtained by training a classification model through multiple negative clusters generated from negative key sentence clustering; determining the target evaluation type of the individual sentence based on the evaluation types set by the classification results corresponding to the positive and negative factor models and the classification results of the individual sentences; and determining a service evaluation for customer service based on a preset evaluation system containing evaluation types and the target evaluation type of the individual sentences. This implementation can identify objective problems existing in conversational services based on the conversation and effectively improve the accuracy of conversation analysis.

Description

Technical Field

[0001] This invention relates to the field of customer service technology, and in particular to a method and apparatus for processing conversation information. Background Technology

[0002] On e-commerce platforms, conversations between customer service representatives and consumers can reflect the quality of customer service and the reasons for consumer satisfaction or dissatisfaction. Therefore, analyzing or processing these conversations helps to evaluate and improve customer service, thereby enhancing the consumer experience.

[0003] Currently, customer service evaluation methods mainly use a binary classification model to divide the conversation between customer service representatives and consumers into conversations where consumers are dissatisfied or satisfied, in order to assess whether consumers are satisfied with customer service. However, this method cannot obtain the specific reasons for consumer dissatisfaction and leads to low accuracy in conversation analysis results. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a session information processing method and apparatus that can identify objective problems existing in session services based on sessions and effectively improve the accuracy of session analysis.

[0005] To achieve the above objectives, according to one aspect of the present invention, a customer service session processing method is provided, comprising:

[0006] Extract multiple individual sentences from the conversation generated by customer service;

[0007] For each of the above sentences, the sentences are classified using a preset positive factor model and a negative factor model respectively; wherein, the positive factor model is obtained by training a classification model through multiple positive clusters formed by clustering positive key sentences, and the negative factor model is obtained by training a classification model through multiple negative clusters formed by clustering negative key sentences.

[0008] Based on the evaluation type set for each classification result corresponding to the positive factor model and the negative factor model, and the classification result of each sentence, determine the target evaluation type to which each sentence belongs;

[0009] Based on a pre-set evaluation system that includes evaluation types and the target evaluation type to which each sentence belongs, a service evaluation is determined for the customer service.

[0010] Optionally, the above customer service conversation handling method further includes:

[0011] For each identified positive or negative session sample, the following operation is performed repeatedly until the first preset condition for stopping the loop is met:

[0012] Delete one or more sample sentences included in the positive or negative conversation sample, and construct a new sample conversation using the remaining sample sentences; use a preset binary classification prediction model to predict whether the new sample conversation belongs to a positive evaluation or a negative evaluation.

[0013] After the loop stops, based on the prediction results of each loop, the positive key sentence of the positive conversation sample or the negative key sentence of the negative conversation sample is determined.

[0014] Optionally, the above customer service conversation handling method further includes:

[0015] Clustering is performed on multiple positive key sentences from multiple positive conversation samples and multiple negative key sentences from multiple negative conversation samples, respectively.

[0016] From the clustering results, select multiple positive clusters and multiple negative clusters that meet the preset clustering conditions;

[0017] The classification model is trained by selecting multiple positive clusters and multiple negative clusters that meet the preset clustering conditions.

[0018] Optionally, the above customer service conversation handling method further includes:

[0019] For each positive key sentence included in each of the positive clusters and each negative key sentence included in each of the negative clusters, the following operation is performed iteratively until a preset second condition for stopping the loop is met:

[0020] The positive or negative key sentences are segmented into words.

[0021] Based on the word segmentation results, delete one or more words included in the positive key sentence or the negative key sentence, and construct a new key sentence using the remaining words;

[0022] The new key sentence is predicted using the positive factor model and the negative factor model;

[0023] After stopping the loop, based on the prediction results of each loop, determine the keywords included in the positive key sentence or the keywords included in the negative key sentence;

[0024] Based on the keywords included in each positive key sentence in the same positive cluster, an evaluation type corresponding to the classification result of the positive cluster is set;

[0025] Based on the keywords included in each negative key sentence within the identified negative cluster, an evaluation type corresponding to the classification result of the negative cluster is set.

[0026] Optionally, the evaluation system includes multiple evaluation levels that are related, wherein the evaluation type set for each classification result is set in the last evaluation level of the evaluation system, and the other evaluation levels are set with the reasons for generating the evaluation;

[0027] The step of determining the service evaluation for the customer service includes: based on multiple evaluation levels that are related and the reasons for generating evaluations set by other evaluation levels, finding the reasons for generating evaluations that are associated with the target evaluation type, and determining the reasons for generating evaluations that are associated with the target evaluation type as the reasons for the service evaluation of the customer service.

[0028] Optionally, the above customer service conversation handling method further includes:

[0029] Extract customer service representatives who indicated negative feedback from customer service questionnaires;

[0030] Get the customer service conversation statement that indicates negative feedback;

[0031] The process of splitting multiple sentences from the conversational statements generated by customer service includes: splitting multiple sentences from the conversational statements of customer service indicating negative feedback.

[0032] Optionally, determining the positive key sentences of the positive conversation sample or the negative key sentences of the negative conversation sample includes:

[0033] Filter out the prediction results that indicate that the new sample session corresponding to the positive session sample belongs to the negative evaluation, and determine the missing sample sentence of the new sample session corresponding to the filtered prediction result as the positive key sentence of the positive session sample.

[0034] Filter out the prediction results that indicate that the new sample session corresponding to the negative session sample belongs to the positive evaluation, and determine the missing sample sentence of the new sample session corresponding to the filtered prediction result as the negative key sentence of the negative session sample.

[0035] Optionally, determining the keywords included in the positive key sentence or the keywords included in the negative key sentence includes:

[0036] New key sentences corresponding to prediction results that are inconsistent with the classification results of the positive or negative key sentences are selected, and the missing words in the selected new key sentences are determined to be the keywords of the positive or negative key sentences.

[0037] Secondly, embodiments of the present invention provide a customer service conversation processing device, comprising: a statement splitting module, a classification module, and a service evaluation module, wherein...

[0038] The statement splitting module is used to split multiple single sentences from the conversational statements generated by customer service.

[0039] The classification module is used to classify each sentence using a preset positive factor model and a negative factor model; wherein the positive factor model is obtained by training a classification model through multiple positive clusters formed by clustering positive key sentences, and the negative factor model is obtained by training a classification model through multiple negative clusters formed by clustering negative key sentences.

[0040] The service evaluation module is used to determine the target evaluation type of each sentence based on the evaluation type set for each classification result corresponding to the positive factor model and the negative factor model, as well as the classification result of each sentence; and to determine the service evaluation for the customer service based on a preset evaluation system containing evaluation types and the target evaluation type of each sentence.

[0041] One embodiment of the above invention has the following advantages or beneficial effects: Since the positive factor model is obtained by training a classification model through multiple positive clusters formed by clustering positive key sentences, and the negative factor model is obtained by training a classification model through multiple negative clusters formed by clustering negative key sentences, the positive factor model and the negative factor model form a multi-classification model, and the positive factor model and the negative factor model have relatively high classification accuracy. Therefore, by classifying the individual sentences in the conversation statements generated by customer service through the positive factor model and the negative factor model respectively, and based on the evaluation type set for each classification result corresponding to the positive factor model and the classification result of each individual sentence, the target evaluation type to which each individual sentence belongs can be determined more accurately. This allows the service evaluation for customer service to more realistically reflect customer service, realize the determination of objective problems existing in the conversation service based on the conversation, and effectively improve the accuracy of conversation analysis.

[0042] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0043] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0044] Figure 1 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0045] Figure 2 This is a schematic diagram of the main flow of the customer service session processing method according to an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of the main process of finding key sentences according to an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of the main process for determining key sentences according to an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of the main process of training a classification model according to an embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram illustrating the main process of searching keywords and setting evaluation types according to an embodiment of the present invention;

[0050] Figure 7 This is a schematic diagram of the evaluation system according to an embodiment of the present invention;

[0051] Figure 8 This is a schematic diagram of the main flow of a customer service session processing method according to another embodiment of the present invention;

[0052] Figure 9 This is a schematic diagram of the main modules of a customer service session processing device according to an embodiment of the present invention;

[0053] Figure 10 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0054] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0055] In existing technologies, customer service is typically evaluated through questionnaires. However, questionnaires offer limited information; for example, they can only indicate customer satisfaction or dissatisfaction, or provide resolution and non-resolution rates, without revealing the specific reasons for changes in resolution rates. This makes it difficult to quickly identify shortcomings in customer service and, more importantly, to guide improvements. Furthermore, existing technologies cannot pinpoint the true reasons for customer dissatisfaction, such as stock shortages, slow delivery, or slow logistics—reasons beyond customer service.

[0056] To address the issues of insufficient objectivity and low accuracy in existing customer service evaluations, this invention provides a method and apparatus for processing session information. Figure 1 An exemplary system architecture 100 is shown in which the session information processing method of embodiments of the present invention can be applied.

[0057] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, a customer service server 105, and a session processing server 106. The network 104 serves as the medium for providing communication links between the customer service server 105 and the terminal devices 101, 102, and 103, and between the session processing server 106 and the customer service server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0058] Users interact with customer service server 105 via network 104 using terminal devices 101, 102, and 103 to send customer service evaluation questionnaires and record conversations or logs between users and customer service via terminal devices 101, 102, and 103. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and social media platforms (for example only).

[0059] The session processing server 106 can be a server that provides various services. It is a backend management server that obtains customer service dialogue content or customer service dialogue logs from the customer service server 105 and provides support for processing and analyzing the customer service dialogue content or customer service dialogue logs (for example only).

[0060] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to desktop computers, laptops, smartphones, tablets, etc.

[0061] It should be noted that the session information processing method provided in this embodiment of the invention is generally performed by the session processing server 106 described above. Accordingly, the session information processing device is installed on the session processing server 106.

[0062] It should be understood that Figure 1 The number of terminal devices, networks, customer service servers, and session processing servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, customer service servers, and session processing servers can be included.

[0063] Figure 2 This is a schematic diagram of the main flow of a customer service session processing method according to an embodiment of the present invention. Figure 2 As shown, the customer service session processing method may include the following steps:

[0064] Step S201: Extract multiple individual sentences from the conversation statements generated by customer service;

[0065] Customer service can be provided by either human or chatbot.

[0066] Customer service conversations generally refer to the dialogue between customer service representatives and consumers during the customer service process. This dialogue can be a text conversation between customer service representatives and consumers conducted directly through instant messaging tools, or it can be a text conversation converted from a voice conversation.

[0067] Breaking down a dialogue into individual sentences typically involves treating a complete dialogue as a single sentence. For example, if a period, question mark, or exclamation mark marks the end of a sentence, then the sentences can be broken down according to the punctuation mark. For sentences without punctuation, the sentences are broken down according to the dialogue partners. For instance, if the dialogue between customer service representative A and customer B is interactive (i.e., the dialogue sequence is ABABA-…), then each of A's dialogue segments is a single sentence, and each of B's ​​dialogue segments is also a single sentence.

[0068] Step S202: For each sentence, classify the sentence using the preset positive factor model and negative factor model respectively; wherein, the positive factor model is obtained by training the classification model through multiple positive clusters formed by clustering positive key sentences, and the negative factor model is obtained by training the classification model through multiple negative clusters formed by clustering negative key sentences.

[0069] In this model, positive key sentences are selected from historical conversation statements that indicate the problem has been resolved by customer service, while negative key sentences are selected from historical conversation statements that indicate the problem has not been resolved by customer service. Both the positive and negative factor models are multi-class classification models. A positive cluster is used to train one classification branch of the multi-class model, and a negative cluster is used to train another classification branch.

[0070] Among them, the positive clusters and negative clusters are obtained through existing text clustering techniques.

[0071] Step S203: Based on the evaluation type set for each classification result corresponding to the positive factor model and the negative factor model, and the classification result of each sentence, determine the target evaluation type to which each sentence belongs;

[0072] Step S204: Based on the preset evaluation system that includes evaluation types and the target evaluation type to which each sentence belongs, determine the service evaluation for customer service.

[0073] This process allows for a comprehensive examination of the classification results of each individual sentence and the target evaluation type to determine the service evaluation of customer service, so that the service evaluation can more objectively and truthfully reflect customer service.

[0074] exist Figure 2 In the illustrated embodiment, the positive factor model is obtained by training a classification model through multiple positive clusters derived from positive key sentence clustering, and the negative factor model is obtained by training a classification model through multiple negative clusters derived from negative key sentence clustering. This allows the positive and negative factor models to form a multi-classification model with relatively high classification accuracy. Therefore, by classifying individual sentences in the customer service conversations using the positive and negative factor models respectively, and based on the evaluation type set for each classification result of the positive and negative factor models and the classification result of each individual sentence, the target evaluation type of each individual sentence can be determined more accurately. This enables the service evaluation for customer service to more realistically reflect customer service, realize the identification of objective problems in the conversation service based on the conversation, and effectively improve the accuracy of conversation analysis.

[0075] In addition, the technical solution provided by this invention can not only provide the result of customer service, i.e., whether it has been resolved or not, but also provide the reason for the result of customer service, i.e. the above-mentioned service evaluation, so as to better guide the improvement of customer service.

[0076] Furthermore, to ensure the accuracy of the positive and negative key sentences used in constructing the positive and negative factor models, and to effectively improve the accuracy of the trained positive and negative factor models, the above customer service conversation processing method can be further improved as follows: Figure 3 As shown, for each identified positive or negative session sample, the following steps S301 and S302 are executed repeatedly until a preset first condition for stopping the loop is met:

[0077] The first condition can be to stop after finding the key sentence, or to stop after all possible new sample sessions have been constructed.

[0078] Step S301: Delete one or more sample sentences included in the positive or negative conversation sample, and construct a new sample conversation using the remaining sample sentences;

[0079] The positive and negative conversation samples are a complete conversation between a customer service representative and a consumer, obtained from historical conversation statements. Furthermore, the positive and negative conversation samples are initially classified using an existing binary classification model; the positive conversation samples represent conversations where the problem has been resolved, as initially identified by the binary classification model, while the negative conversation samples represent conversations where the problem has not been resolved, as initially identified by the binary classification model.

[0080] For example, Table 1 shows an example of a negative session sample.

[0081] Table 1

[0082]

[0083] The process of repeatedly executing step S301 can be as follows: First, delete a single sentence sequentially to construct a new sample session. For example, delete sentence number 1, and construct new sample sessions with the remaining sentences 2-13. Delete sentence number 2, and construct new sample sessions with the remaining sentences 2-13, and so on. Delete sentence number 13, and construct new sample sessions with the remaining sentences 1-12. Then, delete combinations of two single sentences sequentially to construct new sample sessions. For example, delete sentences number 1 and 2, and construct new sample sessions with the remaining sentences 3-13. Delete sentences number 1 and 3, and construct new sample sessions with the remaining sentences 2 and 4-13, and so on. Delete sentences number 1 and 13, and construct new sample sessions with the remaining sentences 2-12. For example, deleting sentences numbered 2 and 3 leaves sentences numbered 1 and 4 through 13, creating a new sample conversation. Similarly, deleting sentences numbered 2 and 4 leaves sentences numbered 1, 3, and 4 through 13, and so on. Deleting sentences numbered 2 and 13 leaves sentences numbered 1 and 3 through 12, and so on. Sequentially deleting combinations of three sentences creates a new sample conversation. For example, deleting sentences numbered 1, 2, and 3 leaves sentences numbered 4 through 13, and so on. There are also combinations of four, five, six, seven, and eight sentences deleted sequentially.

[0084] Step S302: Use a pre-defined binary classification prediction model to predict whether a new sample session is a positive or negative evaluation;

[0085] The preset binary classification prediction model directly identifies sample sessions as either resolved positive evaluations or unresolved negative evaluations.

[0086] For example, by processing the negative session samples given in Table 1 through the above steps S301 and S302, the following Table 2 is obtained.

[0087] Table 2

[0088]

[0089] In Table 2, a new session with a number of 0 indicates that the sentence corresponding to the original sentence number has been deleted, while a new session with a number of 1 indicates that the sentence corresponding to the original sentence number has been retained. m represents the total number of new sessions created.

[0090] Step S303: After stopping the loop, determine the positive key sentence of the positive conversation sample or the negative key sentence of the negative conversation sample based on the prediction results of each loop.

[0091] For example, by analyzing the statements in Table 1 through the above process, it is found that after deleting the second, ninth, and eleventh statements, the type of the conversation statement changes from unresolved to resolved. Therefore, the second, ninth, and eleventh statements in the negative conversation sample in Table 1 can reflect the specific reasons why the problem has not been resolved. These second, ninth, and eleventh statements are the negative key sentences in Table 1.

[0092] The above process can accurately identify positive and negative key sentences, eliminate the interference of non-key sentences on the training process, and effectively improve the accuracy of the trained positive and negative factor models.

[0093] In addition, such as Figure 4 As shown, the specific implementation of determining the positive key sentences of a positive conversation sample or the negative key sentences of a negative conversation sample may include the following steps:

[0094] Step S401: Filter out the prediction results that indicate that the new sample conversation corresponding to the positive conversation sample belongs to the negative evaluation, and determine the missing sample sentence of the new sample conversation corresponding to the filtered prediction result as the positive key sentence of the positive conversation sample.

[0095] Step S402: Filter out the new sample sessions corresponding to the negative session samples that are positive evaluation prediction results, and determine the missing sample sentences of the new sample sessions corresponding to the filtered prediction results as the negative key sentences of the negative session samples.

[0096] The specific implementation of determining the keywords included in the positive key sentence or the keywords included in the negative key sentence may include: filtering out new key sentences corresponding to prediction results that are inconsistent with the classification results of the positive key sentence or the negative key sentence, and determining the missing words in the filtered new key sentences as keywords of the positive key sentence or the negative key sentence.

[0097] Furthermore, such as Figure 5 As shown, the above customer service conversation handling method may further include the following steps:

[0098] Step S501: Cluster the multiple positive key sentences from multiple positive conversation samples and the multiple negative key sentences from multiple negative conversation samples respectively;

[0099] This clustering process is implemented using existing text clustering techniques, which will not be elaborated upon here.

[0100] Step S502: Select multiple positive clusters and multiple negative clusters that meet the preset clustering conditions from the clustering results;

[0101] The preset clustering conditions can be set according to actual needs. For example, the preset clustering conditions can be clusters with no less than 1000 session samples. The preset clustering conditions can also be clusters that select the top N session samples by their sample count.

[0102] Step S503: Train the classification model using multiple positive clusters and multiple negative clusters that meet the preset clustering conditions.

[0103] Clustering can ensure the consistency of samples used to train each classification branch, thereby effectively improving the classification accuracy of positive factor models and negative factor models.

[0104] Furthermore, the above customer service conversation processing method may further include: for each positive key sentence included in each positive cluster and for each negative key sentence included in each negative cluster, cyclically executing the following... Figure 6 The steps S601 to S603 shown continue until the preset second condition for stopping the loop is met:

[0105] The second condition can be to stop after finding the keyword, or it can be to stop after all new key sentences have been constructed.

[0106] Step S601: Segment the positive or negative key sentences into words;

[0107] The word segmentation process can be implemented using existing word segmentation methods. Taking the second sentence in Table 1 above, "When I placed the order, it was clearly in stock, but now they are calling to say it is out of stock," as an example, the word sequence number and segmentation results after word segmentation are shown in columns 1 and 2 of Table 3 below.

[0108] Table 3

[0109] Word number Sentence segmentation Sentence Construction 1 Sentence Construction 2 Sentence Construction 3 ...... Construct sentence m-1 Construct sentence m 1 Place an order 0 1 1 1 1 2 of 1 0 1 1 1 3 when 1 1 1 1 1 4 Mingming 1 1 1 1 1 5 In stock 1 1 1 1 1 6 ， 1 1 1 1 1 7 Now 1 1 1 ...... 1 1 8 again 1 1 1 ...... 1 1 9 beat 1 1 1 ...... 1 1 10 Telephone 1 1 1 ...... 0 1 11 Come 1 1 1 ...... 0 0 12 explain 1 1 1 ...... 0 0 14 Out of stock 1 1 0 ...... 1 0 Prediction results Product out of stock Product out of stock Product out of stock Other ...... Product out of stock Product out of stock

[0110] Step S602: Based on the word segmentation results, delete one or more words included in the positive or negative key sentence, and construct a new key sentence using the remaining words;

[0111] The specific implementation of this step is as follows: After deleting a word in sequence, the remaining words construct new key sentences. For example, in Table 3, deleting word number 1, words numbered 2-14 construct new key sentences (i.e., constructed sentence 1 in Table 3); deleting word number 2, words numbered 1 and 3-14 construct new key sentences (i.e., constructed sentence 2 in Table 3), and so on. Then, after deleting two words in sequence, the remaining words construct new key sentences. For example, deleting words numbered 1 and 2 in Table 3 below, words numbered 3-14 construct new key sentences; deleting words numbered 1 and 3, words numbered 2 and 4-14 construct new key sentences. 14. Construct new key sentences, etc., by deleting words with indices 2 and 3, words with indices 1 and 4-14, etc.; then, after sequentially deleting three words, the remaining words construct new key sentences; after sequentially deleting four words, the remaining words construct new key sentences; after sequentially deleting five words, the remaining words construct new key sentences; after sequentially deleting six words, the remaining words construct new key sentences; after sequentially deleting seven words, the remaining words construct new key sentences; after sequentially deleting eight words, the remaining words construct new key sentences, and so on. In other words, this process can produce sentences composed of any combination of words.

[0112] Step S603: Predict new key sentences using positive and negative factor models;

[0113] This step involves classifying the newly constructed key sentences and filtering out those whose types have changed. The missing words in these new key sentences become the keywords of the key sentences. For example, in the new key sentences constructed in Table 3, when the 14th word is missing, the classification type is no longer out of stock. Therefore, the 14th word "out of stock" becomes the keyword of the key sentence "It was clearly in stock when I placed the order, but now they're calling to say it's out of stock."

[0114] Step S604: After stopping the loop, determine the keywords included in the positive key sentence or the keywords included in the negative key sentence based on the prediction results of each loop;

[0115] Step S605: Based on the keywords included in each positive key sentence in the identified positive cluster, set the evaluation type corresponding to the classification result of the positive cluster;

[0116] For example, the two clusters obtained through the above process, along with their key sentences and keywords, are shown in Table 4 below. By summarizing different keywords within the same cluster, an evaluation type corresponding to that cluster can be obtained. This summarization process can be done manually.

[0117] Table 4

[0118]

[0119] Step S606: Based on the keywords included in each negative key sentence in the identified negative cluster, set the evaluation type corresponding to the classification result of the negative cluster.

[0120] The process of redefining the evaluation type described above enables standardized management of the classification results.

[0121] In addition, the evaluation system may include multiple evaluation levels that are related, wherein the evaluation type set for each classification result is set at the last evaluation level of the evaluation system, and the other evaluation levels are set with reasons for generating the evaluation; accordingly, the specific implementation method for determining service evaluation for customer service may include: based on the multiple evaluation levels that are related and the reasons for generating the evaluation set by the other evaluation levels, finding the reasons for generating the evaluation associated with the target evaluation type, and determining the reasons for generating the evaluation associated with the target evaluation type as the reasons for the service evaluation for customer service.

[0122] like Figure 7The diagram illustrates the structure of an evaluation system. This system includes a first level: external factors – overall shopping experience and internal factors – customer service shopping experience. The second level, corresponding to the external factors – overall shopping experience, can be divided into negative categories such as delivery, after-sales service, products, order-related issues, and promotions. The second level, corresponding to the internal factors – customer service shopping experience, can be divided into negative categories such as service baseline, service attitude, and service capability. Furthermore, the second level corresponding to positive categories is omitted for both external factors – overall shopping experience and internal factors – customer service shopping experience. The third level includes: delivery delays and stockouts; and after-sales issues such as after-sales problems, inability to provide price protection, and invoice irregularities. The reasons for the negative customer service evaluation can be categorized as follows: 1) Abnormal product packaging, 2) Abnormal gifts, and 3) Defective products; 4) Abnormal payment, balance, and credit card transactions; 5) Abnormal flash sales, coupons, discounts, full-reduction offers, and customer reviews; 6) Positive external factors such as good product quality, fast logistics, and low costs; 7) Service bottom lines such as leaking company secrets and diverting customers to other platforms; 8) Poor service attitude, ignoring complaints, passing the buck, and failing to soothe emotions; 9) Lack of consensus in communication, unfulfilled promises, and repetitive use of the same solution; and 10) Positive internal factors such as attentive service, consensus in communication, and fulfillment of promises. For example, if the evaluation type is determined to be "out of stock," then based on the related evaluation levels (delivery, negative types, and external factors - overall shopping experience), the reason for the customer service evaluation can be determined as: negative delivery issues within the external factor - overall shopping experience.

[0123] Furthermore, the aforementioned customer service conversation processing method may further include: extracting customer service representatives indicating negative feedback from customer service questionnaires; obtaining the conversational statements of the customer service representatives indicating negative feedback; and correspondingly, a specific implementation method for splitting multiple individual sentences from the conversational statements generated by the customer service representatives may include: splitting multiple individual sentences from the conversational statements of the customer service representatives indicating negative feedback. In other words, this application primarily aims to further analyze the reasons for negative feedback on customer service representatives who have been pre-evaluated as having negative feedback, thereby reducing workload while improving customer service.

[0124] In summary, the customer service conversation processing procedure described above mainly consists of two parts: one part involves constructing positive and negative factor models, setting classification types for each branch of these models, and determining the evaluation system; the other part is the evaluation of the customer service conversation. The following example illustrates the processing steps involved in the customer service conversation processing method in detail. Figure 8 As shown, the customer service session processing may include the following steps:

[0125] Step S801: Construct positive factor model and negative factor model, set evaluation types for each classification branch of positive factor model and negative factor model, and construct evaluation system for evaluation types;

[0126] The specific implementation of this step may include: retrieving positive key sentences from positive conversation samples and negative key sentences from negative conversation samples respectively; clustering the positive and negative key sentences respectively; training classification models using the key sentences in the clusters to obtain positive factor models and negative factor models. Further, keywords are retrieved from the positive and negative key sentences respectively, and evaluation types are set for the integrated keywords.

[0127] Step S802: Separate multiple individual sentences from the conversation statements generated by customer service;

[0128] For example, the single sentences obtained by splitting the conversation statements of an instance are shown in Table 5 below.

[0129] Table 5

[0130]

[0131]

[0132] Step S803: For each sentence in Table 5, classify the sentence using the preset negative factor model;

[0133] For conversational statements that have been identified as negative evaluations, the negative factor model is used to classify the individual statements directly.

[0134] Step S804: Based on the evaluation type set for each classification result corresponding to the negative factor model and the classification result of each sentence, determine the target evaluation type to which each sentence belongs;

[0135] The classification types of each sentence are shown in Table 5.

[0136] Step S805: Based on the preset evaluation system that includes evaluation types and the target evaluation type to which each sentence belongs, determine the service evaluation for customer service.

[0137] For the target evaluation types in Table 5, namely delivery delay and out-of-stock items, the service evaluation is a negative evaluation caused by external factors - negative delivery reasons under the overall shopping experience, rather than internal factors of customer service.

[0138] Figure 9 This is a schematic diagram of the structure of a customer service conversation processing device provided in an embodiment of the present invention. Figure 9As shown, the customer service conversation processing device 900 may include: a statement splitting module 901, a classification module 902, and a service evaluation module 903, wherein,

[0139] The statement splitting module 901 is used to split multiple single sentences from the conversation statements generated by customer service.

[0140] The classification module 902 is used to classify each sentence using a preset positive factor model and a negative factor model respectively. The positive factor model is obtained by training a classification model through multiple positive clusters formed by clustering positive key sentences, and the negative factor model is obtained by training a classification model through multiple negative clusters formed by clustering negative key sentences.

[0141] The service evaluation module 903 is used to determine the target evaluation type of each sentence based on the evaluation type set for each classification result corresponding to the positive factor model and the negative factor model, as well as the classification result of each sentence; and to determine the service evaluation for customers based on the preset evaluation system containing evaluation types and the target evaluation type of each sentence.

[0142] In embodiments of the present invention, such as Figure 9 As shown, the aforementioned customer service conversation processing device 900 may further include: a key sentence retrieval module 904, wherein,

[0143] The key sentence retrieval module 904 is used to, for each determined positive or negative conversation sample, repeatedly delete one or more sample sentences included in the positive or negative conversation sample, construct a new sample conversation using the remaining sample sentences, and use a preset binary classification prediction model to predict whether the new sample conversation belongs to a positive evaluation or a negative evaluation, until a preset first condition for stopping the loop is met; after stopping the loop, based on the prediction results of each loop, determine the positive key sentence of the positive conversation sample or the negative key sentence of the negative conversation sample.

[0144] In embodiments of the present invention, such as Figure 9 As shown, the aforementioned customer service conversation processing device 900 may further include: a training module 905, wherein,

[0145] Training module 905 is used to cluster multiple positive key sentences from multiple positive conversation samples and multiple negative key sentences from multiple negative conversation samples respectively; select multiple positive clusters and multiple negative clusters that meet the preset clustering conditions from the clustering results; and train classification models using the selected multiple positive clusters and multiple negative clusters that meet the preset clustering conditions respectively.

[0146] In embodiments of the present invention, such as Figure 9As shown, the aforementioned customer service conversation processing device 900 may further include: a keyword retrieval module 906 and a model setting module 907, wherein,

[0147] The keyword retrieval module 906 is used to iteratively perform word segmentation on each positive key sentence included in each positive cluster and each negative key sentence included in each negative cluster, for each positive key sentence included in each positive cluster and for each negative key sentence included in each negative cluster. Based on the word segmentation results, one or more words included in the positive key sentence or negative key sentence are deleted, and new key sentences are constructed using the remaining words. The new key sentences are predicted using positive factor models and negative factor models until a preset second condition for stopping the loop is met. After stopping the loop, the keywords included in the positive key sentence or the keywords included in the negative key sentence are determined based on the prediction results of each loop.

[0148] The model setting module 907 is used to set the evaluation type of the classification result corresponding to the positive cluster based on the keywords included in each positive key sentence in the same positive cluster; and to set the evaluation type of the classification result corresponding to the negative cluster based on the keywords included in each negative key sentence in the same negative cluster.

[0149] In this embodiment of the invention, the evaluation system includes multiple evaluation levels that are related to each other. The evaluation type for each classification result is set at the last evaluation level of the evaluation system, and the other evaluation levels are set with the reasons for generating the evaluation.

[0150] The service evaluation module 903 is further used to find the reasons for generating evaluations associated with the target evaluation type based on multiple evaluation levels that have a relationship and the reasons for generating evaluations set by other evaluation levels, and to determine the reasons for generating evaluations associated with the target evaluation type as the reasons for service evaluations for customer service.

[0151] In this embodiment of the invention, the statement splitting module 901 is further configured to extract customer service indicating negative feedback from the customer service questionnaire; obtain the conversational statements of the customer service indicating negative feedback; and split multiple single sentences from the conversational statements of the customer service indicating negative feedback.

[0152] In this embodiment of the invention, the key sentence retrieval module 904 is further configured to filter out prediction results indicating that the new sample session corresponding to the positive session sample belongs to the negative evaluation, and determine that the missing sample sentence in the new sample session corresponding to the filtered prediction results is the positive key sentence of the positive session sample; and to filter out prediction results indicating that the new sample session corresponding to the negative session sample belongs to the positive evaluation, and determine that the missing sample sentence in the new sample session corresponding to the filtered prediction results is the negative key sentence of the negative session sample.

[0153] In this embodiment of the invention, the keyword retrieval module 906 is further used to filter out new key sentences corresponding to prediction results that are inconsistent with the classification results of positive key sentences or negative key sentences, and to determine that the missing words in the filtered new key sentences are keywords of positive key sentences or negative key sentences.

[0154] The following is for reference. Figure 10 It shows a schematic diagram of the structure of a computer system 1000 equipped with an electronic prescription processing device, which is suitable for implementing embodiments of the present invention. Figure 10 The server shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0155] like Figure 10 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage section 1008 into a random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the system 1000. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0156] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 1010 as needed so that computer programs read from it can be installed into storage section 1008 as needed.

[0157] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by central processing unit (CPU) 1001, it performs the functions defined in the system of this invention.

[0158] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0159] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor may be described as including a statement splitting module, a classification module, and a service evaluation module. The names of these modules do not necessarily limit the module itself; for example, the statement splitting module may also be described as "a module that splits multiple single sentences from conversational statements generated by customer service."

[0160] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0161] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: splitting multiple single sentences from conversational statements generated by customer service; classifying each single sentence using preset single sentence criteria; wherein, a positive factor model is obtained by training a classification model through multiple positive clusters derived from positive key sentence clustering, and a negative factor model is obtained by training a classification model through multiple negative clusters derived from negative key sentence clustering; determining the target evaluation type to which each single sentence belongs based on the evaluation type set for each classification result corresponding to the positive factor model and the negative factor model, and the classification result of each single sentence; and determining a service evaluation for customer service based on a preset evaluation system containing evaluation types and the target evaluation type to which each single sentence belongs.

[0162] According to the technical solution of the present invention, since the positive factor model is obtained by training a classification model through multiple positive clusters formed by clustering positive key sentences, and the negative factor model is obtained by training a classification model through multiple negative clusters formed by clustering negative key sentences, the positive factor model and the negative factor model form a multi-classification model, and the positive factor model and the negative factor model have relatively high classification accuracy. Therefore, by classifying the individual sentences in the conversation statements generated by customer service through the positive factor model and the negative factor model respectively, and based on the evaluation type set for each classification result corresponding to the positive factor model and the classification result of each individual sentence, the target evaluation type to which each individual sentence belongs can be determined more accurately. This allows the service evaluation for customer service to more realistically reflect customer service, realize the determination of objective problems existing in the conversation service based on the conversation, and effectively improve the accuracy of conversation analysis.

[0163] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A customer service conversation processing method, characterized in that, include: Extract multiple individual sentences from the conversation generated by customer service; For each of the above sentences, the sentences are classified using a preset positive factor model and a negative factor model respectively; wherein, the positive factor model is obtained by training a classification model through multiple positive clusters formed by clustering positive key sentences, and the negative factor model is obtained by training a classification model through multiple negative clusters formed by clustering negative key sentences. Based on the evaluation type set for each classification result corresponding to the positive factor model and the negative factor model, and the classification result of each sentence, determine the target evaluation type to which each sentence belongs; Based on a pre-set evaluation system that includes evaluation types and the target evaluation type to which each sentence belongs, a service evaluation is determined for the customer service.

2. The customer service conversation processing method according to claim 1, characterized in that, Also includes: For each identified positive or negative session sample, the following operation is performed repeatedly until the first preset condition for stopping the loop is met: Delete one or more sample sentences included in the positive or negative conversation sample, and construct a new sample conversation using the remaining sample sentences; use a preset binary classification prediction model to predict whether the new sample conversation belongs to a positive evaluation or a negative evaluation. After the loop stops, based on the prediction results of each loop, the positive key sentence of the positive conversation sample or the negative key sentence of the negative conversation sample is determined.

3. The customer service session processing method according to claim 1 or 2, characterized in that, Also includes: Clustering is performed on multiple positive key sentences from multiple positive conversation samples and multiple negative key sentences from multiple negative conversation samples, respectively. From the clustering results, select multiple positive clusters and multiple negative clusters that meet the preset clustering conditions; The classification model is trained by selecting multiple positive clusters and multiple negative clusters that meet the preset clustering conditions.

4. The customer service session processing method according to claim 1 or 2, characterized in that, Also includes: For each positive key sentence included in each of the positive clusters and each negative key sentence included in each of the negative clusters, the following operation is performed iteratively until a preset second condition for stopping the loop is met: The positive or negative key sentences are segmented into words. Based on the word segmentation results, delete one or more words included in the positive key sentence or the negative key sentence, and construct a new key sentence using the remaining words; The new key sentence is predicted using the positive factor model and the negative factor model; After stopping the loop, based on the prediction results of each loop, determine the keywords included in the positive key sentence or the keywords included in the negative key sentence; Based on the keywords included in each positive key sentence in the same positive cluster, an evaluation type corresponding to the classification result of the positive cluster is set; Based on the keywords included in each negative key sentence within the identified negative cluster, an evaluation type corresponding to the classification result of the negative cluster is set.

5. The customer service conversation processing method according to claim 1, characterized in that, The evaluation system includes multiple evaluation levels that are related to each other. The evaluation type for each classification result is set at the last evaluation level of the evaluation system, and the other evaluation levels are set with the reasons for generating the evaluation. The step of determining the service evaluation for the customer service includes: based on multiple evaluation levels that are related and the reasons for generating evaluations set by other evaluation levels, finding the reasons for generating evaluations that are associated with the target evaluation type, and determining the reasons for generating evaluations that are associated with the target evaluation type as the reasons for the service evaluation of the customer service.

6. The customer service conversation processing method according to claim 1, characterized in that, Also includes: Extract customer service representatives who indicated negative feedback from customer service questionnaires; Get the customer service conversation statement that indicates negative feedback; The process of splitting multiple sentences from the conversational statements generated by customer service includes: splitting multiple sentences from the conversational statements of customer service indicating negative feedback.

7. The customer service conversation processing method according to claim 2, characterized in that, The step of determining the positive key sentence of the positive conversation sample or the negative key sentence of the negative conversation sample includes: Filter out the prediction results that indicate that the new sample session corresponding to the positive session sample belongs to the negative evaluation, and determine the missing sample sentence of the new sample session corresponding to the filtered prediction result as the positive key sentence of the positive session sample. Filter out the prediction results that indicate that the new sample session corresponding to the negative session sample belongs to the positive evaluation, and determine the missing sample sentence of the new sample session corresponding to the filtered prediction result as the negative key sentence of the negative session sample.

8. The customer service conversation processing method according to claim 3, characterized in that, The determination of the keywords included in the positive key sentence or the keywords included in the negative key sentence includes: New key sentences corresponding to prediction results that are inconsistent with the classification results of the positive or negative key sentences are selected, and the missing words in the selected new key sentences are determined to be the keywords of the positive or negative key sentences.

9. A customer service conversation processing device, characterized in that, include: The module includes a statement splitting module, a classification module, and a service evaluation module. The statement splitting module is used to split multiple single sentences from the conversational statements generated by customer service. The classification module is used to classify each sentence using a preset positive factor model and a negative factor model; wherein the positive factor model is obtained by training a classification model through multiple positive clusters formed by clustering positive key sentences, and the negative factor model is obtained by training a classification model through multiple negative clusters formed by clustering negative key sentences. The service evaluation module is used to determine the target evaluation type of each sentence based on the evaluation type set for each classification result corresponding to the positive factor model and the negative factor model, as well as the classification result of each sentence; and to determine the service evaluation for the customer service based on a preset evaluation system containing evaluation types and the target evaluation type of each sentence.

10. An electronic device, characterized in that, include: one or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

11. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

Images