Method and apparatus for detecting a repeat resolution scheme for a customer service session

Abstract

The application discloses a kind of methods and devices for detecting repeated scheme of customer service conversation, it is related to computer technical field.The specific embodiment of the method includes: obtaining the session sentence set included in the customer service conversation to be detected, the session sentence set includes the customer sentence and customer service sentence of multiple round conversations;The similarity between each two customer service sentences is calculated respectively, and the customer service sentence that similarity meets set threshold is saved to repeated scheme candidate set;For each candidate customer service sentence in the repeated scheme candidate set, obtain the context session of the candidate customer service sentence, the context session includes customer sentence and customer service sentence;According to the context session of each candidate customer service sentence, the repeated scheme detection is carried out to the customer service conversation to be detected.This embodiment makes the detection result of repeated scheme more accurate under the premise of guaranteeing recall rate, greatly simplifies model, facilitates production deployment, and improves customer service quality.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for detecting duplicate patterns in customer service conversations. Background Technology

[0002] Repeated or single solution quality inspection is a quality inspection method for customer service in the e-commerce customer service field. Its business definition is: the detection of customer service conversations in which the customer service representative sends the same or similar solution to the customer more than twice in order to solve a customer's problem. Existing repeated or single solution quality inspection methods are generally divided into two types: (1) using semantic matching technology to judge the similarity of each sentence of the customer service reply. If two or more sentences are similar, the customer conversation is a repeated solution or a single solution; (2) performing deep learning modeling on a complete conversation, with the customer's and customer service's words being input into a model at the same time to determine whether the conversation is a repeated solution or a single solution.

[0003] In the process of realizing this invention, the inventors discovered at least the following problems in the prior art:

[0004] Method (1) only analyzes customer service statements without considering the conversation scenario, resulting in inaccurate detection results for duplicate or single scenarios; Method (2) models the entire conversation, which is complex, has low detection efficiency, and is not conducive to production deployment in scenarios with a large number of conversations, thus limiting its application. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method and apparatus for detecting duplicate schemes in customer service conversations. This method can perform preliminary screening or recall of duplicate schemes based on the similarity between customer service statements, ensuring a high recall rate. While maintaining a high recall rate, the method further detects duplicate schemes in the customer service conversation by selecting only the context of each candidate customer service statement in the duplicate scheme candidate set. This approach considers the conversational context of the customer service statements, making the detection results more accurate. Furthermore, it eliminates the need to model the entire conversation, greatly simplifying the model, improving the detection efficiency, facilitating production deployment, and enhancing customer service quality.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for detecting repetitive patterns in customer service conversations is provided, comprising:

[0007] Obtain the set of conversation statements included in the customer service conversation to be detected, wherein the set of conversation statements includes customer statements and customer service statements from multiple rounds of conversation;

[0008] Calculate the similarity between each pair of customer service statements, and save the customer service statements whose similarity meets the set threshold to the duplicate solution candidate set.

[0009] For each candidate customer service statement in the candidate set of repeated solutions, obtain the context session of the candidate customer service statement, the context session including the customer statement and the customer service statement;

[0010] The customer service session to be detected is subjected to duplicate scheme detection based on the context of each candidate customer service statement.

[0011] Optionally, before calculating the similarity between each pair of customer service statements, the method further includes: deleting a specified customer service statement from the set of conversation statements, wherein the specified customer service statement is a preset non-scheme customer service statement that conforms to business rules.

[0012] Optionally, the similarity between the two customer service statements is calculated based on the length of the longest common subsequence between the two customer service statements and the string length of the two customer service statements.

[0013] Optionally, the similarity between each pair of customer service statements is calculated separately, and customer service statements whose similarity meets a set threshold are saved to the duplicate scheme candidate set, including: calculating the similarity between each pair of customer service statements separately, saving two customer service statements whose similarity meets a set threshold to a first candidate set; performing a union operation on multiple first candidate sets with intersection to obtain a duplicate scheme candidate set, wherein the duplicate scheme candidate set contains no fewer than three candidate customer service statements.

[0014] Optionally, if all first candidate sets have no overlap, the customer service session to be detected does not include duplicate schemes.

[0015] Optionally, the customer service session to be detected is subjected to duplicate scheme detection based on the context session of each candidate customer service statement, including: determining whether each pair of candidate customer service statements is a duplicate scheme based on the context session of each candidate customer service statement; if there are not less than three candidate customer service statements in the duplicate scheme candidate set that are duplicate schemes, then the customer service session to be detected includes a duplicate scheme.

[0016] Optionally, determining whether two pairs of candidate customer service statements are duplicate schemes based on the context of each candidate customer service statement includes: extracting features from the context of each candidate customer service statement to obtain feature vectors for the two candidate customer service statements; inputting the feature vectors of the two candidate customer service statements into a pre-trained first judgment unit to determine whether the two candidate customer service statements are scheme statements; concatenating the feature vectors of the two candidate customer service statements and inputting them into a pre-trained second judgment unit to determine whether the two candidate customer service statements are duplicates; if the two candidate customer service statements are scheme statements and are duplicates, then the two candidate customer service statements are duplicate schemes.

[0017] Optionally, feature extraction is performed on the context of the two candidate customer service statements to obtain feature vectors for the two candidate customer service statements, including: encoding each sentence in the context of the two candidate customer service statements to obtain a set of sentence vectors; and inputting the set of sentence vectors into a natural language processing encoder for processing to obtain feature vectors for the two candidate customer service statements.

[0018] Optionally, the context session includes the customer statement and customer service statement in the round session where the candidate customer service statement is located, as well as the customer statement in the next round.

[0019] According to another aspect of the present invention, an apparatus for detecting repetitive patterns in customer service sessions is provided, comprising:

[0020] The conversation statement acquisition module is used to acquire the set of conversation statements included in the customer service conversation to be detected. The set of conversation statements includes customer statements and customer service statements from multiple rounds of conversation.

[0021] The candidate statement determination module is used to calculate the similarity between each pair of customer service statements and save customer service statements whose similarity meets the set threshold to the duplicate solution candidate set.

[0022] The context session acquisition module is used to acquire the context session of each candidate customer service statement in the candidate set of repeated solutions.

[0023] The duplicate scheme detection module is used to perform duplicate scheme detection on the customer service session to be detected based on the context of each candidate customer service statement.

[0024] According to another aspect of the present invention, an electronic device for detecting duplicate patterns in customer service sessions is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for detecting duplicate patterns in customer service sessions provided in the embodiments of the present invention.

[0025] According to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method for detecting duplicate patterns in customer service sessions provided in the embodiments of the present invention.

[0026] An embodiment of the above invention has the following advantages or beneficial effects: by acquiring a set of conversational statements included in the customer service session to be detected, the set of conversational statements includes customer statements and customer service statements from multiple rounds of conversation; calculating the similarity between each pair of customer service statements, and saving customer service statements whose similarity meets a set threshold to a duplicate scheme candidate set; for each candidate customer service statement in the duplicate scheme candidate set, acquiring the contextual session of the candidate customer service statement, the contextual session includes both customer statements and customer service statements; and performing duplicate scheme detection on the customer service session to be detected based on the contextual session of each candidate customer service statement, the technical solution realizes preliminary screening or preliminary recall of duplicate schemes based on the similarity between customer service statements, which can guarantee the recall rate of duplicate schemes; under the premise of guaranteeing the recall rate, by selecting only the contextual session of each candidate customer service statement in the duplicate scheme candidate set to further perform duplicate scheme detection on the customer service session to be detected, on the one hand, the conversational scenario of the customer service statements is considered, making the detection results of duplicate schemes more accurate; on the other hand, there is no need to model the complete session, which greatly simplifies the model, improves the detection efficiency of the scheme, facilitates production deployment, and improves the quality of customer service.

[0027] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0028] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0029] Figure 1 This is a schematic diagram of the main steps of a method for detecting duplicate patterns in customer service conversations according to an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of the principle architecture of the duplicate scheme detection model according to an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of the main modules of an apparatus for detecting repetitive patterns in customer service sessions according to an embodiment of the present invention;

[0032] Figure 4 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0033] Figure 5This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0034] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0035] The acquisition, storage, use, and processing of data in the technical solution of this invention all comply with the relevant provisions of national laws and regulations.

[0036] The inventors have analyzed the two existing repetitive or single-method quality inspection methods as follows:

[0037] (1) Semantic matching technology is used to determine the similarity of each sentence in the customer service reply. If two or more sentences are similar, the customer conversation is considered to be a duplicate solution or a single solution. The threshold for determining whether two sentences are semantically similar using semantic matching technology is not easy to determine. If the threshold is too high, it is easy to miss the recall. If the threshold is too low, the recalled sentences are more likely to be semantically dissimilar. The customer's feedback on the solution is not considered. Even if the solution is semantically similar, it is not necessarily a duplicate solution from a business perspective. For example, in the following scenarios: when the customer is unclear about the solution, the customer service will answer the customer service multiple times when the customer confirms again; when the customer accepts the solution, the customer service will give multiple similar replies; the customer service will give similar replies to different questions. It can be seen that method (1) only analyzes the customer service sentences and does not combine the conversation scenario, which leads to inaccurate detection results of duplicate solutions or single solutions.

[0038] (2) Deep learning modeling is performed on a complete conversation, with both customer and customer service utterances input into a single model to determine whether the conversation is a repetitive or single-solution scenario. For this scenario, conversation-dimensional modeling is usually very complex. Actual data analysis shows that similar customer service scenarios often span many rounds of dialogue. Conversation-dimensional modeling itself requires a large amount of data, and the proportion of repetitive or single-solution scenarios is very low. In order to obtain more positive samples, the number of labeled conversations will be even greater. The number of conversations per day for human customer service in the e-commerce field is large, and complex models are not conducive to production deployment. Therefore, it can be seen that method (2) models the complete conversation, which is complex, has low scheme detection efficiency, and is not conducive to production deployment in scenarios with a large number of conversations, thus limiting its application.

[0039] To address the aforementioned technical problems, this invention provides a method for detecting duplicate scenarios in customer service conversations. This method is a simple and efficient quality inspection method for duplicate or single scenarios, improving the accuracy of duplicate scenario detection while maintaining recall. It solves the problem of effectively modeling duplicate or single scenarios, and by leveraging the concept of recall + model-based judgment, this method can be applied to production environments, thereby improving customer service quality.

[0040] Figure 1 This is a schematic diagram illustrating the main steps of a method for detecting duplicate patterns in customer service sessions according to an embodiment of the present invention. Figure 1 As shown, the method for detecting duplicate patterns in customer service sessions according to this embodiment of the invention mainly includes the following steps S101 to S104.

[0041] Step S101: Obtain the set of conversation statements included in the customer service session to be tested. The set of conversation statements includes customer statements and customer service statements from multiple rounds of conversation. For a single customer service session, it includes multiple rounds of interactive conversation between the customer and the customer service representative. One round of the conversation consists of consecutive customer statements followed by a customer service statement. Let T be the number of rounds of conversation. i Let i represent the i-th round of a conversation. Then, a round of a conversation can be further represented as (Q i1 Q i2 ,…,Q ij A i1 A i2 ,…A ik ), Q ij Represents the i-th round of conversation T i The j-th client statement in the text, A ik Represents the i-th round of conversation T i The k-th customer service statement in [T1,T2,…,T] i ,…,T n This refers to the entire customer service conversation. When performing duplicate scheme detection on a customer service conversation, the process involves checking whether the conversation contains duplicate schemes based on the set of conversation statements included in the conversation to be tested. This set of conversation statements includes both customer statements and customer service statements from multiple rounds of conversation.

[0042] Step S102: Calculate the similarity between every two customer service statements, and save the customer service statements whose similarity meets the set threshold to the duplicate scheme candidate set. According to the technical solution of the present invention, when performing duplicate scheme detection on the customer service session to be detected, the duplicate schemes are first initially screened or initially recalled based on the similarity between the customer service statements to obtain a duplicate scheme candidate set that needs further processing.

[0043] According to one embodiment of the present invention, before step S102, that is, before calculating the similarity between each pair of customer service statements, specified customer service statements in the set of conversation statements can be deleted. These specified customer service statements are preset non-scheme customer service statements that conform to business rules. That is, before performing preliminary screening of duplicate schemes, the customer service conversation to be detected can be preprocessed by deleting non-scheme customer service statements that conform to business rules from the set of conversation statements included in the customer service conversation to be detected. The preprocessing is for customer service statements, and its purpose is to delete certain customer service statements that conform to business rules from the conversation, preventing these statements from entering the subsequent process. The preset non-scheme customer service statements that conform to business rules include, for example, the following:

[0044] 1) Sincere confirmation type, for example: Do you have any other questions you would like to ask?

[0045] 2) Polite responses: For example: Sorry to keep you waiting.

[0046] 3) Customer waiting: For example: Please wait a moment, let me take a look.

[0047] 4) Requesting information: For example: Could you please provide your order number?

[0048] 5) Exceptionally long customer service replies: For example, a product introduction.

[0049] Since these customer service statements that conform to business rules often appear multiple times in a single customer service session, and these statements are not solutions to the problem, deleting these statements in advance can effectively reduce the amount of duplicate solutions in the initial screening or initial recall, while improving the accuracy of subsequent models in detecting duplicate solutions.

[0050] According to an embodiment of the present invention, step S102 calculates the similarity between every two customer service statements and saves customer service statements whose similarity meets a set threshold to a duplicate scheme candidate set. In specific implementation, this may include: calculating the similarity between every two customer service statements and saving two customer service statements whose similarity meets a set threshold to a first candidate set; performing a union operation on multiple first candidate sets with intersection to obtain a duplicate scheme candidate set, wherein the duplicate scheme candidate set contains no fewer than three candidate customer service statements. Specifically, it iterates through all customer service statements in the conversation statement set, calculates the similarity between every two customer service statements, and when the similarity is greater than a pre-set threshold, the two customer service statements are used as the first candidate set for a duplicate scheme, such as (A...). 12 A 31Next, the union of the first candidate sets that intersect is taken. The sentences in the union are the recalled sentences, resulting in a duplicate solution candidate set. Therefore, the duplicate solution candidate set includes at least three candidate customer service statements. Then, the customer service statements in the obtained duplicate solution candidate set can be input into the model for identification to determine whether they are duplicate solutions. In the specific implementation process, a customer service session to be detected may have multiple first candidate sets, and the duplicate solution candidate set obtained after taking the union may also have multiple sets. In the subsequent processing, each duplicate solution candidate set will be processed separately.

[0051] Specifically, if all the first candidate sets have no overlap, then the customer service session to be detected does not include duplicate schemes. If all the first candidate sets have no overlap, it means that there are only two customer statements with a similarity greater than the set threshold. In this case, it does not meet the determination criteria for duplicate or single schemes of the present invention, and it can be directly determined that the customer service session to be detected does not include duplicate schemes.

[0052] In one embodiment of the present invention, the Longest Common Subsequence (LCS) technique is used for preliminary screening or recall of duplicate schemes. The similarity between two customer service statements is calculated based on the length of the longest common subsequence between the two customer service statements and the string length of the two customer service statements. Specifically, the similarity S between the two customer service statements is... ij The calculation formula is as follows:

[0053] S ij =2*len(LCS(A) ik A jl ) / (len(A ik )+len(A jl ),

[0054] Here, len(.) represents the length of the string, and LCS(.,.) represents the longest common subsequence.

[0055] Based on the similarity between customer service statements, step S102 is used to perform preliminary screening or preliminary recall of duplicate solutions, which can ensure the recall rate of duplicate solutions.

[0056] Step S103: For each candidate customer service statement in the duplicate solution candidate set, obtain the context session of the candidate customer service statement. The context session includes the customer statement and the customer service statement. At least one duplicate solution candidate set can be obtained according to step S102, and then each duplicate solution candidate set will be processed separately. Specifically, firstly, for each candidate customer service statement in the duplicate solution candidate set, obtain the context session of the candidate customer service statement. In the embodiments of the present invention, the context session includes the customer statement and the customer service statement in the round session where the candidate customer service statement is located, and the customer statement in the next round. Assume that a duplicate solution candidate set includes three candidate customer service statements, namely A... iu A jv and A kw For candidate customer service statement A iu In other words, obtain the session T in the current round. i Customer statement set Q i (For example, including: Q) i1 Q i2 ,…,Q ij (etc.) and customer service statement set A i (For example, including: A) i1 A i2 ,…A ik (etc.), and the next round T after its current round. i+1 Customer statement set Q i+1 (For example, including: Q) (i+1)1 Q (i+1)2 ,…,Q (i+1)m (etc.); for candidate customer service statement A jv In other words, obtain the session T in the current round. j Customer statement set Q j Customer service statement set A j And the next round T after its current round. j+1 Customer statement set Q j+1 For candidate customer service statement A kw The same method is used to obtain its context session, which will not be elaborated here.

[0057] According to an embodiment of the present invention, by selecting only the contextual session of each candidate customer service statement in the candidate set of duplicate schemes, duplicate scheme detection is further performed on the customer service session to be detected. On the one hand, the session scenario of the customer service statement is taken into account, making the detection results of duplicate schemes more accurate; on the other hand, there is no need to model the complete session, which greatly simplifies the model, improves the detection efficiency of the scheme, facilitates production deployment, and improves the quality of customer service.

[0058] Step S104: Perform duplicate scheme detection on the customer service session to be detected based on the context session of each candidate customer service statement. According to an embodiment of the present invention, when performing duplicate scheme detection on the customer service session to be detected based on the context session of each candidate customer service statement, specifically: First, determine whether each pair of candidate customer service statements is a duplicate scheme based on the context session of each candidate customer service statement; if there are not less than three candidate customer service statements in the duplicate scheme candidate set that are duplicate schemes, then determine that the customer service session to be detected includes a duplicate scheme.

[0059] According to one embodiment of the present invention, when determining whether two pairs of candidate customer service statements are duplicate solutions based on the context of each candidate customer service statement, it is necessary to determine whether the candidate customer service statements are duplicates and whether the two candidate customer service statements are solution statements. For each candidate customer service statement in the duplicate solution candidate set obtained in step S102, they only have certain similarities in terms of text, but they are not necessarily duplicates in terms of semantics. At the same time, considering the business definition dimension and the context, they are not necessarily duplicate answers to user questions. Finally, not all of these candidate customer service statements are solutions. Therefore, the present invention needs to further determine whether two candidate customer service statements are duplicates and whether two candidate customer service statements are solution statements to determine whether two pairs of candidate customer service statements are duplicate solutions.

[0060] Specifically, feature vectors for the two candidate customer service statements can be obtained by extracting features from the context of the two candidate customer service statements. These feature vectors are then input into a pre-trained first judgment unit to determine whether the two candidate customer service statements are solution statements. The feature vectors of the two candidate customer service statements are then concatenated and input into a pre-trained second judgment unit to determine whether the two candidate customer service statements are duplicates. If the two candidate customer service statements are solution statements and are duplicates, then the two candidate customer service statements are duplicate solutions. In an embodiment of the invention, a model can be trained in advance using training data to obtain a first judgment unit for determining whether a customer service statement is a solution statement and a second judgment unit for determining whether two candidate customer service statements are duplicates. The determination of whether two candidate customer service statements are duplicate solutions can then be made based on the judgment results of the first and second judgment units.

[0061] According to the technical solution of the present invention, although the customer service session to be detected has been preprocessed before step S102 by deleting non-scheme customer service statements that conform to the business rules from the set of session statements included in the customer service session to be detected, a pre-trained first judgment unit is still needed to accurately determine whether the candidate customer service statement is a scheme statement in order to perform duplicate scheme detection more accurately. Similarly, although the initial screening and initial recall of duplicate schemes have been performed in step S102, a pre-trained second judgment unit is still needed to accurately determine whether the candidate customer service statement is repeated in order to perform duplicate scheme detection more accurately. Specifically, the first judgment unit and the second judgment unit can be implemented, for example, as a binary classifier.

[0062] Specifically, when extracting features from the context of the two candidate customer service statements to obtain their feature vectors, the process includes: encoding each sentence in the context of the two candidate customer service statements to obtain a set of sentence vectors; and inputting the set of sentence vectors into a natural language processing encoder for processing to obtain the feature vectors of the two candidate customer service statements. By encoding each sentence in the context of the two candidate customer service statements to obtain a set of sentence vectors, and then inputting the set of sentence vectors into a natural language processing encoder for processing, semantic analysis can be performed using the context when generating the feature vectors of the candidate customer service statements, thereby more accurately determining whether the two candidate customer service statements are duplicate solutions in the business scenario.

[0063] In an embodiment of the present invention, step S104, when performing duplicate scheme detection on the customer service session to be detected based on the context of each candidate customer service statement, can be performed, for example, by using a pre-trained duplicate scheme detection model. This duplicate scheme detection model can be obtained by labeling a large amount of sample data and then performing machine learning. Referring to the foregoing embodiments, suppose a duplicate scheme candidate set includes three candidate customer service statements, namely A... iu A jv and A kw Then, three samples can be generated by combining them in pairs, and each sample can be labeled to indicate whether the two candidate customer service statements are a solution and whether they are duplicates. In practice, since duplicate solutions are more likely to occur between adjacent rounds of conversation, it is not necessary to combine candidate customer service statements from rounds that are far apart to generate samples for labeling, in order to reduce training data and improve model training efficiency. Suppose that based on the three candidate customer service statements A in this duplicate solution candidate set... iu A jv and A kw Two samples A were generated. iu and A jv A jv and Akw For sample A iu and A jv Based on the context of the sample, i.e., the i-th round of conversation T i The customer service statements and customer statements, as well as the customer statements in rounds i+1 and j+1, need to be labeled by the expert. iu and A jv Whether it is a solution, and whether they are duplicates. Similarly, for A... jv and A kw The same operation is performed. By using machine learning and training on labeled samples obtained after labeling a large number of samples, a duplicate scheme detection model can be obtained.

[0064] Figure 2 This is a schematic diagram of the principle architecture of the repetition detection model according to an embodiment of the present invention. Figure 2 As shown, the implementation principle of the duplicate scheme detection model in this embodiment of the invention is as follows: First, each sentence in the context of each candidate customer service statement in the duplicate scheme candidate set is encoded into a sentence vector by the same gated recurrent unit (GRU); then, it is processed by a transformer encoder based on natural language processing to obtain A. iu and A jv The corresponding feature vector; then, A iu and A jv The feature vectors are each input into a binary classifier activated by the same cross-entropy loss function sigmoid (i.e., the first decision unit), and A is judged. iu and A jv Whether it is a solution statement, and also through the feature concatenation layer, A iu and A jv The feature vectors are concatenated into a single vector, which is then input into a binary classifier activated by the sigmoid cross-entropy loss function (i.e., the second decision unit) to determine A. iu and A jv Is it a duplicate?

[0065] Figure 3 A schematic diagram of the main modules of an apparatus for detecting repetitive patterns in customer service sessions according to an embodiment of the present invention. Figure 3 As shown, the apparatus 300 for detecting duplicate patterns in customer service sessions according to an embodiment of the present invention mainly includes a session statement acquisition module 301, a candidate statement determination module 302, a context session acquisition module 303, and a duplicate pattern detection module 304.

[0066] The conversation statement acquisition module 301 is used to acquire a set of conversation statements included in the customer service conversation to be detected, wherein the set of conversation statements includes customer statements and customer service statements from multiple rounds of conversation;

[0067] The candidate statement determination module 302 is used to calculate the similarity between each pair of customer service statements and save the customer service statements whose similarity meets the set threshold to the duplicate scheme candidate set.

[0068] The context session acquisition module 303 is used to acquire the context session of each candidate customer service statement in the duplicate scheme candidate set, wherein the context session includes the customer statement and the customer service statement.

[0069] The duplicate scheme detection module 304 is used to perform duplicate scheme detection on the customer service session to be detected based on the context of each candidate customer service statement.

[0070] According to an embodiment of the present invention, the apparatus 300 for detecting duplicate patterns in customer service conversations further includes a preprocessing module (not shown in the figure), which is used to delete specified customer service statements in the conversation statement set before calculating the similarity between each pair of customer service statements. The specified customer service statements are preset non-pattern customer service statements that conform to business rules.

[0071] According to another embodiment of the present invention, the similarity between the two customer service statements is calculated based on the length of the longest common subsequence between the two customer service statements and the string length of the two customer service statements.

[0072] According to another embodiment of the present invention, the candidate statement determination module 302 can also be used to: calculate the similarity between each pair of customer service statements, save the two customer service statements whose similarity meets a set threshold to a first candidate set; perform a union operation on multiple first candidate sets with intersection to obtain a duplicate scheme candidate set, wherein the duplicate scheme candidate set contains no less than three candidate customer service statements.

[0073] According to another embodiment of the present invention, if all first candidate sets have no overlap, then the customer service session to be detected does not include duplicate schemes.

[0074] According to another embodiment of the present invention, the duplicate scheme detection module 304 can also be used to: determine whether each pair of candidate customer service statements is a duplicate scheme based on the context of each candidate customer service statement; if there are not less than three candidate customer service statements in the duplicate scheme candidate set that are duplicate schemes, then the customer service session to be detected includes a duplicate scheme.

[0075] According to another embodiment of the present invention, when the duplicate scheme detection module 304 determines whether two pairs of candidate customer service statements are duplicate schemes based on the context of each candidate customer service statement, it can also be used to: extract features from the context of the two candidate customer service statements to obtain feature vectors of the two candidate customer service statements; input the feature vectors of the two candidate customer service statements into a pre-trained first judgment unit to determine whether the two candidate customer service statements are scheme statements; concatenate the feature vectors of the two candidate customer service statements and input them into a pre-trained second judgment unit to determine whether the two candidate customer service statements are duplicates; if the two candidate customer service statements are scheme statements and are duplicates, then the two candidate customer service statements are duplicate schemes.

[0076] According to another embodiment of the present invention, when the duplicate scheme detection module 304 extracts features from the context conversations of the two candidate customer service statements to obtain feature vectors of the two candidate customer service statements, it can also be used to: encode each sentence in the context conversations of the two candidate customer service statements to obtain a set of sentence vectors; input the set of sentence vectors into a natural language processing encoder for processing to obtain feature vectors of the two candidate customer service statements.

[0077] According to another embodiment of the present invention, the context session includes the customer statement and customer service statement of the round in which the candidate customer service statement is located, as well as the customer statement of the next round.

[0078] According to the technical solution of the present invention, by obtaining a set of conversation statements included in the customer service session to be detected, the set of conversation statements includes customer statements and customer service statements from multiple rounds of conversation; calculating the similarity between each pair of customer service statements, and saving customer service statements whose similarity meets a set threshold to a duplicate scheme candidate set; for each candidate customer service statement in the duplicate scheme candidate set, obtaining the context session of the candidate customer service statement, the context session includes both customer statements and customer service statements; and performing duplicate scheme detection on the customer service session to be detected based on the context session of each candidate customer service statement, the technical solution realizes preliminary screening or preliminary recall of duplicate schemes based on the similarity between customer service statements, which can guarantee the recall rate of duplicate schemes; under the premise of ensuring the recall rate, by selecting only the context session of each candidate customer service statement in the duplicate scheme candidate set to further perform duplicate scheme detection on the customer service session to be detected, on the one hand, the conversation scenario of customer service statements is considered, making the detection results of duplicate schemes more accurate; on the other hand, there is no need to model the complete session, which greatly simplifies the model, improves the detection efficiency of the scheme, facilitates production deployment, and improves the quality of customer service.

[0079] Figure 4An exemplary system architecture 400 is shown, which can be applied to the method or apparatus for detecting duplicate patterns in customer service sessions according to embodiments of the present invention.

[0080] like Figure 4 As shown, system architecture 400 may include terminal devices 401, 402, and 403, a network 404, and a server 405. Network 404 serves as the medium for providing communication links between terminal devices 401, 402, and 403 and server 405. Network 404 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0081] Users can use terminal devices 401, 402, and 403 to interact with server 405 via network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, and 403, such as e-commerce service applications, customer service applications, and communication service software (for example only).

[0082] Terminal devices 401, 402, and 403 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0083] Server 405 can be a server providing various services, such as a backend management server supporting customer service sessions submitted by users using terminal devices 401, 402, and 403 (for example only). The backend management server can process received data such as customer service session duplication detection requests to obtain a set of conversation statements included in the customer service session to be detected. This set of conversation statements includes customer statements and customer service statements from multiple rounds of conversation. It can calculate the similarity between every two customer service statements and save customer service statements whose similarity meets a set threshold to a duplication candidate set. For each candidate customer service statement in the duplication candidate set, it obtains the context session of the candidate customer service statement, which includes both customer statements and customer service statements. Based on the context session of each candidate customer service statement, it performs duplication detection and other processing on the customer service session to be detected and feeds back the processing results (e.g., duplication detection results – for example only) to the terminal device.

[0084] It should be noted that the method for detecting duplicate patterns in customer service sessions provided in this embodiment of the invention is generally executed by server 405, and correspondingly, the device for detecting duplicate patterns in customer service sessions is generally located in server 405.

[0085] It should be understood that Figure 4The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0086] The following is for reference. Figure 5 It shows a schematic diagram of the structure of a computer system 500 suitable for implementing terminal devices or servers of the present invention. Figure 5 The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0087] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 502 or programs loaded from storage section 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the system 500. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0088] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.

[0089] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the functions defined above in the system of this invention.

[0090] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0092] The units or modules described in the embodiments of the present invention can be implemented in software or hardware. The described units or modules can also be housed in a processor; for example, a processor can be described as including a session statement acquisition module, a candidate statement determination module, a context session acquisition module, and a duplicate pattern detection module. The names of these units or modules do not necessarily limit the specific unit or module itself; for example, the duplicate pattern detection module can also be described as "a module for performing duplicate pattern detection on the customer service session to be detected based on the context session of each candidate customer service statement."

[0093] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: acquiring a set of conversational statements included in a customer service session to be detected, the set of conversational statements including customer statements and customer service statements from multiple rounds of conversation; calculating the similarity between every two customer service statements, and saving customer service statements whose similarity meets a set threshold to a duplicate scheme candidate set; for each candidate customer service statement in the duplicate scheme candidate set, acquiring the context session of the candidate customer service statement, the context session including customer statements and customer service statements; and performing duplicate scheme detection on the customer service session to be detected based on the context session of each candidate customer service statement.

[0094] According to the technical solution of the present invention, by obtaining a set of conversation statements included in the customer service session to be detected, the set of conversation statements includes customer statements and customer service statements from multiple rounds of conversation; calculating the similarity between each pair of customer service statements, and saving customer service statements whose similarity meets a set threshold to a duplicate scheme candidate set; for each candidate customer service statement in the duplicate scheme candidate set, obtaining the context session of the candidate customer service statement, the context session includes both customer statements and customer service statements; and performing duplicate scheme detection on the customer service session to be detected based on the context session of each candidate customer service statement, the technical solution realizes preliminary screening or preliminary recall of duplicate schemes based on the similarity between customer service statements, which can guarantee the recall rate of duplicate schemes; under the premise of ensuring the recall rate, by selecting only the context session of each candidate customer service statement in the duplicate scheme candidate set to further perform duplicate scheme detection on the customer service session to be detected, on the one hand, the conversation scenario of customer service statements is considered, making the detection results of duplicate schemes more accurate; on the other hand, there is no need to model the complete session, which greatly simplifies the model, improves the detection efficiency of the scheme, facilitates production deployment, and improves the quality of customer service.

[0095] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for detecting duplicate patterns in customer service conversations, characterized in that, include: Obtain the set of conversation statements included in the customer service conversation to be detected, wherein the set of conversation statements includes customer statements and customer service statements from multiple rounds of conversation; Calculate the similarity between each pair of customer service statements, and save the customer service statements whose similarity meets the set threshold to the duplicate solution candidate set. For each candidate customer service statement in the candidate set of repeated solutions, obtain the context session of the candidate customer service statement, the context session including the customer statement and the customer service statement; The detection of duplicate schemes for the customer service session to be detected is performed based on the context session of each candidate customer service statement, including: extracting features from the context sessions of the two candidate customer service statements to obtain feature vectors for the two candidate customer service statements. The feature vectors of the two candidate customer service statements are respectively input into a pre-trained first judgment unit to determine whether the two candidate customer service statements are scheme statements; the feature vectors of the two candidate customer service statements are concatenated and input into a pre-trained second judgment unit to determine whether the two candidate customer service statements are repeated; if the two candidate customer service statements are scheme statements and are repeated, then the two candidate customer service statements are repeated schemes; if there are not less than three candidate customer service statements in the repeated scheme candidate set that are repeated schemes, then the customer service session to be detected includes repeated schemes.

2. The method according to claim 1, characterized in that, Before calculating the similarity between each pair of customer service statements, the following steps are also included: Delete the specified customer service statement from the set of conversation statements. The specified customer service statement is a preset non-scheme customer service statement that conforms to business rules.

3. The method according to claim 1, characterized in that, The similarity between the two customer service statements is calculated based on the length of the longest common subsequence between the two customer service statements and the string length of the two customer service statements.

4. The method according to claim 1, characterized in that, Calculate the similarity between every two customer service statements, and save customer service statements whose similarity meets a set threshold to a duplicate solution candidate set, including: Calculate the similarity between each pair of customer service statements, and save the two customer service statements whose similarity meets the set threshold to the first candidate set; Multiple first candidate sets with intersection are combined to obtain a duplicate scheme candidate set, wherein the duplicate scheme candidate set contains no fewer than three candidate customer service statements.

5. The method according to claim 4, characterized in that, If all first candidate sets have no overlap, then the customer service session to be detected does not include duplicate schemes.

6. The method according to claim 1, characterized in that, Feature vectors for the two candidate customer service statements are obtained by extracting features from the context of each statement. Each sentence in the context of the two candidate customer service statements is encoded to obtain a set of sentence vectors; The set of sentence vectors is input into a natural language processing encoder for processing to obtain the feature vectors of the two candidate customer service statements.

7. The method according to claim 1, characterized in that, The context session includes the customer statement and customer service statement in the round session in which the candidate customer service statement is located, as well as the customer statement in the next round.

8. An apparatus for detecting repetitive patterns in customer service conversations, characterized in that, include: The conversation statement acquisition module is used to acquire the set of conversation statements included in the customer service conversation to be detected. The set of conversation statements includes customer statements and customer service statements from multiple rounds of conversation. The candidate statement determination module is used to calculate the similarity between each pair of customer service statements and save customer service statements whose similarity meets the set threshold to the duplicate solution candidate set. The context session acquisition module is used to acquire the context session of each candidate customer service statement in the duplicate scheme candidate set. The context session includes the customer statement and the customer service statement. The duplicate scheme detection module is used to perform duplicate scheme detection on the customer service session to be detected based on the context of each candidate customer service statement. The duplicate scheme detection module is further configured to: extract features from the context of the two candidate customer service statements to obtain feature vectors for the two candidate customer service statements; input the feature vectors of the two candidate customer service statements into a pre-trained first judgment unit to determine whether the two candidate customer service statements are scheme statements; concatenate the feature vectors of the two candidate customer service statements and input them into a pre-trained second judgment unit to determine whether the two candidate customer service statements are duplicates; if the two candidate customer service statements are scheme statements and are duplicates, then the two candidate customer service statements are duplicate schemes; if there are at least three candidate customer service statements in the duplicate scheme candidate set that are duplicate schemes, then the customer service session to be detected includes duplicate schemes.

9. An electronic device for detecting repetitive patterns in customer service conversations, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images