A routing method and apparatus for a session

Abstract

The present invention discloses a routing method and device for conversations, which relate to the field of artificial intelligence technology. A specific embodiment of the method includes: identifying the intent tags of each historical conversation, so as to route the historical conversation to the corresponding consultation unit; identifying conversation negative samples from the historical conversation according to the occurrence frequency of each intent tag in each consultation unit; constructing conversation positive samples through the conversation negative samples, and retraining the routing model based on the conversation negative samples and the conversation positive samples; deploying the retrained routing model online, and using the retrained routing model to route the current conversation to the target consultation unit. This embodiment can solve the technical problem of insufficient allocation accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and in particular, to a method and device for routing conversations. Background Art

[0002] An online customer service system is a general term for web-based instant messaging software. Compared with other instant messaging software (such as QQ, MSN, etc.), it enables seamless integration with websites and other channels (apps, WeChat, Facebook, etc.), providing a platform for websites and other applications to communicate with visitors; visitors can have conversations through the customer service SDK integrated by the enterprise without installing any software when accessing enterprise applications.

[0003] Currently, large customer service consultation centers need to face millions of users every day, and different users will consult various questions according to different intentions. From a technical perspective, how to accurately obtain users' questions, form categorized tags, and route them to the customer service consultation unit that can professionally answer the questions is an important core difficulty. The difficulties are mainly reflected in the following aspects:

[0004] 1. When customers call in, they do not accurately provide their consultation intentions. There is a lot of external information, which is complex and difficult to identify the true intentions of users;

[0005] 2. Since the true intentions of users cannot be accurately identified, the routing rules set manually often fail, and some users will be routed to the wrong consultation unit, and the customer service cannot answer their questions;

[0006] 3. For existing routing models, the training and update cycle is long, and it is easy to become invalid just after going online. Therefore, the problem of inaccurate routing cannot be solved in a timely manner.

[0007] Therefore, the existing conversation routing method has the technical problem of insufficient allocation accuracy. Summary of the Invention

[0008] In view of this, embodiments of the present invention provide a method and device for routing conversations to solve the technical problem of insufficient allocation accuracy.

[0009] To achieve the above object, according to one aspect of the embodiments of the present invention, a method for routing conversations is provided, including:

[0010] Identifying the intent tags of each historical conversation, thereby routing the historical conversation to the corresponding consultation unit;

[0011] Identifying conversation negative samples from the historical conversations according to the occurrence frequencies of each intent tag in each consultation unit;

[0012] Construct session positive samples from the session negative samples, and retrain the routing model based on the session negative samples and the session positive samples;

[0013] Deploy the retrained routing model online, and use the retrained routing model to route the current session to the target consultation unit.

[0014] Optionally, identifying the intent labels of each historical session, so as to route the historical session to the corresponding consultation unit, includes:

[0015] For each historical session, input multiple session short texts of the user and the customer service into the text summary model to output the session long text; wherein, the session long text is composed of several session short texts with the top weights, and the several session short texts are arranged in the order from early to late;

[0016] Input the session long text into the text classification model to output the probability of each intent label, and take the intent label with the highest probability as the intent label of the session long text;

[0017] According to the mapping relationship between the intent label and the consultation unit, route the historical session to the consultation unit corresponding to the intent label; wherein, each intent label corresponds to a consultation unit.

[0018] Optionally, identifying session negative samples from the historical sessions according to the occurrence frequency of each intent label in each consultation unit, includes:

[0019] For each consultation unit, calculate the occurrence frequency of each intent label in the consultation unit, and screen out at least one intent label with the lowest frequency or intent labels with a frequency lower than the frequency threshold, so as to identify historical sessions with suspected routing errors;

[0020] Identify session negative samples according to the matching results of the historical sessions with suspected routing errors in other consultation units.

[0021] Optionally, identifying session negative samples according to the matching results of the historical sessions with suspected routing errors in other consultation units, includes:

[0022] Judge whether the intent label of the historical session with suspected routing error can be matched in other consultation units;

[0023] If so, identify the session with suspected routing error as a session with confirmed routing error, and store the session with confirmed routing error, its intent label, and the consultation unit in the sample pool as session negative samples;

[0024] If not, identify the session with suspected routing error as a session with correct routing.

[0025] Optionally, there is at least one same attribute between the consultation unit and the other consultation units; wherein, the at least one same attribute includes category and / or user attribute.

[0026] Optionally, before identifying the intent label of each historical session and thus routing the historical session to the corresponding consultation unit, it further includes:

[0027] Using the short text of the conversation between the user and the customer as input and the weight of the short text of the conversation as output to train a long short-term memory neural network model, thereby obtaining a text summary model.

[0028] Optionally, before identifying the intent label of each historical session and thus routing the historical session to the corresponding consultation unit, it further includes:

[0029] Using the long text of the conversation as input and the intent label of the long text of the conversation as output to train a distributed gradient boosting model, thereby obtaining a text classification model.

[0030] Optionally, the session positive samples include: the session with the confirmed routing error and its intent label, and the other consultation units.

[0031] Optionally, retraining the routing model based on the session negative samples and the session positive samples includes:

[0032] For each sample, using the intent label of the first sentence of the user in the sample, the attribute features of the user, the attribute features of the consultation unit, and the consultation unit identifier as input, and the sample result as output to retrain the routing model;

[0033] Wherein, the sample result includes: positive or negative.

[0034] Optionally, using the retrained routing model to route the current session to the target consultation unit includes:

[0035] Inputting the intent label of the first sentence of the user in the current session, the attribute features of the user, the attribute features of each consultation unit, and the identifiers of each consultation unit into the retrained routing model to output the probability of correct allocation corresponding to each consultation unit;

[0036] Routing the current session to the target consultation unit with the highest probability of correct allocation.

[0037] In addition, according to another aspect of the embodiments of the present invention, there is provided a routing device for a session, including:

[0038] An identification module, configured to identify the intent label of each historical session, so as to route the historical session to the corresponding consultation unit;

[0039] A mining module, configured to identify session negative samples from the historical sessions according to the occurrence frequency of each intent label in each consultation unit;

[0040] A training module, configured to construct session positive samples through the session negative samples, and retrain the routing model based on the session negative samples and the session positive samples;

[0041] A routing module, configured to deploy the retrained routing model online, and use the retrained routing model to route the current session to the target consultation unit.

[0042] Optionally, the identification module is further configured to:

[0043] For each historical session, input multiple session short texts of the user and the customer service into a text summary model to output a session long text; wherein, the session long text is composed of several session short texts with the top weights, and the several session short texts are arranged in the order from the earliest to the latest;

[0044] Input the session long text into a text classification model to output the probability of each intent label, and use the intent label with the highest probability as the intent label of the session long text;

[0045] According to the mapping relationship between the intent label and the consultation unit, route the historical session to the consultation unit corresponding to the intent label; wherein, each intent label corresponds to a consultation unit.

[0046] Optionally, the mining module is further configured to:

[0047] For each consultation unit, calculate the occurrence frequency of each intent label in the consultation unit, and filter out at least one intent label with the lowest frequency or intent labels with a frequency lower than the frequency threshold, so as to identify historical sessions with suspected routing errors;

[0048] Identify session negative samples according to the matching results of the historical sessions with suspected routing errors in other consultation units.

[0049] Optionally, the mining module is further configured to:

[0050] Determine whether the intent label of the historical session with suspected routing error can be matched in other consultation units;

[0051] If so, identify the session with suspected routing error as a session with confirmed routing error, and store the session with confirmed routing error, its intent label, and the consultation unit in the sample pool as a negative session sample;

[0052] If not, identify the session with suspected routing error as a session with correct routing.

[0053] Optionally, there is at least one same attribute between the consultation unit and the other consultation units; wherein, the at least one same attribute includes category and / or user attribute.

[0054] Optionally, the training module is further configured to:

[0055] Before identifying the intent label of each historical session and routing the historical session to the corresponding consultation unit, use the short text of the conversation between the user and the customer as the input and the weight of the short text of the conversation as the output to train a long short-term memory neural network model, so as to obtain a text summary model.

[0056] Optionally, the training module is further configured to:

[0057] Before identifying the intent label of each historical session and routing the historical session to the corresponding consultation unit, use the long text of the conversation as the input and the intent label of the long text of the conversation as the output to train a distributed gradient boosting model, so as to obtain a text classification model.

[0058] Optionally, the positive session samples include: the session with confirmed routing error, its intent label, and the other consultation units.

[0059] Optionally, the training module is further configured to:

[0060] For each sample, use the intent label of the user's first sentence, the user's attribute features, the attribute features of the consultation unit, and the consultation unit identifier in the sample as the input, and the sample result as the output to retrain the routing model;

[0061] wherein, the sample result includes: positive or negative.

[0062] Optionally, the routing module is further configured to:

[0063] Input the intent label of the user's first sentence, the user's attribute features, the attribute features of each consultation unit, and the identifiers of each consultation unit in the current session into the retrained routing model to output the correct allocation probability corresponding to each consultation unit;

[0064] Route the current session to the target consultation unit with the highest correct allocation probability.

[0065] According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including:

[0066] One or more processors;

[0067] A storage device for storing one or more programs,

[0068] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the above embodiments.

[0069] According to another aspect of the embodiments of the present invention, there is also provided a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, the method described in any of the above embodiments is implemented.

[0070] One embodiment of the above invention has the following advantages or beneficial effects: Since the session negative samples are identified from the historical sessions according to the occurrence frequency of each intention label in each consultation unit, and the session positive samples are constructed through the session negative samples, and then the routing model is retrained based on the session negative samples and session positive samples, the technical problem of insufficient high allocation accuracy in the prior art is overcome. The embodiments of the present invention identify session negative samples from historical sessions based on natural language understanding technology and neural network models, and retrain the online routing model based on the negative samples to improve the allocation accuracy of the routing model. Therefore, it is possible to change the allocation method following the business development without the intervention of business and R & D, effectively suppressing the accuracy drift problem of the routing model, and helping to improve the experience of user consultation and business personnel.

[0071] The further effects of the above non-conventional alternative manners will be described in conjunction with the specific embodiments below. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The drawings are used to better understand the present invention and do not constitute an improper limitation to the present invention. Among them:

[0073] Figure 1 is a schematic diagram of the main process of the routing method of the session according to the embodiments of the present invention;

[0074] Figure 2 is a schematic diagram of the main process of the routing method of the session according to a reference embodiment of the present invention;

[0075] Figure 3 is a schematic diagram of the main process of the routing method of the session according to another reference embodiment of the present invention;

[0076] Figure 4 is a schematic diagram of the main modules of the routing device of the session according to the embodiments of the present invention;

[0077] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention can be applied;

[0078] Figure 6 is a schematic structural diagram of a computer system of a terminal device or a server suitable for implementing embodiments of the present invention. Detailed implementation manners

[0079] The following makes an illustration of exemplary embodiments of the present invention with reference to the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following.

[0080] A customer service routing and allocation system refers to that when a user enters the consultation system through a consultation entrance, the system background routes the user to a suitable customer service consultation unit through various parameter information. The customer service of this consultation unit is usually dedicated to answering a certain type of questions, such as after-sales questions, pre-sales questions, or more complex after-sales questions of home appliance products, return questions of clothing products, etc. Especially in the e-commerce industry, thousands of consultation units can be formed by different categories and question types. Therefore, how to make the routing more accurate becomes an important technical problem.

[0081] Currently, there are mainly two routing methods:

[0082] The first one is the routing rule type: It requires managers to configure a large number of routing rules in advance, such as:

Those who meet the conditions of a premium member and enter from the home appliance product details page go to consultation unit A

Those who meet the conditions of being easy to complain and enter from the after-sales service page go to consultation unit B

[0083] Second, the intelligent routing model type: This type of model-based routing breaks the shackles of rule conditions to a certain extent. Through a pre-set intention exploration robot, it has a short conversation with the user to guide the user to state their intention, then uses a text classification model to identify the intention, and based on the intention and other external parameters, directly selects the consulting unit to go to by the model. This type of model is a black box model, and the decision-making basis for routing is not exposed externally. The training of the model requires a large amount of manual annotation, that is, using actual samples to select the corresponding consulting unit, and then training a model through a multi-class neural network. Therefore, the costs for annotation, update, and verification are extremely high. Moreover, due to the model being a black box, the operation staff has no idea whether the classification result is correct and can only rely on the manual feedback of the customer service, resulting in the model update taking up to several months and requiring a large number of engineers for maintenance, with high labor costs.

[0084] To solve the technical problems existing in the prior art, an embodiment of the present invention proposes a routing method for conversations. This method identifies conversation negative samples from historical conversations based on natural language understanding technology and a neural network model, realizes automated and intelligent monitoring of routing results, and then retrains the routing model using the conversation negative samples, thereby improving the accuracy of routing allocation.

[0085] Figure 1 It is a schematic diagram of the main process of the routing method for conversations according to an embodiment of the present invention. As an embodiment of the present invention, as Figure 1 shown, the routing method for conversations may include:

[0086] Step 101, identify the intention label of each historical conversation, so as to route the historical conversation to the corresponding consulting unit.

[0087] First, obtain historical conversations from the database. Each historical conversation includes the text record of the conversation between the user and the customer service after the user is assigned a consulting unit, and each historical conversation is presented in the form of multiple short texts (i.e., conversation flows); then, based on natural language understanding technology and a neural network model, respectively identify the intention label of each historical conversation, so as to route the historical conversation to the corresponding consulting unit according to the intention label of the historical conversation. It should be noted that if the conversation between the user and the customer service is in voice mode, the voice conversation content is converted into a text record to facilitate intention recognition of the conversation content.

[0088] Optionally, step 101 may include: for each historical session, inputting multiple session short texts between the user and the customer service into a text summary model to output a session long text; wherein, the session long text is concatenated in order of decreasing weight of the session short texts; inputting the session long text into a text classification model to output the probability of each intent label, and taking the intent label with the highest probability as the intent label of the session long text; routing the historical session to the consultation unit corresponding to the intent label according to the mapping relationship between the intent label and the consultation unit; wherein, each intent label corresponds to a consultation unit.

[0089] In an embodiment of the present invention, the text summary model is used to abstract the session short texts between the user and the customer service into a session long text. Specifically, for each historical session, multiple session short texts between the user and the customer service in this session are input into the text summary model together. The text summary model performs processing such as invalid sentence deletion, keyword weighting, and sentence recombination on these session short texts, and finally outputs a session long text; wherein, the session long text is composed of several session short texts with higher weights, and the several session short texts are arranged in the order from earliest to latest in time.

[0090] Optionally, before step 101, it further includes: using the session short texts between the user and the customer as input and the weights of the session short texts as output to train a long short-term memory neural network model (LSTM), thereby obtaining a text summary model. In order to accurately abstract the session long text, it is necessary to pre-train the text summary model. Specifically, the weights of a batch of session short texts are labeled, that is, the importance of each session short text is graded, and invalid small talk sentences are removed. For example: greetings and other sentences have a weight of 0, and valid sentences have a higher weight. After training, the input of the text summary model is multiple independent session short texts, and the output is a long text composed of the first x session short texts with the highest importance in the order from earliest to latest in time.

[0091] In an embodiment of the present invention, the text classification model is used to identify the intent label of the session long text, that is, to identify the intent label of the historical session. Specifically, the abstracted session long text is input into the text classification model. The text classification model outputs the probability of each intent label, and takes the intent label with the highest probability as the intent label of the session long text. Since each intent label corresponds to a consultation unit, the session long text can be assigned to a certain consultation unit according to the mapping relationship between the intent label and the consultation unit, that is, the historical session is routed to this consultation unit.

[0092] It should be noted that, in an embodiment of the present invention, one intent label corresponds to one consultation unit, but one consultation unit can be mapped to multiple intent labels, that is, multiple intent labels can exist in each consultation unit.

[0093] Optionally, before step 101, it further includes: using the long conversation text as the input and the intent label of the long conversation text as the output to train a distributed gradient boosting model (xgboost), so as to obtain a text classification model. In order to accurately identify the intent of the text, it is necessary to pre-train a text classification model. Specifically, a batch of texts are annotated with intents. The intent types can be arbitrarily expanded according to business development, with at least 2 types and up to thousands of types. After training, the input of the model is a single long text, and the output is the hit probability of each intent label. The intent label with the highest probability is taken as the intent label of the long conversation text.

[0094] Step 102, identify session negative samples from the historical conversations according to the occurrence frequency of each intent label in each consultation unit.

[0095] After routing each historical conversation to the corresponding consultation unit, multiple intent labels may appear in each consultation unit. Therefore, by counting the occurrence frequency of each intent label in each consultation unit, based on at least one intent label with the lowest frequency or intent labels with a frequency lower than the frequency threshold, session negative samples can be screened out.

[0096] Optionally, step 102 may include: for each consultation unit, calculate the occurrence frequency of each intent label in the consultation unit, screen out at least one intent label with the lowest frequency or intent labels with a frequency lower than the frequency threshold, so as to identify historical conversations with suspected routing errors; according to the matching results of the historical conversations with suspected routing errors in other consultation units, identify session negative samples. In the embodiments of the present invention, since each historical conversation will be routed to the corresponding consultation unit, N historical conversations will appear in each consultation unit, that is, there are a total of N intent labels. The occurrence frequencies of these N intent labels are counted and sorted, and the lowest X intent labels or intent labels with a frequency lower than the frequency threshold are screened out. The historical conversations corresponding to these intent labels are identified as historical conversations with suspected routing errors; then, according to the matching results of these historical conversations with suspected routing errors in other consultation units, session negative samples are identified from these historical conversations with suspected routing errors.

[0097] Optionally, according to the matching results of the historical sessions with suspected routing errors in other consultation units, session negative samples are identified, including: determining whether the intent tags of the historical sessions with suspected routing errors can be matched in other consultation units; if so, identifying the sessions with suspected routing errors as sessions with confirmed routing errors, and storing the sessions with confirmed routing errors, their intent tags, and the consultation units in the sample pool as session negative samples; if not, identifying the sessions with suspected routing errors as sessions with correct routing. After discovering the historical sessions with suspected routing errors, the historical sessions with suspected routing errors are matched in other consultation units. If the historical sessions with suspected routing errors also appear in other consultation units, it indicates that there is a more suitable consultation unit for the historical sessions with suspected routing errors, and then the sessions with suspected routing errors are identified as sessions with confirmed routing errors; if the historical sessions with suspected routing errors do not appear in other consultation units, the sessions with suspected routing errors are temporarily identified as sessions with correct routing. Finally, the sessions with confirmed routing errors are stored in the sample pool as session negative samples, waiting to retrain the routing model. Among them, the session negative samples at least include the sessions with confirmed routing errors, their intent tags, and the current consultation unit.

[0098] It should be noted that if the historical sessions with suspected routing errors appear frequently in other consultation units, it indicates that the probability of the historical sessions with suspected routing errors being session negative samples is relatively high; if the historical sessions with suspected routing errors appear less frequently in other consultation units, it indicates that the probability of the historical sessions with suspected routing errors being session negative samples is relatively low. If the historical sessions with suspected routing errors are matched in multiple other consultation units, the consultation unit with the highest appearance frequency of the historical sessions with suspected routing errors is used as the matched consultation unit, which helps to improve the accuracy of mining session negative samples.

[0099] Optionally, there is at least one same attribute between the consultation unit and the other consultation units; where the at least one same attribute includes category and / or user attribute. To quickly match other consultation units, the traversal objects can be reduced, and only the other consultation units with partially the same attributes as the current consultation unit need to be matched, such as other consultation units belonging to the same category and / or the same user attribute as the current consultation unit, so that a smaller number of other consultation units can be selected from thousands of consultation units.

[0100] Step 103, construct session positive samples through the session negative samples, and retrain the routing model based on the session negative samples and the session positive samples.

[0101] Session negative samples are mined from historical sessions through step 101 and step 102 to determine whether the allocation result of the routing model deployed online has drifted. Optionally, the proportion of session negative samples can be judged. If it is confirmed that the sessions with routing errors account for a certain proportion of all historical sessions (such as higher than 5%, 8% or 10%, etc.), the update process of the routing model is triggered. Generally speaking, as the business develops, the accuracy of the routing model will gradually drift. By automatically starting the model update by confirming the proportion of sessions with routing errors, the accuracy drift can be effectively suppressed.

[0102] Before retraining the routing model, it is necessary to construct session positive samples according to the session negative samples first. It can be seen from step 102 that the sessions with confirmed routing errors appear in both the current consultation unit and other consultation units. Therefore, other consultation units can be used as session positive samples, that is, the session positive samples include: the sessions with confirmed routing errors and their intent labels, and the other consultation units (more appropriate consultation units for the historical sessions suspected of routing errors).

[0103] Optionally, retraining the routing model based on the session negative samples and the session positive samples includes: for each sample, taking the intent label of the user's first sentence, the user's attribute features, the attribute features of the consultation unit, and the consultation unit identifier in the sample as inputs, and taking the sample result as the output to retrain the routing model; where the sample result includes: positive or negative. In the embodiments of the present invention, according to the different types of user buried point data, the features required for training the routing model can be stored in different storage spaces respectively.

[0104] Among them, the features required for training the routing model are mainly divided into:

[0105] 1) The intent label of the user's first sentence, that is, the intent label of the first sentence the user said when making an inbound consultation. The user intent recognition model can be obtained by training a long short-term memory neural network model, and the intent label of the first sentence the user said when making an inbound consultation is recognized through the user intent recognition model.

[0106] 2) The user's attribute features mainly include the user attribute snapshots collected by the client and the background when the current session occurs, such as the product category consulted in this session, the user's risk value at this time, the status of the user's consultation order at this time, etc.

[0107] 3) The attribute features of the consultation unit are mainly the most common unit-level intent label in the consultation unit and other inherent attributes of the consultation unit (such as the product category undertaken, etc.), which are used to adjust the allocation method of the consultation unit in the next round of routing model training.

[0108] The feature collection and logging background is responsible for receiving business request data from various clients (such as real-time chat data, consultation entry points, consultation orders, etc.), as well as user attribute data and attribute features of consultation units from the business background.

[0109] The routing model is used to route the consultation session to a more appropriate consultation unit. In an embodiment of the present invention, the intent label of the user's first sentence in the sample, the user's attribute features, the attribute features of the consultation unit, and the consultation unit identifier are used as inputs, and the sample result is used as the output to retrain the routing model; the output result of the session positive sample is 1, and the output result of the session negative sample is 0. Optionally, the routing model can be a distributed gradient boosting model, and the routing model obtained by training with the distributed gradient boosting model can accurately route the session to the appropriate consultation unit.

[0110] Step 104, deploy the retrained routing model online, and use the retrained routing model to route the current session to the target consultation unit.

[0111] After retraining the routing model, deploy the new routing model online. Optionally, before online deployment, verify its assignment accuracy on the test set. If the accuracy is met, it will be automatically launched to update the routing model online.

[0112] Optionally, using the retrained routing model to route the current session to the target consultation unit includes: inputting the intent label of the user's first sentence in the current session, the user's attribute features, the attribute features of each consultation unit, and the identifiers of each consultation unit into the retrained routing model to output the probability of correct assignment corresponding to each consultation unit; routing the current session to the target consultation unit with the highest probability of correct assignment. When a user makes an incoming consultation, first, the intent label of the user's first sentence is identified through the user intent recognition model, and then the intent label of the user's first sentence in the current session, the user's attribute features, the attribute features of each consultation unit, and the identifiers of each consultation unit are input into the retrained routing model. The routing model outputs the probability of correct assignment corresponding to each consultation unit, and the consultation unit with the highest probability is the most appropriate consultation unit for the current session, and the current session is routed to this consultation unit.

[0113] According to the various embodiments described above, it can be seen that the embodiments of the present invention identify session negative samples from historical conversations according to the occurrence frequency of each intention tag in each consultation unit, construct session positive samples through the session negative samples, and thus solve the technical problem of insufficient allocation accuracy in the prior art by retraining the routing model based on the session negative samples and session positive samples. The embodiments of the present invention identify session negative samples from historical conversations based on natural language understanding technology and neural network models, and retrain the online routing model based on the negative samples to improve the allocation accuracy of the routing model. Therefore, it is possible to change the allocation method following the business development without the intervention of business and R & D, effectively suppress the accuracy drift problem of the routing model, and help improve the experience of user consultation and business personnel.

[0114] Figure 2 It is a schematic diagram of the main process of the session routing method according to a reference embodiment of the present invention. As another embodiment of the present invention, as Figure 2 shown, the session routing method may include:

[0115] Step 201: Use the short text of the conversation between the user and the customer as the input and the weight of the short text of the conversation as the output to train a long short-term memory neural network model, thereby obtaining a text summary model.

[0116] Specifically, weight is marked for a batch of short texts of conversations, that is, the importance of each short text of the conversation is graded, such as ineffective small talk sentences. For example, the weight of greetings and the like is 0, and the weight of effective sentences is higher. Use the short text of the conversation between the user and the customer as the input and the weight of the short text of the conversation as the output to train a long short-term memory neural network model, thereby obtaining a text summary model

[0117] Step 202: Use the long text of the conversation as the input and the intention tag of the long text of the conversation as the output to train a distributed gradient boosting model, thereby obtaining a text classification model.

[0118] Intentions are marked for a batch of texts. The intention types can be expanded arbitrarily according to business development, at least 2, and up to thousands. Use the long text of the conversation as the input and the intention tag of the long text of the conversation as the output to train a distributed gradient boosting model, thereby obtaining a text classification model.

[0119] Step 203: Input multiple short texts of the conversation between the user and the customer service into the text summary model to output the long text of the conversation.

[0120] After training, the input of the text summary model is multiple independent short texts of the conversation, and the output is a long text composed of the top x short texts of the conversation with the highest importance in the order from the earliest to the latest in time.

[0121] Step 204: Input the session long text into the text classification model to output the probability of each intent label, and take the intent label with the highest probability as the intent label of the session long text.

[0122] Input the abstracted session long text into the text classification model. The text classification model outputs the probability of each intent label, and takes the intent label with the highest probability as the intent label of the session long text.

[0123] Step 205: According to the mapping relationship between the intent label and the consultation unit, route the historical session to the consultation unit corresponding to the intent label.

[0124] One intent label corresponds to one consultation unit, but one consultation unit can be mapped with multiple intent labels, that is, multiple intent labels can exist in each consultation unit.

[0125] Step 206: Identify session negative samples from the historical session according to the occurrence frequency of each intent label in each consultation unit.

[0126] After routing each historical session to the corresponding consultation unit respectively, multiple intent labels may appear in each consultation unit. Therefore, by counting the occurrence frequency of each intent label in each consultation unit, session negative samples can be screened out based on at least one intent label with the lowest frequency or intent labels with frequencies lower than the frequency threshold.

[0127] Step 207: Construct session positive samples through the session negative samples.

[0128] Step 208: Retrain the routing model based on the session negative samples and the session positive samples.

[0129] Specifically, for each sample, take the intent label of the user's first sentence, the user's attribute features, the attribute features of the consultation unit, and the consultation unit identifier in the sample as the input, and take the sample result as the output to retrain the routing model; where the sample result includes: positive or negative.

[0130] Step 209: Deploy the retrained routing model online.

[0131] Before online deployment, verify its assignment accuracy on the test set. If the accuracy is met, it will be automatically launched online to update the routing model online.

[0132] Step 210: Use the retrained routing model to route the current session to the target consultation unit.

[0133] Input the intent label of the user's first sentence in the current session, the user's attribute characteristics, the attribute characteristics of each consultation unit, and each consultation unit identifier into the retrained routing model to output the correct allocation probability corresponding to each consultation unit; route the current session to the target consultation unit with the highest correct allocation probability. When a user calls in for consultation, first identify the intent label of the user's first sentence through the user intent recognition model, and then input the intent label of the user's first sentence in the current session, the attribute characteristics of this user, the attribute characteristics of each consultation unit, and each consultation unit identifier into the retrained routing model. The routing model outputs the correct allocation probability corresponding to each consultation unit, and the consultation unit with the highest probability is the most suitable consultation unit for the current session. Route the current session to this consultation unit.

[0134] In addition, the specific implementation content of the session routing method in a referenceable embodiment of the present invention has been described in detail in the above-mentioned session routing method, so the repeated content will not be described here again.

[0135] Figure 3 It is a schematic diagram of the main process of the session routing method according to another referenceable embodiment of the present invention. As another embodiment of the present invention, as Figure 3 shown, step 102 may specifically include:

[0136] Step 302, for each consultation unit, calculate the occurrence frequency of each intent label in the consultation unit, and screen out at least one intent label with the lowest frequency or intent labels with a frequency lower than the frequency threshold, so as to identify historical sessions with suspected routing errors.

[0137] Since each historical session will be routed to the corresponding consultation unit, there will be N historical sessions in each consultation unit, that is, there are a total of N intent labels. Statistically sort the occurrence frequencies of these N intent labels, screen out X intent labels with the lowest frequency or intent labels with a frequency lower than the frequency threshold, and identify the historical sessions corresponding to these intent labels as historical sessions with suspected routing errors.

[0138] Step 302, determine whether the intent label of the historical session with suspected routing error can be matched in other consultation units; if so, execute step 303; if not, execute step 304.

[0139] Step 303, identify the session with suspected routing error as a session with confirmed routing error, and store the session with confirmed routing error, its intent label, and the consultation unit in the sample pool as a session negative sample.

[0140] After a historical session with a suspected routing error is found, the historical session with the suspected routing error is matched in other consultation units. If the historical session with the suspected routing error also appears in other consultation units, it indicates that there is a more suitable consultation unit for the historical session with the suspected routing error. Then, the session with the suspected routing error is identified as a session with a confirmed routing error.

[0141] Then, the session with the confirmed routing error is stored in the sample pool as a session negative sample, waiting to retrain the routing model. Among them, the session negative sample at least includes the session with the confirmed routing error, its intent label, and the current consultation unit.

[0142] Step 304, identify the session with the suspected routing error as a session with a correct routing.

[0143] If the historical session with the suspected routing error does not appear in other consultation units, the session with the suspected routing error is temporarily identified as a session with a correct routing.

[0144] If the historical session with the suspected routing error appears frequently in other consultation units, it indicates that the probability of the historical session with the suspected routing error being a session negative sample is relatively high. If the historical session with the suspected routing error appears less frequently in other consultation units, it indicates that the probability of the historical session with the suspected routing error being a session negative sample is relatively low. If the historical session with the suspected routing error is matched in multiple other consultation units, the consultation unit with the highest occurrence frequency of the historical session with the suspected routing error is used as the matched consultation unit, which helps to improve the accuracy of mining session negative samples.

[0145] In addition, the specific implementation content of the session routing method in another referenceable embodiment of the present invention has been described in detail in the above session routing method, so the repeated content will not be described here.

[0146] Figure 4 It is a schematic diagram of the main modules of a session routing device according to an embodiment of the present invention, as Figure 4 shown. The session routing device 400 includes an identification module 401, a mining module 402, a training module 403, and a routing module 404. Among them, the identification module 401 is used to identify the intent label of each historical session, so as to route the historical session to the corresponding consultation unit. The mining module 402 is used to identify session negative samples from the historical sessions according to the occurrence frequency of each intent label in each consultation unit. The training module 403 is used to construct session positive samples through the session negative samples and retrain the routing model based on the session negative samples and the session positive samples. The routing module 404 is used to deploy the retrained routing model online and use the retrained routing model to route the current session to the target consultation unit.

[0147] Optionally, the recognition module 401 is further configured to:

[0148] For each historical session, input multiple session short texts of the user and the customer service into a text summary model to output a session long text; wherein, the session long text is composed of several session short texts with the top weights, and the several session short texts are arranged in ascending order of time;

[0149] Input the session long text into a text classification model to output the probability of each intent label, and use the intent label with the highest probability as the intent label of the session long text;

[0150] According to the mapping relationship between the intent label and the consultation unit, route the historical session to the consultation unit corresponding to the intent label; wherein, each intent label corresponds to a consultation unit.

[0151] Optionally, the mining module 402 is further configured to:

[0152] For each consultation unit, calculate the occurrence frequency of each intent label in the consultation unit, and filter out at least one intent label with the lowest frequency or intent labels with a frequency lower than the frequency threshold, so as to identify historical sessions with suspected routing errors;

[0153] According to the matching results of the historical sessions with suspected routing errors in other consultation units, identify session negative samples.

[0154] Optionally, the mining module 402 is further configured to:

[0155] Determine whether the intent label of the historical session with suspected routing error can be matched in other consultation units;

[0156] If so, identify the session with suspected routing error as a session with confirmed routing error, and store the session with confirmed routing error, its intent label, and the consultation unit into the sample pool as a session negative sample;

[0157] If not, identify the session with suspected routing error as a session with correct routing.

[0158] Optionally, there is at least one same attribute between the consultation unit and the other consultation units; wherein, the at least one same attribute includes category and / or user attribute.

[0159] Optionally, the training module 403 is further configured to:

[0160] Before identifying the intent label of each historical session to route the historical session to the corresponding consultation unit, use the short text of the conversation between the user and the customer as the input and the weight of the short text of the conversation as the output to train a long short-term memory neural network model, so as to obtain a text summary model.

[0161] Optionally, the training module 403 is further configured to:

[0162] Before identifying the intent label of each historical session to route the historical session to the corresponding consultation unit, use the long text of the conversation as the input and the intent label of the long text of the conversation as the output to train a distributed gradient boosting model, so as to obtain a text classification model.

[0163] Optionally, the positive conversation samples include: the conversation with the confirmed routing error and its intent label, and the other consultation units.

[0164] Optionally, the training module 403 is further configured to:

[0165] For each sample, use the intent label of the first sentence of the user in the sample, the attribute features of the user, the attribute features of the consultation unit, and the consultation unit identifier as the input, and the sample result as the output to retrain the routing model;

[0166] Wherein, the sample result includes: positive or negative.

[0167] Optionally, the routing module 404 is further configured to:

[0168] Input the intent label of the first sentence of the user in the current conversation, the attribute features of the user, the attribute features of each consultation unit, and the identifiers of each consultation unit into the retrained routing model to output the probability of correct allocation corresponding to each consultation unit;

[0169] Route the current conversation to the target consultation unit with the highest probability of correct allocation.

[0170] It should be noted that the specific implementation content of the conversation routing device in the present invention has been described in detail in the above-mentioned conversation routing method, so the repeated content will not be described here.

[0171] Figure 5 An exemplary system architecture 500 to which the conversation routing method or the conversation routing device of the embodiments of the present invention can be applied is shown.

[0172] As Figure 5As shown, the system architecture 500 may include terminal devices 501, 502, 503, a network 504, and a server 505. The network 504 is used to provide a medium for communication links between the terminal devices 501, 502, 503 and the server 505. The network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0173] Users can use the terminal devices 501, 502, 503 to interact with the server 505 through the network 504 to receive or send messages, etc. Various communication client applications may be installed on the terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (for example only).

[0174] The terminal devices 501, 502, 503 may be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, and desktop computers, etc.

[0175] The server 505 may be a server that provides various services, such as a background management server that supports shopping websites browsed by users using the terminal devices 501, 502, 503 (for example only). The background management server may analyze and process data such as item information query requests received, and feedback the processing results to the terminal devices.

[0176] It should be noted that the session routing method provided by the embodiments of the present invention is generally executed by the server 505. Correspondingly, the session routing device is generally set in the server 505. The session routing method provided by the embodiments of the present invention may also be executed by the terminal devices 501, 502, 503. Correspondingly, the session routing device may be set in the terminal devices 501, 502, 503.

[0177] It should be understood that Figure 5 the numbers of terminal devices, networks, and servers in

[0178] are merely illustrative. According to the implementation requirements, there may be any number of terminal devices, networks, and servers. Figure 6 The following refers to Figure 6 which shows a schematic structural diagram of a computer system 600 of a terminal device suitable for implementing the embodiments of the present invention.

[0179] As Figure 6As shown, computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage section 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0180] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 610 as needed so that a computer program read from it can be installed into the storage section 608 as needed.

[0181] Specifically, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowchart can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program that includes a computer program carried on a computer-readable medium, and the computer program contains program codes for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above functions defined in the system of the present invention are executed.

[0182] It should be noted that the computer-readable medium shown in the present invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of a computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, in which the computer-readable program code is carried. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, and this computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0183] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer programs according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram can represent a module, a program segment, or a part of code, and the above-mentioned module, program segment, or part of code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0184] The modules involved in the embodiments of the present invention can be implemented in software or in hardware. The described modules can also be provided in a processor. For example, it can be described as: a processor includes an identification module, a mining module, a training module, and a routing module. In some cases, the names of these modules do not limit the modules themselves.

[0185] As another aspect, the present invention also provides a computer-readable medium. The computer-readable medium can be included in the device described in the above embodiments; or it can exist alone and not be assembled into the device. The above computer-readable medium carries one or more programs. When the one or more programs are executed by a device, the device implements the following method: identifying the intent tags of each historical session, so as to route the historical session to the corresponding consultation unit; identifying session negative samples from the historical session according to the occurrence frequency of each intent tag in each consultation unit; constructing session positive samples through the session negative samples, and retraining the routing model based on the session negative samples and the session positive samples; deploying the retrained routing model online, and using the retrained routing model to route the current session to the target consultation unit.

[0186] Since the technical means of identifying session negative samples from historical sessions according to the occurrence frequency of each intent tag in each consultation unit, constructing session positive samples through the session negative samples, and then retraining the routing model based on the session negative samples and the session positive samples are adopted, the technical problem of insufficient high allocation accuracy in the prior art is overcome. The embodiments of the present invention identify session negative samples from historical sessions based on natural language understanding technology and neural network models, and retrain the routing model online based on the negative samples to improve the allocation accuracy of the routing model. Therefore, it can change the allocation method following the business development without the intervention of business and R & D, effectively inhibit the accuracy drift problem of the routing model, and help improve the experience of user consultation and business personnel.

[0187] The above specific embodiments do not limit the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A routing method for a session, characterized in that, Including: Identifying the intent tags of each historical session, so as to route the historical session to the corresponding consultation unit; Identifying session negative samples from the historical sessions according to the occurrence frequency of each intent tag in each consultation unit; Constructing session positive samples through the session negative samples, and retraining the routing model based on the session negative samples and the session positive samples; Deploying the retrained routing model online, and using the retrained routing model to route the current session to the target consultation unit; Identifying session negative samples from the historical sessions according to the occurrence frequency of each intent tag in each consultation unit, including: For each consultation unit, calculating the occurrence frequency of each intent tag in the consultation unit, and screening out at least one intent tag with the lowest frequency or intent tags with a frequency lower than the frequency threshold, so as to identify historical sessions with suspected routing errors; Identifying session negative samples according to the matching results of the historical sessions with suspected routing errors in other consultation units.

2. The method according to claim 1, wherein Identifying the intent tags of each historical session, so as to route the historical session to the corresponding consultation unit, including: For each historical session, inputting multiple session short texts of the user and the customer service into the text summary model to output a session long text; wherein, the session long text consists of several session short texts with higher weights, and the several session short texts are arranged in the order from earliest to latest in time; Inputting the session long text into the text classification model to output the probability of each intent tag, and taking the intent tag with the highest probability as the intent tag of the session long text; According to the mapping relationship between the intent tag and the consultation unit, routing the historical session to the consultation unit corresponding to the intent tag; wherein, each intent tag corresponds to a consultation unit.

3. The method according to claim 1, wherein Identifying session negative samples according to the matching results of the historical sessions with suspected routing errors in other consultation units, including: Judging whether the intent tag of the historical session with suspected routing error can be matched in other consultation units; If so, identifying the session with suspected routing error as a session with confirmed routing error, and storing the session with confirmed routing error, its intent tag, and the consultation unit into the sample pool as session negative samples; If not, identifying the session with suspected routing error as a session with correct routing.

4. The method according to claim 3, wherein There is at least one same attribute between the consultation unit and the other consultation units; wherein, the at least one same attribute includes category and / or user attribute.

5. The method according to claim 2, characterized in that Before identifying the intent tags of each historical session so as to route the historical session to the corresponding consultation unit, it further includes: Taking the session short texts of the user and the customer as input and the weights of the session short texts as output, training a long short-term memory neural network model to obtain a text summary model.

6. The method according to claim 2, characterized in that, Before identifying the intent tags of each historical session so as to route the historical session to the corresponding consultation unit, it further includes: Taking the session long text as input and the intent tag of the session long text as output, training a distributed gradient boosting model to obtain a text classification model.

7. The method according to claim 3, wherein The session positive samples include: the sessions that confirm routing errors, their intent labels, and the other consulting units.

8. The method according to claim 1, wherein Re-training the routing model based on the session negative samples and the session positive samples includes: For each sample, taking the intent label of the user's first sentence in the sample, the user's attribute features, the attribute features of the consulting unit, and the consulting unit identifier as inputs, and taking the sample result as the output to re-train the routing model; Wherein, the sample result includes: positive or negative.

9. The method according to claim 8, wherein, Using the re-trained routing model to route the current session to the target consulting unit includes: Inputting the intent label of the user's first sentence in the current session, the user's attribute features, the attribute features of each consulting unit, and the identifiers of each consulting unit into the re-trained routing model to output the correct allocation probability corresponding to each consulting unit; Routing the current session to the target consulting unit with the highest correct allocation probability.

10. A routing device for a session, characterized in that, Includes: An identification module, configured to identify the intent label of each historical session, so as to route the historical session to the corresponding consulting unit; A mining module, configured to identify session negative samples from the historical sessions according to the occurrence frequency of each intent label in each consulting unit; A training module, configured to construct session positive samples through the session negative samples, and re-train the routing model based on the session negative samples and the session positive samples; A routing module, configured to deploy the re-trained routing model online, and use the re-trained routing model to route the current session to the target consulting unit; The mining module is further configured to: For each consulting unit, calculate the occurrence frequency of each intent label in the consulting unit, and screen out at least one intent label with the lowest frequency or intent labels with a frequency lower than the frequency threshold, so as to identify historical sessions with suspected routing errors; Identifying session negative samples according to the matching results of the historical sessions with suspected routing errors in other consulting units.

11. An electronic device, characterized in that, Includes: One or more processors; A storage device, configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-9.

12. A computer-readable medium having a computer program stored thereon, characterized in that, The program, when executed by the processor, implements the method according to any one of claims 1-9.

Images