Improving natural language interfaces by processing usage data

Through the automation of semantic processing technology and natural language interfaces, the problems of long navigation time and high cost in customer-company interactions are solved, efficient and accurate customer support is achieved, and operational costs are reduced.

CN111801674BActive Publication Date: 2024-12-03ASAPP INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980012983.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-02-12
Filing Date
2019-01-22
Publication Date
2024-12-03
Estimated Expiration
2039-05-15

AI Technical Summary

Technical Problem

The prior art has problems of long navigation time and high cost in the interaction between customers and companies, especially when handling multiple types of requests, it is difficult for natural language interfaces to achieve efficient and accurate response.

Method used

By using semantic processing technology, we can automate natural language interfaces, use customer usage data for intent classification and action processing, automate customer support processes, and provide semantic processing services through third-party companies to reduce costs.

Benefits of technology

Improves the response speed and accuracy of customer requests, reduces the company's operating costs, and provides customers with a better experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111801674B_ABST
    Figure CN111801674B_ABST
Patent Text Reader

Abstract

Third-party companies can help companies provide natural language interfaces for their customers. In order to implement the company's natural language interface, a configuration including information for intents, such as lists, seed messages, and hierarchical information, can be received. The configuration can be used to train an intent classifier, and the natural language interface can be deployed for use by customers. Usage data of the natural language classifier can be collected, and the usage data can be used to improve the natural language interface. Messages corresponding to intents can be clustered into clusters of similar messages, and a prototype message can be obtained for each cluster to provide a human-understandable description of the cluster. Information about clusters can be used to improve the natural language interface, such as by utilizing clusters to create new intents or moving clusters to other intents.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application (ASAP-0010-WO) claims priority to, and is a continuation of, U.S. Patent Application No. 15 / 894,504 (ASAP-0010-U01), filed on February 12, 2018, entitled “UPDATING NATURALLANGUAGE INTERFACES BY PROCESSING USAGE DATA”.

[0003] The above application is incorporated herein by reference in its entirety. Technical Field

[0004] The present invention relates to using semantic processing to improve customer support. Background Art

[0005] Companies need to interact with customers efficiently in order to provide services to their customers. For example, customers may need to obtain information about the company's services, may have questions about bills, or may need technical support from the company. Companies interact with customers in a variety of different ways. A company may have a website that customers can browse to perform various actions. A company may have an application ("app") that runs on a user's device (such as a smartphone or tablet) that provides services similar to a website. A company may have a phone number that customers can call to obtain information or talk to a customer service representative via an interactive voice response. Companies may also use various social media services (such as Facebook or Twitter) to respond to customers.

[0006] Some existing technologies for allowing customers to interact with companies can be cumbersome for customers. Navigating to the correct page on a website or app, or navigating to a voice menu on a phone call can be time consuming. Some existing technologies for allowing customers to interact with companies can be expensive for companies to implement. Hiring customer service representatives to manually respond to requests and answer phone calls can be a huge expense. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The following detailed description of the invention and certain embodiments thereof may be understood by reference to the following drawings:

[0008] Figure 1A is an example user interface for a natural language interface, and Figure 1B is a flow diagram of an example implementation of a natural language interface.

[0009] Figure 2Ais an example user interface for a natural language interface, and Figure 2B is a flow diagram of an example implementation of a natural language interface.

[0010] Figure 3 is an example system for a third-party company that provides services to multiple companies.

[0011] Figure 4 is a flow chart of an example implementation of configuring and improving a natural language interface.

[0012] Figure 5 is a flow diagram of an example implementation of configuring a natural language interface.

[0013] Figure 6 is a sample intent list and hierarchical information related to the intent list.

[0014] Figure 7 is a flow chart of an example implementation of a natural language interface by a client.

[0015] Fig. 8A and 8B are examples of graphs and simplified graphs that can be used through a natural language interface.

[0016] Fig.9A is an example of usage data corresponding to an intent, and Fig. 9B is an example of clustered usage data for an intent.

[0017] Fig.10 is an example cluster of usage data for intent.

[0018] Fig.11 is a flow diagram of an example implementation for improving a natural language interface by processing usage data.

[0019] Fig.12 is an example system that third-party companies can use to help companies configure and improve natural language interfaces.

[0020] Fig.13 is an exemplary computing device that can be used to configure and improve a natural language interface. DETAILED DESCRIPTION

[0021] Described herein are techniques for implementing a natural language interface to respond to user requests. Although the techniques described herein can be used for a wide variety of users and requests, for clarity of presentation, the following example will be used: a company provides a response to a customer request. However, the techniques described herein are not limited to customers and companies, responses can be provided to requests from users who are not customers, and the responses can come from any entity or person. The techniques described herein can use, or be combined with, any of the techniques described in U.S. Patent No. 9,715,496 or U.S. Patent Application Serial No. 15 / 789,241, each of which is incorporated herein by reference in its entirety for all purposes ("Incorporated Patents").

[0022] A natural language interface may be used to provide a fully automated experience for customers. A natural language interface may use semantic processing to allow customers to obtain desired information or desired support by making requests using natural language. Using natural language to make requests may allow customers to receive a resolution to their requests more quickly than other methods for making requests. In some instances, a customer's request may be resolved with a single interaction. For example, where a customer asks, "When will my package arrive?", the response may be, "Your package is scheduled to arrive on Wednesday and has a tracking number of 1234."

[0023] Implementing, diagnosing, and improving natural language interfaces can be challenging for companies. Customers may use a wide variety of different expressions to express a concept. For example, a customer who wants to change his address may state: "I want to change my address" or "I have moved." Additionally, a company may want to handle many different types of requests, such as address changes, billing requests, refunds, requests for new services, technical support, and any other issues that can typically be handled by a human customer support representative. Between the large number of different types of requests and the large number of ways to express each request, it may be challenging to implement a natural language interface that works properly for most methods of expressing each type of request. A company may desire an accurate natural language interface because it can reduce the company's costs and can also provide a better experience for the company's customers.

[0024] The techniques described herein improve the ability of a company to initially create a natural language interface and later improve the natural language interface based on usage data from customers who use the natural language interface. The techniques described herein allow a company to identify portions of a natural language interface that perform well (e.g., have a low error rate) and portions of the natural language interface that can be improved.

[0025] Usage data for customers using a natural language interface can include large amounts of data that can be challenging for a computer or human to review to understand why parts of the natural language interface are working well or not working well. Techniques described herein include clustering the usage data into clusters with similar usage data (e.g., different ways of expressing the same request), and selecting a prototype message for each cluster. By using information about clusters of usage data and their prototype messages, the usage data can be processed more efficiently to improve the operation of the natural language interface, as described in more detail below.

[0026] Figure 1A illustrates an example automated customer support session on a customer device, and Figure 1B is a flow chart of an example implementation of providing automated customer support. Figure 1B In the flowcharts and other flowcharts herein, the ordering of steps is exemplary, and other orders are possible, not all steps are required, steps can be combined or subdivided (in whole or in part), and in some implementations, some steps can be omitted or other steps can be added. The methods described by any flowchart described herein can be implemented, for example, by any computer or system described herein.

[0027] exist Figure 1A , the customer has used a customer device 100 to access a user interface for obtaining support from the company. The customer device 100 may be any appropriate device, such as a smart phone, a tablet, a wearable device, or an IoT device. The customer may request support using any appropriate technology, such as typing or speaking a message to an app running on the customer device (e.g., the company's app, or a third-party app created to handle customer requests), typing or speaking a message on a web page, sending a text message, or sending an email. As used herein, a text message includes any message sent as text, including, but not limited to, messages sent using SMS (Short Message Service) or a dedicated application (e.g., Facebook Messenger, Apple iMessage, Google Hangouts, or WhatsApp). Figure 1A In the example above, the customer has entered the message 110 "My address is incorrect".

[0028] The user interface described throughout the present disclosure may be of any type, including at least a mobile application, a web-based interface, a text input (texting) interface (e.g., via the receipt, response and / or prompt of text), a special program operating on a computer and / or any hardware device, a computer executable medium, and / or a display accessible to a client, a customer service representative, and / or a company agent or representative. The user interface includes any display device configured to depict the elements of the user interface, and / or an input / output device configured to receive responses from the user of any aspect of the embodiments described herein. Without limitation, in certain embodiments, the user interface includes one or more of a network communication, a graphic display, a mobile device, a keyboard, a mouse and / or a touch screen. In certain embodiments, in embodiments of the present disclosure, the same or different or reconfigured user interface may be provided to various users. For example, a first user interface may be provided to a user for training a natural language interface, a second user interface may be provided to a user (e.g., at a first company) for data collection or display (e.g., displaying cluster information) about a natural language interface, a third user interface may be provided to a user (e.g., at a second company) for training and / or data collection or display about a second natural language interface, and a fourth user interface may be provided to a customer for obtaining support from a company (e.g., the first company, the second company, or another company). The numbers of the user interface examples are provided to distinguish the user interfaces, and the numbers do not imply any order or relationship between the interfaces. Any given interface infrastructure (e.g., a computer executable medium operating on a server and / or network communications) may be utilized in one or more interfaces, and in a given embodiment, any one or more of these example user interfaces may be present or absent.

[0029] exist Figure 1B At step 150, the company receives a message from a customer. At step 160, the company processes the message to determine the intent of the message. For example, the company can process the message using an intent classifier that provides a score for each intent in a list of available intents and can select the intent with the highest score as the intent of message 110. At step 170, the company obtains an action corresponding to the determined intent of message 110. The action can correspond to any interaction sequence defined by the company. For example, the action can correspond to launching an application by a user (e.g., to change an address), providing a message with requested information to a user, directing a user to a resource (e.g., a web page), or connecting a user to a customer service representative. In this example, the selected action corresponds to launching an application by a user to change the user's address. At step 180, the action is implemented by the user.

[0030] The term "intent" as used herein should be understood broadly, and any appropriate intent may be used. For example, an intent may include the following user intents: getting help with a problem (e.g., help with a reduction in a faulty product or service), making a request (e.g., requesting account status, product replacement, service disconnection), entering information (e.g., a change in address, a change in purchase order information, changing the status of a previous request), or reporting an emergency (e.g., downed power lines, gas smell, weather conditions). For any particular application, a person skilled in the art may use available information about a company (such as its products, services, and customers) to determine an appropriate list of intents. In some implementations, which will be described in more detail below, the list of intents may be modified, such as by changing the intent or adding a new intent. In some implementations, a message may have more than one intent (e.g., a longer message with a compound sentence or multiple sentences), or multiple messages may be processed to determine the intent (e.g., where a customer partially enters a message and completes it in a subsequent message).

[0031] exist Figure 1A In FIG. 1 , the actions implemented are shown by boxes 120 and 130, box 120 is used to notify the user that the address change application is starting, and box 130 is the next step in the address change application. The customer can then continue the action to complete the address update (not in Figure 1A ).

[0032] Figure 2A illustrates another example automated customer support session on a customer device, and Figure 2B is a flow chart of this example implementation for providing automated customer support. Figure 2A In FIG. 2 , a customer has used a customer device 200 to access a user interface for obtaining support from a company, and the customer has entered a message 210 “My address is incorrect.”

[0033] exist Figure 2B At step 250, the company receives a message from a customer. At step 260, the company processes the message to determine one or more possible intents for the message. For example, multiple intents may be selected because there is some uncertainty about the correct intent (e.g., if the scores of the two highest scoring intents are close to each other). At step 260, the company may present information about the possible intents to the customer, such as Figure 2A As shown in box 220 of .

[0034] Then, clarification information may be received from the client to determine the final intent. For example, the intents may be presented to the client, and the client may then select the intent that best matches its request. As used herein, clarification information refers to any information received from the client that is used to determine the final intent from one or more possible intents.

[0035] In box 220, the customer has selected the "change address" intent. At step 265, the company receives clarifying information from the customer (e.g., the selection of an intent), and determines that the final intent is a change address intent. At step 270, the company obtains an action corresponding to the selected intent, such as the address change application described above. At step 275, the company implements the address change application, such as by presenting boxes 230 and 240 to the customer.

[0036] The interaction between a customer and a company may be referred to as a natural language interface because the customer can request assistance from the company by entering a natural language message. As used herein, a natural language interface is any communication interface in which a person communicating using the interface can use the interface to transmit natural language. For example, a natural language interface may include a person entering a typed message or speaking into a microphone.

[0037] A company that provides customer support to its customers may desire to use the services of a third-party company to improve the customer support experience and reduce overall customer support costs. For example, a company may find that using a third-party company's semantic processing service is more cost-effective than implementing its own semantic processing service. Figure 3 Illustrated is an example architecture that a company can use to obtain assistance from a third-party company in providing customer support to its customers.

[0038] Figure 3 A system 301 is illustrated that allows a third-party company 310 to provide customer support services to multiple companies. Figure 3 , third-party company 310 is providing customer support services to Company A 330, Company B 331, and Company C 332. Third-party company 310 may provide customer support services to any number of companies.

[0039] Customers of each company may seek customer support from the company where the support process uses the services of third-party company 310. For example, customer A 320 may be seeking support from company A 330, customer B 321 may be seeking support from company B 331, and customer C 322 may be seeking support from company C 332. It may or may not be obvious to the customers whether they are seeking customer support directly from the company or using the services of third-party company 310 to seek customer support.

[0040] The third party company 310 can help the company provide customer support in various ways. In some implementations, the third party company 310 can help connect customers with customer service representatives working on behalf of the company. For example, the third party company 310 can select a customer service representative, can provide a user interface to the customer to make it easier for the customer to request support, and can provide a user interface to the customer service representative to help the customer service representative respond to the customer's request. The customer service representative can have any appropriate relationship with the company on whose behalf the customer service representative is providing customer support. For example, the customer service representative can be an employee or contractor of the company and only provide customer support to the customers of the company, or the customer service representative can provide services to multiple companies and provide support to the customers of the multiple companies at the same time.

[0041] The communication between the third party company 310, the customer and the company can be constructed in various ways. In some implementations, all communications between the customer and the company can be conducted via the third party company 310, and there may not be any direct connection between the customer and the company. In some implementations, the third party company 310 can communicate with the company, but may not communicate directly with the customer. In some implementations, the customer can communicate directly with the company and also with the third party company 310.

[0042] In the case where the customer is connected to both the Company and the third-party company 310, each of the two connections can be used for different kinds of requests. For example, in the case where the customer interacts with the Company in a way that does not require the services of the third-party company 310 (e.g., browsing the Company's website), the customer can use the network connection with the Company. In the case where the customer interacts with the Company in a way that uses the services of the third-party company 310, the customer can use the network connection with the third-party company. It may not be obvious to the customer whether the customer is using the network connection with the Company or with the third-party company 310.

[0043] Figure 4 is a flow diagram of an example implementation of creating a natural language interface and improving the natural language interface over time by processing usage data of the natural language interface. Figure 4 It may be implemented by the company itself, or it may be implemented by a third party company that provides services to the company for implementing at least some aspects of the natural language interface.

[0044] At step 410, an initial configuration of the natural language interface is determined. For example, the configuration of the natural language interface may include a list of intents that are to be understood by the natural language interface. The configuration may also include an initial set of seed messages for each intent, such as example messages that illustrate how a client may express the intent.

[0045] At step 420, the natural language interface is deployed so that customers can use it. Deployment can include training models (such as training an intent classifier that can classify messages as corresponding to one or more intents) and then implementing them on a server accessible by customers.

[0046] At step 430, the natural language interface is used by the customer and usage data is collected. The usage data may include interactions of the customer with the natural language interface, and each interaction may include: a message received from the customer, one or more intents corresponding to the message as determined by the intent classifier, and an indication of whether the interaction was successful (e.g., whether the message was correctly classified, or whether the customer was able to obtain the desired assistance).

[0047] At step 440, the natural language interface is improved by processing the usage data. For example, it may be determined that the first intent is confused with the second intent, and the configuration should be updated to more clearly distinguish those intents. For another example, it may be determined that the existing intent should be divided into two intents.

[0048] Each of these steps is described in more detail below.

[0049] Configuration of the natural language interface

[0050] The company may need to configure the natural language interface. For example, when implementing the natural language interface for the first time, the company may need to specify the intents that will be handled by the natural language interface (e.g., address change, cancellation of service, etc.). The company may configure the natural language interface itself, or may use the services of a third-party company to configure the natural language interface. In the following, for clarity of presentation, the use of a third-party company is presented, but the same technology can be implemented by the company itself.

[0051] Figure 5 is a flow chart of an example implementation of configuring a natural language interface. At step 510, the third-party company presents a user interface to the company for providing configuration information related to the natural language interface. The third-party company can use any suitable technology to provide the user interface to the company, such as creating a web page that allows the company to provide information to configure the natural language interface.

[0052] At step 520, the third-party company receives information about the list of intents from the company. Information about the list of intents can be specified using any suitable technique. For example, for each intent provided by the company, the company can specify the name of the intent (e.g., a change address intent), a longer description of the intent (e.g., "This intent allows the customer to change their address"), and an action to be performed when the intent is selected (e.g., the address of a web application that will be launched by the customer). The intent can be associated with any suitable action, such as launching a program by the customer (e.g., a web application or smartphone application), providing a message to the customer ("Your package is arriving on Thursday"), providing a resource to the customer (e.g., directing the user to a web page that provides information to the customer), or connecting the customer with a customer service representative (e.g., via a messaging interface or a telephone call).

[0053] At step 530, the third party company receives hierarchical information about the list of intents, such as a graph or tree indicating relationships between the intents. Figure 6 An example hierarchical information of an intent list in a graphical form is illustrated. Figure 6 , intents (e.g., "change address") are shown along the right side of the graph. The top or root node of the graph is on the left side of the graph. The graph can group or categorize intents based on similarity of the subject matter of the intents. Figure 6 Nodes for three categories are illustrated: Billing, Service, and Technical Support. Each of these categories has multiple intents associated with it. For example, the Billing category includes nodes for the intents "Change Address," "Make Payment," and "Cancel Service." An intent may belong to more than one category. For example, the "Cancel Service" intent is included in the Billing category and the Service category. Hierarchical information may include additional levels, such as categories, subcategories, and then intents.

[0054] The hierarchical information can facilitate navigation by the customer or customer service representative to the customer's desired intent. In some cases, the customer may provide a message that does not clearly specify the customer's needs. For example, the customer may transmit the message "need Internet", and it may not be clear from the message whether the customer wants to add Internet-related services or may need Internet-related technical support. As described in more detail below, the hierarchical information can be used to disambiguate the customer's request.

[0055] In some implementations, a company may not provide hierarchical information about the list of intents, and the intents may be used without any hierarchical relationship.

[0056] In some implementations, the intent list may include an "other" intent to capture messages from a client that do not match any other intent provided by the client. For example, the "other" intent may operate as a generic background model.

[0057] At step 540, the third party company receives from the company an example message or seed message for each provided intent. The example messages may be created by a human, or may be extracted from existing logs of customer support sessions. These messages may be provided using any suitable technique, such as by providing a text file for each intent, where each line of the text file is an example message for the intent. Where the list of intents includes an "other" intent, the "other" intent may be seeded with messages that do not match any of the intents provided by the customer.

[0058] At step 550, the third party uses the seed message to train an intent classifier. Any suitable classifier may be used for the intent classifier, such as a neural network, a recursive neural network, a support vector machine, or a logistic regression classifier. The intent classifier may process the message (e.g., the text of the message, or a word embedding corresponding to the message) and output a score for each available intent, wherein the score indicates a match between the message and the intent.

[0059] At step 560, the performance of the intent classifier can be evaluated. Any appropriate evaluation process can be used. For example, the intent classifier can process each of the seed messages to ensure that they are classified as belonging to the intent for which the message is seeded. During the evaluation, the person doing the evaluation may be able to enter messages for the classifier to evaluate to see the highest scoring intent, or the score for each intent, for the entered messages. The intent classifier can also be tested against a test set of messages, each message of the test set has been labeled with an intent, and the error rate for the test set can be calculated and presented to the user. During the evaluation process, the configuration of the natural language interface can be changed. For example, the hierarchy can be changed, intents can be added, intents can be removed or merged, and seed messages can be added, removed, or moved to other intents.

[0060] At step 570, the natural language interface is deployed. For example, the trained intent classifier can be used to deploy software implementing the natural language interface to one or more servers.

[0061] Deployment of Natural Language Interfaces

[0062] After the natural language interface is deployed, customers can use it to get help from the company. The third-party company can deploy the natural language interface for multiple companies at the same time. For example, customers of the first company can use the first natural language interface, customers of the second company can use the second natural language interface, and so on.

[0063] Figure 7is a flow chart of an example implementation of a natural language interface by a company on behalf of a company by a third party company.At step 710, a message is received from a customer, such as by using any of the techniques described above.

[0064] At step 720, one or more intents are selected by processing the message using an intent classifier. Any suitable technique may be used for selecting one or more intents using the output of the classifier. For example, all intents with a score above a threshold may be selected, or the top N intents with the highest scores may be selected. In some instances, a single intent may be selected, and in such instances, steps 730 and 740 may be skipped, and processing may proceed to step 750.

[0065] Any suitable intent classifier may be used. For example, the intent classifier may have been trained using unsupervised, semi-supervised, or supervised techniques. The intent classifier may process feature vectors computed from the words (or possibly n-grams of words) of the message. For example, for each word of the message, a 1-hot vector or word embedding may be obtained to represent the word. In another example, the message may be represented as a bag of words.

[0066] For example, where the intent classifier is a recursive neural network, the recursive neural network may sequentially process feature vectors of the words of the message and compute one or both of an internal state vector or an output vector at each iteration. The final vector (e.g., the internal state vector or the output vector) from the final iteration may then be used to classify the intent of the message. A transformation of the final vector (e.g., a linear transformation, an affine transformation, or a multilayer perceptron) may be computed, where the transformed final vector has a length corresponding to the number of possible intents. The largest element of the transformed final vector may be selected to correspond to the intent of the message (or one or more largest elements may be selected to correspond to one or more intents).

[0067] For another example, where the intent classifier is a convolutional neural network, the convolutional neural network can process a matrix created from feature vectors of the words of the message. The convolutional neural network can perform one or more operations of filtering and max pooling to generate an output vector. A transformation of the output vector (such as the transformation described above) can be calculated, where the transformed output vector has a length corresponding to the number of possible intents. One or more intents can be selected using the transformed output vector as described above.

[0068] For another example, the intent classifier can be a multi-class support vector machine (SVM). Any suitable SVM can be used, such as an SVM that makes classification decisions between an individual intent and all other intents, an SVM that makes classification decisions between all pairs of possible intents, or an SVM that uses a directed acyclic graph to make classification decisions. After performing one or more classification decisions by comparing the intents, the SVM can select one or more intents as corresponding to the message.

[0069] At step 730 , a simplified intent graph is created using the one or more selected intents. Fig. 8A and 8B Illustration of how the selected intent can be used to simplify the intent hierarchy graph (e.g. Figure 6 ) example.

[0070] exist Fig. 8A , a complete intent graph 810 is shown, where the complete intent graph corresponds to the entire hierarchy of intents provided by the company. In this example, A, B, and C represent categories of intents, and A1, A2, A3, B1, B2, C1, C2, and C3 represent individual intents. In the complete intent graph 810, intents A1, A3, and C2 are highlighted because they are the intents selected at step 720. The score or probability for each selected intent is also shown. Fig. 8A In FIG. 8 , a pruned intent graph 820 is shown, where unselected intents are pruned from the graph, and categories with no remaining intents are also pruned. Fig. 8A , a simplified intent graph 830 is shown in which categories with only a single intent are removed and the single intent continues to exist on its own. The simplified intent graph 830 can be used to disambiguate among the selected intents in order to select the final intent for the message, as described in more detail below.

[0071] Figure 8B Another example of generating a simplified intent graph is shown in FIG. Figure 8B , the complete intent graph 850 is the same as the complete intent graph 810 except that a different set of intents are selected at step 730. In the complete intent graph 850, intents A1, A2, and C2 are selected with scores or probabilities as shown. Figure 8B , a pruned intent graph 860 is shown, where unselected intents are pruned away, and categories with no remaining intents are also pruned away. Figure 8B, a simplified intent graph 870 is shown, where the graph is converted into a tree. The simplified intent graph 870 can be used to disambiguate among the selected intents in order to select the final intent for the message, as described in more detail below.

[0072] At step 740, a final intent is selected using the simplified intent graph. Any suitable techniques may be used to select a final intent using the simplified intent graph, and such techniques may depend on the type of device being used by the customer (e.g., the size of the device's display). In the case where the customer is using a personal computer with a large display, a complete intent graph (e.g., complete intent graph 810) may be presented to the customer, along with other information such as the selected intent from step 720, the probability of the selected intent, and any other information about each intent. Depending on the size of the graph and the display, a portion of the complete intent graph may be presented to the customer, and the customer may be able to navigate to other portions of the complete intent graph. The customer may then select the final intent that best matches the customer's needs.

[0073] In some implementations, a simplified intent graph (e.g., simplified intent graph 830 or simplified intent graph 870) may be presented to the customer, along with any information about the intents or categories described above. The simplified intent graph may be sorted so that the intents or categories with the highest probability are shown at the top and the intents or categories with the lowest probability are shown at the bottom. In some implementations, the customer may be able to navigate to categories or intents that are not part of the simplified intent graph. The customer may then select the final intent that best matches the customer's needs.

[0074] In some implementations, instead of presenting the customer with an intent graphic, the customer may be presented with a question to select a final intent. For example, for the simplified intent graphic 830, the customer may be asked to choose between Category A and Intent C2. In the event that the customer selects Category A, the customer may then be asked to choose between Intent A1 and Intent A3. Alternatively, the customer may be asked to choose among Intents A1, A3, and C2. For the simplified intent graphic 870, the customer may similarly be asked to select Category C or Category A, and then be asked to select an intent from the selected category. Alternatively, the customer may be asked to choose among Intents C2, A1, and A2. The number of options in the question may depend on the size of the display of the customer's device.

[0075] In some implementations, a customer service representative can help select the final intent. It may be preferable for the customer service representative to show the full intent graphic, since the customer service representative may be more likely to have a larger display and may be more accustomed to viewing and browsing intent graphics. The customer service representative may also be able to view the message entered by the customer. The customer service representative can select the final intent for the customer after viewing the message from the customer.

[0076] In some implementations or instances, a customer service representative can select a final intent on behalf of a customer without the customer being aware of the customer service representative's involvement. A customer service representative can select an intent to cause an action (e.g., an address change action) to be implemented by the customer, and the action can be implemented immediately without the customer and the customer service representative exchanging messages. In some implementations or instances, the customer and the customer service representative can exchange messages while the customer service representative selects a final intent on behalf of the customer.

[0077] In some implementations, hierarchical information may not be available or may not be used with the intent list, and therefore, a simplified intent graph may not be created. A final intent may be selected from the one or more intents selected at step 720, and steps 730 and 740 may be omitted.

[0078] At step 750, an action corresponding to the final intent is obtained, and at step 760, the action is implemented by the client. These steps may use any of the techniques described above.

[0079] The natural language interface may be similarly used by a large number of customers of the company.As described below, usage data of the natural language interface may be collected and used to improve the natural language interface.

[0080] Improvements to the natural language interface

[0081] Usage data of the natural language interface can be collected and used to improve the natural language interface. Each session with a customer can be referred to as an interaction, and data can be collected for each interaction. Any appropriate data can be collected for an interaction, such as a message sent by the customer, a score calculated by an intent classifier, one or more intents selected by the intent classifier, a final intent (such as, the final intent selected as described above), and an indication of the success or failure of the interaction. The usage data can include data collected from some or all interactions during a time period. The usage data can then be analyzed to determine the performance of the natural language interface, and to determine changes to the natural language interface to improve performance.

[0082] Any appropriate technique may be used to determine an indication of the failure or success of an interaction. In some implementations, the indication may be a Boolean value indicating success or failure. In some implementations, the indication may be a value, such as a value in the range of 0 to 100, where a value above a threshold value (e.g., a threshold value of 50) indicates success. The following are some non-limiting examples of how an indication of success may be determined for an interaction: the highest-scoring intent of the classifier is the final intent for the interaction, the customer completes the action (e.g., the customer completes the address change process), a survey of the customer after the action has been completed (e.g., asking the customer if he is satisfied with the result), comparing the output of the intent classifier with the intent selected by the customer, comparing the output of the intent classifier with the action implemented by the customer, or asking the customer service representative to indicate the success of the interaction if the customer has already interacted with the customer service representative.

[0083] Fig.11 is a flow diagram of an example implementation of processing usage data to improve a natural language interface.At step 1110, usage data of a natural language interface is obtained, such as any of the usage data described above.

[0084] At step 1120, usage data is compiled for each intent in the intent list. Each interaction may be assigned to an intent using any appropriate criteria. For example, an interaction may be assigned to the intent corresponding to the highest score of the intent classifier, or to the final intent selected as described above. Fig.9A shows example usage data that has been compiled for the purpose of changing addresses. Fig.9A , a table shows messages received from clients corresponding to change of address intent, the number of times the message was received, and the success rate of the message (using any of the success indicators described above).

[0085] At step 1130, the messages assigned to the intent are clustered. For any intent, there may be a large number of ways that a person can express that intent. Some ways of expressing an intent may be similar to other ways, and some ways may be completely different from other ways. For example, in Fig.9A In the example above, for the change address intent, the first three messages are similar to each other: "I want to change my address"; "I need to change my address"; and "I want to change my address." Other ways of expressing the intent may be completely different, such as the message "I have moved." When collecting messages corresponding to an intent, there may be a large number of ways to express the intent, and therefore Fig.9A A table in may have hundreds, thousands, or even more unique messages.

[0086] To facilitate the analysis of the intent, the messages corresponding to the intent may be clustered so that messages similar to each other may be in the same cluster, while messages very different from each other may be in different clusters. Any appropriate technique may be used to cluster the messages.

[0087] In some implementations, a message embedding may be computed for each message, where the message embedding is a vector in an N-dimensional vector space that represents the message, but in a way that preserves useful information about the content of the message. Message embeddings may be computed from a training corpus (e.g., stored messages between customers and CSRs) using any suitable technique.

[0088] In some implementations, latent semantic indexing can be used to compute message embeddings. A feature vector (e.g., word count or term frequency-inverse document frequency value) can be computed for each message of a training corpus of messages, a matrix can be created by combining the feature vectors, and the message embedding can be obtained by performing a singular value decomposition of the matrix. See Deerwester, "Improving Information Retrieval with Latent Semantic Indexing," Proceedings of the 51st ASIS Annual Meeting (ASIS '88), Vol. 25 (October 1988).

[0089] In some implementations, message embeddings may be computed using latent Dirichlet allocation and as a mixture of topics from a topic model. See Blei et al., “Latent Dirichlet Allocation,” Journal of Machine Learning Research, vol. 3, pp. 993-1022, (2003).

[0090] In some implementations, the message embedding can be calculated as a paragraph vector, where the paragraph vector is trained to predict the next word of the paragraph given the context of the paragraph. See Le et al., "Distributed Representations of Sentences and Documents", Proceedings of the 31st International Conference on Machine Learning, W&CP vol. 32, Beijing, China, 2014.

[0091] In some implementations, the message embedding can be computed as the final hidden state of a recurrent neural network using an encoder-decoder language model. See Kiros et al., "Skip-Thought Vectors", Proceedings of the 28th International Conference on Neural Information Processing Systems, vol. 2, pp. 3294-3302, 2015.

[0092] Fig.10 Figure 2 shows a conceptual representation of message embedding in vector space. Fig.10 In the example of , the vector space is two-dimensional, but in real applications the number of dimensions will be much higher. Each message is represented by either an "x" or an "o" (the distinction is explained below). After the message embedding has been calculated for each message, the message embeddings can be clustered using any suitable clustering algorithm, such as k-means or agglomerative clustering. Fig.10 In the example, possible message clusters are indicated by dashed ovals. Each cluster may contain similar ways of expressing intent. For example, the messages "I want to change my address"; "I need to change my address"; and "I want to change my address" may be in the same cluster.

[0093] To describe a cluster, a message may be generated or selected from the cluster, and the message may be referred to as a prototype message. Any suitable technique may be used to obtain the prototype message of a cluster. In some implementations, the prototype message may be the most frequent message, the message with the largest count, the message closest to the center of the cluster (e.g., geometrically or by centroid), a message of a selected language type (e.g., the simplest language, the most commonly used words, or the inclusion or exclusion of jargon or selected words), or a randomly selected message. In some implementations, a prototype message may be generated rather than selected, and the prototype message may be different from the messages in the cluster. For example, a prototype message may be generated by processing the messages of the cluster using an encoder neural network, and then generating the prototype message using a decoder neural network. For another example, a prototype message may be generated by creating a parse tree for the messages in the cluster, transforming the parse tree into a single parse tree, and then using the transformed parse tree to generate the prototype message. In Fig.10 In the example above, the prototype message is represented as "o" instead of "x".

[0094] Fig. 9B It is possible to Fig.9AAn example list of clusters for change of address intent identified in the usage data for . Fig. 9B In the example, the first cluster has "I need to change my address" as the prototype message, and the cluster also includes the messages "I want to change my address" and "I want to change my address". Each cluster may include additional information about the cluster, such as the total count of messages in the cluster and the success rate of messages in the cluster.

[0095] At step 1140, information about the intents and clusters is presented to the user to allow the user to improve the natural language interface. In some implementations, the third-party company may present the user interface to one of its own employees, to an employee of a company for which the third-party company provides the natural language interface, or to another person. Any appropriate information about the natural language interface may be presented, such as any of the following: a list of intents, hierarchical information corresponding to the intents, statistical information related to the intents, and information about the clusters (e.g., prototype messages and statistical information for the clusters).

[0096] For example, a third-party company may present a user interface showing the success rate of each intent. A first intent may have a low success rate, and a person may select the first intent to view additional details about the intent. In response, the user interface may present information about the cluster in the first intent, such as a prototype message and statistical information.

[0097] Usage data for a natural language interface may include a very large number of interactions, and therefore there may be a very large number of unique messages. Due to this large number of messages, it may be difficult for a user to view individual messages and understand the performance of the natural language interface. By combining similar messages into clusters and providing prototype messages for the messages in the clusters, it may be easier for a person to understand which aspects of the natural language interface work well and which aspects need improvement. For example, a single intent may have tens of thousands of unique messages, but these messages may be clustered into five clusters that are presented to the user. The user can then view information about each of the five clusters to better understand why some clusters work well and some clusters do not work well.

[0098] The user interface may include controls to allow the user to modify the natural language interface. Any suitable controls for any suitable modifications may be included. For example, the user interface may include controls to create a new intent, controls to move a cluster from a first intent to a second intent, or controls to retrain the natural language interface using usage data. From step 1140, processing may proceed to step 1150, step 1160, or 1160, depending on the user's actions.

[0099] In some implementations, the user may be able to make other changes at step 1140. For example, the user may be able to change the hierarchical information associated with the intent list to improve the ability of a customer or other personnel to navigate the intent hierarchy.

[0100] In some implementations, suggestions may be presented to the user at step 1140. Any appropriate suggestions may be presented, such as changes to an intent or hierarchical information of an intent. In some implementations, where a cluster of intents has a success rate below a threshold, the cluster may be highlighted, or a suggestion may be presented to the user to create a new intent with the cluster or to move the cluster (or a portion thereof) to a new or different intent. For example, a cluster with a low success rate may be compared to clusters of other intents using any appropriate distance metric (e.g., Mahalanobis distance). Where the cluster with the low success rate is close to a cluster for a second intent, then the suggestion may be to move the cluster to the second intent. Where the cluster with the low success rate is not close to any existing clusters, then the suggestion may be to create a new intent with the cluster.

[0101] In some implementations, suggestions may be presented to improve the hierarchical information. For each interaction, the usage data may include navigation information about a user (e.g., a customer or a customer service representative) navigating the intent graph. In the event that the user typically navigates from a first node of the graph to a second node, and there is no direct connection between the first node and the second node, a suggestion may be made to modify the hierarchical information to include a connection between the first node and the second node.

[0102] At step 1150, the third-party company has received input from the user to create a new intent using the cluster. While reviewing information about the cluster of intents, the user may determine that customers frequently request operations that are not covered by any existing intents. For example, from Fig. 9B In the example clusters of , the user may determine that the cluster with the prototype "I am on vacation" is sufficiently different from the prototypes of the other clusters that a new intent should be created for providing services to customers when they go on vacation (e.g., temporarily stopping service or suspending the delivery of packages to customers). The user may specify any appropriate information for the new cluster, such as a name, description, or seed message. The user may also specify that messages from existing clusters ( Fig. 9B Some or all messages of cluster 5) are assigned to the new cluster.

[0103] At step 1155 , the third-party company may update the configuration of the natural language interface, such as by updating the configuration stored in the third-party company's data store.

[0104] At step 1170, the third party may retrain the intent classifier using the configuration and usage data. Previously, intent classifiers were trained using seed messages, which may have been relatively small in number when the natural language interface was first created due to a lack of available data. Once a customer has used the natural language interface, the intent classifier may be trained using some or all of the messages provided by the customer. In training the intent classifier, the messages used to train each intent may be the messages compiled at step 1120, except that a new intent was created at step 1155 using a message cluster specified by the user.

[0105] Returning back to step 1140, the user may instead decide that the natural language interface will likely perform better if the message cluster currently at the first intent is moved to the second intent. For example, there may be an existing intent for temporarily suspending service, and the user may decide that the cluster related to the customer going on vacation should be moved to the intent for temporarily suspending service.

[0106] At step 1160, the third party company receives input from the user to move the message cluster from the first intent to the second intent. At step 1165, the third party company modifies the configuration of the natural language interface so that the message cluster is now assigned to the second intent.

[0107] At step 1170 , the intent classifier is retrained using the updated configuration and usage data as described above.

[0108] Returning back to step 1140, the user may decide not to make any changes to the configuration, but simply to retrain the intent classifier with the usage data. Thus, the process may proceed to step 1170, where the intent classifier is retrained without making any changes to the configuration.

[0109] At step 1175, the retrained intent classifier can be evaluated or tested to ensure that the retrained intent classifier performs well. Any appropriate evaluation or test can be performed. In some implementations, the retrained classifier can be backtested using usage data or using other data. For example, the retrained classifier can be used to obtain classification results for each message from usage data that has been marked with a positive indication of success. It may be desirable that the retrained intent classifier produces the same intent as that produced by the previous classifier. In the event that the retrained intent classifier produces different intents from the usage data for messages exceeding a threshold number, processing can proceed back to step 1140 to allow further action to be taken. In the event that the retrained classifier passes the evaluation or test, processing can proceed to step 1180.

[0110] At step 1180, the natural language interface is deployed using the updated intent classifier trained at step 1170. The customer now has access to the natural language interface with the desired improved performance.

[0111] In some implementations, the user interface may allow the user to access information about individual messages in a cluster in addition to the prototype. For example, the user interface may allow the user to view any message for an intent, such as Fig.9A In some implementations, the user interface may allow a user to provide input regarding one or more messages, such as to create a new intent using one or more messages, or to move one or more messages from a first intent to a second intent.

[0112] Fig.11 The process may be repeated any number of times to improve the natural language interface. For example, it may be performed at regular intervals. Fig.11 The process may be performed more frequently, such as once a week or once a month. In some implementations, the process may be performed more frequently when the natural language interface is first started, and may be performed less frequently after the natural language interface has reached a desired performance level.

[0113] accomplish

[0114] As described above, third-party companies may provide services to companies to help those companies implement natural language interfaces for their customers. Fig.12 is an example system that third-party companies can use to provide these services to other companies. This can be implemented in a variety of ways Fig.12 system, and Fig.12 The system is an illustrative and non-limiting example. For example, other components can be added, components can be removed or merged, and connections between components can be added or removed.

[0115] exist Fig.12 1, a portion of a service that may be provided by a third-party company is enclosed in box 1200. The third-party company may provide services to the company, and company user 1260 may be an employee of the company, or other person who helps the company configure the natural language interface. User 1210, user 1211, and user 1212 may be customers of the company who use the natural language interface to get support from the company. The third-party company may provide services to any number of companies, but for clarity of presentation, only one company is shown.

[0116] The third-party company may have an intent console component 1250 that presents a user interface to the company user 1260 to provide an initial configuration of the natural language interface and to update the configuration at a later time. The intent console component 1250 may receive the configuration information and store it in a configuration data store 1290 so that other components may access it. The intent console may also receive analysis results (e.g., analysis of usage data) from the analysis component 1230 and present the analysis results to the company user 1260.

[0117] The third party company may have a training component 1240 that can train an intent classifier for classifying messages and may also train a clustering model for clustering messages of intent. The training component 1240 may also retrain or update the classifier and model after the usage data has been collected. The training component 1240 may access any of the configuration data store 1290, the usage data store 1280, and the model data store 1270.

[0118] The third party company may have an analysis component 1230 that analyzes the usage data of the natural language, such as any of the analysis described above. The analysis component may access the usage data store 1280.

[0119] The third party company may have a production component 1220 that allows a user to interact with a natural language interface. For example, the production component 1220 may receive a message from a customer, determine the intent of the message, enable an action to be performed by the customer, or connect the customer to a customer service representative. The production component 1220 may access a model data store 1270 and a usage data store 1280.

[0120] Model data store 1270 may store any suitable model or classifier, such as an intent classifier or a cluster model. Usage data store 1280 may store any suitable usage data, such as the usage data described herein. Configuration data store 1290 may store any suitable configuration information for a natural language interface, such as any configuration information described herein.

[0121] Fig.13 1300 for implementing any of the techniques described above. Fig.13 In the present invention, these components are shown as being on a single computing device, but these components can be distributed across multiple computing devices, such as a computing device system including, for example, an end-user computing device (e.g., a smart phone or tablet) and / or a server computing device (e.g., cloud computing).

[0122] The computing device 1300 may include any components typical of a computing device, such as volatile or non-volatile memory 1310, one or more processors 1311, and one or more network interfaces 1312. The computing device 1300 may also include any input and output components, such as a display, a keyboard, and a touch screen. The computing device 1300 may also include various components or modules that provide specific functionality, and these components or modules may be implemented in software, hardware, or a combination thereof. For example, as described above, the computing device 1300 may include any one of the production component 1220, the analysis component 1230, the training component 1240, or the intent console component 1250. Other implementations may include additional components or exclude some of the components.

[0123] Computing device 1300 may include or have access to various data stores, such as model data store 1270, usage data store 1280, and configuration data store 1290. The data store may use any known storage technology, such as files, relational or non-relational databases, or any non-transitory computer-readable medium.

[0124] The methods and systems described herein may be deployed in part or in whole by a machine that executes computer software, program code, and / or instructions on a processor. "Processor" as used herein is intended to include at least one processor, and unless the context clearly indicates otherwise, the plural and singular should be understood to be interchangeable. Any aspect of the present disclosure may be implemented as a method on a machine, as a system or device as part of a machine or associated with a machine, or as a computer program product embodied in a computer-readable medium that executes one or more machines. The processor may be a part of a server, client, network infrastructure, mobile computing platform, fixed computing platform, or other computing platform. The processor may be any type of computing or processing device capable of executing program instructions, codes, binary instructions, etc. The processor may be or include a signal processor, a digital processor, an embedded processor, a microprocessor, or any variant such as a coprocessor (mathematical coprocessor, graphics coprocessor, communication coprocessor, etc.), etc., which may directly or indirectly facilitate the execution of program codes or program instructions stored thereon. In addition, the processor may enable the execution of multiple programs, threads, and codes. Threads may be executed simultaneously to enhance the performance of the processor and facilitate the simultaneous operation of applications. By way of implementation, the methods, program codes, program instructions, etc. described herein may be implemented in one or more threads. A thread may generate other threads, which may have been assigned priorities associated with them; the processor may execute these threads based on priorities or any other order, based on instructions provided in the program code. The processor may include a memory that stores methods, codes, instructions, and programs as described herein and elsewhere. The processor may access a storage medium through an interface that may store methods, codes, and instructions as described herein and elsewhere. The storage medium associated with the processor for storing methods, programs, codes, program instructions, or other types of instructions that can be executed by a computing or processing device may include, but is not limited to, one or more of a CD-ROM, a DVD, a memory, a hard disk, a flash drive, a RAM, a ROM, a cache, and the like.

[0125] The processor may include one or more cores that can enhance the speed and performance of the multiprocessor. In embodiments, the process may be a dual-core processor, a quad-core processor, other chip-level multiprocessors (referred to as dies) that combine two or more independent cores, and the like.

[0126] The method and system described herein can be deployed partially or entirely by a machine executing computer software on a server, client, firewall, gateway, hub, router or other such computer and / or networking hardware. The software program can be associated with a server, which can include a file server, a print server, a domain server, an Internet server, an intranet server, and other variants such as auxiliary servers, host servers, distributed servers, etc. The server can include one or more of a memory, a processor, a computer-readable medium, a storage medium, a port (physical and virtual), a communication device, and an interface that can access other servers, clients, machines and devices through a wired or wireless medium, etc. The method, program or code described herein and elsewhere can be performed by a server. In addition, other equipment required for performing the method as described in the present application can be considered as a part of the infrastructure associated with the server.

[0127] The server can provide an interface to other devices, and the other devices include but are not limited to: client, other servers, printer, database server, print server, file server, communication server, distributed server etc. Additionally, this coupling and / or connection can facilitate the remote execution of the program across the network. Some or all of the networking of these devices can facilitate parallel processing of programs or methods at one or more locations without departing from the scope of the present disclosure. In addition, any device attached to the server by the interface can include at least one storage medium that can store methods, programs, codes and / or instructions. The central repository can provide program instructions to be executed on different devices. In this implementation, the remote repository can serve as a storage medium for program code, instructions and programs.

[0128] The software program may be associated with a client, which may include a file client, a print client, a domain client, an Internet client, an intranet client, and other variants such as an auxiliary client, a host client, a distributed client, and the like. The client may include one or more of a memory, a processor, a computer-readable medium, a storage medium, a port (physical and virtual), a communication device, and an interface capable of accessing other clients, servers, machines, and devices through a wired or wireless medium, and the like. The methods, programs, or codes described herein and elsewhere may be executed by the client. In addition, other devices required to perform the methods described in the present application may be considered part of the infrastructure associated with the client.

[0129] The client can provide an interface to other devices, and the other devices include but are not limited to: server, other clients, printer, database server, print server, file server, communication server, distributed server, etc. Additionally, this coupling and / or connection can facilitate the remote execution of the program across the network. The networking of some or all of these devices can facilitate the parallel processing of programs or methods at one or more locations without departing from the scope of the present disclosure. In addition, any device attached to the client through the interface can include at least one storage medium that can store methods, programs, codes and / or instructions. The central repository can provide program instructions to be executed on different devices. In this implementation, the remote repository can serve as a storage medium for program code, instructions and programs.

[0130] The methods and systems described herein may be deployed in part or in whole through a network infrastructure. The network infrastructure may include elements such as computing devices, servers, routers, hubs, firewalls, clients, personal computers, communication devices, routing devices, and other active and passive devices, modules and / or components as known in the art. The computing and / or non-computing devices associated with the network infrastructure may also include storage media, such as flash memory, buffers, stacks, RAM, ROM, etc., in addition to other components. The processes, methods, program codes, instructions described herein and elsewhere may be executed by one or more of the network infrastructure elements.

[0131] The methods, program codes, and instructions described herein and elsewhere may be implemented on a cellular network having multiple cells. The cellular network may be a frequency division multiple access (FDMA) network or a code division multiple access (CDMA) network. The cellular network may include mobile devices, cell sites, base stations, repeaters, antennas, towers, etc. The cell network may be GSM, GPRS, 3G, EVDO, mesh, or other network types.

[0132] The methods, program codes and instructions described herein and elsewhere may be implemented on or by a mobile device. Mobile devices may include navigation devices, cellular phones, mobile phones, mobile personal digital assistants, laptops, PDAs, netbooks, pagers, e-book readers, music players, etc. These devices may include storage media, such as flash memory, buffers, RAM, ROM, and one or more computing devices, among other components. A computing device associated with a mobile device may be enabled to execute program codes, methods, and instructions stored thereon. Alternatively, a mobile device may be configured to execute instructions in collaboration with other devices. A mobile device may communicate with a base station (which is docked with a server) and be configured to execute program codes. A mobile device may communicate on a peer-to-peer network, a mesh network, or other communication network. The program code may be stored on a storage medium associated with a server and executed by a computing device embedded in the server. A base station may include a computing device and a storage medium. A storage device may store program codes and instructions executed by a computing device associated with a base station.

[0133] Computer software, program code and / or instructions may be stored and / or accessed on machine-readable media, which may include: computer components, devices and recording media that retain digital data used for computation for a certain time interval; semiconductor storage devices, known as random access memory (RAM); mass storage devices, typically used for more permanent storage, such as optical disks, magnetic storage forms such as hard disks, tapes, drums, cards and other types; processor registers, cache memory, volatile memory, non-volatile memory; optical storage devices, such as CDs, DVDs; removable media, such as flash memory (e.g., USB stick or key), floppy disks, magnetic tape, paper tape, punch cards, stand-alone RAM disks, Zip drives, removable mass storage devices, off-line, etc.; other computer memory, such as dynamic memory, static memory, read / write storage devices, variable storage devices, read-only, random access, sequential access, location addressable, file addressable, content addressable, network attached storage, storage area network, bar code, magnetic ink, etc.

[0134] The methods and systems described herein may transform a physical and / or intangible item from one state to another state. The methods and systems described herein may also transform data representing a physical and / or intangible item from one state to another state.

[0135] The elements described and depicted herein (including the flowcharts and block diagrams in the entire figures) imply the logical boundaries between the elements. However, according to software or hardware engineering practices, the depicted elements and their functions can be implemented on a machine through a computer executable medium, which has a processor that can execute program instructions stored thereon as a monolithic software structure, as an independent software module, or as a module using external routines, codes, services, etc., or any combination of these, and all such implementations can be within the scope of the present disclosure. Examples of such machines may include, but are not limited to: personal digital assistants, laptops, personal computers, mobile phones, other handheld computing devices, medical equipment, wired or wireless communication equipment, transducers, chips, calculators, satellites, tablet PCs, e-books, gadgets, electronic devices, devices with artificial intelligence, computing devices, networking equipment, servers, routers, etc. In addition, the elements or any other logical components depicted in the flowcharts and block diagrams can be implemented on a machine capable of executing program instructions. Therefore, although the aforementioned figures and descriptions set forth the functional aspects of the disclosed system, unless explicitly stated or clearly known from the context, the specific software arrangements for implementing these functional aspects should not be inferred from these descriptions. Similarly, it will be appreciated that the individual steps identified and described above may vary, and the order of the steps may be adapted to the specific application of the technology disclosed herein. All such variations and modifications are intended to fall within the scope of the present disclosure. Thus, unless required by a specific application, or explicitly stated or clearly known from the context, the depiction and / or description of the order of the individual steps should not be understood as requiring a specific execution order for these steps.

[0136] The method and / or process described above and steps thereof can be implemented with hardware, software or any combination of hardware and software suitable for a particular application. Hardware can include a general-purpose computer, and / or a special computing device, or a specific computing device, or a specific aspect or component of a specific computing device. These processes can be implemented in one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors or other programmable devices, and internal and / or external memories. These processes can also or alternatively be embodied in a dedicated integrated circuit, a programmable gate array, a programmable array logic, or any other device or device combination that can be configured to process electronic signals. It will be further appreciated that one or more of these processes can be implemented as a computer executable code that can be executed on a machine-readable medium.

[0137] Computer executable code may be created using a structured programming language (such as C), an object-oriented programming language (such as C++), or any other high-level or low-level programming language (including assembly language, hardware description languages, and database programming languages ​​and techniques) that may be stored, compiled, or interpreted to run on one of the above devices, as well as on a processor, a heterogeneous combination of processor architectures, or a combination of different hardware and software, or any other machine capable of executing program instructions.

[0138] Therefore, on the one hand, each method described above and its combination can be embodied in a computer executable code, which performs the steps of the method when executed on one or more computing devices. On the other hand, these methods can be embodied in a system that performs its steps, and can be distributed across devices in a variety of ways, or all functions can be integrated into a dedicated stand-alone device or other hardware. On the other hand, the components for performing the steps associated with the process described above can include any of the hardware and / or software described above. All such permutations and combinations are intended to fall within the scope of the present disclosure.

[0139] Although the present invention has been disclosed in conjunction with the preferred embodiments shown and described in detail, various modifications and improvements thereon will become apparent to those skilled in the art. Therefore, the spirit and scope of the present invention are not limited by the foregoing examples, but should be understood in the broadest sense permitted by law.

[0140] All documents cited herein are incorporated by reference.

Claims

1. A system for implementing a natural language interface to allow a customer to interact with a company, the system comprising: At least one server computer comprising at least one processor and at least one memory, the at least one server computer being configured to: receiving a configuration for a natural language interface of the company, wherein the configuration refers to a list of schematic diagrams and a plurality of seed messages for each intent in the list of intents; training an intent classifier using the configuration, wherein the intent classifier processes a message to classify the message as corresponding to one or more intents in a list of intents; Use the intent classifier to deploy a natural language interface used by the company’s customers; collecting usage data of the natural language interface, wherein the usage data comprises a plurality of interactions, and wherein each interaction comprises a message received from one of a plurality of clients, an intent selected by an intent classifier, and an indication of success of the interaction; The usage data is analyzed in the following ways: obtaining a first set of interactions from the usage data, wherein the first set of interactions corresponds to a first intent in the list of intents, clustering the messages of the first group of interactions into a plurality of clusters, and obtaining a prototype message for each cluster of the plurality of clusters, wherein each prototype message includes an example message representative of an associated cluster; providing a user interface for viewing information about the plurality of clusters, the information including a prototype message for each of the plurality of clusters; receiving input to (i) create a new intent using a first cluster in the plurality of clusters or (ii) move the first cluster in the plurality of clusters to a second intent in a list of intents; updating the configuration using the received input; updating the intent classifier by retraining the intent classifier using the updated configuration and the usage data; as well as Deploy a natural language interface using the updated intent classifier.

2. The system of claim 1, wherein the at least one server computer is configured to: providing a second user interface to the company to allow the company to configure the natural language interface; and The configuration is received from the company.

3. The system of claim 1 , wherein the at least one server computer is configured to: providing a third user interface to the second company to allow the second company to configure the second natural language interface; and A second natural language interface is deployed, wherein the second natural language interface is used by customers of the second company.

4. The system of claim 1, wherein the at least one server computer is configured to: receiving messages from customers seeking support from the Company; selecting two or more intents by processing the message using an intent classifier; receiving clarification information related to the two or more intentions; selecting a final intent from the two or more intents using the clarification information; as well as Implement actions corresponding to the final intent through the customer.

5. The system of claim 4, wherein the at least one server computer is configured to present the hierarchical information of the two or more intents to a customer or a customer service representative.

6. The system of claim 4, wherein the clarifying information is received from the customer or from a customer service representative assisting the customer.

7. The system of claim 1 , wherein the first indication of success of the first interaction is determined by one or more of: asking the customer whether the first interaction was successful; asking the customer service representative assisting the customer whether the first interaction was successful; Make sure the customer completes the action; Compare the output of the intent classifier to the intent selected by the customer; or The output of the intent classifier is compared to the actions implemented by the customer.

8. The system of claim 1, wherein the at least one server computer is configured to: Calculate the success rate of the cluster; using the success rate of the clusters to determine suggestions for presenting for improving the natural language interface; Suggestions are presented (i) to create a new intent using the cluster, or (ii) to move the cluster to another intent.

9. The system of claim 1, wherein the natural language interface processes a message received from a client, determines the intent of the message, uses the determined intent to select an action, and causes the action to be implemented by the client.

10. A computer-implemented method for modifying a customer service message, the method comprising: receiving a configuration for a natural language interface of the company, wherein the configuration refers to a list of schematic diagrams and a plurality of seed messages for each intent in the list of intents; training an intent classifier using the configuration, wherein the intent classifier processes a message to classify the message as corresponding to one or more intents in a list of intents; Use the intent classifier to deploy a natural language interface used by the company’s customers; collecting usage data of the natural language interface, wherein the usage data comprises a plurality of interactions, and wherein each interaction comprises a message received from one of a plurality of clients, an intent selected by an intent classifier, and an indication of success of the interaction; The usage data is analyzed in the following ways: obtaining a first set of interactions from the usage data, wherein the first set of interactions corresponds to a first intent in the list of intents, clustering messages of the first set of interactions into a plurality of clusters, and obtaining a prototype message for each cluster of the plurality of clusters, wherein each prototype message includes an example message representative of an associated cluster; providing a user interface for viewing information about the plurality of clusters, the information including a prototype message for each of the plurality of clusters; receiving input to (i) create a new intent using a first cluster in the plurality of clusters or (ii) move the first cluster in the plurality of clusters to a second intent in a list of intents; updating the configuration using the received input; updating the intent classifier by retraining the intent classifier using the updated configuration and the usage data; as well as Deploy a natural language interface using the updated intent classifier.

11. The method of claim 10, wherein the intent classifier comprises a recurrent neural network.

12. The method of claim 10, wherein the configuration includes specification of an action to be performed by the client for each intent in the list of intents.

13. The method of claim 10, wherein clustering the messages of the first group of interactions comprises: Calculating message embeddings for the first group of interacted messages; as well as Messages of the first set of interactions are clustered using the message embeddings.

14. The method of claim 10, wherein obtaining a prototype message for a cluster in the plurality of clusters comprises: Select the message with the largest count or select the message closest to the center of the cluster.

15. The method of claim 10, wherein obtaining a prototype message for a cluster in the plurality of clusters comprises: Using neural networks to generate prototype messages.

16. The method of claim 10, comprising calculating a success rate for each intent in the intent list.

17. One or more non-transitory computer-readable media comprising computer-executable instructions that, when executed, cause at least one processor to perform actions comprising: receiving a configuration for a natural language interface of the company, wherein the configuration refers to a list of schematic diagrams and a plurality of seed messages for each intent in the list of intents; training an intent classifier using the configuration, wherein the intent classifier processes a message to classify the message as corresponding to one or more intents in a list of intents; Use the intent classifier to deploy a natural language interface used by the company’s customers; collecting usage data of the natural language interface, wherein the usage data comprises a plurality of interactions, and wherein each interaction comprises a message received from one of a plurality of clients, an intent selected by an intent classifier, and an indication of success of the interaction; The usage data is analyzed in the following ways: obtaining a first set of interactions from the usage data, wherein the first set of interactions corresponds to a first intent in the list of intents, clustering messages of the first set of interactions into a plurality of clusters, and obtaining a prototype message for each cluster of the plurality of clusters, wherein each prototype message includes an example message representative of an associated cluster; providing a user interface for viewing information about the plurality of clusters, the information including a prototype message for each of the plurality of clusters; receiving input to (i) create a new intent using a first cluster in the plurality of clusters or (ii) move the first cluster in the plurality of clusters to a second intent in a list of intents; updating the configuration using the received input; updating the intent classifier by retraining the intent classifier using the updated configuration and the usage data; as well as Deploy a natural language interface using the updated intent classifier.

18. The one or more non-transitory computer-readable media of claim 17, wherein the configuration includes hierarchical information of a list of intents.

19. The one or more non-transitory computer-readable media of claim 18, wherein the actions comprise: Receiving input for modifying hierarchical information associated with a list of intents; as well as The configuration is updated using the input to modify hierarchical information associated with the intent list.

20. The one or more non-transitory computer-readable media of claim 18, wherein the actions comprise: Renders suggestions for modifying the hierarchical information of an intent list.

21. The one or more non-transitory computer-readable media of claim 17, wherein the plurality of seed messages for an intent are extracted from existing logs of customer support sessions or created by a human.

22. The one or more non-transitory computer-readable media of claim 17, wherein the actions comprise: The updated intent classifier is tested by obtaining classification results using the updated intent classifier on the message corpus.