GUI features for updating conversational bots

Abstract

Various techniques related to creating and / or updating a chatbot are described herein. A graphical user interface (GUI) is described that facilitates updating a computer-implemented response model of a chatbot based on interactions between a developer and features of the GUI, wherein the GUI depicts a conversation between a user and the chatbot.

Description

Background Art

[0001] A chatbot refers to a computer-implemented system that provides a service, wherein the chatbot is conventionally based on hard-coded rules, and further, wherein people interact with the chatbot through a chat interface. The service can be any suitable service, ranging from functional services to entertainment services. For example, a chatbot can be configured to provide customer service support for a website designed to sell electronic products, a chatbot can be configured to provide jokes in response to requests, etc. In operation, a user provides input to the chatbot through an interface (wherein the interface can be a microphone, a graphical user interface that accepts input, etc.), and the chatbot responds to such input using (based on the input) responses that are identified as helpful to the user. The input provided by the user can be a natural language input, a selection of a key, entering data into a table, an image, a video, location information, etc. The response output by the chatbot in response to the input can be in the form of text, graphics, audio, or other types of human-interpretable content.

[0002] Conventionally, creating a chatbot and updating the deployed chatbot is a difficult task. In an example, when creating a chatbot, the computer programmer is tasked with creating the chatbot in code or using a tree diagram tool through a user interface, where the computer programmer must understand the chatbot's domain of expertise to ensure that the chatbot interacts appropriately with the user. The chatbot can be updated when the user interacts with the chatbot in an unexpected way, or when new functionality is desired; however, in order to update the chatbot, the computer programmer (or another computer programmer who is a domain expert and understands the current operation of the chatbot) must update the code, which can be time-consuming and expensive. Summary of the invention

[0003] The following is a brief summary of the subject matter that is described in greater detail herein. This summary is not intended to limit the scope of the claims.

[0004] Various techniques related to graphical user interface (GUI) features are described herein that are well suited for creating and / or updating chatbots. In an exemplary embodiment, the chatbot may include computer executable code, an entity extractor module, and a response model, the entity extractor module being configured to identify and extract entities in user-provided input, the response model being configured to select output to provide to the user in response to receiving input from the user (wherein the output of the response model is based on the most recently received input, the previous input in the session, and the entities identified in the session). For example, the response model may be an artificial neural network (ANN), such as a recursive neural network (RNN) or other suitable neural network, which is configured to receive input (such as text, location, etc.) and provide output based on such input.

[0005] The GUI features described herein are configured to facilitate training the extractor modules and / or response models referenced above. For example, the GUI features can be configured to present the type of entity and the parameters corresponding thereto to the developer, wherein the entity type can be customized by the developer; and the parameters can indicate whether the entity type can appear in user input, system response, or both; whether the entity type supports multiple values; and whether the entity type is negatable. The GUI features are also configured to present a list of available responses, and are also configured to allow the developer to edit existing responses or add new responses. When the developer indicates that a new response is to be added, the response model will be modified to support the new response. Similarly, when the developer indicates that an existing response is to be modified, the response model will be updated to support the modified response.

[0006] The GUI features described herein are also configured to support adding new training dialogues for chatbots, where developers can provide input for the purpose of training entity extractor modules and / or response models. A training dialogue refers to a session between a chatbot and a developer, which is conducted by the developer to train entity extractor modules and / or response models. When the developer provides input to the chatbot, the GUI feature identifies the entities in the user input identified by the extractor module, and further identifies possible responses of the chatbot. In addition, the GUI feature illustrates the probabilities corresponding to possible responses, so that the developer can understand how the chatbot selects responses, and further indicates to the developer where more training may be expected. The GUI feature is configured to receive input from the developer regarding the correct response from the chatbot, and the interaction between the chatbot and the developer can continue until the training dialogue has been completed.

[0007] In addition, the GUI feature is configured to allow the developer to select previous interactions between the user and the chatbot from the log and train the chatbot based on the previous interactions. For example, a conversation (e.g., a session) between an end user (e.g., different from the developer) and the chatbot can be presented to the developer, wherein the conversation includes inputs proposed by the user and also includes corresponding responses of the chatbot. The developer can select an incorrect response from the chatbot and can notify the chatbot of a different correct response. Then, based on the correct response identified by the developer, the entity extractor module and / or the response model is updated. Therefore, the GUI features described herein are configured to allow the chatbot to be interactively trained by the developer.

[0008] There is more specificity with respect to interactive training of the response model, in that the response model is retrained as the developer presents input regarding the correct response, thereby allowing for incremental retraining of the response model. Further, an ongoing conversation can be reattached to a new retrained response model. As previously mentioned, the output of the response model is based on the most recently received input, the previously received input, the previous response to the previously received input, and the recognized entities. Therefore, corrections made to responses output by the response model may affect future responses of the response model in the conversation; therefore, the conversation can be reattached to the retrained response model so that when the conversation continues, the output from the response model is from the retrained response model.

[0009] The above summary of the invention presents a simplified summary of the invention in order to provide a basic understanding of some aspects of the systems and / or methods discussed herein. This summary of the invention is not an extensive overview of the systems and / or methods discussed herein. Its purpose is not to identify key / determining elements or to describe the scope of such systems and / or methods. Its sole purpose is to present some concepts in a simplified form as a preface to a more detailed description presented later. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a functional block diagram of an exemplary system that facilitates presentation of GUI features on a display of a client computing device operated by a developer, wherein the GUI features are configured to allow the developer to interactively update a chatbot.

[0011] Figures 2 to 23 An exemplary GUI configured to assist a developer in updating a chatbot is depicted.

[0012] Fig.24 is a flow chart illustrating an exemplary method for creating and / or updating a chatbot.

[0013] Fig.25is a flow chart illustrating an exemplary method for creating and / or updating a chatbot.

[0014] Fig.26 is a flow chart illustrating an exemplary method for updating entity extraction tags within a conversation flow.

[0015] Fig. 27 is an exemplary computing system. DETAILED DESCRIPTION

[0016] Various techniques related to GUI features are now described with reference to the accompanying drawings, which are very suitable for creating and / or updating chatbots, wherein the same reference numerals are used to refer to the same elements throughout the text. In the following description, for the purpose of explanation, many specific details are proposed in order to provide a thorough understanding of one or more aspects. However, it may be obvious that such (multiple) aspects can be practiced without these specific details. In other cases, well-known structures and devices are shown in block diagram form to help describe one or more aspects. Further, it is to be understood that the functionality described as being performed by certain system components can be performed by multiple components. Similarly, for example, a component can be configured to perform the functionality described as being performed by multiple components.

[0017] Moreover, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless specified otherwise or clear from the context, the phrase "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, the phrase "X employs A or B" is satisfied by any of the following: X employs A; X employs B; or X employs A and B. Additionally, the articles "a" and "an" used in this application and the appended claims should generally be construed to mean "one or more" unless specified otherwise or clear from the context to be directed to the singular.

[0018] Further, as used herein, the terms "component," "module," and "system" are intended to encompass a computer-readable data storage device that is configured with computer-executable instructions that cause specific functionality to be performed when executed by a processor. Computer-executable instructions may include routines, functions, etc. It is also understood that a component or system may be located on a single device or distributed across multiple devices. Additionally, as used herein, the term "exemplary" is intended to mean serving as an illustration or example of something and is not intended to indicate a preference.

[0019] Reference Figure 1, illustrates an exemplary system 100 for interactively creating and / or modifying a chatbot. A chatbot is a computer-implemented system configured to provide a service. The chatbot can be configured to receive input from a user, such as transcribed voice input, text input provided through a chat interface, location information, indications of selected keys, etc. Thus, the chatbot can be executed, for example, on a server computing device and provide responses to inputs proposed through a chat interface on a web page being viewed on a client computing device. In another example, the chatbot can be executed on a server computing device as part of a computer-implemented personal assistant.

[0020] The system 100 includes a client computing device 102 operated by a developer who will create a new chatbot and / or update an existing chatbot. The client computing device 102 may be a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone, a wearable computing device (e.g., a head-mounted computing device), etc. The client computing device 102 includes a display 104 on which the graphical features described herein are shown on the display 104 of the client computing device 102.

[0021] The system 100 also includes a server computing device 106 that communicates with the client computing device 102 via a network 108 (e.g., the Internet or an intranet). The server computing device 106 includes a processor 110 and a memory 112, wherein the memory 112 has a chatbot development system 114 (bot development system) loaded therein, and further, wherein the bot development system 114 is executable by the processor 110. Although the exemplary system 100 illustrates the bot development system 114 as executing on the server computing device 106, it is to be understood that all or part of the bot development system 114 may alternatively be executed on the client computing device 102.

[0022] The bot development system 114 includes or has access to an entity extractor module 116, wherein the entity extractor module 116 is configured to identify entities in input text provided to the entity extractor module 116, wherein the entities have one or more predefined types. For example, and according to the example presented below, when the chatbot is configured to assist in placing an order for pizza, a user may present the input "I would like to order a pizza with pepperoni and mushrooms." The entity extractor module 116 may identify "pepperoni" and "mushrooms" as entities to be extracted from the input.

[0023] The robot development system 114 also includes or has access to a response model 118 configured to provide outputs, wherein the outputs are functions of inputs received from the user, and further, wherein the outputs are optionally functions of entities identified by the extractor module 116, previous outputs of the response model 118, and / or previous inputs to the response model. For example, the response model 118 may be or include an ANN such as an RNN, wherein the ANN includes an input layer, one or more hidden layers, and an output layer, wherein the output layer includes nodes representing potential outputs of the response model 118. The input layer may be configured to receive inputs from the user as well as state information (e.g., where the user is in the ordering process when the user presents the input). In non-limiting examples, the output nodes may represent potential outputs "yes", "you're welcome", "do you want any other toppings", "$toppings on your pizza", "do you want to order another pizza", "I can't help you, but I can help order a pizza", etc. (where "$toppings" is used for entity replacement, such that a location in the memory 112 is called, and thus the identified entity replaces $toppings in the output). Continuing with the example presented above, after entity extractor module 116 identifies "pepperoni" and "mushrooms" as entities, response model 118 may output data indicating that the most likely correct response is "You have $toppings on your pizza," where "$toppings" (in the output of response model 118) is replaced with the entities "pepperoni" and "mushrooms." Thus, in this example, response model 118 provides the user with the response "You have pepperoni and mushrooms on your pizza."

[0024] The robot development system 114 additionally includes computer executable code 120 that interacts with the entity extractor module 116 and the response model 118. For example, the computer executable code 120 maintains a list of entities proposed by the user, adds entities to the list upon request, removes entities from the list upon request, etc. Additionally, the computer executable code 120 can receive the output of the response model 118 and return entities from the memory 112 when appropriate. Thus, when the response model 118 outputs "Your pizza has $toppings", "$toppings" can be a call to the code 120, which obtains "pepperoni" and "mushrooms" from the entity list in the memory 112, so that "Your pizza has pepperoni and mushrooms" is provided as the output of the chatbot.

[0025] The robot development system 114 additionally includes a graphical user interface (GUI) renderer module 122 configured to cause a GUI to be displayed on the display 104 of the client computing device 102, wherein the GUI is configured to facilitate interactively updating the entity extractor module 116 and / or the response model 118. Various exemplary GUIs are presented herein, wherein the GUI is caused to be displayed on the display 104 of the client computing device 102 by the GUI renderer module 122, and further, wherein such GUI is configured to facilitate a developer operating the client computing device 102 to update the entity extractor module 116 and / or the response model 118.

[0026] The robot development system 114 also includes an updater module 124 that is configured to update the entity extractor module 116 and / or the response model 118 based on input received from the developer while interacting with one or more (multiple) GUIs presented on the display 104 of the client computing device 102. The updater module 124 can make various updates, including but not limited to: 1) training the entity extractor module 116 based on exemplary inputs including entities; 2) updating the entity extractor module 116 to identify new entities; 3) updating the entity extractor module 116 with new types of entities; 4) updating the entity extractor module 116 to discontinue identifying a certain entity or entity type; 5) updating the response model 118 based on a dialog raised by the developer; 6) updating the response model 118 to include new outputs for the response model 118; 7) updating the response model 118 to remove existing outputs from the response model 118; 8) updating the response model 118 based on a user's dialog with the chatbot, etc. In an example, when the response model 118 is an ANN, the updater module 124 may update weights of synapses assigned to the ANN, may enable new input or output nodes in the ANN, may veto input or output nodes in the ANN, and / or the like.

[0027] Now refer to Figures 2 to 23 , illustrates various exemplary GUIs that may be displayed by the GUI renderer module 122 on the display 104 of the client computing device 102. These GUIs illustrate updating an existing chatbot that is configured to help a user order a pizza; however, it is to be understood that the GUIs are exemplary in nature and that the features described herein are applicable to any suitable chatbot that relies on a machine learning model to generate output. Further, the GUIs are well suited for creating and / or training entirely new chatbots.

[0028] Reference only Figure 2, illustrates an exemplary GUI 200, wherein in response to the client computing device 102 receiving an indication from a developer that a selected chatbot is to be updated, the GUI 200 is presented on the display 104 of the client computing device 102. In the exemplary GUI 200, it is indicated that the selected chatbot is configured to assist an end user in ordering a pizza. The GUI 200 includes several buttons: a home button 202, an entity button 204, an action button 206, a training dialogue button 208, and a log dialogue button 210. In response to selecting the home button 202, the GUI 200 is updated to present a list of selectable chatbots. In response to selecting the entity button 204, the GUI 200 is updated to present information about entities recognized by the currently selected chatbot. In response to selecting the action button 206, the GUI 200 is updated to present a list of actions (e.g., responses) of the chatbot. In response to selecting the training dialogue button 208, the GUI 200 is updated to present a list of training dialogues (e.g., dialogues between a developer and a chatbot used in conjunction with training of the chatbot). Finally, in response to selection of the log conversation button 210, the GUI 200 is updated to present a list of logged conversations (eg, conversations between the chatbot and end users of the chatbot).

[0029] Now refer to Figure 3 , illustrates an exemplary GUI 300 in which the GUI renderer module 122 causes the GUI 300 to be presented on the display 104 of the client computing device 102 in response to a developer indicating that the developer wants to view and / or modify the code 120. The GUI 300 may be presented in response to the developer selecting a button (not shown) on the GUI 200. The GUI 300 includes a code editor interface 302 that includes a field 304 for depicting the code 120. The field 304 may be configured to receive input from the developer so that the code 120 is updated by interacting with the code in the field 304.

[0030] Now refer to Figure 4, illustrates an exemplary GUI 400, wherein in response to the client computing device 102 receiving an indication that the developer has selected the entity button 204, the GUI renderer module 122 causes the GUI 400 to be presented on the display 104 of the client computing device 102. The GUI 400 includes a field 402, which includes a new entity button 404, wherein in response to selecting the new entity button 404, the creation of a new entity to be considered by the chatbot is initiated. The field 402 also includes a text input field 406, wherein a query is conducted for an entity that may be included in the text input field 406, and further, wherein (existing) entities are searched based on such a query. The text input field 406 is particularly useful when there are many entities that can be extracted by the entity extractor module 116 (and therefore considered by the chatbot), thereby allowing the developer to relatively quickly identify an entity of interest.

[0031] GUI 400 also includes field 408 that includes identification of entities that may be extracted from user input by entity extractor module 116, and field 408 also includes parameters for such entities. Each entity in field 408 is selectable, wherein selection of an entity results in presentation of a window configured to allow editing of the entity. Figure 4 In the example shown, the chatbot considers three entities (each of type "custom"): "Toppings", "Out of Stock", and "Final". "Toppings" can be multi-valued (e.g., "pepperoni and mushrooms"), and "Final" (representing the final pizza order placed by the user) can also be multi-valued. Further, the entities "Out of Stock" and "Final" are identified as programmable because the values ​​of such entities are only included in the response of the response model 118 (not in the user input), and further, a portion of the output is populated by the code 120. For example, when the code 120 queries the inventory system, the code 120 determines that pepperoni may be out of stock at the pizza shop. Finally, the "Toppings" and "Out of Stock" entities are identified as negatable in field 408; therefore, items can be removed from the list. For example, when the $toppings list includes "mushrooms", the "Toppings" negatable indication that entering "replace mushrooms with peppers" will result in the removal of item "mushrooms" from the $toppings list (and the addition of item "peppers" to the $toppings list). The parameters "programmable," "multi-valued," and "negotiable" are exemplary in nature, as other parameters may be desired.

[0032] Now refer to Figure 5, depicts an exemplary GUI 500, wherein in response to a developer selecting a new entity button 404 in the GUI 400, the GUI renderer module 122 causes the GUI 500 to be presented. The GUI 500 includes a window 502 presented above the GUI 400, wherein the window 502 includes a drop-down menu 504. The drop-down menu 504, when selected by the developer, describes a list of predefined entity types, such that the developer can select a type for the newly created entity. In other examples, a name can be proposed for the newly created entity in the typing field 506. For example, the developer can assign the name "crust type" to the entity, and then feedback can be proposed that causes the entity extractor module 116 to identify text such as "thin crust", "pan", etc. as a "crust type" entity.

[0033] Window 502 further includes selectable buttons 508, 510, and 512, wherein the buttons are configured to receive developer input regarding whether the new entity is only programmable, multi-valued, and / or negatable, respectively. Window 502 also includes a create button 514 and a cancel button 516, wherein in response to the developer selecting the create button 514, the new entity is created, and in response to selecting the cancel button 516, the new entity is not created.

[0034] Now refer to Figure 6 , illustrates an exemplary GUI 600, wherein in response to the client computing device 102 receiving an indication that the developer has selected the action button 206, the GUI renderer module 122 causes the GUI 600 to be presented on the display 104 of the client computing device 102. The GUI 600 includes a field 602 including a new action button 604, wherein in response to selecting the new action button 604, creation of a new action (e.g., a new response) for the chatbot is initiated. The field 60 also includes a text input field 606, wherein a query is made for actions that may be included in the text input field 606, and further, wherein (existing) actions are searched based on such a query. The text input field 606 is particularly useful when there are many actions for the chatbot, thereby allowing the developer to relatively quickly identify an action of interest.

[0035] GUI 600 also includes field 608, which includes identification of actions currently executable by the chatbot, and also includes parameters for such actions. Each action represented in field 608 is selectable, wherein selection of an action results in presentation of a window configured to allow editing of the selected action. Field 608 includes columns 610, 612, 614, 616, and 618. In Figure 6In the example shown, the chatbot may perform six actions, where the actions may include responses, application programming interface (API) calls, drawing fillable cards, etc. As previously indicated, each action may correspond to an output node of the response model 118. Column 610 includes identifiers of the actions, where the identifiers may include the text of the response, the name of the API call, an identifier of the card (which may be previewed when the icon is selected), etc. For example, the first action may be a first response, and the identifier for the first action may include the text "What would you like on your pizza"; the second action may be a second response, and the identifier for the second action may include the text "You have $Toppings on your pizza"; the third action may be a third response, and the identifier for the third action may be "Do you want anything else?"; the fourth action may be an API call, and the identifier for the fourth action may include the API descriptor "FinalizeOrder"; the fifth action may be a fourth response, and the identifier for the fifth action may be "We are out of $OutStock"; and the sixth action may be the fifth response, and the identifier for the sixth action may be "Do you want $LastToppings?"

[0036] Column 612 includes identification of entities that are required for each action to be available, while column 614 includes identification of entities that do not have to be present for each action to be available. For example, the second action requires that the "Toppings" entity be present, and the "Out of Stock" entity not be present. If these conditions are not met, then the action will be disqualified. In other words, if the user has not yet provided any toppings, and there are toppings that have been identified as out of stock, then the response "You have $toppings on your pizza" is inappropriate.

[0037] Column 616 includes the identification of the entity that the chatbot expects to receive from the user after the action has been presented to the user. Referring again to the first action, the user's reply to the first action (first response) is expected to include the identification of the toppings that the user wants on his or her pizza. Finally, column 618 identifies the value of the "wait" parameter for the action, where the "wait" parameter indicates whether the chatbot should take a subsequent action without waiting for user input. For example, the first action has a wait parameter assigned to it, which indicates that after issuing the first action (first response) to the user, the chatbot will wait for user input before performing another action. In contrast, the second action does not have a wait parameter assigned to it, so the chatbot should perform another action (e.g., output another response) immediately after issuing the second response (and without waiting for the user's reply to the second response). It is to be understood that the parameters identified in columns 610, 612, 614, 616, and 618 are exemplary, as actions can have other parameters associated with them.

[0038] Reference Figure 7 , illustrates an exemplary GUI 700, wherein in response to receiving an indication that a new action button 504 has been selected, the GUI renderer module 122 causes the GUI 700 to be presented on the display 104 of the client computing device 102. The GUI 700 includes a window 702, wherein the window 702 includes a field 704 where a developer can specify a new type of action. Exemplary types include, but are not limited to, "text", "audio", "video", "card", and "API call", wherein an action of type "text" is a text response, an action of type "audio" is an audio response, an action of type "video" is a video response, an action of type "card" is a response that includes an interactive card, and an action of type "API call" is a function in the code defined by the developer, wherein the API call can execute arbitrary code and return text, a card, an image, a video, etc. - or nothing at all. Thus, while the actions described herein are illustrated as being textual in nature, other types of chatbot actions are also contemplated. Further, the field 804 can be a text entry field, a drop-down menu, etc.

[0039] Window 702 also includes a text entry field 708, wherein the developer can state text into the text entry field 708 defining the response. In another example, the text entry field 708 can have a key corresponding thereto that allows the developer to navigate to a file that will be part of the response (e.g., a video file, an image, etc.). Window 702 additionally includes a field 710 that can be populated by the developer with the identification of entities that are expected to be present in the dialog turn proposed by the user for the reply response. For example, if the response is "What toppings would you like on your pizza?", the entity expected in the dialog turn reply will be "ingredients". Window 702 additionally includes a required entity field 712, wherein the developer can propose input that specifies what entities must be in memory to make the response appropriate. Moreover, window 702 includes a disqualified entity field 714, wherein the developer can propose input to such field 704 that identifies when the response is inappropriate based on the entity identification in memory. Continuing with the example presented above, if the entities "cheese" and "pepperoni" are in memory, the response "What toppings would you like on your pizza?" would be inappropriate, so the developer could place the entity "toppings" in the disqualified entity field 714. The developer can interact with the optional checkbox 716 to identify whether to receive user input after submitting the response, or whether another action can be taken immediately after the response. In the example presented above, the developer would decide to select the checkbox 716 because a dialog turn from the user is expected.

[0040] Window 702 also includes a create button 718, a cancel button 720, and an add entity button 722. The create button 718 is selected when the new action is completed, and the cancel button 720 is selected when the creation of the new action is to be canceled. The new entity button 722 is selected when the developer decides to create a new entity that the action depends on to some extent. The updater module 124 updates the response model 118 in response to selecting the create button 718, so that the output nodes of the response model 118 are unmasked and assigned to the newly created action.

[0041] Now refer to Fig. 8A , illustrates yet another exemplary GUI 800, wherein in response to a developer indicating that the developer wants to create a new action, and further in response to the developer indicating that the action will include presenting a template (e.g., a card) to an end user, the GUI renderer module 122 causes the GUI 800 to be presented. The GUI 800 includes a window 702, wherein the window includes fields 704, 712, and 714, a check box 716, and buttons 718, 720, and 722. In field 704, the developer has indicated that the action type is "card," which results in a template field 802 being included in window 702. For example, the template field 802 may be or include a drop-down menu that, when selected by the developer, identifies available templates for a card. In Fig. 8A In the example shown, the template selected by the developer is a delivery address template. In response to the delivery address template being selected, the GUI renderer module 122 causes a preview 804 of the delivery address template to be shown on the display, wherein the preview 804 includes a street field 806, a city field 808, a country field 810, and a submit button 812.

[0042] Go to Figure 8B , illustrates yet another exemplary GUI 850, wherein in response to the client computing device 102 receiving an indication that the developer has selected the new action button 604, and further in response to the developer indicating that the new action will be an API call, the GUI renderer module 122 causes the GUI 850 to be presented on the display 104 of the client computing device 102. Specifically, in the exemplary GUI 850, the developer has selected the action type "API call" in the field 704. In response to selecting the action type "API call", fields 852 and 854 can be presented. Field 852 is configured to receive an identification of an API call. For example, field 852 can include a drop-down menu that, when selected, presents a list of available API calls.

[0043] GUI 850 additionally includes field 854, which is configured to receive parameters that the selected API call is expected to receive. In the pizza ordering example presented herein, the parameters may include an "ingredients" entity. In a non-limiting example, GUI 850 may include multiple fields configured to receive parameters, wherein each of the multiple fields is configured to receive a specific type of parameter (e.g., "ingredients," "crust type," etc.). Although the examples provided above indicate that the parameters are entities, it is to be understood that the parameters may be any suitable parameters, including text, numbers, etc. GUI 850 also includes fields 710, 712, and 714, which are respectively configured to receive an expected entity in a user response to an action, a required entity for the action to be performed (API call), and a disqualification entity for the action.

[0044] Now refer to Fig. 9 , illustrates an exemplary GUI 900, wherein in response to the client computing device 102 receiving an indication that the developer has selected the training dialog button 208, the GUI renderer module 122 causes the GUI 900 to be presented on the display 104 of the client computing device 102. The GUI 900 includes a field 902, the field 902 includes a new training dialog button 904, wherein in response to selecting the new training dialog button 904, the creation of a new training dialog between the developer and the chatbot is initiated. The field 902 also includes a search field 906, wherein a query is performed for training dialogs that may be included in the search field 906, and further, wherein (existing) training dialogs are searched based on such a query. The search field 906 is particularly useful when many training dialogs already exist, thereby allowing the developer to relatively quickly identify one or more training dialogs of interest. The field 902 also includes an entity filter field 908 and an action filter field 910, which allow filtering of existing training dialogs based on entities referenced in the training dialog and / or actions performed in the training dialog. Such fields may be text entry fields, drop-down menus, etc.

[0045] GUI 900 also includes field 912, which includes several rows for existing training dialogues, wherein each row corresponds to a respective training dialogue, and further, wherein each row includes: an identity of a first input from the developer to the chatbot; an identity of a final input from the developer to the chatbot, an identity of the developer's final input to the chatbot, an identity of the chatbot's final response to the developer, and the number of "turns" in the training dialogue (the total number of dialogue turns between the developer and the chatbot, where a dialogue turn is part of a dialogue). Thus, "Input 1" could be "I'm hungry," "Final 1" could be "No, thank you," and "Response 1" could be "Your order is complete." It will be understood that the information in the rows is presented to help the developer distinguish between various training dialogues and find a desired training dialogue, and that any suitable type of information that may assist the developer in performing such a task is contemplated. Fig. 9 In the example shown, the developer has selected the first training session.

[0046] Now go to Fig.10 , depicts an exemplary GUI 1000, wherein in response to the client computing device 102 receiving a request from a developer that the developer has Fig. 9Indicating that the first training dialog is selected in the illustrated field 912, the GUI renderer module 122 causes the GUI 1000 to be presented on the display 104 of the client computing device 102. The GUI 1000 includes a first field 1002 that includes a dialog between a developer and a chatbot, wherein instances of the dialog proposed by the developer are offset to the right side of the field 1002, and instances of the dialog proposed by the chatbot are offset to the left side of the field 1002. An instance of a dialog is an independent portion of a dialog that is presented from a user to a chatbot or from a chatbot to a user. Each instance of the dialog proposed by the chatbot is selectable so that the actions (e.g., responses) performed by the chatbot can be modified for us when retraining the response model 118. In addition, each instance of the dialog proposed by the developer is also selectable so that the input provided to the chatbot can be modified (and the actions of the chatbot can be observed based on the modification of the dialog instance). The GUI 1000 further includes a second field 1004, wherein the second field 1004 includes a branch button 1006, a delete button 1008, and a completion button 1010. When the branch button 1006 is selected, the GUI 1000 is updated to allow the developer to fork the current training dialog and create a new dialog - for example, in a dialog with ten different dialog turns, the developer can select the fifth dialog turn in the dialog (wherein the user participating in the dialog says "yes"; the developer can branch on the fifth dialog turn and propose "no" instead of "yes", thereby creating a new training dialog with five dialog turns, the first four dialog turns being the same as the original training dialog, but the fifth dialog turn being "no" instead of "yes"). When the delete button 1008 is selected (and optionally, the deletion is confirmed by a modal dialog box), the updater module 124 deletes the training dialog so that future outputs of the entity extractor module 116 and / or the response model 118 are not functions of the training dialog. In addition, the GUI 1000 can be updated in response to the user selecting a dialog turn in the field 1002, wherein the updated GUI can facilitate inserting or deleting dialog turns in the training dialog. When the finish button 1010 is selected, the GUI 900 can be presented on the display 104 of the client computing device 102.

[0047] Now refer to Fig.11 , an exemplary GUI 1100 is shown, wherein the GUI presenter module 122 presents the GUI 1100 in response to the developer selecting the new training dialog button 604 in the GUI 600. The GUI 1100 includes a first field 1102, which depicts a chat dialog between the developer and the chatbot. The first field 1102 also includes a text entry field 1104, wherein the developer can state text in the text entry field 1104 to provide to the chatbot.

[0048] The GUI 1100 also includes a second field 1106, wherein the second field 1106 depicts information about an entity identified in a dialog turn stated by the developer (in this example, the dialog turn "I want pizza with cheese and mushrooms"). The second field 1106 includes an area depicting an identification of an entity that already exists in the chatbot's memory; Fig.11 In the example shown, there are currently no entities in memory. The second field 1106 also includes a field 1108, which depicts the most recent conversation entry made by the developer, and further, wherein the entities identified by the entity extractor module 116 (in the conversation turn) are highlighted. Fig.11 In the example shown, the entities "cheese" and "mushroom" are highlighted, indicating that the entity extractor module 116 has identified "cheese" and "mushroom" as "ingredient" entities (with respect to the following Fig.12 1108) presents additional details on how to display entity labels. These entities are selectable in the GUI 1100, so that the developer can notify the entity extractor module 116 of incorrect entity identifications and / or correct entity identifications. Further, the developer can select other text in field 1108 - for example, the developer can select the text "pizza" and indicate that the entity extractor module 116 should have identified the text "pizza" as the "ingredients" entity (although this would be incorrect). Entity values ​​can span multiple consecutive terms, so "pepperoni" can be labeled as a single entity value.

[0049] The second field 1106 also includes a field 1110, where the developer can propose (multiple) alternative inputs to the field 1110 that are semantically equivalent to the dialogue turn shown in the field 1108. For example, the developer can put "cheese and mushrooms on my pizza" in the field 1110, thereby providing the updater module 124 with additional training examples for the entity extractor module 116 and / or the response model 118.

[0050] The second field 1106 additionally includes an undo button 1112, a discard button 1114, and a finish button 1116. When the undo button 1112 is selected, the information stated in the field 1108 is deleted and a "rollback" is taken. When the discard button 1114 is selected, the training session is abandoned and the updater module 124 does not receive information related to the training session. When the finish button 1116 is selected, all information proposed by the developer in the training session is provided to the updater module 124, which then updates the entity extractor module 116 and / or the response model 118 based on the training session.

[0051] The second field 1106 also includes a score action button 1118. When the score action button 1118 is selected, the entity identified by the entity extractor module 116 can be placed in memory, and the response model 118 can be provided with the dialogue turn and the entity. The response model 118 then generates an output based on the entity and the dialogue turn (and optionally, the previous dialogue turn in the training dialogue), where the output can include the probability of an action supported by the chatbot (wherein the output node of the response model 118 represents the action).

[0052] GUI 1100 may optionally include an interactive graphical feature that, when selected, causes presentation of a graphical user interface associated with the Figure 3 , wherein the GUI includes code associated with the identified entity. For example, the code may be configured to determine whether an ingredient is in stock or out of stock, and may be further configured to move an ingredient from in stock to out of stock (or vice versa). Thus, detection of a certain type of entity in a dialog turn may result in a call to the code, and a GUI including such code may be presented (where the developer may edit the code).

[0053] Now go to Fig.12 , shows an exemplary GUI 1200, wherein, in response to a developer in Fig.11 The entity "cheese" is selected in the depicted field 1108, and the GUI renderer module 122 presents the GUI 1200. In response to selecting such an entity, an optional graphical element 1202 is presented, wherein, in response to the graphical element 1202 being selected, feedback is provided to the updater module 124. In the example, the selection of the graphical element 1202 indicates that the entity extractor module 116 should not identify the selected text as an entity. The updater module 124 receives such feedback and updates the entity extractor module 116 based on the feedback. In the example, the updater module 124 receives the feedback in response to the developer selecting a button such as the score action button 1118 or the completion button 1116 in the GUI 1200.

[0054] Fig.12Another exemplary GUI feature is illustrated in which a developer can define classifications to assign to entities. More specifically, in response to the developer selecting the entity "mushroom" using some selection input (e.g., right-clicking the mouse when the cursor is over "mushroom", maintaining contact with the text "mushroom" on a touch-sensitive display using a finger or stylus, etc.), an interactive graphical element 1204 can be presented. The interactive graphical element 1204 can be a drop-down menu, a pop-up window including selectable items, and the like. For example, the entity can be an "ingredients" entity or a "crust type" entity, and the interactive graphical element 1204 is configured to receive input from the developer so that the developer can change or define the entity of the selected text. Furthermore, although not described in Fig.12 , but graphics can be associated with the identified entities to indicate to the developer the classification of the entity (e.g., "ingredients" versus "crust type"). These graphics can include text, assign colors to the text, etc.

[0055] Reference Fig.13 , illustrates an exemplary GUI 1300, which is presented by the GUI presenter module 122 in response to the developer selecting the score action button 1118 in the GUI 1100. The GUI 1300 includes a field 1302 that depicts the identification of entities in the chatbot's memory (e.g., cheese and mushrooms as identified by the entity extractor module 116 from the conversation turn shown in field 1102). Field 1302 also includes the identification of the chatbot's action and the score output by the response model 118 for such action. Fig.13 In the example shown, there may be three actions: action 1, action 2, and action 3. Actions 4 and 5 are disqualified because the entities currently in the memory prevent such actions from being taken. In a non-limiting example, action 1 can be the response "You have $Toppings on your pizza," action 2 can be the response "Do you want anything else?", action 3 can be the API call "FinalizeOrder," action 4 can be the response "We are out of $OutStock," and action 5 can be the response "Do you want $LastTopping?". It can be determined that the response model 118 is configured to not output actions 4 or 5 in this scenario (e.g., these outputs of the response model 118 are obscured) because the memory includes "cheese" and "mushrooms" as entities (which belong to the entity "ingredients," not "out of stock," and thus exclude the output of action 4), and there is no "final" entity in the memory, which excludes the output of action 5.

[0056] The response model 118 has identified response 1 as the most appropriate output. Each possible action (actions 1, 2, and 3) has a selection button corresponding to it; when the developer selects the selection button corresponding to an action, the action is selected for the chatbot. Field 1302 also includes a new action button 1304. Selection of the new action button 1304 causes a window to be presented, wherein the window is configured to receive input from the developer, and further, wherein the input is used to create a new action. The updater module 124 receives an indication to create a new action and updates the response model 118 to support the new action. In an example, when the response model 118 is an ANN, the updater module 124 assigns the output nodes of the ANN to the new action and updates the weights of the synapses of the network based on the feedback from the developer. The "Select" button corresponding to a disqualified action cannot be selected, such as Fig.13 As shown by the dotted line in .

[0057] Now refer to Fig.14 , illustrates an exemplary GUI 1400, wherein the GUI renderer module 122 renders the GUI 1400 in response to the developer selecting a "select" button corresponding to the first action (the most appropriate action identified by the response model 118) in the GUI 1200. In an exemplary embodiment, the updater module 124 immediately updates the response model 118 in response to the select button being selected, wherein updating the response model 118 includes updating the weights of the synapses based on selecting action 1 as the correct action. Field 1102 is updated to reflect that the first action is performed by the chatbot. Because the first action does not require the chatbot to wait for further user input before the chatbot performs another action, field 1302 is further updated to identify the action that the chatbot can take next (and its associated appropriateness). As depicted in field 1302, the response model 118 has identified the most appropriate action (based on the state of the dialog and the entities in memory) as action 2 (responding to "Do you want anything else?"), action 1 and action 3 are the next most appropriate outputs, respectively, and actions 4 and 5 are disqualified due to the entities in memory. As before, the second field 1302 includes "Select" buttons corresponding to actions, wherein the "Select" button corresponding to the disqualifying action cannot be selected.

[0058] Now go to Fig.15 , illustrates yet another exemplary GUI 1500, wherein in response to the developer selecting the second action (the most appropriate action identified by the response model 118) in the GUI 1400, and further in response to the developer stating the dialog turn "remove mushrooms and add peppers" into the text entry field 1104, the GUI renderer module 122 causes the GUI 1500 to be presented. Fig.11As indicated, action 2 indicates that after providing this response to the user in field 1102, the chatbot will wait for user input; in this example, the developer has presented the aforementioned input to the chatbot.

[0059] GUI 1500 includes field 1106 indicating that prior to receiving such input, entity memory included the "ingredient" entities "mushroom" and "cheese". Field 1108 includes text proposed by the developer, wherein the text "mushroom" and "pepper" are highlighted to indicate that entity extractor module 116 has identified such text as entities. Graphical features 1502 and 1504 are graphically associated with the text "mushroom" and "pepper", respectively, to indicate that entity "mushroom" will be removed from memory as an "ingredient" entity, while entity "pepper" will be added to memory as an "ingredient" entity. Graphical features 1502 and 1504 are optional, allowing the developer to change what has been identified by entity extractor module 116. Once the developer makes changes in field 1106, and in response to score action button 1118 being selected, updater module 124 updates entity extractor module 116 based on the developer feedback.

[0060] Now refer to Fig.16 , depicts an exemplary GUI 1600, wherein the GUI renderer module 122 causes the GUI 1600 to be presented in response to the developer selecting the score action button 818 in the GUI 1200. Similar to the GUIs 1300 and 1400, field 1302 identifies actions that the chatbot can perform, and the associated appropriateness of such actions as determined by the response model 118, and further identifies actions that are disqualified due to entities currently in memory. Note that the entity has been updated to reflect that "mushroom" has been removed from memory (illustrated as a strikethrough), while the entity has been updated to reflect that "pepper" has been added to memory (illustrated by bolding or highlighting such text). The text "cheese" remains unchanged.

[0061] Now refer to Fig.17 , illustrating an exemplary GUI 1700. GUI 1700 depicts a scenario in which a developer may be expected to create a new action because the chatbot may lack an appropriate action for the most recent conversation turn that the developer has proposed to the chatbot. Specifically, in response to the chatbot indicating that the order has been completed, the developer has proposed the conversation turn as "OK!". The response model 118 indicates that action 5 (response to "Do you want $Lasttopping?") is the most appropriate action of all actions that the response model 118 can perform; however, it can be determined that the action seems unnatural in light of the rest of the conversation. Therefore, the GUI renderer module 122 receives an indication that the new action button 1004 has been selected at the client computing device 102.

[0062] Now refer to Fig.18 , illustrates an exemplary GUI 1800, wherein in response to the client computing device 102 receiving an indication that the developer has selected the log conversation button 210, the GUI renderer module 122 causes the GUI 1800 to be presented on the display 104 of the client computing device 102. The GUI 1800 is similar to the GUI 900, which depicts a list of selectable training conversations. The GUI 1800 includes a field 1802, which includes a new log conversation button 1804, wherein in response to selecting the new log conversation button 1804, the creation of a new log conversation between the developer and the chatbot is initiated. The field 1802 also includes a search field 1806, wherein a query is performed for log conversations that can be included in the search field 1806, and further, wherein (existing) log conversations are searched based on such a query. The search field 1806 is particularly useful when many log conversations already exist, thereby allowing the developer to relatively quickly identify one or more log conversations of interest. Field 1802 also includes an entity filter field 1808 and an action filter field 1810, which allow filtering of existing log conversations based on entities referenced in the log conversation and / or actions performed in the log conversation. Such fields can be text entry fields, drop-down menus, etc.

[0063] GUI 1800 also includes field 1812, which includes several rows for existing log dialogues, wherein each row corresponds to a respective log dialogue, and further, wherein each row includes: an identification code of the first input from the end user (who may or may not be a developer) to the chatbot; an identification code of the final input from the end user to the chatbot, an identification code of the final response of the chatbot to the end user, and a total number of dialogue turns between the end user and the chatbot. It is to be understood that the information in the rows is stated to help the developer distinguish between the various log dialogues and find the desired log dialogue, and any suitable type of information that can help the developer perform such a task is contemplated.

[0064] Now refer to Fig.19, illustrates an exemplary GUI 1900, wherein in response to the client computing device 102 receiving an indication from the developer that the developer has selected a new log conversation button 904, the GUI renderer module 122 causes the exemplary GUI 1900 to be shown on the display 104 of the client computing device 102, and further interacts with the chatbot. The GUI 1900 is similar or identical to a GUI that may be presented to an end user interacting with a chatbot. The GUI 1900 includes a field 1902, which depicts a log conversation that the developer is creating. The exemplary log conversation depicts conversation turns exchanged between the developer (conversation proposed by the developer is biased to the right) and the chatbot (conversation turns output by the chatbot are biased to the left). In the example, the field 1902 includes an input field 1904, wherein the input field 1904 is configured to receive a new conversation turn from the developer to continue the log conversation. The field 1902 also includes a finish button 1908, wherein selection of the finish button causes the log conversation to be retained (but removed from the GUI 1900).

[0065] Now refer to Fig. 20 , illustrates an exemplary GUI 2000, wherein in response to a developer selecting a log dialogue from a list of selectable log dialogues (e.g., a fourth log dialogue in the list of log dialogues), the GUI presenter module 122 causes the GUI 2000 to be presented on the display 104 of the client computing device 102. The GUI 2000 is configured to allow conversion of the log dialogue to a training dialogue, which the updater module 124 can use to retrain the entity extractor module 116 and / or the response model 118. The GUI 2000 includes a field 2002 that is configured to display the selected log dialogue. For example, the developer can review the log dialogue in the field 2002 and determine that the chatbot did not correctly respond to the dialogue turn from the end user. Additionally, the field 2004 can include a text entry field 2003, in which the developer can state text input to continue the dialogue.

[0066] In the example, the developer may select a conversation turn in field 2002 in which the chatbot stated an incorrect response (e.g., "I can't help you."). Selecting such a conversation turn causes field 2004 in the GUI to be populated with an action that may be output by the response model 118, which is arranged by the calculated appropriateness. As previously described, the developer may specify appropriate actions for the chatbot to perform, create new operations, etc., thereby converting the log conversation into a training conversation. Further, field 2004 may include a "Save as Log" button 2006 - when the developer has not yet proposed any updated actions and desires to convert the log conversation "as is" into a training conversation, button 2006 may be active. The updater module 124 may then update the entity extractor module 116 and / or the response model 118 based on the newly created training conversation. These features allow developers to generate training conversations in a relatively short period of time, because the log conversation can be viewed at any suitable point in the log conversation and converted into a training conversation.

[0067] Additionally, in an example, a developer may decide to edit or delete an action, resulting in a situation where the chatbot is no longer able to perform the action in certain situations where it was previously possible, or is no longer able to perform the action at all. In such an example, it may be determined that the training dialog may be affected; i.e., the training dialog may include an action that is no longer supported by the chatbot (because the developer deleted the action), and the training dialog is therefore out of date. Fig.21 An exemplary GUI 2100 is illustrated, which may be displayed on the display 104 of the client computing device 102 by the GUI renderer module 122, wherein the GUI 2100 is configured to highlight training dialogs that rely on outdated actions. In the exemplary GUI 2100, the first and second training dialogs are highlighted, thereby indicating to the developer that the training dialogs refer to at least one action that is no longer supported by the response model 118 or is no longer supported by the response model 118 in the context of the training dialogs. Thus, the developer can quickly identify which training dialogs must be deleted and / or updated.

[0068] Now refer to Fig. 22, illustrates an exemplary GUI 2200, wherein in response to the client computing device 102 receiving an indication that the developer has selected one of the training dialogs highlighted in the GUI 2100, the GUI renderer module 122 causes the GUI 2200 to be presented on the display 104 of the client computing device 102. In field 1302, an error message is shown, indicating that the response model 118 no longer supports the action previously authorized by the developer. In response to the client computing device 102 receiving an indication that the developer has selected the error message (as indicated by the bold font of the error message), the field 1302 is populated with the available actions currently supported by the response model 118. Further, the disqualified action has an optional "select" button corresponding thereto. Further, the field 1002 includes a new action button 1304. When such button 1304 is selected, the GUI renderer module 122 can cause the GUI 700 to be presented on the display 104 of the client computing device 102.

[0069] Briefly refer to Fig.23 , illustrates an exemplary GUI 2300, wherein in response to a developer creating an action (as depicted in FIG. 8 ) and selecting the action as a correct response, the GUI renderer module 122 causes the GUI 2300 to be presented on the display 104 of the client computing device 102. A delivery address template 2302 is shown on the display, wherein the template 2302 includes a field for entering an address (e.g., where to deliver the pizza), and further, wherein the template 2302 includes a submit button. When the submit button is selected, the contents of the field can be provided to a backend reservation system.

[0070] Figure 24 to Figure 26 An exemplary method related to creating and / or updating a chatbot is illustrated. Although the method is shown and described as a series of actions performed in sequence, it is to be understood and appreciated that the method is not limited by the order of the sequence. For example, some actions may occur in an order different from that described herein. In addition, an action may occur concurrently with another action. Further, in some cases, not all actions may be required to implement the method described herein.

[0071] Furthermore, the actions described herein may be computer-executable instructions that may be implemented by one or more processors and / or stored on one or more computer-readable media. Computer-executable instructions may include routines, subroutines, programs, execution threads, etc. Further, the results of the actions of the methods may be stored in a computer-readable medium, displayed on a display device, etc.

[0072] Now refer to Fig.24, illustrates an exemplary method 2400 for updating a chatbot, wherein the server computing device 106 may perform the method 2400. The method 2400 begins at 2402, and at 2404, an indication is received that a developer wants to create and / or update a chatbot. For example, the indication may be received from a client computing device being operated by the developer. At 2406, a GUI feature is caused to be displayed at the client computing device, wherein the GUI feature includes a dialog between the chatbot and a user. At 2408, a selection of a dialog turn is received from the client computing device, wherein the dialog turn is part of a dialog between the chatbot and the end user. At 2410, in response to receiving the selection of the dialog turn, an updated GUI feature is caused to be displayed at the client computing device, wherein the updated GUI feature includes a selectable feature. At 2412, an indication is received that an optional feature in the selectable features has been selected, and at 2414, in response to receiving the indication, at least one of the entity extractor module or the response model is updated. The method 2400 is completed at 2416.

[0073] Now refer to Fig.25 , illustrates an exemplary method 2500 for updating a chatbot, wherein a client computing device 102 may perform the method 2500. The method 2500 begins at 2502, and at 2504, a GUI is presented on a display 104 of the client computing device 102, wherein the GUI includes a dialog, and further, wherein the dialog includes selectable dialog turns. At 2506, a selection of a dialog turn among the selectable dialog turns is received, and at 2508, an indication of the selected dialog turn is transmitted to the server computing device 106. At 2510, based on feedback from the server computing device 106, a second GUI is presented on the display 104 of the client computing device, wherein the second GUI includes a plurality of potential responses of the chatbot to the selected dialog turn. At 2512, a selection of a response from the potential responses is received, and at 2514, an indication of the selected potential response is transmitted to the server computing device 106. The server computing device updates the chatbot based on the selected response. The method 2500 is completed at 2516.

[0074] Now refer to Fig.26, illustrating an exemplary method 2600 for updating entity extraction labels within a conversation turn. The method 2600 begins at 2602, and at 2604, a conversation between an end user and a chatbot is presented on a display of a client computing device, wherein the conversation includes a plurality of selectable conversation turns (some of which are proposed by the end user and some of which are proposed by the chatbot). At 2606, an indication is received that a developer has selected a selectable conversation turn proposed by the end user. At 2608, in response to the indication being received, an interactive graphical feature is presented on the display of the client computing device, wherein the interactive graphical feature is presented with respect to at least one term in the selected conversation turn. The interactive graphical feature indicates that an entity extraction label has been assigned to the at least one term (or indicates that an entity extraction label has not been assigned to the at least one term). At 2610, an indication is received that the developer has interacted with the interactive graphical feature, wherein the entity extraction label assigned to the at least one term is updated based on the developer interacting with the interactive graphical feature (or the entity extraction label is assigned to the at least one term based on the developer interacting with the interactive graphical feature). The method 2600 is completed at 2612.

[0075] Now go to Fig. 27 , illustrates a high-level diagram of an exemplary computing system 2700 that can be used according to the systems and methods disclosed herein. For example, computing device 2700 can be used in a system configured to create and / or update a chatbot. By way of another example, computing device 2700 can be used in a system that causes certain GUI features to be presented on a display. Computing device 2700 includes at least one processor 2702 that executes instructions stored in memory 2704. For example, the instructions can be instructions for implementing functionality described as being performed by one or more components discussed above or instructions for implementing one or more methods described above. Processor 2702 can access memory 2704 via system bus 2706. In addition to storing executable instructions, memory 2704 can also store response models, model weights, and the like.

[0076] The computing device 2700 additionally includes a data storage device 2708 accessible to the processor 2708 via the system bus 2706. The data storage device 2708 may include executable instructions, model weights, etc. The computing device 2700 also includes an input interface 2710 that allows external devices to communicate with the computing device 2700. For example, the input interface 2710 can be used to receive instructions from an external computer device, from a user, etc. The computing device 2700 also includes an output interface 2712 that enables the computing device 2700 to interface with one or more external devices. For example, the computing device 2700 can display text, images, etc. through the output interface 2712.

[0077] It is contemplated that external devices communicating with the computing device 2700 via the input interface 2710 and the output interface 2712 may be included in an environment that provides substantially any type of user interface with which a user may interact. Examples of user interface types include graphical user interfaces, natural user interfaces, and the like. For example, a graphical user interface may accept input from a user employing (multiple) input devices (such as a keyboard, a mouse, a remote control, and the like) and provide output on an output device (such as a display). Further, a natural user interface may enable a user to interact with the computing device 2700 in a manner that is not constrained by the input devices (such as a keyboard, a mouse, a remote control, and the like). In contrast, a natural user interface may rely on speech recognition, touch and stylus recognition, gesture recognition on and near the screen, air gestures, head and eye tracking, sound and speech, vision, touch, gestures, machine intelligence, and the like.

[0078] Additionally, although illustrated as a single system, it is to be understood that the computing device 2700 may be a distributed system. Thus, for example, multiple devices may communicate via a network connection and may jointly perform the tasks described as being performed by the computing device 2700.

[0079] The various functions described herein can be implemented in hardware, software, or any combination thereof. If implemented in software, the function can be stored on a computer-readable medium or transmitted via a computer-readable medium as one or more instructions or codes. Computer-readable media include computer-readable storage media. Computer-readable storage media can be any available storage medium that can be accessed by a computer. For example, but not limited to, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM, or other optical disk storage, disk storage, or other magnetic storage devices, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer. As used herein, disks and optical disks include: compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and blue-ray disks (BDs), wherein disks typically copy data magnetically, and optical disks typically copy data optically using lasers. Further, the propagated signal is not included in the scope of computer-readable storage media. Computer-readable media also include communication media, which include any media that facilitates the transfer of computer programs from one place to another. For example, a connection can be a communication medium. For example, if the software is transmitted from a website, server or other remote source using coaxial cable, optical cable, twisted pair, digital subscriber line (DSL), or wireless technologies (such as infrared, radio and microwave), then coaxial cable, optical cable, twisted pair, DSL, or wireless technologies (such as infrared, radio and microwave) are included in the definition of communication media. Combinations of the above should also be included within the scope of computer-readable media.

[0080] Alternatively or in addition, the functionality described herein may be at least partially performed by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0081] What has been described above includes examples of one or more embodiments. Of course, for the purposes of describing the aforementioned aspects, it is not possible to describe every conceivable modification and alteration of the above-described apparatus or method, but one of ordinary skill in the art will recognize that many further modifications and permutations of various aspects are possible. Therefore, the described aspects are intended to encompass all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims. Furthermore, to the extent that the term "includes" is used in the detailed description or in the claims, such term is intended to be encompassed in a manner similar to the term "comprising," as "comprising" is to be interpreted when employed as a transitional term in a claim.

Claims

1. A method for updating a chatbot, the method comprising: receiving an indication from a client computing device that the chatbot is to be updated; in response to receiving the indication that the chatbot is to be updated, causing a graphical user interface (GUI) feature to be presented on a display of the client computing device, the GUI feature including a selectable dialog turn pertaining to a dialog including a dialog turn proposed by the chatbot; receiving an indication that the selectable conversation turn has been selected by a user of the client computing device; as well as Based on the indication that the selectable dialog turn has been selected by the user of the client computing device, updating the chatbot, wherein future dialog turns output by the chatbot in response to user input are functions of the update to the chatbot.

2. The method of claim 1 , wherein the chatbot comprises an entity extractor module configured to identify entities in input to the chatbot, and wherein updating the chatbot comprises updating the entity extractor module.

3. The method of claim 1 , wherein the chatbot comprises an artificial neural network configured to select a response of the chatbot to a user input, and further wherein updating the chatbot comprises updating the artificial neural network.

4. The method of claim 3, wherein updating the artificial neural network comprises: Weights assigned to synapses of the artificial neural network are updated.

5. The method of claim 3, wherein updating the artificial neural network comprises assigning new responses to output nodes of the artificial neural network.

6. The method of claim 3, wherein updating the artificial neural network comprises masking output nodes of the artificial neural network.

7. The method according to claim 1, wherein the conversation turn has been proposed by the chatbot, the method further comprising: responsive to receiving the indication that the selectable conversation turn has been selected by the user of the client computing device, causing a second GUI feature to be presented on the display of the client computing device, the second GUI feature comprising selectable potential outputs of the chatbot to the selected conversation turn; as well as An indication is received that an output from the selectable potential outputs has been selected by the user of the client computing device, wherein the chatbot is updated based on the selected output.

8. The method of claim 1, wherein the conversation turn has been initiated by the user of the client computing device, the method further comprising: in response to receiving the indication that the selectable dialog turn has been selected by the user of the client computing device, causing a second GUI feature to be presented on the display of the client computing device, wherein the second GUI feature includes a proposed entity extracted by the chatbot from the dialog turn; as well as An indication is received that the proposed entity was inappropriately extracted from the conversation turn by the chatbot, wherein the chatbot is updated based on the indication that the proposed entity was inappropriately extracted from the conversation turn by the chatbot.

9. The method of claim 1, wherein the conversation was previously conducted between the chatbot and an end user.

10. A server computing device, the server computing device comprising: processor; as well as A memory storing instructions that, when executed by the processor, cause the processor to perform actions including: receiving an indication from a client computing device that a chatbot is to be updated, the client computing device being in network communication with the server computing device; displaying a conversation between the chatbot and an end user of the chatbot based on the indication that the chatbot is to be updated, wherein the conversation includes conversation turns stated by the end user to the chatbot, wherein the conversation includes a first set of conversation turns between the chatbot and the end user, the first set of conversation turns occurring before the conversation turns, wherein the conversation includes a second set of conversation turns between the chatbot and the end user, the second set of conversation turns occurring after the conversation turns, and further wherein the chatbot has a state at the conversation turns; receiving an indication that a developer of the chatbot has selected the conversation turn; after receiving the indication that the developer of the chatbot has selected the dialog turns, receiving an indication that an updated dialog is to be generated, wherein the updated dialog is to include the first set of dialog turns and is to include the second set of dialog turns; Based on the updated dialog the indication will be generated: removing the second group of dialogues from the dialogues so that the chatbot has the state; as well as presenting a text entry field on a display; receiving, after presenting the text entry field on the display, a new conversation turn from the developer through the text entry field to add to the conversation, the new conversation turn immediately following the conversation turn selected by the developer; Based on the new conversation turn, causing a potential response of the chatbot to the new conversation turn to be displayed on a display of the client computing device; receiving, from the developer, an indication that a potential response from among the potential responses is a desired response to the new dialog turn; as well as Based on the indication that the potential response is the desired response to the new conversation turn, the chatbot is updated, wherein future responses output by the chatbot in response to user input are a function of the update of the chatbot.

11. The server computing device of claim 10, wherein the chatbot comprises an artificial neural network, wherein updating the chatbot comprises updating weights assigned to synapses of the artificial neural network based on the indication that the potential response is the desired response to the new conversation turn.

12. The server computing device of claim 11, wherein the artificial neural network is a recurrent neural network.

13. The server computing device of claim 10, wherein the chatbot comprises an artificial neural network, and further wherein the potential response comprises a second potential response, the actions further comprising: receiving a selection of the second potential response from the developer of the chatbot; receiving an indication from the developer that the second potential response is to be removed as a response that the chatbot is capable of outputting; Based on the indication that the second potential response is to be deleted as a response that the chatbot is capable of outputting, removing the second potential response from the artificial neural network.

14. The server computing device of claim 10, wherein the computer-implemented assistant comprises the chatbot.

15. The server computing device of claim 10, wherein the chatbot comprises an entity extraction module, wherein the new dialog turn comprises an entity, and wherein the action comprises: receiving from the developer of the chatbot that the entity was incorrectly identified by the entity extraction module; as well as Based on the indication that the entity was incorrectly identified by the entity extraction module, the entity extraction module is updated.

16. A computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform actions comprising: causing a graphical user interface (GUI) feature to be presented on a display of a client computing device, the GUI feature including a dialog between a user and a chatbot, the dialog including selectable dialog turns; receiving an indication that a conversation turn in the conversation turn has been selected at the client computing device, wherein the conversation turn is output by the chatbot; in response to receiving an indication that a conversation turn has been selected, causing a plurality of possible outputs of the chatbot to be presented on the display of the client computing device; receiving an indication that an output from the plurality of possible outputs has been selected; as well as The chatbot is updated based on the selected output of the plurality of possible outputs.

17. The computer-readable storage medium of claim 16, wherein the chatbot comprises an artificial neural network, and further wherein updating the chatbot comprises updating weights assigned to synapses of the artificial neural network.

18. The computer-readable storage medium of claim 16, wherein the dialog turn is a response to a most recent input from the user.

19. The computer-readable storage medium of claim 16, wherein the dialog turn is not a latest dialog turn output by the chatbot.

Images