Natural language responses of a machine-assisted agent

Abstract

A natural language response is selected from a set of natural language responses using a personal entity analyzer to score a personal entity difference between the natural language query and the natural language response and an edited version of the natural language response. A product entity analyzer is used to score a product entity difference between the natural language response and the edited version of the natural language response. A sentence similarity analyzer is used to score a sentence similarity between the natural language response and the edited version of the natural language response. The set of natural language responses is updated based on the personal entity difference score, the product entity difference score, and the sentence similarity score. In the natural language interaction, the natural language response selected from the set of natural language responses is output.

Description

TECHNICAL FIELD

[0001] The present invention relates generally to methods, systems, and computer program products for natural language response recommendation. More specifically, the present invention relates to methods, systems, and computer program products for natural language responses in machine-assisted agents. BACKGROUND

[0002] Natural language is a scripted (written) or vocal (spoken) language that has a form that is used primarily by humans to communicate with each other or with systems that have a natural language interface. Natural language processing (NLP) is a technology that facilitates the exchange of information between humans and data processing systems. For example, one branch of NLP involves converting human-readable or human-understandable content into machine-usable data. For example, NLP engines are currently available that take input content such as a newspaper article or human speech and produce structured data from the given content, such as an outline of the input content, most important and least important parts, topics, references, dependencies of the content, and the like.

[0003] Another branch of NLP involves answering questions about a topic based on available information about the topic area. This is a branch of cognitive analysis and is also known as a question and answer system (Q and A system). Cognitive analysis is the process of analyzing available information or knowledge to create, infer, deduce, or derive new information.

[0004] Chatbots or conversational interfaces are software that have natural language conversations with human users. Typically, the natural language conversation is in textual form. However, the input to a chatbot can also be converted from another modality (e.g., speech) to text for processing, and then the output of the chatbot is converted back to speech that a human can hear. Chatbots are typically used to assist in customer service or information acquisition.

[0005] Chatbots are typically implemented in two forms. Retrieval-based chatbots use a repository of predefined responses. Specifically, retrieval-based chatbots take as input the context (the conversation so far) and potential responses. To find a good response, the chatbot scores multiple potential responses within the repository and selects the response with the highest score. Generative chatbots can generate responses that are not in the repository of predefined responses. However, today’s generative chatbots are prone to grammatical errors, they require large amounts of training data, they produce irrelevant, generic, or inconsistent responses, and are difficult to optimize

[0006] Agent-assisted tools are a variation of chatbots. Agent-assisted tools receive natural language input and provide one or more recommended natural language responses to the input. Typically, a set of recommended natural language responses is provided in a ranked list. However, an agent performs the final step of selecting a recommended response and, if necessary, editing the response before the tool provides the response to a human user. Moreover, the edited response is not provided to the tool for improving the set of recommended natural language responses. Agent-assisted tools are typically used to assist an agent in interacting with a customer in voice or text. Illustrative embodiments envision that agent-assisted tools also become available in a humanoid form, where the interaction would be a face-to-face or human-to- humanoid interaction between a person and a humanoid.

[0007] Chatbots and agent-assisted tools are collectively referred to as conversational systems. In a conversational system, a natural language utterance provided to the system is referred to as a query. The query need not be a grammatical question, or grammatically correct, but can be any natural language word or phrase. A response is a natural language expression provided by the system in response to the query. The response also need not be a complete sentence or grammatically correct, but can be any natural language word or phrase. The query and the response to the query constitute a query-response pair. Both the query and the response can be in the form of text, voice, or natural language communication. SUMMARY

[0008] Aspects of the invention include a method that scores a personal entity difference between a natural language response to a natural language query and an edited version of the natural language response, the natural language response selected from a set of natural language response recommendations, using a personal entity analyzer that detects instances of a named entity that references a person in a portion of narrative text. Embodiments score a product entity difference between the natural language response and the edited version of the natural language response using a product entity analyzer that detects instances of a named entity that references a product in a portion of narrative text. Embodiments score a sentence similarity between the natural language response and the edited version of the natural language response using a sentence similarity analyzer that measures a similarity between two portions of narrative text. Embodiments update the set of natural language responses based on the personal entity difference score, the product entity difference score, and the sentence similarity score. Embodiments output a natural language response selected from the set of natural language responses in a natural language interaction, such that the natural language interaction uses the selected natural language response in a natural language interaction stream.

[0009] Another aspect includes a computer-usable program product. The computer-usable program product includes one or more computer-readable storage devices and program instructions stored on at least one of the one or more storage devices.

[0010] Another aspect of the present application includes a computer system. The computer system includes one or more processors, one or more computer-readable memories, and one or more computer-readable storage devices, and program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories. BRIEF DESCRIPTION OF DRAWINGS

[0011] Certain novel features believed characteristic of the application are set forth in the appended claims. The application itself, however, as well as a preferred mode of use, further objectives and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, wherein:

[0012] Figure 1 depiction of a data processing system network in which illustrative embodiments can be implemented;

[0013] Figure 2 depiction of a data processing system in which illustrative embodiments can be implemented;

[0014] Figure 3 depiction of an example configuration for natural language response;

[0015] Figure 4 depiction of an example configuration for natural language response improvement in a machine assisted agent according to illustrative embodiments;

[0016] Figure 5 depiction of an example configuration for natural language response improvement in a machine assisted agent according to illustrative embodiments;

[0017] Figure 6 depiction of an example of natural language response improvement in a machine assisted agent according to illustrative embodiments;

[0018] Figure 7 depiction of an example of natural language response improvement in a machine assisted agent according to illustrative embodiments;

[0019] Figure 8 depiction of an example of natural language response improvement in a machine assisted agent according to illustrative embodiments;

[0020] Figure 9 depiction of an example of natural language response improvement in a machine assisted agent according to illustrative embodiments;

[0021] Figure 10 depiction of an example of natural language response improvement in a machine assisted agent according to illustrative embodiments;

[0022] Figure 11A flow diagram depicting an example process for natural language response improvement in machine-assisted agents, in accordance with an illustrative embodiment, is shown in FIG. 1. DETAILED DESCRIPTION

[0023] Illustrative embodiments recognize that, in order to effectively respond to natural language queries, a retrieved agent-assisted tool must have a robust dataset of potential responses from which to choose. An agent-assisted tool can learn a set of potential responses from training on a dataset of natural language interactions (e.g., transcripts of online chats, online question-and-answer forums, transcripts of phone conversations, and other sources of natural language interaction data).

[0024] However, illustrative embodiments recognize that an agent-assisted tool trained in this way will have deficiencies. Because the tool is limited to responses present in the training data, the tool cannot adapt to new situations without additional training. For example, an agent-assisted tool for a customer service role must be updated to assist with queries about every new product and any new versions of existing products. As another example, elements of natural language such as terminology for items and expected levels of politeness can vary across geographies. For example, footwear referred to as "running shoes" or "sneakers" in the United States is commonly referred to as "trainers" in the United Kingdom. Thus, an agent-assisted tool trained using data from one geography can require additional training before being used for a different geography.

[0025] An agent-assisted tool provides recommended responses that an agent can further edit. By editing the responses before sending, an agent can update recommended responses for new products or product releases, adapt recommended responses to context that the tool can not have analyzed correctly, correct recommended responses that include grammatical errors, personalize recommended responses, and generally compensate for any deficiencies in the training or natural language understanding of the tool. However, even if an agent frequently makes similar edits to recommended responses, a pre-trained tool cannot be adapted to learn from the edits. Repeatedly making the same edits—e.g., considering new product pricing—must be performed quickly enough to keep response latency within the range of an interaction that preserves the interaction. Furthermore, making recipient-specific edits while simultaneously interacting with several customers is also error-prone, particularly if response speed must also be maintained.

[0026] Accordingly, illustrative embodiments recognize that there is an unmet need for a set of recommended responses for an agent-assisted tool that is updatable, adaptable, and can learn from edits to the set of recommended responses by an agent.

[0027] Illustrative embodiments recognize that currently available tools or solutions do not address or provide adequate solutions to these needs. Illustrative embodiments used to describe the application generally emphasize and make explicit strong distinctions over the above issues and other issues related to natural language response improvement in machine-assisted agents.

[0028] Embodiments can be implemented as software applications. Applications implementing embodiments can be configured as modifications to existing agent-assisted tools, separate applications that operate in conjunction with existing agent-assisted tools, standalone applications, or some combination thereof.

[0029] In particular, some illustrative embodiments provide a method that scores differences between a recommended response to a natural language query and an edited version of the response, and updates a stored set of recommended responses according to the scored differences.

[0030] Embodiments include a set of recommended natural language responses to a natural language query. The set of recommended responses can be extracted from a dataset of natural language interactions, machine-generated, human-generated, or generated by another method or combination of methods. Embodiments seek to improve the set of recommended responses using edits to responses by an agent while servicing a particular query.

[0031] Embodiments receive a recommended natural language response from the set of recommended responses for analysis. Embodiments also receive an edited version of the same recommended natural language response for analysis. Embodiments can also be configured to receive the natural language query that the natural language response is intended to respond to, if available, to aid in the analysis. Embodiments can also be configured to receive any previous queries and responses within the same interaction as the analyzed recommended response, if available, to aid in the analysis. Embodiments can also be configured to receive metadata about the interaction that the recommended response is a part of, if available, to aid in the analysis. The query, previous queries and responses, and metadata provide additional context for use in the analysis.

[0032] As used herein, a named entity is a real-world object that can be expressed with a proper name. Named entities also include placeholders that represent where a particular named entity can be inserted into a template. A personal entity refers to a named entity that refers to a particular person or legal entity (e.g., a company). Thus, for example, “John,” “John Doe,” “IBM Corporation,” and “{first_name}” (a placeholder in a template) are all personal entities. A product entity is a named entity that refers to a particular product, product version, or other product-related term. Thus, for example, if Product X is a product and Gold is a version of Product X, then “Product X,” “Product X Gold,” and “Gold edition” are all product entities.

[0033] As used herein, a named entity analyzer detects instances of named entities in a portion of narrative text. As such, a personal entity analyzer detects instances of personal entities in a portion of narrative text, and a product entity analyzer detects instances of product entities in a portion of narrative text. Techniques for extracting and classifying named entities, including personal entities and product entities, are currently commercially available. For example, several such techniques are the Natural Language Toolkit (NLTK), Stanford Named Entity Recognition (NER), and Watson Natural Language Understanding. (Watson is a registered trademark of International Business Machines Corporation in the United States and other countries.)

[0034] Embodiments use a personal entity analyzer to detect one or more instances of personal entities in a recommended response and in an edited recommended response. Embodiments then compare the instances of personal entities detected in the recommended response to the instances of personal entities detected in the edited recommended response to determine a number of instances of personal entities that have changed. For example, if the recommended response is “Hi, {first_name}” and the edited version is “Hi, Jenny,” “{first_name}” has changed to “Jenny,” so the edited version contains one personal entity change. As another example, an edit from “Hi, {first_name}” to “Hi, John Doe” of a recommended response results in two personal entity changes: “{first_name}” has been replaced with “John” (a first name), and “Doe” (a last name of a person) has been added. Embodiments use the number of instances of personal entities that have changed as a personal entity difference score.

[0035] Embodiments use a product entity analyzer to detect one or more instances of product entities in a recommended response and in an edited recommended response. Embodiments then compare the instances of product entities detected in the recommended response to the instances of product entities detected in the edited recommended response to determine a number of instances of product entities that have changed. For example, if the recommended response is “Product X Gold companion text analysis” and the edited version is “Product X Professional companion text analysis,” “Gold” has changed to “Professional,” so the edited version contains one product entity change. As another example, an edit from “Product X Gold costs 100 dollars” to “Product X Professional costs 200 dollars” of a recommended response results in two product entity changes: “Gold” has changed to “Professional,” and the cost has changed from 100 dollars to 200 dollars. Embodiments use the number of instances of product entities that have changed as a product entity difference score.

[0036] Embodiments use a sentence similarity analyzer to score the similarity between a recommended response and an edited recommended response. Sentence similarity determines a measure of semantic similarity of two portions of narrative text. The portions of narrative text need not be grammatical sentences, but can be any grouping of natural language text that includes at least one word. One way to compute sentence similarity is to convert the recommended response and the edited recommended response into respective numerical representations, such as vectors, and then compare the numerical representations. For example, cosine similarity measures the angle between two vectors, thereby measuring the similarity between the vectors. When the angle between two vectors is 0, the two vectors are most similar or identical, and as the angle between the two vectors increases above 0, the two vectors are less and less similar. Converting a unit of narrative text, such as a word or a sentence, into a vector representation is called a computation and embedding corresponding to the unit of narrative text. Techniques for measuring sentence similarity by computing a sentence embedding for each sentence, and then computing the similarity between these two embeddings are known. Techniques for measuring sentence similarity by computing a term embedding for each term in each sentence, combining the term embeddings for each sentence, and then computing the similarity between these two combinations of term embeddings are also known.

[0037] Embodiments classify edited recommended responses based on the individual entity difference score, the product entity difference score, and the sentence similarity score. Embodiments then update the set of recommended responses based on the embodiment classification of the type of edit performed to the edited recommended response.

[0038] Embodiments use edits to recommended responses that are to be ignored as one of the edit classifications. Embodiments consider a single individual entity change to indicate that the recommended response has been edited to incorporate personalization, such as adapting a template response to a particular named person. Because such personalization is unlikely to improve a set of recommended responses intended to apply to all customers, embodiments ignore edits in which only one individual entity is changed, i.e., the individual entity difference score is equal to one.

[0039] Embodiments use edits to recommended responses that are considered to be improvements to the recommended response as one of the edit classifications. Embodiments consider a single product entity change to indicate an improvement to the recommended response, such as adding or correcting product information. To determine whether the edit represents an addition or a correction, and to take the appropriate action for each, embodiments use the context of the recommended response.

[0040] In particular, embodiments perform product entity analysis on the query immediately prior to recommending a response, i.e., the query to which the recommended response is responsive. If the query includes the same product entity as the recommended response, but the product entity is changed to create the edited response, embodiments assume that the edited response means a feature update to the recommended response and replace the recommended response with the edited version in the set of recommended responses. For example, for the query, "Does Product X Gold have text analytics?", the recommended response can be "Product X Gold comes with text analytics" and the edited version of the recommended response can be "Product X Professional comes with text analytics." Here, because both the query and the recommended response include the same product entity ("Product X Gold"), but in the edited response, "Product X Gold" is changed to "Product X Professional," the implementation assumes that the change means a feature update, possibly because Product X Gold has been replaced by Product X Professional. As a result, the embodiment replaces the recommended response with the edited version in the set of recommended responses.

[0041] Conversely, if the query includes the same product entity as the edited response but not the original recommended response, embodiments assume that the edited response means a product update to the recommended response and add the edited version to the set of recommended responses while retaining the original recommended response. For example, for the query, "Does Product X Professional have text analytics?", the recommended response can be "Product X Gold has text analytics" and the edited version of the recommended response can be "Product X Professional has text analytics." Here, because the product entity in the query ("Product X Professional") is changed to "Product X Gold" in the recommended response and then changed back to "Product X Professional" in the edited response, the embodiment assumes that the changes mean an update to the product knowledge in the set of recommended responses to cover the features of Product X Professional. As a result, the embodiment adds the edited response to the set of recommended responses.

[0042] Embodiments use edits to a recommended response that are considered low-meaning changes to the recommended response as one of the edit classifications. Embodiments consider a low-meaning change to be a change between the recommended response and the edited response that has a sentence similarity score above a threshold similarity score (determined in the manner described herein). A non-limiting example of a threshold similarity score is 0.9, where the similarity score is determined using a 0-1 range. Embodiments assume that when two responses have a similarity score above a high threshold, the two responses have substantially the same meaning and differ only by minor grammatical or stylistic changes. An example of a minor grammatical change can be editing a recommended response, “The price of Product X is (plural)…” to “The price of Product X is (singular)…”. An example of a minor stylistic change can be editing a recommended response, “Hello, my name is…” to “Hi, my name is…”. Embodiments use grammatical parsing to determine whether a grammatical change is an improvement or a defect. Grammatical parsing identifies a portion of speech for each word in a portion of narrative text, such as a response or an edited response, and uses one or more parse trees to identify whether the text being evaluated passes a set of grammatical rules. There are several commercially available grammatical parsers, for example, the Stanford Parser by the Stanford Natural Language Processing Group. If a parser determines that one is grammatically valid and the other is not, embodiments can delete the original response or the edited version.

[0043] Embodiments also consider low-meaning changes to be changes that have a personal entity difference score above one, meaning that there is more than one personal entity difference between the recommended response and the edited response. For example, if a recommended reply is, “Hi, {first_name}, can I have your email address?” edited to “Hi, John Doe, can I have your email address?” the personal entity difference score for this change would be two because “{first_name}” is changed to “John” and “Doe” is added (an additional personal entity).

[0044] If the embodiment classifies the edit as a low meaning change, the embodiment AB tests both the recommended response and the edited version. As used herein, AB testing is an experiment in which two or more variants of a response are randomly presented to users and statistical analysis is used to determine which variant performs better for a given goal. One non-limiting example of a goal is customer satisfaction with the response or overall interaction. Another non-limiting example of a goal is the rate at which customers purchase a product after an interaction that includes a response variant. Once data is collected for a statistically significant number of interactions that include the variants, the embodiment retains the variant(s) that perform best in the set and deletes the remaining variants. In this way, the embodiment uses AB testing to determine which of a plurality of semantically similar responses performs best according to a goal and retains the best of the set of semantically similar responses.

[0045] The embodiment uses edits to the recommended response that are considered high meaning changes to the recommended response as one of the edit classifications. The embodiment considers a high meaning change to be a change in which the sentence similarity score (determined in the manner described herein) between the recommended response and the edited recommended response is below a threshold similarity score (e.g., 0.9 using a 0-1 range). The embodiment assumes that when two responses have a similarity score below the threshold, the two responses do not have substantially the same meaning. For example, editing the recommended response “The best feature of Product X is its text analysis capabilities” to “The best feature of Product X is its numerical analysis package” is a non-limiting example of a high meaning change because “text analysis capabilities” is not more than a threshold semantic similarity to “numerical analysis package.” Another non-limiting example of a high meaning change is editing the recommended response “The price of Product X is $100” to “The price of Product X is $100. And the main feature is…” because the edited response includes an additional sentence detailing feature information.

[0046] The embodiment also considers a high meaning change to be a change in which the product entity difference score is above one, meaning that there is more than one product entity difference between the recommended response and the edited response. For example, if the recommended response “The price of Product X Gold is $100” is edited to “The price of Product X Professional is $200,” the product entity difference score for this change would be two because “Product X Gold” is changed to “Product X Professional” and the price is changed.

[0047] If the embodiment classifies the edit as a high meaning change, the embodiment retains the recommended response and adds the edited version to the set of responses recommended. Retaining both versions can keep the set of recommended responses up to date because information within the responses, such as product versions, features, and pricing, change over time.

[0048] Embodiments can be configured to include additional or fewer editorial categories. Embodiments can also be configured to perform different or additional actions in response to the editorial categories. Embodiments can be configured to implement changes to the set of recommended responses based on specific geographic or non-geographic limitations. Limiting changes to specific geographies ensures that geography-specific updates are implemented only for the geography to which they apply. A non-limiting example of a geography-specific update is price information for a specific local currency. Another non-limiting example of a geography-specific update is implementing a customer being addressed by their first name in one region while maintaining the customer being addressed by their title and last name in another region.

[0049] Embodiments also include additional functionality. In particular, embodiments determine a response timing for natural language interactions. The response timing is within a range of time between an upper threshold and a lower threshold, where the thresholds are selected to avoid responses to queries that are too fast or too slow. Both too fast and too slow responses detract from the interactivity of the interaction. The embodiment forms a set of edited recommended responses from which an edited response is selected, and the selected edited response is automatically output on behalf of the agent in the interactive session - all within the selected response time.

[0050] The manner in which the agent-assisted tools described herein are unavailable in currently available methods in the technical field in which conversational systems strive. The method of the embodiments described herein, when implemented as executing on a device or data processing system, includes substantial advances in the functionality of the device or data processing system in analyzing and scoring differences between recommended responses and edited versions of responses to natural language queries, and updating stored sets of recommended responses according to the scored differences.

[0051] The illustrative embodiments are described, by way of example only, with respect to certain types of entities, analyses, similarities, differences, scores, thresholds, queries, responses, devices, data processing systems, environments, components, and applications. Any specific manifestations of these and other similar products are not intended to be limiting to the invention. Any suitable manifestation of these and other similar products can be selected in the scope of the illustrative embodiments.

[0052] Furthermore, the illustrative embodiments can be implemented with respect to any type of data, data source, or access to a data source that is in accordance with any other computer technology. In the context of the present invention, the term "data store" can refer to either a computer- readable memory device or a database or database structure located on a storage medium. In the context of the present invention, computer-readable storage media can include any medium that participates in providing data (for example, computer-readable instructions) that can be accessed by a computer. In the context of the present invention, computer-readable storage media can include storage media such as RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired data in an non-transitory fashion or that can be accessed by a computer.

[0053] The illustrative embodiments are described using specific code, designs, architectures, protocols, layouts, schematics, and tools only as examples and are not limiting to the illustrative embodiments. In particular, each of the illustrative embodiments describes an illustrative embodiment as a single implementation. However, the algorithms described in these embodiments can be implemented in a number of manners including as a regular expression, as an application specific circuit, as a firmware routine, as an object-oriented software routine, as a standalone software routine, as part of another software routine, etc. In addition, the flow diagrams and message sequence charts are generally set forth as logical flow of operations representing various actions by various entities. While these logical operations have been shown as a flow diagram, it is understood that these are merely representative of logical operations supported by the various entities and can be used to facilitate the various embodiments.

[0054] The examples in this disclosure are only used for clarity of description and are not limited to the illustrative embodiments. Additional data, operations, actions, tasks, activities, and manipulations will be conceived from this disclosure and are intended to be within the scope of the illustrative embodiments.

[0055] Any advantages listed herein are only examples and are not intended to be limiting of these illustrative embodiments. Additional or different advantages can be realized by specific illustrative embodiments. Furthermore, a particular illustrative embodiment can have some, all, or none of the advantages listed above.

[0056] With reference to the appended drawings, in particular Figure 1 and 2 these figures are example diagrams of data processing environments that can implement the illustrative embodiments. Figure 1 and 2 are examples only and are not intended to assert or imply any limitation with regard to the environments in which different embodiments can be implemented. A particular implementation can make many modifications to these depicted environments based on the following description.

[0057] Figure 1 A block diagram of a data processing system network is shown in which illustrative embodiments can be implemented. Data processing environment 100 is a network of computers in which the illustrative embodiments can be implemented. Data processing environment 100 contains network 102. Network 102 is the medium used to provide communications links between various devices and computers connected together within data processing environment 100. Network 102 can include connections, such as wire, wireless communication links, or fiber optic cables.

[0058] Clients or servers are only example roles of certain data processing systems connected to network 102 and are not intended to exclude other configurations or roles for these data processing systems. Servers 104 and 106 are coupled to storage unit 108, which provides non-volatile storage for server 104 and 106. Software applications can be executed on any computer in the data processing environment 100. Clients 110, 112, and 114 are also coupled to network 102. The data processing system, such as server 104 or 106, or clients 110, 112, or 114, can contain data and can have software applications or software tools executed thereon.

[0059] By way of example, and not by way of limitation, Figure 1 Certain components are depicted as being useful in example implementations of the embodiments. For example, servers 104 and 106 and clients 110, 112, 114 are depicted as servers and clients by way of example only and not by way of limitation to a client-server architecture. As another example, embodiments can be distributed across several data processing systems and data networks shown, while another embodiment can be implemented on a single data processing system within the scope of the illustrative embodiments. Data processing systems 104, 106, 110, 112, and 114 also represent example nodes in clusters, partitions, and other configurations suitable for implementing the embodiments.

[0060] Device 132 is an example of a device described herein. For example, device 132 can take the form of a smartphone, tablet computer, laptop computer, client 110 in a stationary or portable form, wearable computing device, or any other suitable device. Any software application described as being executed in another data processing system Figure 1 may be configured to be executed in device 132 in a similar manner. Any data or information described as being stored or produced in another data processing system Figure 1 may be configured to be stored or produced in device 132 in a similar manner.

[0061] Application 105 implements the embodiments described herein. Application 105 can be executed in any of servers 104 and 106, clients 110, 112, and 114, and device 132. A set of recommended responses can be stored in storage 108 or any other suitable location.

[0062] Servers 104 and 106, storage unit 108, and clients 110, 112, and 114 and device 132 can be coupled to network 102 using wired connections, wireless communication protocols, or other suitable data connectivity. Clients 110, 112, and 114 can be, for example, personal computers or network computers.

[0063] In the depicted example, server 104 can provide data, such as boot files, operating system images, and applications to clients 110, 112, and 114. Clients 110, 112, and 114 can be clients to server 104 in this example. Clients 110, 112, 114, or some combination thereof, can include their own data, boot files, operating system images and applications. Data processing environment 100 can include additional servers, clients, and other devices that are not shown.

[0064] In the depicted example, data processing environment 100 can be the Internet. Network 102 can represent a collection of networks and gateways that are used by the Figure 1 is intended as an example, rather than as a limitation on the architectures of different illustrative embodiments.

[0065] In other uses, data processing environment 100 can be a part of a larger system, such as a grid computing system, or utility computing system. Distributed computing techniques can be employed, where tasks or modules can be distributed across multiple processors or data processing apparatuses. In these illustrative examples, the data processing environment can also be implemented as a massively parallel computer system. Servers 104, 106, and 108, as well as clients 110, 112, and 114, can be implemented as a portion of a massively parallel computer system. In these illustrative examples, the data processing environment can be a cloud computing environment.

[0066] Referring to Figure 2 , this diagram depicts a block diagram of a data processing system that can implement the illustrative embodiments. Data processing system 200 is an example of a computer, such as server 104 and 106 or clients 110, 112, and 114 in Figure 1 , or another type of device in which computer usable program code or instructions implementing the processes can be located for the illustrative embodiments.

[0067] Data processing system 200 also represents a configuration of data processing system or a configuration therein, such as Figure 1the data processing system 132 in FIG. 1, where the computer usable program code or instructions implementing the processes of the illustrative embodiments can be located. The data processing system 200 is described in the context of what is currently considered to be a computer, without limitation. Other devices form implementations, e.g., Figure 1 The data processing system 200 can be modified, e.g., by adding a touch interface, or even removing certain described components from the data processing system 200, without departing from the general description of the data processing system 200, where the devices 132 in FIG. 1 can be located. The operation and functionality of the data processing system 200 described herein.

[0068] In the depicted example, the data processing system 200 employs a hub architecture including a north-bride and memory controller hub (NB / MCH) 202 and a south-bridge and input / output (I / O) controller hub (SB / ICH) 204. A processing unit 206, a main memory 208, and a graphics processor 210 are coupled to the north-bridge and memory controller hub (NB / MCH) 202. The processing unit 206 can contain one or more processors and can be implemented using one or more heterogeneous processor systems. The processing unit 206 can be a multi-core processor. In certain implementations, the graphics processor 210 can be coupled to the NB / MCH 202 through an accelerated graphics port (AGP).

[0069] In the depicted example, a local area network (LAN) adapter 212 is coupled to the south-bridge and I / O controller hub (SB / ICH) 204. An audio adapter 216, a keyboard and mouse adapter 220, a modem 222, a read only memory (ROM) 224, a universal serial bus (USB) and other port 232, and a PCI / PCIe device 234 are coupled to the south-bridge and I / O controller hub 204 through bus 238. Hard disk drive (HDD) or solid state drive (SSD) 226 and CD-ROM 230 are coupled to the south-bridge and I / O controller hub 204 through bus 240. The PCI / PCIe device 234 can include, for example, an Ethernet adapter, a

[0070] Memories such as main memory 208, ROM 224, or flash memory (not shown) are some examples of computer usable storage devices. Hard disk drive or solid state drive 226, CD-ROM 230, and other similarly available devices are some examples of computer usable storage devices that include computer usable storage media.

[0071] The operating system runs on the processing unit 206. The operating system coordinates and provides Figure 2 The operating system controls the various components within data processing system 200. The operating system may be a commercially available operating system for any type of computing platform, including but not limited to server systems, personal computers, and mobile devices. Object-oriented or other types of programming systems may operate in conjunction with the operating system and provide calls to the operating system from programs or applications executing on data processing system 200.

[0072] Operating systems, object-oriented programming systems, and applications or programs such as Figure 1 The instructions for the application 105 in FIG. 2 are located on a storage device (such as in the form of code 226A on hard drive 226) and may be loaded into at least one of one or more memories (such as main memory 208) for execution by processing unit 206. The processes of the exemplary embodiments may be performed by processing unit 206 using computer-implemented instructions, which may be located in a memory such as, for example, main memory 208, read-only memory 224, or one or more peripheral devices.

[0073] Furthermore, in one embodiment, code 226A may be downloaded from remote system 201B via network 201A, where similar code 201C is stored on storage device 201D. In another embodiment, code 226A may be downloaded to remote system 201B via network 201A, where the downloaded code 201C is stored on storage device 201D.

[0074] Figures 1-2 The hardware in may vary depending on the implementation. In addition to or instead of Figures 1-2 The hardware depicted in FIG. 3 may use other internal hardware or peripheral devices, such as flash memory, equivalent non-volatile memory, or optical disk drives, etc. In addition, the processes of the example embodiments may be applied to a multi-processor data processing system.

[0075] In some illustrative examples, data processing system 200 can be a personal digital assistant (PDA), which is generally configured with flash memory to provide non-volatile memory for storing operating system files and / or user-generated data. A bus system can include one or more buses, such as a system bus, an I / O bus, and a PCI bus. Of course, the bus system can be implemented using any type of communications fabric or architecture that provides for a transfer of data between different components or devices attached to the fabric or architecture.

[0076] A communication unit can include one or more devices for transmitting and receiving data, such as a modem or network adapter. A memory can be, for example, main memory 208 or a cache, such as the cache found in northbridge and memory controller hub 202. A processing unit can include or be a processor or CPU.

[0077] Figures 1-2 The examples depicted in the middle and above are not meant to imply architectural limitations. For example, data processing system 200 can be a tablet computer, a laptop computer, or a telephone device in addition to taking the form of a mobile or wearable device.

[0078] Where a computer or data processing system is described as a virtual machine, virtual device, or virtual component, the virtual machine, virtual device, or virtual component operates in the manner of data processing system 200 using a virtualized representation of some or all of the components depicted in data processing system 200. For example, in a virtual machine, virtual device, or virtual component, processing unit 206 is represented as a virtualized instance of all or a certain number of the hardware processing units 206 available in a host data processing system, main memory 208 is represented as a virtualized instance of all or a portion of the main memory 208 available in the host data processing system, and disk 226 is represented as a virtualized instance of the disk 226 available in the host data processing system. In such a case, the host data processing system is represented by data processing system 200.

[0079] Referring to Figure 3 , this figure depicts a block diagram of an example configuration for natural language response improvement in machine assisted agents, in accordance with an illustrative embodiment. Application 300 is an example of application 105 in Figure 1 and is executed in any of servers 104 and 106, clients 110, 112, and 114, and device 132 in Figure 1 .

[0080] The application 300 receives the recommended natural language response from the set of recommended responses for analysis. The application 300 also receives an edited version of the same recommended natural language response for analysis. The application 300 can also be configured to receive (if available) a natural language query intended to be a response to the natural language response to aid in the analysis. The application 300 can also be configured to receive (if available) any previous queries and responses within the same interaction that the recommended response is a part of, as well as metadata about the interaction. The queries, previous queries and responses, and metadata provide additional context for the analysis.

[0081] The edit detection module 310 analyzes the recommended response and the edited version of the recommended response and determines a personal entity difference score, a product entity difference score, and a sentence similarity score for the changes between the response and the edited version. The edit classification module 320 classifies the edited recommended response based on the personal entity difference score, the product entity difference score, and the sentence similarity score. The edit implementation module 330 updates the set of recommended responses based on the classification by the module 320 of the type of edits performed on the edited recommended response.

[0082] Referring to Figure 4 the figure depicts a block diagram of an example configuration for natural language response improvement in a machine-assisted agent, in accordance with an illustrative embodiment. Specifically, Figure 4 more details of the module 310 in Figure 3 are provided.

[0083] The personal entity module 410 uses a personal entity analyzer to detect one or more instances of a personal entity in the recommended response and the edited recommended response. The module 410 then compares the instances of the personal entity detected in the recommended response to the instances of the personal entity detected in the edited recommended response to determine a number of personal entity instances that have changed. The module 410 uses the number of personal entity instances that have changed as the personal entity difference score.

[0084] The product entity engine 420 uses a product entity analyzer to detect one or more instances of a product entity in the recommended response and the edited recommended response. The module 420 then compares the instances of the product entity detected in the recommended response to the instances of the product entity detected in the edited recommended response to determine a number of product entity instances that have changed. The module 420 uses the number of product entity instances that have changed as the product entity difference score.

[0085] The similarity engine 430 uses a sentence similarity analyzer to score the similarity between the recommended response and the edited recommended response. Specifically, the module 430 converts the recommended response and the edited recommended response to corresponding vectors and then computes the cosine similarity between the vectors.

[0086] Referring to Figure 5 , this figure depicts a block diagram of an example configuration for natural language response improvement in a machine-assisted agent, in accordance with an illustrative embodiment. Specifically, Figure 5 More details of the modules 330 in Figure 3 are provided.

[0087] If the product entity difference score is equal to one, the query includes the same product entity as the recommended response, but the product entity is changed to create an edited response, and the response replacement module 510 replaces the recommended response with the edited version in the set of recommended responses. If the product entity difference score is equal to one and the query includes the same product entity as the edited response but does not include the original recommended response, the response addition module 520 adds the edited version to the set of recommended responses while preserving the original recommended response.

[0088] If the sentence similarity score between the recommended response and the edited recommended response, determined in the manner described herein, is above a threshold similarity score, the application 300 determines that a low- meaning change has been made to the recommended response. The application 300 also considers a low- meaning change to be a change in which the personal entity difference score is above one, meaning that there is more than one personal entity difference between the recommended response and the edited response. If the application 300 classifies the edit as a low- meaning change, the AB test module 530 AB tests both the recommended response and the edited version. Once data has been collected for a statistically significant number of interactions including the variants, the module 530 retains one or more of the variants that performed best in the recommended set of responses and deletes the remaining variants.

[0089] If the sentence similarity score between the recommended response and the edited recommended response, determined in the manner described herein, is below a threshold similarity score, the application 300 determines that a high- meaning change has been made to the recommended response. The application 300 also considers a high- meaning change to be a change in which the product entity difference score is above one, meaning that there is more than one product entity difference between the recommended response and the edited response. If the application 300 classifies the edit as a high- meaning change, the response addition module 520 retains the recommended response and adds the edited version to the set of responses recommended.

[0090] Referring to Figure 6 , this figure depicts an example of natural language response improvement in a machine-assisted agent, in accordance with an illustrative embodiment. This example can be performed using the application 300 in Figure 3 .

[0091] As described, the recommended response 610 is "Hi, {first_name}" and the edited response 612 corresponding to the response 610 is "Hi, Jenny." Here, because "{first_name}" has changed to "Jenny," the application 300 determines the classification 614: editing the response 610 to the response 612 has resulted in a single personal entity change. The result 616 depicts the result of the classification 614: ignore the edit and the recommended response 610 remains in the set of recommended responses.

[0092] Referring to Figure 7 , the figure depicts an example of natural language response improvement in a machine-assisted agent, according to an illustrative embodiment. The example can be performed using the application 300 in Figure 3 .

[0093] As described, for the query 710, "Does Product X Gold come with text analytics?" the recommended response 712 is "Product X Gold comes with text analytics" and the edited response 714 is "Product X Professional comes with text analytics." Here, because both the query 710 and the recommended response 712 include the same product entity ("Product X Gold"), but in the edited response 714, "Product X Gold" is changed to "Product X Professional," the application 300 produces the classification 716: a single product entity change means a feature update, possibly because Product X Gold has been replaced by Product X Professional. The result 718 depicts the result of the classification 716: the application 300 replaces the recommended response 712 with the edited response 714 in the set of recommended responses.

[0094] However, for the query 720, "Does Product X Professional come with text analytics?" the recommended response 722 is "Product X Gold comes with text analytics" and the edited response 724 is "Product X Professional comes with text analytics." Here, because the product entity in the query 720 ("Product X Professional") is changed to "Product X Gold" in the recommended response 722 and then changed back to "Product X Professional" in the edited response 724, the application 300 produces the classification 726: a single product entity change means updating product knowledge in the set of recommended responses to cover the features of Product X Professional. The result 728 depicts the result of the classification 726: the application 300 adds the edited response 724 to the set of recommended responses.

[0095] Referring to Figure 8FIG. 3 depicts an example of natural language response improvement in a machine- assisted agent, according to an illustrative embodiment. This example can be performed using the application 300 in Figure 8 FIG. 4 depicts an example of natural language response improvement in a machine- assisted agent, according to an illustrative embodiment. This example can be performed using the application 300 in

[0096] Figure 8 An example of a minor syntax change is depicted: the recommended response 810, "The price of Product X is (plural)...," has been changed to the edited response 812, "The price of Product X is (singular)...". The application 300 has determined the classification 814: the sentence similarity score between the recommended response 810 and the edited response 812 is above the threshold similarity score, so this is a low- meaning change. The results 816 depict the results of the classification 814: the application 300 performs an AB test for the recommended response 810 and the edited response 812 to determine which is best.

[0097] Figure 9 An example of a minor style change is also depicted: the recommended response 820, "Hello, my name is...," has been changed to the edited response 822, "Hi, my name is...". The application 300 has determined the classification 824: the sentence similarity score between the recommended response 820 and the edited response 822 is above the threshold similarity score, so this is a low- meaning change. The results 826 depict the results of the classification 824: the application 300 performs an AB test for the recommended response 820 and the edited response 822.

[0098] Figure 3 An example of multiple personal entity changes is also depicted: the recommended response 830, "Hi, {first_name}, can I have your email address?", has been changed to the edited response 832, "Hi, John Doe, can I have your email address?". The application 300 has determined the classification 834: the personal entity difference score for this change would be two because "{first_name}" is changed to "John" and "Doe" is added (an additional personal entity). So, this is a low- meaning change. The results 836 depict the results of the classification 834: the application 300 performs an AB test for the recommended response 830 and the edited response 832 to determine which is best.

[0099] Referring to FIG. 5, an example of natural language response improvement in a machine- assisted agent is depicted, according to an illustrative embodiment. This example can be performed using the application 300 in Figure 9 FIG. 6 depicts an example of natural language response improvement in a machine- assisted agent, according to an illustrative embodiment. This example can be performed using the application 300 in Figure 9 FIG. 7 depicts an example of natural language response improvement in a machine- assisted agent, according to an illustrative embodiment. This example can be performed using the application 300 in

[0100] Figure 9The recommended response 910, "Best feature of Product X is its text analysis capabilities," and the edited response 912, "Best feature of Product X is its numerical analysis package," are depicted. The application 300 has determined a classification 914: this is a high-implication change because "text analysis capabilities" is not more semantically similar to its threshold similarity than "numerical analysis package." The result 916 depicts the result of the classification 914: the application 300 adds the edited response 912 to the recommended response set.

[0101] Figure 10 The recommended response 920, "Price of Product X is $100," and the edited response 922, "Price of Product X is $100. And the main feature is…," are also depicted. The application 300 has determined a classification 924: this is a high-implication change because the edited response 922 includes an additional sentence detailing feature information. The result 926 depicts the result of the classification 924: the application 300 adds the edited response 922 to the recommended response set.

[0102] Figure 3 The example of multiple product entity changes is also depicted: the recommended response 930, "Price of Product X Gold is $100," is changed to the edited response 932, "Price of Product X Professional is $200." The application 300 has determined a classification 934: the product entity difference score for this change is two because "Product X Gold" is changed to "Product X Professional" and the price is changed. Thus, this is a high-implication change. The result 936 depicts the result of the classification 934: the application 300 adds the edited response 932 to the recommended response set.

[0103] Referring to Figure 11 the figure depicts a flowchart of an example process for natural language response improvement in a machine-assisted agent, according to an illustrative embodiment. The process 1000 can be implemented in the application 300 in Figure 11 .

[0104] In block 1002, the application receives a natural language query, a recommended response to the natural language query, and an edited version of the recommended response. In block 1004, the application scores a personal entity difference between the recommended response and the edited version using a personal entity analyzer. In block 1006, the application scores a product entity difference between the recommended response and the edited version using a product entity analyzer. In block 1008, the application scores a sentence similarity between the recommended response and the edited version using a sentence similarity analyzer. In block 1010, the application updates a natural language response recommendation set based on the personal entity difference, the product entity difference, and the sentence similarity. The application then ends.

[0105] Referring to Figure 10FIG. 1 depicts a flow diagram of an example process for natural language response improvement in a machine-assisted agent, in accordance with an illustrative embodiment. Specifically, ​ More details of block 1010 in FIG. 1 are depicted. ​ More details of block 1010 in FIG. 1 are depicted.

[0106] In block 1102, the application checks whether a single individual entity change is detected. If yes (the “Yes” path of block 1102), in block 1104, the application ignores the edit and then ends. If no (the “No” path of block 1102), in block 1106, the application checks whether a single product entity has changed to update a feature. If yes (the “Yes” path of block 1106), in block 1108, the application replaces the recommended response with the edited version and then ends. If no (the “No” path of block 1106), in block 1110, the application checks whether a single product entity has changed to update a product. If yes (the “Yes” path of block 1110), in block 1112, the application adds the edited response to the set of recommended responses and then ends. If no (the “No” path of block 1110), in block 1114, the application checks whether a high- meaning change is detected. If yes (the “Yes” path of block 1114), in block 1112, the application adds the edited response to the set of recommended responses and then ends. If no (the “No” path of block 1114), in block 1116, the application AB tests the recommended version and the edited version of the response and then ends.

[0107] Accordingly, computer-implemented methods, systems or apparatuses, and computer program products for natural language response improvement in machine-assisted agents and other related features, functions, or operations are provided in illustrative embodiments. When an embodiment or a portion thereof is described as implemented in an application, a computer-implemented method, system or apparatus, computer program product, or portions thereof are adapted or configured for use in a suitable and comparable manner with the kind of apparatus.

[0108] Where an embodiment is described as implemented in an application, it is contemplated within the scope of the illustrative embodiments that the application is delivered in a software as a service (SaaS) model. In a SaaS model, the capability of the application implementing an embodiment is provided under a subscription and made available to customers over a network, such as the Internet. Customers can access the application by a thin client interface such as a web browser (e.g., a web-based e-mail) or other light-weight client-based application. A customer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, or storage of the cloud infrastructure. In some cases, customers can not even manage or control the capabilities of the SaaS application. In some other cases, the SaaS implementation of the application can allow for a limited amount of customer-specific application configuration settings for which a customer can manage or control.

[0109] The present application can be a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0110] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non- exhaustive list of more specific examples of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0111] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0112] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuits, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and procedural programming languages such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0113] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0114] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0115] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0116] The computer program product can have signal recorded thereon in a variety of forms. Such forms include, but are not limited to, radio frequency signals, infrared signals, ultraviolet signals, signals on a carrier wave, digital signals, and the like. Such signals can be recorded on the computer readable medium of the computer program product by a method such as, but not limited to, recording the signals using the recording methodology as described above. The computer program product can be recorded on the computer readable medium using one or more of the various methods as are currently available, or which can become available in the future. The computer program product of the present application can be implemented by using the computer readable medium directly, by use of machines, or by any combination thereof.

Claims

1. A computer-implemented method comprising: scoring personal entity differences between a natural language response to a natural language query and an edited version of the natural language response using a personal entity analyzer, the natural language response selected from a recommended set of natural language responses, wherein the personal entity analyzer detects instances of named entities referencing a person in a portion of a narrative text; scoring product entity differences between the natural language response and the edited version of the natural language response using a product entity analyzer, wherein the product entity analyzer detects instances of named entities referencing a product in a portion of narrative text; scoring sentence similarity between the natural language response and the edited version of the natural language response using a sentence similarity analyzer, wherein the sentence similarity analyzer measures similarity between two portions of narrative text; updating a natural language response set based on the personal entity difference score, the product entity difference score, and the sentence similarity score; as well as In the natural language interaction, a natural language response selected from the natural language response set is output, so that the natural language interaction uses the selected natural language response within the natural language interaction flow.

2. The computer-implemented method of claim 1 , wherein: The updating includes replacing a natural language response with an edited version of the natural language response in the set of natural language responses.

3. The computer-implemented method of claim 2, wherein: The replacing is performed in response to the product entity difference score being equal to one.

4. The computer-implemented method of claim 1 , wherein: The updating includes adding the edited version of the natural language response to a set of natural language responses.

5. The computer-implemented method of claim 4, wherein: The adding is performed in response to the product entity difference score being equal to one.

6. The computer-implemented method of claim 4, wherein: The adding is performed in response to the similarity score being below a threshold.

7. The computer-implemented method of claim 1 , wherein: The updates include: comparing result data responsive to a real-time natural language query with the natural language response, and comparing result data responsive to the real-time natural language query with an edited version of the natural language response; and Based on the result data, one of the natural language response and an edited version of the natural language response is retained in the set of natural language responses.

8. The computer-implemented method of claim 7, wherein: The comparing and retaining are performed in response to the similarity score being above a threshold.

9. A computer-usable program product comprising one or more computer-readable storage devices and program instructions stored on at least one of the one or more storage devices, wherein the stored program instructions include: program instructions for using a personal entity analyzer to score personal entity differences between a natural language response to a natural language query and an edited version of the natural language response, the natural language response being selected from a recommended set of natural language responses, wherein the personal entity analyzer detects instances of named entities referencing a person in a portion of narrative text; program instructions for scoring product entity differences between the natural language response and the edited version of the natural language response using a product entity analyzer, wherein the product entity analyzer detects instances of named entities referencing a product in a portion of narrative text; program instructions for scoring sentence similarity between the natural language response and the edited version of the natural language response using a sentence similarity analyzer, wherein the sentence similarity analyzer measures similarity between two portions of narrative text; program instructions for updating the natural language response recommendation set based on the person entity difference score, the product entity difference score, and the sentence similarity score; as well as Program instructions for outputting a natural language response selected from a set of natural language responses in a natural language interaction so that the natural language interaction uses the selected natural language response within the natural language interaction flow.

10. The computer usable program product of claim 9, wherein: The update includes program instructions to replace the natural language response with an edited version of the natural language response in a set of natural language responses.

11. The computer usable program product of claim 10, wherein: The replacing is performed in response to the product entity difference score being equal to one.

12. The computer usable program product of claim 9, wherein: The update includes program instructions to add an edited version of the natural language response to the set of natural language responses.

13. The computer usable program product of claim 12, wherein: The adding is performed in response to the product entity difference score being equal to one.

14. The computer usable program product of claim 12, wherein: The adding is performed in response to the similarity score being below a threshold.

15. The computer usable program product of claim 9, wherein: The updates include: program instructions for comparing result data responsive to a real-time natural language query and the natural language response with result data responsive to the real-time natural language query and the edited version of the natural language response; and Program instructions for retaining one of the natural language response and an edited version of the natural language response in the set of natural language responses based on result data.

16. The computer usable program product of claim 15, wherein: The comparing and retaining are performed in response to the similarity score being above a threshold.

17. The computer usable program product of claim 9, wherein the computer usable code is stored on a computer readable storage device in a data processing system, and wherein the computer usable code is transmitted from a remote data processing system over a network.

18. The computer-usable program product of claim 9, wherein the computer-usable code is stored in a computer-readable storage device in a server data processing system, and wherein the computer-usable code is downloaded over a network to a remote data processing system for use in a computer-readable storage device associated with the remote data processing system.

19. A computer system comprising one or more processors, one or more computer-readable memories, and one or more computer-readable storage devices, and program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, the stored program instructions comprising: program instructions for scoring personal entity differences between a natural language response to a natural language query and an edited version of the natural language response using a personal entity analyzer, the natural language response selected from a recommended set of natural language responses, wherein the personal entity analyzer detects instances of named entities referencing a person in a portion of narrative text; program instructions for scoring product entity differences between the natural language response and the edited version of the natural language response using a product entity analyzer, wherein the product entity analyzer detects instances of named entities referencing a product in a portion of narrative text; program instructions for scoring sentence similarity between the natural language response and the edited version of the natural language response using a sentence similarity analyzer, wherein the sentence similarity analyzer measures similarity between two portions of narrative text; program instructions for updating the natural language response recommendation set based on the person entity difference score, the product entity difference score, and the sentence similarity score; as well as Program instructions for outputting, in a natural language interaction, a natural language response selected from a set of natural language responses such that the natural language interaction uses the selected natural language response within a flow of the natural language interaction.

20. The computer system of claim 19, wherein: The updating includes program instructions to replace the natural language response with an edited version of the natural language response in the set of natural language responses.

Images