Metrics for automated evaluation of conversational responses

Abstract

Example generally relate to automated evaluation of machine-generated conversational responses. Context-message response n-tuples are extracted from at least one conversational data source to generate a multi-reference response set. Responses in the multi-reference response set include a context-message data pair and a rating. The rating indicates a quality of the response with respect to the context-message data pair. A response evaluation engine generates a metric score for a machine-generated response based on an evaluation metric and the multi-reference response set. The metric score indicates a quality of the machine-generated conversational response with respect to a user-generated message and a context of the user-generated message. Based on the metric score, a response generation system of a computing device, such as a digital assistant, is optimized and adjusted to improve accuracy, quality, and relevance of responses output to a user.

Description

BACKGROUND

[0001] Response generation systems, also known as dialog systems or conversational agents, are becoming increasingly prevalent in a variety of systems and devices. Response generation systems include applications and computer systems designed to interpret natural language input messages and output natural language responses. However, these systems frequently output low quality responses that are not actually relevant or appropriate to the conversation.

[0002] While some machine translation evaluation utilizes metrics, there is currently no method or metric for automatically judging the quality of responses generated in human- machine conversation systems. Machine translation can also be referred to as automated language translation. Without a metric for evaluating the quality of machine-generated responses, response generation systems cannot be automatically optimized to improve the quality of machine-generated responses.

[0003] To improve the quality of responses, human users are required to manually review and evaluate the quality of each machine-generated response and manually adjust the response generation system in an attempt to improve response quality. However, manual human evaluation can be prohibitively expensive. Manual human evaluation results can also be inconsistent. Additionally, manual evaluation and tuning cannot scale with production-scale response generation systems that have hundreds or thousands of parameters to be adjusted for optimization of the system. This manual review process is also time-consuming, cumbersome, tedious, inefficient, and suboptimal. SUMMARY

[0004] Examples of the disclosure provide a metric for automatically scoring machine-generated conversational responses. In one example, a computer-implemented method for automatic evaluation of machine-generated responses is provided. An extraction component extracts candidate response n-grams from at least one conversation data source to form a multi-reference response set. Each response in the multi-reference response set includes a selected context-message data pair and a rating for each response relative to the selected context-message data pair. A response evaluation engine computes an evaluation metric based on a selected machine-generated conversational response and the multi-reference response set. The response evaluation engine generates a metric score. The metric score indicates a quality of the selected machine-generated conversational response relative to the selected context-message data pair associated with the selected machine-generated response.

[0005] This summary is provided to introduce a selection of concepts, in a simplified form, that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0006] Figure 1is an example block diagram illustrating a computing device for automated evaluation of machine-generated responses.

[0007] Figure 2 is an example block diagram illustrating a computing device for extracting a set of reference responses from one or more conversational data sources.

[0008] Figure 3 is an example diagram illustrating a context-message-response triple data set.

[0009] Figure 4 is an example block diagram illustrating biasing responses based on context including sentiment of a user.

[0010] Figure 5 is another example block diagram illustrating biasing responses based on context including sentiment of a user.

[0011] Figure 6 is an example block diagram illustrating responses extracted from conversational data sources based on context-message pairs.

[0012] Figure 7 is an example block diagram illustrating additional context-message-response triples mined from conversational data sources.

[0013] Figure 8 is an example block diagram illustrating generation of a multi-reference response set.

[0014] Figure 9 is an example block diagram illustrating a response evaluation engine.

[0015] Figure 10 is an example block diagram illustrating response evaluation engine training.

[0016] Figure 11 is an example block diagram illustrating response evaluation engine generation of metric scores for machine-generated responses.

[0017] Figure 12 is a table illustrating number of triples, average, minimum, and maximum number of references for debugging and testing.

[0018] Figure 13 is a flow diagram illustrating operations of a computing device for generating metric scores for machine-generated responses.

[0019] Figure 14 is a flow diagram illustrating operations of a computing device for extracting a plurality of candidate responses from one or more conversational data sources.

[0020] Figure 15 is a flow diagram illustrating operations of a computing device for selecting a multi-reference response set.

[0021] Figure 16 is a flowchart illustrating operations of a computing device for accessing ratings for each response in a set of multi-reference responses.

[0022] Figure 17 is an example block diagram of a digital assistant.

[0023] Corresponding reference numbers indicate corresponding parts throughout the drawings. DETAILED DESCRIPTION

[0024] Referring to the figures, example implementations of the disclosure are for context-sensitive measures of quality that automatically evaluate machine-generated responses in a human-conversational system. The response evaluation measures enable efficient, automatic evaluation of machine-generated responses in a variety of outputs, such as textual responses, audio responses, and visual format responses.

[0025] In some examples, the extraction component extracts content-message-response n-tuples from at least one conversational data source to form candidate response n-tuples, where n is an integer. In some examples, the context-message-response n-tuples are context-message-response triples. A context-message-response triple is a data set that includes a message, a response to the message, and a context in which the message occurs. This context-sensitive extraction of candidate responses improves the accuracy and quality of the automatic evaluation engine.

[0026] However, the context-message-response n-tuples are not limited to triples. In other examples, the context-message-response n-tuples can be message-response data pairs. In other words, the context can not exist or the context can be a null value. In such cases, the context-message-response tuples can not include a context value.

[0027] The extraction component selects a response from the candidate context-message-response triples that correspond to the selected message and context. The selected response forms a set of reference responses. In other words, the extraction component identifies and selects responses from content-message-response triples that have the same or similar message and context. The context-sensitive selection of reference responses and the context-sensitive extraction of candidate responses improves the consistency of the response evaluation process.

[0028] Aspects of the disclosure enable automated evaluation of machine-generated responses by a response evaluation engine. The response evaluation engine computes a response evaluation metric based on the machine-generated response, the context associated with the machine-generated message and the message, and a set of reference responses. The response evaluation metric enables an intrinsic evaluation of the machine-generated response in tasks that admit a diverse range of possible outputs in data-driven machine response generation. This intrinsic evaluation is challenging because the semantic space of possible responses that can be output is broad, resistant to definition, and only weakly constrained by the conversation input.

[0029] Aspects of the disclosure further provide a metric score that indicates the quality of the machine-generated response. The response evaluation metric score further provides more consistent evaluation results and reduces human error that occurs during human-in-the-loop evaluation of machine-generated responses.

[0030] The metric score generated by the response evaluation engine correlates well with human evaluation scores. The metric score can be used for automated training and evaluation of data-driven conversational systems, as well as other text and response generation tasks with intrinsically disparate objectives. Additionally, the response evaluation engine improves user efficiency by eliminating the need for human-in-the-loop evaluation of machine-generated responses.

[0031] In other aspects of the disclosure, the metric score can be used to automatically adjust one or more parameters of the response generation system to improve or otherwise optimize the quality of the machine-generated responses. The metric score can be used to calibrate the response generation system to achieve higher or better metric scores. As the metric score for a response increases, the quality of the response generated by the response generation system improves. This metric score enables the response generation system to generate more natural, higher quality responses. The evaluation can be used to ultimately identify and provide more natural, human-like responses from digital assistants to users.

[0032] Additionally, the metric can be used to optimize machine-generated responses in devices and applications, such as digital assistants, to improve the accuracy of responses and functionality performed by the digital assistant. For example, when a user says "I need more" in the context of the user being in a car, where the user's prior message indicates a low fuel level, the digital assistant makes a more accurate determination that the user's message is associated with the purchase of gasoline. The digital assistant is able to make a more relevant response, such as "Would you like the location of a nearby gas station?" or "Would you like to find a gas station now?"

[0033] The metrics can be used in conjunction with a digital assistant to improve user interaction with the digital assistant and provide improved responses to user questions and requests. The digital assistant can provide assistance in obtaining directions, making a phone call, obtaining recommendations for a restaurant or entertainment venue, scheduling an appointment, updating a contact list / address book, or any other functionality associated with a smartphone, tablet, wearable computing device, or any other user device.

[0034] Optimizing a response generation system with metrics produces more natural responses that more closely correspond to natural, human-generated responses. The metrics provide an evaluation of the quality of machine-generated responses, which enables the generation of higher quality responses, thereby improving the user experience interacting with digital assistants and other devices that utilize the response generation system. Higher quality machine-generated responses also improve the overall efficiency of a personal digital assistant or other device that utilizes the response generation system. This also improves optimization efficiency and reduces the time needed to optimize the response generation system.

[0035] Referring now to the drawings Figure 1 , an example block diagram of a computing device that illustrates automatic evaluation of machine-generated responses is shown. System 100 is a dialog system for determining the quality of a machine-generated response with respect to a user-generated input message.

[0036] In the example of Figure 1 , computing device 102 associated with user 104 represents any device that executes computer executable program code 106 (e.g., as an application, operating system functionality, or both) to implement operations and functionality associated with the computing device. Computing device 102 can include a mobile computing device or any other portable device. In some examples, a mobile computing device includes a mobile phone, a laptop computer, a tablet computer, a computing pad, a netbook, a gaming device, and / or a portable media player. Computing device 102 can also include less portable devices such as a desktop personal computer, a kiosk, a tablet device, an industrial control device, a wireless toll station, and an electric car charging station.

[0037] In other examples, computing device 102 can be embodied within a robot, an automobile, or any other device. Additionally, computing device 102 can represent a group of processing units or other computing devices.

[0038] Computing device 102 has one or more processors 108, one or more memory storage devices 110, a set of sensors 112, and input / output devices 114. Memory storage devices 110 include at least one memory region 116. Memory region 116 includes, for example, but is not limited to, computer executable program code 106, applications 118, response evaluation engine 122, and extraction component 124.

[0039] The processor 108 includes any number of processing units and is programmed to execute computer executable program code 106 for implementing aspects of the disclosure. The computer executable program code 106 includes instructions. The instructions can be executed by one or more processors 108 within the computing device 102 or by processors external to the computing device 102. The one or more processors are programmed to execute the instructions, such as the instructions illustrated in the various figures (such as Figure 8 , Figure 9 , Figure 10 and Figure 11 ).

[0040] In some examples, the processor 108 represents an implementation of analog techniques for performing the operations described herein. For example, the operations can be performed by an analog computing device and / or a digital computing device.

[0041] The computing device 102 also has one or more computer readable storage media, such as a memory area 116. The memory area 116 includes any number of media associated with or accessible by the computing device 102. The memory area 116 can be internal (as shown in Figure 1 ), external (not shown), or both (not shown) to the computing device. In some examples, the memory area 116 includes read only memory and / or memory wired into an analog computing device.

[0042] The memory area 116 stores one or more applications 118, as well as other data. The applications 118 operate when executed by a processor to perform functions on the computing device 102. Exemplary applications 118 include a conversational agent, a mail application, a web browser, a calendar application, an address book application, a messaging program, a media application, a location-based service, a search program, etc. The applications 108 can communicate with a corresponding application or service, such as a web service accessible via a network 120. For example, the applications 118 can represent downloaded client-side applications corresponding to server-side services executing in the cloud.

[0043] The memory area 116 also stores one or more computer executable components. An exemplary component includes a response evaluation engine 122. The response evaluation engine 122 component, when executed by a processor of the computing device 102, causes the processor to generate a metric score indicative of a quality of a machine-generated response. The machine-generated response can include, but is not limited to, a textual response, a voice response, a gesture response, a graphical response, an actuation or motor function response, or any other type of response.

[0044] The memory region 116 can optionally store an extraction component 124. The extraction component 124 is a component for extracting messages, responses, and context data from one or more conversational data sources. The conversational data sources can include social media sources as well as databases that store conversational data.

[0045] In some examples, the extraction component uses an extraction algorithm to cover the space of plausible responses that will be mined from the conversational data sources. Given a test triple (context, message, and response), the extraction component 124 mines other responses that fit with the context and message pair (c t , m t ).

[0046] In one non-limiting example, the extraction component uses an information retrieval system to select a set of fifteen (15) candidate response triples. The information retrieval system is calibrated to select candidate response triples for which both the message and the response are similar to the selected message and response.

[0047] The computing device 102 can include a user interface component 126. The user interface component 126 includes a graphics card for displaying data to and receiving data from the user 104. The user interface component 126 can also include computer-executable instructions (e.g., drivers) for operating the graphics card. Additionally, the user interface component can include a display (e.g., a touch screen display or a natural user interface) and / or computer-executable instructions (e.g., drivers) for operating the display. The user interface component 126 can also include one or more of the following user interface components for providing data to or receiving data from the user: a speaker, a sound card, a camera, a microphone, a vibration motor, one or more accelerometers, a communication module, global positioning system (GPS) hardware, and a light sensor. For example, the user 104 can enter commands or manipulate data by moving the computing device 102 in a particular manner.

[0048] The computing device 102 can include a communication interface component 128. The communication interface component 128 includes a network interface card and / or computer- executable instructions (e.g., drivers) for operating the network interface card. Communication between the computing device 102 and other devices can occur through any wired or wireless connection, using any protocol or mechanism. In some examples, the communication interface is operable with short-range communication technology, such as by using near field communication (NFC) tags.

[0049] Input / output devices 130 include one or more devices for inputting data into computing device 102 and / or one or more devices for outputting data to user 104. Input devices can include, but are not limited to, a keyboard, a mouse, a touch screen, a microphone, a camera, a graphical user interface, a pressure sensor, a touch sensor, or any other type of input device. Output devices can include, but are not limited to, a display screen, a graphical user interface, a speaker, an actuator associated with a robot or other mechanical device, or any other type of output device.

[0050] Computing device 102 can optionally include a digital assistant 132. Digital assistant 132 can also be referred to as a personal information manager. Digital assistant 132 is a component for managing a user's personal information, phone functions, schedules, appointments, contacts, favorite restaurants, travel plans, frequent destinations, and any other personal information. Digital assistant 132 includes a conversational agent or other response generation system capable of communicating with user 104 using natural language responses to user queries and messages.

[0051] In this non-limiting example, computing device 102 can optionally be connected to one or more servers 134 over a network 120. Network 120 can include any type of network connection. In one example, network 120 can be the Internet, an intranet, an Ethernet, or other wireless or hardwired connection over which computing device 102 can send and receive data associated with one or more servers 134. However, other examples need not include network 120.

[0052] Computing device 102 can also optionally store and / or access data stored on a remote data store 136. Data store 136 can be any type of data storage system, such as but not limited to a cloud storage system.

[0053] Figure 2 is an example block diagram illustrating a computing device for extracting a set of reference responses from one or more conversational data sources. Computing device 200 is a device for mining context-message-response n-tuples 202 from one or more conversational data sources 204. A context-message-response n-tuple is a set of data that includes a human-generated message and a user-generated response to the message extracted from a conversational data source. A context-message-response n-tuple can be a context-message-response triad that includes a message, a response, and a context of the message.

[0054] An example of a context-message-response extracted from the conversational data source 204 can include the context "I'm on my way now"; the message "I'll be downstairs waiting"; and the response "I'll keep an eye out for you."

[0055] The extraction component 206 includes algorithms for extracting relevant responses from one or more conversational data sources. The algorithms are used to extract relevant responses from conversational data sources, such as social media. The extracted relevant responses are then used as references with the response evaluation metrics of the response evaluation engine 122.

[0056] The response evaluation metrics are metrics used to evaluate machine-generated responses. The response evaluation metrics directly embed human judgment regarding the quality of a reference sentence into the computation of the full-set level multi-reference algorithm. The response evaluation metrics are automated metrics used to perform evaluations of machine-generated responses.

[0057] The response evaluation metrics can be used to evaluate machine-generated responses in a variety of formats. Example formats for responses that can be evaluated using the response evaluation metrics include, but are not limited to, a textual format, an audio format, or a visual format, such as a video, and combinations of textual, audio, and / or visual formats. The responses can be textual only, responses including textual and audio, and responses including audio and visual components, such as video graphics. In other examples, the evaluation metrics can be used to evaluate responses including textual format, audio format, and visual format elements.

[0058] The written responses are in a textual format. If the responses are in an audio format, the responses can include spoken sounds, such as spoken speech, and non-spoken sounds. The non-spoken sounds can include beeps, whistles, cheers, honks, chimes, bells, buzzes, claps, laughter, cries, musical instrument sounds, or any other non-spoken audio. Responses in a visual format can include physical gestures or articulations of a device, such as a robotic arm or other actuated equipment, video graphics or images on a screen or display, and / or any other visual type of response.

[0059] In some examples, the extraction equation, such as shown in Equation (1) below, provides:

[0060]

[0061] where s is the score, m is the message, r is the response, d is the bag of words similarity function, a controls the influence of similarity between responses, and e is a smoothing factor that avoids zero scores for candidate responses that do not share any words with the reference response. This formula provides a diverse and rational reference. Given a set of candidate triples, human evaluators rate the quality of the responses within the new triples.

[0062] The responses can be rated by the human evaluators on any type of scale. In some examples, the reference string is scored by the human rater for quality on a scale of negative one to positive one to weight the multiple reference responses. However, in other examples, the rating can be on a scale of one (1) to five (5).

[0063] In one non-limiting example, the reference responses that receive a human rating in the higher range are retained. For example, the reference responses that receive a human rating score of four or better on a five (5) point scale can be retained.

[0064] The extraction component 206 is executed by one or more processors to perform the extraction of the context-message-response n-tuples 202. The extraction component 206 then identifies relevant responses from the extracted context-message-response triples for use by a response evaluation engine of the response generated by the judgment machine.

[0065] The conversation data source 204 includes human generated messages and corresponding human generated responses in any available format. Examples of formats include, but are not limited to, a textual format, an audio format, or a visual format such as a video. Thus, the messages and responses that are written are in a textual format. If the responses are in an audio format, the messages and responses are spoken utterances. The messages and responses in a visual format can include gestural messages and / or gestural responses. Examples of gestural responses can include, but are not limited to, a wink, a frown, a thumbs up sign, a hand sign, a wave goodbye, or any other movement, motion, gesture, facial expression, posture, or other movement that indicates a message or a response.

[0066] The messages and responses can likewise include multiple formats. In other words, a human generated spoken message can correspond to a human generated gestural response such as a change in facial expression. For example, a human generated message of "goodbye" can correspond to a human generated response of a "wave goodbye gesture" where the wave indicates a goodbye or farewell response to the spoken utterance of "goodbye."

[0067] In some examples, the extraction component 206 can store the extracted context-message-response n-tuples 202 in a database 208. The database 208 can be any type of data storage device for storing the context-message-response n-tuples 202.

[0068] The extracted context-message-response triplets are candidate context-message-response n-tuples 210. The extraction component 206 selects two or more candidate context-message-response n-tuples 210 corresponding to the context-message data pair 212 to form a multi-reference response set 214. The context-message data pair 212 includes a human-generated message and the context of the message.

[0069] The context of the message includes linguistic context data and non-linguistic context data. The linguistic context data includes message-response data pairs that precede the selected message and the selected machine-generated conversational response in a given conversation. The linguistic context data is the user's past conversation history within a given conversation.

[0070] The non-linguistic context data includes data associated with the user. The non-linguistic data can include the user's likes and dislikes, preferences, favorite songs, disliked songs, the user's friends, important people, birthdays, anniversaries, holidays, appointments, hobbies, scheduled events, etc.

[0071] The non-linguistic context data can also include environmental data derived from one or more sensor devices. For example, the non-linguistic context data can include the current temperature, a weather forecast, the speed or orientation of a car associated with the user, the orientation of a device, location information from a device with global positioning system (GPS) capabilities, camera data, audio data from a microphone or other audio device, pressure sensor data, infrared sensor data, or any other type of non-linguistic data.

[0072] In this example, the context-message data pair 212 is a human-generated message and the conversational context of the message. The context-message data pair 212 is associated with a machine-generated response. Thus, the extraction component extracts context-message-response n-tuples 202 in a context-sensitive manner to form a multi-reference response set 214. In other words, the multi-reference response set 214 includes human-generated responses that correspond to the same or similar message and context pairs as the machine-generated response.

[0073] For example, if a machine-generated response of "Good luck" is generated in response to a context-message data pair that includes "Go to study" and the context "school exam," the extraction component will select context-message-response triplets that include responses that correspond to the same or similar context-message data pairs for messages related to "studying / learning" and contexts related to "exams / tests." In this way, the extraction component 206 performs the extraction and selection of the multi-reference response set in a context-sensitive manner that takes into account the context of each message and the context of the corresponding result identified from the conversation data source 204.

[0074] The conversational data sources 204 can include social media sources 216 and / or past conversational data 218. The social media sources can be any available social media source such as, for example and without limitation, social networking data feeds, video chats, video conferencing, instant messaging, or any other social media source.

[0075] In other examples, the conversational data sources 204 include one or more databases that store context-message-response triples generated by human users. For example, the past conversational data 218 can include context-message-response triples generated by users in past conversations. In still other examples, the conversational data sources 204 include one or more databases that store context-message-response triples that have been extracted from one or more conversational data sources.

[0076] Figure 3 is an example diagram illustrating a context-message-response n-tuple dataset. The context-message-response n-tuples 202 are an example dataset extracted from one or more conversational data sources 204.

[0077] The context-message-response n-tuples are datasets that contain data associated with a response. In some examples, the context-message-response n-tuples are triples. For example, a context-message-response triple of the context-message-response triples includes a message, a corresponding response to the message, and a context of the message. In this example, the context-message-response triple 302 includes a human-generated message 304, a context 306 of the message 304, and a human-generated response 308 to the message. The context-message-response triple 310 includes a context 312, a message 314, and a response 316. The context-message-response triple 318 includes a context 320, a message 322, and a response 324.

[0078] The context-message-response triples can be extracted from social media sources such as, but not limited to, social networking data feeds. The extraction component 206 identifies and selects or filters the context-message-response triples according to the selected messages and the selected contexts (such as the selected context-message pairs 212) to generate the multi-reference response set.

[0079] In some examples, the extraction component 206 mines a set of multiple reference strings. This approach includes mining conversational responses from social media (e.g., social networking data feeds or any other conversational data source).

[0080] In this non-limiting example, the extraction component 206 identifies conversation triples (context, state, response) in social media and / or other conversational data sources. For example, in some social media sources, these triples are identified by the fact that any social media user can address her message to a particular user. Millions of conversation triples can be mined in this way.

[0081] The context in this example can represent a chain of any length that includes message-response pairs that precede a given message and response that occur during the current conversation. The context can also include non-verbal information such as location, date, time of day, and other externally available information related to the conversation. This extraction technique provides at least one reference response for each context-message pair.

[0082] In another example, the extraction component 206 mines other responses that fit the selected context and message pair. To do so, the extraction component 206 uses an information retrieval system to select a large number of candidate triples. To minimize the risk of obtaining a low score from human evaluation, the extraction component 206 is calibrated to select candidate triples that are similar to the original or selected message and response pair for both its message and response.

[0083] Figure 4 is an exemplary block diagram illustrating biasing a response based on a context that includes a user's emotion. The conversation context can include a user's emotion. Emotion refers to a user's emotional state. For example, a user's emotion can be, without limitation, happy, sad, neutral, angry / hostile, calm, anxious, tense, peaceful, passionate, impatient, tired / exhausted, or any other emotional state. In this example, the conversation context includes a neutral emotion for the user. The emotion can be determined based on one or more context data sources. The user's emotion can be determined based on audio data, video data, time of day, temperature, input messages typed by the user, words spoken by the user, gestures made by the user, facial expressions, and any other type of context data.

[0084] For example, audio data can be used to determine the emotion based on the user's voice. The user's emotion can be determined based on a voice inflection, a fluctuation in the voice, a volume, a tone, a variation, or any other factor associated with the user's voice.

[0085] The user's emotion can also be determined based on visual data. For example, the user's emotion can be determined based on facial expressions, pupil dilation, body posture, body movement, and the like.

[0086] Emotions can also be determined based on words spoken or typed by the user. In still other examples, emotions can be determined based on two or more signals received from two or more sensors. For example, emotions can be determined based on a combination of audio data received from an audio sensor, visual data received from one or more cameras, and prior messages / statements of the user during the current session.

[0087] In this non-limiting example, the user emotion is neutral. The response generation system generates a response to the user input message 400 based on the context including the user emotion. The response can be generated in real-time based on the emotion associated with the user and / or other context data or selected from a set of pre-generated responses 402.

[0088] Figure 5 is another example block diagram illustrating biasing responses based on a context including an emotion of a user. In this example, the context data indicates that the user emotion is happy. The response generation system generates a response to the user generated message 500. In this non-limiting example, the response generation engine selects a response from a set of pre-generated responses 502 according to the context including the user emotion. In other embodiments, the response can be generated in real-time.

[0089] Figure 6 is an example block diagram illustrating responses extracted from a conversation data source based on a context-message pair. The extraction component extracts one or more responses from the conversation data source corresponding to the selected context and message. In this non-limiting example, the selected context 602 is based on the prior statement "Doubt it! What do you look like?" The selected message 604 is the user generated message "Look at my pics. I have something there." The set of responses 606 is responses from context-message-response identifications having the same or similar context and message as the context 602 and message 604.

[0090] Figure 7 is an example block diagram illustrating additional context-message-response triples. The context-message-response triples 702, 704, 706, 708, 710, and 712 are non-limiting examples of context-message-response triples mined from one or more conversation data sources.

[0091] Figure 8 is an example block diagram illustrating generation of a multi-reference response set. The extraction component 206 performs one or more queries on a context-message-response conversation triple database 802. The context-message-response conversation triple database 802 can be a database such as the database 108. The database 802 can be a database of context-message-response triples mined from one or more conversation data sources.Figure 2 the database 208 in the message-response repository 202.

[0092] The extraction component 206 performs information retrieval responsive to the candidates 804 to generate multi-reference context-message-responses 806. In other words, the extraction component 206 performs information retrieval to extract or select a single context-message-response triple from the plurality of response candidates 804. The retrieved multi-reference context-message-responses 806 are reference responses that are identified and / or retrieved based on the selected context and the selected human-generated responses associated with a given machine-generated response to be judged by the response evaluation engine 122. The multi-reference context-message-responses 806 can be a set of reference responses, such as Figure 2 the multi-reference response set 214 in the message-response repository 202.

[0093] The more references in the multi-reference response set 214, the greater the correlation with human judgment. However, collecting multiple references is difficult in the case of conversational data. Conversational data collected from naturally occurring conversations typically gives each user-generated message only one response. In other words, there is typically only one single response to a message for each user-generated response.

[0094] In some examples, the extraction component can mine messages that occur multiple times in the conversational data to attempt to find distinct responses to a given message. However, this solution is not ideal because responses often directly or indirectly depend on the dialogue history. And while it is possible to have writers create additional references where the downstream task is relatively unambiguous, this approach presents difficulties in the case of more subjective tasks, such as conversational response generation.

[0095] Accordingly, in other examples, the extraction component mines candidate responses from the conversational data source and has the mined candidate responses rated by one or more judgments. The multi-reference context-message-responses 806 receive annotations 808. The annotations 808 in this example are ratings selected, created, or assigned by one or more human users. In other words, one or more judgments rate each response in the multi-reference context-message-response triple 806 to create the annotations.

[0096] The judgments rate the quality of each of the reference responses using a rating system. The judgments rate the reference responses based on one or more criteria. The criteria for rating the relevance of the responses can include criteria such as, but not limited to, the conversational context, the length of the response, the intent, the sentiment, the content of the response, the syntax, the syntax, the intonation, the accuracy of the response, the similarity of the response to one or more other responses, the overall relevance of the response to the corresponding message, or any other criteria.

[0097] In some examples, the judgment is human judgment. In other examples, the judgment can be computer judgment. Computer judgment can include computer hardware, computer software, or a combination of hardware and software.

[0098] In this example, the rating is a score within a scale between negative one and positive one. In other words, the rating is a score or other indicator on a continuous scale from negative one (-1) to positive one (+1).

[0099] Accordingly, the evaluation metric in this example relies on both positive examples and negative examples to produce a metric score that quantifies the relevance of a machine-generated response to a given message and / or the context of a given message. The positive examples used by the evaluation metric are reference responses that are positively rated. The negative examples used by the metric are poor, negatively rated reference responses.

[0100] However, in other examples, the rating can be a score on a different scale. For example, the annotation 808 can be a rating on a scale from one (1) to five (5), a rating on a scale from one (1) to ten (10), or a rating on any other scale.

[0101] The multi-reference context-message-response 806 is added to the multi-reference context-message-response set 810. The multi-reference context-message-response set 810 in this example includes two or more multi-reference context-message-responses. The multi-reference context-message-response set 810 is used by the response evaluation engine to generate a metric score for a given machine-generated response.

[0102] Figure 9 is an example block diagram illustrating a response evaluation engine. The response evaluation engine 900 is a component executed by one or more processors to compute an evaluation metric 902 based on a machine-generated response, a selected contextually sensitive set 214 of context-message data pairs, and multi-reference responses.

[0103] The evaluation metric 902 is a metric used to score a given conversational response against a set of multi-reference responses 214. In this example, the evaluation metric 902 is a tunable metric.

[0104] The evaluation metric 902 is contextually aware of the conversational context of the machine-generated response that is being evaluated or judged by the response evaluation engine. This context sensitivity stems from the fact that the extraction algorithms of the extraction component 124 consider the context in which the relevant responses were produced to mine these candidate responses from conversational data sources.

[0105] An example evaluation metric, such as the evaluation metric 902, is shown in the following equation (2), which is as follows:

[0106]

[0107] where the fraction (Y, R1,... R N ) is a metric score generated by the response evaluation engine 900.

[0108] As shown in equation (2) above, exp() is the exponential function. The normalized score Z guarantees that the output metric score of the machine-generated response Y is defined between zero (0) and one (1).

[0109] The rating "S i " is a human rating between negative one (-1) and positive one (+1). The overlap (Y, R i ) is the number of n-grams (sequences of words) overlap between the machine-generated response Y and one or more reference responses R i An overlap of zero (0) means there are no common words between the machine-generated response Y and one or more reference responses R i An overlap of one (1) means they are identical.

[0110] The overlap between the candidate reference and positively rated responses (S i > 0) helps to increase the metric score. A positively rated response is a higher quality response. A positively rated response can be referred to as a good response.

[0111] The quality of a response can be judged based on whether the response is relevant or pertinent in the context of a given conversation, whether the response is grammatically correct, whether the response is a natural response that matches or corresponds to a human response, whether the response is syntactically correct, how well the context of the response matches the context of the pair, how well the content of the response matches the content of the pair, how well the language, tone, accent, and syntax of the response matches the language, tone, accent, and syntax of the pair, and / or how well the response matches what a human would say in the same or similar context. Thus, a positively rated response is a response that is relevant to the message and / or context, grammatically correct, syntactically correct, relevant to the entire conversation, and / or the same or similar to a natural human-generated response.

[0112] The overlap with a negatively rated response (S i < 0) helps to decrease the metric score. A negatively rated response can be referred to as a suboptimal response. A negatively rated response can be referred to as a bad response.

[0113] A suboptimal response is a response that contains poor grammar and / or incorrect / inappropriate syntax, is not relevant to a given message, is not appropriate in the context of a given current conversation, and / or is not similar to a natural human-generated response to the same or similar message and / or conversation context.

[0114] Another example response evaluation metric for evaluating a conversational response is shown in the following equations (3)-(5). Given an input message m and a prior conversational history c, the goal of a response generation system is to produce a hypothesis h that is a high quality response to the message. The evaluation metric assumes a set of references {r i,j} are available for the context c and message m, where i e {1...I} is an index within the test set. In this case, the automatic score of the system output h1...h I is defined as shown in the following example equations (3) and (4):

[0115] BP - exp(∑n log pn) (3)

[0116] "n" in equation (3) represents the size of the n-gram order. p in equation (3) represents the n-gram precision.

[0117] The hypothesis is represented by p in equation (4) as follows:

[0118]

[0119] where p represents the hypothesis and n represents the length of the reference. In equation (4), the reference length n is not the same as the n-gram order size n shown in equation (3). Likewise, p in equation (3) represents the n-gram precision while p in equation (4) represents the hypothesis.

[0120] The full set level n-gram precision p n is then defined in the following example equation (5) as:

[0121]

[0122] where g is the number of times the n-gram occurs in the given sentence and g (u, v) is shorthand for min{# g (u), g (v)}.

[0123] The response evaluation metric utilizes a human qualitative judgment w ij e [-1, +1]. This metric is discriminative because it both rewards matches to good reference responses with positive ratings and penalizes matches to bad negative rated reference responses.

[0124] The metric shown in equation (5) weights each n-gram match by the highest scoring reference it occurs in. This weighting can be negative. In some examples, to ensure the denominator never goes to zero (0), the metric assumes that for each i there exists a weight w i,jAt least one reference r that is strictly positive i,j .

[0125] In some examples, the maximum theoretical score is one (1). For instance, if the assumption is a match with the highest weighted reference used for each sentence, the numerator equals the denominator and the metric score becomes one. This metric can be used for automatic evaluation of response generation (such as, but not limited to, conversational response generation, machine translation, and image captioning).

[0126] The response evaluation metric 902 is suitable for Minimum Error Rate Training (MERT). It can also be used in conjunction with Pairwise Ranking Optimization (PRO), Maximum Entropy Model (maxent), Marginal Injection Relaxation Algorithm (MIRA), or any other learning algorithm as an alternative to Minimum Error Rate Training. That is, it can be used to tune the parameters of the response generation system. This has the effect of making the system output more similar to good responses and less similar to poor or suboptimal responses.

[0127] In this example, the response evaluation engine 900 is a component separate from the response generation system. In fact, in some examples, the response evaluation engine 900 may reside on a physical computing device separate from the computing device of the response generation system associated with the generating machine. In such examples, the response evaluation engine 900 may be located via a network (e.g., Figure 1 The network 120 shown here is used to receive machine-generated response and / or context-message pair data from one or more computing devices.

[0128] The session response in this example is a machine-generated response. The response evaluation engine 900 optionally includes a learning model 902 for generating evaluation scores for the machine-generated response. In some examples, the learning model 904 includes a neural network model 906.

[0129] In other examples, the learning model 904 may optionally include functionality associated with a response generation system for generating machine-generated responses that will be scored by the response evaluation engine. In other words, in some examples, the response generation system and the response evaluation engine may be embodied in a single component rather than two separate components.

[0130] Figure 10 This is an exemplary block diagram illustrating the training of a response evaluation engine. The response evaluation engine training 1000 is a component used to generate metric scores for responses generated by a given trained machine. The response evaluation engine training 1000 utilizes context-message-response session triples from a database 1002. The database can be any type of database or data storage device used to store data, such as, but not limited to, [examples not specified]. Figure 4 The context-message-response session triplet database 402 is shown.

[0131] In this example, the response evaluation engine 1000 is trained based on a training set of multi-reference context-message-response triples 1004. The training set of multi-reference context-message-response triples 1004 provides training weights for adjusting one or more parameters associated with the response evaluation engine 1000. Upon completion of this process, the trained response evaluation engine 1006 is ready to generate a metric score for a machine-generated response.

[0132] Figure 11 is an exemplary block diagram illustrating the response evaluation engine generating a metric score for a machine-generated response. The trained response evaluation engine 1006 receives a context-message pair 1100, a machine-generated response 1102, and a test set of multi-reference context-message-response triples 1104 as inputs to the response evaluation engine 1006.

[0133] In some examples, the evaluation metric 902 utilizes n-gram overlap metrics. These n-gram overlap metrics require a plurality of reference strings, such as the multi-reference response set 1108.

[0134] For a given user input message and conversation context, the evaluation metric 902 of the response evaluation engine 1006 assumes a set of one or more reference responses (R1,..., R N ) from the multi-reference response set. Each reference response in the set of reference responses is rated by a human with a score between negative one (-1) and positive one (+1). If the responses are rated on a different scale (e.g., 1 to 5), the scores are normalized in the range between negative one (-1) and positive one (+1).

[0135] The trained response evaluation engine 1006 computes a metric based on the context-message pair 1100, the machine-generated response 1102, and the test set of multi-reference context-message-response triples 1104 to generate a metric score 1110.

[0136] The metric score 1110 indicates a level of quality of the given machine-generated response 1102 with respect to the context-message pair 1100. Machine-generated responses of low quality can arise, for example, when a plurality of machine-generated responses are output acceptable or even required. Tasks with inherently diverse objectives can include, without limitation, machine translation, summary summarization, sentence compression, paraphrase generation, and generation of conversational interactions. A major obstacle for these tasks is the automation of evaluation, as the space of reasonable outputs can be large. Response evaluation metrics provide an automated evaluation of the quality of these outputs.

[0137] The quality of the outputted response indicates, for example, but is not limited to, whether the machine-generated response is the same or similar to a natural, human-generated response to the same or similar message and / or conversational context, whether the response is appropriate given the overall context of the conversation, whether the response is relevant to the message associated with the response, and / or whether the response is grammatically / syntactically correct.

[0138] The metric score can be a score on any scale used to grade the quality of the machine-generated response. In some examples, the metric can range from zero (0) to one (1). In other examples, the metric score can range from one to five. The metric score can be on a scale from one (1) to ten (10). In other examples, the metric score can range from negative one (-1) to positive one (+1). A lower metric score indicates that the response is not relevant to the pair and can be discarded. A higher metric score indicates that the response is relevant to the pair and can be selected as the response outputted to the digital assistant.

[0139] The metric score 1110 in some examples can be used to automatically optimize the response generation system. Using an optimization technique such as, for example, Minimum Error Rate Training (MERT), the parameters of the response generation system are updated to cause the system output to increasingly resemble higher quality responses that are positively rated and less resemble sub-optimal responses that are negatively rated. The metric score 1110 is utilized to update a set of one or more parameters associated with the response generation system. In other examples, the parameter set can be updated based on two or more metric scores associated with one or more machine-generated responses.

[0140] Thus, the response evaluation metric can be used to optimize the response generation system to improve the quality of the machine-generated responses. A high quality machine-generated response is, for example, but is not limited to, a response that is appropriate given the context of the conversation, relevant to the previous user-generated message, grammatically correct, syntactically correct, and / or similar to a natural, human-generated response.

[0141] Figure 12 Table 1200 illustrates the number of triples, the average, minimum, and maximum number of references used for debugging and testing. In this multi-reference extraction example, an information retrieval system is used to select candidate triples. The information retrieval system is calibrated to select candidate triples for which both the message and the response are similar to the selected message and the selected response. Each response in the set of candidate triples is rated for quality. References with a score of four (4) or better on a five (5) point scale are retrained for reference. As shown in Table 1200, this results in an average of 3.58 references per example. The average length of the responses in the multi-reference debugging and testing set is 8.75 and 8.13 tokens, respectively.

[0142] Figure 13is a flow diagram illustrating operations of a computing device for generating a metric score for a machine-generated response. In Figure 13 The process depicted in FIG. 13 can be implemented by a computing device, such as but not limited to the computing device 102 in Figure 1 and / or the computing device 200 in Figure 2 .

[0143] The process begins by extracting candidate response triples from at least one conversation data source to form a multi-reference response set at 1302. An evaluation metric is computed based on the machine-generated response and the multi-reference response set at 1304. A metric score is generated that indicates a quality of the machine-generated response at 1306 and the process then terminates.

[0144] Figure 14 is a flow diagram illustrating operations of a computing device for extracting a plurality of candidate responses from one or more conversation data sources. In Figure 14 The process depicted in FIG. 14 can be implemented by a computing device, such as but not limited to the computing device 102 in Figure 1 and / or the computing device 200 in Figure 2 .

[0145] The process begins by sampling context-message-response triples from a database to select a seed context-message-response triple at 1402. The seed context-message-response triple is annotated at 1404. The annotation can include a rating for each candidate response. In some examples, the rating is a human-generated rating.

[0146] A best candidate response is selected at 1406. The best candidate response can be the response that corresponds to the selected context-message data pair.

[0147] A determination is made as to whether additional response candidates are to be retrieved at 1408. If so, a next best candidate is selected from context-message-response triples from the database at 1410 and the process then terminates.

[0148] Now returning to operation 1408, upon a determination that additional response candidates are not to be retrieved, the process then terminates.

[0149] Figure 15 is a flow diagram illustrating operations of a computing device for selecting a multi-reference response set. In Figure 15 The process depicted in FIG. 15 can be implemented by a computing device, such as but not limited to the computing device 102 in Figure 1 and / or the computing device 200 in Figure 2 .

[0150] The process begins by extracting candidate response triples from a conversation data source at 1502. A response corresponding to the selected context-message pair is selected at 1504 to form a multi-reference response set and the process then terminates.

[0151] Figure 16 is a flowchart illustrating operations of a computing device for accessing ratings for each response in a multi-reference response set. In Figure 16 The process depicted in FIG. 15 can be implemented by a computing device such as, but not limited to Figure 1 computing device 102 in Figure 2 computing device 200 in

[0152] The process begins by receiving a human-generated rating for a reference response at 1602. A determination is made at 1604 as to whether the rating is on a scale of minus one to plus one. If so, the rating is added to the reference response in the multi-reference response set at 1606 and the process then terminates.

[0153] Now returning to operation 1604, upon determining that the rating is not on a scale of minus one to plus one, the rating is normalized at 1606. The normalized rating is a rating that is within the scale of minus one to plus one. The reference response and the rating for the reference response are added to the reference response set at 1608 and the process then terminates.

[0154] Figure 17 is an exemplary block diagram illustrating a digital assistant. The digital assistant 1700 is a device for managing personal information of a user. The digital assistant 1700 can include a set of sensors such as one or more cameras, GPS, audio sensors, touchscreens, temperature sensors, pressure sensors, orientation sensors, and any other type of sensor. The digital assistant 1700 includes a response generation system capable of receiving a user-generated input message and generating a natural language output response. The digital assistant 1700 can generate the output response in one or more formats, including a textual response, an audio response, a graphical response, or any other type of response.

[0155] Additional Examples

[0156] In some examples, the response evaluation engine identifies a number of overlaps between the selected machine-generated conversation response and the selected reference response. The response evaluation engine increases a metric score of the selected machine-generated conversation response upon determining that the rating of the selected reference response is a positive rating. The positive rating indicates that the selected reference response is a good reference response with respect to the selected context-message data pair associated with the selected machine-generated conversation response. The response evaluation engine can decrease the metric score of the selected machine-generated conversation response upon determining that the rating of the selected reference response is a negative rating.

[0157] In other examples, the response evaluation engine generates a first metric score associated with the first machine-generated response. The response evaluation engine updates the parameter set in response to the first machine-generated response to form a modified parameter set. The response evaluation engine generates a second metric score associated with the second machine-generated response. The response evaluation engine then updates the modified parameter set based on the second metric score. The parameter set is incrementally adjusted to increase the metric score.

[0158] Figure 1 and Figure 2 The functionality of various units in Figure 1 and Figure 2 may be performed by other units in Figure 1 and Figure 2 not shown (e.g., processors, web services, servers, applications, computing devices, etc.).

[0159] In some examples, Figure 8 , Figure 10 , Figure 11 , Figure 13 , Figure 14 , Figure 15 and Figure 16 The operations shown in may be implemented as software instructions encoded on a computer readable medium, in hardware programmed or designed to perform the operations, or in both.

[0160] While aspects of the disclosure have been described in reference to various examples, it will be understood that the examples are illustrative of the aspects of the disclosure and are not the only examples that can be implemented by aspects of the disclosure. Changes in, or substitutions of, elements of the examples described in conjunction with the above described concepts can be made without departing from the scope of the disclosure.

[0161] While aspects of the disclosure do not track personally identifiable information, examples have been described with reference to data monitored and / or collected from users. In some examples, users to whom data is aggregated can be provided a notification (e.g., via a dialog box or preference setting) and given an opportunity to give or deny consent for monitoring and / or collection. Consent can take the form of opt-in consent or opt-out consent.

[0162] Example Operating Environment

[0163] Exemplary computer-readable media include flash memories, digital versatile disks (DVDs), compact disks (CDs), floppy disks, and tape. By way of example, and not limitation, computer-readable media include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media are tangible and non-transitory and exclude carrier waves and a signal per se. In this context, computer storage media are implemented in hardware and exclude carrier waves and transmitted signals. Computer storage media for the present disclosure excludes signals per se. Exemplary computer storage media include hard disks, flash memory, and other solid-state memory. In contrast, communication media embody computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and include any information delivery media.

[0164] Although described in connection with an exemplary computing system environment, examples of the disclosure can be practiced with many other general purpose or special purpose computing system environments, configurations or devices.

[0165] Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with aspects of the disclosure include, but are not limited to, mobile computing devices, personal computers, server computers, handheld or laptop devices, multiprocessor systems, gaming consoles, microprocessor-based systems, set top boxes, programmable consumer electronics, mobile telephones, mobile computing and / or communication devices in wearable or accessory form factors (e.g., watches, glasses, headsets, or earphones), network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. Such systems or devices can accept input from a user in any way, including from input devices such as keyboards or pointing devices, via gesture input, proximity input (such as by hovering), and / or via voice input.

[0166] The examples of the disclosure can be described in the general context of computer-executable instructions, such as program modules, being executed by one or more computers or other devices. Generally, program modules include, but are not limited to, routines, programs, objects, components, and data structures that perform particular tasks or implement particular abstract data types. Aspects of the disclosure can be practiced in a variety of system configurations, including, but not limited to, cloud- based computing environments, computer- server network environments, and distributed computing environments. Furthermore, aspects of the disclosure can be practiced in or in association with a computer- network environment including a local area network (LAN), a wide area network (WAN), and the Internet.

[0167] In the example involving a general purpose computer, aspects of the disclosure transform the general purpose computer into a special purpose computing device when configured to execute the instructions described herein.

[0168] The examples illustrated and described herein, as well as examples not specifically described herein but within the scope of aspects of the disclosure, constitute exemplary means for generating a metric score for a machine-generated response. For example, Figure 1 , Figure 2 , Figure 8 , Figure 9 , Figure 10 and Figure 11 The units shown in FIGS. 1-8, such as the operations encoded to perform Figure 8 , Figure 13 , Figure 14 , Figure 15 and Figure 16 constitute exemplary means for generating a multi-reference response set, exemplary means for computing an evaluation metric, and exemplary means for generating a metric score.

[0169] As alternatives to or in addition to the examples described herein, examples include any combination of the following:

[0170] extracting candidate response triples from at least one conversational data source to form a multi-reference response set further comprises extracting context-message-response triples from at least one conversational data source to form a plurality of candidate response triples, wherein a given candidate response triple includes a human-generated message, a conversational context, and a reference response corresponding to the human-generated message, wherein the human-generated message and the conversational context form a context-message data pair

[0171] - the selected context-message data pair includes a selected human-generated message and a selected conversational context of the message, and further includes selecting a response from a plurality of candidate response triples based on the conversational context of the response to form a reference response in the multi-reference response set, wherein the conversational context associated with the reference response corresponds to the conversational context of the selected machine-generated response

[0172] - the selected context-message data pair includes a selected human-generated message and a selected conversational context of the message, and further includes selecting a response from a plurality of candidate response triples based on the conversational context of the response to form a reference response in the multi-reference response set, wherein the conversational context associated with the reference response corresponds to the conversational context of the selected machine-generated response

[0173] - the conversational context of the message includes linguistic context data and non-linguistic context data, wherein the linguistic context data includes message-response data pairs preceding the selected message and the selected machine-generated response in the conversation

[0174] - the context-message-response n-tuples are mined from the at least one conversational data source via a network connection, wherein the at least one conversational data source is at least one of a social media source, wherein the social media source provides conversational data in at least one format, wherein the format of the conversational data includes a textual format, an audio format, or a visual format

[0175] - the rating of a given reference in the multi-reference response set is a human-generated rating, and further includes accessing a rating of a given reference in the multi-reference response set, wherein the rating for the given reference response in the multi-reference response set indicates a quality of the given response with respect to the context-message data pair associated with the given reference response

[0176] - in determining the rating for the given reference response in the multi-reference response set is a rating on a scale other than a minus one to plus one scale, the rating is normalized to form a normalized rating in a range from minus one to plus one, wherein a negative value rating indicates that the given reference response is suboptimal with respect to the selected context-message data pair

[0177] - the multi-reference response set is a test set of multi-reference responses, and further includes training a response evaluation engine based on a training set of multi-reference context-response-message n-tuples mined from the at least one conversational data source, wherein training the response evaluation engine further includes computing an evaluation metric based on the training set of multi-reference context-message-response n-tuples to train a set of weights associated with the response evaluation engine

[0178] - the metric score is a score within a scale from zero to one, and wherein generating the metric score further comprises calculating a number of word sequence overlaps between the selected machine-generated response and reference responses in the set of multiple reference responses, wherein an overlap of zero indicates that there are no common words between the selected machine-generated response and at least one reference response, and wherein an overlap of one indicates that the selected machine-generated response is identical to a reference response

[0179] - determining a rating of the reference response when determining the overlap between the selected machine-generated response and the reference response; increasing the metric score when determining that the rating of the reference response is a positive rating indicating that the reference response is a good reference response with respect to the context-message data pair

[0180] - the response evaluation engine performs to identify a number of overlaps between the selected machine-generated response and the selected reference response; increasing the metric score of the selected machine-generated response when determining that the rating of the selected reference response is a positive rating indicating that the selected reference response is a good reference response with respect to the selected context-message data pair associated with the selected machine-generated response; and decreasing the metric score of the selected machine-generated response when determining that the rating of the selected reference response is a negative rating

[0181] - the response evaluation engine performs to generate a first metric score associated with a first machine-generated response; update the parameter set to form a modified parameter set in response to the first machine-generated response; generate a second metric score associated with a second machine-generated response; and update the modified parameter set based on the second metric score, wherein the parameter set is incrementally adjusted to increase the metric score

[0182] - the response evaluation engine performs to compute the evaluation metric based on a training set of multiple reference context-message-response n-tuples

[0183] - the at least one conversation data source is at least one of the social media sources, wherein the social media sources provide conversation data in at least one format, wherein the format of the conversation data includes a textual format, an audio format, or a visual format

[0184] - wherein the conversation context associated with the reference response corresponds to the conversation context of the selected machine-generated response, wherein the conversation context includes linguistic context data and non-linguistic context data, wherein the linguistic context data includes message-response data pairs in the conversation preceding the selected message and the selected machine-generated response

[0185] - a given reference response in the set of reference responses includes a rating, and wherein the response evaluation engine, when executed, further causes the at least one processor to normalize the rating to form a normalized rating in a range from negative one to positive one, where a negative value rating indicates that the given reference response is sub-optimal with respect to the selected context-message data pair.

[0186] The order of execution or performance of the operations in examples of the disclosure illustrated and described herein need not be performed in the order shown unless otherwise specified. That is, embodiments of the disclosure can include additional or fewer operations than those disclosed herein. For example, it is contemplated that executing or performing a particular operation before, concurrently, or after another operation is within the scope of aspects of the disclosure and examples of the disclosure. In some examples, operations can be performed in parallel.

[0187] In introducing elements of aspects of the disclosure, the articles "a," "an," "the," and "said" are intended to mean that there are one or more of the elements. The terms "comprising," "having" and "including" are intended to be inclusive and mean that there can be additional elements other than the listed elements. The term "exemplary" is intended to mean "an example of." The phrase "one or more of the following: A, B, and C" means "at least one of A and / or at least one of B and / or at least one of C."

[0188] Having thus described aspects of the disclosure in detail, it will be apparent that modifications and variations are possible without departing from the scope of aspects of the disclosure as defined in the appended claims. Since modifications and variations are possible using the above construction, products, and methods, the disclosure as described in the above description and as illustrated in the drawings is intended to be illustrative, and not restrictive. Accordingly, it is intended that the scope of aspects of the disclosure be limited only by the appended claims.

Claims

1. A computer-implemented method for automatically evaluating machine-generated responses, the method comprising: The extraction component of the computing device extracts candidate context-message-response n-tuples from at least one session data source, wherein each candidate context-message-response n-tuple includes a human-generated message, the context of the human-generated message, and a reference response corresponding to the human-generated message; A multi-reference response set is formed by selecting at least one candidate context-message-response n-tuple from the candidate context-message-response n-tuples extracted by the extraction component based on at least one of the message and context associated with the machine-generated response, wherein the responses in the multi-reference response set include a rating for the response relative to the context-message data pair associated with the response; The ratings are normalized to form normalized ratings in the range from negative one to positive one, wherein a negative rating indicates that the multi-reference response in the multi-reference response set is suboptimal. An evaluation metric for the machine-generated response is calculated by at least one processor based on the multi-reference response set; as well as The at least one processor generates a metric score for the machine-generated response based on the evaluation metric, the metric score indicating the quality of the machine-generated response relative to the multi-reference response set; as well as The parameter set associated with the generation of the machine-generated response is automatically updated based on the metric score. The evaluation metric mentioned therein is calculated based on the following formula:

2. The computer-implemented method according to claim 1, further comprising: The response is selected from the extracted candidate context-message-response n-tuples based on the context of the message associated with the response to form a reference response in the multi-reference response set, wherein the message associated with the reference response corresponds to a human-generated message.

3. The computer-implemented method according to claim 1, further comprising: The response is selected from the extracted candidate context-message-response n-tuples based on the session context of the response to form a reference response in the multi-reference response set, wherein the session context associated with the reference response corresponds to the session context of the machine-generated response.

4. The computer-implemented method of claim 3, wherein the session context of the message includes language context data and non-language context data, wherein the language context data includes message-response data pairs prior to the selected message and the selected machine-generated response in the session.

5. The computer-implemented method according to claim 1, further comprising: The candidate context-message-response n-tuple is extracted from the at least one session data source via a network connection, wherein the at least one session data source is at least one social media source, wherein the social media source provides session data in at least one format, wherein the format of the session data includes text format, audio format, or visual format.

6. The computer-implemented method of claim 1, wherein the ratings of each multi-reference response in the multi-reference response set are human-generated ratings, and further comprises: Access the ratings of each of the multi-reference responses in the multi-reference response set.

7. The computer-implemented method of claim 1, wherein the multi-reference response set is a test set of multi-reference responses, and further comprises: The response evaluation engine is trained based on a training set of multi-reference context-response-message n-tuples extracted from the at least one session data source, wherein training the response evaluation engine further includes calculating the evaluation metric based on the training set of multi-reference context-message-response n-tuples to train a weight set associated with the response evaluation engine.

8. The computer-implemented method of claim 1, wherein the metric score is a score on a scale from zero to one, and wherein generating the metric score further comprises: Calculate the number of word sequence overlaps between the machine-generated response and the reference responses in the multi-reference response set, where zero overlap indicates that there are no common words between the machine-generated response and the reference response, and one overlap indicates that the machine-generated response is the same as the reference response.

9. The computer-implemented method according to claim 8, further comprising: When determining the overlap between the response generated by the machine and the reference response, a rating of the reference response is determined; When the rating of the reference response is determined to be a positive rating, the metric score is increased; as well as When the rating of the reference response is determined to be negative, the metric score is reduced.

10. A system for automatically evaluating machine-generated responses, the system comprising: At least one processor; as well as A memory storage device associated with the at least one processor, the memory storage device including a memory region storing a response evaluation engine, wherein the at least one processor executes the response evaluation engine to: The ratings are normalized to form normalized ratings in the range of negative one to positive one, where negative ratings indicate that the multi-reference response in the multi-reference response set is suboptimal. An evaluation metric is calculated for at least one machine-generated response based on a multi-reference response set, a rating set associated with the multi-reference response set, and context data. The multi-reference response set is selected from candidate context-message-response n-tuples based on at least one of the message and context associated with the at least one machine-generated response. The candidate context-message-response n-tuples are extracted from at least one session data source. Each candidate context-message-response n-tuple includes a human-generated message, the context of the human-generated message, and a reference response corresponding to the human-generated message. The responses in the multi-reference response set include a rating for the response relative to the context-message data pair associated with the response. Generate at least one metric score indicating the quality of the response generated by the at least one machine relative to at least one multi-reference response from the set of multi-reference responses; as well as The parameter set associated with the generation of the at least one machine-generated response is automatically updated based on the at least one metric score. The evaluation metric mentioned therein is calculated based on the following formula:

11. The system of claim 10, wherein the metric score is a score on a scale from zero to one, and wherein the at least one processor further executes the response evaluation engine to: Calculate the number of word sequence overlaps between the machine-generated response and the reference responses in the multi-reference response set, where zero overlap indicates that there are no common words between the machine-generated response and the reference response, and one overlap indicates that the machine-generated response is the same as the reference response.

12. The system of claim 10, wherein the at least one processor further executes the response evaluation engine to: Indicates the amount of overlap between the response generated by the machine and the reference response; When the rating of the reference response is determined to be positive, the metric score of the machine-generated response is increased; and When the rating of the reference response is determined to be negative, the metric score of the machine-generated response is reduced.

13. The system of claim 10, wherein the at least one processor further executes the response evaluation engine to: Generate a first metric score associated with the response generated by the first machine; The parameter set is updated in response to the response generated by the first machine to form a modified parameter set; Generate a second metric score that is associated with the response generated by the second machine; as well as The modified parameter set is updated based on the second metric score, wherein the parameter set is incrementally adjusted to increase the metric score.

14. The system of claim 10, wherein the at least one processor further executes the response evaluation engine to: The evaluation metric is computed based on a training set of multi-reference context-message-response n-tuples.

15. One or more computer storage media embodying computer-executable components, said components comprising: Extraction component, when executed, causes at least one processor to: Extract multiple candidate context-message-response n-tuples from at least one session data source, wherein each candidate context-message-response n-tuple includes a human-generated message, the context of the human-generated message, and a reference response corresponding to the human-generated message; as well as At least one candidate context-message-response n-tuple is selected from the plurality of candidate context-message-response n-tuples based on at least one of the messages and contexts associated with the machine-generated response to form a multi-reference response set, wherein the responses in the multi-reference response set include a rating for the response relative to the context-message data pair associated with the response; as well as A response evaluation engine, when executed, causes at least one processor to: The ratings are normalized to form normalized ratings in the range from negative one to positive one, wherein a negative rating indicates that the multi-reference response in the multi-reference response set is suboptimal. A metric score is generated for the machine-generated response based on the multi-reference response set, the session context of the machine-generated response, and an evaluation metric, the metric score indicating the quality of the machine-generated response relative to the multi-reference response set; as well as The set of parameters associated with the generation of the machine-generated response is automatically updated based on the metric score. The evaluation metric mentioned therein is calculated based on the following formula:

16. The computer storage medium of claim 15, wherein the at least one session data source is at least one social media source, wherein the social media source provides session data in at least one format, wherein the format of the session data includes text format, audio format, or visual format.

17. The computer storage medium of claim 15, wherein the response evaluation engine, when executed, further causes at least one processor to: The response is selected from the plurality of candidate context-message-response n-tuples based on the session context of the response to form a reference response in the multi-reference response set, wherein the session context associated with the reference response corresponds to the session context of the machine-generated response, and wherein the session context includes language context data and non-language context data. The language context data includes message-response data pairs in the session prior to the selected message and the machine-generated response.

Images