Self-learning digital assistant

Abstract

Systems, methods, and computer storage media are disclosed for improving digital assistant techniques and extending device functionality by learning mechanisms to improve the ability of computing devices to accurately respond to user commands. A response activity mode may be determined from a set of computing devices obtained during the operational monitoring mode. The operation monitoring mode can be initiated when a user command is determined to be a new command or an unknown command. The response activity pattern may be used to generate a response profile indicating an activity to be performed using one or more user devices to perform a task associated with the user command. When an indication of a previously unknown user command to perform the task is received, the generated response profile can be used to perform the designed task by performing the activity using one or more user devices. In this manner, activities that should be performed to perform the task can be learned based on monitored user activities associated with the one or more user devices.

Description

[0001] This application is a divisional application of the application with the application number 202080024692.7 and the invention title of "Self-learning Digital Assistant" filed on March 19, 2020. Background Art

[0002] People increasingly interact with computing devices and rely on these devices to obtain information, recommendations, and other services to assist them in their daily tasks. For example, an individual can interact with a smart speaker, a smart watch, or their mobile device by speaking a command (such as asking for directions or turning on a light). Many of these "smart" computing devices include smart features for responding to a user's command, such as a digital assistant application running on the computing device. However, these computing devices (or their digital assistant applications) often do not respond to the user's command because they do not know how to perform the task associated with the command. In such cases, these smart devices do not seem smart at all and can frustrate the user. Summary of the Invention

[0003] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used alone to assist in determining the scope of the claimed subject matter.

[0004] Embodiments of the techniques described in this disclosure relate to systems and methods for improving digital assistant technology and the ability of a computing device to accurately respond to user commands by extending the functionality of a digital assistant through a learning mechanism. At a high level, an interactive computing device (such as a smart speaker or a mobile device) or a digital assistant program running thereon can learn to perform a specific response activity associated with a specific user command. The user command can include a request to perform a task, such as turning on a light. For example, when an unknown command is received from a user, instead of ignoring the command or simply responding to the user with an indication that the command is unknown, the computing device monitors the user activity to learn the actions performed by the user. Specifically, when it is determined that a command received from the user is not recognized, the computing system (or digital assistant) can enter a special operation mode, in which the user activity on the computing device that occurs after the unrecognized command is monitored. The user activity can be monitored on the computing device or on another computing device, and the user can consent to the monitoring operation. In this way, the computer system learns the actions to be performed to implement a task in response to a new or previously unknown user command.

[0005] For example, in an embodiment, a user-interactive computing device can receive an indication of a user command to perform a task. When it is determined that the user command does not correspond to a known response action, an operation monitoring mode can be initiated. Specifically, a user command library or a collection of known commands and / or responses can be utilized to determine whether the received user command is known or unknown. Thus, when the user command does not exist in the set or library of known commands, the operation monitoring mode can be initiated. In some embodiments, when it is determined that the user command is unknown, the computing device (or a digital assistant application or software service running on the computing device) can enter a fault state, during which monitoring of user-related activities is performed. The fault state can be characterized by enhanced user activity monitoring performed to learn the response action to be associated with the unknown user command for the system. In this way, the response activity can be learned from the monitored user activities. In some cases, user consent can be obtained before performing the enhanced user activity monitoring, and in some cases, the computing device can provide an indication to the user that the operation monitoring mode (or learning mode) is occurring.

[0006] When initiating the operation monitoring mode, a computing device associated with the user (i.e., the "user device") can employ one or more sensors and / or monitoring software services to generate data related to the user's activities on the (one or more) user devices. Such user activities can be monitored as (one or more) response activity events associated with the (one or more) user devices (which can be performed by the user or under the user's guidance). The (one or more) response activity events can be monitored, tracked, and used to determine a response activity pattern. As described herein, the response activity pattern can include, but is not limited to, patterns based on time, location, content, or other context. In some embodiments, a response activity pattern can be determined based on response activity events related to user activities associated with one or more user devices or otherwise determinable via one or more user devices.

[0007] Based on the determined response activity pattern, a response profile corresponding to the user command can be determined and used to provide an improved user experience. For example, the response profile can specify one or more user commands and corresponding actions to be performed to implement the task associated with the command. In some embodiments, the response profile can be determined based on user activities learned from multiple users, and in some embodiments, the response profile can further include a user-device mapping that enables the actions to be performed on different types of user devices, such as smart lights or smart thermostats of different brands. Examples of the improved user experience further described herein can include an improved user experience of interacting with a computing device based on the computing device understanding a previously unknown user command and causing an action to be performed to implement the task associated with the user command.

[0008] In some embodiments, the response activity pattern can be made available to one or more applications and services that consume the information to provide an improved user experience. For example, the response activity pattern (or information derived from the pattern, such as the actions to be performed to implement a task) can be included in a response profile provided on a server or made accessible to other computing devices, such that the devices can learn to respond to new or unknown commands. In this way, digital assistant technology is improved by enabling a computing device (or a digital assistant running thereon) to learn to respond to new or previously unknown commands, thereby expanding the functionality of the computing device and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Aspects of the present disclosure are described in detail below in conjunction with the accompanying drawings, in which:

[0010] Figure 1 is a block diagram of an exemplary operating environment suitable for implementing the present disclosure;

[0011] Figure 2 is a diagram depicting an exemplary computing architecture suitable for implementing aspects of the present disclosure;

[0012] Figure 3A depicts an example of a conventional voice-based interaction between a user and a digital assistant (which is a smart speaker) operating on a computing device;

[0013] Figure 3B depicts an exemplary voice-based interaction between a user and a computing system according to an embodiment of the present disclosure;

[0014] Figure 4 and Figure 5 each depict a flowchart of a method for a computing system to learn to perform a new task associated with a user command to expand its functionality by monitoring user activities according to an embodiment of the present disclosure; and

[0015] Figure 6 is a block diagram of an exemplary computing environment suitable for implementing embodiments of the present disclosure. Detailed Description

[0016] The subject matter of aspects of the present disclosure is described herein in detail to meet statutory requirements. However, the description itself is not intended to limit the scope of this patent. Rather, the inventors have contemplated that the claimed subject matter might also be embodied in other ways, to include combinations of different steps or steps similar to those described in this document, in conjunction with other present or future technologies. Additionally, although the terms "step" and / or "block" may be used herein to imply different elements of a method employed, the terms should not be construed as implying any particular order among or between the various steps disclosed herein, unless and except when the order of individual steps is explicitly described.

[0017] Aspects of the techniques described herein relate to systems, methods, and computer storage media for improving digital assistant technology and enhancing the ability of a computing device to accurately respond to user commands by expanding device functionality through a learning mechanism. A user command generally can refer to a command, statement, or declaration that indicates or corresponds to a request to perform a particular task or operation. A digital assistant or personal digital assistant generally also may be referred to as a virtual assistant, includes one or more computer programs, and can operate on one or more computing devices or across one or more computing devices. For example, one such virtual assistant is by Corporation's Assistant. Specifically, embodiments of the computing devices (and / or digital assistants or software applications running thereon) described herein can learn to perform tasks in response to specific user commands. For example, during a user interaction with a computing device, the user can provide commands to the computing device by speaking (i.e., voice commands), by gestures, touch, or other input means. The commands can be received by a user interface or sensor, such as a microphone, camera, touch screen, keyboard, or other input means or user interface. The computing device can include smart devices, such as smart speakers, mobile devices with apps or interactive services, digital assistant programs running on a computing device, or other examples of the computing devices described herein. The user commands can include requests to perform specific tasks, such as, for example, but not limited to: turning on a light, playing or recording a TV program, launching an application on a computing device, navigating to a website, scheduling a meeting, booking a meeting or venue, configuring settings on a computing device or appliance, or performing an operation or series of operations using one or more computing devices (e.g., launching an app or navigating to a specific website and initiating an operation, such as a restaurant reservation, travel reservation, purchase, playing a video, retrieving and presenting or reading information (such as a news feed) or one or more other operations). The tasks performed in response to user commands can include a set of one or more operations performed via the computing device or another computing device.

[0018] As previously described, as the prevalence of computing devices has increased, user interactions with computing devices have become more frequent. For example, many people now have smart speakers or other "smart" computing devices in their homes, in their vehicles, or wear such smart devices on their bodies. User interactions in which the user requests the device to perform a task on behalf of the user have also become more common. When a computing device knows a specific user command (whether the command is pre-programmed or learned), the computing device's response to the command may involve performing (or causing to be performed) one or more operations (or tasks) associated with the user command. In some cases, a portion of these operations can be performed by another computing device. For example, if the user command is "turn on the thermostat," the task of adjusting the temperature setting on the thermostat can be performed by a computing device, such as a mobile device on which an app for controlling the thermostat is installed, and / or by another computing device (such as a smart thermostat). The prevalence of these smart computing devices and the frequency of user interactions with them have led users to have an expectation that these devices will respond to the user's commands.

[0019] However, when a computing device does not know a user command, the task (or corresponding operation(s)) will not be initiated and / or performed. When these smart devices are unable to perform the requested task, users can become frustrated. InFigure 3A illustrates an example thereof illustratively, Figure 3A shows the interaction between a user and a conventional computing device (smart speaker). Specifically, Figure 3A shows a series of interactions between user 305 and smart speaker 315. In the first interaction (scene 310), in response to user command 311 to perform the task of having the TV record the Seahawks game, the conventional smart speaker (or more specifically, the conventional digital assistant running on the smart speaker) responds with "I don't understand" at 317. As shown in scene 320, user 305 starts to lose patience with smart speaker 315. In response to user command 321 to tell smart speaker 315 that he is cold (which may mean performing the task of turning up the heat), smart speaker 315 responds with 371 ("I don't understand"). By scene 330, user 305 becomes very annoyed. As shown in scene 330, in response to user command 331 to tell smart speaker 315 to turn off the lights (e.g., perform the task of turning off the lights), smart speaker 315 responds with 371 ("I don't understand"). At scene 340, user 305 becomes even more frustrated. In scene 340, in response to user command 341 to tell smart speaker 315 to turn on the home alarm system (e.g., perform the task of activating the home alarm system), smart speaker 315 responds with 371 ("I don't understand"). By scene 350, user 305 is very angry. In response to user command 351 to tell smart speaker 315 to play his favorite song (e.g., perform the task of playing the user's favorite song), smart speaker 315 responds with 371 ("I don't understand"). In scene 360, user 305 is annoyed with the smart speaker (e.g., smart speaker 315), thinking that the smart speaker is not very smart.

[0020] Generally, when a conventional smart device receives an unrecognized command, the command is ignored by the device (or the program running on the device), or the device (or the program running on the device) may respond with an indication that the command is unknown. Contrary to the conventional techniques and the Figure 3A frustrating experiences depicted therein, embodiments of the techniques described herein improve the operation of computing devices by extending the functionality of the computing devices through a learning mechanism to effectively respond to user commands. In this way, the embodiments described herein also improve the user experience of interacting with these computing devices or digital assistant applications running thereon. Instead of ignoring unknown commands or only responding to the user with an indication that the command is unknown, the embodiments described herein monitor the user activities performed after receiving an unknown command in order to learn the actions performed by the user and associate those actions with the command.

[0021] Specifically, when it is determined that a command received from a user is not recognized, the computing system (or digital assistant) can enter a special operation mode, in which the user's activities are monitored. The user activities can be monitored on the computing device or on one or more other computing devices. Sensors on the computing device and / or on one or more user devices can be used to collect data corresponding to the user activities over time. Based on the monitored user activity data, specific operations associated with performing a task corresponding to the user command can be learned. In this way, the computing system (or digital assistant application) learns the operations to be performed in response to the user command in order to facilitate the execution of the task. In some cases, user consent can be obtained before performing the user activity monitoring, and in some cases, the computing device can provide an indication to the user that an operation monitoring mode (or learning mode) is taking place.

[0022] Thus, at a high level, aspects of the present disclosure relate to techniques that enable a computer system to learn to perform new tasks in response to new or previously unknown user commands. In some embodiments, a task can be understood as a set of one or more computing device operations that a user desires to be performed by one or more user devices. Such operations can involve activities associated with one or more user devices. When it is determined that a user command is not recognized, the computing system can initiate monitoring of the user activities. The monitored user activities can be utilized to determine the operations to be performed in order to execute a task related to the user command. For example, the next time the computer system receives an indication of a previously unknown user command, the system can utilize the knowledge it has learned by performing activities using one or more user devices to execute the specified task. In this way, the computer system learns what tasks to perform in response to new or previously unknown user commands.

[0023] In one embodiment, an indication of a user command to perform a task can be received by a user interactive computing device. User commands or requests can generally be received as voice commands, gestures, input into the computing device through a touch screen, keyboard, camera, microphone, other sensors, or other input units. As described herein, the command can include a request to perform a specific task and / or can correspond to a set of one or more operations to be performed in order to execute the task. The task can be executed by one or more user computing devices.

[0024] In some embodiments, a library or collection of known commands and / or responses can be utilized to determine whether a received command is known or unknown. Additionally, in some instances, determining whether a user command is known or unknown may involve performing a similarity comparison. In these embodiments, aspects of the received user command can be subjected to a similarity comparison to determine its similarity to other known user commands. For example, using feature similarity techniques such as clustering or Word2Vec, a measure of similarity, such as a similarity score, can be determined. In some embodiments, if the similarity score is high or meets a predetermined threshold, it can be determined that the command is similar to a known command, and the operation corresponding to that known command can be executed. However, if the similarity score does not meet the predetermined threshold, it can be determined that the command is not similar to the set of known commands.

[0025] When it is determined that the user command is unknown, an operation monitoring mode can be initiated. In some embodiments, user activities can be monitored via a computing device and other user devices, and the user activities can be identified from the monitored user data, as described herein. In some embodiments, an application or service running on the user device can be used to facilitate user activity monitoring. Alternatively or additionally, an application or service running in the cloud can be used to facilitate the user activity monitoring, which can scan, poll, or otherwise monitor the user device, or can detect online activities associated with the user device, such as http requests or other communication information, or otherwise receive information about user activities from the user device.

[0026] In some cases, when it is determined that the user command is unknown, the computing device (or a digital assistant application or software service running on the computing device) can enter a fault state during which monitoring of user-related activities is performed. For example, the fault state can include initiating a program routine or service for performing monitoring of user activities. In some embodiments, the user can be informed of the fault state (or more generally, the operation monitoring mode), such as by providing a notification or an indication by the device, such as changing the color of a light on the device or changing the color of an app associated with the device, or similarly using the device to emit a sound or a sound provided by an app associated with the device, or otherwise providing the user with a notification that the device is monitoring activities to learn how to respond to unknown commands (e.g., which can be output by the presentation component 220).

[0027] When in a monitoring mode, user activities can be monitored as (one or more) response activity events (which may be performed by a user) with respect to one or more user devices. The response activity events can generally be determined based on at least one monitored user activity operation performed on at least one user device (e.g., the user manually performs a response activity event to perform a desired task using one or more user devices). In some embodiments, the (one or more) response activity events may be related to (one or more) direct interactions of the user with one or more devices during monitoring initiated based on a user command. Additionally or alternatively, the (one or more) response activity events may be related to (one or more) user interactions with one or more devices related to application (or app) - related activities, such as application usage, which may include usage duration, launches, files accessed via or in conjunction with the application, or content associated with the application. The term "application" or app is used broadly herein and generally refers to a computer program or computer application, which may include one or more programs or services and may run on (one or more) user devices or in the cloud.

[0028] As further described herein, in some embodiments, a user device may employ one or more sensors to generate data related to (one or more) response activity events of a (one or more) user via the (one or more) user devices. In some embodiments, these response activity events can occur and be detected during an operation monitoring mode. Specifically, sensors on one or more user devices associated with the user can be used to collect data corresponding to such response activity events over time (e.g., in the monitoring mode).

[0029] The response activity event data can be monitored, tracked, and used to determine response activity patterns. The term "response activity pattern" is used broadly herein and can refer to multiple user interactions with one or more user devices, activities of a user on or associated with one or more user devices, events (including actions) associated with user activities, or any type of response activity event that can be determined via a computing device, where the multiple interactions, actions, events, or activities share common characteristics or features. In some embodiments, these common characteristics or variables can include features that characterize the response activity event, time, location, or other types of context information associated with the response activity event, as further described herein. Examples of response activity patterns can include, but are not limited to, response activity patterns corresponding to user commands that are based on time (e.g., the user command "turn on the lights" at 7:00 a.m. is associated with a response activity pattern of turning on the master bedroom lights, master bathroom lights, and kitchen lights, whereas the user command "turn on the lights" at 5:30 p.m. is associated with a response activity pattern of turning on the kitchen lights, living room lights, and dining room lights), location (e.g., the user command "play music" when at work is associated with a response activity pattern of playing classical music, whereas the user command "play music" when at home is associated with a response activity pattern of playing home music), content (e.g., the user command "it's a bit cold here" is associated with a response activity pattern of turning up the smart thermostat by two degrees, whereas the user command "I'm cold" is associated with a response activity pattern of turning up the smart thermostat by four degrees), or other context, as described herein.

[0030] In some embodiments, the response activity events can be associated with or used to generate a response profile corresponding to the user command and are associated with the task based on the history of the sensed user activities. In this way, when an indication of a user command is received in the future, the (one or more) response activity events associated with the task can be determined and used to cause an operation to be performed to implement the task. These (one or more) response activity events can be associated with the task. Based on this response activity information, the computer system can learn the response activity events (according to the user command) corresponding to the user command and associated with the user activities of the (one or more) user devices involved in performing the task.

[0031] More specifically, based on the determined patterns of response activity events, one or more response profiles regarding the expected interactions (e.g., (one or more) response activity events) associated with one or more user devices corresponding to a task in response to a user command can be generated and used to provide an improved user experience. In some embodiments, the response activity patterns or the response profiles generated therefrom can be made available to one or more applications and services that consume the information to provide an improved user experience in interacting with a computing device or a digital assistant application operating thereon (e.g., understanding previously unknown user commands and causing actions to be performed to implement the task associated with the user command). For example, in one embodiment, the response activity pattern information and / or the generated response profiles can be provided via an application programming interface (API) such that third-party applications and services can use it, such as by determining and / or providing user-related device mappings, suggestions, or other information or services based on the learned response activity patterns.

[0032] Accordingly, in one exemplary aspect, user data is received from one or more data sources. The user data can be received by collecting user data using one or more sensors or components on the one or more user devices associated with the user. Also in combination with Figure 2 the example of user data described with respect to the user data collection component 210 can include information about the one or more user devices and response activity information associated with the user devices (e.g., user interactions with the one or more devices, app usage, online activities, searches, calls, usage durations, and other user interaction data). The received user data can be monitored, analyzed (e.g., feature extraction), and information about the one or more response activity events can be stored (e.g., in a user profile, such as Figure 2 the user profile 240) to facilitate pattern analysis.

[0033] Specifically, user data can be analyzed to detect various features associated with the (one or more) response activity events. The detected user actions or (one or more) response activity events can include, for example, direct interactions with one or more user devices, actions to launch an application, or other actions similar to those described herein, and can be recorded together with associated context data, for example, by utilizing corresponding timestamps, location stamps, and / or associating the (one or more) response activity events with other available context information to record the observed user actions. In some embodiments, such recording can be performed on each user device so that response activity patterns can be determined across devices. Additionally, in some embodiments, cloud-based sources of user activity information, such as an online user calendar or user activities determined from social media posts, emails, etc., can be used. These sources can also be used to provide additional context to the (one or more) response activity events detected on the (one or more) user devices. Examples of context information are described further in conjunction with the context information determiner 284 in Figure 2 The context information determiner 284 in

[0034] Based on the activity log or user activity data, the (one or more) historical response activity events can be determined and provided to the response activity pattern engine. Based on the analysis of the (one or more) historical response activity events and, in some cases, based on current sensor data regarding the (one or more) response activity events, a set of one or more possible response activity patterns can be determined. Specifically, in some embodiments, the response activity pattern engine can analyze the (one or more) historical response activity event information to identify response activity patterns, which can be determined by detecting repetitions of similar user actions or routines.

[0035] In some embodiments, a corresponding confidence weight or confidence score can be determined with respect to the response activity pattern. The confidence score can be based on the strength of the pattern, which can be determined by the number of observations used to determine the pattern, the frequency with which the (one or more) response activity events are consistent with the pattern, the age or freshness of the activity observations, the number of features common to the observations of the (one or more) response activity events that make up the pattern, or similar metrics. In some cases, the confidence score can be considered when providing a personalized user experience or other improved user experience. Additionally, in some embodiments, a minimum confidence score may be required before using the response activity pattern to provide such an experience or other service. For example, in one embodiment, a threshold of 0.6 (or just over fifty percent) is used such that only response activity patterns with a 0.6 (or greater) likelihood of performing the desired task corresponding to the user command are used to initiate a response activity event for task execution. However, in cases where confidence scores and thresholds are used, the determined response activity patterns of the (one or more) response activity events with confidence scores below the threshold can still be monitored because additional observations of the (one or more) response activity events can increase the confidence in a particular pattern.

[0036] In some embodiments, the crowdsourced response activity event history can also be used in combination with the user's own (one or more) response activity event history. For example, for a given user, a set of other users similar to the given user can be identified based on having common characteristics or traits with the given user. This may include other users located near the given user, the given user's social media friends, ownership of similar user devices (which can be determined based on an analysis of the context information associated with the given user), other users having similar response activity events, and so on. Information about the response activity event history from other users can be relied upon to infer patterns of the (one or more) response activity events for the given user. This can be particularly useful in cases where there is little or no (one or more) response activity event history for the given user, such as when the user is a new user. In some embodiments, user activity information from similar users can be imputed to the new user until there is sufficient (one or more) response activity event history available for the new user to determine a statistically reliable response activity pattern, which can be determined based on the number of observations included in the (one or more) response action event history information or the statistical confidence of the determined response activity pattern, as further described herein. In some cases, when the (one or more) response activity event history is from other users, a lower confidence can be assigned to the resulting inferred response pattern for the given user.

[0037] In some embodiments, semantic analysis is performed on the (one or more) response activity events and associated information to characterize aspects of the (one or more) response activity events. For example, features associated with the response activity events can be classified (such as by type, similar time range, or location), and relevant features can be identified for determining similarity or relationship proximity to other response activity events, such as having one or more common characteristics, including category and relevant features. In some embodiments, semantic knowledge representation, such as a relational knowledge graph, can be employed. In some embodiments, the semantic analysis can use rules, such as association or conditional logic, or classifiers.

[0038] The semantic analysis can also be used to characterize information associated with the response activity events, such as determining that a location associated with the user activity corresponds to a center or place of user interest, such as the user's home, office, or gym, based on access frequency. (For example, the user's home center can be determined as the location where the user spends most of the time between 8 PM and 6 AM). Similarly, the semantic analysis can determine the time of day corresponding to work hours, lunch, or commute time.

[0039] In this way, the semantic analysis can provide additional relevant features of the response activity events that can be used to determine response activity patterns. For example, in addition to determining the specific device that the user wishes to activate using a user command such as "play music on the garage speaker", the music category associated with the task of playing music on the garage speaker can also be determined (e.g., using the response activity pattern), such as EDM music. In an embodiment, different categories can have additional response activity events (e.g., bass boost for EDM music). Similarly, the semantic analysis can classify the response activity events as work - or home - related based on the characteristics of the activity (e.g., playing classical music in response to the user command "play music" at a location corresponding to the user's office can be determined as a work - related activity, while playing home music in response to the user command "play music" at a location corresponding to the user's home can be determined as a home - related activity). Then, these aspects characterizing the response activity events can be considered when evaluating the (one or more) response activity events to identify response activity patterns. For example, a pattern of turning the smart thermostat up by two degrees can be determined, where the user typically turns the thermostat up by two degrees (e.g., triggered by a fault state according to a user command such as "it's cold"). Such a response activity event pattern can be associated with the user command that triggers the fault state - "it's cold".

[0040] As previously described, the response activity pattern can be used to generate response profiles corresponding to one or more user commands. Based on these response profiles, various implementations can provide an improved user experience. For example, some embodiments can initiate tasks using a response activity pattern associated with a user's computing device (in response to a received user command) (e.g., by executing response activity events). Some embodiments can be performed by a personal assistant application or service, which can be implemented as one or more computer applications, services, or routines, such as an app running on a mobile device or in the cloud, as further described herein.

[0041] Turning now to Figure 1 , a block diagram is provided showing an exemplary operating environment 100 in which some embodiments of the present disclosure may be employed. It should be understood that such and other arrangements described herein are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, and groupings of functions) can be used in addition to or instead of those shown, and some elements may be omitted together for clarity. Moreover, many of the elements described herein are functional entities, which can be implemented as discrete or distributed components or in combination with other components, and implemented in any suitable combination and location. The various functions described herein as being performed by one or more entities can be performed by hardware, firmware, and / or software. For example, some functions can be performed by a processor executing instructions stored in a memory.

[0042] Among other components not shown, the exemplary operating environment 100 includes: a plurality of user devices, such as user devices 102a and 102b through 102n; a plurality of data sources, such as data sources 104a and 104b through 104n; a server 106; and a network 110. It should be understood that the environment 100 shown in Figure 1 is an example of a suitable operating environment. Each of the components shown in Figure 1 can be implemented via any type of computing device, such as the computing device 600 described in connection with Figure 6 . These components can communicate with each other via the network 110, which can include, but is not limited to, one or more local area networks (LANs) and / or wide area networks (WANs). In an exemplary implementation, the network 110 includes the Internet and / or a cellular network, as well as any one of a variety of possible public and / or private networks.

[0043] It should be understood that within the scope of the present disclosure, any number of user devices, servers, and data sources may be employed within the operating environment 100. Each may include a single device or multiple devices that cooperate in a distributed environment. For example, the server 106 may be provided via multiple devices arranged in a distributed environment that together provide the functionality described herein. Additionally, other components not shown may also be included in the distributed environment.

[0044] The user devices 102a and 102b through 102n may be client devices on the client side of the operating environment 100, while the server 106 may be on the server side of the operating environment 100. The server 106 can include server-side software designed to work in conjunction with client-side software on the user devices 102a and 102b through 102n to implement any combination of the features and functions discussed in the present disclosure. This division of the operating environment 100 is provided to illustrate an example of a suitable environment, and it is not required for the server 106 to remain as a separate entity for any combination of the user devices 102a and 102b through 102n in each implementation.

[0045] The user devices 102a and 102b through 102n may include any type of computing device that a user can use. For example, in one embodiment, the user devices 102a through 102n may be of the type of computing device described herein Figure 6 By way of example and not limitation, the user device may be embodied as a personal computer (PC), laptop computer, mobile or cellular device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, global positioning system (GPS) or device, video player, handheld communication device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, camera, remote control, barcode scanner, computerized measurement device, appliance, consumer electronic device, workstation, smart speaker, smart electronics or device, or any combination of these depicted devices, or any other suitable device.

[0046] The data sources 104a and 104b through 104n may include data sources and / or data systems that are configured to make data available to the operating environment 100 or in conjunction with Figure 2 any of the various components of the system 200 described herein. (For example, in one embodiment, one or more of the data sources 104a through 104n provide to Figure 2The user data collection component 210 provides (or makes available for access) user data. Data sources 104a and 104b to 104n may be discrete from user devices 102a and 102b to 102n and server 106, or may be incorporated into and / or integrated with at least one of these components. In one embodiment, one or more of data sources 104a to 104n include one or more sensors, which may be integrated into or associated with one or more of user devices 102a, 102b, or 102n or server 106. In conjunction with Figure 2 the user data collection component 210 further describes examples of sensed user data available from data sources 104a to 104n.

[0047] The operating environment 100 can be used to implement one or more components of the system 200 described in Figure 2 including components for collecting user data, monitoring user activity events, determining status faults, monitoring response activity events, determining response activity patterns, generating (one or more) response personalization and / or general profiles to provide an improved user experience, and / or initiating response activities triggered by identified commands. Now in conjunction with Figure 1 Reference Figure 2 , there is provided a block diagram showing aspects of an exemplary computing system architecture suitable for implementing embodiments and generally designated as system 200. System 200 represents only one example of a suitable computing system architecture. Other arrangements and elements can be used in addition to or in place of those shown, and some elements may be omitted together for clarity. Further, as with the operating environment 100, many of the elements described herein are functional entities, which may be implemented as separate or distributed components or in combination with other components, and implemented in any suitable combination and location.

[0048] Exemplary system 200 includes the network 110 described in conjunction with Figure 1 and communicatively couples the components of system 200, including user data collection component 210, presentation component 220, user activity monitor 280, activity pattern interface engine 260, fault state detector 270, and storage device 225. User activity monitor 280 (including its components 282, 284, and 286), response activity pattern engine 260 (including its components 262, 264, and 266), user data collection component 210, presentation component 220, fault state detector 270, (one or more) response profile generators 290, and response activity initiator 299 may be embodied as a set of compiled computer instructions or functions, program modules, computer software services, or process arrangements executed on one or more computer systems such as the computing device 600 described in conjunction with Figure 6 the computing device 600.

[0049] In one embodiment, the functions performed by the components of system 200 are associated with one or more personal assistant applications, services, or routines. Specifically, such applications, services, or routines may run on one or more user devices (such as user device 104a), servers (such as server 106), may be distributed across one or more user devices and servers, or may be implemented in the cloud. Additionally, in some embodiments, these components of system 200 may be distributed across a network including one or more servers (such as server 106) and client devices (such as user device 102a) in the cloud, or may reside on a user device such as user device 102a. Further, these components, the functions performed by these components, or the services performed by these components may be implemented at an appropriate abstraction layer(s) of a (one or more) computing system (such as an operating system layer, an application layer, a hardware layer), etc. Alternatively or additionally, these components and / or the functions of the embodiments described herein can be performed at least in part by one or more hardware logic components. By way of example, and not limitation, illustrative types of hardware logic components that can be used include field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), system on a chip (SOC), complex programmable logic devices (CPLD), etc. Additionally, although the functionality has been described herein with respect to specific components shown in the exemplary system 200, it is contemplated that in some embodiments the functionality of these components can be shared or distributed across other components.

[0050] Continue Figure 2 , the user data collection component 210 is generally responsible for collecting from one or more data sources (such as Figure 1Data sources 104a and 104b through 104n) access or receive (and in some cases also identify) user data. In some embodiments, the user data collection component 210 can be used to facilitate the accumulation of user data for a particular user (or in some cases, multiple users including crowdsourced data) for the user activity monitor 280, the response activity pattern engine 260, the (one or more) response profile generators 290, the fault status detector 270, or the response activity initiator 299. The data can be received (or accessed) by the user data collection component 210 and optionally accumulated, reformatted, and / or combined and stored in one or more data stores (such as the storage device 225), where it is available to other components of the system 200. For example, the user data can be stored in or associated with the user profile 240, as described herein. In some embodiments, any personally identifiable data (i.e., user data that specifically identifies a particular user) is not uploaded with the user data from one or more data sources or otherwise provided, is not permanently stored, and / or is not available to components or sub-components of the system 200. In some embodiments, a user can opt in or out of the services provided by the techniques described herein and / or select which user data and / or which user data sources are utilized by these techniques.

[0051] User data can be received from various sources, and the data can be obtained in various formats. For example, in some embodiments, the user data received via the user data collection component 210 can be determined via one or more sensors, which can be on or associated with one or more user devices (such as user device 102a), servers (such as server 106), and / or other computing devices. As used herein, a sensor can include a function, routine, component, or combination thereof for sensing, detecting, or otherwise obtaining information such as user data from a data source 104a, and can be embodied as hardware, software, or both. By way of example and not limitation, user data can include data sensed or determined from one or more sensors (referred to herein as sensor data), such as location information of one or more mobile devices, attributes or characteristics of one or more user data (such as device status, charging data, date / time, or other information derived from a user device such as a mobile device), user activity information (e.g., app usage; online activity; searches; voice data, such as automatic speech recognition; activity logs; communication data, including calls, text messages, instant messages, and emails; website posts; other user data associated with communication events), in some embodiments including user activities occurring on more than one user device, user history, session logs, application data, contact data, calendar and schedule data, notification data, social network data, news (including popular or trending items on a search engine or social network), online game data, e-commerce activities (including from such as video streaming services, gaming services, or Data of an online account), (one or more) user account data (which may include data from user preferences or settings associated with a personal assistant application or service), home sensor data, appliance data, Global Positioning System (GPS) data, vehicle signal data, traffic data, weather data (including forecasts), wearable device data, other user device data (which may include device settings, profiles, network-related information (e.g., network name or ID, domain information, workgroup information, connection data, Wi-Fi network data or configuration data, data about the model, firmware or device, device pairing data, such as when a user pairs a mobile phone with a Bluetooth headset, or other network-related information), gyroscope data, accelerometer data, payment or credit card usage data (which may include data from the user's PayPal account), purchase history data (such as information from the user's Xbox Live, Amazon.com or eBay account), other sensor data that can be sensed or otherwise detected by (one or more) sensor (or other detector) components, including data derived from sensor components associated with the user (including location, movement, orientation, position, user access, user activity, network access, user device charging or other data that can be provided by one or more sensor components), data derived from other data (e.g., location data that can be derived from Wi-Fi, cellular network or IP address data), and almost any other data source that can be sensed or determined as described herein.

[0052] User data can be received by the user data collection component 210 from one or more sensors and / or computing devices associated with the user. Although for the sake of interpretability of the user data collection component 210, it is envisioned that the user data is processed, for example, by sensors or other components not shown, the embodiments described herein do not limit the user data to processed data and may include raw data. In some embodiments, the user data collection component 210 or other components of the system 200 may determine interpretive data from the received user data. Interpretive data corresponds to data used by the components of the system 200 to interpret the user data. For example, interpretive data can be used to provide context to the user data, which can support determinations or inferences made by the components or sub-components of the system 200, such as venue information from location, text corpus from the user's speech (i.e., speech-to-text), or aspects of spoken language understanding. Additionally, it is envisioned that for some embodiments, the components or sub-components of the system 200 may use the user data and / or a combination of the user data and interpretive data to perform the objectives of the sub-components described herein.

[0053] In some aspects, user data can be provided in a user data stream or signal. A "user signal" can be a feed or stream of user data from a corresponding data source. For example, a user signal may come from a smartphone, a home sensor device, a smart speaker, a GPS device (e.g., for location coordinates), a vehicle sensor device, a wearable device, a user device, a gyroscope sensor, an accelerometer sensor, a calendar service, an email account, a credit card account, or other data sources. In some embodiments, the user data collection component 210 continuously, periodically, when it becomes available, or as needed, receives or accesses user-related data.

[0054] Continuing Figure 2 , the exemplary system 200 includes a fault state detector 270. In an embodiment, the fault state detector 270 includes an application or service that processes the collected user data (e.g., user commands) in conjunction with device responses (or lack thereof) to identify faults in the device. User commands can be received as user data collected by the user data collection component 210. In an embodiment, such user commands can include voice commands, gestures, interactions with the device (via the screen), or any other means of inputting commands. When the user device receives an indication of a user command and determines that the command is unknown or not recognized, a fault mode or fault state can be considered to have occurred. For example, the command itself may be unknown, or the task associated with the command may be unknown. For example, in an embodiment, a computer program or service can compare the received user command with a set of known (e.g., predefined) commands or known (e.g., predefined) response actions (those operations designated to be performed in response to a command). The set of known commands and / or response actions can be stored as a list, library, index, and / or in a database, such as the response actions 235 stored in the storage device 225.

[0055] In some embodiments, the fault state detector 270 (or another component of the computing system) can utilize a library or collection of known commands and / or responses to determine whether a received command is known or unknown. In some embodiments, a user command can be determined to be unknown when it is not relevant to a task (e.g., as determined using a library of response actions known to the device based on user commands associated with the task). For example, the user command "It's cold in here" may not be relevant to the task of using a smart thermostat to increase the temperature. In some cases, determining whether a user command is known or unknown may involve performing a similarity comparison. In these embodiments, aspects of the received user command can be compared for similarity to aspects of other known user commands or response actions, such as information stored in the storage device 225. For example, using feature similarity techniques, such as clustering or Word2Vec, a measure of similarity, such as a similarity score, can be determined. In some embodiments, where the similarity score is high or meets a predetermined threshold, the command can be determined to be similar to a known command, and an operation corresponding to that known command can be performed. However, where the similarity score does not meet the predetermined threshold, the command can be determined to not be similar to the set of known commands, which may trigger a fault state during which user activity can be monitored.

[0056] In some embodiments, the fault state detector 270 can initiate an operation monitoring mode when it determines that a received user command is unknown. In some cases, the operation monitoring mode can be responsive to a fault state initiated in the device. By way of example and not limitation, a user can explicitly invoke the device and provide a user command that is not recognized by the device. In some embodiments, the device can be explicitly invoked using a trigger name (e.g., "Ok Google", "Alexa", "Hey, Cortana"). In other embodiments, the device can be explicitly invoked based on a user command (e.g., "Turn on the TV", "It's cold in here", "Play my song"). Upon detecting an unknown command, as determined by the fault state detector 270, an enhanced mode of monitoring user activity can be initiated.

[0057] In some implementations, a user may explicitly enter the monitoring mode (e.g., by explicitly triggering the fault state). For example, this is an operating mode in which the user can teach the device specific operations in response to specific commands. For example, the user may initiate the monitoring mode to teach the computing device how the user desires the device to perform tasks associated with a specific user command. In another example, the user can initiate the monitoring mode to correct (or improve) a situation where the computing device does respond to the user command but the response is incorrect or generally not what the user expects (such as based on the user's preferences). For example, the user may provide feedback to the system, such as saying "no, that's wrong" or "this is not what I want", or may select an option in an application, or provide other feedback indicating an operation performed by the computing device in response to an incorrect user command. Such feedback can initiate the monitoring mode, thereby enabling the system to learn one or more new responses and / or check if another response profile is available (e.g., at a server location with published response profiles) in response to the user command. In such a scenario, the user may want the computing device to perform a different response activity event to perform the task associated with the command rather than the current response activity event. In an embodiment, setting a new response activity mode can replace the old response activity mode (e.g., temporarily or permanently replace the response activity mode). In yet another example, the computing device may first have learned an incorrect response activity for the command (e.g., turning on the light and turning up the heat based on the user command "I'm cold" when the user actually only wants the computing device to turn up the heat). In this way, the system can allow the user to override a previously learned or pre-determined operation performed by the computing device in order to modify (e.g., correct or improve) how the computing device responds to user commands.

[0058] The user activity monitor 280 is generally responsible for monitoring user data to obtain information that can be used to determine user activity information, which can include identifying and / or tracking characteristics (sometimes referred to herein as "variables") or other information about specific user actions and associated context information. Embodiments of the user activity monitor 280 can determine user activity associated with a particular user based on the monitored user data. As previously described, the user activity information determined by the user activity monitor 280 can include user activity information from multiple user devices associated with the user and / or from cloud-based services associated with the user (such as email, calendar, social media, or similar information sources), and can include context information associated with the identified user activity. The user activity monitor 280 can determine current or near-real-time user activity information and can also determine historical user activity information, which in some embodiments can be based on observations of user activity collected over time, access to user logs of past activities (e.g., such as browsing history). Additionally, in some embodiments, the user activity monitor 280 can determine user activity (which can include historical activity) from other similar users (i.e., crowdsourcing), as previously described.

[0059] In some embodiments, the user activity monitor 280 is responsible for monitoring user activity information when the fault state detector 270 detects a fault mode. In such embodiments, the user activity information can be monitored as one or more response activity events. Such one or more response activity events can be activities (e.g., an activity or a set of activities) that a user performs when the device fails to execute a command (e.g., detects / enters a fault mode). The information determined by the user activity monitor 280, including information about one or more current response activity events, one or more response activity events within a defined time range, and / or one or more historical response activity events, can be provided to the response activity pattern engine 260. As previously described, one or more response activity events can be determined by monitoring user data received from the user data collection component 210. In some embodiments, the user data, information about the user activity, and / or one or more response activity events determined based on the user data are stored in a user profile, such as stored in the user profile 240.

[0060] For example, when a failure state triggers enhanced user monitoring, the user activity monitor 280 can closely monitor user activities, such as what activities and / or sets of activities the user performs after a user command fails. Specifically, when the user command "It's cold in here" fails, the user activity monitor 280 can detect that the user increases the temperature on the smart thermostat (e.g., raises it by two degrees). The user activity monitor 280 can also detect that the user turns down the volume of the TV and turns on the hallway light.

[0061] In an embodiment, the user activity monitor 280 includes one or more applications or services that analyze information detected via one or more user devices used by the user and / or cloud-based services associated with the user to determine activity information and related context information. Information about user devices associated with the user can be determined from user data available via the user data collection component 210 and can be provided to the user activity monitor 280, the response activity pattern engine 260, or other components of the system 200.

[0062] More specifically, in some implementations of the user activity monitor 280, user devices can be identified by detecting and analyzing characteristics of the user devices, such as device hardware, software such as an operating system (OS), network-related characteristics, user accounts accessed via the device, and similar characteristics. For example, features of many operating systems can be used to determine information about the user device to provide information about hardware, OS version, network connection information, installed applications, etc.

[0063] Some embodiments of the user activity monitor 280 may determine one or more user devices and user device-related activities associated with a particular user based on the user data, which may include context information associated with the identified user device(s). In an embodiment, the user activity monitor 280 includes one or more applications or services that analyze the user devices used by the user to determine information about the devices and device usage. In some embodiments, the user activity monitor 280 monitors user data associated with the user devices and other relevant information on the user devices across multiple computing devices or in the cloud. Information about the user's user devices may be determined from the user data available via the user data collection component 210 and may be provided to the response activity pattern engine 260 or the response profile generator(s) 290, and other components of the system 200. In some implementations of the user activity monitor 280, user devices may be identified by detecting and analyzing characteristics such as device hardware, characteristics of software such as the OS, network-related characteristics, user accounts accessed via the device, and similar characteristics. For example, features of many operating systems may be used to determine information about the user device to provide information about the hardware, OS version, network connection information, installed applications, etc. Similarly, some embodiments of the user activity monitor 280 or its sub-components may determine the device name or identification (device ID) for each device associated with the user.

[0064] This information about the identified user device(s) associated with the user may be stored in the user profile associated with the user, such as in the user accounts and device(s) 244 of the user profile 240. In an embodiment, the user device may be polled, queried, or otherwise analyzed to determine information about the device. This information may be used to determine the label or identification of the device (e.g., device ID) such that interactions with the user of the device may be identified by the user activity monitor 280 from the user data. In some embodiments, the user may declare or register the device, such as by logging into an account via the device, installing an application on the device, connecting to an online service that queries the device, or otherwise providing information about the device to an application or service. In some embodiments, devices that log into an account associated with the user (such as an account or Net Passport, email account, social network, etc.) are identified and determined to be associated with the user.

[0065] As shown in the exemplary system 200, the user activity monitor 280 may include a user activity detector 282, a context information determiner 284, and an activity feature determiner 286. In some embodiments, the user activity monitor 280, one or more of its sub-components, or other components of the system 200, such as the fault status detector 270, the response activity pattern engine 260, or the (one or more) response profile generators 290, may determine interpretive data based on received user data. The interpretive data can correspond to data used by these components of the system 200 or sub-components of the user activity monitor 280 to interpret the user data. For example, the interpretive data can be used to provide additional context for the user data, which can support the determinations or inferences made by the components or sub-components. Additionally, it is contemplated that embodiments of the user activity monitor 280, its sub-components, and other components of the system 200 may use the user data and / or a combination of the user data and the interpretive data to perform the objectives of the sub-components described herein. Further, although several examples of how the user activity monitor 280 and its sub-components may identify user activity information are described herein, many variations of user activity identification and user activity monitoring are possible in the various embodiments of the present disclosure.

[0066] The user activity detector 282 is generally responsible for determining (or identifying) that a user action or user activity event has occurred. Embodiments of the activity detector 282 may be used to determine the (one or more) current response activity events and / or one or more historical response activity events. Some embodiments of the activity detector 282 may monitor user data for activity-related features or variables corresponding to the (one or more) response activity events, such as indications of applications launched or accessed, files accessed, modified, copied, websites navigated to, online content downloaded and presented or played, devices interacted with, or similar user activities.

[0067] Additionally, some embodiments of the user activity detector 282 extract information about the (one or more) response activity events from the user data, which may include the (one or more) current response activity events, the (one or more) historical response activity events, and / or related information (such as context information). (Alternatively or additionally, in some embodiments, the context information determiner 284 determines and extracts context information. Similarly, in some embodiments, based on the identification of the response activity events determined by the user activity detector 282, the activity feature determiner 286 extracts information about the (one or more) response activity events, such as user activity-related features.) Examples of the extracted user activity information may include app usage, online activities, searches, calls, usage durations, application data (e.g., emails, messages, posts, user status, notifications), or almost any other data related to user interactions with the user device or user activities conducted via the user device. In other components of the system 200, the extracted user activity information determined by the user activity detector 282 may be provided to the user activity monitor 280, the response activity pattern engine 260, or other sub-components of the (one or more) response profile generators 290. Additionally, the extracted user activity may be stored in a user profile associated with the user, such as in the user activity information component 242 of the user profile 240. In some embodiments, the user activity detector 282 or the user activity monitor 280 (or other sub-components thereof) performs merging on the detected (one or more) response activity events. For example, overlapping information may be merged and duplicate or redundant information may be eliminated.

[0068] In some embodiments, the user activity-related features may be analyzed to detect one or more response activity events. For example, in some embodiments, the user activity detector 282 employs a user activity event logic unit (which may include rules, conditions, associations, classification models, or other criteria) to identify one or more response activity events. For example, in one embodiment, the user activity event logic unit may include comparing user activity criteria with the user data to determine that a response activity event has occurred. The activity event logic unit can take many different forms, depending on the mechanism used to identify the response activity event. For example, the user activity event logic unit may be training data for training a neural network that is used to evaluate user data to determine when a response activity event occurs. The activity event logic unit may include a fuzzy logic unit, neural network, finite state machine, support vector machine, logistic regression, clustering, or machine learning techniques, similar statistical classification processes, or combinations thereof to identify one or more response activity events based on the user data. For example, the activity event logic unit may specify the type of one or more user device interaction information associated with the response activity event, such as launching an app, manual input, or other interactions with the device. In some embodiments, a series or sequence of user device interactions may be mapped to a response activity event such that the response activity event can be detected when it is determined that the user data indicates that the series or sequence of user interactions has been performed by the user.

[0069] In some embodiments, the activity event logic unit may specify the type of user device-related activities that are considered response activity events, such as activities that occur when the user logs in to the user device while the user interface is receiving input (e.g., when a computer mouse, touchpad, screen, voice recognition interface, etc. is active), or for example, a specific type of activity of launching an application, modifying a file using an application, opening a browser, and navigating to a website. In this way, the activity event logic unit can be used to distinguish genuine user activities from automated activities of processes running on the user device, such as automatic updates or malware scans. Once the user activities are determined, these features or additional related features can be detected and associated with the detected response activity events for determining activity patterns.

[0070] In some embodiments, the user activity detector 282 runs on or is associated with each user device of the user. The user activity detector 282 may include functions for polling or analyzing aspects of the operating system to determine features related to user activity (e.g., such as installed or running applications or file access and modification), network communications, and / or other user actions that can be detected via the user device including sequences of actions.

[0071] The context information determiner 284 is generally responsible for determining context information related to a response activity event (detected by the user activity detector 282 or the user activity monitor 280), such as context features or variables, related information, and user-related activities associated with the response activity event, and is further responsible for associating the determined context information with the detected user activity. In some embodiments, the context information determiner 284 may associate the determined context information with the response activity event and may also record the context information together with the associated response activity event. Alternatively, the association or recording may be performed by another service. For example, some embodiments of the context information determiner 284 provide the determined context information to the activity feature determiner 286, which determines the activity features of the response activity event and / or related context information.

[0072] Some embodiments of the context information determiner 284 determine context information related to a user action or response activity event, such as an entity identified in or related to the response activity event (e.g., a two-degree increase in temperature). Additionally, the user activity can be associated with the location or venue of the user device. By way of example and not limitation, this can include context features such as location data, which may be represented as a location stamp associated with the response activity event; context information about the location, such as venue information (e.g., this is the user's office location, home location, school, restaurant, mobile theater); Yellow Pages identifier (YPID) information, time, period, and / or date, which may be represented as a time stamp associated with the response activity event; user device characteristics or user device identification information about the device on which the user performs the response activity event; the duration of the user response activity event, (one or more) other user response activity events before and / or after the response activity event (such as a sequence of device interactions, an application usage sequence, such as interactions with one or more devices, a sequence of website usage, such as entering a specific search query); other information about the response activity event, such as an entity associated with the response activity event (e.g., a venue, a person, an object); information detected by (one or more) sensors on the user device associated with the user and concurrent or substantially concurrent with the response activity event (e.g., motion information or physiological information detected on a fitness tracking user device, music listened to via a microphone sensor when the music source is not the user device); or any other information related to the response activity event that can be used to detect a pattern for determining similar events.

[0073] In embodiments that use context information related to a user device, the user device can be identified by detecting and analyzing user devices such as device hardware, characteristics of software such as an OS, network-related characteristics, user accounts accessed via the device, and similar characteristics. For example, as previously described, the capabilities of many operating systems can be used to determine information about the user device to provide information about the hardware, OS version, network connection information, installed applications, and the like. In some embodiments, a device name or identifier (device ID) can be determined for each device associated with a user. This information about the identified user devices associated with the user can be stored in a user profile associated with the user, such as in the (one or more) user accounts and (one or more) devices 244 of the user profile 240. In an embodiment, the user device can be polled, queried, or otherwise analyzed to determine context information about the device. This information can be used to determine a label or identifier for the device (e.g., device ID), such that user activities on one user device can be identified and distinguished from user activities on another user device. Additionally, as previously described, in some embodiments, the user can declare or register the user device, such as by logging into an account via the device, installing an application on the device, connecting to an online service that queries the device, or otherwise providing information about the device to an application or service. In some embodiments, a device that logs into an account associated with the user (such as a Microsoft account or NetPassport, email account, social network, etc.) is identified and determined to be associated with the user.

[0074] In some implementations, the context information determiner 284 can receive user data from the user data collection component 210, parse the data in some cases, and identify and extract context features or variables (which can also be performed by the activity feature determiner 286). The context variables can be stored as a relevant set of context information associated with the user activity and can be stored in a user profile such as in the user activity information component 242. In some cases, one or more activity pattern consumers can use the context information, such as for personalizing content or the user experience, such as when, where, or how to present the content. The context information can also be determined based on the user data of one or more users, and in some embodiments, instead of or in addition to the user activity information for a particular user, the user data can be provided by the user data collection component 210.

[0075] The activity feature determiner 286 is generally capable of being responsible for determining activity-related features (or variables) associated with response activity events that can be used to identify patterns of similar events. The activity features can be determined based on information about the response activity events and / or based on relevant context information. In some embodiments, the activity feature determiner 286 receives one or more response activity events or related information from the user activity monitor 280 (or a sub-component thereof), and analyzes the received information to determine a set of one or more features associated with the one or more response activity events.

[0076] Examples of activity-related features include, but are not limited to: location-related features, such as the location of the one or more user devices during the response activity event, venue-related information associated with the location, or other location-related information; time-related features, such as one or more times of day (or multiple days), days of the week or month, or the duration of the response activity event, or related duration information, such as the duration for which the user uses an application associated with the response activity event; user device-related features, such as device type (e.g., desktop computer, tablet computer, mobile phone, fitness tracker, heart rate monitor), hardware attributes or profiles, OS or firmware attributes, device ID or model, network-related information (e.g., mac address, network name, IP address, domain, workgroup, information about other devices detected on the local network, router information, proxy or VPN information, other network connection information), location / motion / orientation-related information about the user device, power information, such as battery level, time of connecting / disconnecting the charger, and user access / touch information; usage-related features, such as one or more files accessed, app usage (which can also include application data, in-app usage, apps running simultaneously), network usage information, one or more user accounts accessed or otherwise used (such as one or more device accounts, one or more OS-level accounts, or one or more online / cloud service-related account activities, such as an account or Net Passport, (one or more) online storage accounts, email, calendar, or social network accounts); content-related features such as online activities (e.g., searches, websites browsed, purchases, social network activities, communications sent or received, including social media posts); other features that may be detected or sensed at or near the time of or in proximity to the response activity event; or any other features that can be detected or sensed and used to determine a pattern of (one or more) response activity events. The features may also include information about the (one or more) users of the device; other information identifying the user, such as a login password, biometric data, which may be provided by a fitness tracker or biometric scanner; and / or characteristics of the (one or more) users of the device, which may help to distinguish users on a device shared by more than one user. In some embodiments, the user activity event logic unit (described in conjunction with user activity detector 282) may be used to identify specific features from user activity information related to the response activity event.

[0077] Continue Figure 2System 200 is generally responsible for determining a response activity pattern based on user activity information related to one or more response activity events determined from user activity monitor 280 (e.g., after detecting a fault condition). In some embodiments, response activity pattern engine 260 may run on a server, as a distributed application across multiple devices, or in the cloud. At a high level, response activity pattern engine 260 may receive user activity-related information, which may be uploaded from user activity logs from client-side applications or services associated with user activity monitor 280. One or more inference algorithms may be applied to the user activity-related information to determine a set of possible response activity patterns (e.g., related to a specific user command). For example, patterns may be determined based on similar observed instances of user activity or associated context information, which may be referred to as "common features" of user activity-related information that occur after a detected fault condition. The inferred activity pattern information may be provided to one or more response profile generators 290 and / or used to generate pattern-based predictions about an operation set for performing a task associated with a specific user command. In some embodiments, a corresponding confidence level is also determined for the response activity pattern, as described herein. Additionally, the response activity pattern may include a confidence level for initiating a single operation to perform a (future-occurring) task and / or initiating one of a series of operations to perform the task. For example, to perform a task associated with a user command, the response activity event may be an eighty percent likelihood of increasing the temperature on a smart thermostat, a fifteen percent likelihood of turning on a hallway light, and a five percent likelihood of muting the volume on a TV.

[0078] As shown in exemplary system 200, response activity pattern engine 260 may include a semantic information analyzer 262, a feature similarity identifier 264, and a response activity pattern determiner 266. Semantic information analyzer 262 is generally responsible for determining semantic information associated with response activity-related features identified by user activity monitor 280. For example, although a response activity feature may indicate a specific device with which the user is interacting, semantic analysis may determine how to interact with the device or the nature of the interaction. Semantic information analyzer 262 may determine additional response activity-related features that are semantically related to the response activity event, which may be used to identify response activity patterns.

[0079] Specifically, as previously described, semantic analysis can be performed on response activity information that may include the context information to characterize aspects of a user action or response activity event. For example, in some embodiments, activity features associated with a response activity event can be classified or categorized (such as by type, time range or location, work-related, home-related, topic, related entities, one or more other users (such as communication with another user), and / or the relationship of other users to the user (e.g., family member, close friend, work acquaintance, boss, etc.), or other categories), or relevant features can be identified to determine similarity or relationship proximity to other response user activity events, which can indicate patterns (e.g., the number of degrees the thermostat is turned up). In some embodiments, the semantic information analyzer 262 can utilize semantic knowledge representation, such as a relational knowledge graph. The semantic information analyzer 262 can also utilize semantic analysis logic, including rules, conditions, or associations to determine semantic information related to the response activity event. For example, a response activity event related to a user command "It's cold here" that includes interaction with a smart thermostat can be characterized by how many degrees the thermostat increases. Thus, when the smart thermostat increases by two degrees in response to a fault state triggered by the "It's cold here" command, it can be determined that the user activity corresponding to the "It's cold here" command is increasing the smart thermostat by two degrees. Additional semantic information (such as the time of year) can also be used to determine patterns. For example, in the summer, the user activity corresponding to the "It's cold here" command is to increase the smart thermostat by two degrees. However, in the winter, the user activity corresponding to the "It's cold here" command is to increase the smart thermostat by four degrees.

[0080] The semantic information analyzer 262 can also be used to characterize context information associated with the response activity event, such as determining that the location associated with the activity response event corresponds to a center or place of user interest (such as the user's home, office, gym, etc.) based on the frequency of user access. For example, the user's home center can be determined (using the semantic analysis logic unit) as the location where the user spends most of their time between 8 pm and 6 am. Similarly, the semantic analysis can determine the times of day corresponding to work hours, lunch hours, and commuting times. Similarly, the semantic analysis can classify the activity as associated with work or home based on other characteristics of the activity (e.g., playing classical music in response to the user command "play music" at a location corresponding to the user's office can be determined as a work-related activity, while playing home music in response to the user command "play music" at a location corresponding to the user's home can be determined as a home-related activity). In this way, the semantic analysis provided by the semantic information analyzer 262 can provide additional relevant features of the user activity event that can be used to determine the response activity pattern. For example, in the case where the user activity includes playing classical music in response to the command "play music" and the semantic analysis indicates that the user is at work, the response activity pattern (determined by the response activity pattern determiner 266) can be determined to indicate that in response to the user command "play music" while at work, the user wants to play classical music at work.

[0081] The feature similarity recognizer 264 is generally responsible for determining the similarity of the activity features of two or more response activity events (in other words, the activity features characterizing the first response activity event are similar to the activity features characterizing the second response activity event). The activity features can include features related to context information and the features determined by the semantic information analyzer 262. Response activity events with common activity features can be used to identify response activity patterns, which can be determined using the response activity pattern determiner 266.

[0082] For example, in some embodiments, the feature similarity identifier 264 can be used to determine a set of response activity events with common features. In some embodiments, this set of response activity events can be used as an input to the response activity pattern determiner 266, as described below. In some embodiments, the feature similarity identifier 264 can include functionality for determining the similarity of periodic and behavior-based activity features. Periodic features include, for example, features that may occur periodically; for example, a day of the week or month, even / odd days (or weeks), monthly, annually, every other day, every 3 days. For example, behavior features can include user activities that tend to occur at a specific location or activities that occur before or after a given user activity event (or sequence of previous activity events).

[0083] In embodiments where the activity features have values, the similarity between different activity features having the same or approximately the same value can be determined based on specific features. (For example, the timestamps of a first response activity event occurring at 9:01 am on Friday and a second response activity event occurring at 9:07 am on Friday can be determined to have similar or common timestamp features.)

[0084] The response activity pattern determiner 266 is generally responsible for determining the response activity pattern based on the similarities identified in the response activity events. Specifically, the response activity pattern determiner 266 (or the response activity pattern engine 260) can determine the response activity pattern based on the repetition of similar activity features associated with a plurality of observed response activity events. Thus, for example, in the case where the activity features corresponding to two or more response activity events are similar, the response activity pattern can be determined. In some instances, the response activity events may have many corresponding activity features, which can be represented as a feature vector associated with a specific response activity event. Thus, the analysis performed by the response activity pattern determiner 266 can involve comparing the activity features of the feature vectors from multiple response activity events.

[0085] In some embodiments, the pattern inference logic 230 can be used to determine the response activity pattern. The pattern inference logic can include rules, associations, conditions, prediction, and / or classification models, or pattern inference algorithms. The pattern inference logic 230 can take many different forms, depending on the specific response activity pattern or the mechanism used to identify the response activity pattern, or to identify the feature similarities between the observed response activity events to determine the pattern corresponding to performing a task associated with a user command. For example, some embodiments of the pattern inference logic unit 230 can employ machine learning mechanisms to determine the feature similarities, or other statistical measures to determine that the response activity events belong to an exemplary set of response actions that support the determined response activity pattern, as further described below. The response activity information can be received from the user activity monitor 280, and the information about the identified similar features can be received from the feature similarity recognizer 264. In some embodiments, the (one or more) response activity patterns determined by the response activity pattern determiner 266 can be stored as user-response activity patterns 248 in the user profile 240. In additional embodiments, the (one or more) response activity patterns determined by the response activity pattern determiner 266 can be stored as response activity patterns 256 in the response profile 250.

[0086] In some embodiments, the response activity pattern determiner 266 provides a pattern of user activity (e.g., (one or more) response activity events) and an associated confidence score regarding the strength of the response activity pattern, which may reflect the likelihood that the response activity pattern satisfactorily performs a task associated with a particular user command. More specifically, in some embodiments, a corresponding confidence weight or confidence score may be determined with respect to the determined response activity pattern. The confidence score may be based on the strength of the pattern, which may be based on the number of observations (of a particular response activity event) used to determine the response activity pattern, the frequency with which the user actions are consistent with the response activity pattern, the age or freshness of the response activity event observations, the number of similar features, the feature type, and / or the degree of similarity or a similar measure of features common to the response activity observations that make up the pattern.

[0087] In some cases, the confidence score may be considered when generating a response profile (e.g., using (one or more) response profile generators 290). For example, in some embodiments, a minimum confidence score may be required before using the response activity pattern to generate a response profile corresponding to a user command. In one embodiment, a threshold of 0.6 (or just over fifty percent) is used such that only response activity patterns with a 0.6 (or greater) likelihood of performing the desired task by initiating the response activity events (as indicated by the response activity pattern) may be provided. However, in cases where confidence scores and thresholds are used, the determined response activity patterns with confidence scores less than the threshold may still be monitored and updated based on additional response activity event observations, as the additional observations may increase the confidence in a particular pattern.

[0088] Some embodiments of the response activity pattern determiner 266 determine a pattern according to the exemplary method described below, where each instance of a response activity event has a corresponding historical value of a tracked activity feature (variable) that forms the pattern, and where the response activity pattern determiner 266 may evaluate the distribution of the tracked variables for the pattern. In the following example, the tracked variable for the response activity event is a timestamp corresponding to an observed instance of the response activity event. However, it will be appreciated that conceptually, the following can be applied to different types of historical values for tracked activity features (variables).

[0089] Some timestamps (i.e., values of a given tracked variable) can be represented as and mapped to a two-dimensional histogram of hours and days of the week. The two-dimensional histogram can include a summation of user-device interaction instances, such as:

[0090]

[0091] The histogram can be used to determine a derivative histogram. For example, a histogram for a day of the week can correspond to: h j = ∑ i h ij . A one-hour histogram for a day can correspond to: h i = ∑ j h ij . As another example, one or more histograms can be determined for a specific semantic time resolution in the following form: h iC = ∑ j∈C h ij . Any of various semantic time resolutions can be employed, such as weekdays and weekends, or morning, afternoon, and evening. An example of the latter is C ∈ {morning, afternoon, evening}, morning = {9, 10, 11}, afternoon = {12, 13, 14, 15, 16}, and evening = {21, 22, 23, 24}.

[0092] An additional data structure for representing events can include the number of distinct timestamps having at least one timestamp in each calendar week, which can be represented as:

[0093]

[0094] As an example, can represent the number of distinct timestamps during the 2nd three-week period of available timestamps. N (j) can be used to represent the number of j-week timestamps available in the tracking data; for example, N (3) represents the number of three-week periods available in the timestamps.

[0095] A response activity pattern determiner 266 (or a response activity pattern engine 260) can generate a confidence score that quantifies the level of certainty that a particular response activity pattern is formed by historical values in the tracking variables. In the following example, the above principle is applied using Bayesian statistics. In some implementations, confidence scores can be generated for corresponding tracking variables indexed by time intervals of different resolutions. For timestamps, examples include 9 am on Tuesday, weekday mornings, and Wednesday afternoons. The confidence scores can be calculated by applying a Dirchlet polynomial model and computing the posterior predictive distribution of each time period histogram. When doing so, the prediction for each bin in a particular histogram can be given by:

[0096]

[0097] where K represents the number of bins, α 0 is a parameter encoding the strength of prior knowledge, and Then, the pattern prediction is the segment in the histogram corresponding to i*, and its confidence is given by x i* For example, consider a histogram where morning = 3, afternoon = 4, and evening = 3. Using α 0 = 10, the pattern prediction is afternoon, and the confidence score is According to various implementations, more observations lead to an increase in the confidence score, indicating an increase in the confidence of the prediction. For example, consider a histogram where morning = 3000, afternoon = 4000, and evening = 3000. Using a similar calculation, the confidence score is

[0098] Similarly, in some implementations, confidence scores can be generated for corresponding tracking variables indexed by time periods and multiple timestamps. Examples include 1 visit per week and 3 visits every 2 weeks. Using a Gaussian posterior, confidence scores can be generated for the patterns at each time period resolution, denoted as j. This can be done by adopting the following formula:

[0099] where In the above formula, σ 2 is the sample variance, and and μ 0 are the parameters of the formula. The confidence score can be calculated by taking a fixed interval around the prediction at multiple timestamps and calculating the cumulative density:

[0100] where

[0101] For example, consider the following observations: and N (1) = 4 and N (2) = 2. Using μ 0 = 1 and μ (1) = 4.075, and conf 1 = 0.25. Additionally, μ (2) = 10.31 and conf 2 = 0.99. In the previous example, although there are fewer timestamps available for the two-week period, the reduced variability in the user signal leads to an increased confidence in the presence of the pattern.

[0102] Once it has been determined that a response activity pattern exists or that the confidence score for a response activity pattern is high enough (e.g., meets a threshold), the response activity pattern determiner 266 can identify that multiple response activity events correspond to the response activity pattern. As another example, the response activity pattern determiner 266 can determine that the response activity pattern may perform a desired task in response to a user command, where one or more confidence scores for one or more tracking variables meet the threshold.

[0103] In some embodiments, a pattern of response activities or a pattern of response activity events can be determined by monitoring one or more activity characteristics, as previously described. These monitored activity characteristics can be determined based on what were previously described as tracking variables or data described in connection with the user data collection component 210. In some cases, the variables can represent contextual and / or semantic similarities between multiple user actions (response activity events). In this way, a pattern can be identified by detecting a variable or characteristic common to multiple user actions that occur after a fault state detected for a user command. More specifically, a characteristic associated with a first user action after a fault state detected for a user command can be correlated with a characteristic of a second user action after the fault state detected for the user command to determine a possible pattern. The identified pattern of characteristics may become stronger (i.e., more likely or more predictable) with more frequent repetition of user activity observations after a fault state (or after a user command) detected for the user commands that make up the pattern. Similarly, a particular characteristic becomes more closely associated with the response activity pattern as it is repeated.

[0104] In some embodiments, such as the exemplary embodiment shown in system 200, the response profile generator(s) 290 can generate one or more response profiles (e.g., stored as response activity pattern 256 and / or user response activity pattern 248) using the response activity pattern determined by the response activity pattern determiner 266. At a high level, the response profile generator(s) 290 can associate the response activity pattern with a corresponding user command. Specifically, the response profile generator(s) 290 can receive user activity information and / or activity characteristics associated with the user activity, and determine the user command in response to which the user activity has occurred. In an embodiment, the response profile generator(s) 290 can include functionality for generating a response profile as further described below, the response profile being used to initiate one or more operations to perform a task in response to a user command (or a similar command) (e.g., by executing one or more user activity events including the user command). The response profile generally includes information associating one or more commands with one or more specific operations to be performed to implement the command (or the task associated with the command). In an embodiment, such information can be generic and / or user-based. In another embodiment, such information can be specific to a particular user device (e.g., a particular brand of smart thermostat).

[0105] In one embodiment, the response profile generator(s) 290 uses features or patterns common between historical user activity and the most recent user activity determined by the response activity pattern determiner. For example, a response activity pattern based on the similarity of periodic features, from a set of historical user actions, from those historical user actions that have common periodic features with the current or most recent user action. Additionally, the response profile generator(s) 290 can determine the user command to be executed as the task corresponding to the response activity pattern. For example, if the most recent user action includes adjusting the thermostat in accordance with the user command (“I'm cold”), then determining behavioral feature similarity will include identifying those historical user actions having features indicating that the user adjusted the thermostat in accordance with the user command (the same or a similar user command—“I'm cold,” “It's cold here,” “Turn up the temperature”).

[0106] To generate and / or update a response profile, the (one or more) response profile generators 290 are capable of selecting response action events from a subset of historical response action events that are most similar (or sufficiently similar) to a particular current or most recent user action event, which can be selected to determine a response activity pattern. In some embodiments, the selection process uses a similarity threshold such as the one described above to identify those historical response action events that meet the similarity threshold. (Although the term "selection" is used, it is contemplated that the selection is performed by a computer-related process that does not require a person to perform the selection.) The response activity pattern can be associated with the (one or more) particular user commands. For example, the user commands "I'm cold", "It's cold here", "Turn up the temperature" can all be associated with the same task and use the same response activity pattern based on the similarity threshold to execute.

[0107] Exemplary system 200 may also include a storage device 225. The storage device 225 generally stores information, which includes data, computer instructions (e.g., software program instructions, routines or services), logical units, profiles, and / or models used in the embodiments described herein. In an embodiment, the storage device 225 includes a data store (or computer data memory). Additionally, although depicted as a single data storage component, the storage device 225 may be embodied as one or more data stores or may be in the cloud.

[0108] As shown in exemplary system 200, the storage device 225 may include an activity pattern inference logic unit 230, response actions 235, user profile 240, and response profile 250, as previously described. In Figure 2 An exemplary embodiment of the user profile 240 is illustratively provided. Exemplary user profile 240 includes information associated with a particular user, such as user activity information 242, information about the user's account and devices 244, user preferences 246, and user response activity pattern 248. The information stored in the user profile 240 can be used by the response activity pattern engine 260 or other components of the exemplary system 200.

[0109] As previously described, the user activity information 242 generally includes user information about user actions or activity events, related context information, activity characteristics, or other information determined via the user activity monitor 280, and may include historical or current user activity information. The user account and devices 244 generally include information about user devices accessed, used, or otherwise associated with the user, and / or information related to the user's account associated with the user, e.g., online or cloud-based accounts (e.g., email, social media), such as Net Passport, other accounts such as entertainment or game-related accounts (e.g., Xbox Live, Netflix, online game subscription accounts), user data associated with the accounts such as user emails, texts, instant messages, calls, other communications, and other content; social network accounts and data such as news feeds; online activities; and calendars, appointments, app data, other user accounts, etc. Some embodiments of the user account and device 244 may store information across one or more databases, knowledge graphs, or data structures. As previously described, the information stored in the user account and device 244 may be determined from the user data collection component 210 or the user activity monitor 280 (including one of its sub-components).

[0110] User preferences 246 generally include user settings or preferences associated with user activity monitoring. By way of example and not limitation, such settings may include user preferences regarding specific activities (and related information) that the user wishes to be explicitly monitored or not monitored, or categories of activities to be monitored or not monitored, crowdsourcing preferences such as whether to use crowdsourcing information, or whether the user's activity pattern information can be shared as crowdsourcing data; preferences regarding which activity patterns consumers can consume the user's activity pattern information; and thresholds, and / or notification preferences, as described herein. As previously described, the user response activity pattern 248 may include one or more user patterns generated by the (one or more) response profile generators 290 based on the (one or more) response activity patterns determined by the response activity pattern determiner 266, and may also include a confidence score associated with the pattern and / or information related to the activity pattern, such as context information or semantic information. The user response activity pattern 248 includes personalized user response activity pattern content determined from the user's personal content, such as a user-unique response activity pattern. For example, the user command "Activate my room" corresponds to a personalized user response activity pattern that includes response activities that include turning on the lights in the office, raising the temperature in the office by two degrees, and turning on the TV in the office at volume 20.

[0111] As shown in the exemplary system 200, the storage device 225 includes the activity pattern inference logic unit 230, the response actions 235, the user profile 240, and the response profile 250, as previously described. In Figure 2An exemplary embodiment of the response profile 250 is illustratively provided. In this example, the response profile 250 includes information associated with responses and / or user commands, such as response commands 252, device mappings 254, response activity patterns 256, and user feedback / weighting 258. In an embodiment, the responses and / or user commands 252 in the response profile 250 can be generic (i.e., capable of performing operations on one or more different devices to perform the same or similar tasks). In another embodiment, the responses and / or user commands in the response profile 250 can be specific to a user or user device (e.g., specific to a user with a particular smart thermostat). In an implementation, the information stored in the response profile 250 can be used by the response activity pattern engine 260 or other components of the exemplary system 200.

[0112] The response profile 250 can generally include one or more user commands and corresponding operations to be performed to implement the tasks associated with the commands. Such corresponding operations can be, for example, response activity events and / or response activity patterns corresponding to the user commands. In an implementation, the response profile can contain user commands, (one or more) associated response activity events, and / or (one or more) associated tasks (in any combination). In other embodiments, the response profile can also include user-device mappings that enable the operations to be performed on different types of user devices (e.g., various brands of user devices - smart thermostats - that perform the same / similar actions).

[0113] In some embodiments, the response profile can be determined from learned user activities. Such user activities can be learned from multiple users. For example, the response profile (corresponding to a specific user command) can be associated with response activity events. The response profile can also be associated with tasks based on the history of sensed user activities related to the user command (e.g., obtained during an operation monitoring mode). For example, but not limited to, (one or more) response profiles can be generated based on the determined pattern of response activity events regarding the desired interactions (e.g., (one or more) response activity events) associated with one or more user devices corresponding to a task in response to a user command, and used to provide an improved user experience. Based on the response activity information, the computer system can learn the response activity events corresponding to the user commands and associated with the user activities of (one or more) user devices involved in performing the tasks (according to the user commands).

[0114] In an embodiment, when an indication of a user command is received in the future, the response profile 250 can be used to determine the response activity event(s) associated with the task(s) involved with the user command. In such an embodiment, the response activity event(s) can be used to cause an operation to be performed to implement the task (e.g., using one or more user devices). In an implementation, the response activity pattern or the response profile generated therefrom can be used by one or more applications and services that consume this information to provide an improved user experience interacting with the computing device or the digital assistant application running thereon (e.g., understanding previously unknown user commands and causing an operation to be performed to implement the task associated with the user command). For example, in one embodiment, the response activity pattern information and / or the generated response profile can be provided via an API so that third-party applications and services can use it, such as by determining and / or providing device mappings, suggestions, or other information or services related to the user based on the learned response activity pattern. In some embodiments, the response profile can also include a user-device mapping that specifies how the operation is to be performed based on different types of user devices, such as smart lights or smart thermostats of different smart brands. For example, the response profile can specify one or more user commands and the corresponding operations to be performed to implement the task associated with the command (for one or more specific user devices).

[0115] As previously described, the response command 252 generally includes the user command(s) that can indicate or correspond to a specific task desired to be performed by the user issuing the command. In some embodiments, the response command can include a set of user commands having corresponding response activity patterns (or response actions that can be determined according to the response activity pattern) for performing an operation to implement the task. Thus, the user command can correspond to one or more actions initiated by one or more user devices to perform the task. For example, the command "It's cold in here" can correspond to the task of raising the temperature a few degrees on a smart thermostat.

[0116] The device mapping 254 generally includes information about the user devices accessed, used, or otherwise associated with the response, and / or information about the devices involved with the response activity. Some embodiments of the device mapping 254 can store information across one or more databases, knowledge graphs, or data structures. As previously described, the information stored in the device mapping 254 can be determined from the user data collection component 210 or the user activity monitor 280 (including one of its sub-components). Specifically, the device mapping 254 can associate the response command with the corresponding response activity pattern across various user devices. The device mapping can be provided by a company, a user, and / or some combination of the two.

[0117] In an embodiment, the device map 254 can be used to teach a previously unknown user command to a specific user device. For example, a first user device has the function of understanding that the user command "raise the temperature" causes the task of turning up the smart thermostat by two degrees. A second user device that inherently does not have the function of understanding the user command "raise the temperature" can learn the response activity pattern from the pairing of the user command and the response activity pattern (e.g., from the response action 235). In this mode, in an embodiment, even when a specific user device did not previously understand the user command, the specific user device can learn from the device map 254 to perform the task (e.g., via the response activity initiator 299).

[0118] In another embodiment, the device map 254 can transform a user command to apply the corresponding task to a specific user device. For example, various user devices may implement the response activity pattern in different ways to perform the same task. The device map 254 can associate a general command (e.g., from the response activity pattern) with a specific action for a specific user device (corresponding to the response activity pattern). For example, if the task is to raise the temperature, one type of smart thermostat uses one operation to raise the temperature, while another type of smart thermostat uses a different type of operation to raise the temperature. In an embodiment, the device map 254 ensures that a specific user device can understand how to implement the response activity pattern on the specific device.

[0119] The response activity pattern 256 generally includes a response activity pattern associated with a user command (e.g., the response command 252). Specifically, the response activity pattern stored in the response activity pattern 256 can include both a specific pattern and a general or universal pattern (e.g., a response activity pattern applicable to a group of users). As previously described, the response activity pattern 256 can include one or more response activity patterns determined by the response activity pattern determiner 266, and can also include a confidence score associated with the pattern and / or information related to the activity pattern, such as context information or semantic information. The response activity pattern can specify a set of operations to be performed in response to a user command, as previously described, and can be specified in the response command 252. Specifically, as described herein, the set of operations includes operations "learned" from the monitored user activities that determine the response pattern. In some embodiments, the set of operations can include only a single operation, and in other embodiments, the set of operations can include a series or sequence of operations to be performed.

[0120] In some embodiments, the set of operations may further include prompting the user to input information regarding specific variables or options associated with the operation. For example, for the user command "book a restaurant", if the user has not provided a specific restaurant, cuisine, or the like, when performing the operation learned from the user activity response to the command, the system may ask the user, for example, "Is there a specific restaurant you would like to book?" or similarly, "Do you have a particular cuisine you like when booking dinner?" or further, "Do you have a preferred dinner booking time?" In some embodiments, these prompts can also be learned, such as where the response activity pattern shows a strong pattern for a specific operation feature type or category (e.g., launching a specific app or contacting a specific website, such as the OpenTable app, which is used to book restaurants), but the feature values (i.e., parameter values) for that feature type vary (e.g., have high variance or divergence). In other words, the response activity pattern shows that a specific operation is always performed (e.g., launching the restaurant booking app), but the settings or values associated with those operations will vary (e.g., the specific restaurant selected in the app will vary). Thus, in cases where differences or divergences in the user response patterns are shown in aspects of the operation to be performed, the user can be prompted for input regarding the operation.

[0121] In some embodiments, weights can be applied to the response activity patterns. For example, user feedback can indicate that a particular response activity pattern (or more generally, a particular response profile 250) is good (e.g., correct) or bad (e.g., incorrect, incomplete, or problematic). The system can utilize this feedback (and / or weights) to determine whether it should continue to learn the response actions for a specific command, or whether it has sufficiently learned the response to the command. (For example, if the user feedback is positive, it can be inferred that the system's response to the user command satisfies the user.) Additionally, feedback or weights can be used to rank the available response profiles. Specifically, in some embodiments where the response profiles are available to other users, it is contemplated that multiple response profiles can be created for the same or similar commands. Thus, user feedback can be used to score or rank the response profiles, such that those response profiles with higher scores or weights are more likely to be utilized. Accordingly, the response profile 250 can include user feedback / weighting 258, which can represent the weight applied to a particular response profile 250 or response activity pattern 256.

[0122] In some embodiments, corresponding confidence weights or confidence scores can be associated with response activity patterns. The confidence score can be based on the strength of the response pattern. In some cases, the confidence score can be based on the number of observations (of a particular response activity event) used to determine the response activity pattern, the frequency with which the user's actions are consistent with the response activity pattern, the age or freshness of the user activity observations, the number of similar features, the feature types, and / or the degree of similarity of features common to the response activity observations that make up the pattern, or similar measurements.

[0123] In some cases, the confidence score can be considered when generating a response profile (e.g., using the (one or more) response profile generators 290). For example, in some embodiments, a minimum confidence score may be required before using the response activity pattern to generate a response profile corresponding to a user command. In one embodiment, the number of times a response activity pattern is established can indicate the strength of the response activity pattern. For example, if the response activity pattern corresponding to the user command "I am cold" is to perform the task of turning up the thermostat and has been determined 10,000 times (e.g., for 10,000 users and / or instances of the user command "I am cold"), the response activity pattern is relatively strong. On the other hand, if the response activity pattern corresponding to the user command "I am cold" is to perform the task of turning up the thermostat and has been determined twice (e.g., for two users and / or instances of the user command "I am cold"), the response activity pattern is relatively weak. However, even when using confidence scores and thresholds, the determined response activity patterns with confidence scores below the threshold can still be monitored and updated based on additional response activity event observations, because additional observations can increase the confidence in a particular pattern.

[0124] In other cases, direct feedback can be obtained from the user. For example, when a fault state is detected (e.g., determined by the fault state detector 270), the user device can request clarification from the user. Specifically, in one embodiment, when the user command "I am cold" triggers a fault state of the user device (e.g., failure to increase the temperature using the smart thermostat), the user device can request clarification and / or feedback from the user. The clarification and / or feedback can include a direct question to the user, such as: "Do you want me to increase the temperature? I know you are cold." In some implementations, the user's answer (e.g., yes or no) to such a direct question can be used to weight the response activity pattern. For example, if the user answers "yes", then the response activity pattern of increasing the temperature can be weighted more heavily to indicate a high confidence score. If the user answers "no", then the response activity pattern of increasing the temperature can be weighted more heavily downward to indicate a lower confidence score.

[0125] In some embodiments, direct feedback observed from the user can also be responsive to an action of the user device. For example, if the user command "I am cold" causes the user device to turn on the hallway light, the user can state "I said I am cold, not turn on the light." Such a statement can be used to determine that the executed response activity event (e.g., turning on the light) is incorrect (not performing the task associated with the user command correctly). In such a case, the response activity event of turning on the light in response to the user command "I am cold" can be downweighted.

[0126] In some embodiments, a response profile can be generated by associating a response activity pattern (e.g., from response activity pattern 256) with a user command (e.g., response command 252), which can be stored in response action 235. Response action 235 can be a collection or library of responses associated with user commands (e.g., response activity patterns for performing tasks). In other embodiments, response action 235 can be stored as a list or data set. In this way, by way of example and not limitation, response action 235 can map the association between various user commands and various actions performed by one or more user devices to perform tasks corresponding to the user commands. In an implementation, in an initial state, response action 235 can include, for example, immediately available response profiles. In some embodiments, the immediately available response profiles can be pre-programmed response activities. Response action 235 can generally also include learned response profiles (e.g., generated by (one or more) response profile generators 290). Learned profiles can be added by monitoring user actions in response to the absence of a user command in response action 235. For example, when there is no user command in response action 235, a fault state can be triggered (at fault state detector 270). From the user actions monitored after initiating the fault state (e.g., using user activity monitor 280), a response activity pattern can be determined (e.g., using response activity pattern engine 260). In some of these embodiments, such a response activity pattern can be associated with a task performed by one or more user devices in response to an indication of a user command to generate a response profile (e.g., using (one or more) response profile generators 290). The response profile can be added to response action 235. In this way, in an embodiment, when a future indication of a user command corresponding to the response profile is received, the response profile can be used to perform a desired task in response to the user command.

[0127] The response activity initiator 299 typically initiates one or more tasks in response to a user command. The one or more tasks can be based on a response activity pattern (e.g., response activity pattern 256 and / or user response activity pattern 248) associated with the user command (stored in the response action 235). In an embodiment, the response activity initiator 299 can select which response activity(ies) to initiate based on the confidence strength of the (one or more) response activity patterns associated with performing tasks based on the indication of the user command. For example, a minimum confidence score may be required before the response activity initiator 299 initiates a response activity pattern. For example, in one embodiment, a threshold of 0.6 (or just over fifty percent) is utilized such that only response activity patterns with a 0.6 (or greater) likelihood of corresponding to the desired task are considered.

[0128] The exemplary system 200 also includes a presentation component 220, which is generally responsible for presenting content and related information to the user, such as content related to user feedback. The presentation component 220 can include one or more applications or services on the user device, across multiple user devices, or in the cloud. For example, in one embodiment, the presentation component 220 manages presenting content to the user across multiple user devices associated with the user. Based on the content logic unit, device characteristics, associated logic centers, inferred user logical location, and / or other user data, the presentation component 220 can determine on which user device(s) to present the content and the context of the presentation, such as how (or in what format and how much content, which can depend on the user device or context), and when to present. Specifically, in some embodiments, the presentation component 220 applies the content logic unit to the device characteristics, associated logic centers, inferred logical location, or sensed user data to determine aspects of content presentation. For example, clarifications and / or feedback requests can be presented to the user via the presentation component 220.

[0129] In some embodiments, the presentation component 220 generates user interface features associated with clarifications and / or feedback requests. Such features can include interface elements (such as graphical buttons, sliders, menus, audio cues, alerts, warnings, vibrations, pop-up windows, notification bar or status bar items, in-app notifications, or similar features for interfacing with the user), queries, and prompts.

[0130] As previously described, in some embodiments, a personal assistant service or application that operates in conjunction with the presentation component 220 determines when and how (e.g., when it is determined that the user is at a particular logical location) to present content. In such embodiments, content that includes a content logic unit can be understood as a recommendation to the presentation component 220 (and / or the personal assistant service or application) as to when and how to present a notification, which can be overridden by the personal assistant application or the presentation component 220.

[0131] Turning now Figure 3B to, another exemplary embodiment of the techniques described herein is illustratively provided. In Figure 3B the exemplary embodiment shown contrasts with the exemplary conventional interaction depicted in Figure 3A . Specifically, in scenario 380, the same user 305 is shown interacting with a smart speaker 385 (an example of a user device on which a digital assistant application is running). However, contrary to the result in the example of Figure 3A , in Figure 3B , the smart speaker 385 is able to perform the task indicated by the user command 381 ("Hey Cortana, turn on the lights"). Specifically, in scenario 380, the smart speaker 385 responds 399 by letting the user know that it has learned to turn on the lights in the home.

[0132] In this example, after the user command 381, the smart speaker 385 is able to check the user command 381 against a library and / or collection of known (predetermined) commands. The smart speaker 385 (or the digital assistant or computer program running thereon) then determines at least a response activity event and / or a response activity pattern associated with the task and / or the user command. Here, the response activity event and / or the response activity pattern is determined as one or more operations of one or more smart lights in the user's home to turn on the lights.

[0133] As further described herein, in Figure 3BIn some embodiments of the example depicted herein, a user device (e.g., smart speaker 385) or a digital assistant application running thereon can learn to respond to the command 381 "turn on the light" by monitoring the user activity of user 305. In another embodiment, and as further described herein, the smart speaker 385 (or the digital assistant application running thereon) can learn to respond to the command 381 by receiving updated instructions. For example, the smart speaker 385 can access a server of the response profile (or can receive additional response profiles via a device update process), thereby enabling the smart speaker 385 to respond to the user command 381. The response profile can be generated based on monitoring the user activities of other users who make the same or similar commands. In this way, the smart speaker 385 (or a similar user device having a digital assistant application running thereon) can be enabled to correctly respond to a user command on the first instance when the user provides the command, because the device has effectively learned from other users. Although Figure 3B the exemplary embodiments are implemented on a smart speaker, it is contemplated that this embodiment and any smart speaker described herein can be implemented using a digital assistant running on a user device (such as the user's mobile computing device or other user device 102a, as described in Figure 1 ).

[0134] Now turning to Figure 4 and Figure 5 , aspects of exemplary process flows 400 and 500 of embodiments of the present disclosure are illustratively depicted. Process flows 400 and 500 each include methods (sometimes referred to herein as method 400 and method 500) that can be executed to implement many exemplary embodiments described herein. For example, process flow 400 or process flow 500 can be used to facilitate a computing system learning to associate an unknown user command with an appropriate task, such as performing a task desired by the user (using device operations) based on the user command.

[0135] Overviews of process flow 400 and process flow 500 are depicted in Figure 4 and Figure 5 respectively, and the blocks of process flows 400 and 500 corresponding to the actions (or steps) to be performed (as opposed to the information on which the actions are to be performed) can be executed by one or more computer applications or services, which in some embodiments include digital assistants running on one or more user devices (such as user device 104a), servers (such as server 106), can be distributed across multiple user devices and / or servers, or can be implemented in the cloud. In one embodiment, the functions performed by the blocks or steps of process flows 400 and 500 are performed by components of system 200 described in conjunction with Figure 2 .

[0136] Steering Figure 4 provides a flowchart of an exemplary method 400 for determining response activity events associated with performing a task indicated in a user command. At step 410, a first indication of a user command to perform a task can be received. The user command can be characterized as a voice command, a gesture, etc. The task can be associated with the user command. Such a task can be implemented by one or more of the user's computing devices. For example, when the user command is "raise the thermostat", the task of raising the thermostat can be implemented by one or more of the user's computing devices. In some embodiments, the first indication of the user command received at step 410 can be received by a user interactive computing device. The user interactive computing device can receive the first indication of the user command from a sensor, such as the sensor 103 described in connection with Figure 1 the sensor 103 described.

[0137] At step 420, the first indication of the user command can be used to initiate an operation monitoring mode. Embodiments of step 420 can initiate the operation monitoring mode when it is determined that the user command does not correspond to a predetermined response action. The predetermined response action can correspond to performing the task by initiating an operation by one or more user devices, including the user interactive computing device that received the first indication of the user command. These predetermined response actions can be stored as a list, a library, and / or in a database. The predetermined response actions can be stored, for example, in the response action 235 described in connection with Figure 2 the response action 235 described. The user command can be checked against a set of predetermined response actions to determine that there is no corresponding action or operation to perform for a particular user command.

[0138] In response to determining that the user command does not correspond to a predetermined response action, the user interactive computing device can enter a fault state. Specifically, a fault state can occur when, for example, an indication of the user command is received at step 410 and the user command does not correspond to a predetermined response action. When the user interactive computing device enters the fault state, the operation monitoring mode can be initiated. A fault state detector, such as the fault state detector 270 described in connection with Figure 2 the fault state detector 270 described, can be used to detect the fault state.

[0139] At step 430, user activity can be monitored to determine a response activity. Embodiments of step 430 monitor user data associated with at least one user device to identify or detect user activity (sometimes referred to herein as user actions) that occurs after the first indication of the user command is received. Sensor data can be used, using information about the user device associated with the user, to monitor user activity.

[0140] In one embodiment, at step 430, information detected via one or more user devices used by a user and / or cloud-based services associated with the user can be analyzed to determine activity information and related context information. Information about user devices associated with the user can be determined based on user data available via, for example, user data collection component 210 and can be provided to, for example, user activity monitor 280, both of which are described in conjunction with Figure 2 The monitored user data can provide information that can be used to determine the user activity of one or more devices, including identifying and / or tracking features or other information about specific user actions and related context information.

[0141] In one embodiment, user activity performed on at least one device can be determined based on user data available via user data collection component 210, such as that described in conjunction with Figure 2 user activity monitor 280. For example, as previously described, information about user devices can be sensed from or otherwise detected in user data, such as via one or more sensors associated with the user devices, or by detecting and analyzing user device-related information in the user data to determine the above information to determine user devices such as device hardware, characteristics of software such as the OS, network-related characteristics, user accounts accessed via the device, and similar characteristics. In one embodiment, the detected user devices (such as user devices 102a through 102n) can be polled, interrogated, or otherwise analyzed to determine information about the devices. In some implementations, this information can be used to determine a label or identification of the device (e.g., device ID) such that user interactions with one device can be identified from user interactions on another device. In some embodiments of step 410, a user device can be inferred based on a user command, such as in cases where the task of the user command is performed by a specific device, e.g., a user command associated with a task involving temperature can infer that the user device is a smart thermostat, and a user command associated with a task involving playing music can infer that the user device is a smart speaker.

[0142] Some embodiments of step 430 can use at least one sensor to monitor user activity. Specifically, user activity can be monitored based on sensor data from one or more sensors associated with a set of user devices to determine the user activity performed. Some implementations of step 430 can be performed using user activity monitor 280, such as that described in Figure 2 system 200. Additionally, some embodiments of step 430 can use activity event logic units to detect response activity events, as described in conjunction with user activity detector 282. In conjunction with Figure 2The user activity monitor 280 in [the context] provides additional details of the embodiment of step 420.

[0143] The embodiment of step 430 uses monitored user data associated with at least one user device to identify or detect user activities (sometimes referred to herein as user actions) that occur after a first indication of a received user command. The identified user activities can be used to determine response activity events. In some cases, a response activity event can include a series or sequence of user actions and / or interactions with one or more user devices. A response activity event can include at least one user activity performed on at least one user device associated with a task (e.g., the user manually performs a user activity that includes the response activity event, which causes the desired task to be performed using one or more user devices).

[0144] At block 440, the response activity event can be associated with a task. Based on user activity information (e.g., user activities performed on at least one user device, monitored at block 430), the computer system can associate the response activity event with a task (from the user command). In an embodiment, the response activity event can be associated with a task involving one or more user devices. The task can be based on performing the response activity event on the one or more user devices. In some embodiments, block 440 can also include associating the response activity with the user command or associating the task with the user command.

[0145] In an embodiment, at block 440, a record of the association between the response activity event and the task can be stored in a data store. The data store can include multiple records about activity events, where each record can include information about a specific activity event, and the specific activity event includes one or more activity characteristics associated with the activity event. In some embodiments, the multiple records in the activity event data store include records of other activity events determined according to steps 410 to 430 of method 400. In some instances, some of the other records can include information (including associated activity characteristics) about activity events derived from other users determined to be similar to a particular user, as previously described. In an embodiment, the data store includes a user profile and can be stored in a user activity information component, such as the user activity information component 242 of the user profile 240 described in conjunction with Figure 2 as described. In another embodiment, the data store includes a response profile and can be stored in a response activity information component, such as the response profile 250 described in conjunction with Figure 2 as described.

[0146] In an embodiment, multiple records that can analyze the relevance of response activity events to a task are used to determine a response activity pattern. The response activity pattern can be used to determine the operations to be performed to implement the task. The response activity pattern can be associated with or used to generate a response profile corresponding to a user command, and is associated with the task based on the history of sensed response activity event(s) (including the response activity pattern). In this way, when an indication of a user command is received in the future, the response activity event(s) associated with the task can be determined (using the response profile) and used to cause the execution of operations to implement the task.

[0147] In an embodiment, the response profile can be stored and made available to other user-interactive computing devices (e.g., referring to Figure 2 the response action 235 described). For example, the response profile can be published on a server accessible to other user-interactive devices. The stored response commands can include a list of one or more user commands that have known response activity patterns for performing operations to implement the task. The published response profile can also include an indication of at least one user device determined based on monitoring user activity. The published response profile can allow a particular user device to learn to perform a task even if the particular user device did not previously understand the user command (e.g., using the device mapping 254, as discussed in reference to Figure 2 ). In some embodiments, a cloud system (such as the cloud system described above) and / or cloud services can be utilized to perform method 400 to provide an improved or enhanced user experience (such as the published response profile) to multiple services running on many different user devices.

[0148] At step 450, a second indication of a user command to perform the task can be received. The user command and task in step 450 can be the same as or similar to the user command and task in step 410. Similar user commands can result in the same task as the user command (e.g., "I'm cold", "It's cold here", "Turn up the temperature", all result in the task of turning up the temperature). In some embodiments, the second indication of the user command received at step 450 can be received by the user-interactive computing device that received the first indication of the user command at step 410. In other embodiments, the second indication of the user command received at step 450 can be received by a user-interactive computing device different from the user-interactive computing device that received the first indication of the user command at step 410. The second indication of the user command can be received from a sensor, such as in combination with Figure 1The described sensor 103. Thus, an embodiment of step 450 may receive a second indication of a user command to perform the task indicated by the first indication received in step 410. Additionally, an embodiment of step 450 may be performed after step 440, which, as described above, may associate a response activity event with the task.

[0149] At step 460, based on the second indication of the user command, the response activity event associated with the task can be determined. The response activity event can be determined based on predefined response actions stored as a list, library, and / or database. For example, the predefined response actions can be based on the association of the response activity event with the task at step 440. The predefined response actions can be stored, for example, in the response action 235 described in connection with Figure 2 The described response action 235. The second indication of the user command can be checked against a set of predefined response actions (e.g., referring to Figure 2 The described response action 235).

[0150] At step 470, the operation can perform the task. In an embodiment, the response activity event determined at step 470 can indicate the operations to be performed to execute the task associated with the indication of the user command. For example, some embodiments may cause the operation to perform a response activity event associated with one or more user computing devices (in response to a received user command). In some embodiments, the operation of performing a task in response to a user command can be based on a (generic or user-based) response profile for initiating a task on one or more devices.

[0151] Now referring to Figure 5 , a flowchart of an exemplary method 500 for determining possible future user actions based on patterns of user activity is provided. At step 510, a first indication of a user command to perform a task can be received. The first indication of the user command can be received as previously described with reference to Figure 2 Or Figure 4 . In an embodiment, the first indication of the user command can be received by a user interactive computing device. The user interactive computing device can receive the first indication of the user command from a sensor, such as the sensor 103 described in connection with Figure 1 .

[0152] At step 520, a monitoring session can be started. The monitoring session to be initiated can be determined based on the user command indicated in step 510. For example, the embodiment of step 420 can initiate an operation monitoring mode when it is determined that the user command does not correspond to a predetermined response action. The predetermined response action can correspond to performing a task by initiating an operation through one or more user devices, including the user interactive computing device that received the first indication of the user command. In response to determining that the user command does not correspond to the predetermined response action, the user interactive computing device can enter a fault state (i.e., because the user command is not recognized as explained by the fault state detector 270 as in Figure 2 ). The monitoring session can be performed as previously referenced Figure 2 or Figure 4 .

[0153] At step 530, a response activity event can be determined. In an embodiment, the response activity event can be determined based on user activity associated with at least one user device from the monitoring session. Determining the response activity event based on user activity associated with user devices from the monitoring session can be performed as previously referenced Figure 2 or Figure 4 . At step 540, the response activity event can be associated with the user command. For example, the response activity event can be monitored, tracked, and used to determine a response activity pattern associated with the user command. The response activity pattern can refer to multiple user interactions with one or more user devices, user activities on or related to one or more user devices, events (including actions) related to user activities, or any type of response activity event that can be determined via a computing device, where the multiple interactions, actions, events, or activities share a common characteristic or feature.

[0154] In some embodiments, step 540 may further determine a confidence score for associating a response activity event (e.g., based on a response activity pattern) with a user command indicating that the response activity event is associated with a task (of a user command). For example, the confidence score may be based on the strength of the pattern, which may be determined by the number of observations used to determine the pattern, the frequency with which the (one or more) response activity events are consistent with the pattern, the age or freshness of the activity observations, the number of features common to the (one or more) response activity event observations that make up the pattern, or similar metrics. In some instances, the confidence score may be considered when providing a personalized user experience or other improved user experience. Additionally, in some embodiments, a minimum confidence score may be required before using a response activity pattern to provide such an experience or other service. For example, in one embodiment, a threshold of 0.6 (or just over fifty percent) is utilized such that only response activity patterns having a 0.6 (or greater) likelihood of performing the desired task corresponding to the user command are used to initiate a response activity event for performing the task. However, in cases where confidence scores and thresholds are used, response activity patterns determined for (one or more) response activity events having a confidence score less than the threshold may still be monitored because additional observations of the (one or more) response activity events may increase the confidence in a particular pattern.

[0155] In some embodiments, a response activity event can be characterized as semantic information, such as time, location, or other contextual information associated with the response activity event, as further described herein. Example associations of response activity events corresponding to user commands include time (e.g., a user command of "turn on the lights" at 7:00 a.m. involves a response activity pattern of turning on the master bedroom lights, master bathroom lights, and kitchen lights, whereas a user command of "turn on the lights" at 5:30 p.m. involves a response activity pattern of turning on the kitchen lights, living room lights, and dining room lights), location (e.g., a user command of "play music" at work involves a response activity pattern of playing classical music, whereas a user command of "play music" at home involves a response activity pattern of playing home music), content (e.g., a user command of "it's a bit cold in here" involves a response activity pattern of turning up the smart thermostat by two degrees, whereas a user command of "I'm cold" involves a response activity pattern of turning up the smart thermostat by four degrees), or other context, as described herein.

[0156] At step 550, a second indication of the user command to perform the task can be received. In an embodiment, the second indication of the user command can be received after associating the response activity event with the user command. The second indication of the user command can be as previously referenced Figure 2 or Figure 4Received as described. In an embodiment of step 550, the same and / or similar user commands as those received at step 510 can be received. In some embodiments, a second indication of the user command received at step 550 can be received by the user interactive computing device that received the first indication of the user command at step 510. In other embodiments, a second indication of the user command received at step 550 can be received by a user interactive computing device different from the user interactive computing device that received the first indication of the user command at step 510. The second indication of the user command can be received from a sensor, such as the sensor 103 incorporated Figure 1 as described.

[0157] At step 560, a response activity event associated with the task can be determined. Determining the response activity event associated with the task can be performed as previously referenced Figure 2 or Figure 4 as described. The response activity event can be determined based on the second indication of the received user command. In an embodiment, the response activity event can be determined based on predetermined response actions stored as a list, library, and / or in a database. For example, the predetermined response actions can be based on the association of the response activity event with the task at step 540. The predetermined response actions can be stored, for example, in the response action 235 incorporated Figure 2 as described. The second indication of the user command can be checked against a set of predetermined response actions (e.g., the response action 235 described with reference to Figure 2 ).

[0158] Various implementations have been described. Now, an exemplary computing environment suitable for implementing the embodiments of the present disclosure is described. Referring to Figure 6 , an exemplary computing device is provided and is generally referred to as computing device 600. Computing device 600 is only one example of a suitable computing environment and is not intended to imply any limitation as to the scope of use or functionality of the embodiments of the present disclosure. Computing device 600 should not be construed as having any dependency or requirement on any one or combination of the components shown.

[0159] Embodiments of the present disclosure may be described in the general context of computer code or machine-usable instructions, including computer-usable or computer-executable instructions executed by a computer or other machine, such as a personal data assistant, a smart phone, a tablet PC, or other handheld device, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., referring to code that performs specific tasks or implements specific abstract data types. Embodiments of the present disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices. Embodiments of the present disclosure may also be practiced in a distributed computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including memory storage devices.

[0160] Reference Figure 6 , computing device 600 includes a bus 610 that directly or indirectly couples the following devices: a memory 612, one or more processors 614, one or more presentation components 616, one or more input / output (I / O) ports 618, one or more I / O (I / O) components 620, and an exemplary power supply 622. Bus 610 represents one or more buses (such as an address bus, a data bus, or a combination thereof). Although each block is shown as a line for clarity Figure 6 of the various blocks, in fact, these blocks represent logical units and not necessarily actual components. For example, one may consider a presentation component, such as a display device, as an I / O component. Similarly, a processor has memory. The inventors recognize this as the nature of the art and reiterate Figure 6 that the diagrams herein are merely illustrative of exemplary computing devices that can be used in conjunction with one or more embodiments of the present disclosure. No distinction is made between categories such as "workstation", "server", "laptop computer", "handheld device", etc., because all of these are contemplated within Figure 6 the scope of and are referred to as "computing device".

[0161] Computing device 600 generally includes various computer-readable media. Computer-readable media can be any available media that can be accessed by computing device 600 and includes volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to: RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disks (DVDs) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by computing device 600. Computer storage media does not include the signal itself. Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0162] Memory 612 includes computer storage media in the form of volatile and / or non-volatile memory. The memory can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid state memory, hard disk drives, optical disk drives, etc. Computing device 600 includes one or more processors 614 that read data from various entities such as memory 612 or I / O components 620. One or more presentation components 616 present data indications to a user or other device. In some implementations, presentation component 220 of system 200 can be embodied as presentation component 616. Other examples of presentation components can include display devices, speakers, printing components, vibration components, etc.

[0163] The I / O port 618 allows the computing device 600 to be logically coupled to other devices, including the I / O components 620, some of which may be built-in. Exemplary components include microphones, joysticks, gamepads, satellite dishes, scanners, printers, wireless devices, etc. The I / O components 620 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by the user. In some instances, the input can be transmitted to appropriate network elements for further processing. The NUI can implement any combination of speech recognition, touch and stylus recognition, face recognition, biometric recognition, gesture recognition both on and near the screen, air gestures, head and eye tracking, and touch recognition associated with the display on the computing device 600. The computing device 600 can be equipped with a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, and combinations thereof, for gesture detection and recognition. Additionally, the computing device 600 can be equipped with an accelerometer or gyroscope capable of detecting motion. The output of the accelerometer or gyroscope can be provided to the display of the computing device 600 to present immersive augmented reality or virtual reality.

[0164] Some embodiments of the computing device 600 may include one or more radio devices 624 (or similar wireless communication components). The radio devices 624 transmit and receive radio or wireless communications. The computing device 600 can be a wireless terminal adapted to receive communications and media via various wireless networks. The computing device 600 can communicate via wireless protocols such as Code Division Multiple Access (“CDMA”), Global System for Mobile Communications (“GSM”), or Time Division Multiple Access (“TDMA”) to communicate with other devices. The radio communication can be a short-range connection, a long-range connection, or a combination of both short-range and long-range radio connections. When we refer to “short” and “long” types of connections, we do not mean the spatial relationship between two devices. Instead, we generally refer to short-range and long-range as different categories or types of connections (i.e., primary and secondary connections). By way of example and not limitation, a short-range connection can include a connection to a device that provides access to a wireless communication network (e.g., a mobile hotspot), such as a WLAN connection using the 802.11 protocol; a Bluetooth connection to another computing device is a second example of a short-range connection or a near-field communication connection. By way of example and not limitation, a long-range connection can include a connection using one or more of the CDMA, GPRS, GSM, TDMA, and 802.16 protocols.

[0165] Many different arrangements of the various components depicted and components not shown are possible without departing from the scope of the following claims. Embodiments of the present disclosure have been described with the intent to be illustrative rather than restrictive. Alternative embodiments will become apparent to the reader of the present disclosure after and because of reading the present disclosure. Alternative ways of implementing the foregoing can be accomplished without departing from the scope of the following claims. Specific features and subcombinations have utility and can be employed without reference to other features and subcombinations and are contemplated within the scope of the claims.

[0166] Thus, in one aspect, embodiments of the present disclosure relate to a computerized system, comprising: one or more sensors configured to provide sensor data; one or more processors; and a computer storage memory having computer-executable instructions thereon, the computer-executable instructions when executed by the processors implement a method. The method includes: receiving a first indication of a user command to perform a task; based on the first indication of the user command, determining to initiate an operation monitoring mode; after associating a response activity event with the task, receiving a second indication of a user command to perform the task; based on the second indication of the user command, determining the response activity event associated with the task; and causing an operation to be performed to implement the task.

[0167] In some embodiments of the system, the monitoring mode includes monitoring user activity using at least one sensor to determine a response activity event, the response activity event including at least one user activity performed on at least one user device; and associating the response activity event with the task. Further, in some embodiments, determining to initiate the operation monitoring mode includes, for example but not limited to: determining that the user command does not correspond to a predetermined response action initiated by a user interactive computing device; and in response to determining that the user command does not correspond to the predetermined response action, causing the user interactive computing device to enter a fault state, wherein the user interactive computing device is operable to learn new response actions based on monitoring user activity occurring after the first indication of the received user command using the at least one sensor.

[0168] In another aspect, embodiments of the present disclosure relate to a computerized system comprising: one or more sensors configured to provide sensor data, each sensor being associated with a user device in a set of user devices associated with a user; one or more processors; and a computer storage memory having computer-executable instructions stored thereon, the computer-executable instructions, when executed by the processor, implementing a method. The method includes: receiving a first indication of a user command to perform a task; based on the first indication of the user command, determining to initiate monitoring; monitoring user activity to determine a response activity event, the response activity event including at least one user activity performed on at least one user device; determining the response activity event based on the user activity; associating the response activity event with the task; after associating the response activity event with the task, receiving a second indication of the user command to perform the task; based on the second indication of the user command, determining the response activity event associated with the task; and causing an operation to be performed to implement the task.

[0169] In yet another aspect, embodiments of the present disclosure relate to a computerized method. The method includes: receiving a first indication of a user command to perform a task; and based on the user command, starting a monitoring session to monitor user activity associated with at least one user device. The method further includes: determining a response activity event based on the user activity associated with the at least one user device from the monitoring session; and associating the response activity event with the user command. Additionally, the method includes: after associating the response activity event with the user command, receiving a second indication of the user command to perform the task; based on the second indication of the user command, determining the response activity event associated with the user command; and based on the response activity event, causing an operation to be performed to implement the task.

Claims

1. A system, comprising: at least one processor; and a computer storage memory having computer-executable instructions stored thereon, the computer-executable instructions, when run by the at least one processor, implement a method, the method comprising: receiving a first indication of a user command to perform a task; determining that the user command is not recognized; based on the user command not being recognized, triggering an increased monitoring level compared to a monitoring level when the first indication is received, the increased monitoring level comprising: monitoring user activity to determine a first response activity event comprising at least one user activity operation performed by the user on at least one user device; determining first context information associated with the first response activity event; and associating the first response activity event with the task, wherein associating the first response activity event with the task comprises associating the first context information with the first response activity event; after associating the first response activity event with the task, receiving a second indication of the user command to perform the task; determining second context information based on the received second indication of the user command; comparing the first context information with the second context information; based on the comparison of the first context information with the second context information, determining that the user command associated with the second indication is recognized; and based on the user command associated with the second indication being recognized, causing at least one of the following to be performed: (1) the task or (2) the at least one user activity operation.

2. The system according to claim 1, wherein, the method further comprises: storing a record of the association of the first response activity event with the task in a data store; determining a plurality of response activity patterns comprising associations of the performed task with the at least one user activity operation based on an analysis of a plurality of records of the association of response activity events with the task; and using the response activity pattern and the second indication of the user command to determine the association of the first response activity event with the task.

3. The system according to claim 2, wherein, the method further comprises: generating a response profile based on the determined first response activity pattern; and storing the response profile for use on other user interactive computing devices.

4. The system according to claim 3, wherein, storing the response profile for use on the other user interactive device comprises: publishing the response profile on a server accessible by the other user interactive device.

5. The system according to claim 4, wherein, the method further comprises: determining an indication of the at least one user device based on monitoring the user activity; and including the indication of the at least one user device in the published response profile.

6. The system according to claim 1, wherein, The first context information differentiates between the first response activity event associated with the user command and a second response activity event associated with the user command.

7. The system according to claim 1, wherein, the at least one user device on which the user activity is monitored is the same as or included in the system.

8. The system according to claim 1, wherein, the first indication of the user command for performing the task includes at least one of the following: a verbal command, a gesture, or a physical interaction.

9. The system according to claim 1, wherein, monitoring the user activity to determine the first response activity event includes monitoring a plurality of user devices.

10. The system according to claim 1, wherein, determining that the user command is not recognized includes: determining that the user command does not correspond to a predetermined response action to be initiated by the at least one user device; and in response to determining that the user command does not correspond to the predetermined response action, causing the at least one user device to enter a fault state, wherein the at least one user device is operable to learn new response actions based on the monitoring of the user activity occurring after the first indication of the received user command.

11. The system according to claim 1, wherein, the first context information is determined based on user-specific context features.

12. A computer-implemented method, comprising: receiving a first indication of a user command for performing a task; determining that the user command is not recognized, wherein determining that the user command is not recognized includes: determining that the user command does not correspond to a predetermined response action to be initiated by at least one user device; and in response to determining that the user command does not correspond to the predetermined response action, causing the at least one user device to enter a fault state; triggering a monitoring session with an increased monitoring level to monitor user activity associated with the at least one user device based on the user command not being recognized and the at least one device entering the fault state, as compared to a monitoring level when the first indication was received; determining a first response activity event based on the user activity associated with the at least one user device from the monitoring session, the first response activity event including at least one user activity operation performed by the user on the at least one user device; associating the first response activity event including the at least one user activity operation performed by the user with the user command, wherein associating the first response activity event with the user command includes associating first context information with the first response activity event; after associating the first response activity event with the user command, receiving a second indication of the user command for performing the task; determining the first response activity event associated with the user command based on the second indication of the user command and the first context information; and Based on the first response activity event, causing the at least one user activity operation to be performed to facilitate the execution of the task.

13. The computer-implemented method according to claim 12, wherein, starting the monitoring session based on the user command includes: comparing the user command with one or more predetermined commands; based on the comparison, determining that the user command does not exist in the one or more predetermined commands; and initiating the monitoring session.

14. The computer-implemented method according to claim 12, further comprising: storing a record of the association between the first response activity event and the user command in a data store, the data store including a plurality of records of the association between response activity events and at least one corresponding user command; based on an analysis of the plurality of records of the association between response activity events and the at least one corresponding user command, determining a response activity pattern including a plurality of associations between the at least one corresponding user command and the at least one user activity operation performed; and using the response activity pattern and the second indication of the user command to determine the first response activity event.

15. The computer-implemented method according to claim 14, further comprising: determining a confidence score for the response activity pattern, the confidence score being determined based on the number of the plurality of records having a similar association between the user command and the at least one user activity operation performed, wherein the confidence score indicates the probability that the first response activity event is associated with the user command; generating a response profile based on the determined first response activity pattern; and associating the response profile with a time or a location, wherein the task corresponds to the time or the location.

16. The computer-implemented method according to claim 12, wherein, the first context information distinguishes between the first response activity event associated with the user command and a second response activity event associated with the user command.

17. A computer storage medium having computer-executable instructions embedded thereon, the computer-executable instructions causing computing operations to be performed when run by at least one computer processor, comprising: receiving a first indication of a user command to perform a task; determining that the user command is not recognized; based on the user command not being recognized, triggering an increased monitoring level compared to the monitoring level when the first indication was received, the increased monitoring level including: monitoring user activity to determine a first response activity event including at least one user activity operation performed by the user on at least one user device; determining first context information associated with the first response activity event; and associating the first response activity event with the task, wherein associating the first response activity event with the task includes associating the first context information with the first response activity event; after associating the first response activity event with the task, receiving a second indication of the user command to perform the task; Determine second context information based on a second indication of the received user command; Compare the first context information with the second context information; Based on the comparison of the first context information and the second context information, determine that the user command associated with the second indication is recognized; and Based on the recognition of the user command, cause the at least one user activity operation to be executed to facilitate the execution of the task.

18. The computer storage medium according to claim 17, wherein, the operation further includes: Storing a record of the association between the first response activity event and the task in a data store; Determining a response activity pattern including multiple associations of the executed task and the at least one user activity operation based on an analysis of multiple records of the association between the response activity event and the task; and Using the response activity pattern to determine the first response activity event.

19. The computer storage medium according to claim 17, wherein, Causing the at least one user activity operation to be executed to facilitate the execution of the task includes initiating the at least one user activity operation on the at least one user device.

20. The computer storage medium according to claim 17, wherein, Determining that the user command is not recognized includes: Determining that the user command does not correspond to a predetermined response action to be initiated by the at least one user device; and In response to determining that the user command does not correspond to the predetermined response action, causing the at least one user device to enter a fault state, wherein the at least one user device is operable to learn new response actions based on the monitoring of the user activity occurring after the first indication of the received user command.

Images