Self-learning digital assistant

Abstract

A response activity pattern can be determined from a set of computing devices obtained during an operation monitoring mode. When a user command is determined to be a new command or an unknown command, the operation monitoring mode can be initiated. The response activity pattern can be used to generate a response profile, which indicates an activity to be performed using one or more user devices to perform a task associated with the user command. When an indication of a previously unknown user command to perform a task is received, the generated response profile can be used to perform the designed task by performing the activity using one or more user devices. In this way, the activity that should be performed to perform the task can be learned based on the monitored user activity associated with the one or more user devices.

Description

Background Art

[0001] People are increasingly interacting with computing devices and relying on them for information, recommendations, and other services to assist them in completing their daily tasks. For example, individuals can interact with smart speakers, smart watches, or their mobile devices by speaking commands, such as asking for directions or turning on a light. Many of these "smart" computing devices include intelligent functions for responding to the user's commands, such as digital assistant applications running on the computing devices. However, these computing devices (or their digital assistant applications) often do not respond to the user's commands because they do not know how to perform the tasks associated with the commands. In such cases, these smart devices do not seem to be smart at all and frustrate the user. Summary of the invention

[0002] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used alone as an aid in determining the scope of the claimed subject matter.

[0003] Embodiments of the technology described in the present disclosure relate to systems and methods for improving digital assistant technology and improving the ability of computing devices to accurately respond to user commands by extending the functionality of digital assistants through learning mechanisms. At a high level, an interactive computing device (such as a smart speaker or mobile device) or a digital assistant program running thereon is able to learn to perform specific response activities associated with specific user commands. The user command may include a request to perform a task, such as turning on a light. For example, when an unknown command is received from a user, the computing device does not ignore the command or respond to the user with only an indication that the command is unknown, but monitors user activities to learn operations performed by the user. Specifically, when it is determined that a command received from a user is not recognized, the computing system (or digital assistant) can enter a special operating mode in which the user's activities on the computing device that occur after the unrecognized command are monitored. User activities can be monitored on a computing device or on another computing device, and the user can consent to the monitoring operation. In this way, the computer system learns the operations to be performed to implement the task in response to new or previously unknown user commands.

[0004] For example, in an embodiment, a user interactive computing device can receive an indication of a user command to perform a task. When it is determined that the user command does not correspond to a known response action, an operation monitoring mode can be initiated. Specifically, a user command library or a set of known commands and / or responses can be used to determine whether the received user command is known or unknown. Therefore, when the user command does not exist in the set or library of known commands, the operation monitoring mode can be initiated. In some embodiments, when it is determined that the user command is unknown, the computing device (or a digital assistant application or software service running on the computing device) can enter a fault state, during which monitoring of user-related activities is performed. The fault state can be characterized by enhanced user activity monitoring performed for the system to learn the response action to be associated with the unknown user command. In this way, the response activity can be learned from the monitored user activity. In some cases, user consent can be obtained before performing enhanced user activity monitoring, and in some cases, the computing device can provide the user with an indication that an operation monitoring mode (or learning mode) is occurring.

[0005] When initiating an operational monitoring mode, a computing device associated with a user (i.e., a "user device") may employ one or more sensors and / or monitoring software services to generate data related to the user's activities on (one or more) user devices. Such user activities can be monitored as (one or more) response activity events (which may be performed by the user or under the guidance of the user) associated with one or more user devices. The (one or more) response activity events may be monitored, tracked, and used to determine a response activity pattern. As described herein, the response activity pattern may include, but is not limited to, patterns based on time, location, content, or other contexts. In some embodiments, a response activity pattern may be determined based on a response activity event involving user activities associated with one or more user devices or otherwise determinable via one or more user devices.

[0006] Based on the determined response activity pattern, a response profile corresponding to the user command can be determined and used to provide an improved user experience. For example, a response profile can specify one or more user commands and corresponding operations to be performed to implement the task associated with the command. In some embodiments, the response profile can be determined based on user activities learned from multiple users, and in some embodiments, the response profile can also include a user-device mapping that enables the operation to be performed on different types of user devices, such as smart lights or smart thermostats of different brands. An example of an improved user experience further described herein may include an improved user experience of interacting with a computing device based on the computing device understanding a previously unknown user command and causing an operation to be performed to implement a task associated with the user command.

[0007] In some embodiments, the response activity pattern can be made available to one or more applications and services that consume the information to provide an improved user experience. For example, the response activity pattern (or information derived from the pattern, such as operations to be performed to implement a task) can be included in a response profile provided on a server or made accessible to other computing devices so that the device can learn to respond to new or unknown commands. In this way, digital assistant technology is improved by enabling a computing device (or a digital assistant running thereon) to learn to respond to new or previously unknown commands, thereby expanding the functionality of the computing device and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The following describes various aspects of the present disclosure in detail with reference to the accompanying drawings, wherein:

[0009] Figure 1 is a block diagram of an exemplary operating environment suitable for implementing the present disclosure;

[0010] Figure 2 is a diagram depicting an exemplary computing architecture suitable for implementing aspects of the present disclosure;

[0011] Figure 3A An example of a conventional voice-based interaction between a user and a digital assistant operating on a computing device, which is a smart speaker, is depicted;

[0012] Figure 3B depicts an exemplary speech-based interaction between a user and a computing system according to an embodiment of the present disclosure;

[0013] Figure 4 and Figure 5 each depicts a flow chart of a method for a computing system to expand its functionality by learning to perform new tasks associated with user commands based on monitoring user activity according to an embodiment of the present disclosure; and

[0014] Figure 6 is a block diagram of an exemplary computing environment suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0015] The subject matter of various aspects of the present disclosure is specifically described herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this patent. On the contrary, the inventors have contemplated that the claimed subject matter may also be embodied in other ways to include steps different from the steps described in this document or combinations of similar steps in conjunction with other existing or future technologies. In addition, although the terms "step" and / or "block" may be used herein to imply different elements of the method employed, the terms should not be interpreted as implying any particular order among or between the various steps disclosed herein, unless and in addition to the order of the individual steps being explicitly described.

[0016] Various aspects of the technology described herein relate to systems, methods, and computer storage media for improving digital assistant technology and improving the ability of computing devices to accurately respond to user commands by extending device functionality through learning mechanisms. User commands can generally refer to commands, statements, or declarations that indicate or correspond to a request to perform a specific task or operation. A digital assistant or personal digital assistant, which may also be generally referred to as a virtual assistant, includes one or more computer programs and can operate on or across one or more computing devices. For example, one such virtual assistant is provided by Corporation Assistant. Specifically, embodiments of the computing devices described herein (and / or digital assistants or software applications running thereon) can learn to perform tasks in response to specific user commands. For example, during user interaction with a computing device, a user can provide commands to the computing device by speaking (i.e., voice commands), by gestures, touch, or other input means. The command can be received by a user interface or sensor, such as a microphone, camera, touch screen, keyboard, or other input means or user interface. The computing device may include a smart device, such as a smart speaker, a mobile device with an app or interactive service, a digital assistant program running on a computing device, or other examples of computing devices described herein. The user command may include a request to perform a specific task, such as, for example, but not limited to: turning on the lights, playing or recording a TV program, launching an application on a computing device, navigating to a website, arranging a meeting, booking a meeting or venue, configuring settings on a computing device or appliance, or using one or more computing devices to perform an operation or a series of operations (e.g., launching an app or navigating to a specific website and initiating an operation, such as a restaurant reservation, a travel reservation, a purchase, playing a video, retrieving, and presenting or reading information (such as a news feed) or (one or more) other operations). The task performed in response to a user command may include a set of one or more operations performed via the computing device or another computing device.

[0017] As previously described, as the popularity of computing devices increases, user interactions with computing devices are becoming more frequent. For example, many people now have smart speakers or other "smart" computing devices at home, in their vehicles, or wear such smart devices on their bodies. User interactions in which a user requests the device to perform a task on behalf of the user are also becoming more common. When a computing device knows a particular user command (whether the command is pre-programmed or learned), the computing device's response to the command may involve performing (or causing to be performed) one or more operations (or tasks) associated with the user command. In some cases, a portion of these operations may be performed by another computing device. For example, if the user command is "turn on the thermostat", the task of adjusting the temperature setting on the thermostat may be performed by a computing device, such as a mobile device on which an app for controlling the thermostat is installed, and / or by another computing device (such as a smart thermostat). The ubiquity of these smart computing devices and the frequency with which users interact with them have created an expectation among users that these devices will respond to the user's commands.

[0018] However, when the computing device is unaware of the user command, the task (or (one or more) corresponding operations) will not be initiated and / or performed. Users may become frustrated when these smart devices are unable to perform the requested task. Figure 3A An example thereof is illustratively depicted in Figure 3A The interaction between a user and a conventional computing device (smart speaker) is shown. Specifically, Figure 3A A series of interactions between a user 305 and a smart speaker 315 are shown. In the first interaction (scenario 310), in response to the user's command 311 to perform the task of having the TV record the Seahawks game, a conventional smart speaker (or more specifically, a conventional digital assistant running on the smart speaker) responds at 317 with "I don't understand." As shown in scenario 320, the user 305 begins to lose patience with the smart speaker 315. In response to the user's command 321 to tell the smart speaker 315 that he is cold (which may mean performing the task of turning up the heat), the smart speaker 315 responds 371 ("I don't understand"). By scenario 330, the user 305 becomes very annoyed. As shown in scenario 330, in response to the user's command 331 to tell the smart speaker 315 to turn off the lights (e.g., perform the task of turning off the lights), the smart speaker 315 responds 371 ("I don't understand"). At scenario 340, the user 305 becomes even more frustrated. In scene 340, in response to the user's command 341 telling the smart speaker 315 to turn on the home alarm system (e.g., performing the task of activating the home alarm system), the smart speaker 315 responds 371 ("I don't understand"). By scene 350, the user 305 is very angry. In response to the user's command 351 telling the smart speaker 315 to play his favorite songs (e.g., performing the task of playing the user's favorite songs), the smart speaker 315 responds 371 ("I don't understand"). In scene 360, the user 305 is annoyed with the smart speaker (e.g., smart speaker 315) and thinks that the smart speaker is not very smart.

[0019] Typically, when a conventional smart device receives an unrecognizable command, the command is ignored by the device (or a program running on the device), or the device (or a program running on the device) may respond with an indication that the command is unknown. Figure 3A In contrast to the frustrating experience depicted in , embodiments of the technology described herein improve the operation of computing devices by extending the functionality of computing devices through learning mechanisms to effectively respond to user commands. In this way, the embodiments described herein also improve the user experience of interacting with these computing devices or digital assistant applications running thereon. Instead of ignoring unknown commands or responding to the user with only an indication that the command is unknown, the embodiments described herein monitor user activities performed after receiving an unknown command in order to learn the operations performed by the user and associate those operations with the command.

[0020] Specifically, when it is determined that the command received from the user is not recognized, the computing system (or digital assistant) can enter a special operating mode in which the user's activities are monitored. User activities can be monitored on a computing device or on one or more other computing devices. Sensors on a computing device and / or one or more user devices can be used to collect data corresponding to user activities over time. Based on the monitored user activity data, specific operations associated with performing tasks corresponding to the user commands can be known. In this way, the computing system (or digital assistant application) learns the operations to be performed in response to the user commands in order to facilitate the execution of the tasks. In some cases, user consent can be obtained before performing the user activity monitoring, and in some cases, the computing device can provide the user with an indication that an operation monitoring mode (or learning mode) is occurring.

[0021] Therefore, at a high level, various aspects of the present disclosure relate to technologies that enable a computer system to learn to perform new tasks in response to new or previously unknown user commands. In some embodiments, a task can be understood as a collection of one or more computing device operations that a user wishes to be performed by one or more user devices. Such operations can involve activities associated with one or more user devices. When it is determined that a user command is not recognized, the computing system can initiate monitoring of the user's activities. Such monitored user activities can be used to determine the operations to be performed in order to perform tasks related to the user commands. For example, the next time the computer system receives an indication of a previously unknown user command, the system can implement its learned knowledge by using one or more user devices to perform activities to perform the specified task. In this way, the computer system learns what tasks to perform in response to new or previously unknown user commands.

[0022] In one embodiment, an indication of a user command to perform a task can be received by a user interactive computing device. The user command or request can typically be received as a voice command, gesture, input into a computing device via a touch screen, keyboard, camera, microphone, other sensor, or other input unit. As described herein, the command can include a request to perform a specific task and / or can correspond to a set of one or more operations to be performed in order to perform the task. The task can be performed by one or more user computing devices.

[0023] In some embodiments, a library or set of known commands and / or responses can be utilized to determine whether the received command is known or unknown. In addition, in some instances, determining whether a user command is known or unknown may involve performing a similarity comparison. In these embodiments, a similarity comparison can be performed on various aspects of the received user command to determine the similarity with other known user commands. For example, using feature similarity techniques such as clustering or Word2Vec, a measure of similarity, such as a similarity score, can be determined. In some embodiments, when the similarity score is high or meets a predetermined threshold, it can be determined that the command is similar to a known command, and an operation corresponding to the known command can be performed. However, when the similarity score does not meet a predetermined threshold, it can be determined that the command is not similar to the set of known commands.

[0024] When it is determined that the user command is unknown, an operational monitoring mode may be initiated. In some embodiments, user activity may be monitored via a computing device and other user devices, and the user activity may be identified from monitored user data, as described herein. In some embodiments, an application or service running on a user device may be used to facilitate user activity monitoring. Alternatively or additionally, an application or service running in the cloud may be used to facilitate the user activity monitoring, which may scan, poll, or otherwise monitor a user device, or may detect online activity associated with the user device, such as http requests or other communication information, or otherwise receive information about user activity from the user device.

[0025] In some cases, upon determining that a user command is unknown, the computing device (or a digital assistant application or software service running on the computing device) may enter a fault state during which monitoring of user-related activity is performed. For example, the fault state may include initiating a program routine or service for performing monitoring of user activity. In some embodiments, the fault state (or more generally, an operating monitoring mode) may be indicated to the user, such as by providing a notification or by the device providing an indication, such as by changing the color of a light on the device or changing the color of an app associated with the device, or similarly using the device to emit a sound or a sound provided by an app associated with the device, or otherwise providing the user with a notification that the device is monitoring activity to learn how to respond to unknown commands (e.g., which may be output by presentation component 220).

[0026] When in monitoring mode, user activity can be monitored as (one or more) responsive activity events (which can be performed by the user) on one or more user devices. Response activity events can generally be determined based on at least one monitored user activity operation performed on at least one user device (for example, a user manually performs a response activity event to perform a desired task using one or more user devices). In some embodiments, the (one or more) responsive activity events can be related to (one or more) direct interactions of the user with one or more devices during monitoring initiated based on user commands. Additionally or alternatively, the (one or more) responsive activity events can be related to (one or more) user interactions with one or more devices using application (or app) related activities, such as application usage, which can include usage duration, startup, files accessed via or in conjunction with application use, or content associated with the application. The term "application" or app is used broadly herein and generally refers to a computer program or computer application, which can include one or more programs or services and can run on (one or more) user devices or in the cloud.

[0027] As further described herein, in some embodiments, the user device may employ one or more sensors to generate data related to the response activity events of (one or more) users via (one or more) user devices. In some embodiments, these response activity events can occur and be detected during the operation monitoring mode. Specifically, sensors on one or more user devices associated with the user may be used to collect data corresponding to such response activity events over time (e.g., in monitoring mode).

[0028] The response activity event data can be monitored, tracked, and used to determine response activity patterns. The term "response activity pattern" is used broadly herein and may refer to multiple user interactions with one or more user devices, activities by users on or related to one or more user devices, events (including actions) related to user activities, or any type of response activity event that can be determined via a computing device, wherein multiple interactions, actions, events, or activities share common features or characteristics. In some embodiments, these common features or variables may include features that characterize response activity events, times, locations, or other types of contextual information associated with the response activity events, as further described herein. Examples of response activity patterns may include, but are not limited to, response activity patterns corresponding to user commands based on time (e.g., a user command “turn on the lights” at 7:00 a.m. is associated with a response activity pattern of turning on the master bedroom lights, master bathroom lights, and kitchen lights, whereas a user command “turn on the lights” at 5:30 p.m. is associated with a response activity pattern of turning on the kitchen lights, living room lights, and dining room lights), location (e.g., a user command “play music” when at work is associated with a response activity pattern of playing classical music, whereas a user command “play music” when at home is associated with a response activity pattern of playing family music), content (e.g., a user command “it’s a little cold in here” is associated with a response activity pattern of turning up the smart thermostat by two degrees, whereas a user command “I’m cold” is associated with a response activity pattern of turning up the smart thermostat by four degrees), or other context, as described herein.

[0029] In some embodiments, the response activity event can be associated with a response profile corresponding to the user command or used to generate a response profile, and is associated with the task based on the history of the sensed user activity. In this way, when an indication of a user command is received in the future, the (one or more) response activity events associated with the task can be determined and used to cause an operation to be performed to implement the task. These (one or more) response activity events can be associated with the task. Based on the response activity information, a computer system can learn response activity events (based on the user command) that correspond to the user command and are associated with the user activity of the (one or more) user devices involved in performing the task.

[0030] More specifically, based on the determined pattern of response activity events, (one or more) response profiles regarding expected interactions associated with one or more user devices corresponding to a task in response to a user command (e.g., (one or more) response activity events) can be generated and used to provide an improved user experience. In some embodiments, the response activity pattern or the response profile generated therefrom can be available to one or more applications and services that consume the information to provide an improved user experience for interacting with a computing device or a digital assistant application operating thereon (e.g., understanding previously unknown user commands and causing operations to be performed to implement tasks associated with the user commands). For example, in one embodiment, the response activity pattern information and / or the generated response profile can be provided via an application programming interface (API) so that third-party applications and services can use it, such as by determining and / or providing device mappings, recommendations, or other information or services related to the user based on the learned response activity pattern.

[0031] Thus, in one exemplary aspect, user data is received from one or more data sources. The user data may be received by collecting the user data using one or more sensors or components on (one or more) user devices associated with the user. Figure 2 Examples of user data described by the user data collection component 210 may include information about (one or more) user devices and response activity information associated with the user devices (e.g., user interactions with (one or more) devices, app usage, online activities, searches, calls, usage duration, and other user interaction data). The received user data may be monitored, analyzed (e.g., feature extraction) and information about (one or more) response activity events may be stored (e.g., in a user profile, such as Figure 2 user profile 240) to facilitate pattern analysis.

[0032] Specifically, user data can be analyzed to detect various features associated with (one or more) response activity events. The detected user actions or (one or more) response activity events may include actions such as direct interaction with one or more user devices, application launches, or other actions similar to those described herein, which can be recorded together with associated contextual data, for example by utilizing corresponding timestamps, location stamps, and / or associating (one or more) response activity events with other available contextual information to record the observed user actions. In some embodiments, such recording can be performed on each user device so that response activity patterns can be determined across devices. In addition, in some embodiments, cloud-based user activity information sources can be used, such as online user calendars or user activities determined from social media posts, emails, etc. These sources can also be used to provide other context to (one or more) response activity events detected on (one or more) user devices. Examples of contextual information are combined in Figure 2 Context information determiner 284 in is further described. In some embodiments, user activity logs from multiple user devices and available user activity information from cloud-based sources can be combined to represent (one or more) composite response activity event histories.

[0033] Based on the activity log or user activity data, (one or more) historical response activity events can be determined and provided to a response activity pattern engine. Based on the analysis of (one or more) historical response activity events, and in some cases, based on current sensor data about (one or more) response activity events, a set of one or more possible response activity patterns can be determined. Specifically, in some embodiments, the response activity pattern engine can analyze (one or more) historical response activity event information to identify response activity patterns, which can be determined by detecting repetitions of similar user actions or routines.

[0034] In some embodiments, a corresponding confidence weight or confidence score may be determined with respect to the response activity pattern. The confidence score may be based on the strength of the pattern, which may be determined by the number of observations used to determine the pattern, the frequency of (one or more) response activity events being consistent with the pattern, the age or freshness of the activity observations, the number of features common to (one or more) response activity event observations constituting the pattern, or similar measurements. In some cases, the confidence score may be considered when providing a personalized user experience or other improved user experience. In addition, in some embodiments, a minimum confidence score may be required before using the response activity pattern to provide such an experience or other service. For example, in one embodiment, a threshold of 0.6 (or just over fifty percent) is used, so that only response activity patterns with a 0.6 (or greater) probability of performing the desired task corresponding to the user command are used to initiate response activity events that perform the task. However, in the case of using confidence scores and thresholds, the determined response activity patterns of (one or more) response activity events with confidence scores less than the threshold can still be monitored, because additional observations of (one or more) response activity events can increase confidence for a specific pattern.

[0035] In some embodiments, crowdsourced response activity event history can also be used in combination with the user's own (one or more) response activity event history. For example, for a given user, a collection of other users similar to the given user can be identified based on having common features or characteristics with the given user. This may include other users located near the given user, social media friends of the given user, ownership of similar user devices (which can be determined based on analysis of contextual information associated with the given user), other users with similar response activity events, and the like. Information about the response activity event history from other users can be relied on to infer the pattern of (one or more) response activity events for the given user. This may be particularly useful in the case where there is almost no (one or more) response activity event history for the given user, such as when the user is a new user. In some embodiments, user activity information from similar users can be attributed to the new user until there are enough (one or more) response activity event histories available for the new user to determine a statistically reliable response activity pattern, which can be determined based on the number of observations contained in the (one or more) response action event history information or the statistical confidence of the determined response activity pattern, as further described herein. In some cases, where the response activity event history(s) are from other users, a lower confidence may be assigned to the resulting inferred response pattern for the given user.

[0036] In some embodiments, semantic analysis is performed on the (one or more) response activity events and associated information to characterize various aspects of the (one or more) response activity events. For example, features associated with response activity events can be classified (such as by type, similar time range or location), and related features can be identified for determining similarity or relational proximity with other response activity events, such as having one or more common characteristics, including categories and related features. In some embodiments, semantic knowledge representations can be used, such as relational knowledge graphs. In some embodiments, the semantic analysis can use rules, logic such as associations or conditions, or classifiers.

[0037] The semantic analysis may also be used to characterize information associated with the responsive activity event, such as determining that the location associated with the user activity corresponds to a center or place of interest to the user, such as the user's home, office, gym, based on visit frequency. (For example, the user's home center may be determined to be the location where the user spends most of his time between 8pm and 6am.) Similarly, the semantic analysis may determine the time of day corresponding to work hours, lunch, commuting time.

[0038] In this way, the semantic analysis can provide other relevant features of the response activity event that can be used to determine the response activity pattern. For example, in addition to determining that the user wants to activate a specific device using a user command such as "play music on the garage speaker", it is also possible to determine the music category associated with the task of playing music on the garage speaker (e.g., using the response activity mode), such as EDM music. In an embodiment, different categories can have additional response activity events (e.g., bass boost for EDM music). Similarly, the semantic analysis can classify the response activity event as being associated with work or home based on the characteristics of the activity (e.g., playing classical music in response to the user command "play music" at a location corresponding to the user's office can be determined as a work-related activity, while playing family music in response to the user command "play music" at a location corresponding to the user's home can be determined as a family-related activity). These aspects that characterize the response activity event can then be considered when evaluating (one or more) response activity events to identify the response activity pattern. For example, a pattern of turning up a smart thermostat by two degrees can be determined, where the user typically turns the thermostat up by two degrees (e.g., triggering a fault state based on a user command such as "it's cold"). Such a response activity event pattern can be associated with a user command - "it's cold" - that triggers a fault condition.

[0039] As previously described, the response activity mode can be used to generate a response profile corresponding to (one or more) user commands. Based on these response profiles, various implementations can provide an improved user experience. For example, some embodiments may use a response activity mode associated with a user's computing device (in response to a received user command) (e.g., by executing a response activity event) to initiate a task. Some embodiments may be performed by a personal assistant application or service, which may be implemented as one or more computer applications, services, or routines, such as an app running on a mobile device or the cloud, as further described herein.

[0040] Now turn to Figure 1 , a block diagram showing an exemplary operating environment 100 in which some embodiments of the present disclosure can be adopted is provided. It should be understood that this and other arrangements described in this article are only proposed as examples. In addition to those shown or in place of those shown, other arrangements and elements (e.g., machines, interfaces, functions, orders, and functional groups) can be used, and for the sake of clarity, some elements can be omitted together. In addition, many elements described in this article are functional entities, which can be implemented as discrete or distributed components or combined with other components, and are implemented in any suitable combination and position. The various functions described in this article as being performed by one or more entities can be performed by hardware, firmware and / or software. For example, some functions can be performed by a processor that performs instructions stored in a memory.

[0041] Among other components not shown, the exemplary operating environment 100 includes: a plurality of user devices, such as user devices 102a and 102b to 102n; a plurality of data sources, such as data sources 104a and 104b to 104n; a server 106; and a network 110. It should be understood that in Figure 1 The environment 100 shown in FIG. 1 is an example of a suitable operating environment. Figure 1 Each of the components shown in the figure may be implemented via any type of computing device, such as a combination of Figure 6 The computing device 600 described. These components can communicate with each other via a network 110, which can include, but is not limited to, one or more local area networks (LANs) and / or wide area networks (WANs). In an exemplary implementation, the network 110 includes the Internet and / or a cellular network, as well as any of a variety of possible public and / or private networks.

[0042] It should be understood that within the scope of the present disclosure, any number of user devices, servers, and data sources may be employed within the operating environment 100. Each may include a single device or multiple devices cooperating in a distributed environment. For example, the server 106 may be provided via multiple devices arranged in a distributed environment, which collectively provide the functionality described herein. In addition, other components not shown may also be included in the distributed environment.

[0043] The user devices 102a and 102b to 102n can be client devices on the client side of the operating environment 100, while the server 106 can be on the server side of the operating environment 100. The server 106 can include server-side software designed to work in conjunction with the client-side software on the user devices 102a and 102b to 102n to implement any combination of features and functions discussed in this disclosure. This division of the operating environment 100 is provided to illustrate one example of a suitable environment, and it is not required for every implementation that the server 106 and any combination of the user devices 102a and 102b to 102n remain as separate entities.

[0044] User devices 102a and 102b to 102n may include any type of computing device that a user can use. For example, in one embodiment, user devices 102a to 102n may be Figure 6 Types of computing devices described. By way of example and not limitation, a user device may be embodied as a personal computer (PC), a laptop computer, a mobile or mobile device, a smartphone, a tablet computer, a smart watch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a global positioning system (GPS) or device, a video player, a handheld communication device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a camera, a remote control, a barcode scanner, a computerized measurement device, an appliance, a consumer electronic device, a workstation, a smart speaker, an intelligent electronic device or device, or any combination of these depicted devices, or any other suitable device.

[0045] Data sources 104a and 104b to 104n may include data sources and / or data systems configured to make data available to operating environment 100 or in conjunction with Figure 2 Any of the various components of the described system 200. (For example, in one embodiment, one or more data sources 104a to 104n provide Figure 2The user data collection component 210 provides (or makes available for access to) user data.) The data sources 104a and 104b to 104n may be separate from the user devices 102a and 102b to 102n and the server 106, or may be incorporated and / or integrated into at least one of these components. In one embodiment, one or more data sources 104a to 104n include one or more sensors, which may be integrated into or associated with one or more user devices 102a, 102b or 102n or the server 106. Figure 2 The user data collection component 210 further describes examples of sensed user data available from the data sources 104a to 104n.

[0046] Operating environment 100 can be used to implement Figure 2 One or more components of the system 200 described in , including components for collecting user data, monitoring user activity events, determining state failures, monitoring response activity events, determining response activity patterns, generating (one or more) response personalized and / or generic profiles to provide an improved user experience, and / or initiating response activities triggered by identified commands. Now in conjunction with Figure 1 refer to Figure 2 , a block diagram illustrating aspects of an exemplary computing system architecture suitable for implementing embodiments and generally designated as system 200 is provided. System 200 represents only one example of a suitable computing system architecture. Other arrangements and elements can be used in addition to or in place of those shown, and some elements may be omitted altogether for clarity. Furthermore, as with operating environment 100, many of the elements described herein are functional entities that may be implemented as separate or distributed components or in conjunction with other components, and in any suitable combination and location.

[0047] The exemplary system 200 includes a combination of Figure 1 The network 110 described herein and which communicatively couples components of the system 200, including a user data collection component 210, a presentation component 220, a user activity monitor 280, an activity pattern interface engine 260, a fault condition detector 270, and a storage device 225. The user activity monitor 280 (including its components 282, 284, and 286), the response activity pattern engine 260 (including its components 262, 264, and 266), the user data collection component 210, the presentation component 220, the fault condition detector 270, the response profile generator (s) 290, and the response activity initiator 299 may be embodied as a set of compiled computer instructions or functions, program modules, computer software services, or in one or more computer systems (such as in conjunction with Figure 6 The process arrangement is executed on the described computing device 600).

[0048] In one embodiment, the functions performed by the components of system 200 are associated with one or more personal assistant applications, services, or routines. Specifically, such applications, services, or routines can be run on one or more user devices (such as user device 104a), servers (such as server 106), can be distributed on one or more user devices and servers, or implemented in the cloud. In addition, in some embodiments, these components of system 200 can be distributed across a network including one or more servers (such as server 106) and client devices (such as user device 102a) in the cloud, or can reside on a user device such as user device 102a. In addition, these components, the functions performed by these components, or the services performed by these components can be implemented at (one or more) appropriate abstract layers (such as operating system layer, application layer, hardware layer) of (one or more) computing systems. Alternatively or additionally, these components and / or the functions of the embodiments described herein can be performed at least in part by one or more hardware logic components. For example, but not limited to, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), etc. Additionally, while functionality is described herein with respect to specific components shown in the exemplary system 200, it is contemplated that in some embodiments the functionality of these components can be shared or distributed across other components.

[0049] continue Figure 2 , the user data collection component 210 is generally responsible for collecting data from one or more data sources (such as Figure 1The user data collection component 210 may be used to facilitate the accumulation of user data for a particular user (or, in some cases, multiple users of crowdsourced data) for a user activity monitor 280, a response activity pattern engine 260, a response profile generator (s) 290, a fault state detector 270, or a response activity initiator 299. The data may be received (or accessed) by the user data collection component 210, and optionally accumulated, reformatted, and / or combined, and stored in one or more data stores (such as storage device 225), where it is available to other components of the system 200. For example, the user data may be stored in or associated with the user profile 240, as described herein. In some embodiments, any personally identifying data (i.e., user data that specifically identifies a particular user) is not uploaded from one or more data sources with the user data, or otherwise provided, is not permanently stored, and / or is not available to components or subcomponents of the system 200. In some embodiments, users may opt in or out of services provided by the techniques described herein, and / or select which user data and / or which sources of user data are utilized by these techniques.

[0050] User data may be received from a variety of sources, where the data may be available in a variety of formats. For example, in some embodiments, user data received via the user data collection component 210 may be determined via one or more sensors, which may be on or associated with one or more user devices (such as user device 102a), servers (such as server 106), and / or other computing devices. As used herein, a sensor may include functionality, routines, components, or combinations thereof for sensing, detecting, or otherwise obtaining information such as user data from a data source 104a, and may be embodied as hardware, software, or both. By way of example and not limitation, user data may include data sensed or determined from one or more sensors (referred to herein as sensor data), such as location information of mobile device(s), attributes or characteristics of user data(s) (such as device status, charging data, date / time, or other information derived from a user device such as a mobile device), user activity information (e.g., app usage; online activity; searches; voice data, such as automatic voice recognition; activity logs; communication data, including calls, text messages, instant messages, and emails; website posts; other user data associated with communication events), including in some embodiments user activity occurring on more than one user device, user history, session logs, application data, contact data, calendar and schedule data, notification data, social network data, news (including popular or trending items on search engines or social networks), online gaming data, e-commerce activity (including from websites such as Video streaming services, gaming services, or Xbox the user's online accounts), (one or more) user account data (which may include data from user preferences or settings associated with the personal assistant application or service), home sensor data, appliance data, global positioning system (GPS) data, vehicle signal data, traffic data, weather data (including forecasts), wearable device data, other user device data (which may include device settings, profiles, network-related information (e.g., network name or ID, domain information, workgroup information, connection data, Wi-Fi network data or configuration data, data about the model, firmware or device, device pairing, such as when a user pairs a mobile phone with a Bluetooth headset, or other network-related information), gyroscope data, accelerometer data, payment or credit card usage data (which may include data from a user's PayPal account), purchase history data (such as data from a user's Xbox Live, Amazon.com, or eBay account), other sensor data that can be sensed or otherwise detected by (one or more) sensor (or other detector) components, including data derived from sensor components associated with the user (including location, motion, orientation, position, user access, user activity, network access, user device charging, or other data that can be provided by one or more sensor components), data derived based on other data (for example, location data that can be derived from Wi-Fi, cellular network, or IP address data), and virtually any other data source that can be sensed or determined as described herein.

[0051] The user data can be received by the user data collection component 210 from one or more sensors and / or a computing device associated with the user. Although it is contemplated that the user data is processed, for example, by a sensor or other components not shown, for the interpretability of the user data collection component 210, the embodiments described herein do not limit the user data to processed data and may include raw data. In some embodiments, the user data collection component 210 or other components of the system 200 can determine explanatory data from the received user data. The explanatory data corresponds to data used by the components of the system 200 to interpret the user data. For example, the explanatory data can be used to provide context to the user data, which can support determinations or inferences made by components or subcomponents of the system 200, such as venue information from a location, a text corpus from a user's voice (i.e., speech to text), or various aspects of spoken language understanding. In addition, it is contemplated that for some embodiments, components or subcomponents of the system 200 can use user data and / or a combination of user data and explanatory data for executing the goals of the subcomponents described herein.

[0052] In some aspects, user data may be provided in a user data stream or signal. A "user signal" can be a feed or stream of user data from a corresponding data source. For example, a user signal may come from a smartphone, a home sensor device, a smart speaker, a GPS device (e.g., for location coordinates), a vehicle sensor device, a wearable device, a user device, a gyroscope sensor, an accelerometer sensor, a calendar service, an email account, a credit card account, or other data source. In some embodiments, the user data collection component 210 receives or accesses user-related data continuously, periodically, when it becomes available, or as needed.

[0053] continue Figure 2 , the exemplary system 200 includes a fault state detector 270. In an embodiment, the fault state detector 270 includes an application or service that processes collected user data (e.g., user commands) in conjunction with device responses (or lack of responses) to identify faults in the device. User commands may be received as user data collected by the user data collection component 210. In an embodiment, such user commands may include verbal commands, gestures, interactions with the device (via a screen), or any other way of inputting commands. A fault mode or fault state may be considered to have occurred when the user device receives an indication of a user command and determines that the command is unknown or unrecognized. For example, the command itself may be unknown, or the task associated with the command may be unknown. For example, in an embodiment, a computer program or service may compare the received user command with a set of known (e.g., predetermined) commands or known (e.g., predetermined) response actions (those operations specified to be performed in response to a command). The set of known commands and / or response actions may be stored as a list, library, index, and / or in a database, such as the response action 235 stored in the storage device 225.

[0054] In some embodiments, the fault state detector 270 (or another component of the computing system) can use a library or set of known commands and / or responses to determine whether the received command is known or unknown. In some embodiments, when the user command is not related to the task (e.g., as determined based on the user command associated with the task using a library of response operations known to the device), the user command can be determined to be unknown. For example, the user command "It's cold here" may not be related to the task of turning up the temperature using a smart thermostat. In some cases, determining whether the user command is known or unknown may involve performing a similarity comparison. In these embodiments, a similarity comparison can be performed on various aspects of the received user command to determine the similarity with various aspects of other known user commands or response actions, such as information stored in the storage device 225. For example, using feature similarity techniques, such as clustering or Word2Vec, a measure of similarity, such as a similarity score, can be determined. In some embodiments, when the similarity score is high or meets a predetermined threshold, it can be determined that the command is similar to a known command, and an operation corresponding to the known command can be performed. However, where the similarity score does not satisfy the predetermined threshold, it may be determined that the command is not similar to the set of known commands, which may trigger a fault state during which user activity may be monitored.

[0055] In some embodiments, the fault state detector 270 can initiate an operation monitoring mode when it is determined that the received user command is unknown. In some cases, the operation monitoring mode can be responsive to a fault state initiated in the device. By way of example and not limitation, a user can explicitly call a device and provide a user command that is not recognized by the device. In some embodiments, a trigger name (e.g., "Ok Google", "Alexa", "Hey, Cortana") can be used to explicitly call the device. In other embodiments, the device can be explicitly called based on a user command (e.g., "Turn on the TV", "It's cold here", "Play my songs"). When an unknown command is detected, as determined by the fault state detector 270, an enhanced mode of monitoring user activity can be initiated.

[0056] In some implementations, the user can explicitly enter the monitoring mode (e.g., by explicitly triggering the fault state). For example, this is an operating mode in which the user can teach the device specific operations in response to a specific command. For example, the user can initiate the monitoring mode to teach the computing device how the user wants the device to perform tasks associated with a specific user command. In another instance, the user can initiate the monitoring mode to correct (or improve) the situation in which the computing device does respond to the user command but the response is incorrect or generally not what the user expects (such as based on the user's preferences). For example, the user can provide feedback to the system, such as saying "No, that's wrong" or "This is not what I want", or can select an option in an application, or provide other feedback indicating the operation performed by the computing device in response to the user command being incorrect. Such feedback can initiate the monitoring mode, thereby enabling the system to learn (one or more) new responses and / or check whether there is another response profile available (e.g., at a server location with a published response profile) to respond to the user command. In such a scenario, the user may want the computing device to perform a different response activity event to perform a task associated with the command instead of the current response activity event. In an embodiment, setting a new response activity pattern can replace an old response activity pattern (e.g., temporarily or permanently replacing the response activity pattern). In yet another example, the computing device may first learn an incorrect response activity for the command (e.g., turning on the lights and turning up the heat based on a user command “I’m cold” when the user actually just wanted the computing device to turn up the heat). In this way, the system can allow the user to override a previously learned or predetermined action performed by the computing device in order to modify (e.g., correct or improve) how the computing device responds to user commands.

[0057] The user activity monitor 280 is generally responsible for monitoring user data to obtain information that can be used to determine user activity information, which may include identifying and / or tracking features (sometimes referred to herein as "variables") or other information about specific user actions and related contextual information. Embodiments of the user activity monitor 280 may determine user activities associated with a specific user based on the monitored user data. As previously described, the user activity information determined by the user activity monitor 280 may include user activity information from multiple user devices associated with the user and / or from cloud-based services (such as email, calendar, social media, or similar information sources) associated with the user, and may include contextual information associated with the identified user activities. The user activity monitor 280 may determine current or near real-time user activity information, and may also determine historical user activity information, which in some embodiments may be determined based on observations of user activities collected over time, access to user logs of past activities (e.g., such as browsing history). In addition, in some embodiments, the user activity monitor 280 may determine user activities (which may include historical activities) from other similar users (i.e., crowdsourcing), as previously described.

[0058] In some embodiments, the user activity monitor 280 is responsible for monitoring user activity information when the fault state detector 270 detects a fault mode. In such an embodiment, the user activity information can be monitored as (one or more) response activity events. Such (one or more) response activity events can be activities (e.g., activities or collections of activities) performed by the user when the device fails to implement a command (e.g., detect / enter a fault mode). The information determined by the user activity monitor 280 includes information about (one or more) current response activity events, (one or more) response activity events within a defined time range, and / or (one or more) historical response activity events, which can be provided to the response activity mode engine 260. As previously described, (one or more) response activity events can be determined by monitoring user data received from the user data collection component 210. In some embodiments, the user data, information about the user activity, and / or (one or more) response activity events determined based on the user data are stored in a user profile, such as stored in the user profile 240.

[0059] For example, when a fault condition triggers enhanced user monitoring, the user activity monitor 280 can closely monitor user activity, e.g., what activity and / or set of activities the user performs after a user command fails. Specifically, when the user command "it's cold in here" fails, the user activity monitor 280 can detect that the user increases the temperature on the smart thermostat (e.g., by two degrees). The user activity monitor 280 can also detect that the user turns down the volume on the TV and turns on the hallway lights.

[0060] In an embodiment, the user activity monitor 280 includes one or more applications or services that analyze information detected via one or more user devices used by a user and / or a cloud-based service associated with the user to determine activity information and related context information. Information about a user device associated with a user can be determined from user data available via the user data collection component 210 and can be provided to the user activity monitor 280, the response activity pattern engine 260, or other components of the system 200.

[0061] More specifically, in some implementations of the user activity monitor 280, the user device may be identified by detecting and analyzing characteristics of the user device, such as device hardware, software such as an operating system (OS), network-related characteristics, user accounts accessed via the device, and the like. For example, information about the user device may be determined using functions of a number of operating systems to provide information about hardware, OS version, network connection information, installed applications, and the like.

[0062] Some embodiments of the user activity monitor 280 can determine one or more user devices and user device related activities associated with a specific user based on the user data (which may include context information associated with the identified (one or more) user devices). In an embodiment, the user activity monitor 280 includes one or more applications or services that analyze the user devices used by the user to determine information about the device and device use. In some embodiments, the user activity monitor 280 monitors the user data associated with the user device and other relevant information on the user device across multiple computing devices or in the cloud. Information about the user device of the user can be determined from the user data available via the user data collection component 210, and a response activity mode engine 260 or (one or more) response profile generator 290, as well as other components of the system 200, can be provided. In some implementations of the user activity monitor 280, the user device can be identified by detecting and analyzing user devices such as device hardware, characteristics of software such as OS, network-related characteristics, user accounts accessed via the device, and similar characteristics. For example, the functions of many operating systems can be used to determine information about the user device to provide information about hardware, OS version, network connection information, installed applications, etc. Similarly, some embodiments of the user activity monitor 280 or subcomponents thereof may determine a device name or identification (device ID) for each device associated with a user.

[0063] This information about the identified (one or more) user devices associated with the user may be stored in a user profile associated with the user, such as stored in the (one or more) user accounts and (one or more) devices 244 of the user profile 240. In an embodiment, the user device may be polled, queried, or otherwise analyzed to determine information about the device. This information may be used to determine a label or identification (e.g., a device ID) of the device so that user interactions with the device may be identified from user data by the user activity monitor 280. In some embodiments, a user may declare or register a device, such as by logging into an account via the device, installing an application on the device, connecting to an online service that queries the device, or otherwise providing information about the device to an application or service. In some embodiments, logging into an account associated with the user (such as a An account or Net Passport, email account, social network, etc.) is identified and determined to be associated with the user.

[0064] As shown in exemplary system 200, user activity monitor 280 may include user activity detector 282, context information determiner 284, and activity feature determiner 286. In some embodiments, user activity monitor 280, one or more of its subcomponents, or other components of system 200, such as fault state detector 270, response activity mode engine 260, or (one or more) response profile generator 290, may determine explanatory data based on received user data. Explanatory data can correspond to data used by these components of system 200 or subcomponents of user activity monitor 280 to interpret user data. For example, explanatory data can be used to provide other contexts for user data, which can support determination or inference made by a component or subcomponent. In addition, it is contemplated that embodiments of user activity monitor 280, its subcomponents, and other components of system 200 may use user data and / or a combination of user data and explanatory data to execute the goals of the subcomponents described herein. Additionally, although several examples of how user activity monitor 280 and its subcomponents might identify user activity information are described herein, many variations of user activity identification and user activity monitoring are possible in various embodiments of the present disclosure.

[0065] The user activity detector 282 is generally responsible for determining (or identifying) that a user action or user activity event has occurred. Embodiments of the activity detector 282 may be used to determine (one or more) current response activity events and / or one or more historical response activity events. Some embodiments of the activity detector 282 may monitor user data for activity-related features or variables corresponding to (one or more) response activity events, such as indications of applications launched or accessed, files accessed, modified, copied, websites navigated to, online content downloaded and presented or played, devices interacted with, or similar user activities.

[0066] In addition, some embodiments of the user activity detector 282 extract information about (one or more) response activity events from the user data, which may include (one or more) current response activity events, (one or more) historical response activity events, and / or related information (such as context information). (Alternatively or additionally, in some embodiments, the context information determiner 284 determines and extracts the context information. Similarly, in some embodiments, based on the identification of the response activity event determined by the user activity detector 282, the activity characteristic determiner 286 extracts information about (one or more) response activity events, such as user activity related characteristics.) Examples of the extracted user activity information may include app usage, online activity, searches, calls, usage duration, application data (e.g., emails, messages, posts, user status, notifications), or virtually any other data related to user interactions with the user device or user activities conducted via the user device. Among other components of the system 200, the extracted user activity information determined by the user activity detector 282 may be provided to the user activity monitor 280, the response activity pattern engine 260, or other subcomponents of the response profile generator (one or more) 290. In addition, the extracted user activities may be stored in a user profile associated with the user, such as in the user activity information component 242 of the user profile 240. In some embodiments, the user activity detector 282 or the user activity monitor 280 (or other subcomponents thereof) performs merging of the detected (one or more) responsive activity events. For example, overlapping information may be merged and duplicate or redundant information may be eliminated.

[0067] In some embodiments, the user activity related features can be analyzed to detect (one or more) response activity events. For example, in some embodiments, the user activity detector 282 uses a user activity event logic unit (which may include rules, conditions, associations, classification models or other criteria) to identify (one or more) response activity events. For example, in one embodiment, the user activity event logic unit may include comparing the user activity criteria with the user data to determine that a response activity event has occurred. The activity event logic unit can take a variety of different forms, depending on the mechanism used to identify the response activity event. For example, the user activity event logic unit can be training data for training a neural network, which is used to evaluate user data to determine when a response activity event occurs. The activity event logic unit may include a fuzzy logic unit, a neural network, a finite state machine, a support vector machine, a logistic regression, a clustering or machine learning technique, a similar statistical classification process, or a combination of these to identify (one or more) response activity events based on user data. For example, the activity event logic unit can specify the type of (one or more) user device interaction information associated with the response activity event, such as launching an app, manual input, or other interactions with the device. In some embodiments, a series or sequence of user device interactions may be mapped to a response activity event such that the response activity event may be detected upon determining that the user data indicates that the series or sequence of user interactions has been performed by the user.

[0068] In some embodiments, the activity event logic unit can specify types of user device-related activities that are considered to be responsive to activity events, such as activities that occur when a user is logged into the user device and a user interface is receiving input (e.g., when a computer mouse, touchpad, screen, voice recognition interface, etc. is active), or specific types of activities such as launching an application, modifying a file using an application, opening a browser, and navigating to a website. In this way, the activity event logic unit can be used to distinguish true user activity from automated activities of processes running on the user device, such as automatic updates or malware scanning. Once user activity is determined, these features or additional related features can be detected and associated with detected responsive activity events for determining activity patterns.

[0069] In some embodiments, the user activity detector 282 runs on or in association with each user device for a user. The user activity detector 282 may include functionality to poll or analyze various aspects of the operating system to determine characteristics associated with user activity (e.g., such as installed or running applications or file access and modification), network communications, and / or other user actions detectable via a user device including a sequence of actions.

[0070] The context information determiner 284 is generally responsible for determining context information related to the response activity event (detected by the user activity detector 282 or the user activity monitor 280), such as context features or variables associated with the response activity event, related information, and user-related activities, and is further responsible for associating the determined context information with the detected user activity. In some embodiments, the context information determiner 284 can associate the determined context information with the response activity event and can also record the context information with the associated response activity event. Alternatively, the association or recording can be performed by another service. For example, some embodiments of the context information determiner 284 provide the determined context information to the activity feature determiner 286, which determines the activity features and / or related context information of the response activity event.

[0071] Some embodiments of the context information determiner 284 determine context information related to a user action or response activity event, such as an entity identified in the response activity event or associated with the event (e.g., the temperature increased by two degrees). In addition, the user activity can be associated with the location or venue of the user device. By way of example and not limitation, this can include context features such as location data, which can be represented as a location stamp associated with the response activity event; context information about the location, such as venue information (e.g., is this the user's office location, home location, school, restaurant, mobile theater); Yellow Pages Identifier (YPID) information, time, period and / or date, which can be represented as a timestamp associated with the response activity event; user device characteristics or user device identification information about the device on which the user performs the response activity event; the duration of the user response activity event, (one or more) other user response activity events before and / or after the response activity event ( Such as a sequence of device interactions, a sequence of application usage, such as interactions with one or more devices, a sequence of website usage, such as entering a specific search query); other information about the response activity event, such as an entity associated with the response activity event (e.g., a place, a person, an object); information detected by (one or more) sensors on a user device associated with the user that is concurrent or substantially concurrent with the response activity event (e.g., motion information or physiological information detected on a fitness tracking user device, listening to music detected via a microphone sensor when the music source is not the user device); or any other information related to the detectable response activity event that can be used to determine a pattern of similar events.

[0072] In an embodiment using contextual information related to a user device, the user device can be identified by detecting and analyzing user devices such as device hardware, characteristics of software such as an OS, network-related characteristics, user accounts accessed via the device, and similar characteristics. For example, as previously described, information about the user device can be determined using functions of many operating systems to provide information about hardware, OS versions, network connection information, installed applications, and the like. In some embodiments, a device name or identification (device ID) can be determined for each device associated with a user. This information about the identified user device associated with the user can be stored in a user profile associated with the user, such as in (one or more) user accounts and (one or more) devices 244 of a user profile 240. In an embodiment, the user device can be polled, queried, or otherwise analyzed to determine contextual information about the device. This information can be used to determine a label or identification (e.g., device ID) of the device, so that user activity on one user device can be identified and distinguished from user activity on another user device. In addition, as previously described, in some embodiments, a user can declare or register a user device, such as by logging into an account via the device, installing an application on the device, connecting to an online service that queries the device, or otherwise providing information about the device to an application or service. In some embodiments, a device logged into an account associated with the user (such as a Microsoft account or NetPassport, an email account, a social network, etc.) is identified and determined to be associated with the user.

[0073] In some implementations, the context information determiner 284 may receive user data from the user data collection component 210, parse the data in some cases, and identify and extract context features or variables (which may also be performed by the activity feature determiner 286). Context variables may be stored as a related set of context information associated with the user activity and may be stored in a user profile such as in the user activity information component 242. In some cases, one or more activity mode consumers may use the context information, such as for personalizing content or user experience, such as when, where, or how content is presented. Context information may also be determined based on user data of one or more users, which in some embodiments may be provided by the user data collection component 210 instead of or in addition to user activity information for a particular user.

[0074] The activity characteristics determiner 286 can generally be responsible for determining activity-related features (or variables) associated with the responsive activity event that can be used to identify patterns of similar events. The activity characteristics can be determined based on information about the responsive activity event and / or based on relevant contextual information. In some embodiments, the activity characteristics determiner 286 receives (one or more) responsive activity events or related information from the user activity monitor 280 (or its subcomponents), and analyzes the received information to determine a set of one or more features associated with (one or more) responsive activity events.

[0075] Examples of activity-related features include, but are not limited to: location-related features, such as the location of (one or more) user devices during the response activity event, venue-related information associated with the location, or other location-related information; time-related features, such as (one or more) times of day (or days), days of a week or month, or duration of the response activity event, or related duration information, such as the length of time the user uses an application associated with the response activity event; user device-related features, such as device type (e.g., desktop computer, tablet computer, mobile phone, fitness tracker, heart rate monitor), hardware attributes or profiles, OS or firmware attributes, device ID or model, network-related information (e.g., mac address, network information about the user's device, such as the device name, IP address, domain, workgroup, information about other devices detected on the local network, router information, proxy or VPN information, other network connection information), location / motion / direction related information about the user's device, power information, such as battery power, time of connecting / disconnecting the charger, and user access / touch information; usage-related characteristics, such as accessed (one or more) files, app usage (which may also include application data, in-app usage, and applications running simultaneously), network usage information, accessed or otherwise used (one or more) user accounts (such as (one or more) device accounts, (one or more) OS-level accounts, or (one or more) online / cloud service-related account activities, such as account or Net Passport, (one or more) online storage account, email, calendar or social networking account); content-related characteristics, such as online activities (e.g., searches, websites browsed, purchases, social networking activities, communications sent or received, including social media posts); other characteristics that may be detected at or near the time of the responsive activity event or the responsive activity event; or any other characteristics that can be detected or sensed and used to determine a pattern of (one or more) responsive activity events. Characteristics may also include information about (one or more) users using the device; other information that identifies the user, such as a login password, biometric data, which may be provided by a fitness tracker or biometric scanner; and / or characteristics of (one or more) users using the device, which may help distinguish users on a device shared by more than one user. In some embodiments, the user activity event logic unit (described in conjunction with the user activity detector 282) can be used to identify specific characteristics from the user activity information related to the responsive activity event.

[0076] continue Figure 2In the system 200, the response activity pattern engine 260 is generally responsible for determining the response activity pattern based on the user activity information related to (one or more) response activity events determined from the user activity monitor 280 (for example, after the fault state is detected). In some embodiments, the response activity pattern engine 260 can be run on a server, across multiple devices as a distributed application or in the cloud. At a high level, the response activity pattern engine 260 can receive user activity related information, which can be uploaded from the user activity log of the service associated with the client-side application or the user activity monitor 280. One or more reasoning algorithms can be applied to the user activity related information to determine a set of possible response activity patterns (for example, related to a specific user command). For example, a pattern can be determined based on similar observation instances of user activity or associated context information, which can be referred to as a "common feature" of user activity related information occurring after a detected fault state. The inferred activity pattern information can be provided to (one or more) response profile generators 290 and / or used to generate a pattern-based prediction of a set of operations for performing tasks associated with a specific user command. In some embodiments, a corresponding confidence level is also determined for the response activity pattern, as described herein. In addition, the response activity pattern may include a confidence level of initiating a single action to perform a (future occurring) task and / or initiating one of a series of actions to perform the task, e.g., an eighty percent likelihood that the response activity event will be to increase the temperature on the smart thermostat, a fifteen percent likelihood that the response activity event will be to turn on the hallway lights, and a five percent likelihood that the response activity event will be to mute the volume on the TV, in order to perform the task associated with the user command.

[0077] As shown in the exemplary system 200, the response activity pattern engine 260 may include a semantic information analyzer 262, a feature similarity identifier 264, and a response activity pattern determiner 266. The semantic information analyzer 262 is generally responsible for determining semantic information associated with the response activity-related features identified by the user activity monitor 280. For example, while the response activity features may indicate a specific device with which the user interacted, the semantic analysis may determine how to interact with the device or the characteristics of the interaction. The semantic information analyzer 262 may determine additional response activity-related features that are semantically related to the response activity event, which may be used to identify a response activity pattern.

[0078] Specifically, as previously described, semantic analysis can be performed on the response activity information, which may include the context information, to characterize various aspects of the user action or response activity event. For example, in some embodiments, the activity features associated with the response activity event can be classified or categorized (such as by type, time range or location, work-related, family-related, theme, related entity, (one or more) other users (such as communications from another user) and / or the relationship of other users to the user (e.g., family members, close friends, work acquaintances, bosses, etc.), or other categories), or related features can be identified for determining similarities or relationship proximity with other response user activity events, which can indicate a pattern (e.g., the degree to which the thermostat is turned up). In some embodiments, the semantic information analyzer 262 can utilize semantic knowledge representations, such as relational knowledge graphs. The semantic information analyzer 262 can also utilize semantic analysis logic, including rules, conditions, or associations, to determine semantic information related to the response activity event. For example, a response activity event associated with a user command "It's cold here" that includes interacting with a smart thermostat can be characterized by how many degrees the thermostat increases. Thus, when the smart thermostat increases by two degrees in response to a fault condition triggered by a "it's cold in here" command, it can be determined that the user activity corresponding to the "it's cold in here" command is increasing the smart thermostat by two degrees. Additional semantic information, such as the time of year, can also be used to determine the pattern. For example, in the summer, the user activity corresponding to the "it's cold in here" command is to increase the smart thermostat by two degrees. However, in the winter, the user activity corresponding to the "it's cold in here" command is to increase the smart thermostat by four degrees.

[0079] The semantic information analyzer 262 can also be used to characterize the contextual information associated with the response activity event, such as determining that the location associated with the activity response event corresponds to a center or place of interest to the user (such as the user's home, office, gym, etc.) based on the frequency of user visits. For example, the user's home center can be determined (using a semantic analysis logic unit) to be the location where the user spends most of his time between 8 pm and 6 am. Similarly, the semantic analysis can determine the time of day corresponding to work hours, lunch time, and commuting time. Similarly, the semantic analysis can classify the activity as being associated with work or family based on other characteristics of the activity (for example, playing classical music in response to a user command "play music" at a location corresponding to the user's office can be determined as a work-related activity, while playing family music in response to a user command "play music" at a location corresponding to the user's home can be determined as a family-related activity). In this way, the semantic analysis provided by the semantic information analyzer 262 can provide other relevant features of user activity events that can be used to determine response activity patterns. For example, in a case where the user activity includes playing classical music in response to a command “play music” and the semantic analysis indicates that the user is working, it can be determined that the response activity pattern (by the response activity pattern determiner 266) indicates that in response to the user command “play music” while working, the user wants to play classical music while working.

[0080] The feature similarity identifier 264 is generally responsible for determining the similarity of activity features of two or more response activity events (in other words, the activity features characterizing a first response activity event are similar to the activity features characterizing a second response activity event). The activity features may include features related to contextual information and features determined by the semantic information analyzer 262. Response activity events having common activity features may be used to identify response activity patterns, which may be determined using the response activity pattern determiner 266.

[0081] For example, in some embodiments, the feature similarity identifier 264 can be used to determine a set of responsive activity events having common features. In some embodiments, the set of responsive activity events can be used as an input to a responsive activity pattern determiner 266, as described below. In some embodiments, the feature similarity identifier 264 can include functionality for determining similarities of periodicity-based and behavior-based activity features. Periodic features include, for example, features that may occur periodically; for example, features of a certain day of the week or month, even / odd days (or weeks), monthly, yearly, every other day, every 3 days. For example, behavioral features can include behaviors such as user activities that tend to occur at a particular location or activities that occur before or after a given user activity event (or a sequence of previous activity events).

[0082] In embodiments where activity features have values, similarities between different activity features having the same value or approximately the same value may be determined based on the particular feature. (For example, a timestamp of a first responding activity event occurring at 9:01 AM on Friday and a timestamp of a second responding activity event occurring at 9:07 AM on Friday may be determined to have similar or common timestamp features.)

[0083] The response activity pattern determiner 266 is generally responsible for determining the response activity pattern based on the similarities identified in the response activity events. Specifically, the response activity pattern determiner 266 (or the response activity pattern engine 260) can determine the response activity pattern based on the repetition of similar activity features associated with multiple observed response activity events. Therefore, for example, in the case where the activity features corresponding to two or more response activity events are similar, the response activity pattern can be determined. In some instances, the response activity event may have many corresponding activity features, which can be represented as feature vectors associated with specific response activity events. Therefore, the analysis performed by the response activity pattern determiner 266 can involve comparing the activity features of the feature vectors from multiple response activity events.

[0084] In some embodiments, the response activity pattern can be determined using pattern inference logic 230. The pattern inference logic can include rules, associations, conditions, prediction and / or classification models, or pattern inference algorithms. The pattern inference logic 230 can take many different forms, depending on the specific response activity pattern or the mechanism for identifying the response activity pattern, or identifying the feature similarity between the observed response activity events to determine the pattern corresponding to the task associated with the user command. For example, some embodiments of the pattern inference logic unit 230 can use machine learning mechanisms to determine feature similarity, or other statistical measures to determine that the response activity event belongs to an exemplary set of response actions that support the determined response activity pattern, as further described below. The response activity information can be received from the user activity monitor 280, and information about the identified similar features can be received from the feature similarity identifier 264. In some embodiments, the (one or more) response activity patterns determined by the response activity pattern determiner 266 can be stored in the user profile 240 as a user-response activity pattern 248. In another embodiment, the (one or more) response activity patterns determined by the response activity pattern determiner 266 can be stored in the response profile 250 as a response activity pattern 256.

[0085] In some embodiments, the response activity pattern determiner 266 provides a pattern of user activity (e.g., (one or more) response activity events) and an associated confidence score regarding the strength of the response activity pattern, which can reflect the likelihood that the response activity pattern satisfactorily performs the task associated with a particular user command. More specifically, in some embodiments, a corresponding confidence weight or confidence score can be determined regarding the determined response activity pattern. The confidence score can be based on the strength of the pattern, which can be determined based on the number of observations (of a particular response activity event) used to determine the response activity pattern, the frequency with which user actions are consistent with the response activity pattern, the age or freshness of the response activity event observations, the number of similar features, the type of features, and / or the degree of similarity or similar measurements of features common to the response activity observations that constitute the pattern.

[0086] In some cases, the confidence score may be considered when generating a response profile (e.g., using (one or more) response profile generators 290). For example, in some embodiments, a minimum confidence score may be required before using the response activity pattern to generate a response profile corresponding to a user command. In one embodiment, a threshold of 0.6 (or just over fifty percent) is used so that only response activity patterns that have a 0.6 (or greater) likelihood of performing the desired task by initiating the response activity event (as indicated by the response activity pattern) can be provided. However, where confidence scores and thresholds are used, determined response activity patterns with confidence scores less than the threshold may still be monitored and updated based on additional response activity event observations, as the additional observations may increase confidence in the particular pattern.

[0087] Some embodiments of the response activity pattern determiner 266 determine a pattern according to the exemplary method described below, wherein each instance of a response activity event has a corresponding historical value of a tracked activity feature (variable) that forms a pattern, and wherein the response activity pattern determiner 266 can evaluate the distribution of the tracking variable for the pattern. In the following example, the tracking variable for the response activity event is a timestamp corresponding to the observed instance of the response activity event. However, it will be appreciated that, conceptually, the following can be applied to different types of historical values ​​for tracking activity features (variables).

[0088] Some timestamps (i.e., the value of a given tracking variable) can be represented as and mapped to a two-dimensional histogram of hours and days of the week. The two-dimensional histogram can include a summation of user-device interaction instances, such as:

[0089]

[0090] This histogram can be used to determine the derivative histogram. For example, the histogram for a day of the week may correspond to: h j =∑ i h ij . A histogram for one hour in a day can correspond to: h i =∑ j h ij As another example, one or more histograms may be determined for a particular semantic temporal resolution in the following form: iC =∑ j∈C h ij Any of a variety of semantic temporal resolutions can be employed, such as weekdays and weekends, or morning, afternoon, and evening. An example of the latter is C∈{morning, afternoon, evening}, with morning = {9, 10, 11}, afternoon = {12, 13, 14, 15, 16}, and evening = {21, 22, 23, 24}.

[0091] An additional data structure for representing events can include the number of distinct timestamps with at least one timestamp per calendar week, which can be represented as:

[0092]

[0093] As an example, The number of different timestamps during the second three-week period that can represent available timestamps. N (j) Can be used to denote the number of j-week timestamps available in the trace data; for example, N (3) Indicates the number of three-week time slots available at the stated timestamp.

[0094] The response activity pattern determiner 266 (or the response activity pattern engine 260) can generate a confidence score that quantifies the level of certainty that a particular response activity pattern is formed by historical values ​​in the tracking variables. In the following example, the above principles are applied using Bayesian statistics. In some implementations, confidence scores can be generated for corresponding tracking variables indexed by time intervals of different resolutions. For timestamps, examples include Tuesday morning at 9 o'clock, weekday mornings, and Wednesday afternoon. The confidence score can be calculated by applying a Dirchlet polynomial model and calculating the posterior predictive distribution of each time period histogram. In doing so, the prediction for each bin in a particular histogram can be given by the following formula:

[0095]

[0096] where K represents the number of segments, α0 is a parameter encoding the strength of prior knowledge, and Then, the mode prediction is the segment in the histogram corresponding to i*, and its confidence is given by As an example, consider a histogram where morning = 3, afternoon = 4, evening = 3. Using α0 = 10, the mode prediction is afternoon, and the confidence score is According to various implementations, more observations result in an increased confidence score, which indicates an increased confidence in the prediction. For example, consider a histogram where morning = 3000, afternoon = 4000, and evening = 3000. Using a similar calculation, the confidence score is

[0097] Likewise, in some implementations, confidence scores can be generated for corresponding tracking variables indexed by time period and multiple timestamps. Examples include 1 visit per week and 3 visits per 2 weeks. Using Gaussian posteriors, confidence scores can be generated for patterns at each time period resolution, denoted as j. This can be done by employing the following formula:

[0098] in,

[0099] In the above formula, σ 2 is the sample variance, and and μ0 are parameters of the formula. The confidence score can be calculated by taking a fixed interval around multiple timestamp predictions and calculating the cumulative density:

[0100] in,

[0101] As an example, consider the following observation: and N (1) =4 and N (2) =2. Using μ0 = 1 and μ (1) =4.075, and conf1 = 0.25. In addition, μ (2) =10.31 and conf2 =0.99. In the previous example, although fewer timestamps were available for the two-week period, the reduced variation in the user signal resulted in increased confidence that the pattern existed.

[0102] Having determined that the response activity pattern exists, or that the confidence score for the response activity pattern is sufficiently high (e.g., satisfies a threshold), the response activity pattern determiner 266 may identify that a plurality of response activity events correspond to the response activity pattern. As another example, the response activity pattern determiner 266 may determine that the response activity pattern is likely to perform a desired task in response to a user command, wherein one or more confidence scores for one or more tracking variables satisfy a threshold.

[0103] In some embodiments, the pattern of the response activity or the pattern of the response activity event can be determined by monitoring one or more activity features, as previously described. These monitored activity features can be determined based on the data previously described as tracking variables or in combination with the user data collection component 210. In some cases, the variables can represent the contextual similarity and / or semantic similarity between multiple user actions (response activity events). In this way, the pattern can be identified by detecting the common variables or features of multiple user actions occurring after the failure state detected for the user command. More specifically, the features associated with the first user action after the failure state detected for the user command can be related to the features of the second user action after the failure state detected for the user command to determine the possible pattern. The identified feature pattern may become stronger (i.e., more likely or more predictable), repeating the user activity observation more frequently after the failure state detected for the user command constituting the pattern (or after the user command). Similarly, a specific feature becomes more closely associated with the response activity pattern when it is repeated.

[0104] In some embodiments, such as the exemplary embodiment shown in system 200, (one or more) response profile generators 290 can use the response activity pattern determined by the response activity pattern determiner 266 to generate (one or more) response profiles (e.g., stored as response activity pattern 256 and / or user response activity pattern 248). At a high level, (one or more) response profile generators 290 can associate the response activity pattern with the corresponding user command. Specifically, (one or more) response profile generators 290 can receive user activity information and / or activity characteristics associated with the user activity, and determine the user command to which the user activity occurred in response. In an embodiment, (one or more) response profile generators 290 can include functionality for generating response profiles as further described below, and the response profiles are used to initiate one or more operations in response to a user command (or a similar command) to perform a task (e.g., by executing (one or more) user activity events that include the user command). The response profiles typically include information that associates (one or more) commands with one or more specific operations to be performed to implement the command (or the task associated with the command). In an embodiment, such information may be general and / or user-based. In another embodiment, such information may be specific to a particular user device (e.g., a particular brand of smart thermostat).

[0105] In one embodiment, the response profile generator(s) 290 use features or patterns that are common between historical user activities and recent user activities determined by the response activity pattern determiner. For example, response activity patterns based on similarity of periodic features, from a set of historical user actions, from those historical user actions that have common periodic features with the current or recent user action. In addition, the response profile generator(s) 290 can determine the user command to be performed as a task corresponding to the response activity pattern(s). For example, if the recent user action includes adjusting the thermostat in accordance with a user command ("I'm cold"), then determining behavioral feature similarity will include identifying those historical user actions that have features that indicate that the user adjusted the thermostat in accordance with a user command (same or similar user command - "I'm cold", "It's cold in here", "Turn up the temperature").

[0106] In order to generate and / or update a response profile, (one or more) response profile generators 290 can select, from a subset of historical response action events, the response action events that are most similar (or similar enough) to a particular current or recent user action event, which can be selected to determine a response activity pattern. In some embodiments, the selection process uses a similarity threshold such as that described above to determine those historical response action events that meet the similarity threshold. (Although the term "selection" is used, it is contemplated that the selection is performed by a computer-related process that does not require an individual to perform the selection.) The response activity pattern can be associated with (one or more) specific user commands. For example, the user commands "I'm cold", "It's cold here", "Turn up the temperature" can all be associated with the same task and executed using the same response activity pattern based on the similarity threshold.

[0107] The exemplary system 200 may also include a storage device 225. The storage device 225 generally stores information including data, computer instructions (e.g., software program instructions, routines, or services), logical units, profiles, and / or models used in the embodiments described herein. In an embodiment, the storage device 225 includes a data store (or computer data storage). In addition, although depicted as a single data store component, the storage device 225 may be embodied as one or more data stores or may be in the cloud.

[0108] As shown in exemplary system 200, storage device 225 may include activity pattern inference logic unit 230, as previously described, response actions 235, user profiles 240, and response profiles 250. Figure 2 An exemplary embodiment of a user profile 240 is illustratively provided in . The exemplary user profile 240 includes information associated with a particular user, such as user activity information 242, information about user accounts and devices 244, user preferences 246, and user response activity patterns 248. The information stored in the user profile 240 may be used in the response activity pattern engine 260 or other components of the exemplary system 200.

[0109] As previously described, user activity information 242 generally includes user information about user actions or activity events, relevant contextual information, activity characteristics, or other information determined via user activity monitor 280, and may include historical or current user activity information. User accounts and devices 244 generally include information about user devices accessed, used, or otherwise associated with a user, and / or information relating to user accounts associated with a user, such as online or cloud-based accounts (e.g., email, social media), such as Net Passport, other accounts, such as entertainment or gaming related accounts (e.g., Xbox Live, Netflix, online gaming subscription accounts), user data associated with the account, such as user emails, texts, instant messages, calls, other communications, and other content; social network accounts and data, such as news feeds; online activities; and calendars, appointments, application data, other user accounts, etc. Some embodiments of user accounts and devices 244 can store information across one or more databases, knowledge graphs, or data structures. As previously described, the information stored in user accounts and devices 244 can be determined from the user data collection component 210 or the user activity monitor 280 (including one of its subcomponents).

[0110] The user preferences 246 generally include user settings or preferences associated with user activity monitoring. By way of example and not limitation, such settings may include user preferences regarding specific activities (and related information) that the user wishes to be explicitly monitored or not monitored, or categories of activities to be monitored or not monitored, crowdsourcing preferences, such as whether to use crowdsourced information, or whether the user's activity pattern information can be shared as crowdsourced data; preferences regarding which activity pattern consumers can consume the user's activity pattern information; and thresholds, and / or notification preferences, as described herein. As previously described, the user response activity pattern 248 may include one or more user patterns generated by (one or more) response profile generators 290 based on (one or more) response activity patterns determined by the response activity pattern determiner 266, and may also include confidence scores associated with the patterns and / or information associated with the activity patterns, such as contextual information or semantic information. The user response activity pattern 248 includes personalized user response activity pattern content determined from the user's personal content, such as a response activity pattern unique to the user. For example, the user command "activate my room" corresponds to a personalized user response activity pattern, which includes response activities, including turning on the lights in the office, raising the temperature in the office by two degrees, and turning on the TV in the office at volume 20.

[0111] As shown in exemplary system 200, storage device 225 includes activity pattern inference logic unit 230, as previously described, response actions 235, user profiles 240, and response profiles 250. Figure 2An exemplary embodiment of a response profile 250 is illustratively provided in . In this example, the response profile 250 includes information associated with responses and / or user commands, such as response commands 252, device mappings 254, response activity patterns 256, and user feedback / weightings 258. In an embodiment, the responses and / or user commands 252 in the response profile 250 can be generic (i.e., can be executed on one or more different devices to perform the same or similar tasks). In another embodiment, the responses and / or user commands in the response profile 250 can be specific to one user or user device (e.g., specific to a user with a specific smart thermostat). In an implementation, the information stored in the response profile 250 can be used in the response activity pattern engine 260 or other components of the exemplary system 200.

[0112] The response profile 250 may generally include one or more user commands and corresponding operations to be performed to implement the tasks associated with the commands. Such corresponding operations can be, for example, response activity events and / or response activity patterns corresponding to user commands. In an implementation, a response profile can contain user commands, (one or more) associated response activity events, and / or (one or more) associated tasks (in any combination). In other embodiments, the response profile may also include user-device mappings that enable the operations to be performed on different types of user devices (e.g., various brands of user devices that perform the same / similar actions - smart thermostats).

[0113] In some embodiments, the response profile can be determined from learned user activities. Such user activities can be learned from multiple users. For example, a response profile (corresponding to a specific user command) can be associated with a response activity event. The response profile can also be associated with a task based on a history of sensed user activities related to the user command (e.g., obtained during an operation monitoring mode). For example, but not limited to, (one or more) response profiles can be generated based on a pattern of determined response activity events associated with one or more user devices corresponding to a task in response to a user command (e.g., (one or more) response activity events), and used to provide an improved user experience. Based on the response activity information, the computer system can learn response activity events corresponding to the user command and associated with the user activity of (one or more) user devices involved in performing the task (according to the user command).

[0114] In an embodiment, when an indication of a user command is received in the future, the response profile 250 can be used to determine (one or more) response activity events associated with the task involving the user command. In such an embodiment, (one or more) response activity events can be used to cause an operation to be performed to implement a task (e.g., using one or more user devices). In an implementation, the response activity pattern or the response profile generated therefrom can be used to consume the information to provide one or more applications and services that interact with a computing device or a digital assistant application running thereon with an improved user experience (e.g., understanding previously unknown user commands and causing the execution of operations to implement tasks associated with the user command). For example, in one embodiment, the response activity pattern information and / or the generated response profile can be provided via an API so that third-party applications and services can use it, such as by determining and / or providing device mappings, recommendations, or other information or services related to the user based on the learned response activity pattern. In some embodiments, the response profile may also include a user-device mapping that specifies how to perform the operation based on different types of user devices, such as smart lights or smart thermostats of different smart brands. For example, a response profile may specify one or more user commands and corresponding operations (for one or more specific user devices) to be performed to implement the tasks associated with the commands.

[0115] As previously described, the response command 252 generally includes (one or more) user commands that may indicate or correspond to a specific task that is desired to be performed by the user issuing the command. In some embodiments, the response command may include a set of user commands with a corresponding response activity mode (or a response action that may be determined based on the response activity mode) for performing operations to implement the task. Thus, the user command may correspond to initiating one or more actions by one or more user devices to perform the task. For example, the command "It's cold in here" may correspond to the task of raising the temperature by a few degrees on a smart thermostat.

[0116] Device mapping 254 typically includes information about user devices that are accessed, used, or otherwise associated with a response, and / or information related to devices associated with the response activity. Some embodiments of device mapping 254 may store information across one or more databases, knowledge graphs, or data structures. As previously described, the information stored in device mapping 254 may be determined from user data collection component 210 or user activity monitor 280 (including one of its subcomponents). Specifically, device mapping 254 can associate response commands with corresponding response activity patterns across various user devices. Device mapping can be provided by a company, a user, and / or some combination of the two.

[0117] In an embodiment, device mapping 254 can be used to teach previously unknown user commands to a particular user device. For example, a first user device has the functionality to understand that a user command to "raise the temperature" results in a task to raise the smart thermostat by two degrees. A second user device that does not inherently have the functionality to understand the user command to "raise the temperature" is able to learn the response activity pattern from the user command and response activity pattern pairing (e.g., from response action 235). In this mode, in an embodiment, even when a particular user device did not previously understand the user command, the particular user device is able to learn from device mapping 254 to perform a task (e.g., via response activity initiator 299).

[0118] In another embodiment, the device mapping 254 can convert the user command to apply the corresponding task to a specific user device. For example, various user devices may implement the response activity mode in different ways to perform the same task. The device mapping 254 can associate a general command (e.g., from a response activity mode) with a specific action (corresponding to the response activity mode) for a specific user device. For example, if the task is to increase the temperature, one type of smart thermostat uses one operation to increase the temperature, while another type of smart thermostat uses a different type of operation to increase the temperature. In an embodiment, the device mapping 254 ensures that a specific user device can understand how to implement the response activity mode on the specific device.

[0119] The response activity pattern 256 generally includes a response activity pattern associated with a user command (e.g., response command 252). Specifically, the response activity pattern stored in the response activity pattern 256 can include both a specific pattern and a general or universal pattern (e.g., a response activity pattern applicable to a group of users). As previously described, the response activity pattern 256 may include one or more response activity patterns determined by the response activity pattern determiner 266, and may also include a confidence score associated with the pattern and / or information related to the activity pattern, such as contextual information or semantic information. The response activity pattern may specify a set of operations to be performed in response to a user command, as previously described, and may be specified in the response command 252. Specifically, as described herein, the operation set includes operations "learned" from the monitored user activities that determine the response pattern. In some embodiments, the operation set may include only a single operation, and in other embodiments, the operation set may include a series or sequence of operations to be performed.

[0120] In some embodiments, the set of operations may also include prompting the user for input regarding specific variables or options associated with the operation. For example, for a user command "book a restaurant," if the user has not provided a specific restaurant, cuisine, or similar information, when performing an operation learned from the user activity response to the command, the system may ask the user, for example, "Is there a specific restaurant you would like to book?" Or similarly, "Do you have a particular cuisine that you prefer when booking dinner?" Or further, "Do you have a favorite dinner reservation time?" In some embodiments, these prompts may also be learned, such as where the response activity pattern shows a strong pattern for a specific operation feature type or category (e.g., launching a specific app or contacting a specific website, such as the OpenTable application, which is used to book a restaurant), but the feature values ​​(i.e., parameter values) for that feature type are variable (e.g., have high variance or divergence). In other words, the response activity pattern shows that specific operations are always performed (e.g., launching a restaurant reservation application), but the settings or values ​​associated with those operations will vary (e.g., the specific restaurant selected in the application will vary). Therefore, in the case where various aspects of the operation to be performed show differences or divergences between user response patterns, the user may be prompted for input regarding the operation.

[0121] In some embodiments, weighting can be applied to response activity patterns. For example, user feedback can indicate that a particular response activity pattern (or more generally, a particular response profile 250) is good (e.g., correct) or bad (e.g., incorrect, incomplete, or problematic). The system can use this feedback (and / or weighting) to determine whether it should continue to learn a response action for a particular command, or whether it has fully learned the response to the command. (For example, if the user feedback is positive, it can be inferred that the system's response to the user command satisfies the user.) In addition, feedback or weighting can be used to rank available response profiles. Specifically, in some embodiments where response profiles are available to other users, it is contemplated that multiple response profiles can be created for the same or similar commands. Therefore, user feedback can be used to score or rank the response profiles, so that those response profiles with higher scores or weights are more likely to be used. Therefore, the response profile 250 can include user feedback / weighting 258, which can represent the weight applied to a particular response profile 250 or response activity pattern 256.

[0122] In some embodiments, a corresponding confidence weight or confidence score may be associated with the response activity pattern. The confidence score may be based on the strength of the response pattern. In some cases, the confidence score may be based on the number of observations (of a particular response activity event) used to determine the response activity pattern, the frequency with which the user's actions are consistent with the response activity pattern, the age or freshness of the user's activity observations, the number of similar features, the type of features, and / or the degree of similarity of features common to the response activity observations that make up the pattern, or similar measurements.

[0123] In some cases, the confidence score may be considered when generating a response profile (e.g., using (one or more) response profile generators 290). For example, in some embodiments, a minimum confidence score may be required before using the response activity pattern to generate a response profile corresponding to a user command. In one embodiment, the number of times a response activity pattern is established can indicate the strength of the response activity pattern. For example, if the response activity pattern corresponding to the user command "I am cold" is that the task of performing the thermostat increase has been determined 10,000 times (e.g., for 10,000 users and / or instances of the user command "I am cold"), the response activity pattern is relatively strong. On the other hand, if the response activity pattern corresponding to the user command "I am cold" is that the task of performing the thermostat increase has been determined twice (e.g., for two users and / or instances of the user command "I am cold"), the response activity pattern is relatively weak. However, even where confidence scores and thresholds are used, determined response activity patterns having confidence scores less than the threshold may still be monitored and updated based on additional response activity event observations, as additional observations may increase confidence in a particular pattern.

[0124] In other cases, direct feedback can be obtained from the user. For example, when a fault state is detected (e.g., determined by the fault state detector 270), the user device can ask the user for clarification. Specifically, in one embodiment, when the user command "I am cold" triggers a fault state of the user device (e.g., failure to turn up the temperature using the smart thermostat), the user device can ask the user for clarification and / or feedback. Clarification and / or feedback can include direct questions to the user, such as: "Do you want me to turn up the temperature? I know you are cold." In some implementations, the user's answer to such a direct question (e.g., yes or no) can be used to weight the response activity mode. For example, if the user answers "yes", then the response activity mode of turning up the temperature can be more heavily weighted to indicate a strong confidence score. If the user answers "no", the response activity mode of turning up the temperature can be more heavily weighted downward to indicate a lower confidence score.

[0125] In some embodiments, direct feedback observed from the user can also be responsive to the action of the user device. For example, if a user command "I'm cold" causes the user device to turn on the hallway light, the user can state "I said I was cold, not turning on the light." Such a statement can be used to determine that the response activity event performed (e.g., turning on the light) was incorrect (the task associated with the user command was not performed correctly). In such a case, the response activity event of turning on the light in response to the user command "I'm cold" can be weighted down.

[0126] In some embodiments, a response profile can be generated by associating a response activity pattern (e.g., from response activity pattern 256) with a user command (e.g., response command 252), which can be stored in response action 235. Response action 235 can be a collection or library of responses associated with user commands (e.g., response activity patterns that perform tasks). In other embodiments, response action 235 can be stored as a list or data set. In this way, by way of example and not limitation, response action 235 can map the association between various user commands and various actions performed by one or more user devices to perform tasks corresponding to the user commands. In an implementation, the initial state, response action 235 can include, for example, an immediately available response profile. In some embodiments, the immediately available response profile can be a preprogrammed response activity. Response action 235 can also generally include a learned response profile (e.g., generated by (one or more) response profile generators 290). A learned profile can be added by monitoring user actions in response to the absence of a user command in response action 235. For example, when there is no user command in the response action 235, a fault state can be triggered (at the fault state detector 270). From the user actions monitored after the fault state is initiated (e.g., using the user activity monitor 280), a response activity pattern can be determined (e.g., using the response activity pattern engine 260). In some of these embodiments, such a response activity pattern can be associated with a task performed by one or more user devices in response to an indication of a received user command to generate a response profile (e.g., using (one or more) response profile generators 290). The response profile can be added to the response action 235. In this way, in an embodiment, when a future indication of a user command corresponding to the response profile is received, the response profile can be used to perform a desired task in response to the user command.

[0127] The response activity initiator 299 typically initiates one or more tasks in response to a user command. The one or more tasks can be based on a response activity pattern (e.g., response activity pattern 256 and / or user response activity pattern 248) associated with the user command (stored in the response action 235). In an embodiment, the response activity initiator 299 can select what response activity (or activities) to initiate based on the confidence strength of the response activity pattern (or patterns) associated with performing a task based on the indication of the user command. For example, a minimum confidence score may be required before the response activity initiator 299 initiates a response activity pattern. For example, in one embodiment, a threshold of 0.6 (or just over fifty percent) is utilized so that only response activity patterns with a probability of 0.6 (or greater) corresponding to the desired task can be considered.

[0128] The exemplary system 200 also includes a presentation component 220, which is generally responsible for presenting content and related information to the user, such as content related to user feedback. The presentation component 220 may include one or more applications or services on a user device, across multiple user devices, or in the cloud. For example, in one embodiment, the presentation component 220 manages presenting content to the user across multiple user devices associated with the user. Based on content logic units, device features, associated logical centers, inferred user logical locations, and / or other user data, the presentation component 220 can determine on which (which) user devices to present content, and the context of the presentation, such as how (or in what format and how much content, which can depend on the user device or context) to present, when to present. Specifically, in some embodiments, the presentation component 220 applies the content logic unit to device features, associated logical centers, inferred logical locations, or sensed user data to determine various aspects of content presentation. For example, clarification and / or feedback requests can be presented to the user via the presentation component 220.

[0129] In some embodiments, presentation component 220 generates user interface features associated with clarification and / or feedback requests. Such features can include interface elements (such as graphical buttons, sliders, menus, audio cues, alerts, warnings, vibrations, pop-up windows, notification bar or status bar items, in-application notifications, or similar features for interfacing with a user), queries, and prompts.

[0130] As previously described, in some embodiments, a personal assistant service or application operating in conjunction with the presentation component 220 determines when and how to present content (e.g., present when it is determined that the user is at a particular logical location). In such embodiments, content comprising a content logical unit can be understood as a recommendation to the presentation component 220 (and / or the personal assistant service or application) as to when and how to present a notification, which can be overridden by the personal assistant application or presentation component 220.

[0131] Now turn to Figure 3B , illustratively provides another exemplary embodiment of the technology described in this article. Figure 3B The exemplary embodiment shown in FIG. Figure 3A Specifically, in scenario 380, the same user 305 is shown interacting with a smart speaker 385 (an example of a user device on which a digital assistant application is running). Figure 3A The opposite is true in the example Figure 3B In the example of FIG. 3 , the smart speaker 385 is able to perform the task indicated by the user command 381 (“Hey Cortana, turn on the lights”). Specifically, in the scenario 380 , the smart speaker 385 responds 399 by letting the user know that it has learned to turn on the lights in the home.

[0132] In this example, following a user command 381, the smart speaker 385 is able to check the user command 381 against a library and / or set of known (predetermined) commands. The smart speaker 385 (or a digital assistant or computer program running thereon) then at least determines a responsive activity event and / or responsive activity pattern associated with the task and / or user command. Here, the responsive activity event and / or responsive activity pattern is determined to be one or more actions of one or more smart lights in the user's home turning on the lights.

[0133] As further described herein, Figure 3BIn some embodiments of the examples depicted in , a user device (e.g., a smart speaker 385) or a digital assistant application running thereon may learn to respond to command 381 “turn on the lights” by monitoring the user activity of user 305. In another embodiment, and as further described herein, the smart speaker 385 (or a digital assistant application running thereon) may learn to respond to command 381 by receiving updated instructions. For example, the smart speaker 385 may access a server of response profiles (or may receive additional response profiles via a device update process), thereby enabling the smart speaker 385 to respond to user commands 381. The response profile may be generated based on monitoring the user activity of other users who make the same or similar commands. In this way, the smart speaker 385 (or a similar user device with a digital assistant application running thereon) may be enabled to correctly respond to a user command on the first instance in which the user provides the command because the device has effectively learned from other users. Although Figure 3B The exemplary embodiment of the present invention is implemented on a smart speaker, but it is contemplated that this embodiment and any smart speaker described herein can be used on a user device (such as a user's mobile computing device or other user device 102a, such as in Figure 1 This is accomplished by a digital assistant running on a computer (described in ).

[0134] Now turn to Figure 4 and Figure 5 , illustratively depicting aspects of exemplary process flows 400 and 500 of embodiments of the present disclosure. Process flows 400 and 500 each include methods (sometimes referred to herein as methods 400 and 500) that can be performed to implement many of the exemplary embodiments described herein. For example, process flow 400 or process flow 500 can be used to facilitate a computing system learning to associate unknown user commands with appropriate tasks, such as performing a user desired task (using a device operation) based on the user command.

[0135] An overview of process flow 400 and process flow 500 is provided in Figure 4 and Figure 5 The blocks of process flows 400 and 500, depicted in FIG. 1 , and corresponding to actions (or steps) to be performed (as opposed to information on which the actions are to be performed), may be performed by one or more computer applications or services, including, in some embodiments, a digital assistant running on one or more user devices (such as user device 104a), a server (such as server 106), may be distributed across multiple user devices and / or servers, or may be implemented in the cloud. In one embodiment, the functions performed by the blocks or steps of process flows 400 and 500 are performed by a combination of Figure 2 The components of the system 200 are described.

[0136] Steering Figure 4 , a flowchart illustrating an exemplary method 400 for determining a response activity event associated with performing a task indicated in a user command is provided. At step 410, a first indication of a user command to perform a task can be received. The user command can be characterized as a voice command, a gesture, etc. A task can be associated with a user command. Such a task can be implemented by one or more of the user's computing devices. For example, when the user command is "turn up the thermostat," the task of turning up the thermostat can be implemented by one or more of the user's computing devices. In some embodiments, the first indication of the user command received in step 410 can be received by a user interactive computing device. The user interactive computing device can receive the first indication of the user command from a sensor, such as a sensor in combination with Figure 1 Sensor 103 is described.

[0137] At step 420, the first indication of the user command can be used to initiate an operation monitoring mode. An embodiment of step 420 can initiate an operation monitoring mode when it is determined that the user command does not correspond to a predetermined response action. The predetermined response action can correspond to performing a task by initiating an operation by one or more user devices (including a user interactive computing device that received the first indication of the user command). These predetermined response actions can be stored as a list, library, and / or in a database. The predetermined response actions can be stored, for example, in conjunction with Figure 2 In the described response action 235. The user command can be checked against a set of predetermined response actions to determine that there is no corresponding action or operation to be performed for the particular user command.

[0138] In response to determining that the user command does not correspond to a predetermined response action, the user interactive computing device can enter a fault state. Specifically, a fault state may occur when an indication of a user command is received, such as, for example, at step 410, and the user command does not correspond to a predetermined response action. When the user interactive computing device enters the fault state, an operational monitoring mode can be initiated. The user interactive computing device can be used, for example, in conjunction with Figure 2 The fault condition detector 270 is described to detect the fault condition.

[0139] At step 430, user activity can be monitored to determine a response activity. Embodiments of step 430 monitor user data associated with at least one user device to identify or detect a user activity (sometimes referred to herein as a user action) that occurs after receiving the first indication of a user command. User activity can be monitored using sensor data using information about a user device associated with a user.

[0140] In one embodiment, at step 430, information detected via one or more user devices used by the user and / or cloud-based services associated with the user can be analyzed to determine activity information and related contextual information. Information about the user devices associated with the user can be determined based on user data available via, for example, user data collection component 210 and can be provided to, for example, user activity monitor 280, both of which are combined with Figure 2 The monitored user data can provide information that can be used to determine user activity of one or more devices, including identifying and / or tracking characteristics or other information about specific user actions and related contextual information.

[0141] In one embodiment, user activities performed on at least one device may be determined based on user data available via user data collection component 210, such as in conjunction with Figure 2 As described by the user activity monitor 280. For example, as previously described, information about the user device can be sensed or otherwise detected from the user data, such as by one or more sensors associated with the user device, or the above information can be determined by detecting and analyzing the user device related information in the user data to determine the user device such as device hardware, characteristics of the software such as the OS, network related characteristics, user accounts accessed via the device, and similar characteristics. In one embodiment, the detected user devices (such as user devices 102a to 102n) can be polled, queried, or otherwise analyzed to determine information about the device. In some implementations, the information can be used to determine a label or identification (e.g., a device ID) of the device so that user interactions with one device can be identified from user interactions on another device. In some embodiments of step 410, the user device can be inferred based on the user command, such as the case where the task of the user command is performed by a specific device, for example, a user command associated with a task involving temperature can infer that the user device is a smart thermostat, and a user command associated with a task involving playing music can infer that the user device is a smart speaker.

[0142] Some embodiments of step 430 may use at least one sensor to monitor user activity. Specifically, user activity can be monitored based on sensor data from one or more sensors associated with the set of user devices to determine the user activity performed. Some implementations of step 430 may be performed using user activity monitor 280, such as in Figure 2 In addition, some embodiments of step 430 may use an activity event logic unit to detect a response activity event, as described in conjunction with user activity detector 282. Figure 2User activity monitor 280 in provides additional details of an embodiment of step 420.

[0143] An embodiment of step 430 uses monitored user data associated with at least one user device to identify or detect user activities (sometimes referred to herein as user actions) that occur after a first indication of a received user command. The identified user activities can be used to determine a response activity event. In some cases, a response activity event may include a series or sequence of user actions and / or an interaction with one or more user devices. A response activity event may include at least one user activity performed on at least one user device associated with a task (e.g., a user manually performs a user activity including a response activity event, which results in the use of one or more user devices to perform a desired task).

[0144] At box 440, the response activity event can be associated with the task. Based on user activity information (e.g., user activity performed on at least one user device, monitored at box 430), the computer system can associate the response activity event with the task (from the user command). In an embodiment, the response activity event can be associated with a task involving one or more user devices. The task can be based on executing the response activity event on the one or more user devices. In some embodiments, box 440 can also include associating the response activity with the user command or associating the task with the user command.

[0145] In an embodiment, at box 440, the record of the association between the response activity event and the task can be stored in a data store. The data store may include multiple records about activity events, wherein each record may include information about a specific activity event, and the specific activity event includes one or more activity characteristics associated with the activity event. In some embodiments, the multiple records in the activity event data store include records of other activity events determined according to steps 410 to 430 of method 400. In some instances, some of the other records may include information about activity events derived from other users determined to be similar to the particular user (including associated activity characteristics), as previously described. In an embodiment, the data store includes user profiles and may be stored in a user activity information component, such as in combination with Figure 2 The user activity information component 242 of the user profile 240 described above. In another embodiment, the data store includes a response profile and may be stored in the response activity information component, such as in conjunction with Figure 2 Response profile 250 is described.

[0146] In an embodiment, multiple records of the association of response activity events with tasks can be analyzed to determine a response activity pattern. The response activity pattern can be used to determine the operation to be performed to implement the task. The response activity pattern can be associated with a response profile corresponding to a user command or used to generate a response profile and associated with the task based on a history of sensed (one or more) response activity events (including the response activity pattern). In this way, when an indication of a user command is received in the future, the (one or more) response activity events associated with the task can be determined (using the response profile) and used to cause the operation to be performed to implement the task.

[0147] In an embodiment, the response profile can be stored and made available to other users of interactive computing devices (e.g., reference Figure 2 The response profile can be published on a server accessible to other user interactive devices. The stored response commands can include a list of one or more user commands having known response activity patterns for performing operations to implement the task. The published response profile can also include an indication of at least one user device determined based on monitoring user activity. The published response profile can allow a particular user device to learn to perform a task even when the particular user device did not previously understand the user command (e.g., using device mapping 254, as described in reference to FIG. 2 ). Figure 2 In some embodiments, method 400 may be performed using a cloud system (such as the cloud system described above) and / or a cloud service to provide an improved or enhanced user experience (such as published response profiles) to multiple services running on many different user devices.

[0148] At step 450, a second indication of a user command to perform the task can be received. The user command and task in step 450 can be the same or similar to the user command and task of step 410. Similar user commands can result in the same task as the user command (e.g., "I'm cold," "It's cold in here," "Turn up the temperature," all result in the task of turning up the temperature). In some embodiments, the second indication of the user command received in step 450 can be received by the user interactive computing device that received the first indication of the user command in step 410. In other embodiments, the second indication of the user command received in step 450 can be received by a different user interactive computing device than the user interactive computing device that received the first indication of the user command in step 410. The second indication of the user command can be received from a sensor, such as in conjunction with a sensor. Figure 1The sensor 103 described. Therefore, an embodiment of step 450 can receive a second indication of a user command to perform the task indicated by the first indication received in step 410. In addition, an embodiment of step 450 can be performed after step 440, as described above, which can associate a response activity event with the task.

[0149] At step 460, based on the second indication of the user command, the response activity event associated with the task can be determined. The response activity event can be determined based on a predetermined response action stored as a list, library, and / or database. For example, the predetermined response action can be based on the association of the response activity event with the task at step 440. The predetermined response action can be stored, for example, in conjunction with Figure 2 The response action 235 described herein can be compared to a predetermined response action set (e.g., reference Figure 2 Described response action 235) is used to check the second indication of the user command.

[0150] At step 470, an operation can perform the task. In an embodiment, the response activity event as determined at step 470 can indicate an operation to be performed to perform the task associated with the indication of the user command. For example, some embodiments can cause an operation to perform a response activity event associated with one or more user computing devices (in response to the received user command). In some embodiments, the operation to perform the task in response to the user command can be based on a (generic or user-based) response profile for initiating a task on one or more devices.

[0151] Reference now Figure 5 , a flowchart illustrating an exemplary method 500 for determining possible future user actions based on a pattern of user activity is provided. At step 510, a first indication of a user command to perform a task may be received. The first indication of the user command may be as previously described with reference to Figure 2 or Figure 4 In an embodiment, the first indication of the user command may be received by a user interactive computing device. The user interactive computing device is capable of receiving the first indication of the user command from a sensor, such as in conjunction with Figure 1 Sensor 103 is described.

[0152] At step 520, a monitoring session can be initiated. A determination to initiate a monitoring session can be made based on the user command indicated in step 510. For example, an embodiment of step 420 can initiate an operational monitoring mode upon determining that the user command does not correspond to a predetermined response action. The predetermined response action can correspond to performing a task by initiating an operation by one or more user devices (including the user interactive computing device that received the first indication of the user command). In response to determining that the user command does not correspond to the predetermined response action, the user interactive computing device can enter a fault state (i.e., because, as in combination with Figure 2 The user command was not recognized as explained by the fault condition detector 270 in FIG. Figure 2 or Figure 4 Perform a monitoring session as described.

[0153] At step 530, a response activity event may be determined. In an embodiment, the response activity event may be determined based on user activity associated with at least one user device from the monitoring session. Determining a response activity event based on user activity associated with a user device from the monitoring session can be as previously described with reference to Figure 2 or Figure 4 At step 540, the response activity event can be associated with the user command. For example, the response activity event can be monitored, tracked and used to determine a response activity pattern associated with the user command. A response activity pattern can refer to multiple user interactions with one or more user devices, user activities on or related to one or more user devices, events (including actions) related to user activities, or any type of response activity event that can be determined via a computing device, wherein multiple interactions, actions, events, or activities share common features or characteristics.

[0154] In some embodiments, step 540 may also determine a confidence score for associating a response activity event (e.g., based on a response activity pattern) with a user command indicating that the response activity event is associated with a task (of a user command). For example, the confidence score may be based on the strength of the pattern, which may be determined by the number of observations used to determine the pattern, the frequency with which (one or more) response activity events are consistent with the pattern, the age or freshness of the activity observations, the number of features common to (one or more) response activity event observations constituting the pattern, or similar measurements. In some instances, the confidence score may be considered when providing a personalized user experience or other improved user experience. In addition, in some embodiments, a minimum confidence score may be required before using a response activity pattern to provide such an experience or other service. For example, in one embodiment, a threshold of 0.6 (or just over fifty percent) is used so that only response activity patterns with a probability of 0.6 (or greater) of performing the desired task corresponding to the user command are used to initiate response activity events that perform the task. However, when using confidence scores and thresholds, determined response activity patterns for (one or more) response activity events with confidence scores less than the threshold may still be monitored because additional observations of (one or more) response activity events can increase the confidence in a particular pattern.

[0155] In some embodiments, the response activity event can be characterized as semantic information, such as time, location, or other contextual information associated with the response activity event, as further described herein. Example associations of response activity events corresponding to user commands are time (e.g., a user command at 7:00 a.m. to "turn on the lights" involves a response activity mode that turns on the master bedroom lights, master bathroom lights, and kitchen lights, whereas a user command at 5:30 p.m. to "turn on the lights" involves a response activity mode that turns on the kitchen lights, living room lights, and dining room lights), location (e.g., a user command at work to "play music" involves a response activity mode that plays classical music, whereas a user command at home to "play music" involves a response activity mode that plays family music), content (e.g., a user command to "it's a little cold in here" involves a response activity mode that turns the smart thermostat up two degrees, whereas a user command to "I'm cold" involves a response activity mode that turns the smart thermostat up four degrees), or other context, as described herein.

[0156] At step 550, a second indication of a user command to perform the task can be received. In an embodiment, the second indication of the user command can be received after associating the response activity event with the user command. The second indication of the user command can be as previously described with reference to Figure 2 or Figure 4In an embodiment of step 550, the same and / or similar user command as received at step 510 can be received. In some embodiments, the second indication of the user command received at step 550 can be received by a user interactive computing device that received the first indication of the user command at step 510. In other embodiments, the second indication of the user command received at step 550 can be received by a user interactive computing device that is different from the user interactive computing device that received the first indication of the user command at step 510. The second indication of the user command can be received from a sensor, such as a sensor that is combined with Figure 1 Sensor 103 is described.

[0157] At step 560, a response activity event associated with the task can be determined. Determining a response activity event associated with the task can be as previously described with reference to Figure 2 or Figure 4 The response activity event may be determined based on the second indication of the received user command. In an embodiment, the response activity event may be determined based on a predetermined response action stored as a list, library, and / or in a database. For example, the predetermined response action can be based on an association of the response activity event with the task at step 540. The predetermined response action can be stored, for example, in conjunction with Figure 2 The response action 235 described herein can be compared to a predetermined response action set (e.g., reference Figure 2 Described response action 235) is used to check for a second indication of a user command.

[0158] Having described various implementations, an exemplary computing environment suitable for implementing embodiments of the present disclosure is now described. Figure 6 , an exemplary computing device is provided and is generally referred to as computing device 600. Computing device 600 is merely one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the disclosure. Computing device 600 should not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated.

[0159] Embodiments of the present disclosure may be described in the general context of computer code or machine-usable instructions, including computer-usable or computer-executable instructions, such as program modules, executed by a computer or other machine (such as a personal data assistant, smart phone, tablet PC or other handheld device). Typically, program modules include routines, programs, objects, components, data structures, etc., referring to codes that perform specific tasks or implement specific abstract data types. Embodiments of the present disclosure may be practiced in various system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. Embodiments of the present disclosure may also be practiced in a distributed computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including memory storage devices.

[0160] refer to Figure 6 , computing device 600 includes bus 610 that directly or indirectly couples the following devices: memory 612, one or more processors 614, one or more presentation components 616, one or more input / output (I / O) ports 618, one or more I / O components 620, and an exemplary power supply 622. Bus 610 represents one or more buses (such as an address bus, a data bus, or a combination thereof). Although shown with lines for clarity, Figure 6 , but in reality, these boxes represent logical units and not necessarily actual components. For example, one can consider a presentation component such as a display device to be an I / O component. Similarly, a processor has memory. The inventors hereby recognize that this is the nature of the art and reiterate that Figure 6 The diagrams of the present disclosure are merely illustrative of exemplary computing devices that can be used in conjunction with one or more embodiments of the present disclosure. No distinction is made between categories such as "workstation," "server," "laptop," "handheld device," etc., as all are contemplated herein. Figure 6 within the scope of and with reference to a “computing device”.

[0161] The computing device 600 typically includes various computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device 600 and include volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media. Computer storage media include both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to: RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage device, cassette, tape, disk storage device or other magnetic storage device, or any other medium that can be used to store the required information and can be accessed by the computing device 600. Computer storage media do not include the signal itself. Communication media typically embody computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier or other transmission mechanism, and include any information transfer medium. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example and not limitation, communication media include wired media such as a wired network or direct wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0162] Memory 612 includes computer storage media in the form of volatile and / or non-volatile memory. Memory may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, and the like. Computing device 600 includes one or more processors 614 that read data from various entities such as memory 612 or I / O components 620. (One or more) presentation components 616 present data indications to a user or other device. In some implementations, presentation component 220 of system 200 may be embodied as presentation component 616. Other examples of presentation components may include a display device, a speaker, a printing component, a vibration component, and the like.

[0163] I / O port 618 allows computing device 600 to be logically coupled to other devices, including I / O components 620, some of which may be built-in. Exemplary components include microphones, joysticks, game controllers, satellite dishes, scanners, printers, wireless devices, etc. I / O components 620 may provide a natural user interface (NUI) that processes air gestures, voices, or other physiological inputs generated by a user. In some instances, the input may be transmitted to an appropriate network element for further processing. NUI may implement any combination of voice recognition, touch and stylus recognition, facial recognition, biometrics, gesture recognition on and near the screen, air gestures, head and eye tracking, and touch recognition associated with a display on computing device 600. Computing device 600 may be equipped with a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, and a combination of these, for gesture detection and recognition. In addition, computing device 600 may be equipped with an accelerometer or gyroscope capable of detecting motion. The output of the accelerometer or gyroscope may be provided to a display of computing device 600 to present an immersive augmented reality or virtual reality.

[0164] Some embodiments of the computing device 600 may include one or more radio devices 624 (or similar wireless communication components). The radio device 624 sends and receives radio or wireless communications. The computing device 600 may be a wireless terminal suitable for receiving communications and media over various wireless networks. The computing device 600 may communicate via wireless protocols such as code division multiple access (“CDMA”), global system for mobile (“GSM”), or time division multiple access (“TDMA”) to communicate with other devices. Radio communications may be short-range connections, long-range connections, or a combination of both short-range and long-range wireless telecommunication connections. When we refer to “short” and “long” types of connections, we are not referring to the spatial relationship between two devices. Instead, we generally refer to short range and long range as different categories or types of connections (i.e., primary and secondary connections). By way of example and not limitation, a short-range connection may include a connection to a device that provides access to a wireless communication network (e.g., a mobile hotspot). A connection such as a WLAN connection using 802.11 protocols; a Bluetooth connection to another computing device is a second example of a short-range connection or near field communication connection. By way of example and not limitation, a long-range connection may include a connection using one or more of CDMA, GPRS, GSM, TDM A, and 802.16 protocols.

[0165] Many different arrangements of the various components depicted, as well as components not shown, are possible without departing from the scope of the following claims. Embodiments of the present disclosure have been described, intended to be illustrative rather than restrictive. Alternative embodiments will become apparent to the reader of the present disclosure after and as a result of reading the present disclosure. Alternative ways of implementing the foregoing can be accomplished without departing from the scope of the following claims. Certain features and subcombinations have utility and may be employed and are contemplated to be within the scope of the claims without reference to other features and subcombinations.

[0166] Thus, in one aspect, embodiments of the present disclosure relate to a computerized system, comprising: one or more sensors configured to provide sensor data; one or more processors; and a computer storage memory having computer executable instructions thereon, the computer executable instructions implementing a method when executed by the processor. The method comprises: receiving a first indication of a user command to perform a task; determining to initiate an operational monitoring mode based on the first indication of the user command; after associating a response activity event with the task, receiving a second indication of a user command to perform the task; determining a response activity event associated with the task based on the second indication of the user command; and causing an operation to be performed to implement the task.

[0167] In some embodiments of the system, the monitoring mode includes monitoring user activity using at least one sensor to determine a response activity event, the response activity event including at least one user activity performed on at least one user device; and associating the response activity event with the task. In addition, in some embodiments, determining to initiate the operational monitoring mode includes, for example but not limited to: determining that a user command does not correspond to a predetermined response action initiated by the user interactive computing device; and in response to determining that the user command does not correspond to the predetermined response action, causing the user interactive computing device to enter a fault state, wherein the user interactive computing device is operable to learn a new response action based on monitoring user activity occurring after a first indication of a received user command using the at least one sensor.

[0168] In another aspect, an embodiment of the present disclosure relates to a computerized system, comprising: one or more sensors configured to provide sensor data, each sensor being associated with a user device in a set of user devices associated with a user; one or more processors; and a computer storage memory having computer executable instructions stored thereon, the computer executable instructions implementing a method when executed by the processor. The method comprises: receiving a first indication of a user command to perform a task; determining to initiate monitoring based on the first indication of the user command; monitoring user activity to determine a response activity event, the response activity event comprising at least one user activity performed on at least one user device; determining a response activity event based on the user activity; associating the response activity event with the task; after associating the response activity event with the task, receiving a second indication of a user command to perform the task; determining the response activity event associated with the task based on the second indication of the user command; and causing an operation to be performed to implement the task.

[0169] In yet another aspect, an embodiment of the present disclosure relates to a computerized method. The method includes: receiving a first indication of a user command to perform a task; and based on the user command, starting a monitoring session to monitor user activity associated with at least one user device. The method also includes: determining a response activity event based on the user activity associated with the at least one user device from the monitoring session; and associating the response activity event with the user command. In addition, the method includes: after associating the response activity event with the user command, receiving a second indication of the user command to perform the task; based on the second indication of the user command, determining the response activity event associated with the user command; and based on the response activity event, causing an operation to be performed to implement the task.

Claims

1. A user interactive computing device comprising: one or more processors; as well as A computer storage memory having stored thereon computer executable instructions which, when executed by the one or more processors, implement a method comprising: receiving a first indication of a user command to perform a task; determining that the user command is not recognized; Based on the user command not being recognized, triggering a higher level of monitoring than the level of monitoring when the first indication is received, the higher level of monitoring comprising: monitoring user activity to determine a first responsive activity event, the first responsive activity event comprising at least one user activity action performed by a user on at least one user device; determining first context information associated with the first response activity event; associating the first response activity event including the at least one user activity action performed by the user with the task, wherein associating the first response activity event with the task comprises: associating the first context information with the first response activity event, and wherein the first context information distinguishes between the first response activity event associated with the user command and a second response activity event associated with the user command; after associating the first response activity event with the task, receiving a second indication of the user command to perform the task; Determine current context information; determining, based on the second indication of the user command, the first context information, and the current context information, the first responsive activity event associated with the task and including the at least one user activity operation performed by the user; and The at least one user activity operation is caused to be performed to facilitate performance of the task.

2. A user interactive computing device according to claim 1, wherein: The method further comprises: storing a record of the association of the first response activity event with the task in a data store; determining a response activity pattern based on an analysis of a plurality of records of associations of response activity events with the task, the response activity pattern comprising a plurality of associations of the task with the at least one user activity operation performed; and The first response activity event associated with the task is determined using the response activity pattern and the second indication of the user command.

3. A user interactive computing device according to claim 2, wherein: The method further comprises: A response profile is generated based on the determined first response activity pattern.

4. A user interactive computing device according to claim 3, wherein: The method further comprises: The response profile is stored to be available to other user interactive computing devices.

5. A user interactive computing device according to claim 4, wherein: Storing the response profile to be available for use with other user interactive devices includes: The response profile is published on a server accessible to the other user interactive devices.

6. A user interactive computing device according to claim 5, wherein: The method further comprises: determining an indication of the at least one user device based on monitoring the user activity; and The indication of the at least one user equipment is included in the published response profile.

7. The user interactive computing device of claim 1, wherein: The at least one user device on which the user activity is monitored is the user interactive computing device.

8. The user interactive computing device of claim 1, wherein: The first indication of the user command to perform the task is received as a voice command.

9. The user interactive computing device of claim 1, wherein: Monitoring the user activity to determine the first responsive activity event includes monitoring a plurality of user devices.

10. The user interactive computing device of claim 1, wherein: Determining that the user command is not recognized includes: determining that the user command does not correspond to a predetermined response action to be initiated by the user-interactive computing device; and In response to determining that the user command does not correspond to a predetermined response action, the user interactive computing device is caused to enter a fault state, wherein the user interactive computing device is operable to learn a new response action based on the monitoring of the user activity occurring after the first indication of the user command was received.

11. A computerized system comprising: one or more processors; as well as A computer storage memory having stored thereon computer executable instructions, the computer executable instructions when executed by the one or more processors implementing a method comprising: receiving a first indication of a user command to perform a task; determining that the user command is not recognized; triggering a higher level of monitoring than that at the time of receiving the first indication based on the user command not being recognized; monitoring user activity at the high level of monitoring to determine a first responsive activity event, the first responsive activity event comprising at least one user activity performed by a user on at least one user device from a set of user devices associated with the user; associating the first response activity event including the at least one user activity action performed by the user with the task, wherein associating the first response activity event with the task comprises: associating first context information with the first response activity event, and wherein the first context information distinguishes between the first response activity event associated with the user command and a second response activity event associated with the user command; After associating the first response activity event with the task, receiving a second indication of the user command to perform the task; determining, based on the second indication of the user command and the first context information, the first responsive activity event associated with the task and including the at least one user activity operation performed by the user; and The at least one user activity operation is caused to be performed to facilitate performance of the task.

12. The computerized system of claim 11, further comprising: storing a record of the association of the first response activity event with the task in a data store; determining a response activity pattern based on an analysis of a plurality of records of associations of response activity events with the task, the response activity pattern comprising a plurality of associations of the task with the at least one user activity operation performed; as well as The first response activity event is determined using the response activity pattern.

13. The computerized system of claim 12, wherein: Causing execution of the at least one user activity operation to facilitate performance of the task includes: The at least one user activity operation is initiated on the at least one user device.

14. The computerized system of claim 11, wherein: The user command includes at least one of: a verbal command, a gesture, and a physical interaction.

Images