Computer speech recognition and semantic understanding according to activity patterns

Abstract

Embodiments of the present disclosure relate to computer speech recognition and semantic understanding according to activity patterns. A user activity pattern can be determined using signal data from a set of computing devices. The activity pattern can be used to infer a user intent regarding a user's interaction with the computing devices or to predict a user's likely future action. In one implementation, the set of computing devices are monitored to detect user activity using sensors associated with the computing devices. Activity features associated with the detected user activity are determined and used to identify an activity pattern based on a plurality of user activities having similar features. Examples of user activity patterns can include patterns based on time, location, content, or other context. The inferred user intent or predicted future action can be used to facilitate understanding of a user's speech or determining a user's semantic understanding.

Description

[0001] This application is a divisional application of the invention patent application with the international application date of November 4, 2016, entered into the Chinese national phase on May 14, 2018, the Chinese national application number is 201680066544.5, and the invention name is "Computer Speech Recognition and Semantic Understanding According to Activity Patterns". TECHNICAL FIELD

[0002] Embodiments of the present disclosure generally relate to the field of computers, and in particular to computer speech recognition and semantic understanding according to activity patterns. BACKGROUND

[0003] People increasingly interact with computing devices and rely on these devices to obtain information, recommendations, and other services to help them with their daily tasks. But it remains a difficult technical problem for computing devices to understand the utterances and intentions of users in these interactions. In such interactions, users often become frustrated because their computerized personal assistant applications or services fail to understand them, their intentions, or anticipate their needs.

[0004] Meanwhile, many users of computing devices have repetitive usage patterns. For example, a user can launch an email application on their mobile device every weekday morning, then start work, browse a favorite news website at lunchtime, call a close friend or family member on the way home from work, or use their laptop to plan an annual vacation around May. By learning these user activity patterns, computerized personal assistant applications and services can provide an improved user experience, including improved understanding of the user's speech and their intentions. SUMMARY

[0005] This summary is provided to introduce a selection of concepts that are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] Embodiments described in this disclosure are directed to understanding user speech and inferring user intentions with respect to user interactions performed via a user's computing device. In particular, embodiments can determine likely spoken utterances or intentions of a user with respect to interactions with a computing device based on a history of sensed user activity, or can predict likely future actions that a user is about to perform. Data corresponding to user activity can be gathered over time using sensors of one or more of the user's computing devices. From this historical user activity information, a computer system can learn user activity patterns associated with the computing device. By analyzing the user activity patterns, future user actions can be predicted or user intentions can be inferred.

[0007] For example, a computing device associated with a user ("user device") can employ one or more sensors to generate data related to user activity on one or more user devices. User activity can be monitored, tracked, and used to determine user activity patterns. As described herein, examples of user activity patterns can include, but are not limited to, time-based, location-based, content-based, or other context-based activity patterns. In some embodiments, activity patterns can be determined based on user activity related to browsing, application usage, or other relevant user activity associated with one or more user devices or user activity that can otherwise be determined via one or more user devices.

[0008] Based on the determined user activity patterns, a user's intent can be inferred with respect to predictions of user interaction with a computing device and / or user activity that is determined and used to provide an improved user experience. Examples of improved user experiences described further herein can include improved speech recognition or improved semantic understanding of a user that can be used to more accurately parse, disambiguate, and / or understand a user's speech or other aspects of input from a user and related enhanced computer experiences such as personalization. In some embodiments, user activity patterns or inferred intent of future activity determined therefrom can be used by one or more applications and services that consume this information to provide an improved user experience. Some embodiments can be incorporated into (or operate in conjunction with) automatic speech recognition (ASR) and / or language modeling components or services. In this way, operating in conjunction with (or as a component of) speech recognition and interpreting operations, a user's utterance can be parsed in such a way that is consistent with learned user activity patterns.

[0009] Thus, as will be described further, in one embodiment, user activity of one or more user devices associated with a user can be identified and monitored. In some embodiments, user activity monitoring can be facilitated using applications or services running on the monitored user devices. User actions or "activity events" such as visiting a website, launching an application, or other actions similar to those described herein can be detected and recorded with related contextual data, for example, by recording observed user actions with timestamps, location stamps, and / or associating activity events with other available contextual information. From the recorded user activity, historical user activity information can be determined and provided to an inference engine. Based on analysis of the historical user activity, user activity patterns can be determined. In some embodiments, semantic analysis is performed on activity events and related information to characterize various aspects of the activity events. Semantic analysis can be used to characterize information associated with activity events and can provide other related features of activity events that can be used to identify patterns.

[0010] User activity patterns can be used to infer user intent or predict future activities of a user. Based on these predictions or inferred user intent, various implementations can provide personalized computing experiences and other services tailored to the user, such as improving speech recognition; incorporating the user's routine work into suggestions and notifications; improving semantic understanding by the user's computer device; or other examples as described herein. In this way, embodiments of the disclosure improve the operation of computing devices, thus providing more accurate speech recognition, improved resolution and disambiguation, more accurate understanding of the user's intent, and thus improving the user's experience using the computing device. BRIEF DESCRIPTION OF DRAWINGS

[0011] Aspects of the disclosure are described below in detail with reference to the accompanying drawings, in which

[0012] Figure 1 is a block diagram of an example operating environment suitable for implementations of the present disclosure;

[0013] Figure 2 is a diagram depicting an example computing architecture suitable for implementing aspects of the present disclosure;

[0014] Figure 3 Aspects of an example system for determining user activity patterns based on browser and application activity across multiple user devices in accordance with embodiments of the present disclosure are illustratively depicted;

[0015] Figure 4 is a flow diagram depicting a method for inferring user activity patterns in accordance with embodiments of the present disclosure;

[0016] Figure 5 is a flow diagram depicting a method for determining possible future user actions based on patterns of user activity in accordance with embodiments of the present disclosure;

[0017] Figure 6 is a flow diagram depicting a method for performing disambiguation to determine user intent in accordance with embodiments of the present disclosure;

[0018] Figure 7 is a flow diagram depicting a method for performing speech recognition to determine an action the user desires to perform in accordance with embodiments of the present disclosure;

[0019] Figure 8 is a flow diagram depicting aspects of a computing architecture for an automatic speech recognition system suitable for implementing embodiments of the present disclosure; and

[0020] Figure 9 is a block diagram of an example computing environment suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0021] The subject matter disclosed herein is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this patent. Rather, the inventors have contemplated that the claimed subject matter might also be embodied in other ways, to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or future technologies. Moreover, although the terms "step" and / or "block" might be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein commensurate with the purposes of the patent application.

[0022] Aspects of the present disclosure relate to understanding user speech to infer user intent associated with user interactions with computing devices. In particular, embodiments described herein can determine a user's likely verbal expression or intent with respect to an interaction with a user device based on a history of sensed user activity, or can predict a user's likely future actions. Data corresponding to user activity can be collected over time using sensors on one or more of the user devices associated with the user. From this historical user activity information, a computer system can learn user activity patterns associated with the user devices. By analyzing the user activity patterns, future user actions can be predicted or user intent can be inferred. In some cases, the user activity patterns can be analyzed together with sensor data collected by the user devices, and the user's intent inferred based on determining a likely intent that is consistent with the determined user activity patterns.

[0023] As further described herein, in some embodiments, a user device can employ one or more sensors to generate data related to a user's activities via one or more user devices. User activities can be monitored, tracked, and used to determine a user activity pattern. The term "activity pattern" is used broadly herein and can refer to a plurality of user interactions implemented using one or more user devices, activities by a user on or in connection with one or more user devices, events (including actions) related to a user's activities, or any type of user activity determinable via a computing device, where the plurality of interactions, actions, events, or activities share common characteristics or properties. In some embodiments, as further described herein, these common characteristics or variables can include characteristics that characterize a user's activities, times, locations, or other contextual information associated with a user's activities. As described herein, examples of user activity patterns can include, but are not limited to, activity patterns based on time (e.g., a user browses his bank's website to view his account balance at the beginning of each month), location (e.g., a user turns down the volume on his phone when arriving at work in the morning), content (e.g., a user typically browses news-related websites followed by his social media-related websites), or other contexts.

[0024] In some embodiments, user activities can be related to a user's browsing activities, such as websites, categories of websites, or sequences of websites and / or categories of websites visited by a user, and user activities associated with browsing activities. Additionally or alternatively, user activities can be related to a user's application (or app)-related activities, such as application usage, which can include duration of use, launches, files accessed via or in connection with an application, or content associated with an application. The term application or app is used broadly herein and generally refers to a computer program or computer application, which can include one or more programs or services, and can run on one or more devices of a user or in the cloud.

[0025] Based on the determined user activity patterns, inferences can be made about the user's intent regarding the user's interaction with the computing device or the user's activity, the inferences determined and used to provide improved user experiences. Examples of these improved user experiences, described further below, include personalized services such as tailoring content for the user, improving speech recognition or improving the user's semantic understanding, which can be used to perform disambiguation or other aspects of understanding input from the user. In some embodiments, the user activity patterns or inferences made from them of future activity can be made available to one or more applications and services that consume the information and provide improved user experiences, such as voice controlled user interfaces, which can be components of vehicles, smart appliances or other devices for which it is advantageous to have semantic understanding of the user's intent, or can be incorporated into language models or spoken language understanding models (SLU), or similar computer processes, components or services. In one embodiment, the activity pattern information can be provided via an API so that third party applications and services can use it based on the learned activity patterns, such as by providing speech to text services, voice operations or controls, timely recommendations, suggestions or other information or services relevant to the user. One embodiment can be incorporated into (or operate in conjunction with) automatic speech recognition (ASR) and / or language modeling components or services. In this way, operating in conjunction with (or as a component of) speech recognition and interpretation operations, a user's utterance can be parsed in such a way that is consistent with the learned user activity patterns. Such embodiments can provide significant advantages to personal digital assistant services and provide improvements to speech recognition and understanding technology by enabling computer recognized speech functionality to "learn" or evolve to understand the user.

[0026] Thus, at a high level, in one embodiment, user data is received from one or more data sources. The user data can be received by collecting user data using one or more sensors or components on one or more user devices associated with the user. Also described in connection with the components 210 are examples of user data that can include information about one or more user devices, user activity associated with the user devices (e.g., app usage, online activity, searches, calls, usage duration and other user interaction data), network related data such as network IDs, connection data or other network related information, application data, contact data, calendar and social network data, or virtually any other source of user data that can be sensed or determined by a user device or other computing device. The received user data can be monitored and information about the user's activity can be stored in a user profile, such as the user profile 240. Figure 2 Examples of user data described in connection with the components 210 can include information about one or more user devices, user activity associated with the user devices (e.g., app usage, online activity, searches, calls, usage duration and other user interaction data), network related data such as network IDs, connection data or other network related information, application data, contact data, calendar and social network data, or virtually any other source of user data that can be sensed or determined by a user device or other computing device. The received user data can be monitored and information about the user's activity can be stored in a user profile, such as the user profile 240. Figure 2 Examples of user data described in connection with the components 210 can include information about one or more user devices, user activity associated with the user devices (e.g., app usage, online activity, searches, calls, usage duration and other user interaction data), network related data such as network IDs, connection data or other network related information, application data, contact data, calendar and social network data, or virtually any other source of user data that can be sensed or determined by a user device or other computing device. The received user data can be monitored and information about the user's activity can be stored in a user profile, such as the user profile 240.

[0027] In some embodiments, user activity of one or more user devices is monitored based on an identification of the one or more user devices that can be determined from user data. In some embodiments, user activity monitoring can be facilitated using applications or services running on the monitored user devices. Alternatively or additionally, user activity monitoring can be facilitated using applications or services running in the cloud that can scan user devices, or detect online activity associated with user devices, such as http requests or other communication information, or otherwise receive information about user activity from user devices.

[0028] User data can be analyzed to detect various features associated with user actions. Detected user actions or "activity events" (which can include actions such as websites visited, applications launched, or other actions similar to those described herein) can be recorded with associated contextual data, for example, by recording observed user actions with corresponding timestamps, location stamps, and / or associating activity events with other available contextual information. In some embodiments, such recording can be performed on each user device, such that activity patterns can be determined across devices. Further, in some embodiments, cloud-based sources of user activity information can be used, such as online user calendars or user activities determined from social media posts, emails, and the like. These sources can also be used to provide other context to user activity detected on one or more user devices. In conjunction with Figure 2 Examples of contextual information extraction are further described in conjunction with contextual information extractor 284 of FIG. 2B. In some embodiments, user activity logs from multiple user devices and available user activity information from cloud-based sources can be combined, representing a composite user activity history. User activity logs including corresponding contextual information can be stored in a user profile associated with the user, such as user profile 240 of FIG. 2A. Figure 2

[0029] From the activity logs or user activity data, historical user activity information can be determined and provided to an inference engine. Based on analysis of historical user activity, and in some cases current sensor data regarding user activity, a set of one or more possible user activity patterns can be determined. In particular, in some embodiments, the inference engine can analyze historical user activity information to identify user activity patterns, which can be determined by detecting repetition of similar user actions or routines.

[0030] ​In some embodiments, a corresponding confidence weight or confidence score can be determined with respect to a user activity pattern. The confidence score can be based on the strength of the pattern, which can be determined by the number of observations used to determine the pattern, the frequency with which the user activity coincides with the pattern, the age or freshness of the activity observations, the number of features common to the activity observations that make up the pattern, or similar measures. In some instances, the confidence score can be considered when providing a personalized user experience or other improved user experience. Further, in some embodiments, a minimum confidence score can be required before using an activity pattern to provide such an experience or other service. For example, in one embodiment, a threshold of 0.6 (or just over 50%) is utilized, such that only activity patterns that predict the likelihood of a user activity with a 0.6 (or greater) can be considered. Nonetheless, with the use of confidence scores and thresholds, user activity patterns that are determined to have a confidence score below the threshold can still be monitored, as additional observations of user activity can increase the confidence in a particular pattern.

[0031] In some embodiments, a crowdsourced user activity history can also be utilized in conjunction with a user's own activity history. For example, for a given user, a set of other users that are similar to the given user can be identified based on having features or characteristics in common with the given user. This can include other users located near the given user, social media friends of the given user, work colleagues (which can be determined from an analysis of contextual information associated with the given user), other users with similar user activity patterns, etc. Information from the other users regarding their user activity history can be relied upon to infer user activity patterns for the given user. This can be particularly useful in instances where the given user has little user activity history, such as in the case of a new user. In some embodiments, as described further herein, user activity information from similar users can be imputed to a new user until sufficient user history is available for the new user to determine a statistically reliable user pattern prediction, which can be determined based on the number of observations included in the user activity history information or the statistical confidence of the determined user activity patterns. In some instances, where user activity history is recorded from other users, the resulting inferred activity patterns for the given user can be assigned a lower confidence.

[0032] In some embodiments, semantic analysis is performed on activity events and associated information to characterize various aspects of the activity events. For example, features associated with activity events can be categorized (such as, for example, by type, similar time ranges, or locations), and relevant features can be identified for determining similarity or relational closeness to other activity events, such as by having one or more of the characteristics including category and relevant features in common. In some embodiments, a semantic knowledge representation such as a relational knowledge graph can be employed. In some embodiments, semantic analysis can use rules, logic such as associations or conditions, or classifiers.

[0033] Semantic analysis can also be used to characterize information associated with activity events, such as determining that a location associated with an activity corresponds to a hub or venue of interest to the user (such as the user's home, place of work, gym, etc.) based on frequency of access. (For example, the user's home hub can be determined to be the location where the user spends most of their time between 8pm and 6am.) Similarly, semantic analysis can determine times of day that correspond to work hours, lunchtime, commute times, etc.

[0034] As such, semantic analysis can provide other relevant features of activity events that can be used to determine patterns. For example, in addition to determining that a user accesses a particular website at a particular time (such as accessing CNN.com at lunchtime), the category of the website can be determined, such as a news-related website. Similarly, semantic analysis can categorize activities as being associated with work or home based on characteristics of the activities (for example, a batch of online searches about chi-square distribution that occur during work hours at a location corresponding to the user's office can be determined to be work-related activities, while streaming a movie at a location corresponding to the user's home on a Friday night can be determined to be a home-related activity). These aspects characterizing user activity events can be considered when evaluating a user's activity history to identify patterns. For example, a pattern of accessing news-related websites at lunchtime can be determined where the user regularly accesses news-related websites at lunchtime but only occasionally accesses CNN.com as one of the news-related websites.

[0035] As previously described, user activity patterns can be used to infer user intent or predict a user's future activity. Based on these predicted or inferred user intent, various implementations can provide an improved user experience. For example, some embodiments can provide timely delivery or presentation of relevant content, or modify content that would otherwise be presented; incorporate a user's routine tasks into recommendations and notifications; improve speech recognition and improve semantic understanding of the user's computer device; or other examples of improved user experiences described herein. For example, acoustic information corresponding to a user's speech (such as spoken commands, queries, questions, or other data from other user interactions with a user device) (which may include visual information from gestures that can be sensed by sensors associated with the user device (such as a camera)) can be received and interpreted based on historical user activity patterns. For example, this information can be interpreted to enable a personal assistant application to respond or invoke an action on the user's behalf (such as setting a reminder, navigating to a website, locating a document for the user, or other tasks that can be performed via the user's device). As further described herein, some embodiments can be performed by a personal assistant application or service, which can be implemented as one or more computer applications, services, or routines, such as apps running on a mobile device or in the cloud.

[0036] Now go to Figure 1 , a block diagram is provided, which shows an example operating environment 100 in which some embodiments of the present disclosure can be adopted. It should be understood that this and other arrangements described herein are set forth as examples only. For the sake of clarity, other arrangements and elements (e.g., machines, interfaces, functions, sequences, and functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted completely for the sake of clarity. Further, many elements described herein are functional entities that may be implemented as discrete components or distributed components or in combination with other components to implement, as well as in any suitable combination and in any suitable location. The various functions described herein as being performed by one or more entities may be performed by hardware, firmware, and / or software. For example, some functions may be implemented by a processor executing instructions stored in a memory.

[0037] In addition to other components not shown, the example operating environment 100 includes several user devices, such as user devices 102a and 102b to 102n; several data sources, such as data sources 104a and 104b to 104n; a server 106; sensors 103a and 107; and a network 110. It should be understood that Figure 1 The depicted environment 100 is an example of a suitable operating environment. Figure 1 Each of the components shown in FIG can be connected to any type of computing device (such as, for example, a computer system) Figure 9The described computing device 900) to implement. These components can communicate with one another over a network 110, which can include, without limitation, one or more local area networks (LANs) and / or wide area networks (WANs). In an example implementation, the network 110 includes the Internet and / or a cellular network among any of a variety of possible public and / or private networks.

[0038] It should be appreciated that any number of user devices, servers, and data sources can be employed within the operating environment 100 within the scope of the present disclosure. Each of these can include a single device or multiple devices cooperating in a distributed environment. For example, the server 106 can be provided via multiple devices arranged in a distributed environment that collectively provide the functionality described herein. Additionally, other components not shown can also be included within the distributed environment.

[0039] The user devices 102a and 102b-102n can be client devices on the client side of the operating environment 100, while the server 106 can be on the server side of the operating environment 100. The server 106 can include server-side software designed to work in conjunction with client-side software on the user devices 102a and 102b-102n to implement any combination of the features and functionality discussed in the present disclosure. This partitioning of the operating environment 100 is provided to illustrate one example of a suitable environment, and each implementation does not require that any combination of the server 106 and the user devices 102a and 102b-102n remain as separate entities.

[0040] The user devices 102a and 102b-102n can include any type of computing device capable of being used by a user. For example, in one embodiment, the user devices 102a-102n can be of the type of computing devices described herein with respect to the computing device 900. By way of example and not limitation, the user devices can be embodied as a personal computer (PC), a laptop computer, a mobile or mobile device, a smartphone, a tablet computer, a smartwatch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a global positioning system (GPS) or device, a video player, a handheld communication device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a camera, a remote control, a barcode scanner, a computerized measuring device, an appliance, a consumer electronic device, a workstation, or any combination of these depicted devices, or any other suitable device. Figure 9

[0041] The data sources 104a and 104b-104n can include data sources and / or data systems configured to make data available to the operating environment 100 or in conjunction with the operating environment 100. For example, the data sources 104a and 104b-104n can include data sources that provide data to the operating environment 100 via the network 110. In one embodiment, the data sources 104a and 104b-104n can include data sources that provide data to the operating environment 100 via the network 110 and / or via a direct connection. Figure 2 ​Any of the various components of the system 200 described. (For example, in one embodiment, one or more data sources 104a to 104n may be provided to Figure 2 The user data collection component 210 provides (or makes available for access to) user data. The data sources 104a and 104b to 104n may be separate from the user devices 102a and 102b to 102n and the server 106, or may be incorporated into and / or integrated into at least one of those components. In one embodiment, one or more of the data sources 104a to 104n include one or more sensors that may be integrated into or associated with one or more of the user devices 102a, 102b or 102n or the server 106. Figure 2 Examples of sensed user data provided by the data sources 104a to 104n are further described with reference to the user data collection component 210 .

[0042] The operating environment 100 can be used to implement Figure 2 The system 200 described in Figure 3 One or more components of the described system 300 include components for collecting user data, monitoring activity events, determining activity patterns, consuming activity pattern information to provide an improved user experience, generating personalized content, and / or presenting notifications and related content to users. Figure 2 , combined with Figure 1 , a block diagram is provided that illustrates aspects of an example computing system architecture suitable for implementing embodiments and generally designated as system 200. System 200 represents only one example of a suitable computing system architecture. Other arrangements and elements may be used in addition to or in place of those shown, and some elements may be omitted entirely for clarity. Further, as with operating environment 100, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components and in any suitable combination and location.

[0043] The example system 200 includes a network 110 that incorporates Figure 1The components of system 200 are described and communicatively coupled, including user data collection component 210, presentation component 220, user activity monitor 280, activity pattern inference engine 260, activity pattern consumer 270, and storage 225. User activity monitor 280 (including components 282, 284, and 286 thereof), activity pattern inference engine 260 (including components 262, 264, 266, 267, and 269 thereof), user data collection component 210, presentation component 220, and activity pattern consumer 270 can be embodied as compiled computer instructions or functional sets, program modules, computer software services, or arrangements of processes executing on one or more computer systems, such as, for example, in conjunction with the computing device 900 described Figure 9 The computing device 900 described.

[0044] In one embodiment, the functions performed by the components of system 200 are associated with one or more personal assistant applications, services, or routines. In particular, such applications, services, or routines can operate on one or more user devices, such as user device 102a, can be distributed across one or more user devices and servers, such as server 106, or can be implemented in the cloud. Also, in some embodiments, these components of system 200 can be distributed across a network, which includes one or more servers, such as server 106, and client devices, such as user device 102a; distributed in the cloud; or can reside on a user device, such as user device 102a. Also, these components, the functions performed by these components, or the services performed by these components can be implemented on one or more appropriate abstraction layers of one or more computing systems, such as an operating system layer, an application layer, a hardware layer, etc. Alternatively or additionally, the functions of these components and / or embodiments described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), etc. Additionally, although the functions are described herein with respect to the specific components shown in example system 200, it is contemplated that, in some embodiments, the functions of these components can be shared or distributed across other components.

[0045] Continuing with reference Figure 2 to FIG. 1, user data collection component 210 is generally responsible for accessing or receiving (and, in some cases, also identifying) data from various sources, such as Figure 1user data of one or more data sources 104a and 104b through 104n. In some embodiments, the user data collection component 210 can be employed to facilitate the accumulation of user data for a particular user (or, in some cases, multiple users including crowd-sourced data) of the user activity monitor 280, the activity pattern inference engine 260, or the activity pattern consumer 270. Data can be received (or accessed) by the user data collection component 210 and, optionally, accumulated, reformatted, and / or combined, and stored in one or more data stores, such as the store 225, where it is available to other components of the system 200. For example, as described herein, user data can be stored in or associated with the user profile 240. In some embodiments, any personally-identifying data (i.e., user data that specifically identifies a particular user) is not uploaded or otherwise provided with user data from one or more data sources, not permanently stored, and / or not made available to the user activity monitor 280 and / or the activity pattern inference engine 260.

[0046] User data can be received from a variety of sources, where the data can be obtained in a variety of formats. For example, in some embodiments, user data received via the user data collection component 210 can be determined via one or more sensors, such as the sensors 103a and 107 in Figure 1 ​a video streaming service, a gaming service, or

[0047] In some aspects, user data can be provided in a user data stream or signal. A "user signal" can be a feed or stream of user data from a corresponding data source. For example, a user signal can be from a smartphone, a home sensor device, a GPS device (e.g., for location coordinates), a vehicle sensor device, a wearable device, a user device, a gyroscope sensor, an accelerometer sensor, a calendar service, an email account, a credit card account, or other data source. In some embodiments, the user data collection component 210 receives or accesses data continuously, periodically, or as needed.

[0048] ​The user activity monitor 280 is generally responsible for monitoring information in user data that can be used to determine user activity information, which can include identifying and / or tracking features (sometimes referred to herein as "variables") or other information about particular user actions and related contextual information. Embodiments of the user activity monitor 280 can determine user activity associated with a particular user from the monitored user data. As previously described, the user activity information determined by the user activity monitor 280 can include user activity information from multiple user devices associated with a user and / or from cloud-based services associated with a user (such as email, calendar, social media, or similar information sources), and it can include contextual information associated with the identified user activity. The user activity monitor 280 can determine current or near real-time user activity information, and it can also determine historical user activity information, which in some embodiments can be determined based on observations of user activity collected over time, accessing a user's log of past activity (such as, for example, a browsing history). Further, in some embodiments, the user activity monitor 280 can determine user activity from other similar users (i.e., crowd-sourcing), which can include historical activity, as previously described.

[0049] In some embodiments, the information determined by the user activity monitor 280 can be provided to the activity pattern inference engine 260, which includes information about current context and historical access (historical observations). Some embodiments can also provide user activity information, such as current user activity, to one or more activity pattern consumers 270. As previously described, user activity features can be determined by monitoring user data received from the user data collection component 210. In some embodiments, user data and / or information about user activity determined from the user data is stored in a user profile, such as the user profile 240.

[0050] In embodiments, the user activity monitor 280 includes one or more applications or services that analyze information detected via one or more user devices used by a user and / or cloud-based services associated with a user to determine activity information and related contextual information. Information about user devices associated with a user can be determined from user data provided via the user data collection component 210, and it can be provided to the user activity monitor 280, the activity pattern inference engine 260, or other components of the system 200.

[0051] More specifically, in some implementations of the user activity monitor 280, a user device can be identified by detecting and analyzing characteristics of the user device, such as device hardware, software such as an operating system (OS), network-related characteristics, user accounts accessed via the device, and similar characteristics. For example, functionality of many operating systems can be used to determine information about a user device to provide information about hardware, OS version, network connection information, installed applications, and the like.

[0052] Some embodiments of the user activity monitor 280 or its subcomponents can determine a device name or identification (device ID) for each device associated with a user. This information about the identified user devices associated with a user can be stored in a user profile associated with the user, such as in the user accounts and devices 244 of the user profile 240. In embodiments, a user device can be polled, interrogated, or otherwise analyzed to determine information about the device. This information can be used to determine a label or identification for the device (e.g., a device ID) so that the user activity monitor 280 can identify user interactions with the device from user data. In some embodiments, a user can declare or register a device, such as by logging into an account via the device, installing an application on the device, connecting to an online service that interrogates the device, or otherwise providing information about the device to an application or service. In some embodiments, devices that log into an account associated with a user, such as a Microsoft account or a web passport, an email account, a social network, and the like, are identified and determined to be associated with the user.

[0053] As shown in the example system 200, the user activity monitor 280 includes a user activity detector 282, a contextual information extractor 284, and an activity feature determiner 286. In some embodiments, the user activity monitor 280, one or more of its subcomponents, or other components of the system 200, such as the activity pattern consumer 270 or the activity pattern inference engine 260, can determine interpretive data from received user data. Interpretive data corresponds to data used by these components of the system 200 or subcomponents of the user activity monitor 280 to interpret user data. For example, interpretive data can be used to provide other context for user data that can support determinations or inferences made by the components or subcomponents. Also, embodiments of the user activity monitor 280, its subcomponents, and other components of the system 200 can be envisioned to use user data and / or user data in conjunction with interpretive data to achieve the purposes of the subcomponents described herein. Additionally, although several examples of how the user activity monitor 280 and its subcomponents identify user activity information are described herein, many variations of user activity identification and user activity monitoring are possible in various embodiments of the present disclosure.

[0054] Generally, the user activity detector 282 is responsible for determining (or identifying) user actions or activity events that have occurred. Embodiments of the activity detector 282 can be used to determine current user activity or one or more historical user actions. Some embodiments of the activity detector 282 can monitor activity-related features or variables in user data corresponding to user activity, such as indications of applications launched or accessed, files accessed, modified, copied, etc., websites navigated to, online content downloaded and rendered or played, or similar user activity.

[0055] Additionally, some embodiments of the user activity detector 282 extract information about user activity from user data, which can include current user activity, historical user activity, and / or related information such as contextual information. (Alternatively or additionally, in some embodiments, the contextual information extractor 284 determines and extracts contextual information. Similarly, in some embodiments, the activity feature determiner 286 extracts information about user activity based on the identification of activities determined by the user activity detector 282, such as user activity-related features). Examples of extracted user activity information can include app usage, online activity, searches, calls, usage duration, application data (e.g., emails, messages, posts, user status, notifications, etc.), or almost any other data related to user interaction with or activity via a user device. In addition to other components of the system 200, the extracted user activity information determined by the user activity detector 282 can be provided to the user activity monitor 280, the activity pattern inference engine 260, or other subcomponents of one or more activity pattern consumers 270. Further, the extracted user activity can be stored in a user profile associated with the user, such as in the user activity information component 242 of the user profile 240. In some embodiments, the user activity detector 282 or the user activity monitor 280 (or other subcomponents thereof) performs consolidation on the detected user activity information. For example, overlapping information can be merged and duplicated, or redundant information can be eliminated.

[0056] In some embodiments, user activity related features can be interpreted to determine that a user activity has occurred. For example, in some embodiments, the user activity detector 282 employs user activity event logic, which can include rules, conditions, associations, classification models, or other criteria to identify user activities. For example, in one embodiment, the user activity event logic can include comparing user activity criteria to user data to determine that an activity event has occurred. Depending on the mechanism used to identify activity events, the activity event logic can take many different forms. For example, the user activity event logic can be training data for a neural network used to train a neural network for evaluating user data to determine when an activity event has occurred. The activity event logic can include fuzzy logic, neural networks, finite state machines, support vector machines, logistic regression, clustering or machine learning techniques, similar statistical classification processes, or combinations of these to identify activity events from user data. For example, the activity event logic can specify types of user device interaction(s) information associated with an activity event, such as navigating to a website, composing an email, or launching an application. In some embodiments, a series or sequence of user device interactions can be mapped to an activity event, such that the activity event can be detected when it is determined that user data indicates that the user has performed the series or sequence of user interactions.

[0057] In some embodiments, the activity event logic can specify types of user device related activities that are considered to be user activities, such as when a user logs into a user device, activities that occur when a user interface is receiving input (e.g., when a computer mouse, touchpad, screen, voice recognition interface, etc. is active), or certain types of activities such as launching an application, using an application to modify a file, opening a browser and navigating to a website. In this way, the activity event logic can be used to distinguish real user activities from automated activities of processes running on the user device, such as automatic updates or malware scans. Once user activities are determined, these features or additional related features can be detected and associated with the detected activities for use in determining activity patterns.

[0058] In some embodiments, the user activity detector 282 runs on or in association with each user device of a user. The user activity detector 282 can include polling or analyzing aspects of the operating system to determine user activity related features, such as, for example, installed or running applications or file access and modification, network communications, and / or other user actions that can be detected via the user device including sequences of actions.

[0059] In some embodiments, such as the embodiment shown in system 200, the user activity detector 282 includes subcomponents, including an application activity recording pipeline 283 and a browsing activity recording pipeline 285. These recording pipelines can be embodied as client-side applications or services that run on each user device associated with the user, and in some embodiments can run in conjunction with or within (or as part of) an application, such as within a browser or as a browser plug-in or extension. Generally speaking, the application activity recording pipeline 283 manages recording of a user's application (or app) activity, such as application downloads, launches, accesses, usage (which may include duration), file accesses via the application, and user activity within the application (which may include application content). Generally speaking, the browsing activity recording pipeline 285 manages recording of a user's browsing activity, such as websites visited, social media activity (which may include browsing via a particular browser or such), and browsing activity recording pipeline 285. app, app, app, 285 ), content downloaded, files accessed, and other browsing-related user activity. In some embodiments, each browser on a user device is associated with an instance of the browsing activity recording pipeline 285, or alternatively, a plug-in or service provides browsing information to a single instance of the browsing activity recording pipeline 285 on the user device. In some embodiments, the app and browsing activity recording pipelines 283 and 285 may also perform the functions described in conjunction with the context information extractor 284, such as recording timestamps, location stamps, user device-related information, or other contextual information associated with the recorded app activity or browsing activity. In some embodiments, the app and browsing activity recording pipelines 283 and 285 upload the recorded user activity information to the activity pattern inference engine 260 and / or store the recorded activity information in a user profile associated with the user, such as the user activity information component 242 of the user profile 240.

[0060] Generally, the context information extractor 284 is responsible for determining context information related to user activity (detected by the user activity detector 282 or the user activity monitor 280), such as context features or variables associated with the user activity, related information, and user-related activity, and further responsible for associating the determined context information with the detected user activity. In some embodiments, the context information extractor 284 can associate the determined context information with the related user activity, and can also record the context information with the associated user activity. Alternatively, the association or recording can be performed by another service. For example, some embodiments of the context information extractor 284 provide the determined context information to the activity feature determiner 286, which determines activity features of the user activity and / or related context information.

[0061] Some embodiments of the context information extractor 284 determine context information related to a user action or activity event, such as entities identified in or related to the activity (e.g., recipients of a group email sent by the user), which can include nicknames used by the user (e.g., "Mom" and "Dad" refer to specific entities that can be identified by their actual names in the user's contacts), that the user activity is associated with a location or venue of the user's device. By way of example and not limitation, this can include context features such as location data, which can be represented as a location stamp associated with the activity; context information about the location, such as venue information (e.g., this is the user's office location, home location, school, restaurant, mobile theater, etc.), yellow pages identifier (YPID) information, time, day and / or date, which can be represented as a time stamp associated with the activity; user device characteristics or user device identification information about the device on which the user performed the activity; duration of the user activity, one other user activity / multiple other user activities before and / or after the user activity (such as a sequence of websites visited, a sequence of online searches made, a sequence of applications, and a sequence of website usage, such as browsing to a bank and then accessing an Excel spreadsheet file to record financial information, etc.), other information about the activity (such as entities associated with the activity (e.g., a venue, a person, an object, etc.)), which can include nicknames or expressions of individuals used by the user (and in some examples, created by the user), or terms used by (and in some examples, created by) the user or the user's acquaintances (e.g., a name for a venue that is specific to the user (and not everyone), such as "Dikla's house," "Shira's car," "my Seattle friends," etc.), information detected by one or more sensors on the user device associated with the user concurrent or substantially concurrent with the user activity (e.g., motion information or physiological information detected on a fitness tracking user device, listening to music, which can be detected via a microphone sensor if the source of the music is not the user device), or any other information related to detectable user activity that can be used to determine a user activity pattern.

[0062] In embodiments that use context information related to user devices, the user devices can be identified by detecting and analyzing characteristics of the user devices, such as device hardware, software such as operating systems (OS), network-related characteristics, user accounts accessed via the device, and similar features. For example, as previously described, the functionality of many operating systems can be used to determine information about the user device to provide information about the hardware, OS version, network connection information, installed applications, and the like. In some embodiments, a device name or identification (device ID) can be determined for each device associated with a user. This information about the identified user devices associated with a user can be stored in a user profile associated with the user, such as in one or more user accounts and one or more devices 244 of user profile 240. In embodiments, a user device can be polled, interrogated, or otherwise analyzed to determine context information about the device. This information can be used to determine a label or identification for the device (e.g., a device ID) so that user activity on one user device can be recognized and distinguished from user activity on another user device. Further, as previously described, in some embodiments, a user can claim or register a user device, such as by logging into an account via the device, installing an application on the device, connecting to an online service that interrogates the device, or otherwise providing information about the device to an application or service. In some embodiments, a device that is logged into an account associated with a user, such as an account or network passport, email account, social network, and the like, is identified and determined to be associated with the user.

[0063] In some implementations, context information extractor 284 can receive user data from user data collection component 210, parse the data in some instances, and identify and extract context features or variables (which can also be performed by activity feature determiner 286). The context variables can be stored as a relevant set of context information associated with a user activity, and can be stored in a user profile, such as in user activity information component 242. In some cases, the context information can be used by one or more activity pattern consumers, such as for personalizing content or user experience, such as when, where, or how to present content. In some embodiments, instead of or in addition to user activity information for a particular user, context information can be determined from user data of one or more users, which can be provided by user data collection component 210.

[0064] ​The activity feature determiner 286 is generally responsible for determining activity-related features (or variables) associated with user activity that can be used to identify user activity patterns. Activity features can be determined from information about user activity and / or from related contextual information. In some embodiments, the activity feature determiner 286 receives user activity or related information from the user activity monitor 280 (or subcomponents thereof) and analyzes the received information to determine a set of one or more features associated with the user activity.

[0065] Examples of activity-related features include, but are not limited to, location-related features such as location of one or more user devices during user activity, venue-related information associated with a location, or other location-related information; time-related features such as one or more times of day or days, date of the week or month of user activity, or duration of activity; or related duration information such as how long a user uses an application associated with the activity; user device-related features such as device type (e.g., desktop, tablet, mobile phone, fitness tracker, heart rate monitor, etc.), hardware attributes or profile, OS or firmware attributes, device ID or model; network-related information (e.g., mac address, network name, IP address, domain, workgroup, information about other devices detected on a local network, router information, proxy or VPN information, other network connection information, etc.); location / motion / position-related information about a user device; power information such as battery level, time connected / disconnected from a charger, and user access / touch information; usage-related features such as one or more files accessed, app usage (which can also include application data, in-app usage, concurrently running applications), network usage information, one or more user accounts accessed or otherwise used (such as one or more device accounts, one or more OS-level accounts, or online / cloud service-related account activity (such as account or network passport, one or more online storage accounts, email, calendar, or social network accounts), etc.); content-related features such as online activity (e.g., searches, websites browsed, purchases, social network activity, communications sent or received including social media posts); other features that can be detected contemporaneously with or near the time of user activity; or any other features that can be detected or sensed and used to determine user activity patterns. Features can also include information about one or more users using the device; other information identifying a user such as login password, biometric data that can be provided by a fitness tracker or biometric scanner; and / or characteristics of one or more users using the device that can be useful to distinguish between users on a device shared by more than one user. In some embodiments, specific features can be identified from user activity information using user activity event logic (described in connection with the user activity detector 282).

[0066] Continued Figure 2 The system 200, the activity pattern inference engine 260 is generally responsible for determining user activity patterns based on user activity information determined from the user activity monitor 280. In some embodiments, the activity pattern inference engine 260 can run on a server, as a distributed application across multiple devices, or in the cloud. At a high level, the activity pattern inference engine 260 can receive user activity related information, which can be uploaded from user activity logs from client-side applications or services associated with the user activity monitor 280. One or more inference algorithms can be applied to the user activity related information to determine a set of possible user activity patterns. For example, a pattern can be determined based on similar observed instances of user activity or associated contextual information, which can be referred to as "common features" of the user activity related information. The inferred activity pattern information can be provided to the activity pattern consumer 270 and / or used to generate pattern-based predictions regarding one or more possible future user actions. In some embodiments, as described herein, a corresponding confidence is also determined for the pattern (or pattern-based prediction). Further, the activity pattern (or prediction of future actions based on the pattern) can include a single (future occurring) user activity that is likely to occur, a sequence of future user actions, or probabilities for more than one future action, e.g., a likelihood of eighty percent that the next action will be to browse to website A, a likelihood of fifteen percent that the next action will be to launch a music player application, and a likelihood of five percent that the next action will be to browse to website B.

[0067] As shown in the example system 200, the activity pattern inference engine 260 includes a semantic information analyzer 262, a feature similarity identifier 264, and an activity pattern determiner 266. The semantic information analyzer 262 is generally responsible for determining semantic information associated with user activity related features identified by the user activity monitor 280. For example, while a user activity feature can indicate a particular website visited by the user, semantic analysis can determine a website category, related websites, a theme or topic, or other entity associated with the website or user activity. The semantic information analyzer 262 can determine additional user activity related features that are semantically related to the user activity, which can be used to identify user activity patterns.

[0068] In particular, as previously described, semantic analysis is performed on user activity information, which can include contextual information, to characterize various aspects of user actions or activity events. For example, in some embodiments, activity features associated with an activity event can be classified or categorized (such as by type, time range or location, work-related, home-related, topic, related entities, one or more other users (such as being a communication to or from another user), and / or another user's relationship to the user (e.g., family member, close friend, work acquaintance, boss, etc.), or other categories) or relevant features can be identified for determining similarity or relational proximity to other user activity events, which can indicate a pattern. In some embodiments, semantic information analyzer 262 can utilize a semantic knowledge representation such as a relational knowledge graph. Semantic information analyzer 262 can also utilize semantic analysis logic, which includes rules, conditions, or associations to determine semantic information related to user activity. For example, a user activity event that includes an email sent to someone who works with the user can be characterized as a work-related activity. Thus, if a user sends emails every Sunday night to people who work with her (but not necessarily to the same person), it can be determined (using activity pattern determiner 266) that the user performs work-related activities every Sunday night. Thus, it can be appropriate to show the user notifications (such as reminders) related to the user's work on Sunday night, since the user has a work pattern on Sunday night. (Here, a notification service is one example of activity pattern consumer 270).

[0069] The semantic information analyzer 262 can also be used to characterize contextual information associated with user activity events, such as determining that a location associated with an activity corresponds to a center of interest or a venue of interest to the user (such as the user's home, work, gym, etc.) based on the frequency of user visits. For example, the user's home center can be determined (using semantic analysis logic) to be the location where the user spends the majority of time between 8pm and 6am. Similarly, semantic analysis can determine times of day that correspond to work hours, lunch hours, commute times, etc. Similarly, based on other characteristics of the activity, semantic analysis can categorize the activity as being associated with work or home (e.g., a batch of online searches about chi-squared distribution that occur at a location corresponding to the user's office during work hours can be determined to be a work-related activity, while streaming a movie at a location corresponding to the user's home on a Friday night can be determined to be a home-related activity). In this way, the semantic analysis provided by the semantic information analyzer 262 can provide other relevant characteristics of user activity events that can be used to determine user activity patterns. For example, where a user activity includes visiting CNN.com at lunchtime and semantic analysis determines that the user visited a news-related website at lunchtime, a user activity pattern can be determined (by the activity pattern determiner 266) that indicates that the user routinely visits news-related websites at lunchtime, but only occasionally visits CNN.com as one of those news-related websites.

[0070] The feature similarity identifier 264 is generally responsible for determining the similarity of activity features of two or more user activity events (in other words, characterizing activity features of a first user activity event that are similar to activity features characterizing a second user activity event). Activity features can include features related to contextual information and features determined by the semantic information analyzer 262. Activity events having common activity features can be used to identify activity patterns, which can be determined using the activity pattern determiner 266.

[0071] For example, in some embodiments, the feature similarity identifier 264 can incorporate one or more pattern-based predictors 267 (sub-components of the activity pattern determiner 266) to determine a set of user activity events having common features. In some embodiments, this set of user activity events can be used as input to the pattern-based predictors, as described below. In some embodiments, the feature similarity identifier 264 includes functionality to determine similarity of periodic and behavioral-based activity features. Periodic features include, for example, features that can occur periodically; e.g., a certain day of the week or month, even / odd day (or week), monthly, yearly, every other day, every third day, etc. Behavioral features can include behaviors such as user activities that tend to occur with certain locations or activities, for example, before or after a given user activity event (or sequence of previous activity events).

[0072] In embodiments where activity features have values, based on a particular feature, similarity can be determined among different activity features having the same value or approximately the same value. (For example, a timestamp of a first activity occurring at 9:01 AM on Friday and a timestamp of a second activity occurring at 9:07 AM on Friday can be determined to have similar or common timestamp features.)

[0073] The activity pattern determiner 266 is generally responsible for determining user activity patterns based on similarities identified in the user activity information. Specifically, the activity pattern determiner 266 (or activity pattern inference engine 260) can determine user activity patterns based on repetition of similar activity features associated with multiple observed activity events. Thus, for example, an activity pattern can be determined where activity features corresponding to two or more activity events are similar. In some instances, an activity event can have many corresponding activity features, which can be represented as a feature vector associated with a particular activity event. As such, the analysis performed by the activity pattern determiner 266 can involve comparing activity features from feature vectors of multiple activity events.

[0074] In some embodiments, activity patterns can be determined using pattern inference logic 230. The pattern inference logic can include rules, associations, conditions, predictive and / or classification models, or pattern inference algorithms. The pattern inference logic 230 can take many different forms depending on the particular activity pattern or mechanism used to identify activity patterns or to identify characteristic similarities between observed activity events to determine a pattern. For example, as described further below, some embodiments of the pattern inference logic 230 can employ machine learning mechanisms to determine characteristic similarities, or other statistical measures for determining activity events that belong to a "set of example user actions" that support a determined activity pattern. User activity information can be received from the user activity monitor 280, and information about identified similar characteristics can be received from the characteristic similarity identifier 264. In some embodiments, one or more user patterns determined by the activity pattern determiner 266 can be stored as inferred user routines 248 in the user profile 240.

[0075] In some embodiments, the activity pattern determiner 266 provides a user activity pattern and an associated confidence score regarding the strength of the user pattern, which can reflect the likelihood that future user activity will follow the pattern. More specifically, in some embodiments, a corresponding confidence weight or confidence score can be determined regarding a determined user activity pattern. The confidence score can be based on the strength of the pattern, which can be determined based on the number of observations (of particular user activity events) used to determine the pattern, the frequency with which the user's actions coincide with the activity pattern, the age or freshness of the activity observations, the number of similar characteristics, the type of characteristics, and / or the degree of similarity of the characteristics common to the activity observations that make up the pattern, or similar measures.

[0076] In some instances, the confidence score can be considered when providing determined activity patterns to the activity pattern consumer 270. For example, in some embodiments, a minimum confidence score can be required before the activity pattern consumer 270 uses the activity pattern to provide an improved user experience or other service. In one embodiment, a threshold of 0.6 (or just over 50%) is utilized, such that only activity patterns with a likelihood of predicting user activity of 0.6 (or greater) can be provided. Nonetheless, because additional observations can increase the confidence for a particular pattern, determined user activity patterns with a confidence score less than the threshold can still be monitored and updated based on additional activity observations, with the use of a confidence score and threshold.

[0077] Some embodiments of the activity pattern determiner 266 determine a pattern from the example approach described below, in which each instance of a user activity event has a corresponding historical value of a tracked activity feature (variable) that forms the pattern, and in which the activity pattern determiner 266 can evaluate a distribution of the tracked variable for the pattern. In the following example, the tracked variable of a user activity event is a timestamp corresponding to an observed instance of the user activity event. However, it should be appreciated that the following can be applied conceptually to different types of historical values of a tracked activity feature (variable).

[0078] A timestamp (i.e., a value of a given tracked variable) bag can be represented as and mapped to a two-dimensional histogram of hours and days of the week. The two-dimensional histogram can include a sum of user-device interaction instances, such as:

[0079]

[0080] The histogram can be used to determine a derived histogram. For example, the days of the week histogram can correspond to h j =∑ j h ij . The hours of the day histogram can correspond to h i =∑ j h ij . As a further example, one or more histograms can be determined for a particular semantic time resolution of the form h iC =∑ j∈C h ij . Any of a variety of semantic time resolutions can be employed, such as weekdays and weekends, or morning, afternoon, and evening. An example of the latter is the case C e {morning, afternoon, evening}, morning = {9, 10, 11}, afternoon = {12, 13, 14, 15, 16}, and evening = {21, 22, 23, 24}.

[0081] An additional data structure for representing events can include a number of different timestamps in each calendar week with at least one timestamp, which can be represented as:

[0082]

[0083] As an example, can represent the number of different timestamps during a second three-week period of available timestamps. N (j) can be used to represent the number of j-week timestamps available in the tracked data; for example, N (3) represents the number of three-week periods available in the timestamps.

[0084] The activity pattern determiner 266 (or activity pattern inference engine 260) can generate a confidence score that quantifies the level of certainty of a particular pattern formed by the historical values in the tracked variables. In the following examples, the above principles are applied using Bayesian statistics. In some implementations, a confidence score can be generated for corresponding tracked variables indexed by time intervals of varying resolution. For time stamps, examples include Tuesday 9am, weekday morning, and Wednesday afternoon. The confidence score can be computed by applying a Dirchlet-multinomial model and computing the posterior predictive distribution of each period histogram. In doing so, the prediction for each bin in a particular histogram can be given by:

[0085]

[0086] where K denotes the number of bins, a0is a parameter that encodes the strength of the prior knowledge, and The mode prediction is then the bin of the histogram corresponding to i*whose confidence is given by x i* As an example, consider a histogram where morning = 3, afternoon = 4, and evening = 3. Using a0= 10, the mode prediction is afternoon and the confidence score is According to various implementations, more observations result in an increased confidence score, indicating an increased confidence in the prediction. As an example, consider a histogram where morning = 3000, afternoon = 4000, and evening = 3000. Using similar calculations, the confidence score is

[0087] Further, in some implementations, a confidence score can be generated for corresponding tracked variables indexed by time period and a number of time stamps. Examples include 1 visit per week, and 3 visits per 2 weeks. Using a Gaussian posterior, a confidence score can be generated for each time period resolution of the mode, denoted as j. This can be achieved by taking the following formula: where In the foregoing, a2is the sample variance, and and a0are parameters of the formula. The confidence score can be computed by taking a fixed interval around the number of time stamp predictions and computing the cumulative density:

[0088] where

[0089] As an example, consider the following observations: wl (1) = 10, w2 (1) = 1, w3 (1) = 10, w4 (1) = 10, w4 (1) = 0, wl (2) = 11, and w2(2) = 10. N (1) = 4 and N (2) = 2. Using μ0= 1 and σ0 2 = 10, μ (1) = 4.075, conf1 = 0.25. Further, μ (2) = 10.31 and conf2 = 0.99. In the preceding example, although fewer timestamps are available within the two-week period, the reduced variance in the user signal results in an increased confidence that the pattern exists.

[0090] Having determined that a pattern exists or that a confidence score for a pattern is sufficiently high (e.g., satisfies a threshold), the activity pattern determiner 266 can identify a plurality of user activities that correspond to a user activity pattern for the user. As a further example, the activity pattern determiner 266 can determine that a user can follow a user activity pattern in which one or more of the confidence scores for one or more tracked variables satisfy a threshold.

[0091] In some embodiments, as previously described, a user activity pattern can be determined by monitoring one or more activity features. These monitored activity features can be determined from user data previously described as tracked variables or user data described in connection with the user data collection component 210. In some cases, a variable can represent a contextual similarity and / or a semantic similarity among a plurality of user actions (activity events). As such, a pattern can be identified by detecting a variable or feature that is common to a plurality of user actions. More specifically, a feature associated with a first user action can be correlated with a feature of a second user action to determine a possible pattern. An identified pattern of features can become stronger (i.e., more likely or more predictable) if the user activity observations that make up the pattern are repeated more frequently. Similarly, a particular feature can be more strongly associated with a user activity pattern as the user activity pattern is repeated.

[0092] In some embodiments, such as the example embodiment shown in the system 200, the activity pattern determiner 266 includes one or more pattern-based predictors 267. The pattern-based predictors 267 include one or more predictors for predicting a next user action or a future user action that a user will take based on a pattern, such as a behavioral pattern or a similarity feature. At a high level, the pattern-based predictors 267 receive user activity information and / or activity features associated with user activities and determine a prediction of a next action or a future action that a user will take. In embodiments, the pattern-based predictors 267 include functionality as further described below for performing user activity filtering, determining activity scores, selecting activities based on scores, and determining a particular pattern-based prediction.

[0093] In one embodiment, the pattern-based predictor 267 uses the feature similarity identifier 264 to determine features or patterns that are common between historical user activity and recent user activity. For example, a periodicity feature similarity can be determined from among the set of historical user actions based on those historical actions that share a periodicity feature with the current or recent user action. Thus, for example, if the recent user action occurs on a Monday, on the first day of the month, on an even week, or on a weekday, determining a periodicity feature similarity can include identifying those historical user actions that indicate the user action occurred on a Monday, that have a feature corresponding to the first day of the month (any first day, not just a Monday), or that occurred on an even week or on a weekday. Similarly, a behavior feature similarity can be determined to identify a set of historical user actions that share a particular behavior feature with the current or recent user action. For example, if the recent user action includes visiting a news-related website, determining a behavior feature similarity would include identifying those historical user actions that have a feature indicating the user visited a news-related website.

[0094] User activity filtering can use the feature similarity determination to filter out historical user actions and retain only those historical user actions that share a particular feature (or features) with the current or recent user action. Thus, in some embodiments, each pattern-based predictor 267 can be designed (or tuned) to determine predictions based on a particular feature (or features); for example, there can be a subset of pattern-based predictors 267 used to determine predictions when the feature indicates a weekday, a weekend, or a Monday, or a particular app visit or application usage duration, etc. Such pattern-based predictors 267 can only need those historical user actions that correspond to their prediction model.

[0095] Thus, in some embodiments, for each pattern-based predictor 267, user activity filtering can be performed to determine a set of historical user actions that are relevant to that particular pattern-based predictor 267, which can include, for example, a periodicity-based predictor, a behavior-based predictor (which can include a behavior sequence or a sequence of previous user actions), a unique or unusual behavior feature (such as when a user performs an activity at an unusual time compared to similar historical user activities), or other types of feature-based predictors. (In some embodiments, the feature similarity identifier 264 can determine user activity sequence similarity by determining the Levenshtein distance between a historical sequence of user actions and a recent sequence of user actions (e.g., a sequence of the last K user actions prior to the current action (or a particular recent action))).

[0096] Additionally, in embodiments where user activity filtering can be performed such that each predictor 267 determines a subset of historical user actions that have features corresponding to its prediction criteria, for each predictor 267, for the subset of historical user actions that pass through the filter, a user action scoring can be performed on the subset of features. The user action scoring generally compares the similarity of the features of the current or recent user action and the subset of historical user actions (which can be considered a comparison of context), and scores each user action for the similarity of its features. Specifically, some embodiments score not only those features that were used to determine the subset of historical user actions, but also all (or a more substantial number of) features of the current or recent action and historical actions that are available for comparison. In some embodiments, a Boolean logic procedure is used (i.e., the features must be true or have the same pattern, and if satisfied, a difference between the particular features is determined). For example, the differences can include differences in time-related features, duration of use, sequence distance, etc. In embodiments, these differences are determined and plugged into a sigmoid function. Further, in embodiments, a similarity threshold is used, which can be predetermined, tunable, or adaptive, or can be initially set to a value based on a population of users, or can be adaptive based on empirical information learned about a particular user, and based on a number of historical observations. The similarity threshold can be used to determine whether a particular historical user action is "similar enough" to a particular current or recent user action to be considered for use in determining a prediction. In some embodiments, a vector representing the difference in similarity (or a similarity score) can be determined, for example, in cases where multiple features are evaluated for similarity.

[0097] A user action from the subset of historical user actions that is most similar (or sufficiently similar) to the particular current user action or recent user action can be selected. In some embodiments, the selection process uses a similarity threshold, such as described above, to determine those historical user actions that satisfy the similarity threshold. (Although the term "select" is used, it is contemplated that the selection is performed by a computer-related process that does not require a human to perform the selection.) The selected historical user actions comprise the "example user action" set.

[0098] A prediction of the user's next action (or future action) can be inferred based on (or from) the historical user actions in the example user action set. In embodiments, the predicted next user action (or future action) is the next user action with the highest observation count (i.e., the next user action that is best predicted based on the example user action set). Those historical user actions in the example user action set that are in line with the prediction comprise the "prediction support set." In some embodiments, a prediction probability corresponding to the prediction can be determined, for example, based on a proportion of the size of the prediction support set to the total number of observations (historical user actions in the subset determined by the user action filtering). Also, in some embodiments, the prediction can further include additional information related to the predicted user action(s), such as activity features characterizing the predicted action(s). By way of example and not limitation, if the predicted action is that the user will send an email to a work colleague, the additional activity features can indicate the particular email recipient, the subject of the email, the approximate time that the user will access his email program to compose the email, or other information related to the predicted action. The relevant information can also be determined based on the activity features observed by the prediction support set.

[0099] Some embodiments also determine a prediction significance that can be determined based on a confidence interval (e.g., a binomial confidence interval) or other appropriate statistical measure. Still further, in some embodiments, the prediction confidence can be based on the prediction probability and the prediction significance (e.g., the product of the prediction probability and the prediction significance). Thus, some embodiments of the activity pattern determiner 266 provide each of the pattern-based predictors 267 with the predicted next (or future) user action or series of next (or future) actions.

[0100] Some embodiments determine a particular prediction from one or more pattern-based predictors 267. In embodiments, an ensemble process is utilized in which one or more pattern-based predictors 267 vote, and a selection is determined based on the ensemble-member predictions. Further, in some embodiments, the ensemble-member predictors can be weighted based on learned information about user actions. In one embodiment, once each pattern-based predictor 267 has provided a prediction, the prediction with the highest corresponding confidence is determined to be the next (or future) predicted user action, and can be considered a pattern-based (or history-based) prediction. In some embodiments, the output of the activity pattern determiner 266 can be stored as an inferred user routine 248 in the user profile 240, and in some embodiments can be provided to an activity pattern consumer 270.

[0101] With continued reference to Figure 2 , the example system 200 includes one or more activity pattern consumers 270, which include applications or services that consume activity pattern information to provide an improved user experience. Examples of activity pattern consumers 270 can include, but are not limited to, content personalization services, user intent inference services, automatic speech recognition services, device power management services, and semantic understanding services.

[0102] In particular, a first example activity pattern consumer 270 includes a content personalization service. In one embodiment, a content personalization engine 271 is provided to facilitate providing a personalized user experience. Thus, the content personalization engine 271 can be considered one example of an application or service (or collection of applications or services) that can consume information about user activity patterns, which can include predictions of future user actions as determined by implementations of the present disclosure.

[0103] At a high level, the example content personalization engine 271 is responsible for generating and providing various aspects of a personalized user experience, such as personalized content or customized content delivered to the user. The content can be provided to the user as a personalized notification (such as described in connection with the presentation component 220), can be provided to an application or service of the user (such as a calendar or scheduling application), or can be provided as part of an API that can be consumed by yet another application or service. In one embodiment, the personalized content includes suggesting to the user to perform a relevant activity at an appropriate time before the user manually performs the activity. For example, if the activity pattern indicates that the user visits their bank website and enters financial information into an Excel file near the beginning of the month, the user can be provided with a suggestion asking the user "Would you like to visit your bank website?" Upon a positive response, a browser instance can be provided that automatically navigates to the user's bank website. Further, the specific Excel file can be automatically opened when the user acknowledges the suggestion or when the user navigates to the bank website. Still further, the user can be provided with this content (here, a notification suggestion) at a convenient time, such as when the user is at his home, in the evening, near the beginning of the month, etc.

[0104] In one embodiment, the personalized content can include a notification, which includes a reminder, a recommendation, a suggestion, a request, a communication related data (e.g., an email, an instant message, or a call notification), information that is in some way relevant to the user, or other content provided to the user in a personalized manner. For example, the content can be provided at a time that the user is most likely to want to receive it, such as providing a work related notification at a time that is indicated by the user's pattern to be when the user is about to begin a work related activity, rather than at a time that can be considered, ignored, or be an annoyance.

[0105] In another example of providing personalized content, in the case where the activity pattern indicates that the user typically checks for coupons online when the user is shopping at a particular store, the user can be automatically provided with an online coupon when the user data indicates that the user has entered the store or is likely to go to the store (which can be determined based on a predicted future activity). Further, the particular coupon that is provided can be for items that the user typically purchases (or for similar items from a competitor) based on the user activity pattern information, which can include the user's purchasing habits.

[0106] In some embodiments, the content personalization engine 271 customizes content for a user to provide a personalized user experience. For example, the content personalization engine 271 can generate a personalized notification to be presented to the user, which can be provided to the presentation component 220. Alternatively, in other embodiments, the content personalization engine 271 generates notification content and makes it available to the presentation component 220, which determines when and how (i.e., in what format) to present the notification based on user data or user activity pattern information. For example, if the user activity pattern indicates that the user is likely driving to work at the time that a particular notification is to be presented, it can be appropriate to provide the notification in an audio format, thereby personalizing it to the user's context. In some embodiments, other services or applications operating in conjunction with the presentation component 220 determine or facilitate the determination of when and how to present personalized content. The personalized content can be stored in the user profile 240, such as in the personalized content component 249.

[0107] Some embodiments of the content personalization engine 271 evaluate user content to determine how to provide the content in an appropriate manner. For example, content determined to be work-related can be withheld from presentation to the user until the user activity pattern indicates that the user is likely engaged in work-related activity.

[0108] The second example activity pattern consumer 270 includes a user intent inference service. The intent of a user can be inferred based on user activity pattern information. In particular, every time a user interacts with a computing device, such as when the user engages in a user activity, it can be assumed that the user has an intent. In one embodiment, the user activity pattern can be analyzed along with sensor data collected by the user device regarding a particular user interaction, such as a user request or recommendation. The intent of the user can be inferred based on determining a likely intent that is consistent with the user activity pattern. For example, during the user's lunch hour, the user can perform an online search with the intent to find information about the user's favorite band, R.E.M., and not information about rapid eye movement (REM) sleep. User activity pattern indicating that the user typically browses music entertainment related websites during his lunch hour and / or other activity pattern information about the user indicating that the user recently listened to music by R.E.M. can be used to resolve the user's intent for the query, i.e., search results related to the music group R.E.M. and not results related to rapid eye movement. As another example, assume that while driving home, the user issues a voice request to the user device to "call Pat." However, the user has multiple "Pats" on his contact list, e.g., the user's brother is named Pat and the user has several friends named Pat. Based on an inferred pattern of user activity indicating that the user typically calls his family while driving home, it can be determined that the user's intent is to call his brother Pat, as this intent is consistent with the activity pattern.

[0109] A third example activity pattern consumer 270 includes an automatic speech recognition (ASR) service. In one embodiment of the disclosure, an ASR service is provided that has improved speech recognition and understanding. The improved ASR service can use information from inferred user activity patterns to more accurately parse, disambiguate, or understand user speech. In particular, a user's speech can be parsed to conform to user activity pattern information. For example, assume that for a particular user, the user activity patterns indicate that at lunchtime the user typically browses news-related websites. Assume that at around 12:00 noon one day, the user says a request to go to CNN.com (e.g., "go to see en en dot com") to a personal assistant service such as Siri®. However, the user's request sounds similar in pronunciation to going to CMM.com or going to CMN.com. Moreover, if there is background noise, the user's spoken request can be more difficult to recognize. However, information from inferred user activity patterns that indicates that the user typically browses news-related websites can be used to more accurately parse and understand the user's speech. Here, the improved ASR service can determine that the user likely said "CNN.com" rather than CMN.com or CMM.com because CMN.com and CMM.com are not news-related websites; CNN.com is a news-related website; and navigating to CNN.com at around 12:00 noon conforms to the user activity pattern of browsing news-related websites at lunchtime.

[0110] A fourth example activity pattern consumer 270 includes a device power management service. For example, in one embodiment of the disclosure, a device power management service is provided that regulates power consumption of a user's device based on user activity pattern information. In particular, based on a pattern of when the user typically charges his mobile device, when it is determined that the time until the typical charging time is long enough such that, based on an average consumption of battery energy, the device is expected to run out of battery power before the user typically charges the device, the power management service can implement power saving measures to reduce battery consumption. For example, the power management service can automatically switch the device to a power saving mode. Similarly, if it is determined that the user is located or will be located at a location where the user previously charged his user device (which can be determined based on location context information associated with user activity related to charging the user device), the user can be provided with a notification recommending that the user charge his user device.

[0111] The fifth example activity pattern consumer 270 includes a communication management service. Specifically, network bandwidth and other communication-related resources can be more efficiently utilized based on information derived from user activity patterns. For example, assume that a user has a pattern of watching a high-definition movie (e.g., a 4k movie) streamed or downloaded on Friday nights. One embodiment of the communication management service can cache or buffer the movie prior to the movie being watched so that it can be watched without interruption. (This is particularly useful for users that share network bandwidth with other users that are simultaneously watching a streamed movie, which reduces the available network bandwidth.) In some embodiments, the particular movie (or several movies that can be watched by the user) can be determined from a user's watch list or can be inferred based on information derived from monitored user activity such as user tastes, Internet searches, movies watched by the user's social media friends, etc. As previously described, some embodiments can use contextual information such as the user's calendar information to determine the likelihood that the user will follow the pattern. Thus, if the user's calendar indicates that an event is scheduled for the following Friday night, then it can be determined that the movie does not need to be downloaded prior to the following Friday night because the user is unlikely to watch the movie. Further, in some embodiments, where the user has a data cap or where network speed is capped or otherwise limited, the movie can be downloaded at an earlier time so that the movie is available for viewing on Friday night. Still further, in some embodiments, the user can be prompted whether the user is interested in watching the particular movie prior to the movie being downloaded.

[0112] Other examples of activity pattern consumers 270 can include, but are not limited to: (a) a recommendation service that suggests new content to a user based on user patterns and context information. For example, a user activity pattern indicates that the user listens to music every Friday night. Context information indicates that the user prefers certain bands or music genres. Thus, on a given Friday night, the user is provided with a recommendation to listen to a new artist with a similar style to the user's tastes, (b) a user has an activity pattern of going to music concerts to see bands that perform songs that the user enjoys. A personal assistant application service monitors local concerts and determines that a band that performs songs that the user listens to is coming to town. The personal assistant application automatically purchases a ticket for the user when the ticket first becomes available. Alternatively, the personal assistant service checks the user's calendar to determine that the user is free on the date of the concert, then prompts the user, notifies the user about the concert, and in some embodiments, asks the user if the personal assistant service can purchase a ticket, (c) a user has an activity pattern of watching online movies on Friday nights. A personal assistant service determines that the user reads books of certain genres based on information about the user's book purchases and / or e-reader activity. Based on the user's tastes in books, the user can be recommended movies that the user might enjoy. The recommended movies can be automatically downloaded in a bandwidth-efficient manner before Friday night.

[0113] The example system 200 also includes a presentation component 220, which is generally responsible for presenting content and related information to users, such as personalized content from the content personalization engine 271 or content from other activity pattern consumers 270. The presentation component 220 can include one or more applications or services on a user device, across multiple user devices, or in the cloud. For example, in one embodiment, the presentation component 220 manages the presentation of content to a user across multiple user devices associated with the user. Based on content logic, device characteristics, associated hubs, inferred logical locations of the user, and / or other user data, the presentation component 220 can determine on which user device(s) to present content and the context of that presentation, such as how to present it (or in what format and how much content, which can depend on the user device or context), when to present it, and so on. In particular, in some embodiments, the presentation component 220 applies content logic to device characteristics, associated logical hubs, inferred logical locations, or sensed user data to determine various aspects of content presentation.

[0114] In some embodiments, the presentation component 220 generates user interface features associated with personalized content. These features can include interface elements (such as graphical buttons, sliders, menus, audio cues, warnings, alerts, vibrations, pop-ups, notification or status bar items, in-app notifications, or other similar features for interfacing with a user), queries, and prompts.

[0115] As previously described, in some embodiments, the personal assistant service or application operating in conjunction with the presentation component 220 determines when and how to present content (e.g., present when it is determined that the user is at a particular logical location). In such embodiments, content that includes content logic can be understood as a recommendation to the presentation component 220 (and / or the personal assistant service or application) for when and how to present the notification, which can be overridden by the personal assistant app or the presentation component 220.

[0116] The example system 200 also includes a storage 225. The storage 225 generally stores information including data, computer instructions (e.g., software program instructions, routines, or services), logic, profiles, and / or models used in the embodiments described herein. In embodiments, the storage 225 includes a data storage (or computer data storage). Further, although depicted as a single data storage component, the storage 225 can be embodied as one or more data storages or can be in the cloud.

[0117] As shown in the example system 200, the storage 225 includes the activity pattern inference logic 230 and the user profile 240 as previously described. In Figure 2 One example embodiment of the user profile 240 is illustratively provided. The example user profile 240 includes information associated with a particular user, such as user activity information 242, information about user accounts and devices 244, user preferences 246, inferred user routines 248, and personalized content 249. The information stored in the user profile 240 can be used by the activity pattern inference engine 260 or other components of the example system 200.

[0118] As previously described, the user activity information 242 generally includes user information about user actions or activity events, related context information, activity characteristics, or other information determined via the user activity monitor 280, and can include historical or current user activity information. The user accounts and devices 244 generally include information about user devices accessed, used, or otherwise associated with the user, and / or information about user accounts associated with the user, for example, such as online or cloud-based accounts such as network passports (e.g., email, social media), other accounts such as entertainment or gaming related accounts (e.g., Xbox Live, Netflix, online gaming subscription accounts, etc.), user data related to these accounts such as user emails, texts, instant messages, calls, other communications, and other content; social network accounts and data such as news feeds; online activity; and calendars, appointments, application data, other user accounts, etc. Some embodiments of the user accounts and devices 244 can store information across one or more databases, knowledge graphs, or data structures. As previously described, the information stored in the user accounts and devices 244 can be determined from the user data collection component 210 or the user activity monitor 280, including one of its subcomponents.

[0119] User preferences 246 generally include user settings or preferences associated with user activity monitoring. By way of example and not limitation, such settings can include, as described herein, user preferences regarding specific activities (and related information) that the user desires to be explicitly monitored or not monitored, or categories of activities to be monitored or not monitored; crowd-sourcing preferences such as whether to use crowd-sourced information or not; or whether the user’s activity pattern information can be shared as crowd-sourced data; preferences regarding which activity pattern consumers can consume the user’s activity pattern information; and threshold and / or notification preferences. As previously described, inferred user routines 248 can include one or more user patterns determined by the activity pattern determiner 266, and can also include confidence scores associated with the patterns and / or information related to the activity patterns such as contextual or semantic information. Personalized content 249 includes personalized content such as pending or scheduled notifications determined from the content personalization engine 271.

[0120] Referring now to Figure 3 Aspects of an example system for determining user activity patterns are provided and generally referred to as system 300. The example system 300 can determine user activity patterns based on browsing and application activity from across multiple devices. As shown, the example system 300 includes one or more client devices 302, an inference system 350, and one or more activity pattern consumers 370. In various embodiments, the client devices 302 can be embodied as user devices such as Figure 1The described user devices 102a. The client devices 302 include an app activity logging pipeline 383 and a browsing activity logging pipeline 385. These logging pipelines can be embodied as client-side applications or services running on each user device associated with a user, and in some embodiments can run in conjunction with or inside of an application, such as running inside a browser or as a browser plugin or extension. Generally speaking, the app activity logging pipeline 383 manages the logging of a user's application (or app) activity, such as app downloads, launches, accesses, usage (which can include duration), file accesses via the app, and in-app user activity (which can include app content). Generally speaking, the browsing activity logging pipeline 385 manages the logging of a user's browsing activity, such as websites visited, social media activity (which can include browsing-type activity performed via a particular browser or like app, app, app, app, and the like. In some embodiments, the app activity logging pipeline 383 and the browsing activity logging pipeline 385 further include functionality described in connection with the app activity logging pipeline 283 and the browsing activity logging pipeline 285 of the system 200 in Figure 2

[0121] As shown in the system 300, the inference system 350 includes an activity pattern inference engine 360 that receives app activity and browsing activity from one or more client devices 302 and uses this information to infer activity patterns. The activity pattern inference engine 360 can be embodied as an implementation of the activity pattern inference engine 260 described. Further, some embodiments of the inference system 350 can also include other inference engines (not shown). For example, other inference engines can include components for determining venue visit inferences (i.e., inferring future venue visits or patterns of venue visits); location inferences (i.e., inferring a user's location when explicit location information is not available); or a shadow calendar, which can include inferences about a user's availability, explicit and inferred scheduled events. Figure 2

[0122] The inference system 350 can provide inferred user activity pattern information (including predicted future actions) to an activity pattern consumer 370. In some instances, the activity pattern consumer 370 can be embodied as an implementation of the activity pattern consumer 270 described in connection with Figure 2 ​​The described activity pattern consumer 270. Another example activity pattern consumer 370 of the system 300 includes a browser A 372 (or a service associated with browser A) that navigates to a website according to a predicted user activity pattern; for example, automatically navigating to a user's bank website around the beginning of the month, where the user has a pattern of visiting the website at the beginning of the month. Yet another example activity pattern consumer 370 includes an app B 373 that can launch and / or load files, or perform certain functions (e.g., play a song, compose an email, etc.) according to a predicted user activity pattern.

[0123] Referring now to Figure 4 , a flow diagram is provided that illustrates one example method 400 for inferring user activity patterns. Each block or step of the method 400 and other methods described herein comprises a computational process that can be executed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The method can also be embodied as computer-usable instructions stored on a computer storage media. The methods can be provided by a standalone application, a service, or a hosted service (standalone or in combination with other hosted services) or a plug-in to another product, etc. Thus, the method 400 can be executed by one or more computing devices such as a smartphone or other user device, a server, or by a distributed computing platform such as in the cloud. Activity patterns can be inferred by analyzing signal data (or user data) collected from one or more user devices associated with a user.

[0124] At step 410, a set of user devices associated with a user is identified. Embodiments of step 410 can determine the set of user devices based on monitoring user device related information in user data. In some embodiments, the set of user devices identified in step 410 includes one user device or multiple user devices, such as the user devices 102a-102n described in connection with Figure 1

[0125] In one embodiment, information about user devices associated with a user can be determined from user data provided via the user data collection component 210, such as described in connection with Figure 2 ​information about the user device, such as device hardware, software such as an operating system (OS), network-related characteristics, user accounts accessed via the device, and similar characteristics. In one embodiment, a detected user device, such as user devices 102a-102n, can be polled, interrogated, or otherwise analyzed to determine information about the device. In some implementations, this information can be used to determine a label or identification of the device (e.g., a device ID) such that user interactions with one device can be recognized from user interactions on another device. In some embodiments of step 410, a user device can be identified based on user-provided information, such as a user's declaration or registration of a device, for example, by logging into an account via the user device, installing an application on the device, connecting to an online service that interrogates the device, or otherwise providing information about the device to an application or service.

[0126] At step 420, a set of user devices is monitored to detect user activity events. Embodiments of step 420 monitor user data associated with the set of user devices identified in step 410 to identify or detect user activity events (sometimes referred to herein as user actions). In some instances, an activity event can include a series or sequence of user interactions with one or more user devices.

[0127] Some embodiments of step 420 can include monitoring sensor data using one or more sensors associated with the set of user devices. As described in connection with user activity detector 282, some embodiments of step 420 can use activity event logic to detect user activity events. Further, some implementations of step 420 can be performed using user activity detector components such as those described in system 200. Figure 2 Additional details of embodiments of step 420 are provided in connection with user activity detector 282 in Figure 2

[0128] At step 430, a set of activity characteristics associated with the activity event is determined. When a user activity event is detected or otherwise identified in step 420, embodiments of step 430 determine a set of activity characteristics associated with the detected activity event. Some embodiments of step 430 determine the set of activity characteristics based at least in part on sensor data provided by one or more sensors associated with the set of user devices, including data that can be derived from sensor data such as interpretive data. In some embodiments, sensor data can be analyzed via​Figure 2 The user data collection component described in the Background section is provided. In particular, user data is received and analyzed to determine a set of one or more features associated with user activity, the user data relating to detected activity events, can be determined at least in part from sensor data, and can include interpretive data, contextual data, and / or semantic information relating to the detected activity events.

[0129] As previously described in connection with the user activity monitor 280 and activity feature determiner 286 of Figure 2 The activity-related features (or variables) associated with user activity can be used to identify user activity patterns, as described previously in connection with the user activity monitor 280 and activity feature determiner 286 of Figure 2 Additional details of determining contextual information relating to detected activity events are described below in connection with the contextual information extractor 284 of Figure 2 Further, in some embodiments, the contextual information can include semantic information determined from a semantic analysis performed on the detected activity events and / or one or more activity features associated with the activity events. For example, while a user activity feature can indicate a particular website visited by a user, semantic analysis can determine a category, related websites, a theme or topic, or other entities associated with the website or user activity. From the semantic analysis, additional user activity-related features semantically related to the user activity can be determined and used to identify user activity patterns. Additional details of determining activity features including semantic information relating to detected activity events are described in connection with the semantic information analyzer 262 of

[0130] Examples of activity-related features can include, but are not limited to, location-related features such as location of one or more user devices during, before, and / or after a user activity, venue-related information associated with a location, or other location-related information; time-related features such as time of day or days, day of the week or month of a user activity, or duration of the activity, or related duration information such as how long a user uses an application associated with an activity; content-related features such as a user's online activity (e.g., searches, websites browsed, types or categories of websites, purchases, social network activity, communications sent or received including social media posts, which can include online activity that occurs before or after a detected activity event); other features that can be detected concurrently with or near the time of a user activity; or any other features that can be detected or sensed and used to determine a pattern of user activity, including features that can characterize aspects of an activity, nature or type of an activity, and / or semantic features that constitute user interaction of an activity. Features can also include information about one or more users using the device; and other contextual features such as user device-related features, usage-related features; or other information about a user. By way of example and not limitation, user device-related features can include information about a device type (e.g., desktop, tablet, mobile phone, fitness tracker, heart rate monitor, etc.), hardware attributes or profile, OS or firmware attributes, device ID or model number; network-related information (e.g., mac address, network name, IP address, domain, workgroup, information about other devices detected on a local network, router information, proxy or VPN information, other network connection information, etc.); location / motion / position-related information about a user device; power information such as battery level, time connected / disconnected from a charger, and user access / touch information; usage-related features can include information about one or more files accessed, app usage (which can also include application data, in-app usage, concurrently running applications), network usage information, one or more user accounts accessed or otherwise used (such as one or more device accounts, one or more OS-level accounts, or one or more online / cloud-service-related accounts, such as account or network passport, one or more online storage accounts, email, calendar, or social network accounts, etc.); other information that identifies a user can include login passwords, biometric data that can be provided by a fitness tracker or biometric scanner, and / or characteristics of one or more users using a device, which can be useful to distinguish between users on a device that is shared by more than one user.

[0131] In some embodiments of step 430, the user activity event logic (described in connection with user activity detector 282) can be utilized to identify particular features associated with the detected user activity event. Some implementations of step 430 can be performed using an activity feature determiner component (such as described in system 200 of Figure 2 In some embodiments of step 430, the user activity event logic (described in connection with user activity detector 282) can be utilized to identify particular features associated with the detected user activity event. Some implementations of step 430 can be performed using an activity feature determiner component (such as described in system 200 of Figure 2 Additional details of embodiments of step 430 are provided in connection with activity feature determiner component 286 in

[0132] At step 440, a record of the detected activity event and the activity features associated with the event are stored in an activity event data store. The activity event data store can include a plurality of records regarding activity events, where each record can include information regarding a particular activity event, including one or more activity features associated with the activity event. In some embodiments, the plurality of records in the activity event data store include records of other activity events determined according to steps 410-430 of method 400. In some instances, as previously described, some of the other records can include information regarding activity events (including associated activity features) derived from other users determined to be similar to the particular user. In embodiments, the activity event data store includes a user profile, and can be stored in a user activity information component (such as user activity information component 242 of user profile 240 described in connection with Figure 2

[0133] At step 450, an activity pattern inference engine is used to identify activity patterns. In embodiments of step 450, an activity pattern can be detected based on an analysis of at least a portion of the plurality of activity event records to identify a set of activity events having similar activity features. Accordingly, embodiments of step 450 can include determining a set of user activity events having common features, which can be determined using a feature similarity identifier component (such as feature similarity identifier 264 described in system 200 of Figure 2

[0134] In embodiments, as previously described in connection with activity pattern determiner 266, the set of activity events having similar activity features can include two or more activity events that form the basis of a pattern. In some embodiments, as previously described, the similar activity features can be the same, can be substantially similar, or can be sufficiently similar according to a similarity threshold in two or more of the activity events. Further, some embodiments can determine a corresponding confidence score associated with the activity pattern. In some instances, the confidence score can indicate a likelihood that the user will behave according to the pattern, and can be used to determine whether to provide the activity pattern to an activity pattern consumer service (such as activity pattern consumer service 268 described in connection with Figure 2 ​​The described activity pattern consumer service) provides activity pattern information. In some embodiments, one or more pattern-based predictors, such as described in connection with Figure 2 The described pattern-based predictors.

[0135] Step 450 can be performed by an activity pattern inference engine, such as the activity pattern inference engine 260 or one of its sub-components described in connection with Figure 2 System 200. Additional details of embodiments of step 450 are provided in connection with Figure 2 Activity pattern inference engine 260.

[0136] At step 460, the activity pattern is stored in an activity pattern data store. In embodiments, the activity pattern data store comprises a user profile, and can be stored in an inferred user routine component of the user profile, such as the inferred user routine 248 of the user profile 240 described in connection with Figure 2 The activity pattern can be accessed by one or more computing applications that provide an enhanced or improved computing experience for the user, such as providing personalized services (e.g., personalized content or custom delivery of content to the user, as described herein), improving speech recognition, more efficiently consuming bandwidth or power, or improving the user's semantic understanding, which can be used to perform disambiguation or other aspects of understanding input from the user.

[0137] In some embodiments, once an activity pattern is determined, it can be used to determine that a likely future activity event will occur at a threshold time in the future from a current time (e.g., in 3 hours, 12 hours, one day, five days, two weeks, etc. from the current time, or within these times) by analyzing the activity pattern and / or current or recent information about the user's activities. The likely future activity event can be associated with a location, time, condition, and / or circumstance (e.g., the future activity can occur after a particular type of event, such as after the user performs another activity) and other contextual data that can be used to facilitate an improved user experience. Further, as previously described, in some embodiments, the user activity pattern or inferred user intent or predictions of future activities determined therefrom can be provided to one or more applications and services that consume this information and provide an improved user experience, such as the activity pattern consumers 270, 271, 372, or 373 described in connection with Figure 3 and Figure 2 For example, in embodiments, a modified user experience is provided that can include presenting (or hiding, or delaying) content to the user in a personalized manner, such as in the manner described in connection with the content personalization engine 271 of system 200 Figure 2

[0138] ​In some embodiments, the method 400 can be performed with a cloud system, such as the cloud system described above, and / or cloud services to provide improved or enhanced user experiences, such as personalized content, to a plurality of services that can be running on many different user devices. As such, the system 200 can save significant processing, bandwidth, storage, and computing resources by centralizing certain functions. For example, the user data collection component 210( Figure 5 ) can accumulate user data and interpretive data for a plurality of users or user devices so that each user device does not need separate and redundant data collection and storage. Additionally, the processing and storage of user profile data can be made more secure by disassociating from user devices that are closely associated with the user.

[0139] Referring now to Figure 2 , a flowchart is provided that illustrates one example method 500 for determining a likely future user action based on a user activity pattern. At step 510, an inferred activity pattern for a user is accessed. The inferred activity pattern can be determined as described previously with reference to Figure 3 、 Figure 4 or Figure 2 . In embodiments, the activity pattern can be accessed from an activity pattern data store, such as the inferred user routine 248 of the user profile 240 described in connection with Figure 2 . As mentioned, a user can be associated with multiple activity patterns. A particular inferred pattern can be associated with a confidence level or confidence score that indicates the likelihood that the pattern will be followed (i.e., how likely it is that the prediction based on the pattern is correct).

[0140] At step 520, a likely future activity event is determined based on the activity pattern. Embodiments of step 530 predict a future activity event, which can include one or a series (or sequence) of future user interactions that are likely to occur or that the user is likely to desire to occur. In some embodiments, step 520 can also determine a confidence score associated with the determined likely future activity, the confidence score indicating the likelihood that the activity event will occur or be desired to occur by the user. In some embodiments, the likely future activity can include a context, which can be defined by a location, a time, a user behavior, or other conditions, and can be determined based at least in part on sensor data.

[0141] For example, in some instances, as previously described, a periodic context or a behavioral context is considered to determine a possible future activity event, which can be defined according to periodic characteristics or behavioral characteristics. Thus, a periodic context defines when an activity event associated with an activity pattern can occur. For example, if a pattern indicates that a user performs a particular activity every Monday, Wednesday, and Friday, a future activity event can be determined for any Monday, Wednesday, or Friday. Similarly, a behavioral context defines contextual characteristics that are present when an activity pattern can occur. For example, if a pattern indicates that a user performs a particular activity when visiting a certain location, after another activity, or another condition manifests (e.g., a user checks for online coupons every time the user visits a particular store). A behavioral context can also be defined negatively to identify contextual characteristics that define exceptions to a periodic context. For example, a user can not perform certain activities during a holiday or when the user is not in a central or frequently visited location, such as when the user is traveling or on vacation. In some embodiments that determine a confidence score for a possible future activity, the confidence score can be further determined based on the context.

[0142] In some embodiments, a context can be determined based in part on a current activity event or a recently occurring activity event. Specifically, in some embodiments of step 520, a possible future activity can also be determined based on a current user activity event or a recently occurring user activity event. Some embodiments of step 520 can monitor one or more user devices associated with a user to detect a current user activity event or a recently occurring user activity event. The detected activity event can be used to determine a particular user activity pattern and / or the next activity event or future activity event in the pattern related to a context (e.g., a current situation of the user). Thus, the recency of the detected activity event in such embodiments of step 520 can be based on a length of time such that the context (e.g., the current situation of the user) is still relevant to the activity pattern. Accordingly, the recency can vary according to the particular activity pattern.

[0143] In some embodiments, a current activity event or a recently occurring activity event is determined based on sensor data from one or more user devices. For example, some embodiments of step 520 can include monitoring sensor data using one or more sensors associated with a set of user devices. Some embodiments of step 520 can be performed as described in step 420 of method 400, or can be performed using a user activity detector 282 as described in system 200. Figure 2 In some embodiments, a context is determined based on a current activity event or a recently occurring activity event. For example, in some embodiments of step 520, a context can be determined based on a current activity event or a recently occurring activity event. In some embodiments, a context can be determined based on a current activity event or a recently occurring activity event. Specifically, in some embodiments of step 520, a possible future activity can also be determined based on a current user activity event or a recently occurring user activity event. Some embodiments of step 520 can monitor one or more user devices associated with a user to detect a current user activity event or a recently occurring user activity event. The detected activity event can be used to determine a particular user activity pattern and / or the next activity event or future activity event in the pattern related to a context (e.g., a current situation of the user). Thus, the recency of the detected activity event in such embodiments of step 520 can be based on a length of time such that the context (e.g., the current situation of the user) is still relevant to the activity pattern. Accordingly, the recency can vary according to the particular activity pattern.

[0144] At step 530, based on the determined possible future activity, an enhanced user experience can be provided to the user. Embodiments of step 530 provide an enhanced user experience that can take the form of a customized, modified, or personalized service for the user determined based at least in part on the possible future activity determined in step 520, and can also be determined based on the associated confidence score. Embodiments of step 530 can include providing the enhanced user experience by Figure 6 The one or more activity pattern consumers 270 described provide a service to the user. For example, in embodiments, the enhanced user experience includes one of a recommendation, notification, request, or suggestion related to the possible future activity. In one embodiment, it includes automatically performing an action related to the possible future activity at a time or location that coincides with the inferred activity pattern. For example, in the case of a user who typically browses her bank account around the beginning of the month and copies financial information from a bank website into a spreadsheet, embodiments of step 530 can automatically perform the following: launch a browser, navigate to the bank website, and load the spreadsheet file into the spreadsheet application. In some embodiments, based on context, the automatic operation can be performed at an appropriate time, such as when the user is at home using the computing device (user device) that she typically uses to access the bank website, or when the user is not driving and thus unable to benefit from the personalized user experience. Other examples of enhanced user experiences are described herein, and include, by way of example and not limitation, personalized services (e.g., providing personalized content or customizing content delivered to the user as described herein), improved speech recognition, more efficient bandwidth or power consumption, or improved semantic understanding of the user, which can be used to perform disambiguation or other aspects of understanding input from the user.

[0145] Now turning to Figure 9 a flowchart is provided that illustrates one example method 600 for performing disambiguation to determine a user intent.

[0146] At step 610, an indication of a user interaction with a user device is received. The indication of the user interaction can be based at least in part on sensor data provided by one or more sensors associated with the user device. The user interaction can be associated with a particular user activity that is associated with a user intent (user intent), such as, for example, conducting a search query, requesting to initiate a call (or email, instant message, other communication) with someone, or issuing a voice command.

[0147] At step 620, a set of possible user intents associated with the user interaction is determined. Embodiments of step 620 identify a set of possible user intents that necessitate disambiguation in order to determine the actual intent of the user. For example, where the user interaction is related to a voice command, the set of intents can correspond to different computer interpretations of the user's utterance (e.g., did the user say "CNN.com" and therefore intend to navigate to a news website, or did the user say "CMN.com" and intend to navigate to a Consumer Media Network website). Similarly, where the user is performing a search query, the user's intent is to search for a result (e.g., where the user's search term is R.E.M., does the user intend to search for the music band or for REM sleep).

[0148] At step 630, an inferred user activity pattern is accessed. Embodiments of step 630 can access an inferred user activity pattern as described in step 510 of method 500. Further, a particular inferred user activity pattern accessed can be determined based on a context, such as the context described in step 520. In particular, the context can be determined from a location, a time, a user behavior or activity, and / or the user interaction indicated in step 610 that is relevant to the user's current situation. In some embodiments of step 630, multiple user activity patterns are accessed. As described herein, each user activity pattern can include an associated confidence score that reflects a likelihood that the pattern will be followed.

[0149] At step 640, a possible future activity event is determined based on the activity pattern determined in step 630. Some embodiments of step 640 further determine the possible future activity event based on the context described in connection with step 630. Some embodiments of step 640 can be performed as described in step 520 of method 500.

[0150] At step 650, the user intent that best fits the possible future activity is selected. Embodiments of step 650 determine the user intent from the set of possible user intents determined in step 620 that best fits the possible future activity. Specifically, the fit can be determined by performing an analysis to determine whether performing an operation according to each intent in the set of intents would result in an activity having the same activity characteristics as the activity pattern accessed in step 630. For example, assume that the user has a habit of browsing for a call to a family member while driving home from work, the user has a brother and several friends named Pat, and issues a voice command to "call Pat." The context indicates that the user is driving, has just left his office, and it is the end of the work day. Thus, the set of intents includes an intent to call each of the user's contacts named Pat. But only one of the intents - the intent to call his brother - fits the activity pattern of the user calling a family member while driving home from work. Accordingly, this intent will be selected. In this way, disambiguation is performed.

[0151] In some embodiments, the set of possible user intents determined in step 620 is ranked in step 650 according to the fit to the possible future activity. Specifically, in some embodiments of method 600, more than one user activity pattern is accessed, where each pattern has an associated confidence score, and the set of possible user intents can be ranked according to the likelihood of being the correct user intent. Then, the highest ranked intent can be selected in step 650 and used in step 660.

[0152] At step 660, the activity associated with the user interaction is performed based on the selected intent. Embodiments of step 660 perform the user activity associated with the user intent selected in step 650. Thus, for example, continuing the example described in step 650, the activity includes calling the user's brother Pat.

[0153] Accordingly, we have described various aspects of technology directed to systems and methods for inferring user intent and predicting future user activity associated with a user's computing device, which can be used to provide an enhanced user experience. It should be appreciated that various features, subcombinations, and modifications of the embodiments described herein can be taken and employed within other embodiments without departing from the scope of the disclosure. Moreover, the order and sequence of some of the steps demonstrated in example methods 400, 500, and 600 do not necessarily require any particular order or sequence, and in fact, these steps can occur in various different orders within embodiments thereof. Such variations and their combinations are considered to be within the scope of embodiments of the present disclosure.

[0154] Having described various implementations, an example computing environment suitable for implementing embodiments of the disclosure is now described. Referring to Figure 7 An example computing device is provided and generally referred to as computing device 900. Computing device 900 is merely one example of a suitable computing environment and is not intended to limit the scope or functionality of embodiments of the disclosure. Computing device 900 should also not be interpreted as having any dependency or requirement relating to any one or combination of the illustrated components.

[0155] Embodiments of the disclosure can be described in the general context of computer code or machine-useable instructions, including computer-usable instructions or computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal data assistant, a smart phone, a tablet, or other handheld device. Generally, program modules refer to code, routines, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. Embodiments of the disclosure can be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general- purpose computers, more specialty computing devices, etc. Embodiments of the disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0156] Now turning to Figure 2, a flowchart is provided that illustrates one example method 700 for performing speech recognition to determine an action that a user desires to perform. The user action can be a verbal request or command (e.g., "Cortana, go to the CNN website" or "Cortana, text Ed that I am running late"), a question or query (e.g., "Cortana, what restaurants are near the theater?" or "Cortana, how busy is traffic now?"), or other manner corresponding to a desire or intent of the user. At a high level, embodiments of the method 700 can recognize and understand the user's utterance associated with an action. Specifically, in cases where there can be ambiguity based on the user's particular utterance, embodiments of the method 700 can resolve the ambiguity in a manner that is consistent with the user's activity patterns, then identify an action corresponding to the currently understood utterance, and perform the action. Thus, using the above examples, embodiments of the method 700 can determine that the user can have said "CNN.com" instead of "CMN.com," where the user has a pattern of watching news-related content when speaking. Based on this understanding, a personal assistant application or service (or browser) can navigate to the CNN.com website. Similarly, embodiments of the method 700 can determine that the user can have said "text Ed" instead of "text Ted," where the user has an activity pattern indicating that the user can be meeting a friend Ed. Similarly, some embodiments of the method 700 determine an action that a user desires to perform based on speech recognition of a personalized expression or set of words. For example, a user saying "call mom" can be determined to call a particular contact of the user that the user identifies as "mom," which can be determined based on contextual information. Further, such nicknames or personalized terms and expressions can be identified and learned from monitored activities, including contextual information extracted about activity events (such as described in connection with the contextual information extractor 284 or user activity monitor 280 of FIG. 2). Figure 2

[0157] Thus, at step 710, information associated with the user's utterance is received. The information can be received based at least in part on sensor data provided by one or more sensors associated with the user device. The utterance can correspond to an action that the user wishes to perform via the user device, such as, for example, conducting a search query, requesting a call (or email, instant message, other communication) to be initiated with someone, or issuing a voice command.

[0158] ​At step 720, a set of possible words corresponding to the utterance is determined. Embodiments of step 720 identify a set of words that can have been spoken by the user. In some instances, the set can include only a single word (or each member of the set of words can include only a single word), a phrase, a series of words, or a series of phrases. Thus, the set can represent a set of possible alternative utterances spoken by the user. Using the example above, the set can include {“text Ed”, “text Ted”, “text Red”, “texted”, etc.} In some embodiments of step 720, a context can be determined based on the set of possible words. For example, in the previous example, the context can include the user texting or communicating with another person. In some embodiments, the context can be determined as described in connection with the context information extractor 284 in FIG. 6B. Figure 8

[0159] At step 730, an inferred user activity pattern is accessed. Embodiments of step 730 can access an inferred user activity pattern as described in step 510 of method 500 or 630 of method 600. Further, a particular inferred user activity pattern accessed can be determined based on a context, such as the context described in step 520. In particular, the context can be determined from a location, a time, a user behavior or activity, and / or information associated with the utterance in step 710 that is relevant to the current situation of the user.

[0160] At step 740, a possible future activity event is determined based on the activity pattern determined in step 730. Some embodiments of step 740 further determine the possible future activity event based on the context described in connection with step 730. Some embodiments of step 740 can be performed as described in step 520 of method 500.

[0161] ​At step 750, a subset of the words that best fit the possible future activity event is selected. Embodiments of step 750 determine the words that are most likely to be spoken by the user that best fit the possible future activity determined in step 750 from the set of possible spoken words determined in step 720. Specifically, the best fit can be determined by performing an analysis to determine whether performing an operation according to each action associated with the set of words would result in an action having the same activity characteristics as the activity pattern accessed in step 730. For example, assume that the user has a habit of calling family members while driving home from work, the user has a brother named Pat and a friend named Matt, and the voice command "call Pat" is issued. The context indicates that the user is driving, has just left his office, and it is the end of the work day. Thus, the set of words corresponding to the user's utterance includes at least "call Pat" and "call Matt." But only one of these words - the word calling a brother - fits the user's activity pattern of calling family members while driving home from work. Accordingly, this subset of words (i.e., "call Pat") is selected. In this way, a more accurate parsing of the speech is performed.

[0162] At step 760, an action corresponding to the subset of words is determined. Embodiments of step 760 determine an action corresponding to the subset of words determined in step 750. The action can include one or more services, processes, routines, functions, or similar activities performed or facilitated by the user device, or initiate a control signal to cause a component to perform the action. For example, in the previous example, the action includes initiating a call to Pat (or initiating a control signal to the communication component on the phone to call Pat). Without limitation, other examples of actions determined in step 760 can include other communications (e.g., texting, sending an email, instant messaging, posting, etc., which can also include drafting or creating content for the communication, such as drafting an email message), performing a query, launching an application, navigating to a website, playing specific multimedia content, controlling operation of a device, service, routine, or function, executing a command, request, or any other action that can be performed or facilitated via the user device. In some embodiments of step 760, the action is determined from a set or library of actions that the user device is capable of performing or facilitating.

[0163] At step 770, the action determined in step 760 is performed. Embodiments of step 770 perform the action corresponding to the subset of words determined in step 750. Thus, for example, continuing the example described in step 750, the action includes initiating a phone call to the user's brother, Pat. Some embodiments of step 770 include initiating a control signal to facilitate performance of the action. For instance, in the case where the user asks to turn on a light or adjust a thermostat to be warmer, step 770 can issue a control signal (or a request or a similar communication specifying the action) to a light controller component or a smart thermostat.

[0164] Turning now to Figure 8 , a perspective view 801 is illustratively provided that depicts an example of a computer-implemented speech recognition (or automatic speech recognition (ASR)) and understanding system according to embodiments of the present disclosure. Figure 8 The illustrated ASR system is merely one example of an ASR system suitable for use with embodiments of the present disclosure for determining recognized speech. It is contemplated that other variations of ASR systems or spoken language understanding (SLU) systems (not shown) can be used, including ASR systems that include fewer components than the example ASR system shown here, or ASR systems that include additional components not shown in the example ASR system. Figure 1

[0165] The perspective view 801 shows a sensor 850 that senses acoustic information (audibly spoken words or speech 890) provided by a user speaker 895. The sensor 850 can include one or more microphones or acoustic sensors, which can be embodied as the sensor 103a or 107 or on a user device such as the user device 102 or 104 described in the example of Figure 9 The sensor 850 converts the speech 890 into acoustic signal information 853 that can be provided to a feature extractor 855 (or, in some embodiments, can be provided directly to a decoder 860). In some embodiments, the acoustic signal can undergo pre-processing (not shown) prior to the feature extractor 855. The feature extractor 855 generally performs feature analysis to determine useful features of the parameterization of the speech signal, while reducing noise or otherwise discarding redundant or unwanted information. The feature extractor 855 transforms the acoustic signal into features 858 suitable for use by a model of the decoder 860 (which can include a speech corpus).

[0166] The decoder 860 includes one or more acoustic models (AM) 865 and language models (LM) 870. In embodiments of the present disclosure implemented in conjunction with an ASR (or SLU) system, particular embodiments can be implemented as sub-components of the decoder 860, LM 870, AM 865, or as separate components (not shown) in the example ASR (or SLU) system. ​

[0167] The AM 865 includes a statistical representation of the different sounds that make up a word, which can be assigned a label referred to as a "phenome." The AM 865 models the phenome based on speech features and provides a language model 870 with a corpus or set of words, which can include sequences of words corresponding to the speech corpus, or in some instances, a set of alternative sequences, each corresponding to a possible utterance of the user speaker 895. The LM 870 receives the corpus of words and determines a recognized speech 880, which can include a subset of words (which can be a single word, phrase, or multiple words or phrases) from a set of alternative interpretations (or recognitions) that includes one interpretation (or recognition) of the utterance. In this way, a subset of words representing a possible utterance spoken by the user can be determined.

[0168] Thus, we have described various aspects of technology directed to systems and methods for inferring user intent and predicting future user activity associated with a user's computing device, which can be used to provide an enhanced user experience. It should be appreciated that various features, subcombinations, and modifications of the embodiments described herein can be employed as appropriate and desired. Moreover, the order of the steps in the example methods 400, 500, 600, and 700 illustrated and further described herein need not be followed, or can be changed, and not all steps can be utilized, or some steps can be utilized more than once. Such variations and combinations of steps and features are also considered to be within the scope of the embodiments of the present disclosure.

[0169] Having described various implementations, an example computing environment suitable for implementing embodiments of the present disclosure is now described. With reference to Figure 9 , an example computing device is provided and generally referred to as computing device 900. Computing device 900 is merely one example of a suitable computing environment and is not intended to limit the scope of use or functionality of embodiments of the present disclosure. Nor should computing device 900 be interpreted as having any dependency or requirement relating to any one or combination of the illustrated components.

[0170] Embodiments of the present disclosure may be described in the general context of computer code or machine-usable instructions, which include computer-usable instructions or computer-executable instructions (such as program modules) executed by a computer or other machine (such as a personal data assistant, smart phone, tablet PC or other handheld device). Typically, a program module including routines, programs, objects, components, data structures, etc. refers to a code that performs a specific task or implements a specific abstract data type. Embodiments of the present disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. Embodiments of the present disclosure may also be practiced in a distributed computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed computing environment, program modules may be located in both a local computer storage medium and a remote computer storage medium including a memory storage device.

[0171] like Figure 9 As shown, computing device 900 includes bus 910, which directly or indirectly couples the following devices: memory 912, one or more processors 914, one or more presentation components 916, one or more input / output (I / O) ports 918, one or more I / O components 920, and an illustrative power supply 922. Bus 910 can represent one or more buses (such as an address bus, a data bus, or a combination thereof). Although Figure 9 The various blocks of FIG are shown with lines for clarity, but in reality, these blocks represent logical components and not necessarily physical components. For example, a presentation component such as a display device can be considered an I / O component. In addition, a processor has memory. The inventors recognize that this is the nature of the art and reiterate that Figure 9 The figures are merely illustrative of exemplary computing devices that may be used in conjunction with one or more embodiments of the present disclosure. No distinction is made between categories such as "workstation," "server," "laptop," "handheld device," etc., as they are all encompassed by ​ within the scope of and within references to “computing devices”.

[0172] The computing device 900 generally includes a variety of computer-readable media. Computer-readable media are media that can be accessed by the computing device 900 and include both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computing device 900. Computer storage media excludes signals per se. Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the above should also be included within the scope of computer-readable media.

[0173] The memory 912 includes computer storage media in the form of volatile and / or nonvolatile memory. The memory can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid state memory, hard drives, optical drives, etc. The computing device 900 includes one or more processors 914 that read data from various entities such as the memory 912 or the I / O components 920. The one or more presentation components 916 present data indications to a user or other devices. In some implementations, the presentation component 220 of the system 200 can be embodied as the presentation component 916. Other examples of presentation components can include display devices, speakers, printing components, vibrating components, etc.

[0174] I / O ports 918 allow the computing device 900 to be logically coupled to other devices including the I / O components 920, some of which can be built in. Illustrative components, types and / or devices include a microphone, joystick, game pad, satellite dish, scanner, printer, wireless device, etc. The I / O components 920 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs can be transmitted to an appropriate network element for further processing. A NUI can implement any combination of speech recognition, touch and stylus recognition, facial recognition, biometric recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, and touch recognition associated with displays 920 on the computing device 900. The computing device 900 can be equipped with depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, and combinations of these, for gesture detection and recognition. Additionally, the computing device 900 can be equipped with accelerometers or gyroscopes that enable detection of motion. The output of the accelerometers or gyroscopes can be provided to the display of the computing device 900 to render immersive augmented reality or virtual reality.

[0175] Some embodiments of the computing device 900 can include one or more radios 924 (or similar wireless communication means). The radios 924 transmit and receive radio or wireless communications. The computing device 900 can be a wireless terminal adapted to receive communications and media through various wireless networks. The computing device 900 can communicate via wireless protocols such as code division multiple access ("CDMA"), global systems for mobile communications ("GSM"), or time division multiple access ("TDMA"), among others, to communicate with other devices. The radio communications can be short-range connections, long-range connections, or a combination of short-range radio connections and long-range radio connections. When we refer to "short" and "long" types of connections, we do not mean the spatial relationship between two devices. Rather, we refer to short-range and long-range as different categories or types of connections (i.e., primary and secondary connections). By way of example and not limitation, a short-range connection can include a connection to a device that provides access to a wireless communication network (such as a WLAN connection using an 802.11 protocol) to a device (e.g., a mobile hotspot), a Bluetooth connection to another computing device, or a near field communication connection. By way of example and not limitation, a long-range connection can include a connection using one or more of the following: a CDMA protocol, a GPRS protocol, a GSM protocol, a TDMA protocol, and an 802.16 protocol.

[0176] ​Many different arrangements of the various components depicted, as well as components not shown, are possible without departing from the scope of the following claims. Embodiments of the disclosure have been described with the intent to be illustrative rather than restrictive. Alternative embodiments will become apparent to those skilled in the art upon reading this disclosure without departing from its scope. Alternative ways of implementing the described functionality of the above described embodiments are possible. Certain features and subcombinations are of utility and can be employed without reference to other features and subcombinations and are considered to be within the scope of the claims.

[0177] Thus, in one aspect, embodiments of the disclosure relate to a computerized system comprising one or more sensors configured to provide sensor data; a user activity monitor configured to identify and monitor user devices and user activities associated with the user devices; an activity pattern inference engine configured to determine activity patterns from a plurality of activity events; one or more processors; and a computer storage memory having stored thereon computer executable instructions that, when executed by the processors, implement a method of inferring user activity patterns. The method comprises: (a) identifying a set of user devices associated with a user; (b) monitoring the set of user devices using the user activity monitor to detect user activity events; (c) upon detecting a user activity event, determining a set of activity features associated with the activity event, the set of activity features being determined based at least in part on the sensor data; (d) storing a record of the activity event and the associated activity features in an activity event data store comprising records of a plurality of activity events; (e) using the activity pattern inference engine to identify activity patterns based on analysis of the plurality of activity events to determine a set of activity events having similar activity features; and (f) storing a description of the activity pattern in an activity pattern data store.

[0178] In some embodiments of the system, the activity events comprise one or more of: browsing a website, launching an application, initiating a communication, performing a search query, setting a reminder, or scheduling an event on a calendar, and the set of associated activity features comprises one or more features related to: content associated with the activity event (content features), date or time of the activity event (date-time features), location of the activity event (location features), device usage related features associated with the activity event (device usage features), or contextual information associated with the activity event. Further, in some embodiments, activity features are determined to be similar based on a comparison of activity features of the same category using a similarity threshold predetermined according to the category of the compared activity features, the categories including (by way of example and not limitation) content features, date-time features, location features, device usage features, or other types of activity features described herein.

[0179] In another aspect, embodiments of the present disclosure relate to a computerized system comprising one or more sensors configured to provide sensor data; one or more processors; and a computer storage memory having stored thereon computer executable instructions that, when executed by the processors, implement a method of inferring a possible future user action. The method comprises: (a) accessing an inferred user activity pattern of a user; (b) predicting a possible future activity event based on the activity pattern and a context determined at least in part from the sensor data; and (c) providing an enhanced user experience based on the determined possible future activity event.

[0180] In one embodiment of the system, the provided enhanced user experience comprises one of a recommendation, a notification, a request, or a suggestion related to the possible future activity. In another embodiment, the provided enhanced user experience comprises automatically performing the possible future activity at a time or location that coincides with the inferred user activity pattern. In yet another embodiment, the predicted possible future activity event comprises connecting a user device having a battery to a power source to charge the battery at a future time, and the enhanced user experience comprises a battery power management service that reduces power consumption of the user device based on the future time.

[0181] In yet another embodiment of the system, the provided enhanced user experience comprises a speech recognition service, and at least one of the one or more sensors comprises an acoustic sensor on the user device configured to convert speech into acoustic information. The method for the speech recognition service embodiment further comprises: (i) receiving acoustic information corresponding to a user's verbal interaction with the user device; (ii) determining a plurality of sequences of words corresponding to the verbal interaction based on an analysis of the acoustic information; (iii) selecting a sequence of words that best coincides with the predicted possible future activity event; and (iv) providing the selected sequence of words as recognized speech corresponding to the verbal interaction.

[0182] In yet another aspect, embodiments of the present disclosure relate to a method for performing disambiguation to determine a user intent of a user. The method includes: (a) receiving an indication of a user interaction with a user device, the user interaction being associated with an activity performed by the user device, the indication being based at least in part on sensor data from one or more sensors associated with the user device; (b) determining a set of possible user intents corresponding to the user interaction; (c) accessing an inferred activity pattern of the user; (d) determining a possible future activity event based on the inferred activity pattern; (e) selecting a user intent from the set of possible user intents that best fits the possible future activity event; and (f) performing the activity associated with the user interaction based on the selected intent. In some embodiments of the method, the activity associated with the user interaction includes a search query, and further includes filtering search results to be provided to the user based on the selected user intent.

Claims

1. A computerized system comprising: one or more processors; as well as A computer storage memory having computer-executable instructions stored thereon, the computer-executable instructions, when executed by the one or more processors, implementing a method for disambiguating an indication of a user interaction, the method comprising: a) receiving a set of words forming said indication of said user interaction; b) Record user activities in the pipeline; c) disambiguating the meaning of said set of words forming said indication of said user interaction based on a repeated sequence of events in said pipeline; and d) performing a computerized action based on the disambiguated meaning. 2 . The computerized system of claim 1 , wherein the indication of the user interaction is formed from sensor data comprising user utterances.

3. The computerized system of claim 1, wherein the sequence of events comprises events having similar time periods. The computerized system of claim 3 , wherein the sequence of events comprises events having similar locations.

5. The computerized system of claim 3, wherein the sequence of events includes events involving similar content. The computerized system of claim 1 , wherein the set of words is determined to be consistent with the determined activity event.

7. The computerized system of claim 6, wherein the activity event determined comprises one or more of browsing a website, launching an application, initiating a communication, performing a search query, setting a reminder, or scheduling an event on a calendar.

8. The computerized system of claim 6, wherein a user is provided with an indication of the determined set of words.

9. A computer-implemented method of disambiguating an indication of a user interaction, the method comprising: receiving a set of words forming said indication of said user interaction; recording user activity in a data storage device; determining a user intent for the set of terms forming the indication of the user interaction based at least in part on an activity pattern determined by the user activity in the data store, wherein the activity pattern is a recurring pattern of usage by the user while using a computing device; and Based on the user intent, a computerized action is performed.

10. The method of claim 9, wherein the indication of the user interaction is formed from sensor data comprising user utterances. The method of claim 9 , wherein the activity patterns include user activities having similar time periods. The method of claim 11 , wherein the activity pattern comprises user activities having similar locations. The method of claim 11 , wherein the activity patterns include user activities involving similar content. The method of claim 9 , wherein the set of words is determined to be consistent with the determined activity event.

15. A computerized system comprising: one or more sensors configured to provide sensor data, each sensor associated with a user device from a set of user devices associated with a user, at least one of the one or more sensors comprising an acoustic sensor configured to convert speech into acoustic information; one or more processors; as well as A computer storage memory having computer-executable instructions stored thereon, the computer-executable instructions, when executed by the one or more processors, implementing a method for disambiguating an indication of a user interaction, the method comprising: receiving a set of words forming said indication of said user interaction from sensor data corresponding to a user utterance, Recording user activities in a data storage device, disambiguating the meaning of the set of words forming the indication of the user interaction based at least in part on an activity pattern determined from the user activity in the data store, wherein the activity pattern is a recurring pattern of usage by the user while using a computing device, and Based on the disambiguated meaning, a computerized action is performed.

16. The computerized system of claim 15, wherein the activity patterns include user activities having similar time periods.

17. The computerized system of claim 16, wherein the activity patterns include user activities having similar locations.

18. The computerized system of claim 16, wherein the activity patterns include user activities involving similar content.

19. The computerized system of claim 16, wherein the set of words is determined to be consistent with the determined activity event.

20. The computerized system of claim 15, wherein the user is provided with an indication of the determined set of terms.

Images