A system for interpreting and managing imprecise time expressions

Abstract

The present disclosure relates to a system for interpreting and managing imprecise temporal expressions. Techniques for extracting, identifying, and consuming imprecise temporal elements ("ITE") are disclosed. User input may be received from a client device. A prediction for one or more time intervals indicated by the user input may be generated based on an ITE model. The user input may be associated with the prediction and provided to the client device.

Description

[0001] This application is a divisional application of the invention patent application with application date of November 27, 2017, application number 201780074271.3, and invention name “System for interpreting and managing imprecise time expressions”. Technical Field

[0002] The present disclosure generally relates to techniques for helping users complete tasks and plan future activities. Background Art

[0003] Automated personal assistants (e.g., virtual assistants) can support a range of reactive and proactive scenarios, from answering questions to providing alerts about flight schedules and traffic. Some automated personal assistants can also provide reminder services designed to help people remember future tasks they might otherwise forget. Automated personal assistants can use natural language to access device commands, settings, and integrated services. In many cases, these interactions may require users to communicate dates and / or times. For example, a time indication is required when a user wants to create an appointment, set a reminder, and / or query the weather, sports, news, or request a timeline for a project. Whether issuing commands or retrieving information, conventions and guidance encourage users to construct their utterances as if they were speaking to a real person. However, these types of interpersonal communication often result in imprecise, nuanced, and / or ambiguous expressions of time. Consequently, users often use imprecise expressions of time (ITEs) to express uncertainty about time in communication and planning. ITEs can be useful for communication purposes (e.g., to convey priorities, flexibility, or uncertainty), but modern automated personal assistants lack system support for capturing the intent behind ITE systems and responding appropriately. This can lead to unnatural interactions and undesirable interruptions. For example, a conventional automated personal assistant might interpret "sometime later today" as 12:00 PM, when the user intended 2:00 PM. Therefore, a system is needed that can identify ITEs and generate predictions for one or more approximate time intervals, so that the system can be consumed by a device's automated personal assistant, a prediction service, a scheduling service, and the like. Summary of the Invention

[0004] In an embodiment, a system is disclosed that may include a database and a processor communicatively coupled to the database. The database may store an imprecise temporal expression model. In some instances, an ITE model may be generated by the processor. The processor may be configured to receive user input from a client device. The user input may be received by the system in the form of voice and / or text input. A prediction for one or more time intervals indicated by the user input may be generated based on the ITE model in the database. The user input may include at least one of an ITE, contextual text, and the time of the interaction. In the event that the user input includes an ITE, the processor may be configured to detect the ITE. In some instances, the prediction may be presented to the client device (or server), and a request may be generated to receive user input to confirm the prediction. In some instances, the processor may receive an indication to modify the prediction and, based on the indication, update the ITE model in the database. The prediction may be associated with the user input. The processor may provide the prediction to the client device. The processor may be configured to display the prediction on the client device at a time corresponding to the user input. The prediction may be provided to the client device in various ways, such as via reminders, notifications, text auto-completion, a timeline, a task list, a calendar schedule, a time interval, a ranked list of time intervals, a probability distribution, and the like.

[0005] According to an embodiment, a method is provided in which user input is received from a client device, a prediction can be generated based on an ITE model, the user input can be associated with the prediction, and the prediction can be provided to the client device.

[0006] In an embodiment, a non-transitory computer-readable medium having stored thereon computer-readable instructions executable to cause one or more processors to perform operations is disclosed. User input may be received from a client device. The operations may include generating predictions for one or more time intervals, wherein the one or more time intervals referred to by the user input may be generated based on an ITE model stored in a database. The user input may include at least one of an ITE, contextual text, and a time of interaction. The user input may be associated with a prediction, and the computer-readable instructions may cause the prediction to be provided to the client device.

[0007] Additional features, advantages, and embodiments of the disclosed subject matter may be set forth or will become apparent in view of the following detailed description, drawings, and claims. In addition, it should be understood that the foregoing summary and the following detailed description are exemplary and are intended to provide further explanation without limiting the scope of the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The accompanying drawings, which are included to provide a further understanding of the disclosed subject matter, are incorporated into and constitute a part of this specification. The drawings also illustrate embodiments of the disclosed subject matter and, together with the detailed description, serve to explain the principles of the embodiments of the disclosed subject matter. No attempt has been made to show structural details in greater detail than is necessary for a basic understanding of the disclosed subject matter and the various ways in which it may be practiced.

[0009] Figure 1 FIGURE 1 illustrates an M according to the disclosed embodiment. CT and M NT The creation time P(r CT ) and notification time P(r NT ) is an example of a prior probability of .

[0010] Figure 2 Here is an example of a plot of a sample dataset showing the delay between reminder creation and notification.

[0011] Figure 3 Examples of both per-task type reminders and those outside of these blocks are shown. "Activity" and "Go Somewhere" reminders tend to be created primarily during office hours, whereas, for example, "Communication" and "Housework" reminders tend to be created more in the evening.

[0012] Figure 4 Illustrated is an example of a probability distribution for tasks that are set to notify a user during office hours and tasks that are triggered outside of these hours.

[0013] Figure 5 An example of probability distribution of reminder notification times for “communication / general” and “communication / coordination” tasks according to embodiments disclosed herein is illustrated.

[0014] Figure 6 An example of a box plot that may illustrate the delay between reminder creation and notification time is illustrated (n=125376).

[0015] Figure 7 An example of a delay (eg, lead time) between reminder creation and notification for the “Communicate” subtask is shown.

[0016] Figure 8 An example of creation time and notification time of a task description containing the word “church” or “appointment” based on sample data according to an embodiment disclosed herein is shown.

[0017] Figure 9 Illustrated is an example of a comparison of the average delays for reminders containing the words "appointment" to "laundry" in an example data set.

[0018] Figure 10 P(r) for three different rCTs according to embodiments disclosed herein are shown. NT |r CT ) example.

[0019] Figure 11 An example of an implementation is shown that provides a process for providing predictions for imprecise temporal expressions to a client device (or other computer).

[0020] Figure 12 An example interface of a note-taking or memo program operating on a client device is illustrated.

[0021] Figure 13 is an example interface of a note-taking or memo program operating on a client device, in which text has been entered into a text input box.

[0022] Figure 14 is an example interface for a note-taking or memo program operating on a client device, in which a user has entered the phrase "Patent application filed this week" into a text input box.

[0023] Figure 15 is an example computer suitable for implementing embodiments of the disclosed subject matter.

[0024] Figure 16 An example network arrangement is shown in accordance with an embodiment of the disclosed subject matter.

[0025] Figure 17 Shown are examples of dialogs that may be displayed to a user to receive direct input from the user regarding time intervals that the user has assigned to a project, according to embodiments.

[0026] Figures 18A-18C An example of aggregated responses across multiple users (eg, a group of users) is shown. Figure 18A An example distribution of responses for the term "early next week" is shown. Figure 18B An example distribution of responses for the term "next weekend" is shown. Figure 18C An example distribution of responses for the term "later this week" is shown.

[0027] Figure 19 This figure shows an example of how "next weekend" is interpreted on different days of the week based on crowdsourcing results. The number shown is the proportion of users who include a specific execution date in their interpretation.

[0028] Figure 20 This figure shows an example of how "later this week" is interpreted on different days of the week based on crowdsourcing results. The number shown is the proportion of workers who included a specific execution date in their interpretation.

[0029] Figure 21 is an example of a note-taking or memo application for an automated personal assistant that displays a ranked list of tasks based on various predictions generated from user input, in accordance with embodiments disclosed herein.

[0030] Figure 22 is an example of explicit feedback that can be provided by the user. DETAILED DESCRIPTION

[0031] Modern automated personal assistants often insist on precise date and time specifications, or will strictly disambiguate a limited set of time expressions into predetermined wall-clock times (e.g., mapping "morning" to 7:00 AM). While early resolution of these expressions may be the simplest solution for system designers and implementers, this strategy can lead to a broken user experience. Users are often strategic in their use of time expressions to convey their own uncertainty, commitments, or task priorities. In these scenarios, insisting that users enter a specific time or date can increase the burden of using the system. Similarly, overly literal interpretation of time expressions can lead to reminders or notifications being delivered at inappropriate times (e.g., a user who specifies "this afternoon" may not want or expect a reminder to be delivered at exactly 12:00 PM). Table 1 shows examples of imprecise and precise date or time expressions. Table 2 lists several other types of ITEs. When considered together with a base time (e.g., the time the expression is spoken), a precise date resolves to the entirety of a specific calendar day. ITEs can refer to time expressions that resolve neither to a precise date (i.e., the entirety of a calendar day) nor to a precise time (i.e., a time with specified hours and minutes).

[0032] Table 1

[0033]

[0034]

[0035] Table 2

[0036]

[0037]

[0038] Disclosed herein are systems and methods that (1) receive user input, such as text or natural language, (2) generate predictions for one or more time intervals indicated by the user input, and / or (3) provide prediction services that facilitate data entry and / or delivery of time-related information to the user in a manner that is appropriate given the uncertainties involved. For example, the resulting predictions may represent the system's beliefs about when an event is likely to occur or has occurred, and / or when action should be taken (e.g., a notification, warning, and / or alarm sent). An alarm may refer to visual, tactile, or auditory feedback (e.g., emitting a sound, displaying text or notification, illuminating a light emitting diode, etc.). The predictions may be based on an ITE model that may take into account various factors, such as the time at which an utterance is spoken or written, the content / subject matter of an enclosing utterance or text, and / or details known about the user. An ITE model may refer to a set of distributions that indicate beliefs about a base time or time interval that an ITE or event (e.g., mowing the lawn) may indicate. An ITE model may be generated in various ways, as described below. In short, an ITE model can be developed based on analysis of data logs (such as data logs corresponding to an email corpus and / or an automated personal assistant). More than one ITE model can be generated (e.g., there can be a model for each ITE), and an ITE model can be updated and / or specific to a particular user or group of users.

[0039] Because ITE can refer to the past, the disclosed embodiments can have practical applications beyond reminders or future intent. For example, the disclosed system can analyze the text of a user's past emails to determine at least one interval of (multiple) events that have occurred, and place these events on a timeline that can be displayed to the user. ITE or specific contextual text (e.g., words that potentially indicate a temporal element, such as "Halloween" or "mowing the lawn") can be automatically detected in the text of the email. For example, a user can request that a timeline be generated from past emails associated with specific keywords. ITE associated with a subset of the identified emails can be detected and displayed to the user as a timeline to obtain an overview of the life cycle of a project or the relationship between the phases of a project.

[0040] Predictive services can facilitate temporal expression output. As an example of a predictive service, when composing a to-do task in a calendar application, the system can utilize the current time and the text of the to-do item to automatically suggest deadlines and / or notification times from which the user can choose. As another example, when a user indicates a to-do item, calendar appointment, etc., a speech recognition or data extraction process can utilize an ITE model to refine hypotheses (e.g., an n-best list). For low-certainty items that may become "due" in the near future, an automated personal assistant can seek additional scheduling clarification as part of a subsequent maintenance task. For example, the assistant can request scheduling guidance the next time the user opens the calendar, or as part of a daily briefing delivered at a convenient break time (e.g., at the beginning or end of the day). Regarding a specific certainty.

[0041] The prediction can be returned to the client device and / or server in various ways. The format of the prediction can be provided in a user-observable form, such as a task list, reminder, calendar, timeline, etc. The prediction can also be in the form of a mathematical representation, such as a time interval and / or distribution. The examples of such predictions provided here are not limited to the following. In some instances, for example, the representation can be stored to the client device and / or a database connected to the server. In the following examples 1-14, standard mathematical symbols are applied so that "[a, b]" indicates an interval that includes a and b, and "[a, b)" indicates an interval from a to b but not including all points. Similarly, "(a, b]" indicates an interval that does not include a, but includes all points greater than a up to b and including b. Finally, "(a, b)" includes all points between a and b, but excludes a and b themselves, and "[a, a]" is a "point interval" that includes point a and only a.

[0042] 1) Time span: For example, "next weekend" can be represented (e.g., calculated, stored, displayed, etc.) as [Friday, December 2, 2016, 5:00 PM - Monday, December 4, 2016, 12:00 PM).

[0043] 2) Ranked List: For example, a prediction may include several possible time intervals ranked from most likely to least likely based on aggregated user responses indicating the time interval indicated by a particular user input. Other methods for ranking lists based on the received data may be used in place of or in addition to this process. For example, the term "next weekend" may be presented as a ranked list, such as Ranking 1: [Friday, December 2, 2016, 5:00 PM - Monday, December 4, 2016, 12:00 PM]. Ranking 2: [Friday, December 9, 2016, 5:00 PM - Monday, December 11, 2016, 12:00 PM]. In some configurations, the ranked list may be presented to the user in a dialog box, etc.

[0044] 3) Open interval to the indefinite future: For example, the term "after this weekend" may be provided as (Monday, December 4, 2016, 12:00 noon - infinite future range).

[0045] 4) Open intervals in the indefinite past: For example, the term "before the weekend" may be provided as (infinite past range - Friday, December 02, 2016, 5:00 PM).

[0046] 5) A set of non-contiguous intervals: For example, the term "next week's working hours period" can be provided as a mathematical representation such as {[Monday, December 5, 2016, 9:00 am - Monday, December 5, 2016, 5:00 pm], [Tuesday, December 6, 2016, 9:00 am - Tuesday, December 6, 2016, 5:00 pm], [Wednesday, December 7, 2016, 9:00 am - Wednesday, December 7, 2016, 5:00 pm], [Thursday, December 8, 2016, 9:00 am - Thursday, December 8, 2016, 5:00 pm], [Friday, December 9, 2016, 9:00 am - Friday, December 9, 2016, 5:00 pm]}.

[0047] 6) Sorted list of sets of non-contiguous intervals: For example, the term "next week's working hours" -> sort 1: {[Monday, December 5, 2016, 9:00 am - Monday, December 5, 2016, 5:00 pm], [Tuesday, December 6, 2016, 9:00 am - Tuesday, December 6, 2016, 5:00 pm], [Wednesday, December 7, 2016, 9:00 am - Wednesday, December 7, 2016, 5:00 pm], [Thursday, December 8, 2016, 9:00 am - Thursday, December 8, 2016, 5:00 pm], [Friday, December 9, 2016, 9:00 am - December 09, Friday, 5:00 PM], Sort 2: {[Monday, December 12, 2016, 9:00 AM - Monday, December 12, 2016, 5:00 PM], [Tuesday, December 13, 2016, 9:00 AM - Tuesday, December 13, 2016, 5:00 PM], [Wednesday, December 14, 2016, 9:00 AM - Wednesday, December 14, 2016, 5:00 PM], [Thursday, December 15, 2016, 9:00 AM - Thursday, December 15, 2016, 5:00 PM], [Friday, December 16, 2016, 9:00 AM - Friday, December 16, 2016, 5:00 PM]}.

[0048] 7) Predictions with confidence: For example, the term "next weekend" may be provided as a prediction in a form such as 80% [Friday, December 02, 2016, 5:00 PM - Monday, December 04, 2016, 12:00 PM].

[0049] 8) A list of predictions with confidence in each candidate / subcomponent: For example, the term "next weekend" may be provided as 80% [Friday, December 2, 2016, 5:00 PM - Monday, December 4, 2016, 12:00 PM]. 20%: [Friday, December 9, 2016, 5:00 PM - Monday, December 11, 2016, 12:00 PM].

[0050] 9) A non-exhaustive list of predictions with confidence in each candidate / subcomponent (note that the percentages do not sum to 100): For example, the term "next weekend" may be provided as 50% [Friday, December 2, 2016, 5:00 PM - Monday, December 4, 2016, 12:00 PM]. 20% [Friday, December 9, 2016, 5:00 PM - Monday, December 11, 2016, 12:00 PM].

[0051] 10) A list of predictions with a confidence "score" for each candidate / subcomponent (the sum of the percentages may not equal 100%), where higher / lower scores indicate confidence: for example, the term "next weekend" may have the following "scores" associated with it: score 104.32 [Friday, December 2, 2016, 5:00 PM - Monday, December 4, 2016, 12:00 PM]. Score 43.43 [Friday, December 9, 2016, 5:00 PM - Monday, December 11, 2016, 12:00 PM]. These scores can be based on, for example, analysis of user data aggregated from various sources (e.g., logs from an automated personal assistant).

[0052] 11) A prediction list with an overall confidence that an expression is covered by any of its subcomponents and that the subcomponents sum to 1: For example, the term "next weekend" may be provided with an overall confidence of 95%. 80% [Friday, December 2, 2016, 5:00 PM - Monday, December 4, 2016, 12:00 PM]. 20% [Friday, December 9, 2016, 5:00 PM - Monday, December 11, 2016, 12:00 PM]. In this example, the confidence that the expression (e.g., ITE) is included in the two provided time intervals may be 95%.

[0053] 12) Prediction list with scores where expressions are covered by any one of the subcomponents and the subcomponents sum to 1: Continuing with the previous example, a score of, for example, 92.8 could be provided instead to reflect an overall confidence of 95%.

[0054] 13) Intervals with associated distributions over the interval: For example, for an appropriate choice of parameter λ, and where α = the start of the interval, and for any time t in the interval, an exponentially decaying probability can be expressed as follows: the term "after the weekend" can be expressed as {(Monday, December 4, 2016, 12:00 noon - infinite future range), λe^(-λ(t-α))}. For large λ, this may produce a significantly higher probability at time points shortly after Monday, December 4, 2016, 12:00 noon and decay rapidly. For smaller λ, the probability may decay significantly more slowly. The parameters can be learned from the data as part of the process of learning a temporal representation model. Similarly, distributions other than exponential distributions can be used, including mixtures of distributions.

[0055] 14) The forecast can be expressed as in any of the preceding examples 1-12, except that a distribution (continuous or discrete) can be associated with each bin or subcomponent, as in the preceding example 13.

[0056] Thus, each of the above examples illustrates a prediction that can be generated and provided to a client device and / or a server, where the prediction can be applied or used by other services. For example, a scheduling application can detect "Let's meet up sometime this weekend" as utterances in a video conference between two users. A prediction of "sometime this weekend" in the form of examples 1-14 above can be provided from a user's device to a server that handles communication between the two users' devices. The server can compare the two provided predictions from the client devices to determine any overlap of at least one time interval and propose a meeting time to the two users accordingly to resolve the scheduling conflict during the (multiple) overlapping times corresponding to "sometime this weekend". The time can be proposed via a separate application (e.g., an email program or a calendar application) or through a video conferencing application. The proposed time can be displayed as a dialog box requesting the user to confirm the appointment.

[0057] Several systems have been developed to help remind people of future actions, many of which leverage contextual signals for more accurate reminders. These systems can help generate reminders associated with a range of future actions, including locations, events, activities, people, and times. Two of the most commonly supported types of reminders are location-based and / or time-based. Location-based reminders can be triggered when a person is at or near a location of interest. Time-based reminders can be set and triggered based on time, including reminders based on the time spent on a task. Time-based reminders can provide value to many users, particularly the elderly, users with memory impairments, and / or users seeking to adhere to prescribed medications.

[0058] Memory aids can help users remember past events and information. Users tend to utilize their own memories and use external memory aids to increase the likelihood of recall via recall strategies. Aids can take different forms, including paper to electronic alternatives. An example of a computer-based memory aid is a memory agent, which can use contextual information (e.g., words typed into a text processor) to retrieve similar documents. Users have been shown to use standard computer facilities to support future reminders (e.g., positioning documents in a prominent position on the computer desktop). For various reasons, these uses may be inadequate, including lack of alerts. Other solutions can use machine learning to predict forgetting and the need for reminder events. According to the embodiments disclosed herein, usage patterns and user behavior are examined and / or quantified to enable a better understanding of user needs, development of improved methods for system-user interaction and collaboration, and enhanced understanding of the types of tasks for which memory aids provide value.

[0059] Prospective memory ("PM") can refer to the ability to remember actions to be performed in the future. Successful PM requires recall at the appropriate time. Failures in PM can be related to external factors (such as interruptions). Prospective tasks can be divided into time-based tasks and event-based tasks. Time-based tasks can be tasks that are intended to be performed at a specific future time, while event-based tasks can be performed when a specific situation or event occurs that is triggered by external cues (e.g., people, places, and / or objects). As disclosed herein, research logs of automated personal assistants can represent a rich source of realistic, time-based PM examples. These examples can provide insights into the types and nature of tasks that users may have forgotten to perform.

[0060] As disclosed herein, a method for creating common time-based reminders at scale in natural settings is provided. Second, a classification of types of time-based reminders facilitated by data obtained about reminders created by a large number of users is disclosed. Third, important aspects of the reminder generation process can be characterized, including their nature (e.g., reminders for ongoing activities versus planned activities), and the relationship between reminder text and the time at which reminders and alarms are set to remind. Predictive models can be generated based on patterns discovered from the data obtained about reminders. As previously discussed, predictive models can be used to determine, for example, when a reminder should be triggered.

[0061] The behavior surrounding reminder creation can be examined by observing common tasks associated with setting reminders and determining whether there is a body of common tasks underlying the reminder creation process. Reminders that are frequently observed and across multiple users can be extracted. Reminders can be categorized into task classes to better understand the types of tasks associated with reminders.

[0062] In the left column of Table 3, three examples of common reminders are listed. These examples illustrate the structure often observed in recorded reminders obtained from automated personal assistants. Reminders can be composed as predicate sentences. They can contain a phrase related to the action the user wants to perform (usually a verb phrase) and a referenced object that is the target of the action to be performed.

[0063] Table 3

[0064] remind predicate Remind me to take out the trash Take out (my trash) Remind me to put my clothes in the dryer Put (my clothes in the dryer) Remind me to withdraw cash from the bank Withdraw (I, cash from the bank)

[0065] The conversation for setting a reminder can include a dialog box in which the user and the automated personal assistant can interact in multiple rounds. The user can begin by issuing a command to set a reminder and indicating the reminder. Optionally, the user can specify a notification time for the reminder. The automated personal assistant can then request a specific time (if the user has not already specified one) or provide a summary of the reminder (i.e., a description of the task and a notification time, asking the user to confirm or change the proposed reminder).

[0066] Logs from automated personal assistants can be analyzed to extract information related to reminders. The logs can be preprocessed to include only reminders from the US market, for example. Other datasets (such as email corpora or other sources of natural language usage) can be used to generate an initial ITE model and / or update the ITE model. In an embodiment, the analysis can focus on specific reminders (such as time-based reminders) and remove location (e.g., "Remind me to do X when I am at Y") and less common person-based reminders (e.g., "When I see Y, remind me to give X"). The preprocessing can further focus on those reminders confirmed by the user. In an example of such preprocessing of logs from an automated personal assistant, a sample size of 576,080 reminders from 92,264 users was obtained over a two-month period. For each reminder, the reminder task description and notification time can be extracted. The creation time can be extracted based on the local time of the user's device. Each reminder can be represented by:

[0067] r task : The textual task description of the reminder, which may refer to a phrase encoding a future task or action to be taken, as specified by the user.

[0068] r CT : The creation time of the reminder, which may refer to the time when the user coded the reminder. CT It can be represented as a discretized time value as described below. The timestamp can be extracted from the client's device.

[0069] r NT : The notification time that is set to trigger the alarm. This data may refer to the time when the user wants to be reminded about a future task or action and can be used with r CT The same discrete method is used to represent .

[0070] r ΔT Subtracting the creation time from the notification time can produce a time delta, which can refer to the delay between the creation and notification times of the reminder. Reminders with smaller time deltas can represent short-term or immediate tasks (e.g., "Remind me to take the pizza out of the oven"), while reminders with larger time deltas can represent tasks that are planned further in time (e.g., "Remind me to make a doctor's appointment").

[0071] In an embodiment, common reminders are identified. Common reminders may refer to reminders that are frequently observed across multiple users. Common task types may be identified using data-driven and / or qualitative methods to extract and identify common task types.

[0072] As disclosed herein, frequent task descriptions can be extracted. First, common task descriptions can be extracted by utilizing the predicate (e.g., verb+object) structure described previously. To ensure that the base task descriptions represent a wide range of tasks, the dataset can be narrowed or filtered to include those descriptions that begin with a verb (or multi-word phrase verb), occur, for example, at least 500 times across at least ten users, and have at least 5 objects. In the example dataset, this can produce a set of 52 frequent verbs, which can include 60.9% of the reminders in the example dataset. The relatively small number of verbs covering most of the reminders in the example log data can indicate that there may be many common task types that cause reminder creation. To further analyze the base tasks, the most common objects can be included, for example, by deleting objects that are observed less than five times with the verb. This can produce a set of 2484 unique task descriptions (i.e., verb+object), covering 21.7% of the example log data.

[0073] Common tasks underlying frequent task descriptions can be identified and / or categorized into broader task type classifications. As examples, dimensions that typically separate tasks can be: whether the task represents an interruption or continuation of user activity, the context in which the task is to be performed (i.e., at home, at work), and the (expected) duration of the task. Based on this analysis, frequent task descriptions can be labeled as belonging to one of six broad task types with several subcategories.

[0074] The following describes six task types identified in the example dataset and provides examples of associated verb+object patterns. Additional task types can be identified by performing the above analysis on different datasets. Example objects are shown in descending order of frequency, starting with the most common. Verbs may not be uniquely associated with a single task type, but verb+object pairs can determine the task type (compare, for example, "start the dishwasher" to "start cooking").

[0075] The first task type identified in the example dataset may refer to going somewhere. Approximately 33.0% of frequent tasks may refer to the user moving from one place to another. Within this task type, two subtypes may be distinguished. The first subtype may refer to running errands (83.2% of the dataset), where a reminder may refer to performing a task at a location (e.g., “pick up milk”). Running errands may represent an interruption in user activity, but the scale of the task is relatively small. For example, it may be a task that simply takes up the user's availability. The second subtype may be broader and may represent tasks characterized by context switching (16.8% of the dataset), for example, moving from one context or activity to another (“going to work”, “leaving for the office”), which may have a larger impact on the user's availability. Examples of these subtypes are provided in Table 4 (Running Errands) and Table 5 (Switching Contexts), respectively.

[0076] Table 4

[0077] Example Verbs Example Object catch [something] Laundry, lunch, headphones take [something] Battery pick up [something / someone] Laundry, people, pizza buy [something] Milk, flowers, coffee, pizza bring [something] Laptop, lunch, phone charger put down [something] Cars, dry cleaning, prescriptions Return [something] library books

[0078] Table 5

[0079] Example Verbs Example Object leave(go) somewhere House, work, airport Come [somewhere] House, back to work, come in in [somewhere] At work, at home go somewhere Gym, work, home, appointments Stop at [somewhere] Banks and retailers have(go)[something] Work, appointments

[0080] The second most common reminder type in the example dataset can be described as routine chores (23.8% of the example data). Two subtypes can be identified: repetitive chores (66.5%) and independent chores (33.5%). Both types represent smaller tasks that briefly interrupt the user's activities. Tables 6 and 7 provide examples of these subtypes, respectively; other subtypes may be identified in other datasets.

[0081] Table 6

[0082]

[0083]

[0084] Table 7

[0085] Example Verbs Example Object write [something] Checks, letters, thank you notes change [something] Laundry, oil, air filter cancel [something] subscription Order [something] Pizza, flowers Replace [something] Book, driver's license, passport order [something] Hotels, flights mail [something] Letters, packages, checks submit [something] Timetables, time cards, fees Fill in [something] Application, Timetable, Forms Print [something] Tickets, articles, boarding passes pack [something] Lunch, gym clothes, clothes

[0086] Another common task identified in the example dataset may refer to a reminder to contact (e.g., "call," "make a phone call," "text message") another individual, such as a person (e.g., "Mom," "Jack," "Jenny"), an organization / company, or other ("hairdresser," "doctor's office"). Two subtypes can be identified: the majority of the example data can be referred to as general, unspecified communication (94.7% of the example data) (e.g., "call Mom"), and a smaller portion (5.3% of the example data) can be referred to as coordination or planning tasks (e.g., "make a doctor's appointment"). Both subtypes can represent tasks that briefly interrupt the user's activity. Tables 8 and 9 provide examples of general subtypes and coordination subtypes, respectively.

[0087] Table 8

[0088] Example Verbs Example Object send [something] Email, text, report Email [something] Dad, Mom Text [something] Mom, Dad Call [something] Mom, Dad tell [someone] [something] My wife I love her, happy birthday mom

[0089] Table 9

[0090] Example Verbs Example Object Set [Appointment] Doctor's appointment Make a reservation Doctor's appointments and reservations arrange [appointment] Haircuts and doctor's appointments

[0091] A fourth common task that can be identified is management of ongoing external processes. These types of tasks comprise 12.9% of the sample dataset. For example, manipulation of ongoing external processes can refer to tasks where a user monitors or interacts with something such as laundry or an oven. These tasks are characterized by their brief interruption of the user's activities and may be less extensive than performing household chores. Table 10 provides examples of such tasks.

[0092] Table 10

[0093]

[0094]

[0095] A fifth common category of tasks that can be identified relates to the management of ongoing user activities. This category of reminders is similar to the previous categories; however, rather than the user interacting with an external process, tasks can reflect changes in the user's own activities. These tasks can result in increased costs to user usability and cognitive load. In the example dataset, three subtypes can be identified: prepare (31.4%), start (61.4%), and stop activity (7.2%). Tables 11-13 provide examples of these three subtypes, respectively.

[0096] Table 11

[0097] Example Verbs Example Object prepare [for / for] Work, home

[0098] Table 12

[0099] Example Verbs Example Object Start [some activity] Dinner, cooking, learning make [something] food, breakfast, grocery list to carry out [something] Take a bath and rest play [something] games, consoles, basketball watch [something] TV shows, sports games

[0100] Table 13

[0101] Example Verbs Example Object stop [a certain activity] Read, play complete [something] Homework, laundry, taxes

[0102] Another frequent reminder type may refer to consuming something, most commonly food (e.g., "Eat lunch") or medication (e.g., "Take medicine"). These tasks can be small and range from brief interruptions (e.g., "Take pills") to longer interruptions ("Eat meal"). Table 14 lists examples of this reminder type.

[0103] Table 14

[0104] Example Verbs Example Object eat (take) [something] medicine eat [something] Lunch, dinner, breakfast, pizza eat (have) [something] Lunch, snacks, breakfast

[0105] The time patterns of reminders can be analyzed to determine, for example, when a user creates a reminder, when a reminder is set to notify the user, and the average delay between creation and notification times for different reminders. Based on these determinations, new capabilities can be provided to the reminder service, such as providing information about the likelihood of when certain tasks tend to occur, suggesting notification times (time slot filling), predicting (follow-up) tasks, and / or proactively blocking time on the user's calendar.

[0106] Common time patterns of reminders can be analyzed by discretizing time by dividing each day into, for example, six four-hour buckets, which are (i) late night (e.g., 00:00-04:00), (ii) early morning (e.g., 04:00-08:00), (iii) morning (e.g., 08:00-12:00), (iv) afternoon (e.g., 12:00-16:00), (v) evening (e.g., 16:00-20:00), and (vi) night (e.g., 20:00-00:00). The division of time into days of the day combined with the number of days of the week can produce a 7 by 6 matrix M, whose columns can represent days and whose rows can represent times. Each r CT and r NT can be represented as a cell in the matrix M, that is, M i，j , where i corresponds to the day of the week and j corresponds to the time of day. In addition, M can be distinguished at specific days and times. CT and M NT , which corresponds to a matrix whose cells may contain reminders that are created or set to be notified. Each reminder can be represented as an object r∈M, which has the properties described above, such as: the task description of the reminder (r task ), creation time (r CT ), notification time (r NT ) and time increment (r ΔT ). The time pattern may be checked based on, for example, the number of reminders created, their notifications set and / or per unit. CT and M NT Conditional probabilities are calculated on the cells in , where the creation or notification time of a reminder can be conditioned on, for example, the task type, time, and / or words in the task description.

[0107]

[0108]

[0109] The conditional probability of a notification or creation time, given a word from the task description, can be estimated by dividing the set of reminders containing the term w (created or whose notifications were set at time x) by the total number of reminders containing that word (see, e.g., Equation 1 and / or Equation 2). NT or M CT The probability of each unit in is used to generate the probability distribution on the matrix M (that is, ).

[0110]

[0111] Given the creation time, a probability distribution for the notification time of the reminder can be generated (see, for example, Equation 3) to determine the common pattern of the time period between the creation and notification of the reminder. CT This probability distribution is calculated by dividing the number of reminders in each cell whose notification is set to be triggered at time X by all reminders in the cell.

[0112] For example, for a given subset (e.g., |r CT ∈R: w∈r|or|r CT ∈R:r CT =X|), we can collect the counts and plot the reminder r ΔT to analyze the delay between setting and executing reminders.

[0113] Figure 1 The diagram shows M CT and M NT The creation time P(rCT) and notification time P(r NT ) is an example of a prior probability of . Figure 1 Example dataset based on 576,080 reminders. The first plot 110 is the distribution of reminder creation time, and the plot 120 on the right is the distribution of reminder notification time for all reminders in the example dataset (n=576,080). In the example dataset, scheduling (reminder creation) occurs most often later in the day, rather than during office hours (morning and midday). This may be a reflection of the user's availability. For example, a user may have more time to interact with their mobile device in the evening. Additionally, the end of the day can be considered a natural time period for "closing the day," e.g., looking back and forward at completed tasks and future tasks. Other datasets may show similar or different trends. The disclosed process for analyzing a dataset is not limited to Figure 1 The specific trends shown in the figure.

[0114] Figure 1The plot 120 on the right shows a slightly different pattern with respect to notification times. In the example dataset, users perform tasks (i.e., notification triggers) throughout the day, from morning to evening. This may indicate that users prefer to be reminded of tasks throughout the day in different contexts (e.g., at home and at work). This may also be reflected in the task type classification, where a task may be associated with two contexts. In the example dataset, slightly more notifications are triggered on weekdays than on weekends, and more notifications are triggered at the beginning and end of the work week. This observation may be attributed to the same phenomenon used for reminder creation. Users may tend to adopt reminders for activities that switch between mid-week and weekend contexts. Finally, a comparison of the two plots shows that notification times are slightly more unevenly distributed than creation times, for example, users create reminders late at night when notifications are relatively less likely to be triggered.

[0115] Next, to determine how far in advance users typically plan, the sample dataset is analyzed for the delay between reminder creation and notification. Figure 2 As shown in the figure. The upper histogram shows distinct spikes around five-minute intervals, which may be due to reminders with relative time indications (e.g., "Remind me to pick up my pizza in five minutes"). These intervals may be easier to imagine than more vague time ranges (e.g., "Remind me to pick up my pizza at 6:34 PM."). Figure 2 The middle and bottom histograms in the example dataset clearly illustrate that most reminders in the example dataset have short delays. About 25% of the reminders in the example dataset are set to notify within the same hour (middle histogram), and about 80% of the reminders are set to notify within 24 hours (bottom histogram). There is a small peak at about 8-9 hours in the second plot, which can be explained by reminders that span the evening (e.g., created at the end of the day to be notified early the next day) or across "workdays" (e.g., created in the morning to be notified at the end of the day).

[0116] In summary, analysis of the sample dataset shows that, on average, users tend to set plans in the evening and execute them throughout the day. Furthermore, most of the tasks that drive reminders tend to be short-term tasks that are to be executed within the next 24 hours.

[0117] Next, we analyze the example dataset to determine whether different task types are characterized by different temporal patterns that differ from the global patterns seen in the above analysis. To this end, we labeled the example dataset reminders with a set of 2,484 frequent reminders (e.g., unique task descriptions). This yielded a subset of 125,376 reminders with task type labels that were used for analysis. Other datasets for automated personal assistants may have different percentages and may reveal different patterns for a given set of users.

[0118] The probability distribution of reminder creation time for each task type is analyzed, namely P(r CT |Task Type). Based on this analysis of the sample data and the distribution of results for each task type, two broader groups can be identified: By task type, reminders can be created primarily in the morning and midday boxes (roughly corresponding to typical office hours), or outside of these boxes. Figure 3 Two types of examples are shown: "activity" reminders and "going somewhere" reminders tend to be created primarily during office hours, while "communication" reminders and "housework" reminders tend to be created more easily in the evening. In the example data, reminders related to activities tend to be more frequent on weekends.

[0119] Reminder notification times can also be analyzed by task type, i.e., P(rNT | Task Type). Similar patterns were observed in the sample data used for this analysis, as with reminder creation times. Two types of tasks were observed: tasks set to notify during office hours, and tasks triggered outside of these hours. Figure 4 The diagram shows examples of both types of tasks. "Communicate" and "Go" belong to the former type, while "Housework" and "Manage an ongoing process" belong to the latter. The nature of the tasks provides insight into this distinction: the former tend to be related to work-related tasks (communication, work-related errands), while the majority of the latter tend to represent activities more common in a domestic setting (cooking, cleaning).

[0120] right Figure 5 Further analysis of the communication task subcategories in the dataset illustrates the differences between "Communication / General" and "Communication / Coordination." "Communication / General" tasks tend to be more evenly distributed, while "Communication / Coordination" tasks tend to be more concentrated during office hours. "General" subtasks can also trigger relatively more reminders on weekends, while "Coordination" subtasks tend to be more concentrated on weekdays. The different patterns in the example dataset indicate that these subcategories represent different types of tasks.

[0121] The differences in the lead time between reminder creation and notification were analyzed. Figure 6An overview of the distribution of reminder delays per task type for the example data is shown in Figure 2. In general, the lower the boxplot is on the y-axis, the shorter the lead time, i.e., the shorter the delay between creating the reminder and executing the task. This can be illustrated by comparing, for example, the plot for "Manage an ongoing process" with the "Go" or "Communicate" task types. In the example data, executing the "Manage an ongoing process" task appears to be planned with a much shorter lead time than tasks of other types. Given the nature of the tasks, where an ongoing process tends to represent close monitoring or inspection of a process (e.g., cooking or cleaning tasks), it is logical that the delay would be on the order of minutes rather than hours. "Communicate / coordinate" has the largest average delay in the example data, i.e., it is the task type that people plan the farthest in advance.

[0122] like Figure 7 A more detailed examination of the differences between the "Communication" subtasks, as presented in

[15] , reveals that, for example, the "Communication / General" subtask tends to be more likely to be performed with shorter lead times, as indicated by the peak at 0 hours in the upper histogram. The "Communication / Coordination" subtask is more likely to be performed the next day, as indicated by the peak around the 12-hour mark in the bottom histogram of the example data. Much like the observations made above, the differences in the patterns between the two "Communication" subtasks indicate that the distinction between the subtypes is meaningful. Qualitative analysis reveals differences not only at the semantic level but also in temporal patterns.

[0123] Based on the above analysis, the task description can show different time patterns. For example, a reminder containing the word "pizza" may be triggered around dinner time. The existence of these time patterns can be used to create reminders or predict notification times. Given a term w, the task description can be calculated for M CT or M NT The conditional probability of the unit in (see Equation 1 and Equation 2).

[0124] Figure 8 An example of the creation time and notification time of a task description containing the word "church" or "appointment" is shown, and the creation time and notification time of the task description are based on example data. Figure 8 The plot for "Appointments" shown in indicates a strong pattern around the morning and midday boxes representing office hours. Reminders containing "Church" also show a clear pattern in the sample data; they tend to be created primarily from Saturday evening to Sunday morning, and tend to be set to notify early Sunday morning and morning.

[0125] Figure 9The figure shows an example of a comparison of the average latency of reminders containing the words "appointment" and "laundry" in an example dataset. On average, "appointment" reminders tend to have longer latencies, which is reflected in the nature of the task (it may involve other individuals and therefore require more planning), while "laundry" reminders tend to be more likely to reflect short-term tasks (which can be performed individually). Therefore, different temporal patterns can be observed in the task description words.

[0126] The correlation between reminder creation and notification time can be analyzed and / or determined. Given its creation time, the probability P(r NT |r CT ). Figure 10 An example of the probability distribution of a given reminder notification time given its creation time is illustrated. Figure 10 The plots shown in provide an example of how similar reminders look across different creation times. For example, reminders tend to be most likely to trigger their notifications within the same unit or the next unit, confirming the previous observation from the sample data that most reminders are short-term (i.e., same unit). As the creation time of a reminder moves later in the day, it tends to be more likely to be set to notify the next day. Additionally, Figure 10 The "Friday Evening" plot in

[15] illustrates how a reminder created on Friday evening has a small but significant probability of triggering its notification on Monday morning (i.e., the reminder spans the weekend). These patterns show that the delay between reminder creation and notification time tends to be low on average, but the length of the delay may be unrelated to the creation time.

[0127] The above analysis illustrates examples of four types of temporal patterns in reminder creation and notification times (aggregate patterns, task type related, word-based, and time-based). Based on these observed patterns, a predicted task can be determined. Specifically, the day of the week when the task is most likely to occur (i.e., the predicted r NT ).

[0128] The task of predicting the day of the week when a reminder is set to be notified can be set as a multi-class classification, where each day of the week corresponds to a class. The input of the prediction model can be the task description of the reminder (r task ), creation time (r CT ), and the target class is the notification time of the day of the week (r NT The predictive power of the patterns identified via the word-based and (creation) time-based features in the above analysis can be measured. Specifically, for the word-based features, a tuple of word features can be extracted, and the time-based features can correspond to R CT The time of day (rows) and day of the week (columns) and minutes since the start of the week.

[0129] For example, gradient boosted decision trees can be used for classification. Other similar methods capable of large-scale learning can be employed. For example, gradient boosted decision trees can effectively handle nonlinearities in feature space and heterogeneous features. To address the multi-class nature of our problem, a one versus all classification strategy can be used, where, for example, 7 binary classifiers can be trained and the prediction with the highest confidence can be output as the final prediction. The accuracy can be compared to a more competitive baseline (baseline-same day) of randomly selecting a day of the week (e.g., an accuracy of 1 / 7 ≈ 0.1429) and predicting that the notification will be created on the same day.

[0130] For this example analysis, six months of data were sampled from the automated personal assistant (e.g., January to June 2015). All data was filtered according to the above process, resulting in a total of 1,509,340 reminders. This data was split sequentially: the first 70% (approximately January 1, 2015, to May 7, 2015) formed the training set, and the second 30% (approximately May 8, 2015, to June 30, 2015) formed the test set. The first two months of the training set were used for the above analysis and for parameter tuning before retraining on the entire training set.

[0131] The following describes the prediction performance on the proposed test set. Specifically, the macro and micro average precision over the classes (macro and micro, respectively) are provided on the test data. Three systems are compared: one using time features based on the creation time of the reminder (time only), one using word features (word only), and a model using both types of features (full model). Statistical significance is tested using t-tests, comparing the prediction models to the baseline - same day. The symbols ▲ and ▼ indicate statistically significant differences (greater than and worse than the baseline, respectively) at α = 0.01.

[0132] Table 15 shows the results of the prediction task on the test dataset. The baseline, which predicts that the notification time will be on the same day as the creation time, performs much better than random (at 0.1429). This indicates that in the test data users mainly set reminders to plan short-term events. The "time only" model significantly improves the baseline, indicating that the reminder creation time helps further improve prediction accuracy. As mentioned earlier, tasks scheduled late at night tend to be more likely to be executed on different days, and the use of creation time can help exploit this and more general patterns. The model that uses only features based on the task description (i.e., "only words") performs better than random, but does not outperform the baseline. However, when combined with the time model (i.e., the "full model"), an increase of 8.2% relative to the "time only" model is observed on the test data. From the previous analysis of the test data, it can be concluded that the creation time provides the most information for predicting when the task will be executed, and the task description provides important additional information (mainly when the description is used in conjunction with the creation time of the reminder).

[0133] Table 15

[0134] run micro Macro Reduced errors Complete model <![CDATA[0.6788 ▲ ]]> <![CDATA[0.6761 ▲ ]]> +0.3381 Time only <![CDATA[0.6279 ▲ ]]> <![CDATA[0.6258 ▲ ]]> +0.2333 Words only <![CDATA[0.1777 ▼ ]]> <![CDATA[0.1772 ▼ ]]> -0.6944 Baseline - Same Day 0.5147 0.5165

[0135] Based on the above analysis, ITE is commonly used for various user inputs (e.g., emails). In addition, the frequency of ITE tends to follow a power-law distribution, where the most common expressions occur exponentially more often than the least common expressions. Therefore, automated personal assistants should be effective in resolving long-tail time expressions. Heuristic methods can be developed to resolve the most common ITEs (such as "next week," "this morning," "this afternoon," "this weekend," and "tomorrow morning"). However, more complex time expressions (e.g., a little over a week later, no later than tomorrow noon, around the end of this week or early next week, etc.) may encounter difficulties in extracting ITE and for downstream interpretation and actions (e.g., triggering reminders). The disclosed embodiments even provide a mechanism for predicting at least one time interval for these long-tail time expressions.

[0136] As previously indicated, ITE can exhibit temporal periodicity. For example, "later this week" may most often occur on Mondays and Tuesdays, while "early next week" may most often occur on Fridays. Therefore, the model can consider the temporal context of the user input to better predict the at least one time interval to which the user input refers.

[0137] This was investigated by analyzing the time attributed to ITE in the example dataset. It was observed that the interpretation of ITE varies based on the time of day at which the ITE is considered. For example, when the given reference time is early in the work week, the interpretation of the phrase "next weekend" is characterized by a bimodal distribution ( Figure 19Here, some interpret the phrase to mean the nearest upcoming weekend, while others interpret the phrase to mean the weekend ending next week. However, by Wednesday, most of the ambiguity disappears. Similarly, Figure 20 The results obtained for the interpretation of "later this week" are summarized. Consistent with intuition, "later this week" tends to be most often interpreted as meaning the remainder of the work week (ending on Friday). The data also indicates that from Monday to Wednesday, the distribution of interpretations peaks on Friday. However, starting from Thursday, the peak in the time of interpretation shifts to Saturday. It is possible that the users in the example data set their estimates because, starting from Thursday, Friday is most naturally referred to as simply "tomorrow". These observed periodic regularities mean that date and time can be key context for correctly interpreting ITE. Based on the day of the week, the virtual assistant can adjust its expectations when obtaining ITE as input from the user. For example, when a user says "later this week" on a Friday, this is a rare event based on its past frequency of occurrence, and the system can verify with the user by asking the user what they really mean. Similarly, if the user says "next weekend" on a Monday, this is very ambiguous according to the interpretation model, and the system can also proactively verify with the user.

[0138] The foregoing description provides examples of how imprecise time expression models may be generated and / or consumed by a device or service. An ITE model may be generated based on logs obtained from an automated personal assistant, an email corpus, or any other similar type of data set. The logs may be analyzed to determine time expressions of interest and to identify imprecise time expressions, such as those provided in the various tables above. Based on the identified imprecise time expressions, a probability distribution may be generated indicating the times to which a given percentage of users believe the imprecise time expression corresponds or would like the imprecise time expression to correspond. As in Figures 1-10 As can be seen in the examples provided in , these probability distributions can be represented in a variety of ways. In addition, classification and / or regression analysis not specifically stated above can be used to modify the way in which the model is trained on a given data set. The model can be updated based on the preferences of a particular user or the preferences of a particular group of users. For example, if the user is in a university setting (e.g., identified by an IP address), the model may associate the word "morning" as referring to 11:00 am rather than 8:00 am. The process of generating the ITE model can be performed or calculated by a processor of the client device and / or server (e.g., Equations 1-3). Later on Figure 15-16 Describes various configurations of processors and client / server devices.

[0139] Similarly, a particular user may indicate that the user considers the word "morning" to be 9:30 a.m. For example, a user may indicate a particular time by answering a confirmation dialog that presents one or more predictions. Figure 17 An example of a question that may be displayed to a user and that may receive a direct response from the user regarding what the terms "later this week or early next week" mean to the user is shown. Figures 18A-18C An example of aggregated responses across multiple users (eg, a group of users) is shown. Figure 18A An example distribution of responses to the term "early next week" is shown. Figure 18B An example distribution of responses to the term "next weekend" is shown. Figure 18C An example distribution of responses to the term "later this week" is shown. Figures 18A-18C In the example, the x-axis represents the user's response to the question (e.g., Figure 17 ) and the y-axis represents the time at which the response was made. It can be seen that, for example, for "early next week," when the time of the user response is approximately the 11th, Monday to the 16th, Saturday, most people believe that the phrase belongs to the time interval from approximately the 18th, Monday to the 20th, Wednesday. However, if the user response occurs on the 17th, Sunday, the phrase "early next week" appears to have a bimodal distribution corresponding to approximately the 18th to the 20th and the 24th to the 27th. Therefore, the prediction for "early next week" is an example of an ITE that can have a prediction for one or more time intervals that vary based on the time at which the text or utterance of the ITE is detected. Therefore, when the system detects an utterance or text containing the phrase "early next week," it can predict the time interval based on the time of the utterance or text and the ITE model (for example, in this case, Figure 18A ) to generate predictions.

[0140] As previously described, the ITE model can be updated on a group or specific user basis. In the event that the user does not indicate a specific time interval for a specific time, the aggregated responses from other users can be used to generate a prediction for the time interval corresponding to the specific user input (e.g., "next weekend"). Even if the user has indicated a time, the system can weight a specific distribution to reflect the user's preferences rather than returning a discrete time value. Updates to the model can be performed regardless of whether the client device is online or offline, and can be performed in batches or in real time (e.g., with a persistent connection to the Internet or a cloud service). For example, updates can include additional ITE and / or new / updated probability distributions that can be detected in the user input.

[0141] Figure 11An example of an embodiment is shown that provides a process for providing a client device (or other computer) with a prediction of an imprecise time expression 1100. At 1110, user input can be received from the client device. The user input can correspond to voice and / or text input by the user entered, detected, or received by the client device. For example, the user can issue a command such as "Remind me to go to the store this morning," or the user can compose a note to schedule an appointment with an imprecise time expression (which the system can automatically complete as disclosed herein).

[0142] ITEs can be detected in user input. For example, a client device (e.g., a laptop, smartphone, tablet, wearable device (such as a smartwatch), server, etc.), a device on which an automated personal assistant is operating, and / or a server can identify one of the ITEs by comparing the user input with known time expressions or imprecise time expressions. As another example, the user input can be scanned for time expressions (e.g., weekend, afternoon, etc.) to detect imprecise time expressions. In some instances, the user input can be filtered to remove words that are not related to time expressions. As previously described, candidate ITEs can be identified from existing datasets (e.g., interaction logs, corpora of emails, etc.).

[0143] At 1120, a prediction of one or more time intervals referred to by the user input may be generated based on the ITE model. The user input may include at least one of an ITE, contextual text, and / or a time of interaction. As described above, an ITE may be detected in the user input (e.g., "before next Monday," where the word Monday is a time expression that may be rendered imprecise by the preceding "before next"). Similarly, the contextual text may refer to a phrase that lacks a specific ITE, such as the phrase "mowing the lawn," which is typically associated with an activity occurring on the weekend. The time of interaction may refer to, for example, Figure 17 and Figures 18A-18C . It is worth noting that the user input can have a combination of these features, such as "mow the lawn at the beginning of next week," which may return a different prediction than simply "mow the lawn." Examples of predictions have been described above. The predictions can be calculated by a processor and / or stored in a database, etc., on the client device and / or server.

[0144] An ITE model can be generated based on an initial association of an ITE with one or more time intervals and / or distributions as described above (e.g., via manually generated heuristics, crowdsourcing, etc.). The association of an ITE with an ITE model can refer to a specific ITE with a predicted time interval. This association can be computationally represented and / or utilized in the form provided in Examples 1-14 above. Candidate ITEs can be enumerated from existing datasets (e.g., interaction logs, corpora of emails, etc.). ITEs can be detected using, for example, a template matching process and / or a reconstruction process. In the template matching method, common prefixes or suffixes for a set of known temporal entities can be determined (e.g., learning the word(s) around "Monday" in a message such as "before Monday morning"). The discovered prefixes / suffixes can then be used to help discover new temporal entities (e.g., "before ____ morning" can be detected to obtain "before tomorrow morning"). In the reconstruction method, for example, reconstructions of failed utterances of an automated personal assistant can be detected. For example, a conventional assistant may not be able to recognize "will it rain on Halloween?" The user can rephrase the utterance as "Will it rain on October 31st?" Both utterances begin with "Will it rain...", but the second utterance provides a precise date, allowing the system to infer that October 31st corresponds to Halloween and that Halloween is a temporal entity. According to the embodiments disclosed herein, other methods for extracting and identifying imprecise temporal expressions can be utilized. Based on human-generated heuristics, crowdsourcing, etc., the extracted ITEs can be associated with one or more time intervals and / or distributions over time and date. For example, user surveys can be conducted to determine how users interpret specific imprecise temporal expressions. The above techniques can also be used to identify other phrases that can refer to events, etc. that do not have ITEs (e.g., "mowing the lawn").

[0145] return Figure 11 , the system can generate predictions (e.g., beliefs) based on this mapping of when imprecise temporal expressions are likely or will occur. For example, Figure 4 and Figure 5 As shown in , when the user input is to run an errand, based on the probability distribution for the example model, the user may indicate that the user wants to perform the errand in the afternoon. The system may select a specific time, such as between 1:00 PM and 3:00 PM, which corresponds to the chore. The model may further reflect something like Figure 4 and Figure 5The discrete times in the window are selected based on a time-based probability distribution of . In some configurations, the user may have provided feedback to the model indicating that the user prefers that tasks associated with "noon" refer to 1:00 p.m., in which case the system may select that time. Similarly, other imprecise time expressions like "early next week," "later this week," or "next weekend" can be parsed based on the model. The distribution may vary based on the context of the situation, such as the time of day, the current day, the month, etc. For example, if the user instructs the device to generate a reminder for next weekend and the current day is Saturday, the system can recognize that "weekend" refers to the time between 5:00 p.m. on Friday and 5:00 p.m. on Sunday, and that the user intends to make a reminder next weekend given the current day.

[0146] At 1130, the user input may be associated with the generated prediction. For example, if a user has requested an automated personal assistant to schedule chores for next weekend, a reminder for the chores may be generated, causing the reminder to surface on the user's client device at or near the predicted time (e.g., alerting the user via haptic feedback, visual, or auditory signals). For example, if the confidence level of the prediction is low, the prediction may be displayed on an ambient display (such as a smartphone's lock screen or a smartwatch's face). Associating the prediction with the user input may mean storing a time indication (e.g., at least one interval) along with the user input (e.g., time and task), as well as other data about the prediction (e.g., confidence level of the prediction, time / date of the user input on which the prediction is based), where the time indicates the triggering event that caused the task to surface. As another example, a user may be conversing with a colleague on an instant messaging system or text messaging system. The user may communicate that she would like to meet with the colleague this weekend. The personal assistant may determine an appropriate time for the meeting based on both the user's and her colleague's calendars and the system's interpretation of the imprecise time expression "weekend."

[0147] Finally, process 1100 may include providing the prediction to the client device and / or server at 1140. Providing may refer to displaying auto-complete selections, timelines, reminders at appropriate times, scheduling of appointments, time intervals, probability distributions, and other mechanisms for surfacing predictions. As previously described, the client device and / or server may provide the prediction to other applications that utilize the prediction for services that are not immediately exposed to the user (e.g., the prediction is a time interval provided to a scheduling application). Figure 12An example interface 1200 of a note-taking or memo-taking program operating on a client device is shown. As can be seen, text entry box 1210 does not contain any text. Based on the time of day, day of the week, and so on, the system can predict the imprecise time expression the user might want to use. Selecting auto-complete 1220 saves the user from having to type the entire expression. For example, the list might suggest "this weekend" for Friday, "early tomorrow" for evening, "later today" for morning, and so on. Figure 13 1300 is an example interface of a note-taking program or memo program operating on a client device, in which text has been entered into a text input box 1310. Based on the user's input, suggestions 1320 may be updated accordingly. In this example, mowing the lawn may be identified as a "chore" and something that may be done in the evening or on the weekend. Thus, Figure 12 and Figure 13 Can be Figure 11 An example of how the process illustrated in provides predictions.

[0148] In some instances, a series of predictions may be presented to the user, and the system may generate a request to receive user input to confirm the predictions, such as in Figure 13 In some instances, the user can modify the prediction by selecting a different time or making a correction to the time prediction. The system can adjust the initial imprecise time expression model to reflect that the particular user believes that the imprecise time expression corresponds to a different discrete time or a different time range. For example, the probability distribution can be weighted so that the time indicated by the user is the time associated with the imprecise expression. As another example, the sorted list can be reordered based on the user's indication of what the ITE is to correspond to. The list can be presented to the user on a display of the client device. The list can be color coded based on the ITE model to convey the likelihood that the user wants or needs the information at a given time. The disclosed embodiments can predict whether / when a user is likely to want to receive information given the urgency and uncertainty of the information and present the information via appropriate means (e.g., updating an ambient display, sending a notification, etc.). The prediction can be displayed to the user in a variety of ways and is not limited to Figure 12-14 The specific example illustrated in . Additionally, the ITE model may be updated in a variety of ways other than those explicitly described herein.

[0149] Figure 1414 is an example interface 1400 for a note-taking program or memo program operating on a client device, in which a user has entered the phrase "file a patent application this week" in a text input box 1410. The word "this week" can be detected as an imprecise time expression. The system's interpretation (e.g., a prediction) of the imprecise time expression can be provided as shown at 1420. For example, the prediction can be based on a 95% confidence interval (upper and lower bounds) for the interpretation of "next week." In addition, the prediction can be automatically added to, for example, the user's calendar and / or task list. Thus, the prediction can be provided (e.g., surfaced) at the appropriate time. In some instances, the prediction can be displayed to the user on the client device at a time corresponding to the user input (e.g., "in the morning" can be displayed at 9:00 a.m. the following morning).

[0150] Figure 21 is an example of a note taking application or memo application of an automated personal assistant that displays a ranked list of tasks based on various predictions generated from user input according to embodiments disclosed herein. As previously mentioned, the automated personal assistant can be programmed to recognize a wide range of ITEs. Figure 21 When creating an event in the application shown in the figure, the user can specify a time period. A short list of events related to a given moment can be displayed by time, such as Figure 21 For example, a user can specify the time to be displayed by including a time expression in the text of the item (such as "coffee after get off work"), by selecting from a list of common expressions (e.g., "today," "tomorrow," etc.), or by selecting a specific date or time from a calendar or time picker applet.

[0151] User feedback may be implicit or explicit. For example, Figure 14 In

[15] , if the user manually corrects the start and / or end times, this can be used as training data to improve the model for the input string "this week." This type of feedback is an example of implicit feedback (e.g., taking or not taking a corrective action). Other forms of implicit feedback can be similarly utilized to update the ITE model. Figure 22 is an example of explicit feedback that can be provided by a user. For example, a user can tap and hold item 2210 in display 2200. This can cause dialog box 2220 to appear and enable the user to provide explicit feedback to the system. Other forms of explicit feedback can also be received, such as clicking a "thumbs-up," ignoring a notification, swiping a notification in a specific direction, gestures, etc. Feedback can be aggregated across multiple users to update the ITE model.

[0152] In an embodiment, the system includes a database for storing an imprecise temporal expression model and a processor communicatively coupled thereto. Figure 15 and Figure 16The database and processor configuration are described in more detail. The database may refer to a local storage device, memory, server or cloud-based database, etc. The imprecise time expression model can be compressed to accommodate the reduced amount of memory available in certain devices (such as wearable devices). For example, the processor may be configured to execute Figure 11 The process shown in the figure.

[0153] Embodiments are disclosed that include a non-transitory computer readable medium having computer readable instructions stored thereon, the computer readable instructions being executable to cause one or more processors to perform operations. The operations may be similar to those described above with respect to Figure 11 Describes the operation.

[0154] Embodiments of the presently disclosed subject matter can be implemented in and used with a variety of components and network architectures. Figure 15 2 is an example computer 20 suitable for implementing embodiments of the presently disclosed subject matter (e.g., an electronic device such as a smartphone, tablet, laptop, personal computer, a wearable device such as a smartwatch, a server, etc.). A client device as described above may be a computer 20. The computer 20 includes a bus 21 that interconnects major components of the computer 20, such as a central processing unit 24, memory 27 (typically RAM, but may also include read-only memory ("ROM"), flash RAM, etc.), an input / output controller 28, a user display 22 (such as a display screen via a display adapter), a user input interface 26 (which may include one or more controllers and associated user input devices (such as a keyboard, mouse, etc.) and may be tightly coupled to the I / O controller 28), a fixed storage device 23 (such as a hard drive, a flash memory device, a Fibre Channel network, a SAN device, a SCSI device, etc.), and a removable media component 25 (operable to control and receive optical disks, flash drives, etc.).

[0155] The bus 21 allows data communication between the central processing unit 24 and the memory 27, which, as previously described, may include ROM or flash memory (neither shown) and RAM (not shown). The RAM is typically the main memory into which the operating system and application programs are loaded. The ROM or flash memory may contain, among other code, a basic input and output system (BIOS), which controls basic hardware operations such as interaction with peripheral components. Applications resident in the computer 20 are typically stored on and accessed via a computer-readable medium, such as a hard drive (e.g., fixed storage device 23), an optical drive, a floppy disk, or other storage medium 25.

[0156] The fixed storage device 23 may be integrated with the computer 20 or may be separate and accessible through other interfaces. The network interface 29 may provide a direct connection to a remote server via a telephone link, a direct connection to the Internet via an Internet Service Provider (ISP), or a direct connection to a remote server via a direct network link to the Internet (via a POP (point of presence) or other technology). The network interface 29 may provide such a connection using wireless technology, including a digital cellular telephone connection, a Cellular Digital Packet Data (CDPD) connection, a digital satellite data connection, etc. For example, the network interface 29 may allow the computer to communicate with other computers via one or more local networks, wide area networks, or other networks. Many other devices or components (not shown, for example, a digital camera or speakers) may be connected in a similar manner. In contrast, there need not be a network interface. Figure 15 The present disclosure can be practiced with all components shown in FIG. Components may be interconnected in a manner different from that shown. Figure 15 The operation of the computer shown in is well known in the art and will not be discussed in detail in this application. The code used to implement the present disclosure can be stored in a computer-readable storage medium (such as one or more of the memory 27, fixed storage device 23, and removable medium 25), or stored in a remote storage location.

[0157] Figure 16 An example network arrangement according to an embodiment of the disclosed subject matter is shown. One or more clients 10, 11, such as local computers, smartphones, wearable devices, tablet computing devices, etc., can be connected to other devices via one or more networks 7. The network can be a local network, a wide area network, the Internet, or any other suitable communication network or networks, and can be implemented on any suitable platform including wired networks and / or wireless networks. The clients can communicate with one or more servers 13 and / or databases 15. The devices can be directly accessible by the clients 10, 11, or one or more other devices can provide intermediate access, such as a server 13 providing access to resources stored in a database 15. The clients 10, 11 can also access a remote platform 17 or services provided by the remote platform 17, such as a cloud computing arrangement and services. The remote platform 17 can include one or more servers 13 and / or databases 15.

[0158] More generally, various embodiments of the presently disclosed subject matter may include computer-implemented processes and apparatus for practicing those processes or be implemented in the form thereof. The embodiments may also be implemented in the form of a computer program product having computer program code of instructions embodied in a non-transitory and / or tangible medium (such as a floppy disk, CD-ROM, hard drive, USB (Universal Serial Bus) drive, or any other machine-readable storage medium), wherein when the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for practicing embodiments of the disclosed subject matter. The embodiments may also be implemented in the form of computer program code, for example, whether stored in a storage medium, loaded into and / or executed by a computer, or transmitted over some transmission medium (such as on a wire or cable, through fiber optics, or via electromagnetic radiation), wherein when the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for practicing embodiments of the disclosed subject matter.

[0159] When implemented on a general-purpose microprocessor, the computer program code segments configure the microprocessor to create specific logic circuits. In some configurations, a set of computer-readable instructions stored on a computer-readable storage medium can be implemented by a general-purpose processor, and the set of computer-readable instructions can transform a general-purpose processor or a device containing a general-purpose processor into a special-purpose device configured to implement or execute instructions. Embodiments can be implemented using hardware that can include a processor, such as a general-purpose microprocessor and / or an application-specific integrated circuit (ASIC) that implements all or part of the technology of the embodiments of the disclosed subject matter in a hardware and / or firmware manner. The processor can be coupled to a memory (such as RAM, ROM, flash memory, hard disk, or any other device capable of storing electronic information). The memory can store instructions suitable for being executed by a processor to perform the technology of the embodiments of the disclosed subject matter.

[0160] For purposes of explanation, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the embodiments of the disclosed subject matter to the precise forms disclosed. In light of the above teachings, many modifications and variations are possible. The embodiments were chosen and described in order to explain the principles of the embodiments of the disclosed subject matter and their practical application, thereby enabling those skilled in the art to utilize those embodiments and various embodiments with various modifications (as may be suitable for the specific intended use).

Claims

1. A system for interpreting and managing imprecise time expressions, comprising: a memory for storing an imprecise time expression model, wherein the imprecise time expression model is trained using one or both of classification analysis and regression analysis to indicate a time corresponding to the imprecise time expression based on a type of activity associated with the imprecise time expression; a processor communicatively coupled to the memory and configured to: Receive user input; generating a prediction of one or more time intervals indicated by the user input based on the imprecise time expression model in the memory, wherein the user input includes: an imprecise time element, a type of activity associated with the imprecise time expression, and a time of interaction; associating the user input with the prediction; and In response to the user input, the prediction is provided. 2 . The system of claim 1 , wherein the processor is further configured to display the prediction at a time corresponding to the user input.

3. The system of claim 1 , wherein the processor is further configured to: presenting the forecast; and A request is generated to receive a second user input to confirm the prediction.

4. The system of claim 3, wherein the processor is further configured to: receiving an instruction to modify the forecast; and The imprecise temporal representation model in the memory is updated based on the indication.

5. The system of claim 1, wherein the user input is received in a form selected from the group consisting of voice input and text input.

6. The system of claim 1 , wherein providing the prediction in response to the user input comprises providing one or more of: a reminder, a notification, auto-completion of text, a timeline, a task list, a calendar schedule, a time interval, a ranked list of time intervals, and a probability distribution. The system of claim 1 , wherein the processor is further configured to generate the imprecise temporal expression model.

8. The system of claim 1, wherein the processor is further configured to detect imprecise temporal expressions in the user input.

9. A method for interpreting and managing imprecise time expressions, comprising: receiving user input at the client device; generating, on the client device, a prediction of one or more time intervals indicated by the user input based on an imprecise time expression model, wherein the user input includes at least one of: an imprecise time element, a type of activity associated with the imprecise time expression, and an interaction time, and wherein the imprecise time expression model is trained using one or both of classification analysis and regression analysis to indicate a time corresponding to the imprecise time expression based on the type of activity associated with the imprecise time expression; associating the user input with the prediction; as well as The forecast is provided.

10. The method of claim 9, further comprising displaying the prediction on the client device at a time corresponding to the user input.

11. The method according to claim 9, further comprising: presenting the prediction on the client device; as well as A request is generated to receive a second user input to confirm the prediction.

12. The method according to claim 11, further comprising: receiving an instruction to modify the forecast; as well as The imprecise temporal representation model in a memory of the client device is updated based on the indication.

13. The method of claim 9, wherein the user input is received in a form selected from the group consisting of voice input and text input.

14. The method of claim 9, wherein providing the prediction in response to the user input comprises providing one or more of: a reminder, a notification, auto-completion of text, a timeline, a task list, a calendar schedule, a time interval, a ranked list of time intervals, and a probability distribution.

15. The method according to claim 9, further comprising: The imprecise time expression model is generated.

16. The method according to claim 9, further comprising: The imprecise temporal element in the user input is detected.

17. A non-transitory computer-readable medium having computer-readable instructions stored thereon, the computer-readable instructions being executable to cause one or more processors to perform operations comprising: receiving user input at the client device; generating, on the client device, a prediction of one or more time intervals indicated by the user input based on an imprecise time expression model, wherein the user input includes an imprecise time element, a type of activity associated with the imprecise time expression, and an interaction time, and wherein the imprecise time expression model is trained using one or both of classification analysis and regression analysis to indicate a time corresponding to the imprecise time expression based on the type of activity associated with the imprecise time expression; associating the user input with the prediction; and The forecast is provided.

18. The non-transitory computer-readable medium of claim 17, wherein the processor is further configured to perform operations including displaying the prediction on the client device at a time corresponding to the user input.

19. The non-transitory computer-readable medium of claim 17, wherein the processor is further configured to perform operations comprising: presenting the prediction on the client device; as well as A request is generated to receive a second user input to confirm the prediction.

20. The non-transitory computer-readable medium of claim 17, wherein the processor is further configured to perform operations including detecting an imprecise temporal expression in the user input.

Images