Managing timed tasks using a digital assistant

Abstract

It provides an intelligent automated assistant. [Solution] A system and process are provided for managing timed tasks using a digital assistant. An exemplary method includes a computer system communicating with one or more input devices, which includes receiving natural language voice input via one or more input devices that includes a request for the status of a timed task; identifying an electronic device associated with the timed task based on the natural language voice input; receiving the status of the timed task; and providing an output indicating the status of the timed task.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims priority to U.S. Patent Application No. 18 / 444,324, titled "MANAGING TIMED TASKS WITH A DIGITAL ASSISTANT", filed on February 16, 2024, and U.S. Provisional Patent Application No. 63 / 470,792, titled "MANAGING TIMED TASKS WITH A DIGITAL ASSISTANT", filed on June 2, 2023. The entire content of each of these applications is hereby incorporated by reference into this specification.

[0002] This application generally relates to intelligent automatic assistants, and more specifically, to using intelligent automatic assistants to manage timed tasks.

Background Art

[0003] Intelligent automatic assistants (or digital assistants) can provide a useful interface between human users and electronic devices. With such assistants, users may be able to interact with a device or system using natural language in oral and / or text form. For example, a user can provide an utterance input containing a user request to a digital assistant operating on an electronic device. The digital assistant can interpret the user's intention from the utterance input and make the user's intention actionable as a task. Then, the task can be executed by performing one or more services of the electronic device and return a relevant output response to the user request to the user.

Summary of the Invention

[0004] Exemplary methods are described herein. Exemplary methods include, in a computer system communicating with one or more input devices, receiving a natural language voice input via one or more input devices that includes a request for the status of a timed task; identifying an electronic device associated with the timed task based on the natural language voice input; receiving the status of the timed task; and providing an output indicating the status of the timed task.

[0005] An exemplary method is a computer system that communicates with one or more input devices, which includes receiving a request via one or more input devices for a set of states corresponding to a separate set of timed tasks; identifying a user corresponding to the request; receiving the set of states; providing at least a subset of the set of states in a first order according to a determination that a first timed task of the set of timed tasks is associated with the identified user; and providing at least a subset of the set of states in a second order different from the first order according to a determination that a first timed task of the set of timed tasks is not associated with the identified user.

[0006] Examples of non-temporary computer-readable storage media are disclosed herein. An exemplary non-temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices. The one or more programs include receiving natural language voice input via one or more input devices, which includes a request for the status of a timed task; identifying an electronic device associated with the timed task based on the natural language voice input; receiving the status of the timed task; and providing an output indicating the status of the timed task.

[0007] An exemplary non-temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices. The one or more programs include instructions that, via one or more input devices, receive a request for a set of states corresponding to a separate set of timed tasks; identify a user corresponding to the request; receive the set of states; provide at least a subset of the set of states in a first order according to a determination that a first timed task of the set of timed tasks is associated with the identified user; and provide at least a subset of the set of states in a second order different from the first order according to a determination that a first timed task of the set of timed tasks is not associated with the identified user.

[0008] An exemplary computer system is described herein. The exemplary computer system is configured to communicate one or more input devices and comprises one or more processors and memory for storing one or more programs configured to run by the one or more processors. The one or more programs include receiving natural language voice input via one or more input devices, which includes a request for the status of a timed task; identifying an electronic device associated with the timed task based on the natural language voice input; receiving the status of the timed task; and providing an output indicating the status of the timed task.

[0009] An exemplary computer system is configured to communicate with one or more input devices and comprises one or more processors and memory for storing one or more programs configured to run by the one or more processors. The one or more programs include receiving a request via one or more input devices for a set of states corresponding to a separate set of timed tasks and identifying a user corresponding to the request; receiving a set of states and providing at least a subset of the set of states in a first order according to a determination that a first timed task of the set of timed tasks is associated with the identified user; and providing at least a subset of the set of states in a second order different from the first order according to a determination that a first timed task of the set of timed tasks is not associated with the identified user.

[0010] An exemplary computer system is configured to communicate with one or more input devices and comprises means for receiving natural language voice input via one or more input devices, including a request for the status of a timed task; means for identifying an electronic device associated with the timed task based on the natural language voice input; means for receiving the status of the timed task; and means for providing an output indicating the status of the timed task.

[0011] An exemplary computer system is configured to communicate with one or more input devices and comprises, via one or more input devices, means for receiving requests for sets of states corresponding to separate sets of timed tasks; means for identifying a user corresponding to a request; means for receiving sets of states; means for providing at least a subset of sets of states in a first order according to a determination that a first timed task among the set of timed tasks is associated with the identified user; and means for providing at least a subset of sets of states in a second order different from the first order according to a determination that a first timed task among the set of timed tasks is not associated with the identified user.

[0012] As described herein, by managing timed tasks using the device's digital assistant, users can manage timed tasks across devices, thereby eliminating the need for users to manage timed tasks by interacting with multiple devices individually. In this way, timed task management becomes faster (for example, less time is required to create, cancel, modify, and / or review timed tasks across different devices and / or locations), which further reduces power consumption and improves the device's battery life by enabling users to use the device more quickly and efficiently. [Brief explanation of the drawing]

[0013] [Figure 1] This block diagram shows systems and environments for running a digital assistant, illustrated by various embodiments.

[0014] [Figure 2A] This block diagram shows a portable multifunction device that performs the client-side portion of a digital assistant, in various embodiments.

[0015] [Figure 2B] This is a block diagram showing exemplary components for event processing in various embodiments.

[0016] [Figure 3] This figure shows a portable multifunction device that performs the client-side portion of a digital assistant, according to various embodiments.

[0017] [Figure 4] This is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface, according to various embodiments.

[0018] [Figure 5A]FIG. showing an exemplary user interface for a menu of an application on a portable multifunctional device according to various embodiments.

[0019] [Figure 5B] FIG. showing an exemplary user interface for a multifunctional device having a touch sensing surface separate from a display according to various embodiments.

[0020] [Figure 6A] FIG. showing a personal electronic device according to various embodiments.

[0021] [Figure 6B] FIG. showing a block diagram of a personal electronic device according to various embodiments.

[0022] <ocke diagram of a digital assistant system or a server portion thereof according to various embodiments.

[0023] [Figure 7B] FIG. showing the functions of the digital assistant shown in FIG. 7A according to various embodiments.

[0024] [Figure 7C] FIG. showing a part of an ontology according to various embodiments.

[0025] [Figure 8] Showing an exemplary environment for managing time-limited tasks according to various embodiments.

[0026] [Figure 9A] Showing an exemplary technique for managing time-limited tasks according to various embodiments. [Figure 9B] Showing an exemplary technique for managing time-limited tasks according to various embodiments. [Figure 9C] ​This document presents exemplary techniques for managing time-sensitive tasks through various implementations. [Figure 9D] This document presents exemplary techniques for managing time-sensitive tasks through various implementations. [Figure 9E] This document presents exemplary techniques for managing time-sensitive tasks through various implementations. [Figure 9F] This document presents exemplary techniques for managing time-sensitive tasks through various implementations.

[0027] [Figure 10] This is a flowchart of the process for managing time-limited tasks, illustrated by various implementations.

[0028] [Figure 11] This is a flowchart of the process for managing user-specific time-limited tasks, illustrated by various implementations. [Modes for carrying out the invention]

[0029] The following description of embodiments refers to the attached drawings, which illustrate specific embodiments that can be implemented. It should be understood that other embodiments can be used and structural modifications can be made without departing from the scope of these various embodiments.

[0030] In the following description, terms such as "first," "second," etc., are used to describe various elements, but these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, without deviating from the scope of the various embodiments described, it is possible to call the first input the second input, and similarly, the second input the first input. Both the first input and the second input are inputs, and in some cases are distinct and different inputs.

[0031] The terminology used in the descriptions of the various embodiments described herein is intended solely to describe specific embodiments and not to limit them. In the descriptions of the various embodiments and in the appended claims, the singular forms “a,” “an,” and “the” are intended to also include the plural forms unless the context explicitly indicates otherwise. Furthermore, as used herein, the term “and / or” should be understood to refer to and include any and all possible combinations of one or more of the enumerated items relating to the description. It will be further understood that, as used herein, the terms “includes,” “comprises,” and / or “comprising” specify the presence of the described features, integers, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof.

[0032] The term "if" can be interpreted, depending on the context, as meaning "when," "upon," "in response to determining," or "in response to detecting." Similarly, the phrases "if it is determined" or "if [a stated condition or event] is detected" can be interpreted, depending on the context, as meaning "upon determining," "in response to determining," "upon detecting [the stated condition or event]," or "in response to detecting [the stated condition or event]." 1. System and Environment

[0033] Figure 1 shows block diagrams of System 100 in various embodiments. In some embodiments, System 100 performs a digital assistant. The terms “digital assistant,” “virtual assistant,” “intelligent automated assistant,” or “automated digital assistant” refer to any information processing system that infers user intent by interpreting natural language input in oral and / or textual form and performs actions based on that inferred user intent. For example, in order to act based on inferred user intent, the system performs one or more of the following: identifying a task flow having steps and parameters designed to fulfill the inferred user intent; inputting specific requests from the inferred user intent into the task flow; executing the task flow by calling a program, method, service, API, or similar; and generating an output response to the user in audible (e.g., utterance) and / or visual form.

[0034] Specifically, a digital assistant can accept user requests, at least partially, in the form of natural language commands, requests, opinions, discourses, and / or inquiries. Typically, a user request asks the digital assistant to either provide information or perform a task. A satisfactory response to a user request includes providing the requested information, performing the requested task, or a combination of the two. For example, a user might ask the digital assistant, "Where am I right now?" Based on the user's current location, the digital assistant might respond, "You are in Central Park near the West Gate." The user might also request the performance of a task, such as, "Please invite my friends to my girlfriend's birthday party next week." In response, the digital assistant could affirm the request by saying, "Yes, right away," and then, on behalf of the user, send appropriate calendar invitations to each of the user's friends listed in the user's electronic address book. While performing a requested task, the digital assistant may interact with the user in a continuous conversation involving multiple information exchanges over an extended period. Numerous other ways exist to interact with a digital assistant to request information or to perform various tasks. In addition to providing verbal responses and taking programmed actions, digital assistants can also provide responses in other visual or auditory formats, such as text, alarms, music, videos, and animations.

[0035] As shown in Figure 1, in some embodiments, the digital assistant operates according to a client-server model. The digital assistant includes a client-side portion 102 (hereinafter, "DA client 102") that runs on a user device 104, and a server-side portion 106 (hereinafter, "DA server 106") that runs on a server system 108. The DA client 102 communicates with the DA server 106 through one or more networks 110. The DA client 102 provides client-side functions such as user-responsive input and output processing, and communication with the DA server 106. The DA server 106 provides server-side functions to any number of DA clients 102, each residing on an individual user device 104.

[0036] In some embodiments, the DA server 106 includes a client-responsive I / O interface 112, one or more processing modules 114, data and models 116, and an I / O interface 118 to external services. The client-responsive I / O interface 112 facilitates client-responsive input and output processing by the DA server 106. One or more processing modules 114 use the data and models 116 to process speech input and determine user intent based on natural language input. Furthermore, one or more processing modules 114 perform task execution based on the inferred user intent. In some embodiments, the DA server 106 communicates with external services 120 via a network(s) 110 to complete tasks or retrieve information. The I / O interface 118 to external services facilitates such communication.

[0037] The user device 104 can be any suitable electronic device. In some embodiments, the user device 104 is a portable multifunction device (e.g., device 200 described below in relation to Figure 2A), a multifunction device (e.g., device 400 described below in relation to Figure 4), or a personal electronic device (e.g., device 600 described below in relation to Figures 6A-6B). A portable multifunction device is, for example, a mobile phone that also includes other functions such as a PDA and / or music player function. Specific examples of portable multifunction devices include Apple Watch®, iPhone®, iPod Touch®, and iPad® devices by Apple Inc. (Cupertino, California). Other examples of portable multifunction devices include, but are not limited to, earphones / headphones, speakers, and laptop computers or tablet computers. Furthermore, in some embodiments, the user device 104 is a non-portable multifunction device. Specifically, the user device 104 is a desktop computer, a game console, a speaker, a television, or a television set-top box. In some embodiments, the user device 104 includes a touch-sensitive surface (e.g., a touchscreen display and / or touchpad). Furthermore, the user device 104 optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick. Various embodiments of electronic devices, such as multifunction devices, are described in further detail below.

[0038] Examples of communication networks (one or more) 110 include local area networks (LANs) and wide area networks (WANs), such as the Internet. Communication networks (one or more) 110 are implemented using any known network protocol, including a variety of wired or wireless protocols such as Ethernet, Universal Serial Bus (USB), FireWire®, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth®, Wi-Fi®, Voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol.

[0039] The server system 108 is implemented on one or more standalone data processing devices or on a distributed computer network. In some embodiments, the server system 108 also employs various virtual devices and / or services from third-party service providers (e.g., third-party cloud service providers) to provide the basic computing and / or infrastructure resources of the server system 108.

[0040] In some embodiments, user device 104 communicates with DA server 106 via a second user device 122. The second user device 122 is similar to or identical to user device 104. For example, the second user device 122 is similar to devices 200, 400, or 600 described below in relation to Figures 2A, 4, and 6A-6B. User device 104 is configured to be communicatively coupled to the second user device 122 via a direct communication connection such as Bluetooth, NFC, or BTLE, or via a wired or wireless network such as a local Wi-Fi network. In some embodiments, the second user device 122 is configured to act as a proxy between user device 104 and DA server 106. For example, the DA client 102 of user device 104 is configured to transmit information (e.g., user requests received by user device 104) to the DA server 106 via the second user device 122. The DA server 106 processes the information and returns the relevant data (e.g., data content responding to a user request) to the user device 104 via the second user device 122.

[0041] In some examples, user device 104 is configured to reduce the amount of information transmitted from user device 104 by communicating a shortened request for data to a second user device 122. The second user device 122 is configured to determine any supplementary information to be added to the shortened request and generate a complete request for transmission to the DA server 106. This system architecture can favorably allow a user device 104 with limited communication capabilities and / or limited battery power (e.g., a wristwatch or similar small electronic device) to access services provided by the DA server 106 by using a second user device 122 with higher communication capabilities and / or battery power (e.g., a mobile phone, laptop computer, tablet computer, etc.) as a proxy to the DA server 106. Although only two user devices 104 and 122 are shown in Figure 1, it should be understood that system 100 includes any number and types of user devices configured to communicate with the DA server system 106 in this proxy configuration in some embodiments.

[0042] The digital assistant shown in Figure 1 includes both a client-side portion (e.g., DA client 102) and a server-side portion (e.g., DA server 106), however, in some embodiments, the functions of the digital assistant are executed as a separate application installed on the user device. Furthermore, the allocation of functions between the client and server portions of the digital assistant may vary depending on the implementation. For example, in some embodiments, the DA client is a thin client that provides only user-responsive input and output processing functions, delegating all other functions of the digital assistant to the backend server. 2. Electronic devices

[0043] Here, we turn our attention to embodiments of electronic devices for performing the client-side portion of a digital assistant. Figure 2A is a block diagram of a portable multifunction device 200, which, according to some embodiments, includes a touch-sensitive display system 212. The touch-sensitive display 212 may be referred to for convenience as a “touchscreen” and may be known or referred to as a “touch-sensitive display system.” Device 200 includes memory 202 (optionally including one or more computer-readable storage media), a memory controller 222, one or more processing units (CPUs) 220, a peripheral interface 218, an RF circuit 208, an audio circuit 210, a speaker 211, a microphone 213, an input / output (I / O) subsystem 206, other input control devices 216, and an external port 224. Device 200 optionally includes one or more optical sensors 264. Device 200 optionally includes one or more contact intensity sensors 265 (e.g., touch-sensitive surfaces such as the touch-sensitive display system 212 of device 200) for detecting the intensity of contact on device 200. Device 200 optionally includes one or more tactile output generators 267 for generating tactile output on device 200 (e.g., on touch-sensitive surfaces such as the touch-sensitive display system 212 of device 200 or the touchpad 455 of device 400). These components optionally communicate via one or more communication buses or signal lines 203.

[0044] As used herein and in the claims, the term “strength” of contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of contact on the touch-sensitive surface (e.g., finger contact), or a proxy for the force or pressure of contact on the touch-sensitive surface. The strength of contact has a range of values, including at least four distinct values, and more typically, including several hundred (e.g., at least 256) distinct values. The strength of contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted averaged) to determine the estimated force of contact. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size and / or modification of the contact area detected on the touch-sensing surface, the capacitance and / or modification of the touch-sensing surface adjacent to the contact, and / or the resistance and / or modification of the touch-sensing surface adjacent to the contact may optionally be used as a substitute for the force or pressure of the contact on the touch-sensing surface. In some implementations, the substitute measurement of the contact force or pressure is used directly to determine whether or not an intensity threshold is exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some implementations, the substitute measurement of the contact force or pressure is converted into an estimate of the force or pressure, which is then used to determine whether or not an estimate of the force or pressure exceeds an intensity threshold (e.g., the intensity threshold is a pressure threshold measured in units of pressure). By using the intensity of contact as an attribute of user input, it becomes possible for users to access additional device functions that may otherwise be inaccessible (e.g., on a touch-sensitive display) and / or receive user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical control such as a knob or button) on reduced-size devices where the implementation area for displaying affordances is limited.

[0045] As used herein and in the claims, the term “tactile output” means a physical displacement of the device relative to its previous position, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by the user through the user’s sense of touch. For example, in a situation where the device or a component of the device is in contact with the touch-sensitive surface of the user (e.g., the user’s fingers, palm, or other part of their hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical properties of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) may be optionally interpreted by the user as a “down-click” or “up-click” of a physical actuator button. In some cases, the user may feel a tactile sensation such as a “down-click” or “up-click” even when there is no movement of a physical actuator button associated with a touch-sensitive surface that has been physically pressed (e.g., displaced) by the user’s action. In another embodiment, movement of a touch-sensitive surface may be optionally interpreted or perceived by the user as "roughness" of the touch-sensitive surface, even if there is no change in the smoothness of the touch-sensitive surface. Such user interpretations of touch depend on the user's personal sensory perception, but there are many touch sensory perceptions common to the majority of users. Therefore, when a tactile output is described as corresponding to a user's specific sensory perception (e.g., "up-click," "down-click," "roughness"), unless otherwise stated, the generated tactile output corresponds to the physical displacement of the device or its components that produce the described sensory perception of a typical (or average) user.

[0046] It should be understood that device 200 is merely an example of a portable multifunction device, and that device 200 may optionally have more or fewer components than those shown, may optionally be a combination of two or more components, or may optionally have different configurations or arrangements of those components. The various components shown in Figure 2A may be implemented as hardware, software, or a combination of both hardware and software, including one or more signal processing circuits and / or application-specific integrated circuits.

[0047] Memory 202 includes one or more computer-readable storage media. These computer-readable storage media are, for example, tangible and non-temporary. Memory 202 includes high-speed random-access memory and also includes non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. The memory controller 222 controls access to memory 202 by other components of device 200.

[0048] In some embodiments, the non-temporary computer-readable storage medium of memory 202 is used to store instructions (for example, for performing the processes described below) for use by or in connection with instruction execution systems, devices, or other systems capable of fetching and executing instructions from such instruction execution systems, devices, or devices, such as computer-based systems or systems including processors. In other embodiments, the instructions (for example, for performing the processes described below) are stored in the non-temporary computer-readable storage medium (not shown) of the server system 108, or are shared between the non-temporary computer-readable storage medium of memory 202 and the non-temporary computer-readable storage medium of the server system 108.

[0049] The peripheral interface 218 is used to connect the input and output peripherals of this device to the CPU 220 and memory 202. One or more processors 220 operate or execute various software programs and / or instruction sets stored in memory 202 to perform various functions for the device 200 and process data. In some embodiments, the peripheral interface 218, CPU 220, and memory controller 222 are implemented on a single chip, such as chip 204. In some other embodiments, they are implemented on separate chips.

[0050] The RF (radio frequency) circuit 208 transmits and receives RF signals, also known as electromagnetic signals. The RF circuit 208 converts electrical signals to electromagnetic signals or electromagnetic signals to electrical signals and communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 208 optionally includes well-known circuits for performing these functions, which include, but are not limited to, antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, CODEC chipsets, subscriber identity module (SIM) cards, and memory. The RF circuit 208 optionally communicates wirelessly with networks such as the Internet, also known as the World Wide Web (WWW), intranets, and / or wireless networks such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs), as well as with other devices. The RF circuit 208 optionally includes well-known circuits for detecting near-field communication (NFC) fields, such as short-range radios. Wireless communication is not limited to, optionally, the Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPADA), and long-term evolution.Bluetooth evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE®), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX®, Email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), Instant messaging (e.g., Extensible Messaging and Presence Protocol) Using any of several communication standards, protocols, and technologies, including the XMPP protocol, the Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), the Instant Messaging and Presence Service (IMPS), and / or the Short Message Service (SMS), or any other suitable communication protocol, including a communication protocol not yet developed as of the filing date of this specification.

[0051] The audio circuit 210, speaker 211, and microphone 213 provide an audio interface between the user and the device 200. The audio circuit 210 receives audio data from the peripheral interface 218, converts this audio data into an electrical signal, and transmits this electrical signal to the speaker 211. The speaker 211 converts the electrical signal into human audible sound waves. The audio circuit 210 also receives the electrical signal converted from the sound waves by the microphone 213. The audio circuit 210 converts the electrical signal into audio data and transmits this audio data to the peripheral interface 218 for processing. The audio data is retrieved by the peripheral interface 218 from memory 202 and / or RF circuit 208, and / or transmitted to memory 202 and / or RF circuit 208. In some embodiments, the audio circuit 210 also includes a headset jack (e.g., 312 in Figure 3). The headset jack provides an interface between the audio circuit 210 and detachable audio input / output peripherals such as output-only headphones or headsets that have both output (e.g., headphones for one or both ears) and input (e.g., a microphone).

[0052] The I / O subsystem 206 connects input / output peripherals on device 200, such as the touchscreen 212 and other input control devices 216, to the peripheral interface 218. The I / O subsystem 206 optionally includes a display controller 256, an optical sensor controller 258, an intensity sensor controller 259, a haptic feedback controller 261, and one or more input controllers 260 for other input or control devices. One or more input controllers 260 receive electrical signals from / transmit electrical signals to other input control devices 216. The other input control devices 216 optionally include physical buttons (e.g., push buttons, rocker buttons), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, one or more input controllers 260 are optionally connected to (or not connected to) one of the following: a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. One or more buttons (e.g., 308 in Figure 3) optionally include up / down buttons for volume control of speaker 211 and / or microphone 213. One or more buttons optionally include push buttons (e.g., 306 in Figure 3).

[0053] A quick press of a push button unlocks the touchscreen 212 or initiates a process using gestures on the touchscreen to unlock the device, as described in U.S. Patent Application No. 11 / 322,549, filed December 23, 2005, U.S. Patent No. 7,657,849, “Unlocking a Device by Performing Gestures on an Unlock Image,” the entire process of which is incorporated herein by reference. A long press of a push button (e.g., 306) turns the device 200 on or off. The user can customize the function of one or more buttons. The touchscreen 212 is used to implement virtual or soft buttons and one or more soft keyboards.

[0054] The touch-sensitive display 212 provides input and output interfaces between the device and the user. The display controller 256 receives electrical signals from and / or transmits electrical signals to the touchscreen 212. The touchscreen 212 displays visual output to the user. Visual output includes graphics, text, icons, videos, and any combination thereof (collectively referred to as “graphics”). In some embodiments, some or all of the visual output corresponds to user interface objects.

[0055] The touchscreen 212 has a touch-sensing surface, sensor, or set of sensors that accept user input based on touch and / or tactile contact. The touchscreen 212 and the display controller 256 (together with any associated modules and / or instruction sets in memory 202) detect contact (and any movement or interruption of contact) on the touchscreen 212 and translate the detected contact into interaction with user interface objects displayed on the touchscreen 212 (e.g., one or more soft keys, icons, web pages, or images). In an exemplary embodiment, the point of contact between the touchscreen 212 and the user corresponds to the user's finger.

[0056] The touchscreen 212 uses LCD (liquid crystal display) technology, LPD (polymer light-emitting display) technology, or LED (light-emitting diode) technology, but other display technologies may be used in other embodiments. The touchscreen 212 and the display controller 256 detect contact and its movement or disconnection using any of several currently known or future-developed touch sensing technologies, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, and other proximity sensor arrays or other elements that determine one or more contact points using the touchscreen 212. In exemplary embodiments, projected mutual capacitive sensing technology is used, such as that found in the iPhone® and iPod Touch® from Apple Inc. of Cupertino, California.

[0057] Some embodiments of the touch-sensitive display of the touchscreen 212 are similar to the multi-touch-sensitive touchpads described in U.S. Patents 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.) and / or 6,677,932 (Westerman) and / or U.S. Patent Publication 2002 / 0015024 (A1), each of which is incorporated herein by reference in whole. However, the touchscreen 212 displays visual output from device 200, whereas the touch-sensitive touchpad does not provide visual output.

[0058] The touch-sensitive display in some embodiments of the touchscreen 212 is based on the following applications: (1) U.S. Patent Application No. 11 / 381,313, filed May 2, 2006, "Multipoint Touch Surface Controller"; (2) U.S. Patent Application No. 10 / 840,862, filed May 6, 2004, "Multipoint Touchscreen"; (3) U.S. Patent Application No. 10 / 903,964, filed July 30, 2004, "Gestures For Touch Sensitive Input Devices"; (4) U.S. Patent Application No. 11 / 048,264, filed January 31, 2005, "Gestures For Touch Sensitive Input Devices"; (5) U.S. Patent Application No. 11 / 038,590, filed January 18, 2005, "Mode-Based Graphical User Interfaces For Touch Sensitive Input Such devices are described in (6) U.S. Patent Application No. 11 / 228,758, filed September 16, 2005, "Virtual Input Device Placement On A Touch Screen User Interface", (7) U.S. Patent Application No. 11 / 228,700, filed September 16, 2005, "Operation Of A Computer With A Touch Screen Interface", (8) U.S. Patent Application No. 11 / 228,737, filed September 16, 2005, "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard", and (9) U.S. Patent Application No. 11 / 367,749, filed March 3, 2006, "Multi-Functional Hand-Held Device". All of these applications are incorporated herein by reference in their entirety.

[0059] The touchscreen 212 has a video resolution of, for example, more than 100 dpi. In some embodiments, the touchscreen has a video resolution of about 160 dpi. The user touches the touchscreen 212 using a suitable object or attachment such as a stylus or finger. In some embodiments, the user interface is designed to operate primarily using finger-based touch and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​the finger on the touchscreen. In some embodiments, the device translates coarse finger input into a precise pointer / cursor position or command to perform an action desired by the user.

[0060] In some embodiments, in addition to the touchscreen, the device 200 includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touchscreen, does not display a visual output. The touchpad is either a separate touch-sensitive surface from the touchscreen 212 or an extension of the touch-sensitive surface formed by the touchscreen.

[0061] Device 200 also includes a power system 262 that supplies power to various components. The power system 262 includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in the portable device.

[0062] Device 200 also includes one or more optical sensors 264. Figure 2A shows the optical sensors coupled to an optical sensor controller 258 in the I / O subsystem 206. The optical sensors 264 include charge-coupled devices (CCDs) or complementary metal-oxide semiconductor (CMOS) phototransistors. The optical sensors 264 receive light from the environment projected through one or more lenses and convert that light into data representing an image. In conjunction with the imaging module 243 (also called the camera module), the optical sensors 264 capture still images or video. In some embodiments, the optical sensors are located on the back of Device 200, opposite the touchscreen display 212 on the front of the device, so that the touchscreen display is used as a viewfinder for acquiring still images and / or video. In some embodiments, the optical sensors are located on the front of the device so that an image of the user is acquired for video conferencing while the user views other video conference participants on the touchscreen display. In some embodiments, the position of the optical sensor 264 can be changed by the user (for example, by rotating the lens and sensor within the device housing), so that a single optical sensor 264 can be used for both video conferencing and still image and / or video acquisition, together with the touchscreen display.

[0063] Device 200 also optionally includes one or more contact strength sensors 265. Figure 2A shows a contact strength sensor coupled to a strength sensor controller 259 in the I / O subsystem 206. The contact strength sensor 265 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electric force sensors, pressure-power sensors, optical force sensors, capacitive touch-sensing surfaces, or other strength sensors (e.g., sensors used to measure the force (or pressure) of contact on a touch-sensing surface). The contact strength sensor 265 receives contact strength information (e.g., pressure information, or a proxy for pressure information) from the environment. In some embodiments, at least one contact strength sensor is positioned juxtaposed with or adjacent to a touch-sensing surface (e.g., a touch-sensing display system 212). In some embodiments, at least one contact strength sensor is located on the back of Device 200, opposite the touchscreen display 212 located on the front of Device 200.

[0064] Device 200 also includes one or more proximity sensors 266. Figure 2A shows a proximity sensor 266 coupled to a peripheral interface 218. Alternatively, the proximity sensor 266 is coupled to an input controller 260 in an I / O subsystem 206. The proximity sensor 266 functions as described in U.S. Patent Application No. 11 / 241,839, “Proximity Detector In Handheld Device”, No. 11 / 240,788, “Proximity Detector In Handheld Device”, No. 11 / 620,702, “Using Ambient Light Sensor To Augment Proximity Sensor Output”, No. 11 / 586,862, “Automated Response To And Sensing Of User Activity In Portable Devices”, and No. 11 / 638,251, “Methods And Systems For Automatic Configuration Of Peripherals”. In some embodiments, if the multifunction device is placed near the user's ear (for example, when the user is making a phone call), the proximity sensor turns off and disables the touchscreen 212.

[0065] Device 200 also optionally includes one or more tactile output generators 267. Figure 2A shows a tactile output generator coupled to a tactile feedback controller 261 in the I / O subsystem 206. The tactile output generator 267 optionally includes one or more electroacoustic devices such as a speaker or other audio component, and / or electromechanical devices that convert energy into linear motion, such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component that converts an electrical signal into a tactile output on the device). The contact intensity sensor 265 receives a tactile feedback generation command from the tactile feedback module 233 and generates a tactile output on device 200 that can be sensed by the user of device 200. In some embodiments, at least one tactile output generator is positioned alongside or adjacent to a touch-sensing surface (e.g., a touch-sensing display system 212) and optionally generates a tactile output by moving the touch-sensing surface vertically (e.g., inward / outward from the surface of the device 200) or horizontally (e.g., forward / backward in the same plane as the surface of the device 200). In some embodiments, at least one tactile output generator sensor is located on the back of the device 200, opposite the touchscreen display 212 which is located on the front of the device 200.

[0066] Device 200 also includes one or more accelerometers 268. Figure 2A shows an accelerometer 268 coupled to a peripheral interface 218. Alternatively, the accelerometer 268 is coupled to an input controller 260 in an I / O subsystem 206. The accelerometer 268 operates as described, for example, in U.S. Patent Publication No. 20050190059, “Acceleration-based Theft Detection System for Portable Electronic Devices,” and U.S. Patent Publication No. 20060017692, “Acceleration-based Theft Detection System for Portable Electronic Devices,” both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed on a touchscreen display in portrait or landscape orientation based on an analysis of data received from one or more accelerometers. Device 200 optionally includes, in addition to one or more accelerometers 268, a magnetometer (not shown), and a GPS (or GLONASS or other global navigation system) receiver (not shown) for acquiring information about the location and orientation of Device 200 (e.g., portrait or landscape).

[0067] In some embodiments, the software components stored in memory 202 include an operating system 226, a communications module (or instruction set) 228, a contact / motion module (or instruction set) 230, a graphics module (or instruction set) 232, a text input module (or instruction set) 234, a Global Positioning System (GPS) module (or instruction set) 235, a digital assistant client module 229, and an application (or instruction set) 236. Furthermore, memory 202 stores data and models, such as user data and models 231. In addition, in some embodiments, as shown in Figures 2A and 4, memory 202 (Figure 2A) or memory 470 (Figure 4) stores device / global internal state 257. The device / global internal state 257 includes one or more of the following: an active application state indicating which application is currently active, if there is an active application; a display state indicating which applications, views, or other information occupy different areas of the touchscreen display 212; a sensor state including information obtained from various sensors and input control devices 216 of the device; and location information relating to the device's location and / or orientation.

[0068] An operating system 226 (for example, embedded operating systems such as Darwin, RTXC, LINUX, UNIX®, OS X, iOS, WINDOWS®, or VxWorks) includes various software components and / or drivers that control and manage general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitate communication between various hardware components and software components.

[0069] The communication module 228 facilitates communication with other devices via one or more external ports 224 and also includes various software components for processing data received by the RF circuit 208 and / or external ports 224. The external ports 224 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted to connect to other devices directly or indirectly via a network (e.g., the Internet, Wi-Fi, etc.). In some embodiments, the external ports are multi-pin (e.g., 30-pin) connectors that are the same as and / or compatible with the 30-pin connector used on iPod® (a trademark of Apple Inc.) devices.

[0070] The contact / motion module 230 optionally detects contact with the touchscreen 212 and other touch-sensitive devices (e.g., touchpad or physical click wheel) (in cooperation with the display controller 256). The contact / motion module 230 includes various software components for performing various contact detection operations, such as determining whether contact has occurred (e.g., detecting a finger down event), determining the intensity of the contact (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of contact and tracking movement across the touch-sensitive surface (e.g., detecting one or more events of a finger dragging), and determining whether contact has been terminated (e.g., detecting a finger up event or interruption of contact). The contact / motion module 230 receives contact data from the touch-sensitive surface. Determining the movement of the contact point, represented by a series of contact data, optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These actions can be optionally applied to a single contact (e.g., a single finger contact) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contacts). In some embodiments, the contact / motion module 230 and the display controller 256 detect contact on the touchpad.

[0071] In some embodiments, the contact / motion module 230 uses a set of one or more intensity thresholds to determine whether an action has been performed by a user (for example, to determine whether a user has "clicked" on an icon). In some embodiments, at least a subset of the intensity thresholds is determined according to software parameters (for example, the intensity thresholds can be adjusted without modifying the physical hardware of the device 200, rather than being determined by the activation threshold of a particular physical actuator). For example, the mouse "click" threshold for a trackpad or touchscreen display can be set to one of a wide range of default thresholds without modifying the trackpad or touchscreen display hardware. In addition, some implementations provide the user of the device with software settings to adjust one or more of the set of intensity thresholds (for example, by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once using a system-level click "intensity" parameter).

[0072] The contact / motion module 230 optionally detects gesture input from the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motion, timing, and / or intensity of the detected contact). Therefore, gestures are optionally detected by detecting a specific contact pattern. For example, detecting a finger tap gesture involves detecting a finger down event, followed by a finger up (lift-off) event at the same position (or substantially the same position) as the finger down event (e.g., the position of the icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface involves detecting a finger down event, followed by one or more finger drag events, and then a finger up (lift-off) event.

[0073] The graphics module 232 includes various known software components for rendering and displaying graphics on the touchscreen 212 or other display, including components that modify the visual effects of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual properties). As used herein, the term “graphics” includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (including user interface objects such as soft keys), digital images, videos, and animations.

[0074] In some embodiments, the graphics module 232 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 232 receives one or more codes from an application or the like, as needed, specifying the graphics to be displayed, along with coordinate data and other graphic property data, and then generates screen image data to output to the display controller 256.

[0075] The haptic feedback module 233 includes various software components for generating commands used by a tactile output generator(s) 267, and generates tactile outputs at one or more locations on the device 200 in response to the user's interaction with the device 200.

[0076] The text input module 234, a component of the graphics module 232, provides, in some embodiments, a soft keyboard for entering text in various applications (e.g., contacts 237, email 240, IM 241, browser 247, and any other application requiring text input).

[0077] The GPS module 235 determines the device's location and provides this information for use in various applications (for example, to the phone 238 for use in location-based dialing, to the camera 243 as picture / video metadata, and to applications that provide location-based services such as weather widgets, local yellow pages widgets, and map / navigation widgets).

[0078] The digital assistant client module 229 includes various client-side digital assistant commands to provide client-side functionality for the digital assistant. For example, the digital assistant client module 229 can accept voice input (e.g., speech input), text input, touch input, and / or gesture input through various user interfaces of the portable multifunction device 200 (e.g., microphone 213, accelerometer(s) 268, touch-sensitive display system 212, optical sensor(s) 264, other input control devices 216, etc.). The digital assistant client module 229 can also provide audio (e.g., speech output), visual, and / or tactile shape output, etc., through various output interfaces of the portable multifunction device 200 (e.g., speaker 211, touch-sensitive display system 212, tactile output generator(s) 267, etc.). For example, the output may be provided as voice, sound, alarm, text message, menu, graphic, video, animation, vibration, and / or a combination of two or more of the above. During operation, the digital assistant client module 229 communicates with the DA server 106 using the RF circuit 208.

[0079] User data and model 231 include various user-related data (e.g., user-specific vocabulary data, user preference data, pronunciation of user-specified names, data from the user's electronic address book, to-do lists, shopping lists, etc.) to provide client-side functionality for the digital assistant. Furthermore, user data and model 231 include various models (e.g., speech recognition models, statistical language models, natural language processing models, ontologs, task flow models, service models, etc.) to process user input and determine user intent.

[0080] In some embodiments, the digital assistant client module 229 utilizes various sensors, subsystems, and peripherals of the portable multifunction device 200 to collect additional information from the surrounding environment of the portable multifunction device 200, thereby establishing a context associated with the user, the current user interaction, and / or the current user input. In some embodiments, the digital assistant client module 229 provides the DA server 106 with the context information, or a subset thereof, along with the user input, to assist in inferring the user's intent. In some embodiments, the digital assistant also uses the context information to determine how to prepare and deliver output to the user. The context information is referred to as context data.

[0081] In some embodiments, contextual information associated with user input includes sensor information, such as lighting, ambient noise, ambient temperature, and images or videos of the surrounding environment. In some embodiments, contextual information may also include the physical state of the device, such as the orientation of the device, the location of the device, the temperature of the device, power level, speed, acceleration, motion pattern, and cellular signal strength. In some embodiments, the software state of the DA server 106, such as information on running processes, installed programs, past and present network activity, background services, error logs, and resource usage, as well as information on the software state of the portable multifunction device 200, are provided to the DA server 106 as contextual information associated with user input.

[0082] In some embodiments, the digital assistant client module 229 selectively provides information (e.g., user data 231) stored on the portable multifunction device 200 in response to a request from the DA server 106. In some embodiments, the digital assistant client module 229 also elicits additional input from the user via a natural language dialog or other user interface in response to a request from the DA server 106. The digital assistant client module 229 passes its additional input to the DA server 106 to assist the DA server 106 in intent inference and / or execution of the user's intent expressed in the user request.

[0083] A more detailed description of the digital assistant is provided below with reference to Figures 7A to 7C. It should be noted that the digital assistant client module 229 may contain any number of submodules of the digital assistant module 726, which is described below.

[0084] Application 236 includes the following modules (or sets of instructions), or subsets or supersets thereof: ● Contact module 237 (sometimes called the address book or contact list), ●Telephone module 238, ●Video conferencing module 239, ● Email client module 240, ● Instant messaging (IM) module 241, ●Training support module 242, ● Camera module 243 for still images and / or video, ●Image management module 244, ●Video player module, ● Music player module, ● Browser module 247, ● Calendar module 248, ● A widget module 249, which in some embodiments includes one or more of the following: weather widget 249-1, stock price widget 249-2, calculator widget 249-3, alarm clock widget 249-4, dictionary widget 249-5, and user-created widget 249-6. ●Widget creator module 250 for creating user-created widget 249-6, ● Search module 251, ●Video and music player module 252, which integrates a video player module and a music player module. ●Memo Module 253, ● Map module 254 and / or, ● Online video module 255.

[0085] Other applications 236 stored in memory 202 include other word processing applications, other image editing applications, drawing applications, presentation applications, Java-enabled applications, encryption, digital rights management, speech recognition, and speech duplication.

[0086] In conjunction with the touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, the contact module 237 is used to manage an address book or contact list (for example, stored in the application internal state 292 of the contact module 237 in memory 202 or memory 470), including adding names(s) to and removing names(s) from the address book, and associating names with telephone numbers(s) or email addresses(s) or other information. This includes associating images with names, categorizing and sorting names, and providing telephone numbers or email addresses to initiate and / or facilitate communication via telephone 238, video conferencing module 239, email 240, or IM 241.

[0087] In conjunction with the RF circuit 208, audio circuit 210, speaker 211, microphone 213, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, the telephone module 238 is used to input strings corresponding to telephone numbers, access one or more telephone numbers in the contact module 237, modify the entered telephone number, dial individual telephone numbers, conduct conversations, and disconnect or hang up when the conversation is complete. Thus, wireless communication employs one of several communication standards, protocols, and technologies.

[0088] The video conferencing module 239 works in conjunction with an RF circuit 208, an audio circuit 210, a speaker 211, a microphone 213, a touchscreen 212, a display controller 256, an optical sensor 264, an optical sensor controller 258, a contact / motion module 230, a graphics module 232, a text input module 234, a contact module 237, and a telephone module 238 to include executable commands for starting, running, and ending video conferences between the user and one or more other participants in accordance with the user's commands.

[0089] The email client module 240, in conjunction with the RF circuit 208, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, includes executable commands for creating, sending, receiving, and managing emails in response to user commands. In conjunction with the image management module 244, the email client module 240 makes it extremely easy to create and send emails containing still or video images captured by the camera module 243.

[0090] The instant messaging module 241, in conjunction with the RF circuit 208, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, includes executable commands for inputting character sequences corresponding to instant messages, modifying previously entered characters, sending individual instant messages (for example, using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for telephone-based instant messaging, or XMPP, SIMPLE, or IMPS for internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, sent and / or received instant messages include graphics, photographs, audio files, video files, and / or other attachments supported by MMS and / or Enhanced Messaging Services (EMS). In this specification, “instant messaging” refers to both telephone-based messaging (e.g., messages sent using SMS or MMS) and internet-based messaging (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0091] In conjunction with the RF circuit 208, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, GPS module 235, map module 254, and music player module, the training support module 242 includes executable commands, which create training (e.g., having time, distance, and / or calorie burn goals), communicate with training sensors (sports devices), receive training sensor data, calibrate sensors used to monitor training, select and play music for training, and display, store, and transmit training data.

[0092] The camera module 243, in cooperation with the touchscreen 212, display controller 256, optical sensor(s) 264, optical sensor controller 258, contact / motion module 230, graphics module 232, and image management module 244, includes executable commands for capturing still images or videos (including video streams) and storing them in memory 202, modifying the characteristics of still images or videos, or deleting still images or videos from memory 202.

[0093] The image management module 244 works in conjunction with the touchscreen 212, display controller 256, touch / motion module 230, graphics module 232, text input module 234, and camera module 243 to include executable commands for arranging, modifying (e.g., editing), or otherwise manipulating still images and / or videos, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing them.

[0094] The browser module 247, in conjunction with the RF circuit 208, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, includes executable commands for browsing the internet according to user commands, including searching, linking, receiving, and displaying web pages or parts thereof, as well as attachments and other files linked to web pages.

[0095] The calendar module 248 works in conjunction with the RF circuit 208, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, email client module 240, and browser module 247 to include executable commands for creating, displaying, modifying, and storing a calendar and data associated with the calendar (e.g., calendar entries, to-do lists, etc.) in accordance with user commands.

[0096] In conjunction with the RF circuit 208, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, and browser module 247, the widget module 249 is a mini-application that can be downloaded and used by the user (e.g., weather widget 249-1, stock price widget 249-2, calculator widget 249-3, alarm clock widget 249-4, and dictionary widget 249-5) or created by the user (e.g., user-created widget 249-6). In some embodiments, the widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! Widgets®).

[0097] In conjunction with the RF circuit 208, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, and browser module 247, the widget creation module 250 is used by the user to create widgets (for example, to convert a user-specified portion of a web page into a widget).

[0098] The search module 251 works in conjunction with the touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234 to include executable commands for searching for characters, music, sounds, images, videos, and / or other files in memory 202 that match one or more search criteria (e.g., one or more user-specified search terms) according to user commands.

[0099] The video and music player module 252 works in conjunction with the touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, audio circuit 210, speaker 211, RF circuit 208, and browser module 247 to include executable commands that allow the user to download and play recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, as well as executable commands for displaying, presenting, or otherwise playing videos (for example, on the touchscreen 212 or on an external display connected via the external port 224). In some embodiments, the device 200 optionally includes the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0100] The memo module 253 works in conjunction with the touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234 to include executable commands for creating and managing memos, to-do lists, etc., according to user commands.

[0101] In conjunction with the RF circuit 208, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, GPS module 235, and browser module 247, the map module 254 is used to receive, display, modify, and store maps and map-related data (e.g., driving instructions, store information, other locations in or near a specific place, and other location-based data) according to user commands.

[0102] The online video module 255, in conjunction with the touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, audio circuit 210, speaker 211, RF circuit 208, text input module 234, email client module 240, and browser module 247, includes instructions that enable the user to access, browse, receive (e.g., by streaming and / or downloading), play (e.g., on the touchscreen or on an external display connected via external port 224), send emails with links to specific online videos, and perform other management of online videos in one or more file formats such as H.264. In some embodiments, the instant messaging module 241 is used instead of the email client module 240 to send links to specific online videos. Further descriptions of online video applications can be found in U.S. Provisional Patent Application No. 60 / 936,562, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed June 20, 2007, and U.S. Patent Application No. 11 / 968,067, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed December 31, 2007, the entirety of which is incorporated herein by reference.

[0103] Each of the modules and applications identified above corresponds to a set of executable instructions that perform one or more of the functions described above and the methods described in this application (e.g., the computer-based methods and other information processing methods described herein). Since these modules (e.g., sets of instructions) do not need to be implemented as separate software programs, procedures, or modules, various subsets of these modules can be combined or reconfigured in various embodiments. For example, a video player module can be combined with a music player module in a single module (e.g., a video and music player module 252, Figure 2A). In some embodiments, memory 202 stores a subset of the modules and data structures identified above. Furthermore, memory 202 stores additional modules and data structures not described above.

[0104] In some embodiments, device 200 is a device in which the operation of a default set of functions in the device is performed solely via a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for the operation of device 200, the number of physical input control devices (push buttons, dials, etc.) on device 200 is reduced.

[0105] A default set of functions, performed only through the touchscreen and / or touchpad, optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 200 from any user interface displayed on the device 200 to the main menu, home menu, or root menu. In such embodiments, a “menu button” is implemented using the touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0106] Figure 2B is a block diagram showing exemplary components for event processing according to some embodiments. In some embodiments, memory 202 (Figure 2A) or memory 470 (Figure 4) includes an event sorting unit 270 (e.g., within the operating system 226) and individual applications 236-1 (e.g., any of the aforementioned applications 237-251, 255, 480-490).

[0107] The event sorting unit 270 receives event information and determines the application 236-1 that distributes the event information, and the application view 291 of application 236-1. The event sorting unit 270 includes an event monitor 271 and an event dispatcher module 274. In some embodiments, application 236-1 includes an application internal state 292 that indicates the current application view(s) displayed on the touch-sensitive display 212 when the application is active or running. In some embodiments, a device / global internal state 257 is used by the event sorting unit 270 to determine which application(s) are currently active, and the application internal state 292 is used by the event sorting unit 270 to determine the application view(s) to which the event information is distributed.

[0108] In some embodiments, the application internal state 292 includes additional information such as resume information to be used when the application 236-1 resumes execution, user interface state information that indicates or is ready to display information displayed by the application 236-1, a state queue that allows the user to return to a previous state or view of the application 236-1, and one or more redo / undo queues of previous actions performed by the user.

[0109] The event monitor 271 receives event information from the peripheral interface 218. The event information includes information about sub-events (for example, user touch as part of a multi-touch gesture on the touch-sensitive display 212). The peripheral interface 218 transmits information received from the I / O subsystem 206, or from sensors such as the proximity sensor 266, one or more accelerometers 268, and / or the microphone 213 (via the audio circuit 210). The information received by the peripheral interface 218 from the I / O subsystem 206 includes information from the touch-sensitive display 212 or the touch-sensitive surface.

[0110] In some embodiments, the event monitor 271 sends requests to the peripheral interface 218 at predetermined intervals. In response, the peripheral interface 218 transmits event information. In other embodiments, the peripheral interface 218 transmits event information only when there is a significant event (e.g., reception of input exceeding a predetermined noise threshold and / or exceeding a predetermined duration).

[0111] In some embodiments, the event sorting unit 270 also includes a hit view determination module 272 and / or an active event recognition unit determination module 273.

[0112] The hit view determination module 272 provides a software procedure for determining where in one or more views a sub-event occurred when the touch-sensitive display 212 is displaying two or more views. A view consists of control devices and other elements that the user can see on the display.

[0113] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of an individual application) in which a touch is detected corresponds to the program level of the application or the program level within the view hierarchy. For example, the lowest level view in which a touch is detected is called the hit view, and the set of events recognized as a valid input is determined at least in part on the hit view of the initial touch that initiates a touch-based gesture.

[0114] The hit view determination module 272 receives information related to sub-events of touch-based gestures. When an application has multiple views arranged in a hierarchy, the hit view determination module 272 identifies the hit view as the lowest-level view in the hierarchy from which sub-events should be processed. In most situations, the hit view is the lowest-level view from which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module 272, the hit view typically receives all sub-events related to the same touch or input source identified as the hit view.

[0115] The active event recognition determination module 273 determines which view(s) in the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognition determination module 273 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognition determination module 273 determines that all views, including the physical location of the sub-event, are actively involved views, and therefore all actively involved views should receive a particular sequence of sub-events. In other embodiments, even if the touch sub-event is entirely confined to an area associated with one particular view, higher-level views in the hierarchy still remain actively involved views.

[0116] The event dispatcher module 274 dispatches event information to an event recognition unit (e.g., an event recognition unit 280). In embodiments including an active event recognition unit determination module 273, the event dispatcher module 274 distributes the event information to the event recognition unit determined by the active event recognition unit determination module 273. In some embodiments, the event dispatcher module 274 stores event information retrieved by individual event receiving units 282 in an event queue.

[0117] In some embodiments, the operating system 226 includes an event sorting unit 270. Alternatively, application 236-1 includes an event sorting unit 270. In yet another embodiment, the event sorting unit 270 is a standalone module or part of another module stored in memory 202, such as a contact / motion module 230.

[0118] In some embodiments, application 236-1 includes a plurality of event processing units 290 and one or more application views 291, each containing instructions for handling touch events occurring within a separate view of the application's user interface. Each application view 291 of application 236-1 includes one or more event recognition units 280. Typically, a separate application view 291 includes a plurality of event recognition units 280. In other embodiments, one or more of the event recognition units 280 are part of a separate module, such as a user interface kit (not shown) or a higher-level object from which application 236-1 inherits methods and other properties. In some embodiments, a separate event processing unit 290 includes one or more event data 279 received from a data update unit 276, an object update unit 277, a GUI update unit 278, and / or an event sorting unit 270. The event processing unit 290 utilizes or calls the data update unit 276, the object update unit 277, or the GUI update unit 278 to update the application's internal state 292. Alternatively, one or more application views 291 include one or more event processing units 290. In some embodiments, one or more of the data update unit 276, object update unit 277, and GUI update unit 278 are included in individual application views 291.

[0119] Each individual event recognition unit 280 receives event information (e.g., event data 279) from the event sorting unit 270 and identifies events from the event information. The event recognition unit 280 includes an event receiving unit 282 and an event comparison unit 284. In some embodiments, the event recognition unit 280 includes at least a subset of metadata 283 and event distribution commands 288 (including sub-event distribution commands).

[0120] The event receiving unit 282 receives event information from the event sorting unit 270. The event information includes information about sub-events, such as touches or the movement of touches. Depending on the sub-event, the event information also includes additional information such as the location of the sub-event. If the sub-event relates to the movement of a touch, the event information also includes the velocity and direction of the sub-event. In some embodiments, an event includes a rotation of the device from one orientation to another (e.g., from portrait to landscape, or vice versa), and the event information includes corresponding information about the current orientation of the device (also called the device's orientation).

[0121] The event comparison unit 284 compares the event information with the definition of a default event or sub-event, and based on the comparison, determines the event or sub-event, or determines or updates the state of the event or sub-event. In some embodiments, the event comparison unit 284 includes an event definition 286. The event definition 286 includes definitions of events (e.g., a default sequence of default sub-events), such as event 1 (287-1) and event 2 (287-2). In some embodiments, sub-events within event (287) include, for example, touch start, touch end, touch movement, touch cancellation, and multiple touches. In one embodiment, the definition for event 1 (287-1) is a double tap on a displayed object. A double tap includes, for example, a first touch on the displayed object for a predetermined stage (touch start), a first lift-off for the predetermined stage (touch end), a second touch on the displayed object for the predetermined stage (touch start), and a second lift-off for the predetermined stage (touch end). In another example, the definition of event 2(287-2) is a drag on a displayed object. A drag includes, for example, a touch (or contact) on the displayed object to a predetermined stage, movement of the touch across the touch-sensitive display 212, and lift-off of the touch (end of touch). In some embodiments, the event also includes information about one or more associated event processing units 290.

[0122] In some embodiments, the event definition 287 includes event definitions for individual user interface objects. In some embodiments, the event comparison unit 284 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view where three user interface objects are displayed on the touch-sensitive display 212, when a touch is detected on the touch-sensitive display 212, the event comparison unit 284 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with an individual event processing unit 290, the event comparison unit uses the results of the hit test to determine which event processing unit 290 should be activated. For example, the event comparison unit 284 selects the sub-event and the event processing unit associated with the object that triggers the hit test.

[0123] In some embodiments, the definition of an individual event (287) also includes a delay action that delays the delivery of event information until it is determined whether the sequence of sub-events corresponds to an event type in the event recognition unit.

[0124] If an individual event recognition unit 280 determines that a series of sub-events does not match any of the events in the event definition 286, the individual event recognition unit 280 enters an event impossible, event failed, or event terminated state, and thereafter ignores subsequent sub-events of the touch-based gesture. In this situation, if there are other event recognition units that remain active for the hit view, those event recognition units continue to track and process the sub-events of the ongoing touch-based gesture.

[0125] In some embodiments, an individual event recognition unit 280 includes metadata 283 having configurable properties, flags, and / or lists that indicate to the actively involved event recognition unit how the event distribution system should perform sub-event distribution. In some embodiments, the metadata 283 includes configurable properties, flags, and / or lists that indicate how the event recognition units interact with each other, or how they can interact with each other. In some embodiments, the metadata 283 includes configurable properties, flags, and / or lists that indicate whether sub-events are distributed to various levels in the view hierarchy or program hierarchy.

[0126] In some embodiments, an individual event recognition unit 280 activates an event processing unit 290 associated with an event when one or more specific sub-events of an event are recognized. In some embodiments, the individual event recognition unit 280 delivers event information associated with the event to the event processing unit 290. Activating the event processing unit 290 is separate from sending (and delaying the sending of) sub-events to individual hit views. In some embodiments, the event recognition unit 280 sets a flag associated with the recognized event, and the event processing unit 290 associated with that flag captures the flag and executes a default process.

[0127] In some embodiments, the event distribution command 288 includes a sub-event distribution command that distributes event information about a sub-event without activating an event processing unit. Instead, the sub-event distribution command distributes event information to an event processing unit associated with a set of sub-events, or to an actively involved view. The event processing unit associated with the set of sub-events or the actively involved view receives the event information and executes a predetermined process.

[0128] In some embodiments, the data update unit 276 creates and updates data used in application 236-1. For example, the data update unit 276 updates telephone numbers used in contact module 237 or stores video files used in video player module. In some embodiments, the object update unit 277 creates and updates objects used in application 236-1. For example, the object update unit 277 creates new user interface objects or updates the positions of user interface objects. The GUI update unit 278 updates the GUI. For example, the GUI update unit 278 prepares display information and sends it to graphics module 232 for display on touch-sensitive display.

[0129] In some embodiments, the event processing unit(s) 290 includes or has access to a data update unit 276, an object update unit 277, and a GUI update unit 278. In some embodiments, the data update unit 276, the object update unit 277, and the GUI update unit 278 are contained in a single module of an individual application 236-1 or application view 291. In other embodiments, they are contained in two or more software modules.

[0130] The foregoing description regarding the handling of user touch events on a touch-sensitive display also applies to other forms of user input for operating the multifunction device 200 using input devices, but it should be understood that not all of these begin on the touchscreen. For example, mouse movement and mouse button presses, touch movements such as taps, drags, and scrolls on a touchpad, pen stylus input, device movement, verbal commands, detected eye movements, biometric input, and / or any combination thereof may be optionally used as inputs corresponding to sub-events that define the events to be recognized.

[0131] Figure 3 shows a portable multifunction device 200 having a touchscreen 212 according to several embodiments. The touchscreen optionally displays one or more graphics within a user interface (UI) 300. In this embodiment, and in other embodiments described below, the user can select one or more of the graphics by performing gestures on the graphics using, for example, one or more fingers 302 (not shown in the figure to an exact scale) or one or more styluses 303 (not shown in the figure to an exact scale). In some embodiments, the selection of one or more graphics is performed when the user interrupts contact with that one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, right to left, upward and / or downward) and / or rolling (from right to left, left to right, upward and / or downward) with a finger in contact with the device 200. In some implementations or situations, accidental contact with a graphic does not constitute a selection of that graphic. For example, if the gesture corresponding to selection is a tap, a swipe gesture sweeping over an application icon does not arbitrarily select the corresponding application.

[0132] Device 200 also includes one or more physical buttons, such as a "Home" or menu button 304. As described above, the menu button 304 is used to navigate to any application 236 within the set of applications running on device 200. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touchscreen 212.

[0133] In one embodiment, the device 200 includes a touchscreen 212, a menu button 304, a push button 306 for turning the device on / off and locking the device, one or more volume control buttons 308, a subscriber identification module (SIM) card slot 310, a headset jack 312, and an external docking / charging port 224. The push button 306 is optionally used to turn the device on / off by pressing down and holding the button down for a predetermined period of time, to lock the device by pressing down and releasing the button before a predetermined period of time has elapsed, and / or to unlock the device or to initiate an unlocking process. In an alternative embodiment, the device 200 also accepts verbal input via a microphone 213 to activate or deactivate certain functions. The device 200 also optionally includes one or more contact intensity sensors 265 for detecting the intensity of contact on the touchscreen 212, and / or one or more tactile output generators 267 for generating tactile output to the user of the device 200.

[0134] Figure 4 is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface, according to several embodiments. The device 400 does not need to be portable. In some embodiments, the device 400 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a game system, or a control device (e.g., a home or commercial controller). The device 400 typically includes one or more processing units (CPUs) 410, one or more network or other communication interfaces 460, memory 470, and one or more communication buses 420 that interconnect these components. The communication buses 420 optionally include circuitry (sometimes referred to as a chipset) that interconnects and controls communication between system components. The device 400 includes an input / output (I / O) interface 430, which includes a display 440, and the display 440 is typically a touchscreen display. The I / O interface 430 also optionally includes a keyboard and / or mouse (or other pointing device) 450, as well as a touchpad 455, a tactile output generator 457 (similar to the tactile output generator 267(single or multiple) described above with reference to Figure 2A) for generating tactile output on device 400, and a sensor 459 (e.g., an optical sensor, an accelerometer, a proximity sensor, a touch sensor, and / or a contact intensity sensor (similar to the contact intensity sensor(single or multiple) 265 described above with reference to Figure 2A). The memory 470 includes high-speed random-access memory such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices, and optionally includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. The memory 470 optionally includes one or more storage devices located remotely from the CPU(single or multiple) 410.In some embodiments, memory 470 stores programs, modules, and data structures, or subsets thereof, that are similar to those stored in memory 202 of the portable multifunction device 200 (Figure 2A). Furthermore, memory 470 optionally stores additional programs, modules, and data structures that are not present in memory 202 of the portable multifunction device 200. For example, memory 470 of device 400 optionally stores a drawing module 480, a presentation module 482, a word processing module 484, a website creation module 486, a disk authoring module 488, and / or a spreadsheet module 490, while memory 202 of the portable multifunction device 200 (Figure 2A) optionally does not store these modules.

[0135] Each element identified above in Figure 4 is stored in any one or more of the memory devices mentioned above, in some embodiments. Each module identified above corresponds to an instruction set that performs the function described above. The modules or programs (e.g., instruction sets) identified above do not need to be implemented as separate software programs, procedures, or modules; therefore, various subsets of these modules are combined or otherwise reconfigured in various embodiments. In some embodiments, memory 470 stores a subset of the modules and data structures identified above. Furthermore, memory 470 stores additional modules and data structures not described above.

[0136] Here, we focus on an embodiment of a user interface that can be implemented, for example, in a portable multi-functional device 200.

[0137] Figure 5A shows an exemplary user interface for an application menu on a portable multifunction device 200, according to some embodiments. A similar user interface is implemented on device 400. In some embodiments, the user interface 500 includes the following elements, or subsets or supersets thereof:

[0138] Signal strength indicators (single or multiple) for wireless communication (single or multiple) such as cellular and Wi-Fi signals 502, ●Time 504, ●Bluetooth indicator 505, ●Battery status indicator 506, ●Tray 508 containing icons for frequently used applications, as shown below. ○Optionally including an indicator 514 for the number of missed calls or voicemail messages, an icon 516 of the telephone module 238 labeled "Telephone", ○Optionally includes an indicator 510 for the number of unread emails, an icon 518 of the email client module 240 labeled "Mail", ○ Icon 520 of browser module 247, labeled as "Browser", and ○ Icon 522 for the video and music player module 252, also known as the iPod (trademark of Apple Inc.) module 252, which is labeled as "iPod," and ● Icons of other applications, such as the following: ○ Icon 524 of IM module 241, labeled "Message" ○ Icon 526 of calendar module 248, labeled "Calendar", ○ Icon 528 of image management module 244, labeled "Photo" ○ Icon 530 of camera module 243, labeled "Camera" ○ Icon 532 of online video module 255, labeled "online video" ○ Icon 534 of stock price widget 249-2, labeled "Stock Price" ○ Icon 536 of map module 254, labeled "Map" ○ Icon 538 of weather widget 249-1, labeled "Weather" ○ Icon 540 of the alarm clock widget 249-4, labeled as "Clock" ○ Icon 542 of training support module 242, labeled "Training Support" ○ Icon 544 of memo module 253, labeled as "Memo", and ○ An icon 546 labeled "Settings," which provides access to the settings of device 200 and its various applications 236, for a settings application or module.

[0139] Note that the icon labels shown in Figure 5A are for illustrative purposes only. For example, the icon 522 for the video and music player module 252 may optionally be labeled "Music" or "Music Player." Other labels may optionally be used for various application icons. In some embodiments, the label for an individual application icon may include the name of the application to which that individual application icon corresponds. In some embodiments, the label for a particular application icon may differ from the name of the application to which that particular application icon corresponds.

[0140] Figure 5B shows an exemplary user interface on a device (e.g., device 400 in Figure 4) having a touch-sensitive surface 551 (e.g., tablet or touchpad 455 in Figure 4) separate from the display 550 (e.g., touchscreen display 212). Device 400 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 459) for detecting the intensity of contact on the touch-sensitive surface 551, and / or one or more tactile output generators 457 for generating tactile output to the user of device 400.

[0141] Some of the following embodiments are described with reference to input on a touchscreen display 212 (when the touch-sensing surface and the display are combined), but in some embodiments, the device detects input on a touch-sensing surface separate from the display, as shown in Figure 5B. In some embodiments, this touch-sensing surface (e.g., 551 in Figure 5B) has a principal axis (e.g., 552 in Figure 5B) corresponding to a principal axis (e.g., 553 in Figure 5B) on the display (e.g., 550). According to these embodiments, the device detects contact with the touch-sensing surface 551 at locations corresponding to each location on the display (e.g., 560 and 562 in Figure 5B) (e.g., in Figure 5B, 560 corresponds to 568 and 562 corresponds to 570). In this method, if the touch-sensitive surface is separate from the display, user input detected by the device on the touch-sensitive surface (e.g., 551 in Figure 5B) (e.g., touches 560 and 562, and their movement) is used by the device to operate the user interface on the display of the multifunction device (e.g., 550 in Figure 5B). It should be understood that a similar method may be optionally used for other user interfaces described herein.

[0142] In addition, while the following examples are given primarily with reference to finger input (e.g., finger touch, finger tap gesture, finger swipe gesture), it should be understood that in some embodiments, one or more of the finger inputs may be replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture may optionally be replaced by a mouse click (e.g., instead of touch), followed by a mouse click with cursor movement along the swipe path (e.g., instead of touch movement). As another example, a tap gesture may optionally be replaced by a mouse click (e.g., instead of touch detection and subsequent cessation of touch detection) while the cursor is located over the tap gesture location. Similarly, it should be understood that when multiple user inputs are detected simultaneously, multiple computer mice may optionally be used simultaneously, or mouse and finger touch may optionally be used simultaneously.

[0143] Figure 6A shows an exemplary personal electronic device 600. Device 600 includes a body 602. In some embodiments, device 600 includes some or all of the features described in relation to devices 200 and 400 (e.g., Figures 2A to 4). In some embodiments, device 600 has a touch-sensitive display screen 604, hereafter referred to as touchscreen 604. Alternatively, or in addition to touchscreen 604, device 600 has a display and a touch-sensitive surface. Together with devices 200 and 400, in some embodiments, the touchscreen 604 (or touch-sensitive surface) has one or more intensity sensors that detect the intensity of the applied contact (e.g., touch). One or more intensity sensors on the touchscreen 604 (or touch-sensitive surface) provide output data representing the intensity of the touch. The user interface of device 600 responds to touches based on the intensity of the touch, meaning that touches of different intensity can invoke different user interface behaviors on device 600.

[0144] Techniques for detecting and processing touch intensity can be found, for example, in related applications: International Patent Application No. PCT / US2013 / 040061, filed on 8 May 2013, “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” and International Patent Application No. PCT / US2013 / 069483, filed on 11 November 2013, “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” each of which is incorporated herein by reference in whole.

[0145] In some embodiments, the device 600 has one or more input mechanisms 606 and 608. The input mechanisms 606 and 608 are physical, if included. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, the device 600 has one or more attachment mechanisms. Such attachment mechanisms, if included, can allow the device 600 to be attached to, for example, hats, eyeglasses, earrings, necklaces, shirts, jackets, bracelets, watch bands, chains, trousers, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow the user to wear the device 600.

[0146] Figure 6B shows an exemplary personal electronic device 600. In some embodiments, the device 600 includes some or all of the components described in relation to Figures 2A, 2B, and 4. The device 600 has a bus 612 that operably connects an I / O section 614 to one or more computer processors 616 and memory 618. The I / O section 614 is connected to a display 604 which may have a touch-sensitive component 622 and optionally a touch intensity-sensitive component 624. In addition, the I / O section 614 is connected to a communication unit 630 that receives application and operating system data using Wi-Fi, Bluetooth, near-field communication (NFC), cellular, and / or other wireless communication technologies. The device 600 includes input mechanisms 606 and / or 608. The input mechanism 606 is, for example, a rotatable input device or a pressable and rotatable input device. The input mechanism 608 is, in some embodiments, a button.

[0147] The input mechanism 608 is a microphone in some embodiments. The personal electronic device 600 includes various sensors, such as a GPS sensor 632, an accelerometer 634, a direction sensor 640 (e.g., a compass), a gyroscope 636, a motion sensor 638, and / or a combination thereof, all of which are operably connected to the I / O unit 614.

[0148] The memory 618 of the personal electronic device 600 is a non-temporary computer-readable storage medium that stores computer-executable instructions, and when executed by, for example, one or more computer processors 616, causes the computer processors to perform the following techniques and processes. These computer-executable instructions are also stored and / or transmitted in any non-temporary computer-readable storage medium for use by, or in connection with, instruction execution systems, devices, or other systems capable of fetching instructions from computer-based systems, systems including processors, or other systems capable of executing those instructions. The personal electronic device 600 is not limited to the components and configurations shown in Figure 6B and may include other or additional components in multiple configurations.

[0149] As used herein, the term “affordance” refers to, for example, user-interactive graphical user interface objects displayed on the display screens of devices 200, 400, and 600 (Figures 2A, 4, and 6A-6B). For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) each constitute an affordance.

[0150] As used herein, the term “focus selector” refers to an input element that indicates the current portion of the user interface with which the user is interacting. In some implementations, including a cursor or other location marker, the cursor acts as a “focus selector” so that when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 455 in Figure 4, or touch-sensitive surface 551 in Figure 5B) while the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), that particular user interface element is adjusted according to the detected input. In some implementations, including a touchscreen display (e.g., touch-sensitive display system 212 in Figure 2A, or touchscreen 212 in Figure 5A) that enables direct interaction with user interface elements on a touchscreen display, the touch detected on the touchscreen acts as a “focus selector” so that when an input (e.g., a press input by touch) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, that particular user interface element is adjusted according to the detected input. In some implementations, focus is moved from one area of ​​the user interface to another without corresponding cursor movement or touch movement on the touchscreen display (for example, by using the tab key or arrow keys to move focus from one button to another), and in these implementations, the focus selector moves in accordance with the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is generally a user interface element (or touch on the touchscreen display) controlled by the user to communicate the user's intended interaction with the user interface (for example, by pointing to the device an element of the user interface through which the user intends to interact).For example, the location of a focus selector (e.g., cursor, touch, or selection box) over an individual button while pressure input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen) indicates that the user intends to activate that individual button (rather than other user interface elements displayed on the device's display).

[0151] As used herein and in the claims, the term “characteristic intensity” of a contact refers to the characteristics of that contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a set of intensity samples, i.e., a set of intensity samples collected over a predetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) associated with a predetermined event (e.g., after detection of contact, before detection of lift-off of contact, before or after detection of the start of movement of contact, before detection of the end of contact, before or after detection of an increase in contact intensity, and / or before or after detection of a decrease in contact intensity). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum intensity of the contact, the median intensity of the contact, the average intensity of the contact, the top 10 percent of the contact intensity, half of the maximum intensity of the contact, 90 percent of the maximum intensity of the contact, and so on. In some embodiments, the duration of contact is used when determining characteristic intensity (for example, when characteristic intensity is the average intensity of contact over time). In some embodiments, characteristic intensity is compared to a set of one or more intensity thresholds to determine whether an action has been performed by the user. For example, the set of one or more intensity thresholds includes a first intensity threshold and a second intensity threshold. In this embodiment, contact with a characteristic intensity not exceeding the first threshold results in a first action, contact with a characteristic intensity above the first intensity threshold but not exceeding the second intensity threshold results in a second action, and contact with a characteristic intensity above the second threshold results in a third action. In some embodiments, the comparison of characteristic intensity to one or more thresholds is not used to determine whether to perform the first or second action, but rather to determine whether to perform one or more actions at all (for example, whether to perform individual actions or to refrain from performing individual actions).

[0152] In some embodiments, a portion of the gesture is identified for the purpose of determining characteristic intensity. For example, a touch-sensitive surface receives a series of swipe contacts that transition from a starting location point to an ending location point where the contact intensity increases. In this example, the characteristic intensity of the contact at the ending location is based only on a portion of the series of swipe contacts, and not on the entire swipe contact (e.g., only a portion of the swipe contact at the ending location). In some embodiments, a smoothing algorithm is applied to the intensity of the swipe contact before determining the characteristic intensity of the contact. For example, the smoothing algorithm may optionally include one or more of the following: an unweighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some situations, these smoothing algorithms eliminate narrow spikes or drops in swipe contact intensity for the purpose of determining characteristic intensity.

[0153] The intensity of contact on a touch-sensitive surface is characterized with respect to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to the intensity at which the device performs an action typically associated with clicking a physical mouse button or trackpad. In some embodiments, the deep press intensity threshold corresponds to the intensity at which the device performs an action different from the action typically associated with clicking a physical mouse button or trackpad. In some embodiments, when a contact with a characteristic intensity below the light press intensity threshold (for example, above a nominal contact detection intensity threshold below which contact is no longer detected) is detected, the device moves the focus selector in accordance with the movement of the contact on the touch-sensitive surface without performing an action associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise specified, these intensity thresholds are consistent across different sets of user interface values.

[0154] An increase in the characteristic intensity of contact from an intensity below a light pressure intensity threshold to an intensity between the light and deep pressure intensity thresholds is sometimes referred to as a "light pressure" input. An increase in the characteristic intensity of contact from an intensity below a deep pressure intensity threshold to an intensity above a deep pressure intensity threshold is sometimes referred to as a "deep pressure" input. An increase in the characteristic intensity of contact from an intensity below a contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light pressure intensity threshold is sometimes referred to as detection of contact on the touch surface. A decrease in the characteristic intensity of contact from an intensity above a contact detection intensity threshold to an intensity below a contact detection intensity threshold is sometimes referred to as detection of contact lift-off from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0155] In some embodiments described herein, one or more actions are performed in response to the detection of a gesture including an individual press input, or in response to the detection of an individual press input performed by an individual contact (or multiple contacts), wherein the individual press input is detected at least in part on the detection of an increase in the intensity of the contact (or multiple contacts) above a press input intensity threshold. In some embodiments, the individual action is performed in response to the detection of an increase in the intensity of the individual contact above a press input intensity threshold (e.g., a "downstroke" of the individual press input). In some embodiments, the press input includes an increase in the intensity of the individual contact above a press input intensity threshold, followed by a decrease in the intensity of the contact below the press input intensity threshold, and the individual action is performed in response to the detection of a subsequent decrease in the intensity of the individual contact below the press input threshold (e.g., an "upstroke" of the individual press input).

[0156] In some embodiments, the device employs intensity hysteresis to avoid accidental inputs, which may be referred to as “jitter,” and the device defines or selects a hysteresis intensity threshold that has a predetermined relationship with a press input intensity threshold (for example, the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable percentage of the press input intensity threshold). Thus, in some embodiments, the press input includes an increase in the intensity of an individual contact above the press input intensity threshold, followed by a decrease in the intensity of the contact below the hysteresis intensity threshold corresponding to the press input intensity threshold, and the individual action is performed in response to the detection of a subsequent decrease in the intensity of an individual contact below the hysteresis intensity threshold (for example, an “upstroke” of the individual press input). Similarly, in some embodiments, a press input is detected only when the device detects an increase in contact intensity from an intensity below a hysteresis intensity threshold to an intensity above a press input intensity threshold, and optionally a decrease in contact intensity to an intensity below the hysteresis intensity thereafter, and individual actions are performed in response to the detection of a press input (e.g., an increase in contact intensity or a decrease in contact intensity, depending on the situation).

[0157] For the sake of clarity, the description of an action performed in response to a press input associated with a press input intensity threshold, or a gesture involving a press input, is optionally triggered in response to the detection of any of the following: an increase in contact intensity above the press input intensity threshold, an increase in contact intensity from below the hysteresis intensity threshold to above the press input intensity threshold, a decrease in contact intensity below the press input intensity threshold, and / or a decrease in contact intensity below the hysteresis intensity threshold corresponding to the press input intensity threshold. Furthermore, in examples where an action is described to be performed in response to the detection of a decrease in contact intensity below the press input intensity threshold, the action is optionally performed in response to the detection of a decrease in contact intensity below a hysteresis intensity threshold corresponding to and lower than the press input intensity threshold. 3. Digital Assistant System

[0158] Figure 7A shows block diagrams of the digital assistant system 700 according to various embodiments. In some embodiments, the digital assistant system 700 is implemented on a standalone computer system. In some embodiments, the digital assistant system 700 is distributed across multiple computers. In some examples, some of the modules and functions of the digital assistant are allocated to a server portion and a client portion, the client portion residing on one or more user devices (e.g., devices 104, 122, 200, 400, 600, 810a, 810b, 820a, 830a, 830b), as shown in Figure 1, and communicating with the server portion (e.g., server system 108) through one or more networks. In some embodiments, the digital assistant system 700 is an implementation of the server system 108 (and / or DA server 106) shown in Figure 1. It should be noted that the digital assistant system 700 is merely one embodiment of a digital assistant system, and the digital assistant system 700 may have more or fewer components than those shown, may combine two or more components, or may have different configurations or arrangements of those components. The various components shown in Figure 7A may be implemented as hardware including one or more signal processing circuits and / or application-specific integrated circuits, software instructions executed by one or more processors, firmware, or a combination thereof.

[0159] The digital assistant system 700 includes a memory 702, one or more processors 704, an input / output (I / O) interface 706, and a network communication interface 708. These components can communicate with each other via one or more communication buses or signal lines 710.

[0160] In some embodiments, the memory 702 includes a non-transient computer-readable medium, such as a high-speed random-access memory and / or a non-volatile computer-readable storage medium (e.g., one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices).

[0161] In some embodiments, the I / O interface 706 connects the input / output devices 716 of the digital assistant system 700, such as a display, keyboard, touchscreen, and microphone, to the user interface module 722. The I / O interface 706 works in conjunction with the user interface module 722 to receive and process user input (e.g., voice input, keyboard input, touch input, etc.). In some examples, for example, when the digital assistant is implemented on a standalone user device, the digital assistant system 700 includes any of the components and I / O communication interfaces described in relation to devices 200, 400, 600, 810a, 810b, 820a, 830a, and 830b, respectively, in Figures 2A, 4, 6A-6B, and 8. In some examples, the digital assistant system 700 represents the server portion of the digital assistant implementation, and can interact with the user through a client portion residing on a user device (e.g., devices 104, 200, 400, 600, or 810a, 810b, 820a, 830a, 830b).

[0162] In some embodiments, the network communication interface 708 includes one or more wired communication ports 712 and / or a wireless transceiver circuit 714. The one or more wired communication ports transmit and receive communication signals via one or more wired interfaces, such as Ethernet, Universal Serial Bus (USB), FireWire, etc. The wireless circuit 714 transmits and receives RF signals and / or optical signals to and from the communication network and other communication devices. Wireless communication uses one of several communication standards, communication protocols, and communication technologies, such as GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol. The network communication interface 708 enables communication between the digital assistant system 700 and networks such as the Internet, intranets, and / or wireless networks such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs), and between the digital assistant system 700 and other devices.

[0163] In some embodiments, memory 702, or the computer-readable storage medium of memory 702, stores programs, modules, instructions, and data structures, including all or a subset of the operating system 718, communication module 720, user interface module 722, one or more applications 724, and digital assistant module 726. In particular, memory 702, or the computer-readable storage medium of memory 702, stores instructions for executing the processes described below. One or more processors 704 execute these programs, modules, and instructions, and read from / write to data structures.

[0164] Operating Systems 718 (e.g., embedded operating systems such as Darwin, RTXC, LINUX, UNIX®, iOS, OS X, WINDOWS®, or VxWorks) include various software components and / or drivers for controlling and managing common system tasks (e.g., memory management, storage device control, power management, etc.) and facilitate communication between various hardware, firmware, and software components.

[0165] The communication module 720 facilitates communication between the digital assistant system 700 and other devices via the network communication interface 708. For example, the communication module 720 communicates with the RF circuit 208 of electronic devices such as devices 200, 400, and 600 shown in Figures 2A, 4, and 6A-6B, respectively. The communication module 720 also includes various components for processing data received by the wireless circuit 714 and / or the wired communication port 712.

[0166] The user interface module 722 receives commands and / or input from the user via the I / O interface 706 (e.g., from a keyboard, touchscreen, pointing device, controller, and / or microphone) and generates user interface objects on the display. The user interface module 722 also prepares and delivers outputs to the user (e.g., speech, sound, animation, text, icons, vibration, haptic feedback, light, etc.) via the I / O interface 706 (e.g., through the display, audio channels, speakers, touchpad, etc.).

[0167] Application 724 includes programs and / or modules configured to run by one or more processors 704. For example, if the digital assistant system is implemented on a standalone user device, application 724 includes user applications such as games, calendar applications, navigation applications, or email applications. If the digital assistant system 700 is implemented on a server, application 724 includes, for example, resource management applications, diagnostic applications, or scheduling applications.

[0168] Memory 702 also stores the digital assistant module 726 (or the server portion of the digital assistant). In some embodiments, the digital assistant module 726 includes the following submodules, or subsets or supersets thereof: input / output processing module 728, speech-to-text (STT) processing module 730, natural language processing module 732, dialogue flow processing module 734, task flow processing module 736, service processing module 738, and speech synthesis processing module 740. Each of these modules has access to one or more of the following systems or data and models of the digital assistant module 726, or subsets or supersets thereof: ontology 760, vocabulary index 744, user data 748, task flow model 754, service model 756, and ASR system 758.

[0169] In some embodiments, by using the processing modules, data, and models implemented in the digital assistant module 726, the digital assistant can perform at least some of the following: converting speech input to text; identifying the user's intent expressed in the natural language input received from the user; actively extracting and obtaining the information necessary to fully infer the user's intent (e.g., by clarifying words, games, intentions, etc.); determining a task flow to satisfy the inferred intent; and executing that task flow to satisfy the inferred intent.

[0170] In some embodiments, as shown in Figure 7B, the I / O processing module 728 interacts with the user through the I / O device 716 in Figure 7A, or with a user device (e.g., device 104, 200, 400, or device 600) through the network communication interface 708 in Figure 7A, in order to acquire user input (e.g., speech input) and to provide a response to the user input (e.g., as speech output). The I / O processing module 728 optionally acquires contextual information associated with the user input from the user device, either with the user input or immediately after receiving the user input. The contextual information includes user-specific data, vocabulary, and / or preferences related to the user input. In some embodiments, the contextual information also includes the software and hardware state of the user device at the time the user request is received, and / or information about the user's surrounding environment at the time the user request is received. In some embodiments, the I / O processing module 728 also sends supplementary questions to the user regarding the user request and receives answers from the user. When a user request is received by the I / O processing module 728 and the user request includes speech input, the I / O processing module 728 forwards the speech input to the STT processing module 730 (or speech recognition device) for speech-to-text conversion.

[0171] The STT processing module 730 includes one or more ASR systems 758. These one or more ASR systems 758 can process speech input received via the I / O processing module 728 to generate recognition results. Each ASR system 758 includes a front-end speech preprocessor. This front-end speech preprocessor extracts representative features from the speech input. For example, the front-end speech preprocessor extracts spectral features that characterize the speech input as a representative multidimensional vector sequence by performing a Fourier transform on the speech input. Furthermore, each ASR system 758 includes one or more speech recognition models (e.g., acoustic models and / or language models) and implements one or more speech recognition engines. Examples of speech recognition models include hidden Markov models, Gaussian mixture models, deep neural network models, n-gram language models, and other statistical models. Examples of speech recognition engines include dynamic time stretching-based engines and weighted finite-state transducer (WFST)-based engines. One or more speech recognition models and one or more speech recognition engines are used to process the extracted representative features of a front-end speech preprocessor to generate intermediate recognition results (e.g., phonemes, phoneme sequences, subwords) and finally text recognition results (words, word sequences, token sequences). In some embodiments, the utterance input is processed at least partially by a third-party service or on the user's device (e.g., device 104, 200, 400, or device 600) to generate the recognition results. Once the STT processing module 730 generates a recognition result containing a text string (e.g., a word, a word sequence, or a token sequence), the recognition result is passed to the natural language processing module 732 for intent inference. In some embodiments, the STT processing module 730 generates a number of text representation candidates for the utterance input. Each text representation candidate is a sequence of words or tokens corresponding to the utterance input. In some embodiments, each text representation candidate is associated with a speech recognition confidence score.Based on this speech recognition reliability score, the STT processing module 730 ranks the text representation candidates and provides the n best (e.g., n highest-ranked) text representation candidates (one or more) to the natural language processing module 732 for intent inference (where n is a predetermined integer greater than zero). For example, in one embodiment, only the highest-ranked (n=1) text representation candidate is passed to the natural language processing module 732 for intent inference. In another embodiment, five highest-ranked (n=5) text representation candidates are passed to the natural language processing module 732 for intent inference.

[0172] Further details regarding the speech-to-text processing are described in U.S. Utility Patent Application No. 13 / 236,942, filed September 20, 2011, for "Consolidating Speech Recognition Results," the entire disclosure of which is incorporated herein by reference.

[0173] In some embodiments, the STT processing module 730 includes a vocabulary of recognizable words and / or accesses that vocabulary via the phonetic-to-text conversion module 731. Each vocabulary word is associated with one or more pronunciation candidates for that word, represented by speech-recognition phonetic characters. Specifically, the vocabulary of recognizable words includes words associated with multiple pronunciation candidates. For example, the vocabulary may include:

number

[0174] In some embodiments, pronunciation candidates are ranked based on their generality. For example, pronunciation candidates

number

number

number

number

number

number

[0175] When speech input is received, the STT processing module 730 is used to determine the phonemes corresponding to the speech input (for example, using an acoustic model), and then attempts to determine the words that match those phonemes (for example, using a language model). For example, the STT processing module 730 first determines the sequence of phonemes corresponding to a portion of the speech input.

number

[0176] In some embodiments, the STT processing module 730 uses approximate matching techniques to determine words in a utterance. For example, the STT processing module 730 uses a sequence of phonemes.

number

[0177] The digital assistant's natural language processing module 732 ("natural language processor") retrieves n best text representation candidates (singular or plural) ("word sequences (singular or plural)" or "token sequences (singular or plural)") generated by the STT processing module 730 and attempts to associate each of these text representation candidates with one or more "actionable intentions" recognized by the digital assistant. An "actionable intention" (or "user intention") represents a task that the digital assistant can perform and may have an associated task flow realized within the task flow model 754. This associated task flow is a series of programmed actions and steps that the digital assistant performs to perform that task. The scope of the digital assistant's capabilities is determined by the number and variety of task flows realized and stored within the task flow model 754, or in other words, by the number and variety of "actionable intentions" recognized by the digital assistant. However, the effectiveness of a digital assistant is also judged by its ability to infer the correct "feasible intent(s)" from user requests expressed in natural language.

[0178] In some embodiments, in addition to the sequence of words or tokens obtained from the STT processing module 730, the natural language processing module 732 also receives contextual information associated with the user request, for example, from the I / O processing module 728. The natural language processing module 732 optionally uses this contextual information to clarify, complement, and / or further define the information contained within the text representation candidates received from the STT processing module 730. The contextual information includes, for example, user preferences, the hardware and / or software state of the user device, sensor information collected before, during, or immediately after the user request, and previous interactions (e.g., dialogues) between the digital assistant and the user. As described herein, the contextual information is dynamic in some embodiments and changes with time, location, dialogue content, and other factors.

[0179] In some embodiments, natural language processing is based on, for example, Ontology 760. Ontology 760 is a hierarchical structure containing numerous nodes, each representing either an “executable intent” or an “attribute” or other “attribute” associated with one or more “executable intents.” As described above, an “executable intent” represents a task that a digital assistant can perform; that is, the task is “executable” or can be performed. An “attribute” represents a parameter associated with an implementable intent, or associated with a sub-part of another attribute. Links between implementable intent nodes and attribute nodes within Ontology 760 define how the parameter represented by the attribute node relates to the task represented by the implementable intent node.

[0180] In some embodiments, the ontology 760 consists of actionable intent nodes and attribute nodes. Within the ontology 760, each actionable intent node is directly linked to one or more attribute nodes, or linked via one or more intermediate attribute nodes. Similarly, each attribute node is directly linked to one or more actionable intent nodes, or linked via one or more intermediate attribute nodes. For example, as shown in Figure 7C, the ontology 760 includes a “Restaurant Reservation” node (i.e., an actionable intent node). The attribute nodes “Restaurant,” “Date / Time” (for reservation), and “Number of Participants” are each directly linked to the actionable intent node (i.e., the “Restaurant Reservation” node).

[0181] Furthermore, the attribute nodes "Cuisine," "Price Range," "Phone Number," and "Location" are sub-nodes of the attribute node "Restaurant," and are each linked to the "Restaurant Reservation" node (i.e., an actionable intent node) via the intermediate attribute node "Restaurant." As another example, as shown in Figure 7C, ontology 760 also includes a "Reminder Settings" node (i.e., another actionable intent node). The attribute nodes "Date / Time" (for reminder settings) and "Theme" (for reminders) are each linked to the "Reminder Settings" node. The attribute node "Date / Time" is related to both the task of making a restaurant reservation and the task of setting a reminder, and therefore, within ontology 760, the attribute node "Date / Time" is linked to both the "Restaurant Reservation" node and the "Reminder Settings" node.

[0182] An actionable intent node, along with its linked attribute nodes, is described as a “domain.” In this paper, each domain is associated with a specific actionable intent and refers to a group of nodes (and the relationships between those nodes) associated with that particular actionable intent. For example, ontology 760, shown in Figure 7C, contains an example of a restaurant reservation domain 762 and an example of a reminder domain 764 within ontology 760. The restaurant reservation domain includes the actionable intent node “Restaurant Reservation,” attribute nodes “Restaurant,” “Date / Time,” and “Number of Participants,” as well as subordinate attribute nodes “Cuisine,” “Price Range,” “Phone Number,” and “Location.” The reminder domain 764 includes the actionable intent node “Reminder Settings,” as well as attribute nodes “Theme” and “Date / Time.” In some embodiments, ontology 760 consists of multiple domains. Each domain shares one or more attribute nodes with one or more other domains. For example, the "Date / Time" attribute node is associated with numerous different domains (e.g., scheduling domains, travel booking domains, movie ticket domains, etc.) in addition to the restaurant reservation domain 762 and the reminder domain 764.

[0183] Figure 7C shows two exemplary domains within Ontology 760, but other domains include, for example, “Find a movie,” “Start a phone call,” “Find directions,” “Schedule a meeting,” “Send a message,” “Provide answers to questions,” “Read a list,” “Provide navigation instructions,” and “Provide instructions for a task.” The “Send a message” domain is associated with the actionable intent of “Send a message” and further includes attribute nodes such as “Recipients,” “Message type,” and “Message body.” The attribute node “Recipients” is further defined by sub-attribute nodes such as “Recipient name” and “Message address.”

[0184] In some embodiments, the ontology 760 includes all domains (and therefore, all actionable intentions) that the digital assistant can understand and perform. In some embodiments, the ontology 760 is modified by adding or removing domains or entire nodes, or by modifying the relationships between nodes within the ontology 760.

[0185] In some embodiments, nodes associated with multiple related actionable intentions are clustered under a “higher-level domain” within Ontology 760. For example, the “Travel” higher-level domain contains clusters of attribute nodes and actionable intention nodes related to travel. Actionable intention nodes related to travel include “Book an airline ticket,” “Book a hotel,” “Rent a car,” “Get directions,” and “Find points of interest.” Actionable intention nodes under the same higher-level domain (e.g., the “Travel” higher-level domain) share a number of attribute nodes. For example, actionable intention nodes related to “Book a flight ticket,” “Book a hotel,” “Rent a car,” “Get directions,” and “Find places of interest” share one or more of the attribute nodes “Departure location,” “Destination,” “Departure date / time,” “Arrival date / time,” and “Number of participants.”

[0186] In some embodiments, each node in the ontology 760 is associated with a set of words and / or phrases related to the attribute or actionable intention represented by that node. The distinct set of words and / or phrases associated with each node is the so-called "vocabulary" associated with that node. The distinct set of words and / or phrases associated with each node is stored in the vocabulary index 744 in relation to the attribute or actionable intention represented by that node. For example, returning to Figure 7B, the vocabulary associated with the node relating to the attribute of "restaurant" includes words such as "food," "drink," "cooking," "hungry," "eat," "pizza," "fast food," and "meal." As another example, the vocabulary associated with the actionable intention of "make a phone call" includes words and phrases such as "call," "phone," "dial," "ring," "call this number," and "make a call to." The vocabulary index 744 optionally includes words and phrases from different languages.

[0187] The natural language processing module 732 receives candidate text representations (e.g., strings (single or multiple) or token sequences (single or multiple)) from the STT processing module 730 and determines, for each candidate representation, which node the words in the candidate text representation imply. In some embodiments, if it is found (via the lexical index 744) that a word or phrase in the candidate text representation is associated with one or more nodes in the ontology 760, then that word or phrase “triggers” or “activates” those nodes. Based on the number and / or relative importance of the activated nodes, the natural language processing module 732 selects one of those actionable intentions as the task that the user intends the digital assistant to perform. In some embodiments, the domain with the most “triggered” nodes is selected. In some embodiments, the domain with the highest confidence value (e.g., based on the relative importance of the various triggered nodes) is selected. In some embodiments, the domain is selected based on a combination of the number and importance of the triggered nodes. In some embodiments, additional factors are considered when selecting a node, such as whether the digital assistant has previously interpreted a similar request from the user accurately.

[0188] User data 748 includes user-specific information such as user-specific vocabulary, user preferences, user address, user's default and second languages, user's contact list, and other short-term or long-term information about each user. In some embodiments, the natural language processing module 732 uses this user-specific information to complement the information contained in the user input and to further define the user intent. For example, with respect to the user request "invite my friends to my birthday party," the natural language processing module 732 can access user data 748 to determine who the "friends" are and when and where the "birthday party" will take place, without requiring the user to explicitly provide such information in the user request.

[0189] It should be noted that in some embodiments, the natural language processing module 732 is implemented using one or more machine learning mechanisms (e.g., neural networks). Specifically, one or more machine learning mechanisms are configured to receive text representation candidates and contextual information associated with those text representation candidates. Based on these text representation candidates and associated contextual information, one or more machine learning mechanisms are configured to determine intent confidence scores across a set of feasible intent candidates. Based on the determined intent confidence scores, the natural language processing module 732 can select one or more feasible intent candidates from the set of feasible intent candidates. In some embodiments, an ontology (e.g., ontology 760) is also used to select one or more feasible intent candidates from the set of feasible intent candidates.

[0190] Further details regarding the token string-based ontology search are described in U.S. Utility Patent Application No. 12 / 341,743, filed December 22, 2008, “Method and Apparatus for Searching Using an Active Ontology,” which is incorporated herein by reference in its entirety.

[0191] In some embodiments, once the natural language processing module 732 identifies an actionable intent (or domain) based on a user request, it generates a structured query to represent that identified actionable intent. In some embodiments, this structured query includes parameters for one or more nodes within the domain relating to that actionable intent, with at least some of these parameters being populated with specific information and requirements specified in the user request. For example, the user says, "Make me a dinner reservation at a sushi place at 7." In this case, the natural language processing module 732 can accurately identify the actionable intent as "restaurant reservation" based on the user input. According to the ontology, the structured query for the "restaurant reservation" domain includes parameters such as {cuisine}, {time}, {date}, and {number of people}. In some embodiments, based on the utterance input and the text derived from the utterance input using the STT processing module 730, the natural language processing module 732 generates a partially structured query for a restaurant reservation domain, which includes the parameters {cuisine = "sushi"} and {time = "7pm"}. However, in this embodiment, the information contained in the user's utterance is insufficient to complete the structured query associated with that domain. Therefore, other necessary parameters such as {number of participants} and {date} are not specified in the structured query based on the currently available information. In some embodiments, the natural language processing module 732 adds received contextual information to some of the parameters of this structured query. For example, in some embodiments, if the user requests a "nearby" sushi restaurant, the natural language processing module 732 adds GPS coordinates from the user's device to the {location} parameter in the structured query.

[0192] In some embodiments, the natural language processing module 732 identifies multiple feasible intent candidates for each text representation candidate received from the STT processing module 730. Furthermore, in some embodiments, a separate (partially or completely) structured query is generated for each of the identified feasible intent candidates. The natural language processing module 732 determines an intent confidence score for each feasible intent candidate and ranks them based on that intent confidence score. In some embodiments, the natural language processing module 732 passes the generated structured query(s) containing arbitrary input parameters to the task flow processing module 736 ("task flow processor"). In some embodiments, the task flow processing module 736 is provided with structured queries(s) for the m best (e.g., m highest-ranked) feasible intent candidates (m being a predetermined integer greater than zero). In some embodiments, a structured query(s) relating to m best-feasible intent candidates, along with corresponding textual representation candidates(s), is provided to the task flow processing module 736.

[0193] Further details of user intent inference based on multiple viable intent candidates determined from multiple textual representation candidates of speech input are described in U.S. Utility Patent Application No. 14 / 298,725, filed June 6, 2014, for "System and Method for Inferring User Intent From Speech Inputs," the entire disclosure of which is incorporated herein by reference.

[0194] The task flow processing module 736 is configured to receive a structured query (one or more) from the natural language processing module 732, complete the structured query as needed, and perform the actions required to "complete" the user's final request. In some embodiments, the various steps required to complete these tasks are provided within the task flow model 754. In some embodiments, the task flow model 754 includes steps for obtaining additional information from the user and a task flow for performing actions associated with actionable intents.

[0195] As described above, in order to complete the structured query, the task flow processing module 736 needs to initiate an additional dialogue with the user to obtain additional information and / or to remove ambiguity from potentially ambiguous statements. If such a dialogue is necessary, the task flow processing module 736 calls the dialogue flow processing module 734 to engage in the dialogue with the user. In some embodiments, the dialogue flow processing module 734 determines how (and / or when) to request additional information from the user and receives and processes the user's response. Questions are provided to the user and answers are received from the user via the I / O processing module 728. In some embodiments, the dialogue flow processing module 734 presents the dialogue output to the user via audio and / or visual output and receives input from the user via verbal or physical (e.g., click) responses. Continuing with the above embodiment, when the task flow processing module 736 calls the dialog flow processing module 734 to determine the "number of participants" and "date" information for the structured query associated with the domain "restaurant reservation", the dialog flow processing module 734 generates questions such as "For how many people?" and "On which day?" and passes them to the user. Once the user's answer is received, the dialog flow processing module 734 then passes that information to the task flow processing module 736 to either add the missing information to the structured query or to complete the missing information from the structured query.

[0196] Once the task flow processing module 736 completes the structured query for an actionable intent, it proceeds to execute the final task associated with that actionable intent. Thus, the task flow processing module 736 executes the steps and instructions in the task flow model according to the specific parameters contained within the structured query. For example, the task flow model for the actionable intent of "restaurant reservation" includes steps and instructions to contact a restaurant and actually request a reservation for a specific number of people at a specific time. For example, using a structured query such as {restaurant reservation, restaurant=ABC Cafe, date=3 / 12 / 2012, time=7pm, number of people=5}, the task flow processing module 736 executes the following steps: (1) log on to the ABC Cafe server or a restaurant reservation system such as OPENTABLE®; (2) enter the date, time, and number of people information into a form on the website; (3) submit the form; and (4) enter the calendar item for the reservation into the user's calendar.

[0197] In some embodiments, the task flow processing module 736 employs the assistance of the service processing module 738 ("service processing module") to complete a task requested by user input or to provide a response to information requested by user input. For example, the service processing module 738 performs functions on behalf of the task flow processing module 736, such as making phone calls, setting calendar items, calling map searches, calling or interacting with other user applications installed on the user device, or calling or interacting with third-party services (e.g., restaurant reservation portals, social networking websites, banking portals, etc.). In some embodiments, the protocols and application programming interfaces (APIs) required by each service are specified by individual service models in the service model 756. The service processing module 738 accesses the appropriate service model for the service and generates a request for that service in accordance with that service model and the protocols and APIs required by that service.

[0198] For example, if a restaurant supports an online reservation service, the restaurant submits a service model that specifies the parameters required to make a reservation and an API for communicating the values ​​of those parameters to the online reservation service. When requested by the task flow processing module 736, the service processing module 738 establishes a network connection with the online reservation service using the web address stored in the service model and sends the required reservation parameters (e.g., time, date, number of participants) to the online reservation interface in the format specified by the online reservation service's API.

[0199] In some embodiments, the natural language processing module 732, the dialogue flow processing module 734, and the task flow processing module 736 are used collectively and iteratively to infer and define user intent, acquire information to further clarify and refine that user intent, and ultimately generate a response (i.e., output to the user or completion of a task) that satisfies the user intent. The generated response is a dialogue response to a speech input that satisfies at least partially the user intent. Furthermore, in some embodiments, the generated response is output as a speech output. In these embodiments, the generated response is sent to a speech synthesis processing module 740 (e.g., a speech synthesizer), which can process it to synthesize a dialogue response in speech form. In yet other embodiments, the generated response is data content related to satisfying a user request in the speech input.

[0200] In embodiments where the task flow processing module 736 receives multiple structured queries from the natural language processing module 732, the task flow processing module 736 first attempts to process a first structured query from the received structured queries to complete it and / or to perform one or more tasks or actions represented by it. In some embodiments, the first structured query corresponds to the highest-ranked actionable intent. In other embodiments, the first structured query is selected from the received structured queries based on a combination of the corresponding speech recognition confidence score and the corresponding intent confidence score. In some embodiments, if the task flow processing module 736 encounters an error while processing the first structured query (for example, due to an undeterminable required parameter), the task flow processing module 736 may proceed to select and process a second structured query from the received structured queries that corresponds to a lower-ranked actionable intent. This second structured query is selected based, for example, on the speech recognition confidence score of the corresponding text representation candidate, the intent confidence score of the corresponding actionable intent candidate, any missing required parameters in the first structured query, or any combination thereof.

[0201] The speech synthesis processing module 740 is configured to synthesize speech output for presentation to the user. The speech synthesis processing module 740 synthesizes speech output based on text provided by the digital assistant. For example, the generated dialogue response is in the form of a text string. The speech synthesis processing module 740 converts this text string into audible speech output. The speech synthesis processing module 740 uses, but is not limited to, any appropriate speech synthesis technique to generate speech output from text, including, but is not limited to, waveform concatenation synthesis, unit selection synthesis, diphon synthesis, domain-specific synthesis, formant synthesis, articulation synthesis, synthesis based on hidden Markov models (HMMs), and sine wave synthesis. In some embodiments, the speech synthesis processing module 740 is configured to synthesize individual words based on phoneme sequences corresponding to words. For example, phoneme sequences are associated with words in the generated dialogue response. The phoneme sequences are stored in metadata associated with the words. The speech synthesis processing module 740 is configured to synthesize words in speech format by directly processing phoneme sequences in the metadata.

[0202] In some embodiments, instead of using (or in addition to) the speech synthesis processing module 740, speech synthesis is performed on a remote device (e.g., a server system 108), and the synthesized speech is sent to a user device for output to the user. For example, this can be done in some implementations where the output for the digital assistant is generated on the server system. Also, since the server system generally has more processing power or resources than the user device, it is possible to obtain higher quality speech output than would be practical in the case of client-side synthesis.

[0203] Further details regarding the digital assistant can be found in U.S. Utility Application No. 12 / 987,982, filed on 10 January 2011, entitled "Intelligent Automated Assistant," and in U.S. Utility Application No. 13 / 251,088, filed on 30 September 2011, entitled "Generating and Processing Task Items That Represent Tasks to Perform," the entire disclosure of which is incorporated herein by reference. 4. Techniques for managing time-limited tasks

[0204] Figures 8 and 9A-9F illustrate various techniques for managing timed tasks through different embodiments. It will be understood that the individual embodiments described in Figures 8 and 9A-9F can be combined in any way and at will. These figures are also used to illustrate the processes described below, including process 1000 in Figure 10.

[0205] Figure 8 shows an environment 800 including user 802 and electronic devices 810a-b, 820a, and 830a-b. In the illustrated example, each of devices 810a, 820a, and 830a is an audio output device such as a smart speaker, device 810b is a set-top box coupled to a display device (e.g., a television) (not shown), and device 830b is a thermostat (e.g., a smart thermostat). Although devices 810a, 810b, 820a, 830a, and 830b are shown as the aforementioned device types, it will be understood that each of 810a, 810b, 820a, 830a, and 830b could be any type of device, including, but not limited to, mobile devices, desktop computers, laptop computers, tablets, head-mounted displays (HMDs), smart glasses, IoT devices, speakers, set-top boxes, smartwatches, smart displays (e.g., smart televisions), or any combination thereof. In some examples, devices 810a, 810b, 820a, 830a, and / or 830b may operate in any number of operating modes, including hands-on and hands-free modes, as described in U.S. Patent Application No. 63 / 465,234, filed May 9, 2023, entitled “Multi-modal Digital Assistant,” which is incorporated herein by reference in its entirety.

[0206] In some examples, one or more of the electronic devices 810a, 810b, 820a, 830a, and 830b are associated with a device ecosystem (e.g., a home automation network) and / or are configured to communicate with each other via a network (e.g., Wi-Fi) implemented within the environment 800. Thus, each of the electronic devices 810a-b, 820a, and 830a-b can optionally correspond to a location and / or a device name. A user can, for example, manually assign a name or location to a device. Referring to Figure 8, for example, devices 810a and 810b are named "John's Speaker" and "TV," respectively, and correspond to location 810 (e.g., "Living Room"). In another example, device 820a is named "Bedroom Speaker" and corresponds to location 820 (e.g., "Bedroom"). As yet another example, devices 830a and 830b are named "office speaker" and "thermostat," respectively, and correspond to location 830 (for example, "office").

[0207] In some examples, timed tasks can be managed using one or more devices in environment 800. Generally, timed tasks include tasks that have one or more temporal aspects or parameters, such as timers (e.g., countdowns for a specified or predetermined amount of time) or alarms (e.g., notifications provided at a specified or predetermined time). The examples provided herein concern timed tasks that include timers, but it will be understood that the techniques described herein may also be applicable to other types of timed tasks.

[0208] For example, Figures 9A to 9F illustrate various techniques for managing timed tasks. Figure 9A shows the process by which user 802 can start (e.g., generate, create) a timed task (e.g., a timer). In Figure 9A, user 802 provides user input 804a to electronic device 810a, as indicated by arrow 910. In some examples, user input 804a is natural language voice input indicating a request directed to the digital assistant of electronic device 810a (e.g., "Hey assistant, start a 10-minute timer in the bedroom"). User input 804a may include, for example, a voice trigger to activate the digital assistant (e.g., "Hey assistant"). In some examples, user input 804a may include any request that can be directed to the digital assistant. For example, the user input 804a that states "Start a 10-minute timer in the bedroom" may request the digital assistant of the electronic device 810a to start a timer for a specific duration (e.g., 10 minutes) on the electronic device associated with the specified location (e.g., the bedroom).

[0209] In response to user input 804a, the electronic device 810a identifies an electronic device to perform a timed task. In some examples, the electronic device 804a identifies a device based on one or more parts of user input 804a. User input 804a may include one or more device keywords, such as a device name (e.g., "bedroom speaker"), a device location (e.g., "bedroom"), a device characteristic (e.g., device type), or any combination thereof. Based on the keywords, the electronic device 810a identifies an electronic device to perform the task. In some examples, the electronic device 810a queries one or more devices in the environment 800 to determine if any device in the environment 800 corresponds to a keyword, and if so, the corresponding device is selected to perform the task. In other embodiments, the electronic device 810a retrieves a set of devices in the environment 800 (e.g., a list) (e.g., from a home automation platform to which the devices in the environment 800 are associated and / or registered) and selects a device from the set corresponding to the keyword. In yet another embodiment, electronic device 810a requests that another device (e.g., another device in environment 800, a remote server) identify a device based on a keyword, and electronic device 810a selects the identified device. In the illustrated example, user 802 specifies "bedroom" as the location for a timed task, and electronic device 804a identifies a device 820a associated with the specified location (i.e., bedroom) to perform the timed task.

[0210] In some examples, timed tasks are user-specific. For example, a timed task may be associated with a specific user and / or user account. A timed task may be associated with a user, for example, when the timed task is started. In some examples, if a user requests that the device start a timed task, the electronic device can identify the user making the request and associate the timed task with the user. In some examples, the electronic device identifies the user using biometric information corresponding to the user (e.g., using voice recognition and / or facial recognition). In some examples, the electronic device used to request a timed task is associated with a specific user, and the electronic device identifies the user as the user associated with the device. In the illustrated embodiment, the electronic device 810a can identify user 802 as the one providing user input 804a and associate the started timed task (e.g., a 10-minute timer) with user 802.

[0211] Subsequently, electronic device 810a instructs electronic device 820a to execute a timed task (e.g., a 10-minute timer), as indicated by arrow 912, and electronic device 820a starts the timed task accordingly. Optionally, as indicated by arrows 916 and 918, electronic device 820a provides a response to electronic device 810a indicating that the timed task has been started by electronic device 820a, and electronic device 810a provides output 806a ("Start 10-minute timer in bedroom") signaling that the timed task has been started as requested.

[0212] Figure 9B illustrates the process by which user 802 can modify a timed task. In Figure 9B, user 802 provides user input 804b to electronic device 810a, as indicated by arrow 920. In some examples, user input 804b is natural language voice input indicating a request directed to the digital assistant of electronic device 810b (e.g., "Hey assistant, pause my 10-minute timer"). User input 804b may also include, for example, a voice trigger to activate the digital assistant (e.g., "Hey assistant"). In some examples, user input 804b may include any request that can be directed to the digital assistant. For example, the portion of user input 804b stating "pause my 10-minute timer" may request that the digital assistant of electronic device 810a pause an ongoing timed task.

[0213] In response to user input 804b, electronic device 810a identifies the electronic device associated with the timed task specified by user 802. In some examples, electronic device 810a identifies the electronic device associated with the timed task based on user input 804b. User input 804b may include one or more keywords that specify characteristics of the timed task, such as task duration (e.g., 10 minutes). Based on the keywords, electronic device 810a identifies the electronic device that will perform the timed task. In some examples, electronic device 810a queries one or more devices in environment 800 to determine whether any of the devices in environment 800 is performing the task corresponding to the specified keywords. In other examples, electronic device 810a requests another device (e.g., another device in environment 800, a remote server) to identify the device that will perform the task, and electronic device 810a selects the identified device. Additionally or alternatively, user input 804b may include a device name, location, and / or device characteristics, and electronic device 810a selects the specified device as described. In the illustrated example, user 802 specifies "10 minutes" as the characteristic of the timed task, and electronic device 810a identifies device 820a as the device that will perform the specified timed task.

[0214] In some examples, electronic device 810a identifies electronic devices based on user-specific characteristics of the timed task. For example, in response to user input 804b, electronic device 810a may identify user 802 as the user providing the request. Electronic device 810a can then determine whether the ongoing timed task (and by extension, which electronic device is performing the timed task) is associated with user 802. If so, the electronic device can identify the electronic device performing the timed task referenced by user 802 in user input 804b.

[0215] In some examples, user-specific characteristics are used in combination with user-specified keywords to identify electronic devices. For example, electronic device 810a may use keywords provided by the user to identify a time-limited task from any number of time-limited tasks associated with the user. Electronic device 810a can then identify the device as a device that performs the identified time-limited task. In this way, electronic device 810a can optionally select only from time-limited tasks associated with user 802, so that time-limited tasks not associated with user 802 are not considered. In the illustrated example, the time-limited task referenced by user 802 (e.g., a 10-minute timer) is associated with user 802 (recall that user 802 requested to start a 10-minute timer in the bedroom), and electronic device 810a identifies device 820a as a device that performs the specified time-limited task.

[0216] Subsequently, electronic device 810a instructs electronic device 820a to modify the timed task (e.g., pause the timer) as indicated by arrow 922, and electronic device 820a modifies the timed task accordingly. Optionally, as indicated by arrows 926 and 928, respectively, electronic device 820a provides electronic device 820a with a response indicating that the timed task has been modified by electronic device 810a, and electronic device 810a then provides output 806b ("Your 10-minute timer has been paused") signaling that the timed task has been modified as requested.

[0217] In the illustrated example, user 802 modifies the timed task by pausing it, but it will be understood that the timed task may be modified in other ways. For example, modifying a timed task may include resetting the timed task (e.g., resuming the timed task). Another example is modifying a timed task, which may include resuming the timed task (e.g., if it was previously paused). Yet another example is modifying a timed task, which may include adding time to the timed task (e.g., increasing the remaining time on the timer by a specified amount) or removing time from the timed task (e.g., decreasing the remaining time on the timer by a specified amount).

[0218] Figure 9C illustrates the process by which user 802 can retrieve the status of a timed task. In Figure 9C, user 802 provides user input 804c to electronic device 810a, as indicated by arrow 930. In some examples, user input 804c is a natural language voice input indicating a request directed to the digital assistant in electronic device 810b (e.g., "Hey assistant, get the status of my 10-minute timer"). User input 804c may also include, for example, a voice trigger to activate the digital assistant (e.g., "Hey assistant"). In some examples, user input 804c may include any request that can be directed to the digital assistant. For example, the portion of user input 804c stating "Get the status of my 10-minute timer" may request that the digital assistant in electronic device 810a provide information about the status of an ongoing timed task.

[0219] In response to user input 804c, electronic device 810a identifies the electronic device associated with the timed task specified by user 802. In some examples, the user specifies the timed task and / or device (e.g., by name, location, and / or characteristics), and electronic device 810a identifies the electronic device currently executing the timed task, as described. In the illustrated example, user 802 specifies "10 minutes" as the characteristic of the timed task, and electronic device 810a identifies device 820a as the device that will perform the specified task.

[0220] As explained, in some examples, the electronic device 810a may identify only the timed task associated with the user making the request (and identify the electronic device that will perform the identified timed task). In this example, user 802 is associated with the timed task (e.g., a 10-minute timer) referenced in user input 804c, and therefore the electronic device 810a may identify the timed task referenced by user 802.

[0221] Subsequently, electronic device 810a requests electronic device 820a to provide a state for the timed task (e.g., pause the timer), as indicated by arrow 932, and accordingly, electronic device 820a provides a state for the timed task, as indicated by arrow 936. The state thus provided may include, but is not limited to, any information relating to the timed task, including, the remaining duration of the timed task, the time the timed task is expected to be completed, the location and / or name of the device running the timed task, the time the timed task was started, or any combination thereof. Upon receiving the state for the timed task from electronic device 820a, electronic device 810a provides output 806c, as indicated by arrow 938, which includes the received state ("Your 10-minute timer has 8 minutes remaining").

[0222] Figure 9D illustrates a process by which user 802 can cancel and / or terminate a timed task. In Figure 9D, user 802 provides user input 804d to electronic device 810a, as indicated by arrow 940. In some examples, user input 804d is a natural language voice input indicating a request directed to the digital assistant of electronic device 810b (e.g., "Hey assistant, cancel my 10-minute timer"). User input 804d may include, for example, a voice trigger to activate the digital assistant (e.g., "Hey assistant"). In some examples, user input 804d may include any request that can be directed to the digital assistant. For example, the portion of user input 804d stating "cancel my 10-minute timer" may request the digital assistant of electronic device 810a to cancel an ongoing timed task.

[0223] In response to user input 804d, electronic device 810a identifies the electronic device associated with the timed task specified by user 802. In some examples, the user specifies the timed task and / or device (e.g., by name, location, and / or characteristics), and electronic device 810a identifies the electronic device currently performing the timed task, as described. In the illustrated example, user 802 specifies "10 minutes" as the characteristic of the timed task, and electronic device 810a identifies device 820a as the device performing the specified task.

[0224] As explained, in some examples, the electronic device 810a may identify only the timed task associated with the user making the request (and identify the electronic device that will perform the identified timed task). In this example, user 802 is associated with the timed task (e.g., a 10-minute timer) referenced in user input 804d, and therefore the electronic device 810a may identify the timed task referenced by user 802.

[0225] Subsequently, electronic device 810a instructs electronic device 820a to cancel the timed task, as indicated by arrow 942, and electronic device 820a cancels the timed task accordingly. Optionally, electronic device 820a provides electronic device 820a with a response indicating that the timed task has been canceled by electronic device 810a, as indicated by arrows 936 and 938, respectively, and then electronic device 810a provides output 806d ("Your 10-minute timer has been canceled") signaling that the timed task has been canceled as requested.

[0226] As described, devices in environment 800 can be associated with a device ecosystem. In some examples, a device ecosystem may include, for example, a coordinating device (or set of devices) configured to manage the operation of devices within the device ecosystem. Thus, in some examples, timed tasks may be managed by a coordinating device.

[0227] Figure 9E shows an example where a user requests the status of an ongoing timed task while electronic device 830a is acting as a coordinating device for a device in environment 800. In Figure 9E, user 802 provides user input 804e to electronic device 810a, as indicated by arrow 950. In some examples, user input 804e is a natural language voice input indicating a request directed to the digital assistant of electronic device 810b (e.g., "Hey assistant, get the status of the 10-minute timer"). User input 804e may include, for example, a voice trigger to activate the digital assistant (e.g., "Hey assistant"). In some examples, user input 804e may include any request that can be directed to the digital assistant. For example, the portion of user input 804e stating "get the status of the 10-minute timer" may request that the digital assistant of electronic device 810a provide information regarding the status of an ongoing timed task.

[0228] In response to user input 804e, electronic device 810a requests the timer status from electronic device 830a, as indicated by arrow 952 (recall that device 830a is acting as a regulating device). In some examples, electronic device 810a identifies the device performing the timed task and further indicates to electronic device 830a which device is performing the timed task. In other embodiments, electronic device 810a does not identify which device is performing the timed task, but instead transfers one or more parts (or related information) of user input 804e to electronic device 830a (e.g., "10 minutes" as a characteristic of the timed task), thereby allowing electronic device 830a to identify the device performing the timed task. In the illustrated example, device 820a is the device performing the specified task, and electronic device 830a requests the timed task status from device 820a.

[0229] In some examples, devices (e.g., electronic device 810a, electronic device 830a) may identify only the timed tasks associated with the user making the request (and identify the electronic device that performs the identified timed tasks). In this example, user 802 is associated with the timed task referenced in user input 804d (e.g., a 10-minute timer), and therefore either electronic device 810a or 830a may identify the timed task referenced by user 802.

[0230] Upon receiving a request from device 810a, device 830a requests the state of the timed task from device 820a, as indicated by arrow 954, and accordingly, electronic device 820a provides the state of the timed task, as indicated by arrow 956. Next, electronic device 830a provides the state to electronic device 810a, and electronic device 810a provides output 806e containing the received state ("Your 10-minute timer has 8 minutes remaining"), as indicated by arrows 958 and 959, respectively.

[0231] In some embodiments, electronic devices cannot readily identify themselves based on user input. Therefore, in some examples, the electronic device removes ambiguity from the user input to identify the device. In some examples, disambiguation involves querying the user for additional information (e.g., "Which device in the bedroom did you mean?", "Which timer in the bedroom did you refer to?"). In other examples, disambiguation involves applying one or more prioritization rules to select the "most suitable" device to perform the requested task. In some examples, the prioritization rules specify a hierarchy of device types for performing timed tasks (e.g., a smart speaker selected before a set-top box). In some examples, the prioritization rules specify the selection based on the device's position within a location (e.g., a device located in the center of the location is selected before a device located at the edge or boundary of the location).

[0232] In some examples, a user may want to manage multiple timed tasks simultaneously. The user may start, modify, and / or cancel multiple timed tasks based on a single user input (e.g., "cancel all timers"). In the embodiment shown in Figure 9F, electronic device 820a is running a first timed task (e.g., a 10-minute timer), and electronic device 810b is running a second timed task (e.g., a 15-minute timer). While devices 810b and 820a are running their respective timed tasks, user 802 provides user input 804f to electronic device 810a, as indicated by arrow 960. In some examples, user input 804f is a natural language voice input indicating a request directed to the digital assistant of electronic device 810b (e.g., "Hey Assistant, how much time is left on all the timers?"). User input 804f may include, for example, a voice trigger to activate the digital assistant (e.g., "Hey Assistant"). In some examples, user input 804f may include any request that can be directed to the digital assistant. For example, a portion of user input 804f stating "How much time is left on all the timers?" may request that the digital assistant of the electronic device 810a provide information about the status of all ongoing timed tasks of a specified type, for example.

[0233] In response to user input 804f, electronic device 810a identifies the electronic device currently performing the timed task. In some embodiments, electronic device 810a queries one or more devices in environment 800 to determine whether any of the devices in environment 800 is performing the timed task. In other embodiments, electronic device 810a requests another device (e.g., a coordination device) to identify the device currently performing the timed task.

[0234] In some examples, the electronic device 810a identifies a timed task associated with the user making the request (and identifies the electronic device that performs the identified timed task). In the illustrated example, if user 802 is associated with a first timed task (e.g., a 10-minute timer) but not with a second timed task (e.g., a 15-minute timer), the electronic device 810a may identify only the device performing the first timed task and not the device performing the second timed task. In the illustrated example, if user 802 is associated with both the first timed task (e.g., a 10-minute timer) and the second timed task (e.g., a 15-minute timer), the electronic device 810a may identify both the device performing the first timed task and the device performing the second timed task.

[0235] Subsequently, electronic device 810a requests that the electronic devices performing the timed tasks provide the state of their respective timed tasks. In the illustrated example, electronic devices 820a and 810b are performing timed tasks, and electronic device 810 requests that each of devices 820a and 810b provide a separate state, as indicated by arrows 962 and 964. Accordingly, electronic devices 820a and 810b provide the state of their respective timed tasks, as indicated by arrows 966 and 968. The state thus provided may include, but is not limited to, any information relating to the timed task, including the remaining duration of the timed task, the time the timed task is expected to be completed, the location and / or name of the device performing the timed task, the time the timed task was started, or any combination thereof. Upon receiving the state of the timed tasks, electronic device 810a provides output 806f, as indicated by arrow 970, which includes the received state ("Your 10-minute timer has 8 minutes remaining, and your 15-minute timer has 4 minutes remaining.").

[0236] While the electronic device 810f is described as referring to timed tasks by their characteristics (e.g., "10-minute timer," "15-minute timer"), it will be understood that timed tasks may be referred to using any distinguishing syntax so as to sufficiently remove ambiguity between timers and other timed tasks. For example, timed tasks may be referred to by timer characteristics, device name, device location, and / or any combination of device characteristics. For example, if device 810b performs the same timed task as device 820a (e.g., 10-minute timer), electronic device 810a may distinguish the timers of each device based on the device location (e.g., bedroom and living room) and / or other distinguishing aspects of each device (e.g., device type).

[0237] Figure 10 shows a process 1000 for managing timed tasks using a digital assistant, according to various embodiments. In some embodiments, the process 1000 is executed using a computer system (e.g., a personal electronic device (e.g., a mobile device (e.g., iPhone)), a tablet computer (e.g., iPad), a smartwatch (e.g., Apple Watch), a desktop (e.g., iMac), or a laptop (e.g., MacBook), or a shared electronic device (e.g., a smart TV (e.g., Apple TV) or a smart speaker (e.g., HomePod)) that implements a digital assistant and communicates with one or more input devices (e.g., a touch-sensitive surface (e.g., a touch-sensitive display), a mouse, a keyboard, a remote control, an audio input device, and / or one or more mechanical input devices (e.g., a pressable input mechanism, a button, a rotatable input mechanism, a crown, and / or a dial)). In some embodiments, the process 1000 is stored in a non-temporary (or temporary) computer-readable storage medium and managed by instructions executed by one or more processors of the computer system, such as one or more processors 220 of device 200 (Figure 2A).

[0238] In some embodiments, process 1000 is executed using a client-server system (e.g., system 100), and blocks of process 1000 are divided in any way between a server (e.g., DA server 106) and a client (e.g., user device 104). In other embodiments, blocks of process 1000 are divided between a server and multiple client devices (e.g., mobile phones and smartwatches). Thus, although parts of process 1000 are described herein as being executed by specific devices in a client-server system, it will be understood that process 1000 is not so limited. In other embodiments, process 1000 is carried out using only one client or only multiple client devices. In process 1000, some blocks are optionally combined, the order of some blocks is optionally changed, and some blocks are optionally omitted. In some embodiments, additional steps may be carried out in combination with process 1000.

[0239] The computer system receives natural language voice input (e.g., 804c, 804e, 804f) containing requests about the status of timed tasks via one or more input devices (e.g., as indicated by arrows 930, 950, 960) (1005). In some examples, the computer system is included in a device ecosystem (e.g., a home automation ecosystem), in which the computer system and other devices in the device ecosystem are configured to communicate with each other. In some examples, devices in a device ecosystem may be used to communicate with and / or control other devices within the device ecosystem. For example, a device may be used to start, modify, and / or cancel various timed tasks such as timers and alarms. A user may provide input (e.g., natural language voice input) requesting that a timed task be started, modified, or canceled (e.g., "Hey Assistant, start a 10-minute timer in the kitchen"). In some examples, a user may identify a specific device or location for a timed task, and the computer system (and / or one or more other devices in the device ecosystem) may identify a device in the device ecosystem that corresponds to the device and / or location specified by the user. The timed task can then be started, modified, and / or canceled on the identified device. In some examples, a device may be queried for the status of one or more timed tasks. For example, a user may provide input requesting the status of a timed task, and based on the input, the computer system (and / or one or more other devices in the device ecosystem) may identify the timed task corresponding to the request and initiate a process to retrieve the status of the timed task.

[0240] The computer system identifies an electronic device (e.g., 820a) associated with a timed task based on natural language speech input (1010). In some examples, when performing a process to retrieve the state of a timed task, the computer system identifies the device that will perform the timed task for the request based on one or more user-specified parameters provided in the input, including but not limited to the device name (e.g., How much time is left on the timer for speaker 1?), the device location (e.g., "How much time is left on the timer in the kitchen?"), and / or timer task characteristics (e.g., How much time is left on the 10-minute timer?).

[0241] The computer system requests the status of a timed task associated with an electronic device (for example, as indicated by arrows 932, 952, 962, and 964) (1015). In some examples, once the computer system identifies a device in the device ecosystem that performs the timed task corresponding to the received request, the computer system requests the status of the timed task from the identified electronic device or from another device that can indicate the status of the timed task (for example, a device that coordinates the operation of one or more other devices, including the identified electronic device and / or the computer system).

[0242] The computer system receives the status of the timed task (for example, as indicated by arrows 936, 958, 966, and 968) (1020). In some examples, the identified electronic device returns the status of the timed task to the computer system.

[0243] The computer system provides outputs (e.g., 806c, 806e, 806f) (e.g., audio output, visual output) indicating the status of a timed task (e.g., "Does the timed task have 3 minutes remaining?") (1025). (e.g., as indicated by arrows 938, 959, 970)

[0244] In some embodiments, the timed task is performed on an electronic device. In some embodiments, a computer system identifies the electronic device associated with the timed task. In some examples, the timed task is performed by the identified electronic device (for example, on the identified electronic device), and in response to a request for the state of the timed task, the identified electronic device determines the state of the timed task and provides the determined state to the computer system.

[0245] In some examples, requesting the status of a timed task involves having an electronic device retrieve the status of the timed task from a second electronic device different from the electronic device itself (for example, as indicated by arrow 954). In some examples, the timed task is performed by a second electronic device other than the identified electronic device, and in response to a request for the status of the timed task, the identified electronic device requests the status of the timed task from the second electronic device. Upon receipt, the identified electronic device provides the status to the computer system. In some examples, the identified electronic device is configured to coordinate the operation of a device ecosystem (e.g., a home automation network, multiple communicably coupled devices) including a computer system, and in response to a request for the status of a timed task, the identified electronic device determines which electronic device in the device ecosystem is performing the timed task. The identified electronic device then retrieves the status of the timed task from the device performing the timed task and provides that status to the computer system.

[0246] In some examples, requesting the status of a timed task involves providing an identifier for the timed task (e.g., “10-minute timer” as contained in inputs 804c, 804e). In some examples, when requesting the status of a timed task, the user may identify the timed task, for example, to allow the computer system to eliminate ambiguity between multiple timed tasks. In some examples, the user may specify a location, a device name (the device on which the timed task is running), characteristics of the timed task, or any combination thereof. In some examples, the location is, for example, an area to which the device is assigned according to the device ecosystem configuration (e.g., a room in a house). In some examples, the device name is, for example, a user-specified name for the device assigned according to the device ecosystem configuration. In some examples, timer characteristics are any parameters, values, and / or settings that can identify the timed task. For example, a timer may be identified by its duration (e.g., “10-minute timer”), start time (e.g., “Timer started at 6:00 PM”), end time (e.g., “7:00 AM alarm”), recency (“Cancel recency alarm”), or any combination thereof.

[0247] In some embodiments, the identifier indicates the location of an electronic device.

[0248] In some embodiments, the identifier indicates the name of the electronic device.

[0249] In some examples, the identifier specifies the characteristics of a time-limited task.

[0250] In some examples, before receiving natural language voice input containing a request about the state of a timed task, the computer system receives a request to start a timed task on an electronic device (e.g., input 804a) via one or more input devices. In some examples, before receiving a request about the state of a timed task, the computer system receives a request to start a timed task on an electronic device. In some examples, the request is natural language voice input provided by the user ("Hey Assistant, start a 10-minute timer"). In some examples, the request is provided by another device.

[0251] In some examples, a request to start a timed task on an electronic device specifies the location of the electronic device (e.g., "bedroom"). In some examples, a request to start a timed task can specify one or more aspects of the timed task. In some examples, the user can specify the location, the device name (the device on which the timed task will run), the characteristics of the timed task, or any combination thereof. In some examples, the location is, for example, the area to which the device is assigned according to the device ecosystem configuration (e.g., a room in the house). In some examples, the device name is, for example, the user-specified name of the device assigned according to the device ecosystem configuration. In some examples, the timer characteristics are any parameters, values, and / or settings on which the timed task will run. As an example, the user may specify the duration (e.g., "10-minute timer"), start time (e.g., "timer started at 6:00 p.m."), and end time (e.g., "7:00 a.m. alarm") for the timed task. In some examples, the user may specify that the timed task adopts one or more characteristics of the previous timed task (e.g., "repeat the last timer").

[0252] In some examples, each of a plurality of electronic devices corresponds to the location of the electronic device. In some embodiments, the computer system requires that an electronic device initiate a time-limited task. In some examples, requiring an electronic device to initiate a time-limited task includes identifying the device type of each of the plurality of electronic devices and selecting an electronic device among the plurality of electronic devices based on the identified device type. In some examples, the computer system identifies a device (e.g., a device associated with a time-limited task) based on the type of the device. In some examples, the device type is used to remove ambiguity between devices, such as devices corresponding to the same location (e.g., a living room). For example, when the computer system identifies a first device that is a speaker and a second device that is a set-top box, the computer system can select the first device to perform the time-limited task. In some examples, when the computer system selects a device to perform a time-limited task, the computer system causes the selected device to perform the time-limited task by providing an instruction to the selected device. In some examples, the computer system causes the selected device to perform the time-limited task by providing an instruction to another device (e.g., a device coordinator) that instructs the selected device to perform the time-limited task.

[0253] In some embodiments, the requirement to initiate a time-limited task on an electronic device indicates the name of the electronic device.

[0254] In some examples, the status of a time-limited task indicates the remaining time of the time-limited task. In some examples, the status of the time-limited task indicates the remaining duration of the time-limited task (e.g., "Your timer has 3 minutes left", "Your alarm is set to go off in 25 minutes").

[0255] In some examples, after providing an output indicating the state of a time-limited task, the computer system receives a request to cancel the time-limited task via one or more input devices. In some examples, a user can cancel a time-limited task by providing, for example, a request to cancel the time-limited task (e.g., "Hey Assistant, cancel my 10-minute timer") to the computer system.

[0256] In some examples, the computer system receives a second natural language voice input via one or more input devices, the second natural language voice input including a request regarding the state of a second time-limited task being executed on the computer system. In some examples, the computer system determines the state of the second time-limited task. In some examples, the computer system provides an output (e.g., an audio output, a visual output) indicating the state of the second time-limited task. In some examples, the computer system receives a request regarding the state of a second time-limited task being executed by the computer system. In response, the computer system can optionally provide the state of the second time-limited task without querying any device.

[0257] The operations described above with reference to FIG. 10 can optionally be implemented by the components shown in FIGS. 1-4, FIGS. 6A-6B, and FIGS. 7A-7C, FIG. 8, and FIGS. 9A-9F. For example, the operations of process 1000 can be performed by the devices of environment 800 as described. Based on the components shown in FIGS. 1-4, FIGS. 6A-6B, and FIGS. 7A-7C, FIG. 8, and FIGS. 9A-9F, how other processes are implemented will be apparent to those skilled in the art.

[0258] As described, in some examples, a timed task (e.g., an alarm) is user-specific, so that the timed task is associated with a specific user and / or user account. A timed task may be associated with a user who is identified when the timed task is started. For example, a user may request that a timed task be started (e.g., "Set an alarm for 3 p.m."), and accordingly, an electronic device (or another electronic device) receiving the request may identify the user making the request and associate the identified user with the started timed task. In some examples, associating a user with a timed task in this way involves indicating to one or more other devices (e.g., a coordination device) that the user is associated with the timed task.

[0259] In some examples, electronic devices identify users using biometric information corresponding to the user (e.g., using voice recognition and / or facial recognition). In some examples, electronic devices used to request timed tasks are associated with a specific user, and the electronic device identifies the user as the user associated with the device.

[0260] As explained, in some examples, the way timed tasks are managed is user-specific. For example, a timed task identified in a request may be managed differently based on whether the timed task is associated with the same user making the request. Consider an example where two timed tasks are started: a 3 PM alarm associated with a first user and a 2 PM alarm associated with a second user. The first user may request the status of both the first timed task associated with them (e.g., the 3 PM alarm) and the second timed task not associated with them (e.g., the 2 PM alarm) (e.g., "Hey Assistant, what are the statuses of the 3 PM and 2 PM alarms?"). In at least one example, the electronic device may provide the status only for the timed task associated with the user (e.g., "You have 1 hour and 22 minutes until your 3 PM alarm triggers") and optionally indicate to the user that they are not permitted to receive the status of the second timer (e.g., "The 2 PM alarm is associated with another user"). In other embodiments, the electronic device prioritizes the state of timed tasks associated with a user (e.g., providing it first), and then provides the state of timed tasks not associated with a user (e.g., "You have 1 hour and 22 minutes until the 3 p.m. alarm is triggered, and 22 minutes until the 2 p.m. alarm is triggered"). In some examples, if a timed task does not correspond to a user providing a request, the electronic device identifies the user to whom the timed task corresponds (e.g., "John has 1 hour and 22 minutes until his alarm is triggered").

[0261] As another example, a user might request that an electronic device cancel a timed task. In some cases, if the cancellation requested by the user is associated with a timed task, the electronic device will cancel the timed task as requested. In some cases, if the user requesting the cancellation is not associated with a timed task, the electronic device will not cancel the timed task and will optionally indicate that the user is not authorized to cancel the timed task (e.g., "You cannot cancel another user's alarm"). In some cases, if the user requesting the cancellation is not associated with a timed task, the electronic device will prompt the user to confirm that they intend to cancel the timed task for another user (e.g., "Do you really want to cancel John's 2pm alarm?"). The user can then confirm the cancellation and allow the electronic device to cancel the timed task.

[0262] In some examples, a user can only manage associated timed tasks. For example, a user might request that the electronic device provide the status of all ongoing timed tasks ("Hey Assistant, list all alarms."). In response, the electronic device would provide the status of all timed tasks associated with the user (e.g., "You currently have one 10-minute timer with 8 minutes remaining and a 3pm alarm that will trigger in 1 hour and 221 minutes"), and refrain from providing the status of all timed tasks not associated with the user (e.g., those associated with other users or not associated with the user). In another example, a user might request that the electronic device cancel all ongoing timed tasks (e.g., "Cancel all timers and alarms"). In response, the electronic device would cancel all of the user's timed tasks, but refrain from canceling timed tasks not associated with the user.

[0263] Figure 11 is a flowchart of process 1100 for managing user-specific time-limited tasks in various embodiments. In some embodiments, process 1100 is executed using a computer system (e.g., a personal electronic device (e.g., a mobile device (e.g., iPhone)), a tablet computer (e.g., iPad), a smartwatch (e.g., Apple Watch), a desktop (e.g., iMac), or a laptop (e.g., MacBook), or a shared electronic device (e.g., a smart TV (e.g., Apple TV) or a smart speaker (e.g., HomePod)) that implements a digital assistant and communicates with one or more input devices (e.g., a touch-sensitive surface (e.g., a touch-sensitive display), a mouse, a keyboard, a remote control, an audio input device, and / or one or more mechanical input devices (e.g., a pressable input mechanism, a button, a rotatable input mechanism, a crown, and / or a dial)). In some embodiments, process 1100 is stored in a non-temporary (or temporary) computer-readable storage medium and managed by instructions executed by one or more processors of the computer system, such as one or more processors 220 of device 200 (Figure 2A).

[0264] In some embodiments, process 1100 is executed using a client-server system (e.g., system 100), and blocks of process 1100 are divided in any way between a server (e.g., DA server 106) and a client (e.g., user device 104). In other embodiments, blocks of process 1100 are divided between a server and multiple client devices (e.g., mobile phones and smartwatches). Thus, although parts of process 1100 are described herein as being executed by specific devices in a client-server system, it will be understood that process 1100 is not so limited. In other embodiments, process 1100 is carried out using only one client or only multiple client devices. In process 1100, some blocks are optionally combined, the order of some blocks is optionally changed, and some blocks are optionally omitted. In some embodiments, additional steps may be carried out in combination with process 1100.

[0265] In some examples, a computer system receives requests for a set of states (e.g., one or more) corresponding to a separate set of timed tasks via one or more input devices (1105). In some examples, the computer system is part of a device ecosystem (e.g., a home automation ecosystem), in which the computer system and other devices in the device ecosystem are configured to communicate with each other. In some examples, devices in a device ecosystem may be used to communicate with and / or control other devices within the device ecosystem. For example, a device may be used to start, modify, and / or cancel various timed tasks such as timers and alarms. A user may provide input (e.g., natural language voice input) requesting that one or more timed tasks be started, modified, or canceled ("Hey Assistant, get the states of the bedroom alarm and office timer"). In some examples, a user may identify a specific device or location for a timed task, and the computer system (and / or one or more other devices in the device ecosystem) may identify a device in the device ecosystem corresponding to the device and / or location specified by the user. One or more timed tasks can then be started, modified, and / or canceled on the identified device. In some examples, a device may be queried for the state of one or more timed tasks. A user may, for example, provide input requesting the state of one or more timed tasks, and based on that input, a computer system (and / or one or more other devices in the device ecosystem) may identify the one or more timed tasks corresponding to the request and initiate a process to retrieve the state of the timed task(s).

[0266] In some examples, a computer system identifies the user responding to a request (1110). The computer system may, for example, use biometric information corresponding to the user to determine the user's identity. In some examples, the biometric information is voice information, and the electronic device uses voice recognition to recognize the user. In some examples, the biometric information is facial information, and the electronic device uses facial recognition to identify the user.

[0267] In some examples, the computer system receives a set of states (1115). In some examples, after requesting the state of each timed task in a set of timed tasks, the computer system receives the individual states of each timed task in the set of timed tasks. The states may be received from any number of devices performing timed tasks and / or from a device acting as a coordinating device for a home automation system.

[0268] In some examples, according to the determination that a first timed task in a set of timed tasks is associated with an identified user, the computer system provides at least a subset of the set of states in a first order (1120). In some examples, the computer system determines whether a user is associated with any ongoing timed task and, if so, provides the set of states in a prioritized or filtered manner. In some examples, providing the set of states in a prioritized manner includes first providing the states of the timed task associated with the user, and then providing the states of tasks not associated with the user. In some examples, providing the set of states in a filtered manner includes providing the states of the timed task associated with the user and omitting the states of one or more timed tasks not associated with the user. In some examples, the user is not permitted to receive the states of timed tasks not associated with the user. Thus, in some examples, an electronic device may indicate when the user has requested a state that the user is not permitted to receive.

[0269] In some examples, following a determination that the first timed task in a set of timed tasks is not associated with an identified user, the computer system provides at least a subset of the set of states in a second order different from the first order (1125). In some examples, the computer system provides at least a subset of individual sets of states in a second order different from the first order. In some examples, if the computer system determines that the user is not associated with an ongoing timed task, the computer system provides a set of states in an unprioritized and / or unfiltered manner.

[0270] In some examples, identifying the user responding to a request involves receiving biometric information associated with the user and identifying the user based on that biometric information.

[0271] In some examples, receiving a request about a set of states includes receiving natural language speech input containing the request, biometric information includes voice information associated with the user, and identifying a user based on biometric information includes identifying a user based on voice information.

[0272] In some examples, a user is a first user, and identifying a user involves determining a second user associated with the computer system and determining whether the first user matches the second user.

[0273] In some examples, each state in a subset of the set of states is associated with a time-limited task that is linked to a user.

[0274] In some examples, a subset of the set of states includes a second timed task, the second timed task being associated with a user, a first state associated with the second timed task, and a third timed task, the third timed task not being associated with a user, a second state associated with the third timed task. In some examples, providing at least a subset of the set of states in a first order includes providing a first state, and then providing a second state after providing the first state.

[0275] The operations described above, with reference to Figure 11, can be optionally implemented by the components shown in Figures 1-4, 6A-6B, 7A-7C, 8, and 9A-9F. For example, the operation of process 1100 can be carried out by the devices in environment 800, as described. How other processes can be implemented based on the components shown in Figures 1-4, 6A-6B, 7A-7C, 8, and 9A-9F will be obvious to those skilled in the art.

[0276] According to some embodiments, a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) is provided, which stores one or more programs to be executed by one or more processors of an electronic device, and the one or more programs include instructions for executing any of the methods or processes described herein.

[0277] According to some embodiments, an electronic device (e.g., a portable electronic device) is provided, which comprises means for executing any of the methods or processes described herein.

[0278] According to some embodiments, an electronic device (e.g., a portable electronic device) is provided, which comprises a processing unit configured to execute any of the methods or processes described herein.

[0279] According to some embodiments, an electronic device (e.g., a portable electronic device) is provided, which comprises one or more processors and a memory storing one or more programs for execution by the one or more processors, and the one or more programs include instructions for executing any of the methods or processes described herein.

[0280] The above has been described by reference to specific embodiments for purposes of illustration. However, the above exemplary considerations are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. Embodiments have been chosen and described in order to best explain the principles of the technology and their practical applications, thereby enabling others skilled in the art to best utilize the technology and various embodiments with various modifications suitable for the intended specific uses.

[0281] <着 While the present disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. Such changes and modifications should be understood to fall within the scope of the present disclosure and examples as defined by the claims.

[0282] As described above, one aspect of the technology involves collecting and using data available from various sources to improve the management of dimensional tasks. This disclosure suggests that in some cases, such collected data may include personal information data that uniquely identifies a particular person, or personal information data that can be used to contact a particular person or locate them. Such personal information data may include demographic data, location-based data, telephone numbers, email addresses, Twitter IDs, home addresses, data or records relating to a user's health or fitness level (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0283] This disclosure acknowledges that such use of personal data in the technology may be in the user's best interest. For example, personal data may be used to provide time-limited tasks that are of greater interest to the user (e.g., generated and / or modified according to the user's preferences and / or behavior). Thus, such use of personal data enables the user to have computational control over the content delivered. Furthermore, other uses of personal data that may be in the user's best interest are also conceivable in this disclosure. For example, health and fitness data may be used to provide insights into the user's overall wellness, or as positive feedback to individuals using the technology to pursue wellness goals.

[0284] This disclosure implies that entities involved in the collection, analysis, disclosure, transmission, storage, or other use of such personal data should adhere to a robust privacy policy and / or privacy practices. Specifically, such entities should implement and consistently use a privacy policy and practices that are generally recognized as meeting or exceeding industry or government requirements for the strict confidentiality of personal data. Such policies should be readily accessible to users and should be updated as data collection and / or use changes. Personal data from users should be collected for the lawful and legitimate use of the entity and should not be shared or sold for any other purpose. Furthermore, such collection / sharing should be carried out only after informing and obtaining the user's consent. In addition, such entities should consider taking all necessary steps to protect and secure access to such personal data and to ensure that others with access to the personal data faithfully adhere to those privacy policies and procedures. Furthermore, such entities may undergo third-party evaluations to demonstrate their compliance with widely accepted privacy policies and practices. Furthermore, policies and practices should be adapted to the specific types of personal data collected and / or accessed, and should comply with applicable laws and standards, including jurisdiction-specific considerations. For example, in the United States, the collection or access to certain health data may be subject to federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA). Health data in other countries, on the other hand, may be subject to other regulations and policies and should be addressed accordingly. Therefore, different privacy practices should be maintained in each country with respect to different types of personal data.

[0285] Notwithstanding the foregoing, the Disclosure also envisions embodiments that allow a user to selectively prevent the use of or access to personal data. That is, the Disclosure envisions that hardware and / or software elements may be provided to prevent or prevent access to such personal data. For example, in the case of managing dimensional tasks, the Technology could be configured so that a user can choose to “opt in” or “opt out” of participating in the collection of personal data during or at any time thereafter when registering for the service. In addition to providing “opt-in” and “opt-out” options, the Disclosure envisions providing notices regarding access to or use of personal data. For example, a user may be notified when downloading an app that will access their personal data, and then again immediately before the app accesses their personal data.

[0286] Furthermore, the intent of this disclosure is that personal data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data when it is no longer needed. In addition, where applicable in certain health-related applications, data anonymization can be used to protect user privacy. Anonymization can be facilitated by removing certain identifiers (e.g., date of birth), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data across users), and / or by other means, where necessary.

[0287] Therefore, while this disclosure broadly covers the use of personal data to implement one or more different disclosed embodiments, it is also conceivable that these different embodiments could be implemented without requiring access to such personal data. In other words, the different embodiments of the technology would not be rendered inoperable by the absence of all or part of such personal data. For example, timed tasks could be proposed and / or implemented based on non-personal data such as content requested by a device associated with the user, or a minimum amount of personal information, other non-personal information available on the user device, or publicly available information. [Item 1] It is a method, In a computer system communicating with one or more input devices, Receiving natural language speech input via one or more input devices, including a request about the status of a timed task, Based on the aforementioned natural language speech input, the electronic device associated with the time-limited task is identified, Receiving the state of the aforementioned time-limited task, To provide an output indicating the state of the aforementioned time-limited task, Methods that include... [Item 2] The method according to item 1, wherein the timed task is performed on the electronic device. [Item 3] To request the aforementioned state of the time-limited task is, The method of item 2, comprising causing the electronic device to retrieve the state of the timed task from a second electronic device different from the electronic device. [Item 4] The method according to any one of items 1 to 3, wherein requesting the state of the timed task includes providing an identifier for the timed task. [Item 5] The identifier is the method described in item 4, which indicates the location of the electronic device. [Item 6] The identifier is the method described in item 4, which indicates the name of the electronic device. [Item 7] The identifier is the method described in item 4, which specifies the characteristics of the time-limited task. [Item 8] Before receiving the natural language speech input which includes a request about the status of a timed task, Receiving a request to start the timed task on the electronic device via one or more input devices, The method described in any one of items 1 through 7, further including the method described in any one of items 1 through 7. [Item 9] The method of item 8, wherein the request to initiate the timed task on the electronic device indicates the location of the electronic device. [Item 10] Each of the multiple electronic devices corresponds to the location of the electronic device, and the method is The electronic device requests the commencement of the timed task, Identifying the device type of each of the aforementioned plurality of electronic devices, The method of item 9, further comprising selecting an electronic device from among the plurality of electronic devices based on the identified device type. [Item 11] The request for initiating the timed task on the electronic device is the method described in any one of items 8 to 10, which indicates the name of the electronic device. [Item 12] The state of the timed task is the method described in any one of items 1 to 11, which indicates the remaining time for the timed task. [Item 13] After providing an output indicating the state of the timed task, the system receives a request to cancel the timed task via one or more input devices. The method described in any one of items 1 through 12, further including the method described in any one of items 1 through 12. [Item 14] Receiving a second natural language speech input via one or more input devices, which includes a request for the state of a second time-limited task being performed on the computer system; Determining the state of the second time-limited task, To provide an output indicating the state of the second time-limited task, The method described in any one of items 1 through 13, further including the method described in item 1 through 13. [Item 15] A computer system configured to communicate with one or more input devices, wherein the computer system is One or more processors, The system comprises a memory that stores one or more programs configured to be executed by one or more processors, and the one or more programs are The system receives natural language speech input via one or more input devices, including a request about the status of a timed task. Based on the natural language speech input, the electronic device associated with the timed task is identified. Upon receiving the state of the aforementioned time-limited task, A computer system including instructions that provides an output indicating the state of the aforementioned time-limited task. [Item 16] A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices, wherein the one or more programs are The system receives natural language speech input via one or more input devices, including a request about the status of a timed task. Based on the natural language speech input, the electronic device associated with the timed task is identified. Upon receiving the state of the aforementioned time-limited task, A non-temporary computer-readable storage medium including instructions, which provides an output indicating the state of the aforementioned time-limited task. [Item 17] A computer system configured to communicate with one or more input devices, wherein the computer system is Means for receiving natural language speech input, including a request about the status of a timed task, via one or more input devices, Means for identifying an electronic device associated with the timed task based on the natural language voice input, Means for receiving the state of the aforementioned time-limited task, Means for providing an output indicating the state of the aforementioned time-limited task, A computer system equipped with the following features. [Item 18] A computer system configured to communicate with one or more input devices, wherein the computer system is One or more processors, A computer system comprising: memory for storing one or more programs configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the method described in any one of items 1 to 14. [Item 19] A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices, wherein the one or more programs include instructions that perform the method described in any one of items 1 to 14. [Item 20] A computer system configured to communicate with one or more input devices, wherein the computer system is A computer system comprising means for performing the method described in any one of items 1 through 14. [Item 21] It is a method, In a computer system communicating with one or more input devices, Receiving requests for a set of states corresponding to a separate set of time-limited tasks via one or more input devices, Identifying the user corresponding to the aforementioned request, Receiving the set of the aforementioned states, To provide at least a subset of the set of states in a first order, based on the determination that a first time-limited task among the set of time-limited tasks is associated with the identified user, In accordance with the determination that the first time-limited task among the set of time-limited tasks is not associated with the identified user, the set of states is provided in a second order different from the first order, Methods that include... [Item 22] Identifying the user corresponding to the request is, Receiving biometric information associated with the user, The method according to item 21, comprising identifying the user based on the biometric information. [Item 23] Receiving the request for the set of states includes receiving natural language speech input containing the request, The biometric information includes voice information associated with the user, Identifying the user based on the biometric information includes identifying the user based on the voice information. The method described in item 22. [Item 24] The aforementioned user is the first user, and identifying the aforementioned user is To determine a second user associated with the aforementioned computer system, The method according to any one of items 21 to 23, including determining whether the first user matches the second user. [Item 25] Each state in the subset of the set of states is associated with a time-limited task associated with the user, as described in any one of items 21 to 24. [Item 26] The subset of the set of states is A second time-limited task, wherein the second time-limited task is associated with the user, and a first state associated with the second time-limited task, A third time-limited task, the third time-limited task being not associated with the user, and a second state associated with the third time-limited task, Providing at least a subset of the aforementioned set of states in a first order is: To provide the first state, Providing the first state, and then providing the second state, The method described in any one of items 21 to 24. [Item 27] A computer system configured to communicate with one or more input devices, wherein the computer system is One or more processors, The system comprises a memory that stores one or more programs configured to be executed by one or more processors, and the one or more programs are The system receives requests for a set of states corresponding to a separate set of time-limited tasks via one or more input devices. Identify the user corresponding to the aforementioned request, Receive the set of the aforementioned states, In accordance with the determination that a first time-limited task among the set of time-limited tasks is associated with the identified user, at least a subset of the set of states is provided in a first order: A computer system including instructions that, in accordance with the determination that the first time-limited task among the set of time-limited tasks is not associated with the identified user, provides the set of states, at least a subset of them, in a second order different from the first order. [Item 28] A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices, wherein the one or more programs are The system receives requests for a set of states corresponding to a separate set of time-limited tasks via one or more input devices. Identify the user corresponding to the aforementioned request, Receive the set of the aforementioned states, In accordance with the determination that a first time-limited task among the set of time-limited tasks is associated with the identified user, at least a subset of the set of states is provided in a first order: A non-temporary computer-readable storage medium including instructions, which provides the set of states in a second order different from the first order, according to the determination that the first time-limited task among the set of time-limited tasks is not associated with the identified user. [Item 29] A computer system configured to communicate with one or more input devices, wherein the computer system is Means for receiving requests for a set of states corresponding to a separate set of time-limited tasks via one or more input devices, Means for identifying the user corresponding to the aforementioned request, Means for receiving the aforementioned set of states, Means for providing at least a subset of the set of states in a first order, according to the determination that a first time-limited task among the set of time-limited tasks is associated with the identified user, Means for providing at least a subset of the set of states in a second order different from the first order, based on the determination that the first time-limited task among the set of time-limited tasks is not associated with the identified user, A computer system equipped with the following features. [Item 30] A computer system configured to communicate with one or more input devices, wherein the computer system is One or more processors, A computer system comprising: memory for storing one or more programs configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the method described in any one of items 21 to 26. [Item 31] A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices, wherein the one or more programs include instructions to perform the method described in any one of items 21 to 26. [Item 32] A computer system configured to communicate with one or more input devices, wherein the computer system is A computer system comprising means for performing the method described in any one of items 21 to 26. [Item 1] It is a method, In a computer system communicating with one or more input devices, Receiving natural language speech input via one or more input devices, including a request about the status of a timed task, Based on the natural language speech input, identify the timed task, which is performed on the electronic device, and identify the electronic device associated with the timed task. Receiving the state of the aforementioned time-limited task, To provide an output indicating the state of the aforementioned time-limited task, Methods that include... [Item 2] To request the aforementioned state of the time-limited task is, The method according to item 1, comprising causing the electronic device to retrieve the state of the timed task from a second electronic device different from the electronic device. [Item 3] The method according to either item 1 or 2, wherein requesting the state of the timed task includes providing an identifier for the timed task. [Item 4] The identifier is the method described in item 3, which indicates the location of the electronic device. [Item 5] The identifier is the method described in item 3, which indicates the name of the electronic device. [Item 6] The identifier is the method described in item 3, which specifies the characteristics of the time-limited task. [Item 7] Before receiving the natural language speech input which includes a request about the status of a timed task, The method according to any one of items 1 to 6, further comprising receiving a request to initiate the timed task on the electronic device via one or more input devices. [Item 8] The method of item 7, wherein the request to initiate the timed task on the electronic device indicates the location of the electronic device. [Item 9] Each of the multiple electronic devices corresponds to the location of the electronic device, and the method is The electronic device requests the commencement of the timed task, Identifying the device type of each of the aforementioned plurality of electronic devices, The method of item 8, further comprising selecting an electronic device from among the plurality of electronic devices based on the identified device type. [Item 10] The request for initiating the timed task on the electronic device is the method described in any one of items 7 to 9, which indicates the name of the electronic device. [Item 11] The state of the timed task is the method described in any one of items 1 to 10, which indicates the remaining time for the timed task. [Item 12] After providing an output indicating the state of the timed task, the system receives a request to cancel the timed task via one or more input devices. The method described in any one of items 1 through 11, further including the method described in any one of items 1 through 11. [Item 13] Receiving a second natural language speech input via one or more input devices, which includes a request for the state of a second time-limited task being performed on the computer system; Determining the state of the second time-limited task, To provide an output indicating the state of the second time-limited task, The method described in any one of items 1 through 12, further including the method described in any one of items 1 through 12. [Item 14] A computer system configured to communicate with one or more input devices, wherein the computer system is One or more processors, The system comprises a memory that stores one or more programs configured to be executed by one or more processors, and the one or more programs are Receiving natural language speech input via one or more input devices, including a request about the status of a timed task, Based on the natural language speech input, identify the timed task, which is performed on the electronic device, and identify the electronic device associated with the timed task. Receiving the state of the aforementioned time-limited task, A computer system including instructions that provides an output indicating the state of the aforementioned time-limited task. [Item 15] A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices, wherein the one or more programs are The system receives natural language speech input via one or more input devices, including a request about the status of a timed task. Based on the natural language speech input, identify the timed task, which is performed on the electronic device, and identify the electronic device associated with the timed task. Upon receiving the state of the aforementioned time-limited task, A non-temporary computer-readable storage medium including instructions, which provides an output indicating the state of the aforementioned time-limited task. [Item 16] A computer system configured to communicate with one or more input devices, wherein the computer system is Means for receiving natural language speech input, including a request about the status of a timed task, via one or more input devices, Based on the natural language speech input, means for identifying the electronic device associated with the timed task, wherein the timed task is performed on the electronic device, Means for receiving the state of the aforementioned time-limited task, Means for providing an output indicating the state of the aforementioned time-limited task, A computer system equipped with the following features. [Item 17] A computer system configured to communicate with one or more input devices, wherein the computer system is One or more processors, A memory that stores one or more programs configured to be executed by one or more processors, An electronic device comprising a computer system wherein one or more programs include instructions for performing the method described in any one of items 1 to 13. [Item 18] A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices, wherein the one or more programs include instructions that perform the method described in any one of items 1 to 13. [Item 19] A computer system configured to communicate with one or more input devices, wherein the computer system is A computer system comprising means for performing the method described in any one of items 1 through 13. [Item 20] It is a method, In a computer system communicating with one or more input devices, Receiving requests for a set of states corresponding to a separate set of time-limited tasks via one or more input devices, Identifying the user corresponding to the aforementioned request, Receiving the set of the aforementioned states, To provide at least a subset of the set of states in a first order, based on the determination that a first time-limited task among the set of time-limited tasks is associated with the identified user, In accordance with the determination that the first time-limited task among the set of time-limited tasks is not associated with the identified user, the set of states is provided in a second order different from the first order, Methods that include... [Item 21] Identifying the user corresponding to the request is, Receiving biometric information associated with the user, The method according to item 20, comprising identifying the user based on the biometric information. [Item 22] Receiving the request for the set of states includes receiving natural language speech input containing the request, The biometric information includes voice information associated with the user, Identifying the user based on the biometric information includes identifying the user based on the voice information. The method described in item 21. [Item 23] The aforementioned user is the first user, and identifying the aforementioned user is To determine a second user associated with the aforementioned computer system, The method according to any one of items 20 to 22, comprising determining whether the first user matches the second user. [Item 24] Each state in the subset of the set of states is associated with a time-limited task associated with the user, as described in any one of items 20 to 23. [Item 25] The subset of the set of states is A second time-limited task, wherein the second time-limited task is associated with the user, and a first state associated with the second time-limited task, A third time-limited task, the third time-limited task being not associated with the user, and a second state associated with the third time-limited task, Providing at least a subset of the aforementioned set of states in a first order is: To provide the first state, Providing the first state, and then providing the second state, The method described in any one of items 20 to 23. [Item 26] A computer system configured to communicate with one or more input devices, wherein the computer system is One or more processors, The system comprises a memory that stores one or more programs configured to be executed by one or more processors, and the one or more programs are The system receives requests for a set of states corresponding to a separate set of time-limited tasks via one or more input devices. Identify the user corresponding to the aforementioned request, Receive the set of the aforementioned states, In accordance with the determination that a first time-limited task among the set of time-limited tasks is associated with the identified user, at least a subset of the set of states is provided in a first order: A computer system including instructions that, in accordance with the determination that the first time-limited task among the set of time-limited tasks is not associated with the identified user, provides the set of states, at least a subset of them, in a second order different from the first order. [Item 27] A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices, wherein the one or more programs are The system receives requests for a set of states corresponding to a separate set of time-limited tasks via one or more input devices. Identify the user corresponding to the aforementioned request, Receive the set of the aforementioned states, In accordance with the determination that a first time-limited task among the set of time-limited tasks is associated with the identified user, at least a subset of the set of states is provided in a first order: A non-temporary computer-readable storage medium including instructions, which provides the set of states in a second order different from the first order, according to the determination that the first time-limited task among the set of time-limited tasks is not associated with the identified user. [Item 28] A computer system configured to communicate with one or more input devices, wherein the computer system is Means for receiving requests for a set of states corresponding to a separate set of time-limited tasks via one or more input devices, Means for identifying the user corresponding to the aforementioned request, Means for receiving the aforementioned set of states, Means for providing at least a subset of the set of states in a first order, according to the determination that a first time-limited task among the set of time-limited tasks is associated with the identified user, Means for providing at least a subset of the set of states in a second order different from the first order, based on the determination that the first time-limited task among the set of time-limited tasks is not associated with the identified user, A computer system equipped with the following features. [Item 29] A computer system configured to communicate with one or more input devices, wherein the computer system is One or more processors, A computer system comprising: memory for storing one or more programs configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the method described in any one of items 20 to 25. [Item 30] A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices, wherein the one or more programs include instructions to perform the method described in any one of items 20 to 25. [Item 31] A computer system configured to communicate with one or more input devices, wherein the computer system is A computer system comprising means for performing the method described in any one of items 20 to 25.

Claims

1. It is a method, In a computer system communicating with one or more input devices, Receiving natural language speech input via one or more input devices, including a request about the status of a timed task, Based on the aforementioned natural language speech input, the electronic device associated with the time-limited task is identified, Receiving the state of the aforementioned time-limited task, To provide an output indicating the state of the aforementioned time-limited task, Methods that include...

2. The method according to claim 1, wherein the timed task is performed on the electronic device.

3. To request the aforementioned state of the time-limited task is, The method according to claim 2, comprising causing the electronic device to retrieve the state of the timed task from a second electronic device different from the electronic device.

4. The method according to any one of claims 1 to 3, wherein requesting the state of the timed task includes providing an identifier for the timed task.

5. The method according to claim 4, wherein the identifier indicates the location of the electronic device.

6. The method according to claim 4, wherein the identifier indicates the name of the electronic device.

7. The method according to claim 4, wherein the identifier specifies the characteristics of the time-limited task.

8. Before receiving the natural language speech input which includes a request about the status of a timed task, Receiving a request to start the timed task on the electronic device via one or more input devices, The method according to any one of claims 1 to 7, further comprising:

9. The method of claim 8, wherein the request to start the timed task on the electronic device indicates the location of the electronic device.

10. Each of the multiple electronic devices corresponds to the location of the electronic device, and the method is The electronic device requests the commencement of the timed task, Identifying the device type of each of the aforementioned plurality of electronic devices, The method of claim 9, further comprising selecting an electronic device from among the plurality of electronic devices based on the identified device type.

11. The method according to any one of claims 8 to 10, wherein the request to initiate the timed task on the electronic device indicates the name of the electronic device.

12. The method according to any one of claims 1 to 11, wherein the state of the timed task indicates the remaining time of the timed task.

13. After providing an output indicating the state of the timed task, receive a request to cancel the timed task via one or more input devices. The method according to any one of claims 1 to 12, further comprising:

14. Receiving a second natural language speech input via one or more input devices, which includes a request for the state of a second time-limited task being performed on the computer system; Determining the state of the second time-limited task, To provide an output indicating the state of the second time-limited task, The method according to any one of claims 1 to 13, further comprising:

15. A computer system configured to communicate with one or more input devices, wherein the computer system is One or more processors, The system comprises a memory that stores one or more programs configured to be executed by one or more processors, and the one or more programs are The system receives natural language speech input via one or more input devices, including a request about the status of a timed task. Based on the natural language speech input, the electronic device associated with the timed task is identified. Upon receiving the state of the aforementioned time-limited task, A computer system including instructions that provides an output indicating the state of the aforementioned time-limited task.

16. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices, wherein the one or more programs are The system receives natural language speech input via one or more input devices, including a request about the status of a timed task. Based on the natural language speech input, the electronic device associated with the timed task is identified. Upon receiving the state of the aforementioned time-limited task, A non-temporary computer-readable storage medium including instructions, which provides an output indicating the state of the aforementioned time-limited task.

17. A computer system configured to communicate with one or more input devices, wherein the computer system is Means for receiving natural language speech input, including a request about the status of a time-limited task, via one or more input devices, Means for identifying an electronic device associated with the timed task based on the natural language voice input, Means for receiving the state of the aforementioned time-limited task, Means for providing an output indicating the state of the aforementioned time-limited task, A computer system equipped with the following features.

18. A computer system configured to communicate with one or more input devices, wherein the computer system is One or more processors, A computer system comprising: a memory for storing one or more programs configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the method according to any one of claims 1 to 14.

19. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices, wherein the one or more programs include instructions for performing the method according to any one of claims 1 to 14.

20. A computer system configured to communicate with one or more input devices, wherein the computer system is A computer system comprising means for performing the method described in any one of claims 1 to 14.

21. It is a method, In a computer system communicating with one or more input devices, Receiving requests for a set of states corresponding to a separate set of time-limited tasks via one or more input devices, Identifying the user corresponding to the aforementioned request, Receiving the set of the aforementioned states, To provide at least a subset of the set of states in a first order, based on the determination that a first time-limited task among the set of time-limited tasks is associated with the identified user, In accordance with the determination that the first time-limited task among the set of time-limited tasks is not associated with the identified user, the set of states is provided in a second order different from the first order, Methods that include...

22. Identifying the user corresponding to the request is, Receiving biometric information associated with the user, The method according to claim 21, comprising identifying the user based on the biometric information.

23. Receiving the request for the set of states includes receiving natural language speech input containing the request, The biometric information includes voice information associated with the user, Identifying the user based on the biometric information includes identifying the user based on the voice information. The method according to claim 22.

24. The aforementioned user is the first user, and identifying the aforementioned user is To determine a second user associated with the aforementioned computer system, The method according to any one of claims 21 to 23, comprising determining whether the first user matches the second user.

25. The method according to any one of claims 21 to 24, wherein each state in the subset of the set of states is associated with a timed task associated with the user.

26. The subset of the set of states is A second time-limited task, wherein the second time-limited task is associated with the user, and a first state associated with the second time-limited task, A third time-limited task, the third time-limited task being not associated with the user, and a second state associated with the third time-limited task, Providing at least a subset of the aforementioned set of states in a first order is: To provide the first state, Providing the first state, and then providing the second state, The method according to any one of claims 21 to 24.

27. A computer system configured to communicate with one or more input devices, wherein the computer system is One or more processors, The system comprises a memory that stores one or more programs configured to be executed by one or more processors, and the one or more programs are The system receives requests for a set of states corresponding to a separate set of time-limited tasks via one or more input devices. Identify the user corresponding to the aforementioned request, Receive the set of the aforementioned states, In accordance with the determination that a first time-limited task among the set of time-limited tasks is associated with the identified user, at least a subset of the set of states is provided in a first order: A computer system including instructions that, in accordance with the determination that the first time-limited task among the set of time-limited tasks is not associated with the identified user, provides the set of states, at least a subset of them, in a second order different from the first order.

28. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices, wherein the one or more programs are The system receives requests for a set of states corresponding to a separate set of time-limited tasks via one or more input devices. Identify the user corresponding to the aforementioned request, Receive the set of the aforementioned states, In accordance with the determination that a first time-limited task among the set of time-limited tasks is associated with the identified user, at least a subset of the set of states is provided in a first order: A non-temporary computer-readable storage medium including instructions, which provides the set of states in a second order different from the first order, according to the determination that the first time-limited task among the set of time-limited tasks is not associated with the identified user.

29. A computer system configured to communicate with one or more input devices, wherein the computer system is Means for receiving requests for a set of states corresponding to a separate set of time-limited tasks via one or more input devices, Means for identifying the user corresponding to the aforementioned request, Means for receiving the aforementioned set of states, Means for providing at least a subset of the set of states in a first order, according to the determination that a first time-limited task among the set of time-limited tasks is associated with the identified user, Means for providing at least a subset of the set of states in a second order different from the first order, in accordance with the determination that the first time-limited task among the set of time-limited tasks is not associated with the identified user, A computer system equipped with the following features.

30. A computer system configured to communicate with one or more input devices, wherein the computer system is One or more processors, A computer system comprising: a memory for storing one or more programs configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the method according to any one of claims 21 to 26.

31. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices, wherein the one or more programs include instructions for performing the method according to any one of claims 21 to 26.

32. A computer system configured to communicate with one or more input devices, wherein the computer system is A computer system comprising means for performing the method described in any one of claims 21 to 26.

Images