Activating your Virtual Assistant

Abstract

To provide an efficient method that triggers an intelligent automated assistant.SOLUTION: An electronic device 600 includes a display 604 and a microphone. While the display 604 of the electronic device 600 is in a first power state, when receiving a user input including the movement of the electronic device 600, the power state of the display 604 of the electronic device 600 is changed from the first power state to a second power state, and a virtual assistant session is triggered in accordance with a determination that the power state of the display 604 of the electronic device 600 is changed from the first power state to the second power state.SELECTED DRAWING: Figure 6B

Description

[Technical field]

[0001] The present disclosure relates generally to intelligent automated assistants, and more specifically to techniques for triggering intelligent automated assistants.

[0002] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 129,932, entitled "VIRTUAL ASSISTANT ACTIVATION," filed March 8, 2015, and U.S. Nonprovisional Patent Application No. 14 / 841,449, entitled "VIRTUAL ASSISTANT ACTIVATION," filed August 31, 2015, the contents of which are incorporated herein by reference in their entireties.

[0003] This application is related to U.S. Provisional Patent Application No. 62 / 026,532, entitled "Raise Gesture Detection in a Device," filed July 18, 2014, and U.S. Patent Application No. 12 / 987,982, entitled "Intelligent Automated Assistant," filed January 10, 2011, the contents of which are incorporated herein by reference. [Background technology]

[0004] An intelligent automated assistant (or virtual assistant) provides a useful interface between a human user and an electronic device. An exemplary virtual assistant can enable a user to interact with a device or system using natural language in verbal and / or textual form. A user can provide verbal input including a user request to a virtual assistant running on an electronic device. The virtual assistant interprets the user's intent from the verbal input and executes the user's intent as one or more tasks. The tasks can then be executed by executing one or more services of the electronic device, which can return relevant output in response to the user request. Summary of the Invention

[0005] As virtual assistants become more sophisticated, it may be desirable to solicit their assistance in an increasing number of usage scenarios. Therefore, efficient techniques for triggering virtual assistants are useful. In addition, in situations where a virtual assistant is running on a battery-powered portable electronic device, efficient techniques for triggering a virtual assistant while conserving battery power are particularly useful.

[0006] According to some embodiments, a method for triggering a virtual assistant on an electronic device includes receiving a user input that satisfies a predetermined condition via the input device while the display is on in an electronic device having a display, a microphone, and an input device, and sampling an audio input received via the microphone in accordance with receiving the user input that satisfies the predetermined condition, determining whether the audio input includes a verbal trigger, and determining that the audio input includes a verbal trigger. Triggering a virtual assistant session.

[0007] According to some embodiments, a method for triggering a virtual assistant on an electronic device includes detecting a software event that satisfies a predetermined condition in an electronic device having a display and a microphone, and sampling an audio input received via the microphone in accordance with a determination that a software event that satisfies the condition has been detected. Determine whether the audio input received via the microphone includes an oral trigger, and Trigger a virtual assistant session in accordance with a determination that the audio input includes an oral trigger.

[0008] According to some embodiments, a method for triggering a virtual assistant on an electronic device includes receiving a user input that satisfies a predetermined condition via the input device in an electronic device equipped with a display, a microphone, and an input device, sampling an audio input received via the microphone in accordance with receiving the user input that satisfies the predetermined condition, determining whether the audio input includes a verbal trigger, and determining that the audio input includes a verbal trigger. Triggering a virtual assistant session and providing a haptic output.

[0009] According to some embodiments, a method for triggering a virtual assistant on an electronic device includes, in an electronic device having a display and a microphone, sampling an audio input received via the microphone and determining whether the audio input represents a task; receiving an instruction to stop sampling the audio input while sampling the audio input; and providing an output notifying the user of the received instruction to stop sampling the audio input; and after providing this output, continuing to sample the audio input for a duration and determining whether the audio input includes a task, and then stopping sampling the audio input.

[0010] According to some embodiments, the electronic device includes a display, one or more processors, a memory, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the one or more processors, and the one or more programs receiving user input that satisfies a predetermined condition via the input device while the display is on, and in accordance with the receipt of the user input that satisfies the predetermined condition, sampling audio input received via the microphone, determining whether the audio input includes a verbal trigger, and in accordance with the determination that the audio input includes a verbal trigger, triggering a virtual assistant session. Includes instructions.

[0011] According to some embodiments, the electronic device includes a display, one or more processors, a memory, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the one or more processors, and the one or more programs detecting a software event that satisfies a predetermined condition, sampling audio input received via the microphone in accordance with a determination that a software event that satisfies the condition has been detected, determining whether the audio input received via the microphone includes a verbal trigger, and triggering a virtual assistant session in accordance with a determination that the audio input includes a verbal trigger.

[0012] According to some embodiments, an electronic device comprises a display, one or more processors, a memory, and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs receiving user input via an input device that satisfies a predetermined condition;

[0013] In response to receiving a user input that satisfies a predetermined condition, sample the audio input received via the microphone, determine whether the audio input includes a verbal trigger, and in response to a determination that the audio input includes the verbal trigger, trigger a virtual assistant session and provide a haptic output.

[0014] According to some embodiments, an electronic device comprises a display, one or more processors, a memory, and one or more programs configured to be stored in the memory and executed by the one or more processors, the one or more programs including instructions for sampling audio input received via a microphone, determining if the audio input represents a task, receiving an instruction to stop sampling the audio input while sampling the audio input, providing an output indicating receipt of the received instruction to stop sampling the audio input, and after providing the output, continuing to sample the audio input for a duration and determining if the audio input comprises a task, and thereafter stopping sampling the audio input.

[0015] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs, which, when executed by an electronic device having a display, a microphone, and an input device, cause the electronic device to receive user input that satisfies a predetermined condition via the input device while the display is on, and in accordance with the receipt of the user input that satisfies the predetermined condition, sample the audio input received via the microphone, determine whether the audio input includes a verbal trigger, and trigger a virtual assistant session in accordance with the determination that the audio input includes a verbal trigger.

[0016] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs, which, when executed by an electronic device having a display, a microphone, and an input device, cause the electronic device to detect a software event that satisfies a predetermined condition, and in accordance with a determination that a software event that satisfies the condition has been detected, sample audio input received via the microphone, and determine whether the audio input received via the microphone includes an oral trigger. Trigger a virtual assistant session according to a determination that the audio input includes an oral trigger.

[0017] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs, which, when executed by an electronic device having a display, a microphone, and an input device, cause the electronic device to receive user input that satisfies a predetermined condition via the input device, and in accordance with the receipt of the user input that satisfies the predetermined condition, sample the audio input received via the microphone, determine whether the audio input includes a verbal trigger, and in accordance with the determination that the audio input includes a verbal trigger, trigger a virtual assistant session and provide a haptic output. Includes instructions.

[0018] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs that, when executed by an electronic device having a display, a microphone, and an input device, cause the electronic device to sample audio input received via the microphone, determine if the audio input represents a task, receive, while sampling the audio input, an instruction to stop sampling the audio input, provide an output notifying the receipt of the received instruction to stop sampling the audio input, and, after providing the output, continue sampling the audio input for a duration and determine if the audio input comprises a task, and then stop sampling the audio input.

[0019] According to some embodiments, the electronic device receives a user input that satisfies a predetermined condition via the input device while the display means is on, and samples an audio input received via a microphone in accordance with the receipt of the user input that satisfies the predetermined condition. Means for determining whether the audio input includes a verbal trigger, and a virtual assistant session in accordance with a determination that the audio input includes a verbal trigger. Means for triggering the virtual assistant session.

[0020] According to some embodiments, the electronic device includes means for detecting a software event that satisfies a predetermined condition, and, in accordance with a determination that a software event that satisfies the condition has been detected, sampling audio input received via the microphone; determining whether the audio input received via the microphone includes an oral trigger; and, in accordance with a determination that the audio input includes an oral trigger, triggering a virtual assistant session. Means for doing so include:

[0021] According to some embodiments, the electronic device receives a user input that satisfies a predetermined condition via an input device, and samples the audio input received via the microphone in accordance with the receipt of the user input that satisfies the predetermined condition. Means for determining whether the audio input includes a verbal trigger, and in accordance with a determination that the audio input includes a verbal trigger, Trigger a virtual assistant session and provide a haptic output. Means for providing the haptic output.

[0022] According to some embodiments, an electronic device comprises means for sampling audio input received via a microphone and determining whether the audio input represents a task; means for receiving, while sampling the audio input, a command to stop sampling the audio input; means for providing an output indicating receipt of the received command to stop sampling the audio input; and means for continuing to sample the audio input for a duration after providing the output and determining whether the audio input comprises a task, and thereafter ceasing sampling of the audio input.

[0023] According to some embodiments, the electronic device includes a display, a microphone, an input device, and a processing unit coupled to the display, the microphone, and the input device. The processing unit receives user input that satisfies a predetermined condition via the input device while the display is on, and in accordance with the reception of the user input that satisfies the predetermined condition, samples the audio input received via the microphone, determines whether the audio input includes a verbal trigger, and triggers a virtual assistant session in accordance with the determination that the audio input includes a verbal trigger.

[0024] According to some embodiments, the electronic device includes a display, a microphone, and a processing unit coupled to the display, the microphone, and the input device, and the processing unit detects a software event that satisfies a predetermined condition, and in accordance with a determination that a software event that satisfies the condition has been detected, samples audio input received via the microphone, and determines whether the audio input received via the microphone includes an oral trigger. In accordance with a determination that the audio input includes an oral trigger, the virtual assistant session is triggered.

[0025] According to some embodiments, the electronic device includes a display, a microphone, an input device, and a processing unit coupled to the display, the microphone, and the input device. The processing unit receives user input that satisfies a predetermined condition via the input device, samples audio input received via the microphone in accordance with the receipt of the user input that satisfies the predetermined condition, and determines whether the audio input includes a verbal trigger. Trigger a virtual assistant session and provide a haptic output in accordance with the determination that the audio input includes a verbal trigger.

[0026] According to some embodiments, an electronic device comprises a display, a microphone, an input device, and a processing unit coupled to the display, the microphone and the input device, the processing unit being configured to: sample audio input received via the microphone; determine if the audio input represents a task; while sampling the audio input, receive an instruction to stop sampling the audio input; provide an output notifying of the received instruction to stop sampling the audio input; and, after providing the output, continue sampling the audio input for a duration and determine if the audio input comprises a task, and thereafter stop sampling the audio input. [Brief description of the drawings]

[0027] For a better understanding of the various described embodiments, please refer to the following Detailed Description in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout the drawings, in which:

[0028] [Figure 1] 1 illustrates a system and environment for implementing a digital assistant according to some embodiments.

[0029] [Figure 2A] FIG. 1 is a block diagram illustrating a portable multifunction device implementing a client-side portion of a digital assistant in accordance with some embodiments.

[0030] [Figure 2B] FIG. 2 is a block diagram illustrating example components for event processing according to some embodiments.

[0031] [Diagram 3] FIG. 1 is a block diagram illustrating a portable multifunction device implementing a client-side portion of a digital assistant in accordance with some embodiments.

[0032] [Figure 4] 1 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface in accordance with some embodiments.

[0033] [Figure 5A] 1 illustrates an exemplary user interface for an applications menu on a portable multifunction device in accordance with some embodiments.

[0034] [Figure 5B] 1 illustrates an exemplary user interface for a multifunction device having a touch-sensitive surface that is separate from a display in accordance with some embodiments.

[0035] [Figure 6A]1 illustrates a personal electronic device according to some embodiments.

[0036] [Figure 6B] FIG. 1 is a block diagram illustrating a personal electronic device in accordance with some embodiments.

[0037] [Figure 7A] 1 illustrates a digital assistant system or a server portion thereof according to some embodiments.

[0038] [Figure 7B] 8B illustrates the functionality of the digital assistant shown in FIG. 8A in accordance with some embodiments.

[0039] [Figure 7C] 1 illustrates a portion of an ontology according to some embodiments.

[0040] [Figure 8A] 1 illustrates an example technique for triggering a virtual assistant in accordance with some embodiments. [Figure 8B] 1 illustrates an example technique for triggering a virtual assistant in accordance with some embodiments. [Figure 8C] 1 illustrates an example technique for triggering a virtual assistant in accordance with some embodiments.

[0041] [Figure 9A] 1 illustrates an example technique for triggering a virtual assistant in accordance with some embodiments. [Figure 9B] 1 illustrates an example technique for triggering a virtual assistant in accordance with some embodiments. [Figure 9C] 1 illustrates an example technique for triggering a virtual assistant in accordance with some embodiments.

[0042] [Figure 10A] 1 illustrates an example technique for triggering a virtual assistant in accordance with some embodiments. [Figure 10B]1 illustrates an example technique for triggering a virtual assistant in accordance with some embodiments. [Figure 10C] 1 illustrates an example technique for triggering a virtual assistant in accordance with some embodiments. [Figure 10D] 1 illustrates an example technique for triggering a virtual assistant in accordance with some embodiments.

[0043] [Figure 11A] 1 illustrates an example technique for triggering a virtual assistant in accordance with some embodiments. [Figure 11B] 1 illustrates an example technique for triggering a virtual assistant in accordance with some embodiments. [Figure 11C] 1 illustrates an example technique for triggering a virtual assistant in accordance with some embodiments.

[0044] [Figure 12] FIG. 1 is a flow diagram illustrating a method for triggering a virtual assistant in accordance with some embodiments.

[0045] [Figure 13] FIG. 1 is a flow diagram illustrating a method for triggering a virtual assistant in accordance with some embodiments.

[0046] [Figure 14] FIG. 1 is a flow diagram illustrating a method for triggering a virtual assistant in accordance with some embodiments.

[0047] [Figure 15] FIG. 1 is a flow diagram illustrating a method for triggering a virtual assistant in accordance with some embodiments.

[0048] [Figure 16] FIG. 1 is a functional block diagram of an electronic device in accordance with some embodiments.

[0049] [Figure 17] FIG. 1 is a functional block diagram of an electronic device in accordance with some embodiments.

[0050] [Figure 18] FIG. 1 is a functional block diagram of an electronic device in accordance with some embodiments.

[0051] [Figure 19] FIG. 1 is a functional block diagram of an electronic device in accordance with some embodiments.

[0052] [Figure 20A] 1 illustrates an example technique for interacting with a virtual assistant in accordance with some embodiments. [Figure 20B] 1 illustrates an example technique for interacting with a virtual assistant in accordance with some embodiments. [Figure 20C] 1 illustrates an example technique for interacting with a virtual assistant in accordance with some embodiments. [Figure 20D] 1 illustrates an example technique for interacting with a virtual assistant in accordance with some embodiments. [Figure 20E] 1 illustrates an example technique for interacting with a virtual assistant in accordance with some embodiments.

[0053] [Figure 21] FIG. 1 is a flow diagram illustrating a method for interacting with a virtual assistant in accordance with some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0054] In the following description of the disclosure and embodiments, reference is made to the accompanying drawings in which is shown, by way of illustration, specific embodiments which may be practiced. It is understood that other embodiments and examples may be practiced and changes may be made without departing from the scope of the present disclosure.

[0055] Virtual assistant services can be computationally intensive. For example, a virtual assistant can sample audio input from a user by activating a microphone, its associated circuitry, and corresponding software processes, thereby consuming computing resources. There is a need for an electronic device that provides a virtual assistant service that employs techniques to reduce processor and associated battery power consumption, thereby conserving computing resources that would otherwise be wasted on processing unnecessary user input while providing an effective virtual assistant interface.

[0056] Figures 1, 2A, 7A-7B, and 16-19 below provide descriptions of example devices for performing the techniques for migrating between computing devices. Figures 8A-8C, 9A-9C, 10A-10D, and 11A-11C show example user interfaces for migrating between computing devices. The user interfaces in the figures are also used to explain the processes described below, including the processes of Figures 12-15.

[0057] In the following description, terms such as "first" and "second" are used to describe various elements, but these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first input can be referred to as a second input, and similarly, a second input can be referred to as a first input, without departing from the scope of the various embodiments being described. The first input and the second input can both be outputs, and in some cases, can be separate, distinct inputs.

[0058] The terms used in the description of the various embodiments described herein are for the purpose of describing particular embodiments only and are not intended to be limiting. When used in the description of the various embodiments described and in the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It is also to be understood that the term "and / or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "includes," "including," "comprises," and / or "comprising," as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0059] The term "if" can be interpreted to mean "when," "upon," "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [a stated condition or event] is detected" can be interpreted to mean "upon determining," or "in response to determining," or "upon detecting [the stated condition or event]," or "in response to detecting [the stated condition or event]," depending on the context. 1. System and Environment

[0060] FIG. 1 illustrates a block diagram of a system 100 according to various embodiments. In some embodiments, the system 100 may implement a digital assistant. The terms “digital assistant,” “virtual assistant,” “intelligent automated assistant,” or “automated digital assistant” may refer to any information processing system that interprets natural language input in spoken and / or textual form to infer user intent and perform actions based on the inferred user intent. For example, to execute the inferred user intent, the system may perform one or more of the following: identify a task flow having steps and parameters designed to realize the inferred user intent, populate the task flow with specific requirements from the inferred user intent, execute the task flow by invoking programs, methods, services, APIs, etc., and generate an output response to the user in an audible (e.g., speech) and / or visual form.

[0061] Specifically, a digital assistant may have the ability to accept user requests, at least in part, in the form of natural language commands, requests, statements, predications, and / or queries. Typically, a user request may seek either an informational answer or the performance of a task by the digital assistant. A satisfactory response to a user request may be the provision of the requested informational answer, the performance of the requested task, or a combination of both. For example, a user may ask a digital assistant a question such as "Where am I right now?" Based on the user's current location, the digital assistant may respond with "You are near the West Gate in Central Park." The user may request a task performance, such as, for example, "Invite my friends to my girlfriend's birthday party next week." In response, the digital assistant may acknowledge receipt of the request by stating "Yes, right now" and then send appropriate calendar invitations on behalf of the user to each of the user's friends in the user's electronic address book. During the performance of a requested task, the digital assistant may occasionally interact with the user in an ongoing dialog involving multiple information exchanges over an extended period of time. There are many other ways to interact with a digital assistant to request information or the performance of various tasks. In addition to providing verbal responses and taking programmed actions, the digital assistant can also provide other visual or audio forms of responses, such as text, alerts, music, videos, animations, etc.

[0062] As shown in FIG. 1, in some embodiments, the digital assistant can be implemented according to a client-server model. The digital assistant can include a client-side portion 102 (hereinafter, "DA client 102") that runs on a user device 104, and a server-side portion 106 (hereinafter, "DA server 106") that runs on a server system 108. The DA client 102 can communicate with the DA server 106 through one or more networks 110. The DA client 102 can provide client-side functionality such as user-responsive input and output processing, and communication with the DA server 106. The DA server 106 can provide server-side functionality for any number of DA clients 102, each resident on a respective user device 104.

[0063] In some embodiments, the DA server 106 may include a client-facing I / O interface 112, one or more processing modules 114, data and models 116, and an I / O interface 118 to external services. The client-facing I / O interface 112 may facilitate client-facing input and output processing for the DA server 106. The one or more processing modules 114 may utilize the data and models 116 to process speech input and determine user intent based on natural language input. Furthermore, the one or more processing modules 114 may perform task execution based on the inferred user intent. In some embodiments, the DA server 106 may communicate with external services 120 through the network(s) 110 for task completion or information acquisition. The I / O interface 118 to external services may facilitate such communication.

[0064] User device 104 may be any suitable electronic device. For example, user device may be a portable multifunction device (e.g., device 200 described below with reference to FIG. 2A), a multifunction device (e.g., device 400 described below with reference to FIG. 4), or a personal electronic device (e.g., device 600 described below with reference to FIGS. 6A-6B). A portable multifunction device may be, for example, a mobile phone that also includes other functions, such as PDA and / or music player functions. Particular examples of portable multifunction devices may include iPhone®, iPod Touch®, and iPad® devices by Apple Inc. (Cupertino, California). Other examples of portable multifunction devices may include, without limitation, laptop computers or tablet computers. Furthermore, in some embodiments, user device 104 may be a non-portable multifunction device. In particular, user device 104 may be a desktop computer, a game console, a television, or a television set-top box. In some embodiments, user device 104 may include a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). Additionally, the user device 104 may optionally include one or more other physical user interface devices, such as a physical keyboard, a mouse, and / or a joystick. Various examples of electronic devices, such as multifunction devices, are described in more detail below.

[0065] Examples of communication network(s) 110 include a local area network (LAN) and a wide area network (WAN), such as the Internet. Communication network(s) 110 may be implemented using any known network protocol, including various wired or wireless protocols, such as, for example, Ethernet, Universal Serial Bus (USB), FIREWIRE, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wi-Fi, voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol.

[0066] The server system 108 may be implemented on one or more stand-alone data processing devices or a distributed network of computers. In some embodiments, the server system 108 may also utilize various virtual devices and / or services of third party service providers (e.g., third party cloud service providers) to provide the underlying computing and / or infrastructure resources of the server system 108.

[0067] In some embodiments, the user device 104 may communicate with the DA server 106 via the second user device 122. The second user device 122 may be similar to the user device 104. For example, the second user device 122 may be similar to any one of the devices 200, 400, or 600 described below with reference to FIGS. 2A, 4, and 6A-B. The user device 104 may be configured to communicatively couple to the second user device 122 via a direct communication connection, e.g., Bluetooth, NFC, BTLE, etc., or via a wired or wireless network, e.g., a local Wi-Fi network. In some embodiments, the second user device 122 may be configured to act as a proxy between the user device 104 and the DA server 106. For example, the DA client 102 of the user device 104 may be configured to send information (e.g., a user request received at the user device 104) to the DA server 106 via the second user device 122. The DA server 106 may process the information and return relevant data (eg, data content in response to a user request) to the user device 104 via the second user device 122.

[0068] In some embodiments, the user device 104 may be configured to communicate an abbreviated request for data to the second user device 122 to reduce the amount of information sent from the user device 104. The second user device 122 may be configured to determine additional information and generate and send a full request in addition to the abbreviated request to the DA server 106. This system architecture may allow a user device 104 (e.g., a watch or similar compact electronic device) with limited communication capabilities and / or limited battery power to conveniently access services provided by the DA server 106 using a second user device 122 (e.g., a mobile phone, laptop computer, tablet computer, etc.) with stronger communication capabilities and / or greater battery power as a proxy to the DA server 106. Although only two user devices 104 and 122 are shown in FIG. 1, it should be understood that the system 100 may include any number and type of user devices configured in this proxy configuration to communicate with the DA server system 106.

[0069] Although the digital assistant shown in FIG. 1 can include both a client-side portion (e.g., DA client 102) and a server-side portion (e.g., DA server 106), in some embodiments, the functionality of the digital assistant can be implemented as a standalone application installed on a user device. In addition, the distribution of functionality between the client and server portions of the digital assistant can vary depending on the implementation. For example, in some embodiments, the DA client can be a thin client that provides only user-facing input and output processing functionality and relegates all other functionality of the digital assistant to a back-end server. 2. Electronic Devices

[0070] Attention is now directed to an embodiment of an electronic device for implementing a client-side portion of a digital assistant. FIG. 2A is a block diagram illustrating a portable multifunction device 200 with a touch-sensitive display system 212, according to some embodiments. The touch-sensitive display 212 may be conveniently referred to as a "touch screen" or may be known or referred to as a "touch-sensitive display system." The device 200 includes memory 202 (optionally including one or more computer-readable storage media), a memory controller 222, one or more processing units (CPUs) 220, a peripherals interface 218, an RF circuit 208, an audio circuit 210, a speaker 211, a microphone 213, an input / output (I / O) subsystem 206, other input control devices 216, and an external port 224. The device 200 optionally includes one or more optical sensors 264. The device 200 optionally includes one or more contact intensity sensors 265 for detecting the intensity of a contact on the device 200 (e.g., a touch-sensitive surface such as the touch-sensitive display system 212 of the device 200). Device 200 optionally includes one or more tactile output generators 267 for generating tactile output on device 200 (e.g., generating tactile output on a touch-sensitive surface, such as touch-sensitive display system 212 of device 200 or touchpad 455 of device 400). These components optionally communicate via one or more communication buses or signal lines 203.

[0071] As used herein and in the claims, the term “intensity” of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or a surrogate (proxy) for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four distinct values, and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using a variety of methods and a variety of sensors, or combinations of sensors. For example, one or more force sensors disposed under or adjacent to the touch-sensitive surface are optionally used to measure the force at various points on the touch-sensitive surface. In some implementations, the force measurements of multiple force sensors are combined (e.g., a weighted average) to determine an estimate of the contact force. Similarly, a pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size and / or change in a contact area detected on the touch-sensitive surface, the capacitance and / or change in the capacitance of the touch-sensitive surface proximate the contact, and / or the resistance and / or change in the capacitance of the touch-sensitive surface proximate the contact are optionally used as a surrogate for the force or pressure of the contact on the touch-sensitive surface. In some implementations, the surrogate measure of the force or pressure of the contact is used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the surrogate measure). In some implementations, the surrogate measure of the force or pressure of the contact is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of contact as an attribute of user input allows a user access to additional device functionality that may otherwise be inaccessible by the user on devices of reduced size that have a limited area for displaying affordances (e.g., on a touch-sensitive display) and / or receiving user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls such as knobs or buttons).

[0072] As used herein and in the claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, that is to be detected by a user with the user's sense of touch. For example, in a situation where a device or a component of the device is in contact with a touch-sensitive surface of a user (e.g., the fingers, palm, or other part of the user's hand), the tactile output generated by the physical displacement is interpreted by the user as a tactile sensation that corresponds to a perceived change in a physical property of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "down-click" or "up-click" of a physical actuator button. In some cases, a user will feel a tactile sensation such as a "down-click" or "up-click" even when there is no movement of a physical actuator button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's action. As another example, movement of a touch-sensitive surface is optionally interpreted or sensed by a user as "roughness" of the touch-sensitive surface, even when there is no change in the smoothness of the touch-sensitive surface. Although such user interpretation of touch will depend on the user's personal sensory perception, there are many sensory perceptions of touch that are common to a majority of users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., an "up click," a "down click," "roughness"), unless otherwise noted, the generated tactile output corresponds to a physical displacement of the device, or a component of a device, that produces the described sensory perception of a typical (or average) user.

[0073] It should be understood that device 200 is but one example of a portable multifunction device, and that device 200 optionally has more or fewer components than those shown, optionally combines two or more components, or optionally has a different configuration or arrangement of components. The various components shown in Figure 2A are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing circuits and / or application specific integrated circuits.

[0074] Memory 202 may include one or more computer-readable storage media, which may be tangible and non-transitory. Memory 202 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 222 may control access to memory 202 by other components of device 200.

[0075] In some embodiments, the non-transitory computer-readable storage medium of memory 202 may be used to store instructions (e.g., for performing aspects of methods 1200, 1300, 1400, or 1500 described below) used by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other system that may retrieve and execute instructions from an instruction execution system, apparatus, or device. In other embodiments, the instructions (e.g., for performing aspects of methods 1200, 1300, 1400, or 1500 described below) may be stored in a non-transitory computer-readable storage medium (not shown) of server system 108 or distributed between the non-transitory computer-readable storage medium of memory 202 and the non-transitory computer-readable storage medium of server system 108. In the context of this specification, a "non-transitory computer-readable storage medium" may be any medium that may contain or store a program for use by or with an instruction execution system, apparatus, or device.

[0076] A peripherals interface 218 can be used to couple input and output peripherals of the device to the CPU 220 and memory 202. The one or more processors 220 operate or execute various software programs and / or instruction sets stored in memory 202 to perform various functions and process data for the device 200. In some embodiments, the peripherals interface 218, CPU 220, and memory controller 222 may be implemented on a single chip, such as chip 204. In some other embodiments, they may be implemented on separate chips.

[0077] The RF (Radio Frequency) circuitry 208 transmits and receives RF signals, also referred to as electromagnetic signals. The RF circuitry 208 converts electrical signals to electromagnetic signals or electromagnetic signals to electrical signals and communicates with communication networks and other communication devices via electromagnetic signals. The RF circuitry 208 optionally includes well-known circuitry for performing these functions, including, but not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, and the like. The RF circuitry 208 optionally communicates by wireless communication with the Internet, also referred to as the World Wide Web (WWW), an intranet, and / or wireless networks, such as cellular telephone networks, wireless local area networks (LANs) and / or metropolitan area networks (MANs), and other devices. The RF circuitry 208 optionally includes well-known circuitry for detecting near field communication (NFC) fields, such as by near field communication radio. The wireless communication optionally uses any of a number of communication standards, protocols, and technologies. These include Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), long term evolution (LTE), near field communication (NFC), wideband code division multiple access (W-CDMA), code division multiple access (CDMA), and code division multiple access (CDMA).access (CDMA), time division multiple access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or IEEE 802.11ac), voice over Internet Protocol (VoIP), Wi-MAX, protocols for email (e.g., Internet message access protocol (IMAP) and / or post office protocol (POP)), instant messaging (e.g., extensible messaging and presence protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE)), instant messaging and presence services (e.g., IEEE 802.11b), instant messaging and presence services (e.g., IEEE 802.11c), instant messaging and presence services (e.g., IEEE 802.11d), instant messaging and presence services (e.g., IEEE 802.11g ... Such protocols may include, but are not limited to, Intermediate Message Service (IMPS), and / or Short Message Service (SMS), or any other suitable communications protocol, including communications protocols not yet developed as of the filing date of this document.

[0078] The audio circuitry 210, speaker 211, and microphone 213 provide an audio interface between a user and the device 200. The audio circuitry 210 receives audio data from the peripherals interface 218, converts the audio data to an electrical signal, and transmits the electrical signal to the speaker 211. The speaker 211 converts the electrical signal to sound waves audible to humans. The audio circuitry 210 also receives an electrical signal converted from the sound waves by the microphone 213. The audio circuitry 210 converts the electrical signal to audio data and transmits the audio data to the peripherals interface 218 for processing. The audio data may be retrieved from and / or transmitted to the memory 202 and / or the RF circuitry 208 by the peripherals interface 218. In some embodiments, the audio circuitry 210 further comprises a headset jack (e.g., 312 in FIG. 3). The headset jack provides an interface between audio circuitry 210 and a removable audio input / output peripheral device, such as an output-only headphone or a headset that has both an output (e.g., one- or binaural headphones) and an input (e.g., a microphone).

[0079] The I / O subsystem 206 couples input / output peripherals of the device 200, such as the touch screen 212 and other input control devices 216, to a peripheral interface 218. The I / O subsystem 206 optionally includes one or more input controllers 260 for a display controller 256, a light sensor controller 258, an intensity sensor controller 259, a haptic feedback controller 261, and other input or control devices. The one or more input controllers 260 receive / send electrical signals to / from the other input control devices 216. The other input control devices 216 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, the input controller(s) 260 are optionally coupled to any (or none) of a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. The one or more buttons (e.g., 308 in FIG. 3) optionally include up / down buttons for adjusting the volume of the speaker 211 and / or microphone 213. The one or more buttons optionally include a push button (e.g., 306 in FIG. 3).

[0080] A quick press of the push button may unlock the touch screen 212 or initiate a process of using gestures on the touch screen to unlock the device, as described in U.S. Patent Application Serial No. 11 / 322,549, "Unlocking a Device by Performing Gestures on an Unlock Image," U.S. Patent No. 7,657,849, filed December 23, 2005, which is incorporated by reference in its entirety. A longer press of a push button (e.g., 306) may power the device 200 on or off. The user may customize the functionality of one or more buttons. The touch screen 212 may be used to implement virtual or soft buttons and one or more soft keyboards.

[0081] The touch-sensitive display 212 provides an input and output interface between the device and a user. The display controller 256 receives and sends electrical signals to the touch screen 212. The touch screen 212 displays visual output to the user. This visual output may include graphics, text, icons, video, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output may correspond to user interface objects.

[0082] Touchscreen 212 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on haptic and / or tactile contact. Touchscreen 212 and display controller 256 (along with any associated modules and / or instruction sets in memory 202) detects contacts (and any movement or breaking of contact) on touchscreen 212 and translates the detected contacts into interactions with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on touchscreen 212. In an exemplary embodiment, the point of contact between touchscreen 212 and the user corresponds to the user's finger.

[0083] Touch screen 212 may use liquid crystal display (LCD), light emitting polymer display (LPD), or light emitting diode (LED) technology, although other display technologies may be used in other embodiments. Touch screen 212 and display controller 256 may detect contact and any movement or disruption of that contact using any of a number of touch sensing technologies now known or later developed, including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements to determine one or more points of contact with touch screen 212. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as that found in the iPhone® and iPod Touch® by Apple Inc. of Cupertino, California.

[0084] The touch-sensitive display in some embodiments of touchscreen 212 may be similar to the multi-touch-sensing touchpads described in the following U.S. Patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman), and / or U.S. Patent Application Publication No. 2002 / 0015024 (A1), each of which is incorporated by reference in its entirety herein, except that touchscreen 212 displays visual output from device 200, whereas touch-sensitive touchpads do not provide visual output.

[0085] The touch-sensitive display in some embodiments of touch screen 212 may be as described in the following applications: (1) U.S. Patent Application No. 11 / 381,313, filed May 2, 2006, entitled "Multipoint Touch Surface Controller"; (2) U.S. Patent Application No. 10 / 840,862, filed May 6, 2004, entitled "Multipoint Touchscreen"; (3) U.S. Patent Application No. 10 / 903,964, filed July 30, 2004, entitled "Gestures For Touch Sensitive Input Devices"; (4) U.S. Patent Application No. 11 / 048,264, filed January 31, 2005, entitled "Gestures For Touch Sensitive Input Devices"; (5) U.S. Patent Application No. 11 / 038,590, filed January 18, 2005, entitled "Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices"; (6) U.S. Patent Application No. 11 / 228,758, filed September 16, 2005, entitled "Virtual Input Device No. 11 / 228,700, filed Sep. 16, 2005, entitled "Operation Of A Computer With A Touch Screen Interface," (8) No. 11 / 228,737, filed Sep. 16, 2005, entitled "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard," and (9) No. 11 / 367,749, filed Mar. 3, 2006, entitled "Multi-Functional Hand-Held Device," all of which are incorporated herein by reference in their entirety.

[0086] The touch screen 212 may have a video resolution of more than 100 dpi. In some embodiments, the touch screen has a video resolution of approximately 160 dpi. A user may contact the touch screen 212 using any suitable object or appendage, such as a stylus, a finger, or the like. In some embodiments, the user interface is designed to function primarily through finger-based contact and gestures, which may be less precise than stylus-based input due to the large contact area of ​​a finger on the touch screen. In some embodiments, the device translates the coarse finger-based input into precise pointer / cursor position or commands to perform the user's desired action.

[0087] In some embodiments, in addition to a touchscreen, device 200 may include a touchpad (not shown) for activating or deactivating certain functions. In some embodiments, a touchpad is a touch-sensitive area of ​​a device that, unlike a touchscreen, does not display visual output. A touchpad can be a touch-sensitive surface separate from touchscreen 212 or an extension of the touch-sensitive surface formed by the touchscreen.

[0088] Device 200 also includes a power system 262 for providing power to the various components. Power system 262 may include a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharging system, power failure detection circuitry, power converters or inverters, power status indicators (e.g., light emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0089] The device 200 may also include one or more light sensors 264. FIG. 2A shows a light sensor coupled to a light sensor controller 258 in the I / O subsystem 206. The light sensor 264 may include a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The light sensor 264 receives light from the environment, projected through one or more lenses, and converts the light into data representing an image. In conjunction with an imaging module 243 (also called a camera module), the light sensor 264 can capture still images or video. In some embodiments, the light sensor is located on the back of the device 200, opposite the touchscreen display 212 on the front of the device, so that the touchscreen display can be used as a viewfinder for still and / or video image acquisition. In some embodiments, the light sensor is located on the front of the device so that an image of the user can be obtained for a video conference at the same time that the user sees other video conference participants on the touchscreen display. In some embodiments, the position of the light sensor 264 can be changed by the user (e.g., by rotating the lens and sensor within the device housing), thereby allowing a single light sensor 264 to be used in conjunction with a touch screen display for both video conferencing and capturing still and / or video images.

[0090] Device 200 also optionally includes one or more contact intensity sensors 265. FIG. 2A shows a contact intensity sensor coupled to intensity sensor controller 259 in I / O subsystem 206. Contact intensity sensor 265 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of a contact on a touch-sensitive surface). Contact intensity sensor 265 receives contact intensity information (e.g., pressure information or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is located on or proximate to a touch-sensitive surface (e.g., touch-sensitive display system 212). In some embodiments, at least one contact intensity sensor is located on the back of device 200, opposite touchscreen display 212 located on the front of device 200.

[0091] Device 200 may include one or more proximity sensors 266. Figure 2A shows proximity sensor 266 coupled to peripherals interface 218. Alternatively, proximity sensor 266 may be coupled to input controller 260 in I / O subsystem 206. The proximity sensor 266 may function as described in U.S. patent application Ser. Nos. 11 / 241,839, "Proximity Detector In Handheld Device," 11 / 240,788, "Proximity Detector In Handheld Device," 11 / 620,702, "Using Ambient Light Sensor To Augment Proximity Sensor Output," 11 / 586,862, "Automated Response To And Sensing Of User Activity In Portable Devices," and 11 / 638,251, "Methods And Systems For Automatic Configuration Of Peripherals," all of which are incorporated by reference herein in their entireties. In some embodiments, the proximity sensor turns off and disables the touch screen 212 when the multifunction device is placed near the user's ear (e.g., when the user is talking on the phone).

[0092] Device 200 also optionally includes one or more tactile output generators 267. FIG. 2A shows a tactile output generator coupled to haptic feedback controller 261 in I / O subsystem 206. Tactile output generator 267 optionally includes one or more electroacoustic devices, such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components that convert electrical signals into tactile output on the device). Contact intensity sensor 265 receives haptic feedback generation instructions from haptic feedback module 233 and generates a tactile output on device 200 that can be sensed by a user of device 200. In some embodiments, at least one tactile output generator is collocated with or proximate to a touch-sensitive surface (e.g., touch-sensitive display system 212) and optionally generates a tactile output by moving the touch-sensitive surface vertically (e.g., in / out of the surface of device 200) or laterally (e.g., back and forth in the same plane as the surface of device 200). In some embodiments, the at least one tactile output generator sensor is located on the back of device 200, opposite touchscreen display 212, which is located on the front of device 200.

[0093] Device 200 may also include one or more accelerometers 268. FIG. 2A shows accelerometer 268 coupled to peripherals interface 218. Alternatively, accelerometer 268 may be coupled to input controller 260 in I / O subsystem 206. Accelerometer 268 may function as described in U.S. Patent Application Publication Nos. 20050190059, "Acceleration-based Theft Detection System for Portable Electronic Devices," and 20060017692, "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer," both of which are incorporated herein by reference in their entireties. In some embodiments, information is displayed on a touchscreen display in portrait or landscape orientation based on an analysis of data received from one or more accelerometers. In addition to accelerometer(s) 268, device 200 optionally includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information regarding the location and orientation (e.g., portrait or landscape) of device 200.

[0094] In some embodiments, the software components stored in memory 202 include an operating system 226, a communication module (or instruction set) 228, a touch / motion module (or instruction set) 230, a graphics module (or instruction set) 232, a text input module (or instruction set) 234, a global positioning system (GPS) module (or instruction set) 235, a digital assistant client module 229, and an application (or instruction set) 236. Additionally, memory 202 may store data and models, such as user data and models 231. Additionally, in some embodiments, as shown in FIGS. 2A and 4, memory 202 (FIG. 2A) or memory 470 (FIG. 4) stores device / global internal state 257. The device / global internal state 257 includes one or more of the following: active application state, indicating which application, if any, is currently active; display state, indicating what applications, views, or other information are occupying various regions of the touchscreen display 212; sensor state, including information obtained from the device's various sensors and input control devices 216; and position information regarding the device's position and / or orientation.

[0095] Operating system 226 (e.g., an embedded operating system such as Darwin®, RTXC®, LINUX®, UNIX®, OS X®, iOS®, WINDOWS®, or VxWorks®) includes various software components and / or drivers for controlling and managing common system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware and software components.

[0096] Communications module 228 also includes various software components for facilitating communication with other devices through one or more external ports 224 and for processing data received by RF circuitry 208 and / or external port 224. External port 224 (e.g., Universal Serial Bus (USB), FIREWIRE®, etc.) is adapted to couple to other devices directly or indirectly through a network (e.g., Internet, wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector the same as or similar to the 30-pin connector used on iPod® (trademark of Apple Inc.) devices and / or a compatible multi-pin (e.g., 30-pin) connector.

[0097] Contact / motion module 230, optionally in conjunction with display controller 256, detects contact with touch screen 212 and with other touch-sensing devices (e.g., a touchpad or physical click wheel). Contact / motion module 230 includes various software components for performing various operations related to detecting contact, such as determining whether contact has occurred (e.g., detecting a finger down event), determining the strength of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger up event or an interruption of the contact). Contact / motion module 230 receives contact data from the touch-sensitive surface. Determining the movement of the contact point represented by the set of contact data optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These actions are, optionally, applied to a single contact (e.g., one finger contact) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contacts). In some embodiments, contact / motion module 230 and display controller 256 detect contacts on a touchpad.

[0098] In some embodiments, the contact / motion module 230 uses a set of one or more intensity thresholds for determining whether an action has been performed by a user (e.g., for determining whether a user has “clicked” on an icon). In some embodiments, at least a subset of the intensity thresholds are determined according to software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of a particular physical actuator and may be adjusted without modifying the physical hardware of the device 200). For example, the mouse “click” threshold of a trackpad or touchscreen display may be set to any of a wide range of predefined thresholds without modifying the hardware of the trackpad or touchscreen display. Additionally, in some implementations, a user of the device is provided with a software setting for adjusting one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once via a system-level click “intensity” parameter).

[0099] Contact / motion module 230 optionally detects gesture input by a user. Different gestures on a touch-sensitive surface have different contact patterns (e.g., different movements, timing, and / or intensity of detected contact). Thus, a gesture is optionally detected by detecting a particular contact pattern. For example, detecting a finger tap gesture includes detecting a finger down event (e.g., at an icon location) followed by detecting a finger lift off event at the same location (or substantially the same location) as the finger down event. As another example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger down event followed by one or more finger drag events followed by detecting a finger lift off event.

[0100] Graphics module 232 includes various known software components for rendering and displaying graphics on touchscreen 212 or other display, including components for modifying the visual effects (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, and the like.

[0101] In some embodiments, graphics module 232 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. Graphics module 232 receives one or more codes specifying the graphics to be displayed, including, as appropriate, coordinate data and other graphic characteristic data, from an application or the like, and then generates screen image data for output to display controller 256.

[0102] Haptic feedback module 233 includes various software components for generating instructions used by tactile output generator(s) 267 to generate tactile outputs at one or more locations on device 200 in response to user interaction with device 200. In some embodiments, event monitor 271 sends requests to peripherals interface 218 at predetermined intervals. In response, peripherals interface 218 sends event information. In other embodiments, peripherals interface 218 sends event information only if there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or exceeding a predetermined duration).

[0103] The text input module 234, which may be a component of the graphics module 232, provides a soft keyboard for entering text in various applications (e.g., contacts 237, email 240, IM 241, browser 247, and any other application requiring text input).

[0104] The GPS module 235 determines the location of the device and provides this information for use by various applications (e.g., to the phone 238 for location-based telephony use, to the camera 243 as photo / video metadata, and to applications that provide location-based services, such as a weather widget, a local yellow pages widget, and a maps / navigation widget).

[0105] Digital assistant client module 229 may include various client-side digital assistant instructions to provide the client-side functionality of a digital assistant. For example, digital assistant client module 229 may be capable of receiving voice input (e.g., speech input), text input, touch input, and / or gesture input through various user interfaces of portable multifunction device 200 (e.g., microphone 213, accelerometer(s) 268, touch-sensitive display system 212, light sensor(s) 229, other input control devices 216, etc.). Digital assistant client module 229 may also be capable of providing output in audio (e.g., voice output), visual, and / or tactile form through various output interfaces of portable multifunction device 200 (e.g., speaker 211, touch-sensitive display system 212, tactile output generator(s) 267, etc.). For example, output may be provided as voice, sound, alert, text message, menu, graphic, video, animation, vibration, and / or a combination of two or more of the above. In operation, the digital assistant client module 229 can communicate with the DA server 106 using the RF circuitry 208.

[0106] User data and models 231 may include various data associated with a user (e.g., user-specific vocabulary data, user preference data, user-specific name pronunciation, data from the user's electronic address book, to-do lists, shopping lists, etc.) to provide the client-side functionality of the digital assistant. Additionally, user data and models 231 may include various models for processing user input and determining user intent (e.g., speech recognition models, statistical language models, natural language processing models, ontologies, task flow models, service models, etc.).

[0107] In some embodiments, digital assistant client module 229 may use various sensors, subsystems, and peripherals of portable multifunction device 200 to gather additional information from the surrounding environment of portable multifunction device 200 and establish a context associated with the user, the current user interaction, and / or the current user input. In some embodiments, digital assistant client module 229 may provide context information, or a subset thereof, along with the user input to DA server 106 to help infer the user's intent. In some embodiments, the digital assistant may also use the context information to determine how to prepare and deliver output to the user. Context information may also be referred to as context data.

[0108] In some embodiments, the context information associated with the user input may include sensor information, such as ambient lighting, ambient noise, ambient temperature, images or video, etc. In some embodiments, the context information may also include a physical state of the device, such as device orientation, device location, device temperature, power level, speed, acceleration, motion patterns, cellular signal strength, etc. In some embodiments, information related to the software state of the DA server 106, such as running processes, installed programs, past and current network activity, background services, error logs, resource usage, etc., and the software state of the portable multifunction device 200 may be provided to the DA server 106 as context information associated with the user input.

[0109] In some embodiments, digital assistant client module 229 can selectively provide information (e.g., user data 231) stored on portable multifunction device 200 upon request from DA server 106. In some embodiments, digital assistant client module 229 can also elicit additional input from the user via a natural language dialog or other user interface upon request by DA server 106. Digital assistant client module 229 can pass additional input to DA server 106 to assist DA server 106 in intent inference and / or fulfillment of the user's intent expressed in the user request.

[0110] Further details of the digital assistant are described below with reference to Figures 7A-7C. It should be appreciated that the digital assistant client module 229 can include any number of sub-modules of the digital assistant module 726 described below.

[0111] The application 236 may include the following modules (or sets of instructions), or a subset or superset thereof: ● a contacts module 237 (sometimes called an address book or contact list); ● Telephone module 238, ● Videoconferencing module 239; ● an email client module 240; ● Instant messaging (IM) module 241; ● Training Support Module 242, A camera module 243 for still and / or video images, ● Image management module 244, ●Video player module, ●Music player module, ● Browser module 247, ● Calendar module 248, A widget module 249 that may include one or more of a weather widget 249-1, a stocks widget 249-2, a calculator widget 249-3, an alarm clock widget 249-4, a dictionary widget 249-5, and other widgets acquired by the user, as well as user-created widgets 249-6; A widget creator module 250 for creating user-created widgets 249-6; ●Search module 251 A video and music player module 252 that integrates a video player module and a music player module. ● Memo module 253, Map module 254, and / or ●Online video module 255.

[0112] Examples of other applications 236 that may be stored in memory 202 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA® enabled applications, encryption, digital rights management, voice recognition, and voice duplication.

[0113] In conjunction with the touch screen 212, the display controller 256, the contact / motion module 230, the graphics module 232, and the text input module 234, the contacts module 237 is used to manage an address book or contact list (e.g., stored in the memory 202 or in the application internal state 292 of the contacts module 237 in the memory 470), including adding a name(s) to the address book, deleting a name(s) from the address book, associating phone number(s), email address(es), address(es), or other information with a name, associating a picture with a name, categorizing and sorting names, providing phone numbers or email addresses to initiate and / or facilitate communication by phone 238, videoconferencing module 239, email 240, or IM 241, and the like.

[0114] In conjunction with RF circuitry 208, audio circuitry 210, speaker 211, microphone 213, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, telephone module 238 can be used to enter a series of characters corresponding to a telephone number, access one or more telephone numbers in contacts module 237, modify an entered telephone number, dial a corresponding telephone number, conduct a conversation, and disconnect or hang up when the conversation is completed. As discussed above, wireless communication can use any of a number of communication standards, protocols, and technologies.

[0115] In conjunction with RF circuitry 208, audio circuitry 210, speaker 211, microphone 213, touch screen 212, display controller 256, light sensor 264, light sensor controller 258, contact / motion module 230, graphics module 232, text input module 234, contacts module 237, and telephone module 238, videoconferencing module 239 contains executable instructions for initiating, conducting, and terminating a videoconference between a user and one or more other participants in accordance with the user's commands.

[0116] In conjunction with RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, email client module 240 contains executable instructions for composing, sending, receiving, and managing emails in response to user commands. Email client module 240 in conjunction with image management module 244 greatly facilitates composing and sending emails with attachments of still or video images captured by camera module 243.

[0117] In cooperation with the RF circuitry 208, the touch screen 212, the display controller 256, the contact / motion module 230, the graphics module 232, and the text input module 234, the instant messaging module 241 includes executable instructions for inputting characters corresponding to an instant message, modifying characters already input, sending the corresponding instant message (e.g., using Short Message Service (SMS) or Multimedia Message Service (MMS) protocols for telephone-based instant messaging, or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), and receiving and displaying the received instant message. In some embodiments, the sent and / or received instant messages may include graphics, photos, audio files, video files, and / or other attachments, such as those supported by MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant messaging" refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0118] In conjunction with the RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, GPS module 235, map module 254, and music player module, the training support module 242 includes executable instructions for: creating workouts (e.g., having time, distance, and / or calorie burn goals), communicating with training sensors (sports devices), receiving training sensor data, calibrating sensors used to monitor the workouts, selecting and playing music for the workouts, and displaying, storing, and transmitting the workout data.

[0119] In cooperation with touch screen 212, display controller 256, light sensor(s) 264, light sensor controller 258, contact / motion module 230, graphics module 232, and image management module 244, camera module 243 contains executable instructions to capture still images or video (including video streams) and store them in memory 202, change characteristics of the still images or video, or delete the still images or video from memory 202.

[0120] In conjunction with touch screen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, and camera module 243, image management module 244 contains executable instructions for arranging, modifying (e.g., editing) and otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slide show or album), or storing still and / or video images.

[0121] In conjunction with the RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, the browser module 247 contains executable instructions for browsing the Internet according to user commands, including retrieving, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.

[0122] In conjunction with the RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, email client module 240, and browser module 247, calendar module 248 contains executable instructions for creating, displaying, modifying, and storing calendars and data associated with the calendars (e.g., calendar items, to-do lists, etc.) in accordance with a user's instructions.

[0123] In conjunction with RF circuitry 208, touch screen 212, display controller 256, touch / motion module 230, graphics module 232, text input module 234, and browser module 247, widget module 249 is a mini-application that can be downloaded and used by a user (e.g., weather widget 249-1, stocks widget 249-2, calculator widget 249-3, alarm clock widget 249-4, and dictionary widget 249-5) or created by a user (e.g., user-created widget 249-6). In some embodiments, a widget includes an HTML (HyperText Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, a widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! Widget).

[0124] In conjunction with the RF circuitry 208, the touch screen 212, the display controller 256, the contact / motion module 230, the graphics module 232, the text input module 234, and the browser module 247, the widget creator module 250 can be used by a user to create widgets (e.g., turn user-specified portions of a web page into widgets).

[0125] In cooperation with the touch screen 212, the display controller 256, the contact / motion module 230, the graphics module 232, and the text input module 234, the search module 251 contains executable instructions for searching the memory 202 for text, music, sound, images, video, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with a user's commands.

[0126] In conjunction with touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, audio circuitry 210, speaker 211, RF circuitry 208, and browser module 247, video and music player module 252 contains executable instructions that enable a user to download and play pre-recorded music or other sound files stored in one or more file formats, such as MP3 or AAC files, as well as executable instructions for displaying, presenting, or otherwise playing videos (on touchscreen 212 or on an external display connected via external port 224). In some embodiments, device 200 optionally includes the functionality of an MP3 player, such as an iPod (a trademark of Apple Inc.).

[0127] In conjunction with the touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, the notes module 253 contains executable instructions for creating and managing notes, to-do lists, and the like according to a user's commands.

[0128] In conjunction with RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, GPS module 235, and browser module 247, map module 254 can be used to receive, display, modify, and store maps and data associated with the maps (e.g., driving directions, data about businesses and other points of interest at or near a particular location, and other location-based data) in accordance with a user's instructions.

[0129] In conjunction with touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, audio circuitry 210, speaker 211, RF circuitry 208, text input module 234, email client module 240, and browser module 247, online video module 255 contains instructions that enable a user to access, view, receive (e.g., by streaming and / or downloading), play (e.g., on the touchscreen or on an external display connected via external port 224), and send and otherwise manage emails containing links to particular online videos in one or more file formats, such as H.264. In some embodiments, instant messaging module 241 is used to send links to particular online videos, rather than email client module 240. Further description of online video applications can be found in U.S. Provisional Patent Application No. 60 / 936,562, filed June 20, 2007, entitled "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos," and U.S. Patent Application No. 11 / 968,067, filed December 31, 2007, entitled "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos," the contents of which are incorporated by reference in their entireties herein.

[0130] Each of the above-identified modules and applications corresponds to an executable set of instructions for performing one or more of the above-identified functions and methods (e.g., computer-implemented methods and other information processing methods described herein) described herein. These modules (e.g., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus, in various embodiments, various subsets of these modules may be combined or otherwise rearranged. For example, a video player module may be combined with a music player module into a single module (e.g., video and music player module 252 of FIG. 2A). In some embodiments, memory 202 may store a subset of the above-identified modules and data structures. Additionally, memory 202 may store additional modules and data structures not described above.

[0131] In some embodiments, device 200 is a device in which operation of a predefined set of functions on the device is performed exclusively via a touchscreen and / or touchpad. By using the touchscreen and / or touchpad as the primary input control device for operation of device 200, the number of physical input control devices (e.g., push buttons, dials, etc.) on device 200 can be reduced.

[0132] The set of predefined functions performed solely via the touchscreen and / or touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates device 200 to a main menu, a home menu, or a root menu from any user interface displayed on device 200. In such embodiments, a "menu button" is implemented using the touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device instead of a touchpad.

[0133] 2B is a block diagram illustrating example components for event processing, according to some embodiments. In some embodiments, memory 202 (FIG. 2A) or memory 470 (FIG. 4) includes an event sorter 270 (e.g., in operating system 226) and a corresponding application 236-1 (e.g., any of applications 237-251, 255, 480-490 described above).

[0134] Event sorter 270 receives the event information and determines which application 236-1 and application view 291 of application 236-1 to deliver the event information to. Event sorter 270 includes an event monitor 271 and an event dispatcher module 274. In some embodiments, application 236-1 includes application internal state 292 that indicates the current application view(s) that are displayed on touch-sensitive display 212 when the application is active or running. In some embodiments, device / global internal state 257 is used by event sorter 270 to determine which application(s) are currently active, and application internal state 292 is used by event sorter 270 to determine which application view 291 to deliver the event information to.

[0135] In some embodiments, application internal state 292 includes additional information such as one or more of resume information used when application 236-1 resumes execution, user interface state information indicating information is being displayed or ready to be displayed by application 236-1, a state queue to allow the user to return to a previous state or view of application 236-1, and a redo / undo queue of actions previously performed by the user.

[0136] Event monitor 271 receives event information from peripherals interface 218. The event information includes information about sub-events (e.g., a user's touch on touch-sensitive display 212 as part of a multi-touch gesture). Peripherals interface 218 transmits information it receives from I / O subsystem 206 or sensors such as proximity sensor 266, accelerometer(s) 268, and / or microphone 213 (via audio circuitry 210). Information that peripherals interface 218 receives from I / O subsystem 206 includes information from touch-sensitive display 212 or a touch-sensitive surface.

[0137] In some embodiments, the event sorter 270 also includes a hit view determination module 272 and / or an active event recognizer determination module 273 .

[0138] Hit view determination module 272 provides software procedures for determining where in one or more views a sub-event occurred when touch-sensitive display 212 displays more than one view. A view consists of controls and other elements that a user can see on the display.

[0139] Another aspect of a user interface associated with an application is the set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and in which touch-based gestures occur. The application view (of the corresponding application) in which a touch is detected may correspond to a programmatic level within the programmatic or view hierarchy of that application. For example, the lowest-level view in which a touch is detected may be referred to as the hit view, and the set of events that are recognized as appropriate inputs may be determined, at least in part, based on the hit view of the initial touch that initiates a touch-based gesture.

[0140] The hit view determination module 272 receives information associated with sub-events of a touch-based gesture. If an application has multiple views organized as a hierarchy, the hit view determination module 272 identifies the hit view as the lowest view in the hierarchy that should process the sub-event. In most situations, the hit view is the lowest level view in which the first sub-event occurs (e.g., the first sub-event in a sequence of sub-events that form an event or potential event). Once a hit view is identified by the hit view determination module 272, the hit view typically receives all sub-events associated with the same touch or input source for which it was identified as the hit view.

[0141] The active event recognizer determination module 273 determines which view or views in the view hierarchy should receive the particular sequence of sub-events. In some embodiments, the active event recognizer determination module 273 determines that only the hit view should receive the particular sequence of sub-events. In other embodiments, the active event recognizer determination module 273 determines that all views that contain the physical location of the sub-event are actively participating views, and therefore all actively participating views should receive the particular sequence of sub-events. In other embodiments, even if the touch sub-event is completely confined to the area associated with a particular view, views higher in the hierarchy remain actively participating views.

[0142] Event dispatcher module 274 sends the event information to an event recognizer (e.g., event recognizer 280). In embodiments that include an active event recognizer determination module 273, event dispatcher module 274 delivers the event information to the event recognizers determined by the active event recognizer determination module 273. In some embodiments, event dispatcher module 274 stores event information obtained by a corresponding event receiver 282 in an event queue.

[0143] In some embodiments, operating system 226 includes event sorter 270. Alternatively, application 236-1 includes event sorter 270. In yet other embodiments, event sorter 270 is a stand-alone module or part of another module stored in memory 202, such as contact / motion module 230.

[0144] In some embodiments, application 236-1 includes multiple event handlers 290 and one or more application views 291, each including instructions for handling touch events that occur within a respective view of the application's user interface. Each application view 291 of application 236-1 includes one or more event recognizers 280. Typically, each application view 291 includes multiple event recognizers 280. In other embodiments, one or more of the event recognizers 280 are part of a separate module, such as a user interface kit (not shown) or a higher level object from which application 236-1 inherits methods and other properties. In some embodiments, each event handler 290 includes one or more of a data updater 276, an object updater 277, a GUI updater 278, and / or event data 279 received from event sorter 270. Event handler 290 can utilize or call data updater 276, object updater 277, or GUI updater 278 to update application internal state 292. Alternatively, one or more of the application views 291 include one or more respective event handlers 290. In some embodiments, one or more of a data updater 276, an object updater 277, and a GUI updater 278 are also included in the corresponding application view 291.

[0145] Each event recognizer 280 receives event information (e.g., event data 279) from event sorter 270 and identifies an event from the event information. Event recognizer 280 includes an event receiver 282 and an event comparator 284. In some embodiments, event recognizer 280 further includes at least a subset of metadata 283 and event delivery instructions 288 (which may include sub-event delivery instructions).

[0146] The event receiver 282 receives event information from the event sorter 270. The event information includes information about a sub-event, such as information about a touch or a movement of a touch. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. If the sub-event relates to a movement of a touch, the event information may also include the speed and direction of the sub-event. In some embodiments, the event includes a rotation of the device from one orientation to another (e.g., from portrait to landscape or vice versa), and the event information includes corresponding information about the current orientation of the device (also called the device's attitude).

[0147] The event comparator 284 compares the event information to predefined event or sub-event definitions and determines the event or sub-event or determines or updates the state of the event or sub-event based on the comparison. In some embodiments, the event comparator 284 includes an event definition 286. The event definition 286 includes a definition of an event (e.g., a predefined sequence of sub-events), such as, for example, event 1 (287-1), event 2 (287-2), etc. In some embodiments, the sub-events in an event (287) include, for example, touch start, touch end, touch movement, touch stop, and multiple touches. In one example, the definition of event 1 (287-1) is a double tap on a displayed object. The double tap includes, for example, a first touch on a displayed object for a given stage (touch start), a first lift off (touch end) for the given stage, a second touch on a displayed object for a given stage (touch start), and a second lift off (touch end) for the given stage. In another example, a definition of event 2 (287-2) is a drag operation on a displayed object, including, for example, a touch (or contact) on the displayed object for a given step, a movement of the touch across the touch-sensitive display 212, and a lift-off of the touch (end of the touch). In some embodiments, the event also includes information about one or more associated event handlers 290.

[0148] In some embodiments, event definition 287 includes a definition of the event for each user interface object. In some embodiments, event comparator 284 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view in which three user interface objects are displayed on touch-sensitive display 212, when a touch is detected on touch-sensitive display 212, event comparator 284 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 290, event comparator 284 uses the results of the hit test to determine which event handler 290 to activate. For example, event comparator 284 selects the event handler associated with the sub-event and the object that triggers the hit test.

[0149] In some embodiments, the definition for each event (287) also includes a delay action that delays delivery of the event information until it is determined whether the sequence of sub-events corresponds to the event type of the event recognizer.

[0150] If the respective event recognizer 280 determines that the sequence of sub-events does not match any of the events in the event definition 286, the respective event recognizer 280 enters an event disabled, event failed, or event finished state and thereafter ignores the next sub-event of the touch-based gesture. In this situation, any other event recognizers that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0151] In some embodiments, each event recognizer 280 includes metadata 283 with configurable properties, flags, and / or lists that indicate how the event delivery system performs sub-event delivery to actively participating event recognizers. In some embodiments, metadata 283 includes configurable properties, flags, and / or lists that indicate how event recognizers may or may not interact with each other. In some embodiments, metadata 283 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to various levels in a view or programmatic hierarchy.

[0152] In some embodiments, each event recognizer 280 activates an event handler 290 associated with an event when one or more particular sub-events of the event are recognized. In some embodiments, each event recognizer 280 delivers event information associated with the event to the event handler 290. Activating the event handler 290 is distinct from sending (and postponing sending) the sub-events to the corresponding hit view. In some embodiments, the event recognizer 280 throws a flag associated with the recognized event, and the event handler 290 associated with the flag catches the flag and performs predefined processing.

[0153] In some embodiments, the event delivery instructions 288 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions send the event information to an event handler associated with a set of sub-events or an actively participating view. The event handler associated with the set of sub-events or an actively participating view receives the event information and performs a predetermined process.

[0154] In some embodiments, data updater 276 creates and updates data used by application 236-1. For example, data updater 276 updates phone numbers used by contacts module 237 or stores video files used by video player module. In some embodiments, object updater 277 creates and updates objects used by application 236-1. For example, object updater 277 creates new user interface objects or updates the positions of user interface objects. GUI updater 278 updates the GUI. For example, GUI updater 278 prepares display information and sends it to graphics module 232 for display on the touch-sensitive display.

[0155] In some embodiments, event handler(s) 290 include or have access to data updater 276, object updater 277, and GUI updater 278. In some embodiments, data updater 276, object updater 277, and GUI updater 278 are included in a single module of the respective application 236-1 or application view 291. In other embodiments, they are included in two or more software modules.

[0156] It will be understood that the foregoing discussion regarding event processing of a user's touch on a touch-sensitive display also applies to other forms of user input for operating multifunction device 200 using input devices, not all of which are initiated on a touch screen. For example, mouse movements and mouse button presses, optionally in conjunction with single or multiple keyboard presses or holds, touch movements such as tapping, dragging, scrolling on a touchpad, pen stylus input, device movements, verbal commands, detected eye movements, biometric input, and / or any combination thereof, are optionally utilized as inputs corresponding to sub-events that define the event to be recognized.

[0157] FIG. 3 illustrates a portable multifunction device 200 having a touch screen 212, according to some embodiments. The touch screen optionally displays one or more graphics in a user interface (UI) 300. In this embodiment, as well as in embodiments described below, a user can select one or more of the graphics by making a gesture on the graphics, for example, with one or more fingers 302 (not drawn to scale) or one or more styluses 303 (not drawn to scale). In some embodiments, selection of one or more graphics occurs when the user loses contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (left to right, right to left, upward and / or downward), and / or rolling of a finger in contact with the device 200 (right to left, left to right, upward and / or downward). In some implementations or situations, accidental contact with a graphic does not select the graphic. For example, if the gesture corresponding to selection is a tap, a swipe gesture sweeping over an application icon optionally does not select the corresponding application.

[0158] The device 200 may also include one or more physical buttons, such as a "home" or menu button 304. As described above, the menu button 304 may be used to navigate to any application 236 in a set of applications that may be executed on the device 200. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen 212.

[0159] In one embodiment, device 200 includes a touch screen 212, a menu button 304, a push button 306 for powering the device on / off and locking the device, a volume control button(s) 308, a subscriber identity module (SIM) card slot 310, a headset jack 312, and an external port 224 for docking / charging. The push button 306 powers the device on / off by pressing and holding the button down for a predefined time, locks the device by pressing and releasing the button before the predefined time has elapsed, and / or unlocks the device or initiates an unlocking process. In another embodiment, device 200 also receives verbal input through microphone 213 for activating or deactivating some functions. Device 200 also optionally includes one or more contact intensity sensors 265 for detecting the intensity of a contact on touch screen 212 and / or one or more tactile output generators 267 for generating a tactile output to a user of device 200.

[0160] FIG. 4 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface, according to some embodiments. Device 400 need not be portable. In some embodiments, device 400 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a child's learning toy), a gaming system, or a control device (e.g., a home or commercial controller). Device 400 typically includes one or more processing units (CPUs) 410, one or more network or other communication interfaces 460, memory 470, and one or more communication buses 420 for interconnecting these components. Communication bus 420 optionally includes circuitry (sometimes called a chipset) that interconnects and controls communication between system components. Device 400 includes an input / output (I / O) interface 430 with a display 440, which is typically a touchscreen display. The I / O interface 430 also optionally includes a keyboard and / or mouse (or other pointing device) 450, as well as a touchpad 455, a tactile output generator 457 (e.g., similar to the tactile output generator(s) 267 described above with reference to FIG. 2A) for generating tactile output on the device 400, and sensors 459 (e.g., optical sensors, acceleration sensors, proximity sensors, touch-sensitive sensors, and / or contact intensity sensors similar to the contact intensity sensor(s) 265 described above with reference to FIG. 2A). The memory 470 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile semiconductor storage devices. The memory 470 optionally includes one or more storage devices located remotely from the CPU(s) 410.In some embodiments, memory 470 stores programs, modules, and data structures similar to, or a subset of, the programs, modules, and data structures stored in memory 202 of portable multifunction device 200 (FIG. 2A). Additionally, memory 470 optionally stores additional programs, modules, and data structures not present in memory 202 of portable multifunction device 200. For example, memory 470 of device 400 optionally stores drawing module 480, presentation module 482, word processing module 484, website creation module 486, disk authoring module 488, and / or spreadsheet module 490, while memory 202 of portable multifunction device 200 (FIG. 2A) optionally does not store these modules.

[0161] Each of the above-identified elements of FIG. 4 may be stored in one or more of the memory devices mentioned above. Each of the above-identified modules corresponds to an instruction set that performs the functions described above. The above-identified modules or programs (e.g., instruction sets) need not be implemented as separate software programs, procedures, or modules, and thus, in various embodiments, various subsets of these modules may be combined or otherwise rearranged. In some embodiments, memory 470 may store a subset of the above-identified modules and data structures. Additionally, memory 470 may store additional modules and data structures not described above.

[0162] Attention is now directed to user interface embodiments that may be implemented on portable multifunction device 200, for example.

[0163] 5A shows an exemplary user interface for a menu of applications on portable multifunction device 200 in accordance with some embodiments. A similar user interface may be implemented on device 400. In some embodiments, user interface 500 includes the following elements, or a subset or superset thereof:

[0164] signal strength indicator(s) 502 for wireless communication(s), such as cellular and Wi-Fi signals; ●Time 504, ●Bluetooth indicator 505, ● Battery status indicator 506, A tray 508 containing icons for frequently used applications, such as: ○ An icon 516 for the phone module 238, labeled "Phone", optionally including an indicator 514 of the number of missed calls or voicemail messages; ○ An icon 518 for the email client module 240, labeled "Mail", optionally including an indicator 510 of the number of unread emails; ○ An icon 520 for the browser module 247, labeled "Browser"; and ○ An icon 522 for the video and music player module 252, also called the iPod (trademark of Apple Inc.) module 252, labeled "iPod"; and ●Icons for other applications, such as: ○ Icon 524 for IM module 241, labeled "Messages"; ○ Icon 526 for the Calendar module 248, labeled "Calendar" ○ Icon 528 for the image management module 244, labeled "Photos" ○ An icon 530 for the camera module 243, labeled "camera"; ○ Icon 532 for the online video module 255, labeled "Online Video"; ○ Icon 534 for stock widget 249-2, labeled "Stock Prices" ○ Icon 536 for map module 254, labeled "Map"; ○ Icon 538 for weather widget 249-1, labeled "Weather" ○ Icon 540 for alarm clock widget 249-4, labeled "Clock" ○ Icon 542 for Training Support Module 242, labeled "Training Support"; ○ An icon 544 for the notes module 253, labeled "Notes"; and ○ An icon 546 for a settings application or module, labeled “Settings,” that provides access to settings related to the device 200 and its various applications 236.

[0165] Note that the icon labels shown in FIG. 5A are merely exemplary. For example, icon 522 for video and music player module 252 may optionally be labeled "Music" or "Music Player." Other labels are optionally used for the various application icons. In some embodiments, the label for each application icon includes the name of the application that corresponds to the respective application icon. In some embodiments, the label of a particular application icon is different from the name of the application that corresponds to the particular application icon.

[0166] 5B shows an exemplary user interface on a device (e.g., device 400 of FIG. 4) with a touch-sensitive surface 551 (e.g., tablet or touchpad 455 of FIG. 4) that is separate from display 550 (e.g., touchscreen display 212). Device 400 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 457) for detecting the intensity of a contact on touch-sensitive surface 551 and / or one or more tactile output generators 459 for generating a tactile output to a user of device 400.

[0167] Although some of the following examples are described with reference to input on touch screen display 212 (where the touch-sensitive surface and display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, as shown in FIG. 5B. In some embodiments, this touch-sensitive surface (e.g., 551 in FIG. 5B) has a major axis (e.g., 552 in FIG. 5B) that corresponds to a major axis (e.g., 553 in FIG. 5B) on the display (e.g., 550). According to these embodiments, the device detects contact with touch-sensitive surface 551 (e.g., 560 and 562 in FIG. 5B) at locations that correspond to their respective locations on the display (e.g., in FIG. 5B, 560 corresponds to 568 and 562 corresponds to 570). In this manner, when the touch-sensitive surface is separate from the display, user input (e.g., contacts 560 and 562 and their movement) detected by the device on the touch-sensitive surface (e.g., 551 in FIG. 5B ) is used by the device to operate a user interface on the display (e.g., 550 in FIG. 5B ) of the multifunction device. It is to be understood that similar methods are optionally used for the other user interfaces described herein.

[0168] Additionally, while the following examples are described primarily with reference to finger inputs (e.g., finger contacts, finger tap gestures, finger swipe gestures), it should be understood that in some embodiments, one or more of the finger inputs are replaced with inputs from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced with a mouse click (e.g., instead of a contact) followed by cursor movement along the path of the swipe (e.g., instead of a contact movement). As another example, a tap gesture is optionally replaced with a mouse click while the cursor is positioned over the tap gesture location (e.g., instead of stopping contact detection following contact detection). Similarly, when multiple user inputs are detected simultaneously, it will be understood that multiple computer mice are optionally being used simultaneously, or that a mouse and finger contacts are being used simultaneously.

[0169] FIG. 6A illustrates an exemplary personal electronic device 600. Device 600 includes a body 602. In some embodiments, device 600 may include some or all of the features described with respect to devices 200 and 400 (e.g., FIGS. 2A-4B). In some embodiments, device 600 has a touch-sensitive display screen 604, hereafter touch screen 604. Instead of, or in addition to, touch screen 604, device 600 has a display and a touch-sensitive surface. Similar to devices 200 and 400, in some embodiments, touch screen 604 (or touch-sensitive surface) may include one or more intensity sensors for detecting the intensity of a contact (e.g., a touch) being applied. The one or more intensity sensors of touch screen 604 (or touch-sensitive surface) may provide output data representative of the intensity of the touch. A user interface of device 600 may respond to a touch based on the intensity of the touch, meaning that touches of different intensities may invoke different user interface actions on device 600.

[0170] Techniques for detecting and processing touch intensity can be found, for example, in related applications, International Patent Application No. PCT / US2013 / 040061, filed May 8, 2013, entitled "Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application," and PCT / US2013 / 069483, filed November 11, 2013, entitled "Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships," each of which is incorporated by reference in its entirety.

[0171] In some embodiments, device 600 has one or more input mechanisms 606 and 608. Input mechanisms 606 and 608, if included, can be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, device 600 has one or more attachment mechanisms. Such attachment mechanisms, if included, can allow device 600 to be attached to, for example, hats, eyewear, earrings, necklaces, shirts, jackets, bracelets, watch bands, chains, pants, belts, shoes, wallets, backpacks, etc. These attachment mechanisms can allow device 600 to be worn by a user.

[0172] FIG. 6B illustrates an exemplary personal electronic device 600. In some embodiments, the device 600 may include some or all of the components described in connection with FIGS. 2A, 2B, and 4. The device 600 includes a bus 612 that operably couples an I / O unit 614 to one or more computer processors 616 and a memory 618. The I / O unit 614 may be connected to a display 604 that may have a touch sensing component 622 and, optionally, a touch intensity sensing component 624. Additionally, the I / O unit 614 may be connected to a communication unit 630 for receiving application and operating system data using Wi-Fi, Bluetooth, Near Field Communication ("NFC"), cellular, and / or other wireless communication technologies. The device 600 may include an input mechanism 606 and / or an input mechanism 608. The input mechanism 606 may be, for example, a rotatable input device or a depressible and rotatable input device. The input mechanism 608 may be a button in some examples.

[0173] The input mechanism 608 may be a microphone in some embodiments. The personal electronic device 600 may include various sensors, such as a GPS sensor 632, an accelerometer 634, an orientation sensor 640 (e.g., a compass), a gyroscope 636, a motion sensor 638, and / or combinations thereof, all of which may be operably connected to the I / O section 614.

[0174] The memory 618 of the personal electronic device 600 may be a non-transitory computer-readable storage medium for storing computer-executable instructions that, when executed by one or more computer processors 616, may, for example, cause the computer processors to perform the techniques described below, including methods 1200, 1300, 1400, or 1500 (FIGS. 12-15). The computer-executable instructions may also be stored and / or transmitted to any non-transitory computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other system that may fetch instructions from and execute the instructions from an instruction execution system, apparatus, or device. For purposes of this document, a "non-transitory computer-readable storage medium" may be any medium capable of tangibly containing or storing computer-executable instructions for use by or in connection with an instruction execution system, apparatus, or device. Non-transitory computer-readable storage media may include, but are not limited to, magnetic, optical, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, CDs, DVDs, or optical disks based on Blu-ray technology, and persistent solid-state memory such as flash, solid-state drives, etc. Personal electronic device 600 is not limited to the components and configuration of FIG. 6B and may include other or additional components in multiple configurations.

[0175] As used herein, the term "affordance" refers to a user-interactive graphical user interface object that may be displayed on a display screen of device 200, device 400, and / or device 600 (FIGS. 2, 4, and 6). For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) can each constitute an affordance.

[0176] As used herein, the term "focus selector" refers to an input element that indicates the current portion of a user interface with which a user is interacting. In some implementations involving a cursor or other position marker, when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 455 of FIG. 4 or touch-sensitive surface 551 of FIG. 5B) and the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the cursor acts as a "focus selector" and the particular user interface element is adjusted according to the detected input. In some implementations involving a touchscreen display (e.g., touch-sensitive display system 212 of FIG. 2A or touchscreen 212 of FIG. 5A) that allows direct interaction with user interface elements on the touchscreen display, a contact detected on the touchscreen acts as a "focus selector" such that when an input (e.g., a press input by contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, the particular user interface element is adjusted according to the detected input. In some implementations, focus is moved from one region of the user interface to another region of the user interface without a corresponding cursor movement or contact movement on the touch-screen display (e.g., by using the tab or arrow keys to move focus from one button to another). In these implementations, the focus selector moves to follow the movement of focus between different regions of the user interface. Regardless of the specific form that the focus selector takes, the focus selector is generally a user interface element (or a contact on a touch-screen display) that is controlled by the user to communicate the user's intended interaction with the user interface (e.g., by indicating to the device which element of the user interface the user intends to interact with).For example, location of a focus selector (e.g., a cursor, touch, or selection box) over a corresponding button while a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen) indicates that the user intends to activate that corresponding button (and not other user interface elements shown on the device's display).

[0177] As used herein and in the claims, the term "characteristic intensity" of a contact refers to a characteristic of that contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on a plurality of intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected within a predefined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) relative to a predefined event (e.g., after detecting a contact, before detecting lift-off of the contact, before or after detecting the start of movement of the contact, before detecting the end of the contact, before or after detecting an increase in the intensity of the contact, and / or before or after detecting a decrease in the intensity of the contact). The characteristic intensity of a contact is optionally based on one or more of: a maximum intensity of the contact, a mean value of the intensity of the contact, an average value of the intensity of the contact, the top 10% of the intensity of the contact, half the maximum intensity of the contact, 90% of the maximum intensity of the contact, etc. In some embodiments, the duration of the contact is used to determine the characteristic intensity (e.g., when the characteristic intensity is an average of the intensity of the contact over time). In some embodiments, the characteristic intensity is compared to a set of one or more intensity thresholds to determine whether an action is performed by the user. For example, the set of one or more intensity thresholds may include a first intensity threshold and a second intensity threshold. In this example, a contact having a characteristic intensity that does not exceed the first threshold results in a first action being performed, a contact having a characteristic intensity that exceeds the first intensity threshold and does not exceed the second intensity threshold results in a second action being performed, and a contact having a characteristic intensity above the second threshold results in a third action being performed. In some embodiments, the comparison between the characteristic intensity and one or more thresholds is not used to determine whether to perform the first or second action, but rather to determine whether to perform one or more actions (e.g., to perform the respective action or to omit performing the respective action).

[0178] In some embodiments, a portion of the gesture is identified for purposes of determining the characteristic intensity. For example, a touch-sensitive surface may receive a continuous swipe contact that transitions from a start position to an end position where the intensity of the contact increases. In this example, the characteristic intensity of the contact at the end position may be based on only a portion of the swipe contact (e.g., only the portion of the swipe contact at the end position) rather than the entire continuous swipe contact. In some embodiments, a smoothing algorithm may be applied to the intensity of the swipe contact before determining the characteristic intensity of the contact. For example, the smoothing algorithm optionally includes one or more of an unweighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some circumstances, these smoothing algorithms filter out small increases or decreases in the intensity of the swipe contact for purposes of determining the characteristic intensity.

[0179] The intensity of a contact on the touch-sensitive surface may be characterized against one or more intensity thresholds, such as a contact-detection intensity threshold, a light pressure intensity threshold, a deep pressure intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light pressure intensity threshold corresponds to an intensity that causes the device to perform an action typically associated with a physical mouse button or a click of a trackpad. In some embodiments, the deep pressure intensity threshold corresponds to an intensity that causes the device to perform an action different from the action typically associated with a physical mouse button or a click of a trackpad. In some embodiments, if a contact is detected with a characteristic intensity below the light pressure intensity threshold (e.g., and above a slight contact-detection intensity threshold below which the contact is no longer detected), the device moves a focus selector in accordance with the movement of the contact on the touch-sensitive surface without performing an action associated with the light pressure intensity threshold or the deep pressure intensity threshold. Generally, unless otherwise specified, these intensity thresholds are consistent across different sets of user interface diagrams.

[0180] An increase in the characteristic intensity of the contact from an intensity below the light pressure intensity threshold to an intensity between the light pressure intensity threshold and the deep pressure intensity threshold may be referred to as a "light press" input. An increase in the characteristic intensity of the contact from an intensity below the deep pressure intensity threshold to an intensity above the deep pressure intensity threshold may be referred to as a "deep press" input. An increase in the characteristic intensity of the contact from an intensity below the contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light pressure intensity threshold may be referred to as detecting a contact on the touch surface. A decrease in the characteristic intensity of the contact from an intensity above the contact detection intensity threshold to an intensity below the contact detection intensity threshold may be referred to as detecting a lift-off of the contact from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0181] In some embodiments described herein, one or more actions are performed in response to detection of a gesture including a respective pressure input or in response to detection of a respective pressure input performed on a respective contact (or multiple contacts), where the respective pressure input is detected based at least in part on detection of an increase in intensity of the contact (or multiple contacts) above a pressure input intensity threshold. In some embodiments, the respective actions are performed in response to detection of an increase in intensity of the corresponding contact (e.g., a "downstroke" of the corresponding pressure input) above a pressure input intensity threshold. In some embodiments, the pressure input includes an increase in intensity of the corresponding contact above a pressure input intensity threshold and a subsequent decrease in intensity of the contact below the pressure input intensity threshold, where the corresponding action is performed in response to detection of a subsequent decrease in intensity of the corresponding contact below the pressure input intensity threshold (e.g., an "upstroke" of the corresponding pressure input).

[0182] In some embodiments, the device employs intensity hysteresis to avoid accidental inputs, sometimes referred to as "jitter," and the device defines or selects a hysteresis intensity threshold that has a predefined relationship to the pressure input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units below the pressure input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable percentage of the pressure input intensity threshold). Thus, in some embodiments, the pressure input includes an increase in the intensity of a corresponding contact above the pressure input intensity threshold and a subsequent decrease in the intensity of the contact below a hysteresis intensity threshold that corresponds to the pressure input intensity threshold, and a corresponding action is performed in response to detecting a subsequent decrease in the intensity of the corresponding contact below that hysteresis intensity threshold (e.g., an "upstroke" of the corresponding pressure input). Similarly, in some embodiments, a pressure input is detected only when the device detects an increase in the intensity of the contact from an intensity below the hysteresis intensity threshold to an intensity above the pressure input intensity threshold, and optionally a subsequent decrease in the intensity of the contact to an intensity below the hysteresis intensity, and a corresponding action is performed in response to detection of that pressure input (e.g., an increase in the intensity of the contact or a decrease in the intensity of the contact, as the case may be).

[0183] For ease of explanation, descriptions of actions performed in response to a pressure input associated with a pressure input intensity threshold or in response to a gesture including the pressure input are optionally triggered in response to detection of any of an increase in the intensity of the contact above the pressure input intensity threshold, an increase in the intensity of the contact from an intensity below a hysteresis intensity threshold to an intensity above the pressure input intensity threshold, a decrease in the intensity of the contact below the pressure input intensity threshold, and / or a decrease in the intensity of the contact below a hysteresis intensity threshold corresponding to the pressure input intensity threshold. Additionally, in embodiments described as an action being performed in response to detection of a decrease in the intensity of the contact below a pressure input intensity threshold, the action is optionally performed in response to detection of a decrease in the intensity of the contact below a hysteresis intensity threshold corresponding to and lower than the pressure input intensity threshold. 3. Digital Assistant Systems

[0184] FIG. 7A illustrates a block diagram of a digital assistant system 700 according to various embodiments. In some embodiments, the digital assistant system 700 can be implemented on a stand-alone computer system. In some embodiments, the digital assistant system 700 can be distributed across multiple computers. In some embodiments, some of the modules and functions of the digital assistant can be divided into a server portion and a client portion. In this case, the client portion resides on one or more user devices (e.g., device 104, device 122, device 200, device 400, or device 600) and communicates with the server portion (e.g., server system 108) through one or more networks, for example, as shown in FIG. 1. In some embodiments, the digital assistant system 700 can be an embodiment of the server system 108 (and / or DA server 106) shown in FIG. 1. It should be noted that the digital assistant system 700 is merely an example of a digital assistant system, and that the digital assistant system 700 can have more or fewer components than shown, can combine two or more components, or can have a different configuration or arrangement of components. The various components shown in FIG. 7A may be implemented in the form of hardware, including one or more signal processing circuits and / or application specific integrated circuits, software instructions executed by one or more processors, firmware, or a combination thereof.

[0185] The digital assistant system 700 can include memory 702, one or more processors 704, an input / output (I / O) interface 706, and a network communication interface 708. These components can communicate with each other through one or more communication buses or signal lines 710.

[0186] In some embodiments, memory 702 may include a non-transitory computer-readable medium, such as a high-speed random access memory and / or a non-volatile computer-readable storage medium (e.g., one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices).

[0187] In some examples, the I / O interface 706 can couple the input / output devices 716 of the digital assistant system 700, such as a display, keyboard, touch screen, and microphone, to the user interface module 722. The I / O interface 706 can cooperate with the user interface module 722 to receive user inputs (e.g., voice input, keyboard input, touch input, etc.) and process them accordingly. In some examples, for example, when the digital assistant is implemented on a standalone user device, the digital assistant system 700 can include any of the components and I / O communication interfaces described in connection with device 200, device 400, or device 600 in FIG. 2A, FIG. 4, FIG. 6A-FIG. 6B, respectively. In some examples, the digital assistant system 700 can represent the server portion of an embodiment of a digital assistant and can interact with a user through a client-side portion that resides on the user device (e.g., device 104, device 200, device 400, or device 600).

[0188] In some embodiments, the network communication interface 708 may include wired communication port(s) 712 and / or wireless transmission and reception circuitry 714. The wired communication port(s) may receive and transmit communication signals via one or more wired interfaces, e.g., Ethernet, Universal Serial Bus (USB), FIREWIRE, etc. The wireless circuitry 714 may receive and transmit RF and / or optical signals to / from communication networks and other communication devices. The wireless communication may use any of a number of communication standards, protocols, and technologies, such as GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol. The network communication interface 708 can enable communication between the digital assistant system 700 and devices using a network, such as the Internet, an intranet, and / or a cellular telephone network, a wireless network such as a wireless local area network (LAN), and / or a metropolitan area network (MAN).

[0189] In some embodiments, memory 702, or the computer-readable storage medium of memory 702, can store programs, modules, instructions, and data structures including all or a subset of an operating system 718, a communications module 720, a user interface module 722, one or more applications 724, and a digital assistant module 726. In particular, memory 702, or the computer-readable storage medium of memory 702, can store instructions to perform methods 1200, 1300, 1400, or 1500 described below. One or more processors 704 can execute these programs, modules, and instructions and read / write from / to the data structures.

[0190] Operating system 718 (e.g., Darwin®, RTXC®, LINUX®, UNIX®, iOS, OS X®, WINDOWS®, or an embedded operating system such as VxWorks) may include various software components and / or drivers for controlling and managing common system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between the various hardware, firmware, and software components.

[0191] The communications module 720 can facilitate communications between the digital assistant system 700 and other devices through the network communications interface 708. For example, the communications module 720 can communicate with the RF circuitry 208 of an electronic device such as device 200, device 400, and device 600 shown in FIGS. 2A, 4, and 6A-6B, respectively. The communications module 720 can also include various components for processing data received by the wireless circuitry 714 and / or the wired communications port 712.

[0192] User interface module 722 may receive commands and / or input from a user (e.g., from a keyboard, touch screen, pointing device, controller, and / or microphone) via I / O interface 706 and generate user interface objects on a display. User interface module 722 may also prepare and deliver output (e.g., speech, sound, animation, text, icons, vibration, haptic feedback, light, etc.) to the user via I / O interface 706 (e.g., through a display, audio channel, speaker, touchpad, etc.).

[0193] Applications 724 can include programs and / or modules configured to be executed by one or more processors 704. For example, if the digital assistant system is implemented on a standalone user device, applications 724 can include user applications such as games, calendar applications, navigation applications, or email applications. If the digital assistant system 700 is implemented on a server, applications 724 can include, for example, resource management applications, diagnostic applications, or scheduling applications.

[0194] Memory 702 can also store a digital assistant module 726 (or a server portion of a digital assistant). In some examples, the digital assistant module 726 can include the following submodules, or a subset or superset thereof: an input / output processing module 728, a speech-to-text (STT) processing module 730, a natural language processing module 732, a dialog flow processing module 734, a task flow processing module 736, a service processing module 738, and a speech synthesis module 740. Each of these modules can have access to one or more of the following systems or data and models of the digital assistant module 726, or a subset or superset thereof: an ontology 760, a vocabulary index 744, a user data 748, a task flow model 754, a service model 756, and an ASR system.

[0195] In some examples, using the processing modules, data, and models implemented in digital assistant module 726, the digital assistant can perform at least some of the following: converting voice input to text; identifying the user's intent as expressed in the natural language input received from the user; actively eliciting and obtaining the information necessary to fully infer the user's intent (e.g., by disambiguating words, games, intents, etc.); determining a task flow for fulfilling the inferred intent; and executing the task flow to fulfill the inferred intent.

[0196] In some examples, as shown in FIG. 7B , I / O processing module 728 can interact with a user through I / O device 716 in FIG. 7A or with a user device (e.g., device 104, device 200, device 400, or device 600) through network communication interface 708 in FIG. 7A to obtain user input (e.g., voice input) and provide responses to the user input (e.g., as voice output). I / O processing module 728 can optionally obtain context information associated with the user input from the user device along with or immediately after receiving the user input. The context information can include user-specific data, vocabulary, and / or preferences associated with the user input. In some examples, the context information also includes information about the software and hardware state of the user device at the time the user request is received and / or the user's surrounding environment at the time the user request is received. In some examples, I / O processing module 728 can also send follow-up questions to the user about the user request and receive answers from the user. When a user request is received by the I / O processing module 728, and the user request may include voice input, the I / O processing module 728 may forward the voice input to the STT processing module 730 (or a voice recognizer) for speech-to-text conversion.

[0197] The STT processing module 730 may include one or more ASR systems. The one or more ASR systems may process the speech input received through the I / O processing module 728 and generate recognition results. Each ASR system may include a front-end speech pre-processor. The front-end speech pre-processor may derive representative features from the speech input. For example, the front-end speech pre-processor may perform a Fourier transform on the speech input to derive spectral features that characterize the speech input as a series of representative multi-dimensional vectors. Additionally, each ASR system may include one or more speech recognition models (e.g., acoustic models and / or language models) and may implement one or more speech recognition engines. Examples of speech recognition models may include hidden Markov models, Gaussian mixture models, deep neural network models, n-gram language models, and other statistical models. Examples of speech recognition engines may include dynamic time warping-based engines and weighted finite-state transducer (WFST)-based engines. One or more speech recognition models and one or more speech recognition engines can be used to process the derived representative features of the front-end speech pre-processor to generate intermediate recognition results (e.g., phonemes, phoneme strings, and subwords) and, ultimately, text recognition results (e.g., words, word strings, or token strings). In some implementations, the speech input can be at least partially processed by a third-party service or on the user's device (e.g., device 104, device 200, device 400, or device 600) to generate the recognition results. Once the STT processing module 730 generates a recognition result that includes a text string (e.g., a word, a word string, or a token string), the recognition result can be passed to the natural language processing module 732 for intent inference.

[0198] Further details regarding speech-to-text processing are described in U.S. Utility Patent Application No. 13 / 236,942, for "Consolidating Speech Recognition Results," filed Sep. 20, 2011, the entire disclosure of which is incorporated herein by reference.

[0199] In some implementations, the STT processing module 730 may include and / or have access to a vocabulary of recognizable words via the speech-to-text module 731. Each vocabulary word may be associated with one or more candidate pronunciations of the word represented in the speech-recognition phonetic alphabet. In particular, the vocabulary of recognizable words may include words associated with multiple candidate pronunciations. For example, the vocabulary may include: The word "tomato" may be associated with a candidate pronunciation of TIFF0007625047000001.tif6128. Additionally, vocabulary words may be associated with custom candidate pronunciations based on previous voice input from the user. Such custom candidate pronunciations may be stored within the STT processing module 730 and associated with a particular user via that user's profile on the device. In some examples, candidate pronunciations for a word may be determined based on the spelling of the word and one or more linguistic and / or phonetic rules. In some examples, candidate pronunciations may be generated manually, for example, based on known standard pronunciations.

[0200] In some embodiments, the candidate pronunciations may be ranked based on the commonality of the candidate pronunciations. For example, TIFF0007625047000002.tif6128 because the former is a more commonly used pronunciation (e.g., among all users, for users in a particular geographic region, or for any other suitable subset of users). In some examples, the candidate pronunciations can be ranked based on whether the candidate pronunciation is a custom candidate pronunciation associated with the user. For example, a custom candidate pronunciation can be ranked higher than a standard candidate pronunciation. This can be useful for recognizing proper nouns that have unique pronunciations that deviate from the standard pronunciation. In some examples, the candidate pronunciations can be associated with one or more speech characteristics, such as place of origin, nationality, or ethnicity. For example, the candidate pronunciation TIFF0007625047000003.tif6128 can be associated with the United States, for which the candidate pronunciation TIFF0007625047000004.tif6128 may be associated with the United Kingdom. Additionally, the ranking of the candidate pronunciations may be based on one or more characteristics of the user (e.g., place of origin, nationality, ethnicity, etc.) stored in the user's profile on the device. For example, it may be determined from the user's profile that the user is associated with the United States. Based on the user's association with the United States, the ranking of the candidate pronunciations may be based on one or more characteristics of the user (e.g., place of origin, nationality, ethnicity, etc.) stored in the user's profile on the device. TIFF0007625047000005.tif6128 (associated with the United States) as a candidate pronunciation TIFF0007625047000006.tif6128 (associated with the United Kingdom). In some examples, one of the ranked candidate pronunciations may be selected as the predicted pronunciation (e.g., the most likely pronunciation).

[0201] When a speech input is received, the STT processing module 730 can be used to determine (e.g., using an acoustic model) the phonemes that correspond to the speech input, and then attempt to determine (e.g., using a language model) the words that match the phonemes. For example, the STT processing module 730 may first determine a sequence of phonemes that correspond to a portion of the speech input. If TIFF0007625047000007.tif6128 can be located, then it can determine, based on the lexical index 744, that this string corresponds to the word "tomato."

[0202] In some implementations, the STT processing module 730 may use proximity matching techniques to determine words within an utterance. Thus, for example, the STT processing module 730 may detect a phoneme sequence as being part of a word even if that particular phoneme sequence is not one of the candidate phoneme sequences for that word. It can be determined that TIFF0007625047000008.tif6128 corresponds to the word "tomato."

[0203] The digital assistant's natural language processing module 732 ("natural language processor") can take the string of words or tokens ("token string") generated by the STT processing module 730 and attempt to associate the token string with one or more "actionable intents" recognized by the digital assistant. An "actionable intent" can represent a task that can be performed by the digital assistant and can have an associated task flow implemented in the task flow model 754. The associated task flow can be a sequence of programmed actions and steps that the digital assistant takes to perform a task. The scope of the digital assistant's capabilities can depend on the number and type of task flows implemented and stored in the task flow model 754, or in other words, the number and type of "actionable intents" that the digital assistant recognizes. However, the effectiveness of the digital assistant can also depend on the assistant's ability to infer the exact "actionable intent(s)" from a user request expressed in natural language.

[0204] In some examples, in addition to the string of words or tokens obtained from the STT processing module 730, the natural language processing module 732 can also receive context information associated with the user request, for example from the I / O processing module 728. The natural language processing module 732 can optionally use the context information to clarify, complement, and / or further clarify information contained in the token string received from the STT processing module 730. The context information can include, for example, user preferences, hardware and / or software state of the user device, sensor information collected before, during, or immediately after the user request, previous interactions (e.g., dialogue) between the digital assistant and the user, and the like. As described herein, the context information can be dynamic and can change with time, location, content of the dialogue, and other factors.

[0205] In some embodiments, the natural language processing may be based on, for example, an ontology 760. The ontology 760 is a hierarchical structure that includes a number of nodes, each of which may represent either an "actionable intent" or an "attribute" that is related to one or more of the "actionable intents" or other "attributes." As described above, an "actionable intent" may represent a task that the digital assistant has the ability to perform; that is, it may be "actionable" or may be subject to performance. An "attribute" may represent a parameter associated with an actionable intent or a sub-aspect of another attribute. A link between an actionable intent node and an attribute node in the ontology 760 may define how the parameter represented by the attribute node relates to the task represented by the actionable intent node.

[0206] In some embodiments, ontology 760 may be composed of actionable intent nodes and attribute nodes. Within ontology 760, each actionable intent node may be linked to one or more attribute nodes, either directly or through one or more intermediate attribute nodes. Similarly, each attribute node may be linked to one or more actionable intent nodes, either directly or through one or more intermediate attribute nodes. For example, as shown in FIG. 7C , ontology 760 may include a “Restaurant Reservation” node (i.e., an actionable intent node). The attribute nodes “Restaurant”, “Date / Time” (for reservations), and “Number of Participants” may each be directly connected to the actionable intent node (i.e., the “Restaurant Reservation” node).

[0207] Furthermore, the attribute nodes “cuisine”, “price range”, “phone number”, and “location” may be subnodes of the attribute node “restaurant”, and each may be connected to the “reservation of restaurant” node (i.e., an actionable intent node) through the intermediate attribute node “restaurant”. As another example, as shown in FIG. 7C , ontology 760 may also include a “set reminder” node (i.e., another actionable intent node). The attribute nodes “date / time” (for setting a reminder) and “subject” (for a reminder) may each be linked to the “set reminder” node. Because the attribute node “date / time” may be related to both the task of making a restaurant reservation and the task of setting a reminder, the attribute node “date / time” may be linked to both the “reservation of restaurant” node and the “set reminder” node in ontology 760.

[0208] An actionable intent node, together with its linked concept nodes, can be described as a "domain." In this description, each domain can be associated with a respective actionable intent and refers to a group of nodes (and relationships between nodes) associated with a particular actionable intent. For example, the ontology 760 shown in FIG. 7C can include an example of a restaurant reservation domain 762 and an example of a reminder domain 764 within the ontology 760. The restaurant reservation domain includes an actionable intent node "restaurant reservation," attribute nodes "restaurant," "date / time," and "number of participants," and sub-attribute nodes "cuisine," "price range," "phone number," and "location." The reminder domain 764 can include an actionable intent node "reminder settings," and attribute nodes "subject" and "date / time." In some embodiments, the ontology 760 can be composed of many domains. Each domain can share one or more attribute nodes with one or more other domains. For example, a "date / time" attribute node may be associated with many different domains (eg, a scheduling domain, a travel booking domain, a movie ticket domain, etc.) in addition to a restaurant reservation domain 762 and a reminder domain 764.

[0209] 7C shows two example domains within the ontology 760, other domains (i.e., actionable intents) may include, for example, "find a movie," "make a phone call," "find directions," "schedule a meeting," "send a message," and "provide an answer to a question," "read a list," "provide navigation instructions," "provide task instructions," and the like. The "send a message" domain is associated with the "send a message" actionable intent node and may further include attribute nodes such as "recipient(s)," "message type," and "message body." The attribute node "recipient" may be further defined by sub-attribute nodes such as, for example, "recipient name" and "message address."

[0210] In some examples, ontology 760 can include all domains (and therefore actionable intents) that the digital assistant has the ability to understand and act upon. In some examples, ontology 760 can be modified, such as by adding or removing entire domains or nodes, or by changing relationships between nodes in ontology 760.

[0211] In some embodiments, nodes associated with multiple related actionable intents may be clustered under a “super domain” in the ontology 760. For example, a “travel” super domain may include a cluster of travel-related attribute nodes and actionable intent nodes. The travel-related actionable intent nodes may include “book a flight”, “book a hotel”, “rent a car”, “get directions”, “find places of interest”, etc. Actionable intent nodes under the same super domain (e.g., the “travel” super domain) may share many attribute nodes. For example, the actionable intent nodes for “book a flight”, “book a hotel”, “rent a car”, “get directions”, and “find places of interest” may share one or more of the attribute nodes “departure location”, “destination”, “departure date / time”, “arrival date / time”, and “number of participants”.

[0212] In some examples, each node in the ontology 760 can be associated with a set of words and / or phrases related to the attribute or actionable intent represented by that node. The respective set of words and / or phrases associated with each node can be a so-called "vocabulary" associated with that node. The respective set of words and / or phrases associated with each node can be stored in the vocabulary index 744 in association with the attribute or actionable intent represented by the node. For example, returning to FIG. 7B, the vocabulary associated with a node for the attribute "restaurant" can include words such as "food", "drink", "dish", "hungry", "eat", "pizza", "fast food", "meal", etc. As another example, the vocabulary associated with a node for the actionable intent "initiate a phone call" can include words and phrases such as "call", "phone", "dial", "ring", "call this number", "make a call to", etc. The vocabulary index 744 can optionally include words and phrases in different languages.

[0213] The natural language processing module 732 can receive a token sequence (e.g., a text string) from the STT processing module 730 and determine which nodes are implied by words in the token sequence. In some embodiments, if a word or phrase in the token sequence is found to be associated with one or more nodes in the ontology 760 (via the vocabulary index 744), the word or phrase can "trigger" or "activate" those nodes. Based on the amount and / or relative importance of the activated nodes, the natural language processing module 732 can select one of the possible intents as the task that the user intends the digital assistant to perform. In some embodiments, the domain with the most "triggered" nodes can be selected. In some embodiments, the domain with the highest confidence value (e.g., based on the relative importance of its various triggered nodes) can be selected. In some embodiments, the domain can be selected based on a combination of the number and importance of the triggered nodes. In some embodiments, additional factors are also considered when selecting a node, such as whether the digital assistant has previously correctly interpreted a similar request from the user.

[0214] User data 748 can include user-specific information, such as user-specific vocabulary, user preferences, user addresses, the user's default and secondary languages, the user's contact list, and other short-term or long-term information about each user. In some embodiments, the natural language processing module 732 can use the user-specific information to supplement information contained in the user input to further clarify the user's intent. For example, in response to a user request "invite my friends to my birthday party," the natural language processing module 732 can access user data 748 to determine who the "friends" are and when and where the "birthday party" will be held, instead of asking the user to explicitly provide such information in the user's request.

[0215] Additional details of ontology searching based on token strings are described in U.S. Utility Patent Application No. 12 / 341,743, filed Dec. 22, 2008, for "Method and Apparatus for Searching Using an Active Ontology," the entire disclosure of which is incorporated herein by reference.

[0216] In some examples, once the natural language processing module 732 identifies an actionable intent (or domain) based on the user request, the natural language processing module 732 can generate a structured query to represent the identified actionable intent. In some examples, the structured query can include parameters for one or more nodes in the domain for the actionable intent, where at least a portion of the parameters are populated with the specific information and requirements specified in the user request. For example, a user may say, "Make a dinner reservation for me at a sushi restaurant at 7 o'clock." In this case, the natural language processing module 732 can accurately identify the actionable intent as "restaurant reservation" based on the user input. According to the ontology, a structured query for the "restaurant reservation" domain can include parameters such as {cuisine}, {time}, {date}, {number of participants}, and the like. In some examples, based on the voice input and text derived from the voice input using the STT processing module 730, the natural language processing module 732 can generate a partial structured query for the restaurant reservation domain. In this case, the partial structured query includes the parameter {cuisine="sushi"} and the parameter {time="7pm"}. However, in this example, the user's utterance does not contain enough information to complete the structured query associated with the domain. Therefore, other required parameters such as {number of participants} and {date} may not be specified in the structured query based on the currently available information. In some embodiments, the natural language processing module 732 may populate some parameters of the structured query with received context information. For example, in some embodiments, if the user requests sushi restaurants "near me," the natural language processing module 732 populates the {location} parameter in the structured query with GPS coordinates from the user device 104.

[0217] In some examples, natural language processing module 732 can pass the generated structured query (including any completed parameters) to a task flow processing module 736 ("task flow processor"). Task flow processing module 736 can be configured to receive the structured query from natural language processing module 732, complete the structured query as necessary, and perform the actions required to "complete" the user's ultimate request. In some examples, the various steps required to complete these tasks can be provided within a task flow model 754. In some examples, task flow model 754 can include steps for obtaining additional information from the user and a task flow for performing the actions associated with the actionable intent.

[0218] As described above, to complete a structured query, task flow processing module 736 may need to initiate additional dialogue with the user to obtain additional information and / or disambiguate potentially ambiguous utterances. If such dialogue is required, task flow processing module 736 may invoke dialog flow processing module 734 to engage in a dialogue with the user. In some examples, dialog flow processing module 734 may determine how (and / or when) to ask the user for additional information, receive user responses, and process these. Questions may be provided to the user through I / O processing module 728, and answers may be received from the user. In some examples, dialog flow processing module 734 may present dialog output to the user via audio and / or visual output, and receive input from the user via verbal responses or physical (e.g., click) responses. Continuing with the example above, when task flow processing module 736 invokes dialog flow processing module 734 to determine "number of participants" and "date" information for a structured query associated with the domain "restaurant reservation," dialog flow processing module 734 may generate questions such as "for how many people?" and "on what date?" to pass to the user. Once a response is received from the user, dialog flow processing module 734 can then enter the missing information into the structured query or pass the missing information to task flow processing module 736 to complete the missing information from the structured query.

[0219] Once the taskflow processing module 736 completes the structured query for the actionable intent, the taskflow processing module 736 can proceed to execute the final task associated with the actionable intent. In response, the taskflow processing module 736 can execute steps and instructions in the taskflow model depending on the specific parameters contained in the structured query. For example, a taskflow model for an actionable intent of "restaurant reservation" can include steps and instructions for contacting a restaurant and actually requesting a reservation for a specific number of participants at a specific time. For example, with a structured query for restaurant reservation, restaurant=ABC Cafe, date=3 / 12 / 2012, time=7pm, number of participants=5, etc., the taskflow processing module 736 can execute the steps of (1) logging on to the server of ABC Cafe or a restaurant reservation system such as OPENTABLE®, (2) entering the date, time, and number of participants information into a form on the website, (3) submitting the form, and (4) posting a calendar item for the reservation in the user's calendar.

[0220] In some embodiments, task flow processing module 736 can use the assistance of service processing module 738 ("service processing module") to complete a task requested in a user input or provide an answer for information requested in a user input. For example, service processing module 738 can act on behalf of task flow processing module 736 to make a phone call, set a calendar item, invoke a map search, invoke or interact with other user applications installed on the user device, and invoke or interact with third party services (e.g., restaurant reservation portals, social networking websites, banking portals, etc.). In some embodiments, the protocols and application programming interfaces (APIs) required by each service can be specified by a respective service model in service models 756. Service processing module 738 can access the appropriate service model for the service and generate a request for the service according to the protocols and APIs required by the service associated with the service model.

[0221] For example, if a restaurant enables an online reservation service, the restaurant may submit a service model that specifies the parameters required to make a reservation and an API for communicating the values ​​of the required parameters to the online reservation service. When requested by the task flow processing module 736, the service processing module 738 may establish a network connection with the online reservation service using the web address stored in the service model and transmit the required reservation parameters (e.g., time, date, number of participants) to the online reservation interface in a format conforming to the API of the online reservation service.

[0222] In some examples, the natural language processing module 732, the dialog flow processing module 734, and the task flow processing module 736 can be used collectively and iteratively to infer and clarify the user's intent, obtain information to further clarify and refine the user's intent, and ultimately generate a response (i.e., output to the user or completion of a task) to fulfill the user's intent. The generated response can be a dialog response to the voice input that at least partially fulfills the user's intent. Furthermore, in some examples, the generated response can be output as a voice output. In these examples, the generated response can be sent to a speech synthesis module 740 (e.g., a speech synthesizer), where it can be processed to synthesize a dialog response in voice form. In still other examples, the generated response can be data content related to fulfilling the user request in the voice input.

[0223] The speech synthesis module 740 can be configured to synthesize speech output for presentation to the user. The speech synthesis module 740 synthesizes speech output based on text provided by the digital assistant. For example, the generated dialog response can be in the form of a text string. The speech synthesis module 740 can convert the text string into an audible speech output. The speech synthesis module 740 can use any suitable speech synthesis technique to generate speech output from the text, including, but not limited to, concatenative synthesis, unit selection synthesis, diphone synthesis, domain-specific synthesis, formant synthesis, articulatory synthesis, hidden Markov model (HMM)-based synthesis, and sinusoidal synthesis. In some embodiments, the speech synthesis module 740 can be configured to synthesize individual words based on a phoneme string corresponding to the word. For example, a phoneme string can be associated with a word in the generated dialog response. The phoneme string can be stored in metadata associated with the word. The speech synthesis model 740 can be configured to directly process the phoneme string in the metadata to synthesize the word in a phonetic form.

[0224] In some embodiments, instead of (or in addition to) using the speech synthesis module 740, speech synthesis can be performed on a remote device (e.g., server system 108) and the synthesized speech can be sent to a user device for output to the user. For example, this can be done in some implementations where output for a digital assistant is generated at a server system. Also, because server systems typically have more processing power or resources than user devices, it can be possible to obtain higher quality speech output than would be possible using client-side synthesis.

[0225] Further details regarding digital assistants can be found in U.S. Utility Patent Application No. 12 / 987,982, entitled "Intelligent Automated Assistant," filed January 10, 2011, and U.S. Utility Patent Application No. 13 / 251,088, entitled "Generating and Processing Task Items That Represent Tasks to Perform," filed September 30, 2011, the disclosures of which are incorporated by reference herein in their entireties.

[0226] We now turn our attention to techniques for triggering (e.g., starting) a virtual assistant that may run on an electronic device with a display and a touch-sensitive surface, and an accompanying user interface ("UI"). 4. Exemplary Techniques for Triggering a Virtual Assistant

[0227] 8A-8C show an example technique for triggering a virtual assistant based on whether an input on an electronic device satisfies a predetermined condition, according to some embodiments. These figures are also used to illustrate the processes described below, including the process in FIG.

[0228] 8A illustrates an electronic device 800 of a user 802. Device 800 is, in some embodiments, one of device 104, device 122, device 200, device 400, device 600, and device 1700 (FIGS. 1, 2, 4, 6, and 17). Device 800 has a touchscreen 804. In the illustrated (e.g., lowered) position, touchscreen 804 is powered off to conserve battery use.

[0229] Figure 8B shows a user 802 lifting the electronic device 800 to a raised position for viewing. In response to the movement, the device 800 powers on the touch screen 804 to present a user interface. Techniques for determining device movements, such as the lift movement shown in Figures 8A and 8B, are further described below with reference to Figure 12.

[0230] Following the touch screen 804 being powered on, the device 800 also begins sampling audio input via its microphone to listen for verbal input from the user 802. FIG. 8C shows the user 802 providing a verbal command 810 while the device 800 is sampling the audio input. When the device 800 detects the verbal command 810 while sampling the audio input, the device determines whether the sampled audio input includes a phrase that triggers the virtual assistant service. If the device 800 determines that the sampled input includes a verbal trigger that triggers the virtual assistant, the device 800 triggers the virtual assistant. If the device 800 determines that the sampled audio input does not include a verbal trigger, it can continue to sample the audio input for the verbal trigger for a limited time.

[0231] In the illustrated embodiment, the verbal trigger is the phrase "Hey Siri." Thus, the verbal command 810, when sampled by the device 800, causes the device 800 to trigger the virtual assistant service. However, the command 810 does not include additional language expressing a user request other than the trigger phrase. Thus, when the device 800 triggers the virtual assistant service, it waits for further verbal input by continuing to sample the audio input. While waiting for the additional input, the device 800 optionally displays an indication 812 prompting the user 802 for further commands.

[0232] In this way, the device 800 activates a microphone, associated circuitry, and corresponding software processes for sampling audio input to the virtual assistant based on whether the device detects a particular condition of use. This technique allows the device 800 to disable certain electrical circuits (e.g., microphone circuitry) and / or reduce computer instruction execution (e.g., associated processor power consumption) when the user is unlikely to be actively providing cognitive input to the device, for example, when the device is put down and its display is powered off, thereby reducing overall power consumption by the device. In other words, the technical advantage of battery conservation is achieved in some embodiments by the idea that the user is unlikely to provide verbal input while the device is outside of a particular condition of use.

[0233] 9A-9D show an example technique for triggering a virtual assistant according to some embodiments. These figures are also used to illustrate the processes described below, including the process in FIG.

[0234] FIG. 9A illustrates an electronic device 900 of a user 902. The device 900 is one of the devices 104, 122, 200, 400, 600, and 1600 (FIGS. 1, 2, 4, 6, and 16) in some embodiments. The device 900 has a touch screen 904. The touch screen 904 displays a notification 906 notifying the user 902 of an incoming message that has just been received by the device 900. The display of the notification 906 is triggered by one or more software events associated with the incoming message. Following detection of one or more associated software events, the device 900 begins sampling audio input via its microphone to identify verbal input. Techniques for determining associated software events, such as message notifications as seen in FIGS. 9A and 9B, are further described below with reference to FIG. 13.

[0235] FIG. 9B shows a user 902 providing a verbal command 910 while the device 900 is sampling the audio input. When the device 900 detects the verbal command 910 in response to sampling the audio input, the device determines whether the sampled input includes a verbal trigger for the virtual assistant. In the illustrated embodiment, the verbal trigger is the phrase "Hey Siri." If the device 900 determines that the sampled input includes a verbal trigger that triggers the virtual assistant, the device 900 triggers the virtual assistant. If the device 900 determines that the sampled audio input does not include a verbal trigger, it can continue to sample the audio input for the verbal trigger for a limited time.

[0236] In the illustrated embodiment, the verbal command 910 includes both verbal triggers (e.g., "hey Siri") and natural language input (e.g., "tell the sender I'm not available to chat right now") that the virtual assistant can execute as a computing task. For example, the task is to send a brief reply to the sender of the incoming message 912. Moving to FIG. 9C, in response to sampling the audio input including the command 910, the electronic device 900 initiates and sends a message 916 to the sender 918 of the incoming message 912 to inform the sender that the user 902 is currently unavailable.

[0237] 9A-9C, the present technique of powering on the microphone of device 900 in response to a software event detected by the device may disable certain electrical circuitry (e.g., microphone circuitry) and / or reduce computer instruction execution (e.g., associated processor power consumption) when the user is unlikely to be providing active cognitive input to the device, thereby reducing overall power consumption by the device. In other words, the technical advantage of battery conservation is achieved in some embodiments by the notion that the user is unlikely to be providing verbal input during the absence of a software event that would cause the device to generate output to the user.

[0238] 10A-10D show an example technique for triggering a virtual assistant according to some embodiments. These figures are also used to illustrate the processes described below, including the process in FIG.

[0239] FIG. 10A illustrates an electronic device 1000 of a user 1002. The device 1000, in some embodiments, is one of the devices 104, 122, 200, 400, 600, and 1700 (FIGS. 1, 2, 4, 6, and 17). The device 1000 has a touchscreen 1004. In the illustrated position (e.g., lowered), the touchscreen 1004 is powered off to conserve the battery life of the device 1000. FIG. 10B illustrates a user 1002 lifting the electronic device 1000 to a raised position for viewing. In response to the movement, the device 1000 begins sampling an audio input of the verbal input from the user 1002. In some embodiments, the device 1000 begins sampling this audio input even if the touchscreen 1004 remains off. In some embodiments, the device 1000 begins sampling this audio input regardless of whether the touchscreen 1004 is on or off.

[0240] FIG. 10C shows a user 1002 providing a verbal command 1010 while the device 1000 samples audio input with the touch screen 1004 off. When the device 1000 detects the verbal command 1010 according to the sampling of the audio input, the device determines whether the sampled input includes a verbal trigger for the virtual assistant. In the illustrated embodiment, the verbal trigger is the phrase "Hey Siri." If the device 1000 determines that the sampled input includes a verbal trigger that triggers the virtual assistant, the device 1000 triggers the virtual assistant without turning on the touch screen if the touch screen 1004 is powered off. If the device 1000 determines that the sampled audio input does not include a verbal trigger, it can continue to sample the audio input for the verbal trigger for a period of time, as further described with reference to FIG. 14 below.

[0241] In the illustrated example of FIG. 10C, the sampled verbal commands include verbal triggers (e.g., "hey Siri") and natural language inputs (e.g., "download my email") that the virtual assistant can execute as a task. In some embodiments, the device 1000 provides an output (e.g., a haptic and / or audio output) that acknowledges receipt of the received verbal input. In some embodiments, the acknowledgment output is provided without turning on the touch screen 1004.

[0242] 10D, in response to verbal command 1010, device 1000 initiates the requested task of downloading email. In some embodiments, the task occurs while touchscreen 1004 remains powered off. In some embodiments, device 1000 provides an output (e.g., a tactile and / or audio output) that indicates to the user that verbal input is occurring and / or has occurred. In some embodiments, the output is provided without turning on touchscreen 1004.

[0243] 10A-10D, under some usage conditions, the technique of keeping the touchscreen off and powering on the device's microphone to process the user's verbal input reduces overall power consumption by device 1000 while maintaining the device's ability to process the user's verbal input. In other words, the technical advantage of battery conservation is achieved in some embodiments through efficient control of display activation in display-equipped devices.

[0244] 11A-11C show an example technique for triggering a virtual assistant according to some embodiments. These figures are also used to illustrate the processes described below, including the process in FIG.

[0245] FIG. 11A shows an electronic device 1100 of a user 1102. The electronic device 1100 is, in some embodiments, one of the devices 104, 122, 200, 400, 600, and 1800 (FIGS. 1, 2, 4, 6, and 18). The device 1100 has a touch screen 1104. The touch screen 1104 displays an indication 1106 that the virtual assistant session is active and waiting for the user's verbal command. While the virtual assistant session is active, the microphone of the device 1100 samples the audio input of the user's command. Other user interfaces (not shown) can also indicate that the virtual assistant session is active, such as a virtual assistant transcript that lists previous user inputs and the virtual assistant's response. The touch screen 1104 also displays and includes a cancel affordance 1108 for terminating the virtual assistant session.

[0246] As seen in FIG. 11B, in response to detecting a user's selection of the cancel affordance 1108, the device 1100 reports to the user that the virtual assistant session has ended by replacing the virtual assistant's indication 1106 with a non-virtual assistant's user interface. In an example embodiment, the device 1100 displays a digital clock face 1110. In some embodiments (not explicitly shown in FIG. 11B), the device 1100 powers off the touch screen 1104 upon notifying the user that it has received a user's cancellation of the virtual assistant session.

[0247] However, in some embodiments, even when the device 1100 indicates to the user that the virtual assistant session has ended, the microphone continues to sample the audio input of the user's verbal commands for some time. In some embodiments, the ongoing audio sampling occurs for a predetermined time (e.g., 8 seconds). In some embodiments, the ongoing audio sampling occurs until additional user input, such as a touch input to the touch screen 1104, application activation, etc., is received. Further techniques for determining the duration for which sampling of audio input continues are described below with reference to FIG. 15.

[0248] Referring to FIG. 11C, during ongoing sampling of audio input, the device 1100 triggers the virtual assistant in response to sampled audio input that includes a verbal trigger, such as verbal input 1112 that includes the phrase "hey Siri."

[0249] This technique of continuing to sample audio input for a period of time after the user has terminated the virtual assistant (rather than immediately turning off the microphone) is beneficial. For example, consider a situation in which the user's cancellation of the virtual assistant (e.g., in FIG. 11B) is incorrect and has the effect of putting the device into a state in which it would be waiting for immediate input, such as moving the device before sampling the audio input again. That is, the user's incorrect cancellation would prevent the device's ability to process immediate subsequent verbal input by the user. In such a situation, the technique described above allows the device to respond as if the incorrect user input had not occurred.

[0250] As another example, consider a situation in which a user's cancellation of a virtual assistant requires the device to power cycle its microphone and associated circuitry, thereby delaying the device's ability to process the user's subsequent verbal input. The techniques described above allow the electronic device 1100 to respond to the user while avoiding the power cycle delay, thereby improving response time. Such advantages provide the device 1100 with a high-quality user experience and are particularly useful for portable devices where design compromises are made between processing power and battery consumption.

[0251] 20A-20F show an example technique for processing a user's request to a virtual assistant according to some embodiments. These figures are also used to illustrate the processes described below, including the process in FIG.

[0252] FIG. 20A shows an electronic device 2000 of a user 2002. The electronic device 2000 is, in some embodiments, one of 104, 122, 200, 400, 600, and 1900 (FIGS. 1, 2, 4, 6, and 19). The device 2000 has a touch screen 2004. As shown in FIG. 20A, the device 2000 displays a virtual assistant user interface 2006, which indicates that the device is sampling an audio input of a user's instruction 2010 instructing the device to initiate a message to the recipient.

[0253] 20B, the device 2000 begins to process the verbal command 2010 and identifies computing tasks that can be performed to fulfill the user's command. The time required to execute the user's command varies in different usage scenarios. For example, a slow data service provider may slow down the response time of the device.

[0254] As seen in FIG. 20C, the user 2002 can place the device 2000 in a non-viewing position while the virtual assistant processes the user's instructions. When there is no user input, the touch screen 2004, in the illustrated embodiment, turns off after a period of time to conserve battery power, even though the virtual assistant continues to process the user's instructions 2010.

[0255] 20D, when device 2000 executes the user's instructions to the point where an output should be displayed to the user, device 2000 provides a tactile and / or audible output to notify the user that additional information is now available. For example, in response to user instruction 2010, device 2000 may display a confirmation after launching a messages application and populating an outgoing message and before sending the message to the recipient.

[0256] As seen in FIG. 20E, upon being alerted by the tactile and / or audible output, the user 2002 may raise the device 2000 to a viewing position, which powers on the touchscreen 2004 and shows a message 2012 addressed to recipient "Jen" and including a message body 2012 (both of which correspond to the user's natural language input 2010).

[0257] Although the examples of Figures 20A-20E show a user's command to create a message, the described techniques are not so limited. The virtual assistant of device 2000 may use haptic and / or audible alerts to inform the user whenever it obtains a useful result while the device's screen is off. For example, device 2000 may vibrate to alert the user that navigation directions have been downloaded and are available for viewing. In this way, device 2000 can relieve the user from having to focus on the device while the user's input is being processed. Such techniques also allow device 2000 to immediately power down its touch screen, conserving battery power while its virtual assistant continues to perform tasks. 6. Example process for triggering a virtual assistant

[0258] 12 is a flow diagram illustrating a method 1200 of triggering a virtual assistant on an electronic device according to some embodiments. In some embodiments, the method 1200 is performed on one of the devices 104, 200, 400, 800, and 1600 (FIGS. 1, 2, 4, 8, and 16). In some embodiments, the method 1200 is performed on a device having a touch screen. In some embodiments, the method 1200 is performed on a device having a separate display screen and a touch-sensitive surface. The operations of the method 1200 are optionally combined and / or the order of some operations is optionally changed.

[0259] At block 1202, while the display of the electronic device is on, the device detects user input via an input component of the device. At block 1204, the device determines whether the input meets a predetermined condition. If the input meets the predetermined condition, processing continues to 1206, where the device samples audio input received via a microphone. If the input does not meet the predetermined condition, processing returns to block 1202, where the device can detect another user input.

[0260] In some embodiments, the predetermined condition is that the display screen of the device is on. The device determines whether the display is on by, for example, determining whether its backlight is on.

[0261] In some embodiments, the predetermined condition is device movement, such as lifting the device to a viewing position (as seen in FIGS. 8A and 8B). Whether device movement constitutes a lift movement, in some embodiments, is based on accelerometer readings over time. Whether device movement constitutes a lift movement, in some embodiments, is based on smoothness of accelerometer readings over time. Whether device movement constitutes a lift movement, in some embodiments, is based on sensing a minimum period of dwell time (e.g., a period of relative non-movement or stability).

[0262] In the exemplary lifting movement shown in FIGS. 8A-8B (as well as FIGS. 10A-10B), a user 802 lifts the device 800 from a lowered position in FIG. 8A to a raised position in FIG. 8B. The movement includes acceleration over time in one direction (e.g., vertically along the y-axis) and rotational acceleration over time in another direction (e.g., horizontal rotation about the x-axis), both of which may be determined by the acceleration sensor of the device 800. Optionally, the smoothness of such a lifting movement may be determined by comparing the fluctuations in the acceleration sensor readings over time. Optionally, the lifting movement ends when the user glances at the touch screen of the device in the raised position. The acceleration sensor readings may be used to determine a relatively stationary period when the user 802 looks at the display of the device 800. The device 800, in some embodiments, uses at least a portion of these accelerometer readings in determining whether a predetermined condition is met for sampling audio input via the microphone.

[0263] Additional details regarding techniques for determining a raise gesture in an electronic device are described, for example, in U.S. Provisional Patent Application No. 62 / 026,532, entitled "Raise Gesture Detection in a Device," filed July 18, 2014, the contents of which are incorporated herein by reference. For the sake of brevity, the contents of this application will not be repeated here.

[0264] In some embodiments, the predetermined condition is the activation of an input device, such as a mechanical button, a touch-sensitive button, a rotatable input device, etc. In some embodiments, the predetermined condition is a touch to a touch-sensitive surface or a touch screen (such as touch screen 804 shown in FIG. 8A).

[0265] Optionally, in block 1204, the device determines whether it is operating in a predefined mode (e.g., "do not disturb") in which its audio output is muted. If the device is operating in such a mode, processing returns to block 1202, which means that the device forgoes sampling the audio that triggers the virtual assistant.

[0266] In some embodiments, sampling the audio at block 1206 involves powering on a microphone of the device. In some examples, sampling the audio involves powering on additional circuitry to perform signal processing. In some examples, sampling the audio involves passing audio input received from the microphone to a software recognition algorithm executing on one or more processors of the device.

[0267] After sampling the audio input at block 1206, the device proceeds to block 1208, where it determines whether the sampled audio input includes a verbal trigger that triggers the virtual assistant. If the sampled audio input includes a verbal trigger, processing proceeds to block 1210, where the device triggers the virtual assistant session (e.g., by displaying the user interface 812 shown in FIG. 8C). In some embodiments, the verbal trigger is a predetermined phrase, such as "hey Siri."

[0268] If the sampled audio input does not include a verbal trigger, processing continues to block 1212, where the device determines whether to continue sampling the audio input. If the device does continue sampling the audio input, processing returns to block 1206, where audio sampling continues. If the device does stop sampling the audio input, processing continues to block 1214, where audio sampling stops.

[0269] In some embodiments, the device stops sampling audio input after the device's display turns off. In some embodiments, the device stops sampling audio input after sampling for a threshold duration, such as 8 seconds duration. In some embodiments, the device stops sampling audio after detecting activation of an input device, such as a mechanical button, a touch-sensitive button, a rotatable input device, etc. In some embodiments, the device stops sampling audio after detecting activation of an input device, such as a touch on a touch-sensitive surface or touch screen.

[0270] In some embodiments, the device stops sampling the audio input after an intervening input is received, such as a user input invoking another application installed on the device. In some embodiments, the device stops sampling the audio based on an amount of descent of the electronic device during the time interval as determined from an accelerometer of the electronic device. In some embodiments, the device stops sampling the audio based on a smoothness of the descent of the electronic device during the time interval as determined from an accelerometer of the device. In some embodiments, the device stops sampling the audio input after identifying an audio endpoint in the audio input. Exemplary endpoints include those that indicate that the received audio input is too quiet to be near-field speech (as opposed to background noise), is unlikely to be a human voice, a halt in the user's verbal input, etc.

[0271] In some embodiments, the device stops sampling the audio at block 1214 by powering off a microphone on the device. In some embodiments, the audio sampling stops by powering off associated circuitry that performs signal processing for the microphone. In some embodiments, the audio sampling stops by not processing the audio input received from the microphone with a software recognition algorithm. Optionally, in block 1214, the device provides one or more of a haptic, audio, and visual output indicating the end of the audio sampling.

[0272] It should be understood that the particular order described for the operations in FIG. 12 is merely exemplary, and the order described is not intended to indicate that the operations are the only order in which they may be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein for other methods described herein (e.g., methods 1300 and 1500) are also applicable to method 1200 described above for FIG. 12 in a similar manner. For example, a device performing method 1200 can also perform method 1500 (FIG. 15) to overcome erroneous cancellation by a user of a virtual assistant. For the sake of brevity, these details will not be repeated here.

[0273] 13 is a flow diagram illustrating a method 1300 of triggering a virtual assistant on an electronic device according to some embodiments. In some embodiments, the method 1300 is performed on one of the devices 104, 200, 400, 900, and 1700 (FIGS. 1, 2, 4, 8, and 17). In some embodiments, the method 1300 is performed on a device having a touch screen. In some embodiments, the method 1300 is performed on a device having a separate display screen and a touch-sensitive surface. The operations in the method 1300 are optionally combined and / or the order of some operations is optionally changed.

[0274] At block 1302, the electronic device detects a software event. At block 1304, the device determines whether the software event satisfies a predetermined condition. If the software event satisfies the predetermined condition, processing proceeds to block 1306, where the device begins sampling audio input received via the microphone. If the software event does not satisfy the predetermined condition, processing returns to block 1302, where the device can detect another software event.

[0275] In the illustration of FIG. 9A , a notification 906 of an incoming text message 912 satisfies a predetermined condition that causes the device 900 to sample the audio input. In some embodiments, the predetermined condition is that the software event is an operating system event triggered by the device's operating system. Exemplary software events triggered by the operating system include an alert for a hardware component of the device 900, such as an alert that the device has been connected to a computer via a cable. In some embodiments, the predetermined condition is that the software event is an application-based event triggered by an application running on the device. Exemplary software events triggered by an application of the device include calendar reminders, task reminders, and the like. Exemplary software events that may be considered triggered by the operating system and / or applications of the device 900 include clock events, such as an alarm or timer expiration.

[0276] In some embodiments, the predetermined condition is that the software event is an event triggered by an external source. Exemplary software events triggered by an external source include an incoming call (e.g., a voice or video based, cellular or WiFi based call), a message (e.g., an email, a text message SMS, a multimedia message, iMessage, etc.), a calendar invitation, etc. Exemplary software events triggered by an external source also include application-based notifications and alerts, such as an alert indicating that newly available information is available on a website or service.

[0277] In some embodiments, the predetermined condition is that a software event indicates that a virtual assistant session is active on the device.

[0278] Optionally, in block 1304, the device determines whether it is operating in a predefined mode (e.g., "do not disturb") in which its audio output is muted. If the device is operating in such a mode, processing returns to block 1302, which means that the device forgoes sampling the audio that triggers the virtual assistant.

[0279] Optionally, in block 1304, the device determines that its display screen is on. The device determines whether the display is on, for example, by determining whether its backlight is on. If the device's display is off, the process returns to block 1302, which means that the device forgoes sampling the audio that triggers the virtual assistant.

[0280] After sampling the audio input at block 1306, the device proceeds to block 1308, where it determines whether the sampled audio input includes a verbal trigger that triggers the virtual assistant. If the sampled audio input includes a verbal trigger, processing proceeds to block 1310, where the device triggers a virtual assistant session and, optionally, performs a task based on the user's verbal input (as seen in the example illustrations of FIG. 9B and FIG. 9C). In some embodiments, the verbal trigger is a predetermined phrase, such as "hey Siri."

[0281] If the sampled audio input does not include a verbal trigger, processing continues to block 1312, where the device determines whether to continue sampling the audio input. If the device does continue sampling the audio input, processing returns to block 1306, where audio sampling continues. If the device does stop sampling the audio input, processing continues to block 1314, where audio sampling stops.

[0282] In some embodiments, the device stops sampling audio input after the device's display turns off. In some embodiments, the device stops sampling audio input after sampling for a threshold duration, such as 8 seconds duration. In some embodiments, the device stops sampling audio after detecting activation of an input device, such as a mechanical button, a touch-sensitive button, a rotatable input device, etc. In some embodiments, the device stops sampling audio after detecting activation of an input device, such as a touch on a touch-sensitive surface or touch screen.

[0283] In some embodiments, the device stops sampling the audio input after an intervening input is received, such as a user input invoking another application installed on the device. In some embodiments, the device stops sampling the audio based on an amount of descent of the electronic device during the time interval as determined from an accelerometer of the electronic device. In some embodiments, the device stops sampling the audio based on a smoothness of the descent of the electronic device during the time interval as determined from an accelerometer of the device. In some embodiments, the device stops sampling the audio input after identifying an audio endpoint in the audio input. Exemplary endpoints include those that indicate that the received audio input is too quiet to be near-field speech (as opposed to background noise), is unlikely to be a human voice, a halt in the user's verbal input, etc.

[0284] In some embodiments, the device stops sampling the audio at block 1314 by powering off a microphone on the device. In some embodiments, the audio sampling stops by powering off associated circuitry that performs signal processing for the microphone. In some embodiments, the audio sampling stops by not processing the audio input received from the microphone with a software recognition algorithm. Optionally, in block 1314, the device provides one or more of a haptic, audio, and visual output indicating the end of the audio sampling.

[0285] It should be understood that the particular order described for the operations in FIG. 13 is merely exemplary, and the order described is not intended to indicate that the operations are the only order in which they may be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein for other methods described herein (e.g., methods 1200 and 1500) are also applicable to method 1300 described above for FIG. 13 in a similar manner. For example, a device performing method 1300 can also perform method 1500 (FIG. 15) to overcome erroneous cancellation by a user of a virtual assistant. For the sake of brevity, these details will not be repeated here.

[0286] 14 is a flow diagram illustrating a method 1400 of triggering a virtual assistant on an electronic device according to some embodiments. In some embodiments, the method 1400 is performed on one of the devices 104, 200, 400, 1000, and 1800 (FIGS. 1, 2, 4, 10, and 18). In some embodiments, the method 1400 is performed on a device having a touch screen. In some embodiments, the method 1400 is performed on a device having a separate display screen and a touch-sensitive surface. Operations in the method 1400 are optionally combined and / or the order of some operations is optionally changed.

[0287] At block 1402, the device detects user input via an input component of the device. At block 1404, the device determines whether the input meets a predetermined condition. If the input meets the predetermined condition, processing continues to 1406, where the device samples audio input received via a microphone. If the input does not meet the predetermined condition, processing returns to block 1402, where the device can detect another user input.

[0288] In some embodiments, the predetermined condition is that the device receives user input while its display is off. The device determines if its display is off, for example, by determining if its backlight is on.

[0289] In some embodiments, the predetermined condition is a movement of the device, such as lifting the device to a viewing position (e.g., as seen in FIGS. 8A-8B and 10A-10B). In some embodiments, the predetermined condition is based on an amount of lifting of the electronic device during a time interval, as determined from an accelerometer of the electronic device. In some embodiments, the predetermined condition is based on a smoothness of lifting of the electronic device during a time interval, as determined from an accelerometer of the electronic device. In some embodiments, the predetermined condition includes a minimum period of dwell time in a position following lifting of the electronic device.

[0290] In some embodiments, the predetermined condition is the activation of an input device, such as a mechanical button, a touch-sensitive button, a rotatable input device, etc. In some examples, the predetermined condition is a touch on a touch-sensitive surface or a touch screen.

[0291] Optionally, in block 1404, the device determines whether it is operating in a predefined mode (e.g., "do not disturb") in which its audio output is muted. If the device is operating in such a mode, processing returns to block 1402, which means that the device forgoes sampling the audio that triggers the virtual assistant.

[0292] In some embodiments, sampling the audio at block 1406 involves powering on a microphone of the device. In some embodiments, sampling the audio involves powering on additional circuitry to perform signal processing. In some embodiments, sampling the audio involves passing audio input received from the microphone to a software recognition algorithm running on one or more processors of the device.

[0293] In some embodiments, the device proceeds from block 1402 (wherein input is detected) to block 1406 (wherein audio sampling occurs) without turning on its display (e.g., by not turning on a backlight for the display). Optionally, the device provides at least one of haptic and audio output at block 1406 without powering on its display. In some embodiments, the device provides visual output from a light source other than its display when it begins sampling audio input.

[0294] After sampling the audio input at block 1406, the device proceeds to block 1408, where it determines whether the sampled audio input includes a verbal trigger that triggers the virtual assistant. If the sampled audio input includes a verbal trigger, processing proceeds to block 1410, where the device triggers the virtual assistant session. In some embodiments, the verbal trigger is a predetermined phrase, such as "hey Siri."

[0295] In some embodiments, the device proceeds from block 1402 (where an input is detected) to block 1408 (where a virtual assistant is triggered) with the device's display turned off (e.g., by not turning on the display backlight). In some embodiments, the device provides at least one of haptic and audio output when triggering the virtual assistant at block 1410. In some embodiments, the device provides visual output from a light source other than its display when triggering the virtual assistant at block 1410.

[0296] If the sampled audio input does not include a verbal trigger, processing continues to block 1412, where the device determines whether to continue sampling the audio input. If the device does continue sampling the audio input, processing returns to block 1406, where audio sampling continues. If the device does stop sampling the audio input, processing continues to block 1414, where audio sampling stops.

[0297] In some embodiments, the device stops sampling the audio input after sampling for a threshold duration, such as 8 seconds duration. In some embodiments, the device stops sampling audio after detecting activation of an input device, such as a mechanical button, a touch-sensitive button, a rotatable input device, etc. In some embodiments, the device stops sampling audio after detecting activation of an input device, such as a touch on a touch-sensitive surface or touch screen.

[0298] In some embodiments, the device stops sampling the audio input after an intervening input is received, such as a user input invoking another application installed on the device. In some embodiments, the device stops sampling the audio based on an amount of descent of the electronic device during the time interval as determined from an accelerometer of the electronic device. In some embodiments, the device stops sampling the audio based on a smoothness of the descent of the electronic device during the time interval as determined from an accelerometer of the device. In some embodiments, the device stops sampling the audio input after identifying an audio endpoint in the audio input. Exemplary endpoints include those that indicate that the received audio input is too quiet to be near-field speech (as opposed to background noise), is unlikely to be a human voice, a halt in the user's verbal input, etc.

[0299] In some embodiments, the device ceases sampling the audio at block 1414 by powering off the device's microphone. In some embodiments, audio sampling ceases by powering off associated circuitry that performs signal processing for the microphone. In some embodiments, audio sampling ceases by not processing audio input received from the microphone with software recognition algorithms.

[0300] Optionally, the device provides at least one of haptic and audio output when it stops sampling the audio input without turning on its display, In some embodiments the device provides visual output from a light source other than its display when it stops sampling the audio input.

[0301] It should be understood that the particular order described for the operations in FIG. 14 is merely exemplary, and the order described is not intended to indicate that the operations are the only order in which they may be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein for other methods described herein (e.g., methods 1300 and 1500) are also applicable to method 1400 described above for FIG. 14 in a similar manner. For example, a device performing method 1400 can also perform method 1500 (FIG. 15) to overcome erroneous cancellation by a user of a virtual assistant. For the sake of brevity, these details will not be repeated here.

[0302] 15 is a flow diagram illustrating a method 1500 of triggering a virtual assistant on an electronic device according to some embodiments. In some embodiments, the method 1500 is performed on one of the devices 104, 200, 400, 1100, and 1900 (FIGS. 1, 2, 11, and 19). In some embodiments, the method 1500 is performed on a device having a touch screen. In some embodiments, the method 1500 is performed on a device having a separate display screen and a touch-sensitive surface. The operations in the method 1500 are optionally combined and / or the order of some operations is optionally changed.

[0303] In block 1502, the electronic device samples the audio input to determine whether the audio input represents an executable task, e.g., a user request provided in spoken natural language. While sampling the audio input, the device can provide a UI showing the sampling, such as screen 1106 of FIG.

[0304] In some embodiments, sampling of the audio input occurs when the display of the device is powered on, e.g., when the backlight of the display is illuminated. In some embodiments, the device samples the audio input in response to a movement that meets a predetermined condition, such as lifting the device to a viewing position (as seen in FIGS. 8A-8B, 10A-10B, and 11A). In some embodiments, the predetermined condition is based on an amount of lifting of the electronic device during a time interval, as determined from an accelerometer of the electronic device. In some embodiments, the predetermined condition is based on a smoothness of lifting of the electronic device during a time interval, as determined from an accelerometer of the electronic device. In some embodiments, the predetermined condition includes a minimum period of dwell time in a position following lifting of the electronic device.

[0305] In some embodiments, the predetermined condition is the activation of an input device, such as a mechanical button, a touch-sensitive button, a rotatable input device, etc. In some embodiments, the predetermined condition is a touch on a touch-sensitive surface or a touch screen.

[0306] At block 1504, while sampling the audio input, the electronic device receives an instruction to stop sampling the audio input. In some embodiments, the device is adapted to receive an instruction to stop sampling the audio input from a user of the device. The instruction may be received as a verbal instruction, for example, via a microphone of the device. The instruction may also be received by detecting activation of an input device, such as, for example, a touch on a touch-sensitive surface or touch screen, such as, for example, a touch on affordance 1108 of FIG. 11A. Additionally, the instruction may be received by detecting activation of, for example, a mechanical button, a touch-sensitive button, a rotatable input device, or the like.

[0307] In some embodiments, the device is adapted to receive an instruction to cease sampling the audio input from another electronic device external to the device, the instruction may be received through a communication medium such as cellular communication, Bluetooth communication, WiFi communication, etc.

[0308] In some embodiments, at block 1504, the device prompts the user for verbal input. In some embodiments, at block 1504, the device determines whether the audio input represents a request, and pursuant to determining whether the audio input represents a task, the device determines a user intent based at least in part on the audio input, identifies and performs the task based on the user intent, and further provides at least one of a haptic, audio, and visual output representative of the performance of the task.

[0309] In block 1506, in response to the command to stop sampling the audio input, the device provides an output acknowledgment. This acknowledgment may, for example, report to the user that the virtual assistant has been canceled. In some embodiments, the acknowledgment involves one or more of haptic, audio, and visual output. In some embodiments, the acknowledgment involves turning off the display as seen in FIG. 11B.

[0310] In some embodiments, after receiving the command to stop sampling the audio input at block 1504 and after providing an acknowledgment at block 1506, processing proceeds to block 1508, where the device continues to sample the audio input for a limited period of time (after the device stops sampling the audio input) and determines whether the audio input includes a user task.

[0311] In some embodiments, the device stops sampling audio input after the device's display turns off. In some embodiments, the device stops sampling audio input after continuously sampling for a threshold duration, such as 8 seconds duration. In some embodiments, the device stops sampling audio after detecting activation of an input device, such as a mechanical button, a touch-sensitive button, a rotatable input device, etc. In some embodiments, the device stops sampling audio after detecting activation of an input device, such as a touch on a touch-sensitive surface or touch screen.

[0312] In some embodiments, the device stops sampling the audio input after an intervening input is received, such as a user input invoking another application installed on the device. In some embodiments, the device stops sampling the audio based on an amount of descent of the electronic device during the time interval as determined from an accelerometer of the electronic device. In some embodiments, the device stops sampling the audio based on a smoothness of the descent of the electronic device during the time interval as determined from an accelerometer of the device. In some embodiments, the device stops sampling the audio input after identifying an audio endpoint in the audio input. Exemplary endpoints include those that indicate that the received audio input is too quiet to be near-field speech (as opposed to background noise), is unlikely to be a human voice, a halt in the user's verbal input, etc.

[0313] In some embodiments, after receiving the command to stop sampling the audio input 1504, the device determines whether the command to stop sampling the audio input originates from a user or an external device. Following a determination that the audio input originates from an external device, the device immediately stops sampling the audio input.

[0314] In some embodiments, the device stops sampling the audio at block 1506 by powering off a microphone of the device. In some embodiments, the audio sampling stops by powering off associated circuitry that performs signal processing for the microphone. In some embodiments, the audio sampling stops by not processing the audio input received from the microphone with a software recognition algorithm. Optionally, in block 1506, the device provides one or more of a haptic, audio, and visual output indicating the end of the audio sampling.

[0315] It should be understood that the particular order described for the operations in FIG. 15 is merely exemplary, and the order described is not intended to indicate that the operations are the only order in which they may be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein for other methods described herein (e.g., methods 1200, 1300, 1400, and 2100) are also applicable to method 1500 described above for FIG. 15 in a similar manner. For example, a device performing method 1500 can also perform method 1400 for triggering a virtual assistant session. For the sake of brevity, these details will not be repeated here.

[0316] FIG. 21 is a flow diagram illustrating a method 2100 of triggering a virtual assistant on an electronic device, according to some embodiments. In some embodiments, the method 2100 is performed on one of the devices 104, 200, 400, and 2000 (FIGS. 1, 2, 4, 8, and 20). In some embodiments, the method 2100 is performed on a device having a touch screen. In some embodiments, the method 2100 is performed on a device having a separate display screen and a touch-sensitive surface. The operations of the method 1200 are optionally combined and / or the order of some operations is optionally changed.

[0317] In block 2102, while a given service, such as a virtual assistant, is executed on the device, the device detects a user's natural language input, such as a spoken input. In block 2104, the device executes the user's input by identifying one or more computing tasks to be executed. In block 2106, while executing the one or more tasks, the device powers down its display screen (e.g., by turning off the screen backlight). In block 2108, after completing the one or more computing tasks, the device provides haptic and / or audio output.

[0318] It should be understood that the particular order described for the operations in FIG. 21 is merely exemplary, and the order described is not intended to indicate that the operations are the only order in which they may be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein for other methods described herein (e.g., methods 1200, 1300, 1400, and 1500) are also applicable in a similar manner to method 2100 described above for FIG. 21. For example, a device performing method 2100 can also perform method 1400 for triggering a virtual assistant session. For the sake of brevity, these details will not be repeated here.

[0319] Note that sampling of the audio input at blocks 1206, 1306, 1406, and 1502 (FIGS. 12, 13, 14, and 15) may occur regardless of whether the electronic device is charging. However, such sampling of the audio input may depend on the remaining battery level of the device. In some embodiments, the device may sample the audio input consistent with the process described above, but may forgo sampling of the audio input when the remaining battery level is below a threshold.

[0320] According to some embodiments, FIG. 16 shows a functional block diagram of an electronic device 1600 configured according to the principles of the various described embodiments, including those described with reference to FIG. 8 and FIG. 12. The functional blocks of this device are optionally implemented by hardware, software, or a combination of hardware and software to carry out the principles of the various described embodiments. Those skilled in the art will understand that the functional blocks described in FIG. 16 are optionally combined or separated into sub-blocks to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein.

[0321] 16, electronic device 1600 includes a display unit 1602 configured to display a graphical user interface, an optional touch-sensitive surface unit 1604 for receiving contacts, a microphone unit 1606 for detecting audio input, an optional movement sensor unit 1608 for obtaining movement readings such as acceleration over a time interval, an optional input device unit 1610 such as a button, an optional feedback unit 1612 for providing audio, tactile and / or visual feedback, and a processing unit 1614 coupled to the above-mentioned units. In some embodiments, processing unit 1614 includes a display enablement unit 1616, an input detection unit 1618, an audio sampling unit 1620, an audio processing unit 1622, a virtual assistant service unit 1624, and optionally a feedback enablement unit 1626.

[0322] In some embodiments, the processing unit 1614 receives user input that satisfies a predetermined condition via the input device (e.g., using the display enablement unit 1616) while the display is on, and samples (e.g., using the audio sampling unit 1620) audio input received via the microphone (e.g., from the microphone unit 1606) in accordance with the receipt of the user input that satisfies the predetermined condition (e.g., using the input detection unit 1618), and determines whether the audio input includes an oral trigger (e.g., using the audio processing unit 1622). In accordance with the determination (e.g., using the audio processing unit 1622) that the audio input includes an oral trigger, (e.g., using the virtual assistant service unit 1624) is configured to trigger a virtual assistant session.

[0323] In some embodiments, the predetermined condition is based on an amount of lifting of the electronic device during a time interval, determined from an accelerometer (e.g., the motion sensor unit 1608) of the electronic device. In some embodiments, the predetermined condition is based on a smoothness of lifting of the electronic device during a time interval, determined from an accelerometer (e.g., the motion sensor unit 1608) of the electronic device. In some embodiments, the predetermined condition includes a minimum period of dwell time at a position following lifting of the electronic device. In some embodiments, the predetermined condition includes detecting (e.g., the input detection unit 1618) an activation of a button (e.g., the input device unit 1610) of the electronic device.

[0324] In some embodiments, the predetermined condition includes detecting a touch input to a touch-sensitive surface of the electronic device (eg, touch-sensitive surface unit 1604).

[0325] In some embodiments, the processing unit is further configured to determine whether the display is on (e.g., with the display enabling unit 1616) and, pursuant to determining that the display is on, sampling of the audio input occurs (e.g., with the audio sampling unit 1620). In some embodiments, the processing unit is further configured to determine whether a backlight for the display is on (e.g., with the display enabling unit 1616) pursuant to determining that the display is on.

[0326] In some embodiments, the processing unit is further configured to cease sampling of the audio input after sampling of the audio input has occurred (eg, using the audio sampling unit 1620) for a predetermined duration.

[0327] In some embodiments, the user input satisfying the predetermined condition is a first user input satisfying a first predetermined condition, and the processing unit 1614 is further configured to receive (e.g., using the input detection unit 1618) a second user input satisfying a second predetermined condition while sampling the audio input (e.g., using the audio sampling unit 1620) following receipt of the first user input, and to cease sampling the audio input (e.g., using the audio sampling unit 1620) following receipt of the second user input satisfying the second predetermined condition.

[0328] In some embodiments, the second predetermined condition is based on an amount of descent of the electronic device during a time interval as determined from an accelerometer of the electronic device (eg, using the mobile sensor unit 1608).

[0329] In some embodiments, the first predetermined condition is based on the smoothness of a descent of the electronic device during a time interval, as determined from an accelerometer of the electronic device (eg, using the mobile sensor unit 1608).

[0330] In some embodiments, the second predetermined condition includes detecting activation of a button of the electronic device (e.g., with input detection unit 1618). In some embodiments, the second predetermined condition includes detecting a touch input on a touch-sensitive surface of the electronic device.

[0331] In some embodiments, the processing unit 1614 is further configured to determine (e.g., using the display enable unit 1616) whether the display is off while sampling the audio input upon receipt of user input satisfying a predetermined condition (e.g., using the audio sampling unit 1620), and to cease sampling the audio input (e.g., using the audio sampling unit 1620) upon determining that the display is off.

[0332] In some embodiments, the processing unit 1614 is further configured to identify (e.g., using the audio processing unit 1622) an audio end point within the audio input while sampling the audio input following receipt of user input satisfying a predetermined condition (e.g., using the audio sampling unit 1620), and to cease sampling of the audio input (e.g., using the audio sampling unit 1620) in response to identifying the audio end point.

[0333] In some embodiments, the processing unit 1614 is further configured to generate a haptic output (eg, using the feedback enablement unit 1626 and / or the feedback unit 1612) following the cessation of sampling of the audio input.

[0334] In some embodiments, the processing unit 1614 is further configured to prompt the user for verbal input (eg, using the display unit 1602 and / or the feedback unit 1612) following the triggering of a virtual assistant session (eg, using the virtual assistant service unit 1624) (eg, using the display enablement unit 1616 and / or the feedback enablement unit 1626).

[0335] In some embodiments, the audio input includes a verbal trigger and additional input, and the processing unit 1614 determines the user intent based on at least the additional input in the audio input according to the trigger of the virtual assistant session, and (e.g., using the virtual assistant service unit 1624) performs a task associated with the user intent. It is further configured.

[0336] In some embodiments, the processing unit 1614 is further configured to forgo sampling of the audio input (e.g., using the audio sampling unit 1620) even after receiving user input that satisfies a predetermined condition while the display is on in accordance with a determination (e.g., using the virtual assistant service unit 1624) that the electronic device is operating in a predetermined mode.

[0337] In some embodiments operating in the predetermined mode includes muting an audio output of the electronic device, hi some embodiments the verbal trigger includes a predetermined phrase.

[0338] The operations described above with reference to FIG. 12 are optionally implemented by components shown in FIG. 1, FIG. 2, FIG. 7A-7B, or FIG. 16. For example, the sample audio operation 1202 is optionally performed by the audio circuit 210 (FIG. 2). When audio having a verbal trigger is detected, the device provides a corresponding output using the display controller 256 (FIG. 2) and / or the haptic feedback controller 261 (FIG. 2). It will be clear to one skilled in the art how other processes are implemented based on the components shown in FIG. 1, FIG. 2, FIG. 7A-7B, or FIG. 16.

[0339] Those skilled in the art will understand that the functional blocks described in FIG. 16 may be optionally combined or separated into sub-blocks to implement the principles of the various embodiments described. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For example, the units 1602-1612 may have associated "controller" units operatively coupled with the processing unit 1614 that enables the respective units and operations. These controller units are not shown separately in FIG. 16, but are understood to be within the understanding of those skilled in the art designing devices having the units 1602-1612, such as the device 1600. Thus, the description herein optionally supports combination, separation and / or further definition of the functional blocks described herein.

[0340] According to some embodiments, FIG. 17 illustrates a functional block diagram of an electronic device 1700 configured according to the principles of the various described embodiments, including those described with reference to FIG. 9 and FIG. 13. The functional blocks of this device are optionally implemented by hardware, software, or a combination of hardware and software to carry out the principles of the various described embodiments. Those skilled in the art will understand that the functional blocks described in FIG. 17 are optionally combined or separated into sub-blocks to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein.

[0341] 17, electronic device 1700 includes a display unit 1702 configured to display a graphical user interface, an optional touch-sensitive surface unit 1704 for receiving contacts, a microphone unit 1706 for detecting audio input, an optional movement sensor unit 1708 for obtaining movement readings, such as acceleration, over a time interval, an optional input device unit 1710, such as a button, an optional feedback unit 1712 for providing audio, tactile and / or visual feedback, and a processing unit 1714 coupled to each of the above-mentioned units. In some embodiments, processing unit 1714 includes a display enablement unit 1716, a software event detection unit 1728, an input detection unit 1718, an audio sampling unit 1720, an audio processing unit 1722, a virtual assistant service unit 1724, and optionally a feedback enablement unit 1726.

[0342] In some embodiments, the processing unit 1714 detects a software event that satisfies a predetermined condition (e.g., using the software event detection unit 1728), and in accordance with a determination that a software event that satisfies the condition has been detected, samples the audio input received via the microphone (e.g., the microphone unit 1706) (e.g., using the audio sampling unit 1620), determines whether the audio input received via the microphone includes an oral trigger (e.g., using the audio processing unit 1722), and in accordance with a determination that the audio input includes an oral trigger, the virtual assistant session (e.g., using the virtual assistant service unit 1724) is configured to be triggered.

[0343] In some embodiments, the predetermined condition is that the software event is an event triggered by an application of the electronic device. In some embodiments, the predetermined condition includes that the software event represents a notification. In some embodiments, the predetermined condition includes that the software event represents a calendar notification.

[0344] In some embodiments, the predetermined condition includes the software event representing an incoming calendar invitation from an external device. In some embodiments, the predetermined condition includes the software event representing an incoming transmission triggered from an external device. In some embodiments, the predetermined condition includes the software event representing an incoming phone call from an external device. In some embodiments, the predetermined condition includes the software event representing an incoming text or multimedia message from an external device.

[0345] In some embodiments, the processing unit 1714 is further configured to initiate a response addressed to the external device (eg, using the virtual assistant service unit 1624) following the trigger of the virtual assistant session. An exemplary response is a reply to an incoming message, such as the text message 916 of FIG. 9C. Other exemplary responses include an email message, a multimedia message, a calendar invitation response, and the like. In the example of FIG. 9C, the response text message 916 was automatically sent by the device 900 based on the user's instruction 910. Optionally, after initiating the message 916 (eg, entering recipient information and content in the message), the device can prompt the user for a final confirmation before sending the message to the recipient.

[0346] In some embodiments, the processing unit 1714 is further configured to determine whether the display is on (e.g., using the display enable unit 1716) and, pursuant to determining that the display is on, sampling of the audio input occurs (e.g., using the audio sampling unit 1720).

[0347] In some embodiments, the processing unit 1714 is further configured to determine (e.g., with the display enablement unit 1716) whether a backlight for the display is on following the determination of whether the display is on (e.g., with the display enablement unit 1716).

[0348] In some embodiments, the processing unit 1714 is further configured to cease sampling of the audio input (eg, using the audio sampling unit 1720) after sampling of the audio input has occurred for a predetermined duration.

[0349] In some embodiments, the user input satisfying the predetermined condition is a first user input satisfying a first predetermined condition, and the processing unit 1714 is further configured to receive (e.g., using the input detection unit 1718) a second user input satisfying a second predetermined condition while sampling the audio input (e.g., using the audio sampling unit 1720) following receipt of the first user input, and to cease sampling the audio input (e.g., using the audio sampling unit 1720) following receipt of the second user input satisfying the second predetermined condition.

[0350] In some embodiments, the second predetermined condition is based on an amount of descent of the electronic device during the time interval as determined from an accelerometer of the electronic device (e.g., using the mobile sensor unit 1708). In some embodiments, the first predetermined condition is based on a smoothness of the descent of the electronic device during the time interval as determined from an accelerometer of the electronic device (e.g., using the mobile sensor unit 1708).

[0351] In some embodiments, the second predetermined condition includes detecting (e.g., with input detection unit 1718) the activation of a button of the electronic device (e.g., with input device unit 1710). In some embodiments, the second predetermined condition includes detecting a touch input on a touch-sensitive surface of the electronic device (e.g., with touch-sensitive surface unit 1704).

[0352] In some embodiments, the processing unit 1714 is further configured to determine (e.g., using the display enabling unit 1716) whether the display is off while sampling the audio input (e.g., using the audio sampling unit 1720) upon receipt of user input satisfying a predetermined condition, and to cease sampling the audio input (e.g., using the audio sampling unit 1720) upon determining that the display is off.

[0353] In some embodiments, the processing unit 1714 is further configured to, upon receipt of user input satisfying a predetermined condition, identify an audio end point within the audio input (e.g., using the audio processing unit 1722) while sampling the audio input (e.g., using the audio sampling unit 1720), and, upon identifying the audio end point, cease sampling of the audio input (e.g., using the audio sampling unit 1720).

[0354] In some embodiments, the processing unit 1714 generates a haptic output (using the feedback unit 1712) following the cessation of sampling of the audio input (eg, using the feedback enablement unit 1726).

[0355] In some embodiments, the processing unit 1714 prompts the user for verbal input (eg, using the display unit 1702 and / or the feedback unit 1712) in response to a trigger of the virtual assistant session (eg, using the display enablement unit 1716 and / or the feedback enablement unit 1726).

[0356] In some embodiments, the audio input includes a verbal trigger and additional input, and the processing unit 1714 determines the user intent (e.g., using the virtual assistant service unit 1724) based on at least the additional input in the audio input according to the trigger of the virtual assistant session (e.g., using the virtual assistant service unit 1724), and performs a task associated with the user intent (e.g., using the virtual assistant service unit 1724).

[0357] In some embodiments, the processing unit 1714, pursuant to a determination (e.g., using the virtual assistant service unit 1724) that the electronic device is operating in a predetermined mode, even after receiving a user input (e.g., using the input detection unit 1718) that satisfies a predetermined condition while the display is on, (e.g., using the audio sampling unit 1720). Further configured to forgo sampling of the audio input.

[0358] In some embodiments, operating in the predefined mode includes muting an audio output of the electronic device. In some embodiments, the verbal trigger includes a predefined phrase, such as "hey Siri."

[0359] The operations described above with reference to FIG. 13 are optionally performed by components shown in FIG. 1, FIG. 2, FIG. 7A-7B, or FIG. 17. For example, the sample audio operation 1202 is optionally performed by the audio circuit 210 (FIG. 2). When audio having a verbal trigger is detected, the device provides a corresponding output using the display controller 256 (FIG. 2) and / or the haptic feedback controller 261 (FIG. 2). It will be clear to one skilled in the art how other processes are implemented based on the components shown in FIG. 1, FIG. 2, FIG. 7A-7B, or FIG. 17.

[0360] Those skilled in the art will understand that the functional blocks described in FIG. 17 may be optionally combined or separated into sub-blocks to implement the principles of the various embodiments described. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For example, units 1702-1712 may have associated "controller" units operatively coupled with the processing unit 1714 that enables the respective units and operations. These controller units are not shown separately in FIG. 17 but are understood to be within the understanding of those skilled in the art designing devices having units 1702-1712, such as device 1700. Thus, the description herein optionally supports combination, separation and / or further definition of the functional blocks described herein.

[0361] According to some embodiments, FIG. 18 illustrates a functional block diagram of an electronic device 1800 configured according to the principles of the various described embodiments, including those described with reference to FIG. 10 and FIG. 14. The functional blocks of this device are optionally implemented by hardware, software, or a combination of hardware and software to carry out the principles of the various described embodiments. Those skilled in the art will appreciate that the functional blocks described in FIG. 18 are optionally combined or separated into sub-blocks to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein.

[0362] 18, the electronic device 1800 includes a display unit 1802 configured to display a graphical user interface, an optional touch-sensitive surface unit 1804 for receiving contacts, a microphone unit 1806 for detecting audio input, an optional movement sensor unit 1808 for obtaining movement readings such as acceleration over a time interval, an optional input device unit 1810 such as a button, an optional feedback unit 1812 for providing audio, tactile and / or visual feedback, and a processing unit 1814 coupled to each of the above-mentioned units. In some embodiments, the processing unit 1814 includes a display enablement unit 1816, an input detection unit 1818, an audio sampling unit 1820, an audio processing unit 1822, a virtual assistant service unit 1824, and optionally a feedback enablement unit 1826.

[0363] In some embodiments, the processing unit 1814 receives a user input that satisfies a predetermined condition via an input device (e.g., using the input detection unit 1818), samples the audio input received via the microphone (e.g., using the audio sampling unit 1820) in accordance with the receipt of the user input that satisfies the predetermined condition, determines whether the audio input includes an oral trigger (e.g., using the audio processing unit 1822), and determines that the audio input includes an oral trigger. Trigger a virtual assistant session (e.g., using the virtual assistant service unit 1824) and generate a haptic output (e.g., using the feedback unit 1826).

[0364] In some embodiments, the processing unit 1814 is further configured to receive user input (eg, using the input detection unit 1818) while the display is off, upon receipt of user input that satisfies a predetermined condition.

[0365] In some embodiments, the processing unit 1814 is further configured to trigger the virtual assistant session (eg, using the virtual assistant service unit 1824) without turning on the backlight of the display in accordance with the trigger of the virtual assistant session.

[0366] In some embodiments, the predetermined condition is based on an amount of lifting of the electronic device during the time interval, as determined from an accelerometer of the electronic device (e.g., using the mobile unit sensor 1808). In some embodiments, the predetermined condition is based on a smoothness of lifting of the electronic device during the time interval, as determined from an accelerometer of the electronic device (e.g., using the mobile sensor unit 1808). In some embodiments, the predetermined condition includes a minimum period of dwell time at a position following lifting of the electronic device.

[0367] In some embodiments, the predetermined condition includes detecting activation of a button of the electronic device (e.g., with input detection unit 1818). In some embodiments, the predetermined condition includes detecting a touch input to a touch-sensitive surface of the electronic device.

[0368] In some embodiments, the processing unit 1814 is further configured to cease sampling of the audio input (eg, using the audio sampling unit 1820) after sampling of the audio input has occurred for a predetermined duration.

[0369] In some embodiments, the user input satisfying the predetermined condition is a first user input satisfying a first predetermined condition, and the processing unit 1814 is further configured to receive (e.g., using the input detection unit 1818) a second user input satisfying a second predetermined condition while sampling the audio input (e.g., using the audio sampling unit 1820) following receipt of the first user input, and to cease sampling the audio input (e.g., using the audio sampling unit 1820) following receipt of the second user input satisfying the second predetermined condition.

[0370] In some embodiments, the second predetermined condition is based on an amount of descent of the electronic device during the time interval, as determined from an accelerometer of the electronic device (e.g., with the mobile unit 1808). In some embodiments, the first predetermined condition is based on a smoothness of the descent of the electronic device during the time interval, as determined from an accelerometer of the electronic device (e.g., with the mobile unit 1808).

[0371] In some embodiments, the second predetermined condition includes detecting (e.g., with input detection unit 1818) activation of a button of the electronic device (e.g., with input device unit 1810). In some embodiments, the second predetermined condition includes detecting (e.g., with input detection unit 1818) a touch input on a touch-sensitive surface of the electronic device (e.g., with touch-sensitive surface unit 1804).

[0372] In some embodiments, the processing unit 1814 is further configured to, pursuant to receipt of user input satisfying a predetermined condition, identify an audio end point within the audio input (e.g., using the audio processing unit 1822) while sampling the audio input (e.g., using the audio sampling unit 1820), and, pursuant to identifying the audio end point, cease sampling of the audio input (e.g., using the audio sampling unit 1820).

[0373] In some embodiments, the processing unit 1814 is further configured to provide an output that occurs (using the feedback unit 1826) following the cessation of sampling of the audio input (e.g., using the audio sampling unit 1820). In some embodiments, the processing unit 1814 is further configured to prompt the user (e.g., using the display unit 1802 and / or the feedback unit 1812) for verbal input (e.g., using the display enablement unit 1816 and / or the feedback enablement unit 1826) following the triggering of a virtual assistant session (e.g., using the virtual assistant service unit 1824).

[0374] In some embodiments, the audio input includes a verbal trigger and additional input, and the processing unit 1814 determines the user intent (eg, using the virtual assistant service unit 1824) based on at least the additional input in the audio input in accordance with the trigger of the virtual assistant session (eg, using the virtual assistant service unit 1824), and performs a task associated with the user intent (eg, using the virtual assistant service unit 1824).

[0375] In some embodiments, processing unit 1814 is further configured, upon determining that the electronic device is operating in a predetermined mode, to forgo sampling of the audio input (e.g., with audio sampling unit 1820), even after receiving (e.g., with input detection unit 1818) a user input (e.g., with touch-sensitive surface unit 1804 or input device unit 1810) that satisfies a predetermined condition while the display is on.

[0376] In some embodiments, operating in the predefined mode includes muting an audio output of the electronic device. In some embodiments, the verbal trigger includes a predefined phrase, such as "hey Siri."

[0377] The operations described above with reference to FIG. 14 are optionally implemented by components shown in FIG. 1, FIG. 2, FIG. 7A-7B, or FIG. 18. For example, the sample audio operation 1202 is optionally performed by the audio circuit 210 (FIG. 2). When audio having a verbal trigger is detected, the device provides a corresponding output using the display controller 256 (FIG. 2) and / or the haptic feedback controller 261 (FIG. 2). It will be clear to one skilled in the art how other processes are implemented based on the components shown in FIG. 1, FIG. 2, FIG. 7A-7B, or FIG. 18.

[0378] Those skilled in the art will understand that the functional blocks described in FIG. 19 may be optionally combined or separated into sub-blocks to implement the principles of the various embodiments described. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For example, units 1802-1812 may have associated "controller" units operatively coupled with the processing unit 1814 that enables the respective units and operations. These controller units are not shown separately in FIG. 18 but are understood to be within the understanding of those skilled in the art designing devices having units 1802-1812, such as device 1800. Thus, the description herein optionally supports combination, separation and / or further definition of the functional blocks described herein.

[0379] According to some embodiments, FIG. 19 illustrates a functional block diagram of an electronic device 1900 configured according to the principles of the various described embodiments, including those described with reference to FIG. 8 and FIG. 12. The functional blocks of this device are optionally implemented by hardware, software, or a combination of hardware and software to carry out the principles of the various described embodiments. Those skilled in the art will understand that the functional blocks described in FIG. 19 are optionally combined or separated into sub-blocks to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein.

[0380] 19, the electronic device 1900 includes a display unit 1902 configured to display a graphical user interface, an optional touch-sensitive surface unit 1904 for receiving contacts, a microphone unit 1906 for detecting audio input, an optional movement sensor unit 1908 for obtaining movement readings such as acceleration over a time interval, an optional input device unit 1910 such as a button, an optional feedback unit 1912 for generating audio, tactile and / or visual feedback, and a processing unit 1914 coupled to each of the above-mentioned units. In some embodiments, the processing unit 1914 includes a display enablement unit 1916, an input detection unit 1918, an audio sampling unit 1920, an audio processing unit 1922, a virtual assistant service unit 1924, and optionally a feedback enablement unit 1926.

[0381] In some embodiments, the processing unit 1914 is configured to sample audio input received via a microphone (e.g., with the audio sampling unit 1920), determine whether the audio input represents a task (e.g., with the audio processing unit 1922), receive an instruction (e.g., with the input detection unit 1918) to stop sampling the audio input while sampling the audio input (e.g., with the audio sampling unit 1920), provide an output (e.g., with the feedback unit 1926) indicating receipt of the received instruction to stop sampling the audio input, and, after providing this output, continue to sample the audio input (with the audio sampling unit 1920) for a duration, determine (e.g., with the audio processing unit 1922) whether the audio input comprises a task, and then stop sampling the audio input (e.g., with the audio sampling unit 1920).

[0382] In some embodiments, the processing unit 1914 is further configured to sample the audio input (e.g., using the audio sampling unit 1920) while the display is on in accordance with sampling the audio input (e.g., using the audio sampling unit 1920).

[0383] In some embodiments, the processing unit 1914 is further configured to generate a haptic output (e.g., using the feedback unit 1926) following providing an output (e.g., using the feedback unit 1912) indicating receipt of a received command to end sampling of the audio input.

[0384] In some embodiments, the processing unit 1914 is further configured to provide an audio output (e.g., using the feedback unit 1926) following providing an output (e.g., using the feedback unit 1912) indicating receipt of a received instruction to end sampling of the audio input.

[0385] In some embodiments, the processing unit 1914 is further configured to prompt the user for verbal input (e.g., using the display enablement unit 1916 and the display unit 1902, and / or using the display enablement unit 1926 and the feedback unit 1912) following sampling (e.g., using the audio sampling unit 1920) of the audio received via the microphone.

[0386] In some embodiments, the processing unit 1914 is further configured to determine a user intent (e.g., using the virtual assistant service unit 1924) based on at least a portion of the audio input in accordance with a determination (e.g., using the virtual assistant service unit 1924) that the audio input represents a task, identify and execute a task based on the user intent (e.g., using the virtual assistant service unit 1924), and generate a haptic output (e.g., using the feedback unit 1912) (e.g., using the feedback unit 1926).

[0387] In some embodiments, the processing unit 1914 is further configured to determine (e.g., using the input detection unit 1918) whether the received instruction to stop sampling the audio input is received from a user via the electronic device or from an external device, and following a determination that the received instruction is received from a user via the electronic device, continue sampling the audio input (e.g., using the audio sampling unit 1920), and a determination is made (e.g., using the audio processing unit 1922) whether the audio input includes a task for a certain duration after receiving the instruction to stop sampling; following a determination that the received instruction is received from an external device, the device stops sampling the audio input (e.g., using the audio sampling unit 1920) in response to the received instruction.

[0388] In some embodiments, the verbal trigger includes a predefined phrase, such as "hey Siri."

[0389] The operations described above with reference to FIG. 15 are optionally implemented by components shown in FIG. 1, FIG. 2, FIG. 7A-7B, or FIG. 19. For example, the sample audio operation 1202 is optionally performed by the audio circuit 210 (FIG. 2). When audio having a verbal trigger is detected, the device provides a corresponding output using the display controller 256 (FIG. 2) and / or the haptic feedback controller 261 (FIG. 2). It will be clear to one skilled in the art how other processes are implemented based on the components shown in FIG. 1, FIG. 2, FIG. 7A-7B, or FIG. 19.

[0390] Those skilled in the art will understand that the functional blocks described in FIG. 19 may be optionally combined or separated into sub-blocks to implement the principles of the various embodiments described. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For example, units 1902-1912 may have associated "controller" units operatively coupled with the processing unit 1914 that enables the respective units and operations. These controller units are not shown separately in FIG. 19, but are understood to be within the understanding of those skilled in the art designing devices having units 1902-1912, such as device 1900. Thus, the description herein optionally supports combination, separation and / or further definition of the functional blocks described herein.

[0391] Although the present disclosure and embodiments have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art, and such changes and modifications are to be understood as being included within the scope of the present disclosure and embodiments, as defined by the appended claims.

Claims

1. 1. A method for activating a virtual assistant, comprising: In an electronic device having a display and a microphone, receiving a user input comprising a movement of the electronic device while a display of the electronic device is in a first power state; in response to receiving the user input including the movement of the electronic device, changing the display of the electronic device from the first power state to a second power state, wherein a power consumption of the display in the first power state is less than a power consumption of the display in the second power state; responsive to receiving the user input including the movement of the electronic device, upon determining that the display of the electronic device has changed from the first power state to a second power state; Sampling audio input received via the microphone; and triggering a virtual assistant session in accordance with a determination that the voice input includes a verbal trigger.

2. The method of claim 1 , wherein receiving the user input including the movement of the electronic device includes detecting an amount of lifting of the electronic device in a time interval.

3. The method of claim 1 or 2, wherein the user input further comprises activating a button on the electronic device.

4. The method of claim 1 , wherein the user input further comprises a touch input on a touch-sensitive surface of the electronic device.

5. 5. The method of claim 1, wherein changing the display of the electronic device from the first power state to the second power state comprises changing a brightness of a backlight of the display.

6. The method of any one of claims 1 to 5, further comprising stopping the virtual assistant session after sampling of the voice input at a predetermined time.

7. The method of claim 6, further comprising providing a haptic output following the stopping of the virtual assistant session.

8. 8. The method of claim 1, further comprising: stopping the virtual assistant session in accordance with a determination that the display of the electronic device has changed from the second power state to the first power state.

9. The method according to any one of claims 1 to 8, further comprising providing a prompt for verbal input to the user in response to a trigger of the virtual assistant session.

10. According to the trigger of the virtual assistant session, determining a user's intent based on at least the additional input in the speech input; performing a task associated with the user's intent; and 10. The method of claim 1, further comprising:

11. A computer program product for causing a computer to carry out the method according to any one of claims 1 to 10.

12. A memory storing a computer program according to claim 11; one or more processors capable of executing the computer programs stored in the memory; 1. An electronic device comprising:

13. An electronic device comprising means for carrying out the method according to any one of claims 1 to 10.

Images