Search operation in various user interfaces

Abstract

Systems and processes are provided for providing and updating a user interface for a search function. One example process includes, at a computer system comprising a touch-sensitive display and one or more input devices: receiving, via the one or more input devices, a request to display a user interface; in response to receiving the request to display the user interface, displaying, via the touch-sensitive display, the user interface comprising an affordance comprising an indication of a number of pages of the user interface; and while displaying the user interface: detecting an occurrence of a predetermined condition; and in response to detecting the occurrence of the predetermined condition, updating the affordance to comprise an indication of a search function.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application 63 / 340,190, filed May 10, 2022, and entitled “SEARCH OPERATIONS IN VARIOUS USER INTERFACES,” and U.S. Patent Application 17 / 948,440, filed September 20, 2022, and entitled “SEARCH OPERATIONS IN VARIOUS USER INTERFACES.” The contents of each of these patent applications are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0003] This application relates generally to search user interfaces, and more specifically to accessing and operating a user interface from various user interfaces of an electronic device. BACKGROUND

[0004] Search functionality and user interfaces can allow users to find new topics, applications, and other objects of interest and interact with them. However, accessing search functionality and user interfaces from some user interfaces on an electronic device can be cumbersome. Thus, it is advantageous to provide efficient methods of accessing search functionality and its associated interfaces. SUMMARY

[0005] An example method is disclosed herein. An example method includes, at a computer system that includes a touch-sensitive display and one or more input devices: receiving, via the one or more input devices, a request to display a user interface; in response to receiving the request to display the user interface, displaying, via the touch-sensitive display, the user interface that includes an affordance, the affordance including an indication of a number of pages of the user interface; and while displaying the user interface: detecting an occurrence of a predetermined condition; and in response to detecting the occurrence of the predetermined condition, updating the affordance to include an indication of a search function.

[0006] An example non-transitory computer-readable medium is disclosed herein. An example non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system, the one or more programs including instructions for receiving, via the one or more input devices, a request to display a user interface; in response to receiving the request to display the user interface, displaying, via a touch-sensitive display, the user interface that includes an affordance, the affordance including an indication of a number of pages of the user interface; and while displaying the user interface: detecting an occurrence of a predetermined condition; and in response to detecting the occurrence of the predetermined condition, updating the affordance to include an indication of a search function.

[0007] An example computer system (e.g., electronic device) is disclosed. An example computer system includes one or more processors; a memory; and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for receiving, via the one or more input devices, a request to display a user interface; in response to receiving the request to display the user interface, displaying, via a touch-sensitive display, the user interface including an affordance, the affordance including an indication of a number of pages of the user interface; and while displaying the user interface: detecting an occurrence of a predetermined condition; and in response to detecting the occurrence of the predetermined condition, updating the affordance to include an indication of a search function.

[0008] An example electronic device includes means for receiving, via the one or more input devices, a request to display a user interface; in response to receiving the request to display the user interface, means for displaying, via a touch-sensitive display, the user interface including an affordance, the affordance including an indication of a number of pages of the user interface; and while displaying the user interface: means for detecting an occurrence of a predetermined condition; and in response to detecting the occurrence of the predetermined condition, means for updating the affordance to include an indication of a search function.

[0009] Detecting an occurrence of a predetermined condition while displaying a user interface and updating an affordance to include an indication of a search function allows a user to quickly and efficiently access the search function without having to invoke a separate interface. This allows a user to access a greater variety of tasks while displaying the same interface, thereby increasing the efficiency of the interaction between the user and the electronic device and, thereby, reducing power consumption and preserving battery life.

[0010] An example method is disclosed. An example method includes, at a computer system in communication with a display generation component and one or more input devices: detecting, via the one or more input devices, a user input on a first portion of a user interface element associated with a search operation; and in response to detecting the user input selecting the affordance associated with the search operation, displaying, via the display generation component, a search results user interface, the search results user interface including: a user interface element associated with a search result; and a user interface element associated with a search operation, wherein the user interface element associated with a search operation is displayed in a predetermined location.

[0011] Disclosed herein are example non-transitory computer-readable media. An example non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system, the one or more programs including instructions for detecting, via one or more input devices, user input on a first portion of a user interface element associated with a search operation; and in response to detecting the user input that selects the affordance associated with the search operation, displaying, via a display generation component, a search results user interface including: user interface elements associated with search results; and user interface elements associated with the search operation, wherein the user interface elements associated with the search operation are displayed in a predetermined location.

[0012] Disclosed herein are example computer systems (e.g., electronic devices). An example computer system includes one or more processors; memory; and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for detecting, via one or more input devices, user input on a first portion of a user interface element associated with a search operation; and in response to detecting the user input that selects the affordance associated with the search operation, displaying, via a display generation component, a search results user interface including: user interface elements associated with search results; and user interface elements associated with the search operation, wherein the user interface elements associated with the search operation are displayed in a predetermined location.

[0013] An example computer system includes means for detecting, via the one or more input devices, user input on a first portion of a user interface element associated with a search operation; and in response to detecting the user input that selects the affordance associated with the search operation, means for displaying, via the display generation component, a search results user interface including: user interface elements associated with search results; and user interface elements associated with the search operation, wherein the user interface elements associated with the search operation are displayed in a predetermined location.

[0014] Displaying a search results interface including user interface elements associated with search results and user interface elements associated with a search operation in a predetermined location in response to detecting user input allows a user to quickly and efficiently interact with search functionality and various results. This allows the user to access a greater variety and quantity of information in response to a single input, thereby increasing the efficiency of the interaction between the user and the electronic device and, as a result, reducing power consumption and preserving battery. BRIEF DESCRIPTION OF DRAWINGS

[0015] FIG. 1 FIG. 1 illustrates a block diagram of a system and environment for implementing a digital assistant according to various examples.

[0016] FIG. 2A FIG. 2 illustrates a block diagram of a portable multifunctional device implementing a client-side portion of a digital assistant according to various examples.

[0017] FIG. 2B FIG. 3 illustrates a block diagram of example components for event processing according to various examples.

[0018] FIG. 3 FIG. 4 illustrates a portable multifunctional device implementing a client-side portion of a digital assistant according to various examples.

[0019] FIG. 4 FIG. 5 illustrates a block diagram of an example multifunctional device having a display and a touch-sensitive surface according to various examples.

[0020] FIG. 5A FIG. 6 illustrates an example user interface displaying a menu of applications on a portable multifunctional device in accordance with various examples.

[0021] FIG. 5B FIG. 7 illustrates an example user interface for a multifunctional device having a touch-sensitive surface separate from the display in accordance with various examples.

[0022] FIG. 6A FIG. 8 illustrates a personal electronic device according to various examples.

[0023] FIG. 6B FIG. 9 illustrates a block diagram of a personal electronic device according to various examples.

[0024] FIG. 7A FIG. 10 illustrates a block diagram of a digital assistant system or server portion thereof according to various examples.

[0025] FIG. 7B FIG. 11 illustrates functionality of a digital assistant shown in FIG. 7A FIG. 12 illustrates a portion of a knowledge base according to various examples.

[0026] FIG. 7C FIG. 13 illustrates a portion of a knowledge base according to various examples.

[0027] FIG. 8A to FIG. 8P FIG. 14 illustrates an example user interface for providing search functionality according to some embodiments.

[0028] FIG. 9 FIG. 15 illustrates a process for accessing a search user interface according to various examples.

[0029] FIG. 10 FIG. 16 illustrates a process for updating a search user interface according to various examples. DETAILED DESCRIPTION

[0030] In the following description of examples, reference is made to the accompanying drawings which form a part hereof, and in which is shown, by way of illustration, specific examples that can be practiced. It is to be understood that other examples can be used and structural changes can be made without departing from the scope of the various examples.

[0031] Described herein are user interfaces for providing search functionality and methods for accessing user interfaces for providing search functionality. The user interfaces and methods provide a more efficient and useful interface for a user, thereby reducing the amount of input required to perform a search and, thus, reducing power consumption and increasing battery life of an electronic device and / or computer system.

[0032] Although the following description uses the terms "first," "second," etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, a first input could be termed a second input, and, similarly, a second input could be termed a first input, without departing from the scope of the various described examples. The first input and the second input are both inputs, and in some cases, independent and distinct inputs.

[0033] The terminology used in the description herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various described examples and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0034] The term "if can be interpreted as meaning "when" or "while" or "in response to a determination" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" can be interpreted as meaning "upon a determination" or "in response to a determination" or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0035] 1. SYSTEMS AND ENVIRONMENTS

[0036] FIG. 1A block diagram of a system 100 is shown in accordance with various examples. In some examples, the system 100 implements a digital assistant. The terms "digital assistant," "virtual assistant," "intelligent automated assistant," or "automated digital assistant" refer to any information processing system that interprets natural language input in spoken and / or textual form to deduce user intent and perform actions based on the deduced user intent. For example, to act on the deduced user intent, the system performs one or more of the following steps: identify a task flow having steps and parameters designed to implement the deduced user intent, input specific requirements into the task flow in accordance with the deduced user intent; execute the task flow by invoking programs, methods, services, APIs, etc.; and generate an output response to the user in audible (e.g., spoken) and / or visual form.

[0037] In particular, a digital assistant can accept user requests that are at least partially in the form of natural language commands, requests, statements, narratives, and / or queries. Generally, a user request seeks an informational answer or performance of a task from the digital assistant. A satisfactory response to a user request includes providing the requested informational answer, performing the requested task, or a combination of the two. For example, a user asks a question of the digital assistant, such as "Where am I now?" Based on the user's current location, the digital assistant answers "You are near the Central Park West entrance." The user also requests performance of a task, such as "Please invite my friends to my girlfriend's birthday party next week." In response, the digital assistant can confirm the request by saying "Okay, right away," and then send appropriate calendar invitations to each of the user's friends listed in the user's electronic address book on behalf of the user. During performance of the requested task, the digital assistant sometimes interacts with the user in a continuing conversation involving multiple exchanges of information over a long period of time. There are many other ways in which a user interacts with a digital assistant to request information or perform various tasks. In addition to providing spoken responses and taking programmed actions, a digital assistant also provides responses in other video or audio forms, such as text, reminders, music, videos, animations, etc.

[0038] As FIG. 1 shown, in some examples, a digital assistant is implemented according to a client-server model. The digital assistant includes a client-side portion 102 (hereinafter "DA client 102") that executes on a user device 104, and a server-side portion 106 (hereinafter "DA server 106") that executes on a server system 108. The DA client 102 communicates with the DA server 106 over one or more networks 110. The DA client 102 provides client-side functionality, such as user-facing input and output processing, and communication with the DA server 106. The DA server 106 provides server-side functionality for any number of DA clients 102 that are each located on a respective user device 104.

[0039] In some examples, the DA server 106 includes a client-facing I / O interface 112, one or more processing modules 114, data and models 116, and an external service I / O interface 118. The client-facing I / O interface 112 facilitates client-facing input and output processing of the DA server 106. The one or more processing modules 114 utilize the data and models 116 to process speech input and determine a user intent based on the natural language input. Further, the one or more processing modules 114 perform task execution based on the inferred user intent. In some examples, the DA server 106 communicates with external services 120 over the one or more networks 110 to complete tasks or gather information. The external service I / O interface 118 facilitates such communication.

[0040] The user device 104 can be any suitable electronic device. In some examples, the user device 104 is a portable multifunction device (e.g., a device 200 as described below with reference to FIG. 2A The user device 104 can be any suitable electronic device. In some examples, the user device 104 is a portable multifunction device (e.g., a device 200 as described below with reference to FIG. 4 The user device 104 can be any suitable electronic device. In some examples, the user device 104 is a portable multifunction device (e.g., a device 200 as described below with reference to FIG. 6A to FIG. 6B The user device 104 can be any suitable electronic device. In some examples, the user device 104 is a portable multifunction device (e.g., a device 200 as described below with reference to iPod and devices. Other examples of portable multifunction devices include, but are not limited to, earbuds / headphones, speakers, and laptop or tablet computers. In addition, in some examples, the user device 104 is a non-portable multifunction device. Specifically, the user device 104 is a desktop computer, a game console, a speaker, a television, or a television set-top box. In some examples, the user device 104 includes a touch-sensitive surface (e.g., a touch screen display and / or a touchpad). In addition, the user device 104 optionally includes one or more other physical user-interface devices, such as a physical keyboard, a mouse, and / or a joystick. Various examples of electronic devices, such as multifunction devices, are described in more detail below.

[0041] Examples of the communication network 110 include local area networks (LANs) and wide area networks (WANs), such as the Internet. The communication network 110 is implemented using any known network protocol, including various wired or wireless protocols, such as Ethernet, Universal Serial Bus (USB), FIREWIRE, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi, Voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol.

[0042] The server system 108 is implemented on one or more standalone data processing devices or a distributed computer network. In some examples, the server system 108 also employs various virtual devices and / or services of third-party service providers (e.g., third-party cloud service providers) to provide potential computing resources and / or infrastructure resources of the server system 108.

[0043] In some examples, the user device 104 communicates with the DA server 106 via a second user device 122. The second user device 122 is similar or identical to the user device 104. For example, the second user device 122 is similar to the devices 200, 400, or 600 described below with reference to FIGS. 2, 4, and 6. The user device 104 is configured to be communicatively coupled to the second user device 122 via a direct communication connection (such as Bluetooth, NFC, BTLE, etc.) or via a wired or wireless network (such as a local Wi-Fi network). In some examples, the second user device 122 is configured to act as a proxy between the user device 104 and the DA server 106. For example, the DA client 102 of the user device 104 is configured to transmit information (e.g., user requests received at the user device 104) to the DA server 106 via the second user device 122. The DA server 106 processes the information and returns relevant data (e.g., data content responsive to the user requests) to the user device 104 via the second user device 122. FIG. 2A 、 FIG. 4 and FIG. 6A to FIG. 6B In some examples, the user device 104 communicates with the DA server 106 via a second user device 122. The second user device 122 is similar or identical to the user device 104. For example, the second user device 122 is similar to the devices 200, 400, or 600 described below with reference to FIGS. 2, 4, and 6. The user device 104 is configured to be communicatively coupled to the second user device 122 via a direct communication connection (such as Bluetooth, NFC, BTLE, etc.) or via a wired or wireless network (such as a local Wi-Fi network). In some examples, the second user device 122 is configured to act as a proxy between the user device 104 and the DA server 106. For example, the DA client 102 of the user device 104 is configured to transmit information (e.g., user requests received at the user device 104) to the DA server 106 via the second user device 122. The DA server 106 processes the information and returns relevant data (e.g., data content responsive to the user requests) to the user device 104 via the second user device 122.

[0044] In some examples, the user device 104 is configured to send a truncated request for data to the second user device 122 to reduce the amount of information transmitted from the user device 104. The second user device 122 is configured to determine supplemental information to add to the truncated request to generate a complete request to transmit to the DA server 106. This system architecture can advantageously allow a user device 104 with limited communication capabilities and / or limited battery power (e.g., a watch or similar compact electronic device) to access services provided by the DA server 106 by using a second user device 122 (e.g., a mobile phone, a laptop, a tablet, etc.) with stronger communication capabilities and / or battery power as a proxy to the DA server 106. Although FIG. 1 Although only two user devices 104 and 122 are shown in FIG. 1, it should be understood that in some examples, the system 100 can include any number and type of user devices configured to communicate with the DA server system 106 in this proxy configuration.

[0045] Although FIG. 1 The digital assistant shown in FIG. 1 includes both a client-side portion (e.g., the DA client 102) and a server-side portion (e.g., the DA server 106), in some examples, the functionality of the digital assistant is implemented as a standalone application installed on a user device. Moreover, the division of functionality between the client portion and the server portion of the digital assistant can vary in different implementations. For example, in some examples, the DA client is a thin client that provides only user-facing input and output processing functionality and delegates all other functionality of the digital assistant to a backend server.

[0046] 2. Electronic device

[0047] Attention is now directed to embodiments of an electronic device for implementing the client-side portion of a digital assistant. FIG. 2AA block diagram illustrating portable multifunction device 200 with touch- sensitive display 212 in accordance with some embodiments is shown in Figure 2. Touch- sensitive display 212 is sometimes called a "touch screen" for convenience and is sometimes known as or called a "touch- sensitive display system." Device 200 includes memory 202 (which optionally includes one or more computer-readable storage mediums), memory controller 222, one or more processing units (CPUs) 220, peripherals interface 218, RF circuitry 208, audio circuitry 210, speaker 211, microphone 213, input / output (I / O) subsystem 206, other input control devices 216, and external port 224. Device 200 optionally includes one or more optical sensors 264. Device 200 optionally includes one or more contact intensity sensors 265 for detecting intensity of contacts on device 200 (e.g., a touch- sensitive surface such as touch- sensitive display system 212 of device 200). Device 200 optionally includes one or more tactile output generators 267 for generating tactile outputs on device 200 (e.g., generating tactile outputs on a touch- sensitive surface such as touch- sensitive display system 212 of device 200 or touchpad 455 of device 400). These components optionally communicate over one or more communication buses or signal lines 203.

[0048] As used in the specification and claims, the term "intensity" of a contact on a touch- sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch- sensitive surface, or to a proxy for the force or pressure of the contact on the touch- sensitive surface. The intensity of a contact has a range of values that includes at least four distinct values and more typically includes hundreds of distinct values (e.g., at least 256). Intensity of a contact is, optionally, determined (or measured) using various approaches and various sensors or combinations of sensors. For example, one or more force sensors underneath (or adjacent to) the touch- sensitive surface optionally are used to measure force at various points on the touch- sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., a weighted average) to determine an estimated force at the point of contact. Similarly, a pressure- sensitive tip of a stylus is, optionally, used to determine a pressure of the stylus on the touch- sensitive surface. Alternatively, the size of the contact area detected on the touch- sensitive surface and / or changes thereto, the capacitance of the touch- sensitive surface proximate to the contact and / or changes thereto, and / or the resistance of the touch- sensitive surface proximate to the contact and / or changes thereto are, optionally, used as a substitute for the force or pressure of the contact on the touch- sensitive surface. In some implementations, the substitute measurements for contact force or pressure are used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurements). In some implementations, the substitute measurements for contact force or pressure are converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of a contact as an attribute of a user input allows for user access to additional device functions with additional device features that would otherwise not be accessible by the user.

[0049] As used in the specification and claims, the term "tactile output" refers to physical displacement of a device relative to a previous positioning of the device by a user with the user's sense of touch, physical displacement of a component (e.g., a touch-sensitive surface) of a device relative to another component (e.g., a housing) of the device, or displacement of the component relative to a center of mass of the device that will be detected by a user with the user's sense of touch. For example, in the case of contact between a device or component of a device and a surface of a user that is sensitive to touch (e.g., a finger, palm, or other part of a user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in physical characteristics of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) will optionally be interpreted by a user as a "down click" or "up click" of a physical button even if the touch-sensitive surface does not move. In some cases, a user will feel a tactile sensation such as an "down click" or "up click" even if there is no movement of a physical button associated with the touch-sensitive surface by which the user physically presses (e.g., depresses) the button (e.g., by depressing a key on a keyboard, area of a trackpad, etc.). Similarly, a user will feel a tactile sensation of a "scrubbing" or "roughness" even if there is no change in smoothness of a touch-sensitive surface. While such interpretations of tactile sensations by a user will be subject to the individualized sensory perceptions of the user, there are many tactile sensations that are commonly perceived by most users. Accordingly, while tactile output is described in terms of a particular sensory perception (e.g., "down click", "up click", "scrubbing", "roughness") that a user will feel, unless otherwise stated, the generated tactile output corresponds to a physical displacement of a device or component of the device that will generate that particular sensory perception for a typical (or average) user.

[0050] It should be appreciated that device 200 is only one example of a portable multifunction device, and that device 200 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. The various components illustrated for device 200 are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and / or application specific integrated circuits. FIG. 2A The various components illustrated for device 200 are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0051] Memory 202 includes one or more computer-readable storage mediums. The computer-readable storage mediums are for example, tangible and non-transitory. Memory 202 includes high-speed random access memory and can also include nonvolatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other nonvolatile solid-state storage devices. Memory controller 222 controls access to memory 202 by other components of device 200.

[0052] In some examples, the non-transitory computer-readable storage medium of memory 202 is for storing instructions (e.g., for performing aspects of the processes described below) for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In other examples, the instructions (e.g., for performing aspects of the processes described below) are stored on a non-transitory computer- readable storage medium (not shown) of server system 108, or divided between the non- transitory computer-readable storage medium of memory 202 and the non-transitory computer-readable storage medium of server system 108.

[0053] Peripheral interface 218 is used to couple the device’s input and output peripherals to CPU 220 and memory 202. One or more processors 220 run or execute various software programs and / or instruction sets stored in memory 202 to perform various functions of device 200 and process data. In some embodiments, peripheral interface 218, CPU 220, and memory controller 222 are implemented on a single chip, such as chip 204. In some other embodiments, they are implemented on separate chips.

[0054] RF (radio frequency) circuitry 208 receives and sends RF signals, also called electromagnetic signals. RF circuitry 208 converts electrical signals to / from electromagnetic signals and communicates with communications networks and other communications devices, via the electromagnetic signals. RF circuitry 208 optionally includes well-known circuitry for detecting / handling the RF signals, including, for example, antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, and so forth. RF circuitry 208 optionally communicates with networks, such as the Internet, also referred to as the World Wide Web (WWW), an intranet and / or a wireless network, such as a cellular telephone network, a wireless local area network (LAN) and / or a metropolitan area network (MAN), and other devices by wireless communication using RF signals. RF circuitry 208 optionally includes well-known circuitry for detecting / handling the RF signals, such as a Bluetooth®, Bluetooth® Low Energy (LE), Zigbee®, near-field communication (NFC) circuitry, and / or a wireless local area network (WLAN) such as Wi-Fi circuitry. The wireless communication optionally uses any of a plurality of communications standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), code division multiple access (CDMA), time division multiple access (TDMA), Wideband-CDMA (W-CDMA), Long Term Evolution (LTE), Near Field Communication (NFC), Bluetooth, Bluetooth Low Energy (BLE), a Wireless Fidelity (Wi-Fi) such as IEEE 802.11a, IEEE 802.10b, IEEE 802.10g, IEEE 802.10η, and / or IEEE 802.1 lac, a Wireless LAN (WLAN), a Wi-MAX, a protocol for e-mail (e.g., Internet Message Access Protocol (IMAP) and / or

[0055] Audio circuitry 210, speaker 211, and microphone 213 provide an audio interface between a user and device 200. Audio circuitry 210 receives audio data from peripherals interface 218, converts the audio data to an electrical signal, and transmits the electrical signal to speaker 211. Speaker 211 converts the electrical signal to human-audible sound waves. Audio circuitry 210 also receives electrical signals converted by microphone 213 from sound waves. Audio circuitry 210 converts the electrical signal to audio data and transmits the audio data to peripherals interface 218 for processing. Audio data are retrieved from and / or transmitted to memory 202 and / or RF circuitry 208 by peripherals interface 218 in some embodiments. In some embodiments, audio circuitry 210 also includes a headset jack (e.g., 312 in FIG. 3). The headset jack provides an interface between audio circuitry 210 and removable audio input / output peripherals, such as output-only headphones or a headset with both output (e.g., stereo FIG. 3

[0056] I / O subsystem 206 couples input / output peripherals on device 200, such as touch screen 212 and other input control devices 216, with peripherals interface 218. I / O subsystem 206 optionally includes display controller 256, optical sensor controller 258, intensity sensor controller 259, haptic feedback controller 261, and one or more input controllers 260 for other input or control devices. The one or more input controllers 260 receive / send electrical signals from / to other input control devices 216. The other input control devices 216 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and so forth. In some alternate embodiments, input controller(s) 260 are, optionally, coupled with any (or none) of the following: a keyboard, infrared port, USB port, and a pointer device such as a mouse. The one or more buttons (e.g., 308 in FIG. 3) optionally include an up / down button for FIG. 3 FIG. 3

[0057] ​​​A quick press of the down button disengages a lock of the touch screen 212 or begins a process that uses gestures on the touch screen to unlock the device, as described in U.S. Patent Application 11 / 322,549, "Unlocking a Device by Performing Gestures on an Unlock Image," filed December 23, 2005; U.S. Patent No. 7,657,849, which are hereby incorporated by reference in their entirety. A longer press of the down button (e.g., 306) turns power to the device 200 on or off. The user is able to customi ze a functionality of one or more of the buttons. The touch screen 212 is used to implement virtual or soft buttons and one or more soft keyboards.

[0058] The touch-sensitive display 212 provides an input interface and an output interface between the device and a user. The display controller 256 receives and / or sends electrical signals from / to the touch screen 212. The touch screen 212 displays visual output to the user. The visual output includes graphics, text, icons, video, and any combination thereof (collectively termed "graphics"). In some embodiments, some or all of the visual output or

[0059] The touch screen 212 has a touch-sensitive surface, sensor, or set of sensors that accepts input from the user based on haptic and / or tactile contact. The touch screen 212 and the display controller 256 (along with any associated modules and / or sets of instructions in memory 202) detect contact (and any movement or breaking of the contact) on the touch screen 212 and convert the

[0060] The touch screen 212 uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light-emitting diode) technology, although other display technologies can be used in other embodiments. The touch screen 212 and the display controller 256 can detect contact and any movement or breaking of the contact using any of a plurality of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 212. In an example embodiment, projected mutual capacitance sensing technology is used, such as that found in the iPhone® and iPod touch® from Apple Inc. of Cupertino, California. and iPod touch® devices from Apple Inc. of Cupertino, California.

[0061] In some embodiments, the touch-sensitive display of touch screen 212 is analogous to the multi-touch- sensitive touchpads described in the following U.S. Patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman) and / or U.S. Patent Publication 2002 / 0015024 Al, which are hereby incorporated by reference in their entirety. However, touch screen 212 displays visual output from device 200, whereas a touch-sensitive touchpad does not provide visual output.

[0062] The touch-sensitive display in some embodiments of touch screen 212 is described in the following applications:

[0063] (1) U.S. Patent Application No. 11 / 381,313, "Multipoint TouchSurface Controller," filed May 2, 2006; (2) U.S. Patent Application No. 10 / 840,862, "Multipoint Touchscreen," filed May 6, 2004; (3) U.S. Patent Application No. 10 / 903,964, "Gestures For Touch Sensitive Input Devices," filed July 30, 2004; (4) U.S. Patent Application No. 11 / 048,264, "Gestures For Touch Sensitive Input Devices," filed January 31, 2005; (5) U.S. Patent Application No. 11 / 038,590, "Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices," filed January 18, 2005; (6) U.S. Patent Application No. 11 / 228,758, "Virtual Input Device Placement On A Touch Screen User Interface," filed September 16, 2005; (7) U.S. Patent Application No. 11 / 228,700, "Operation Of A Computer With A Touch Screen Interface," filed September 16, 2005; (8) U.S. Patent Application No. 11 / 228,737, "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard," filed September 16, 2005; and (9) U.S. Patent Application No. 11 / 367,749, "Multi-Functional Hand-Held Device," filed March 3, 2006. All of these applications are incorporated in full by reference herein.

[0064] Touch screen 212 has, for example, a video resolution of greater than 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user can make contact with touch screen 212 using any suitable object or appendage, such as a stylus, finger, and the like. In some embodiments, the user interface is designed to work primarily with finger-based contacts and gestures, which can be less precise than stylus-based input due to the larger area of contact of a finger on the touch screen. In some embodiments, the device translates the rough finger-based input into a precise pointer / cursor position or command for performing the action desired by the user.

[0065] In some embodiments, device 200 includes a touchpad (not shown) in addition to the touch screen. In some embodiments, the touchpad is a touch-sensitive area that is separate from the touch screen 212 and that does not display visual output. The touchpad is a touch-sensitive surface that is separate from the touch screen 212 or that is formed by the touch screen 212 as a force- sensitive area of the touch screen 212.

[0066] Device 200 also includes power system 262 for powering the various components of device 200. Power system 262 includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)) and any other components associated with the generation, management and distribution of power for portable devices.

[0067] Device 200 also includes one or more optical sensors 264. FIG. 2A Optical sensor(s) 264 are coupled to optical sensor controller 258 in I / O subsystem 206. Optical sensor(s) 264 include charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) photoreceptor(s). Optical sensor(s) 264 receive light from the environment, projected through one or more lens(es), and converts the light to data representing an image. In conjunction with imaging module 243 (also called a camera module), optical sensor(s) 264 capture images (e.g., static images, video, and / or live video). In some embodiments, optical sensor(s) are located on the front of device 200, opposite touch screen display 212 on the front of the device so that the touch screen display is used as the viewfinder for the camera module. In some embodiments, optical sensor(s) are located on the front of device 200 so that the user’s image is obtained for videoconferencing while the user views the other video conference participants on the touch screen display. In some embodiments, the position of optical sensor(s) 264 can be changed by the user (e.g., by rotating the lens and the sensor in the device housing) so that a single optical sensor 264 is used with the touch screen display for both video conferencing and still and / or video image capture.

[0068] Device 200 optionally also includes one or more contact intensity sensors 265. Contact intensity sensor(s) 265 reduce the number of contacts that accurately indicates the location of a touch, and improves accuracy of the user’s input. FIG. 2AA contact strength sensor 265 is shown coupled to a strength sensor controller 259 in I / O subsystem 206. The contact strength sensor 265 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric sensors, optical force sensors, capacitive touch-sensitive surfaces, or other strength sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). The contact strength sensor 265 receives contact strength information (e.g., pressure information or a substitute for pressure information) from the environment. In some embodiments, at least one contact strength sensor is arranged juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 212). In some embodiments, at least one contact strength sensor is located on the rear of device 200, opposite to the touchscreen display 212 located on the front of device 200.

[0069] The device 200 also includes one or more proximity sensors 266. FIG. 2A A proximity sensor 266 coupled to a peripheral device interface 218 is shown. Alternatively, the proximity sensor 266 is coupled to an input controller 260 in an I / O subsystem 206. The proximity sensor 266 performs as described in the following U.S. patent applications: No. 11 / 241,839, entitled "Proximity Detector In Handheld Device"; 11 / 240,788, entitled "Proximity Detector In Handheld Device"; 11 / 620,702, entitled "Using Ambient Light Sensor To Augment Proximity Sensor Output"; 11 / 586,862, entitled "Automated Response To And Sensing Of User Activity In Portable Devices"; and 11 / 638,251, entitled "Methods And Systems For Automatic Configuration Of Peripherals", the entire contents of which are incorporated herein by reference. In some implementations, the proximity sensor is turned off and the touchscreen 212 is disabled when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call).

[0070] The device 200 optionally also includes one or more tactile output generators 267. FIG. 2AA haptic output generator coupled to a haptic feedback controller 261 in I / O subsystem 206 is shown. The haptic output generator 267 optionally includes one or more electroacoustic devices such as speakers or other audio components; and / or electromechanical devices for converting energy into linear motion such as motors, solenoids, electroactive polymerizers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components for converting electrical signals into haptic outputs on the device). A contact intensity sensor 265 receives haptic feedback generation instructions from a haptic feedback module 233 and generates a haptic output on device 200 that can be felt by a user of device 200. In some embodiments, at least one haptic output generator is juxtaposed or adjacent to a haptic surface (e.g., haptic display system 212) and optionally generates the haptic output by moving the haptic surface vertically (e.g., in / outward from the surface of device 200) or laterally (e.g., backward and forward in the same plane as the surface of device 200). In some implementations, at least one haptic output generator sensor is located on the rear of the device 200, opposite to the touch screen display 212 located on the front of the device 200.

[0071] The device 200 also includes one or more accelerometers 268. FIG. 2A An accelerometer 268 coupled to a peripheral device interface 218 is shown. Alternatively, the accelerometer 268 is coupled to an input controller 260 in an I / O subsystem 206. The accelerometer 268 performs as described in the following U.S. patent publications: U.S. Patent Publication No. 20050190059, “Acceleration-based Theft Detection System for Portable Electronic Devices” and U.S. Patent Publication No. 20060017692, “Methods and Apparatuses For Operating A Portable Device Based On An Accelerometer,” the entire contents of which are incorporated herein by reference. In some embodiments, information is displayed on a touchscreen display in portrait or landscape view based on analysis of data received from one or more accelerometers. Device 200 optionally includes, in addition to one or more accelerometers 268, a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information about the position and orientation (e.g., portrait or landscape) of device 200.

[0072] In some embodiments, the software components stored in memory 202 include operating system 226, communication module (or set of instructions) 228, contact / motion module (or set of instructions) 230, graphics module (or set of instructions) 232, text input module (or set of instructions) 234, Global Positioning System (GPS) module (or set of instructions) 235, digital assistant client module 229, and applications (or sets of instructions) 236. Further, memory 202 stores data and models, such as user data and models 231. Moreover, in some embodiments, memory 202 stores device / global internal state 257, as shown in FIGS. FIG. 2A ) or 470( FIG. 4 ) as shown in FIGS. FIG. 2A and FIG. 4 Device / global internal state 257 includes one or more of: active application state, indicating which applications, if any, are currently active; display state, indicating what applications, views or other information occupy various regions of touch screen display 212; sensor state, including information obtained from the device's various sensors and input control devices 216; and location information indicating the location and / or attitude of the device.

[0073] Operating system 226 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates wireless communications between various hardware and software components.

[0074] Communication module 228 facilitates communication with other devices over one or more external ports 224 and also includes various software components for handling data received by RF circuitry 208 and / or external port 224. External port 224 (e.g., Universal Serial Bus (USB), FIREWIRE, etc.) is adapted for coupling directly to other devices or indirectly over a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as, or similar to, and / or compatible with, the 30-pin connector used on iPod® (trademark of Apple Inc.), iPhone® and iPad® devices. (Apple Inc. of Cupertino, California) devices.

[0075] Contact / motion module 230 optionally detects contact with touch screen 212 (in conjunction with display controller 256) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). Contact / motion module 230 includes various software components for performing various operations related to detection of contact, such as determining if contact has occurred (e.g., detecting a finger-down event), determining an intensity of the contact (e.g., the force or pressure of the contact or a substitute for the force or pressure of the contact), determining if movement of the contact has occurred (e.g., detecting a finger-dragging event), and determining if the contact has ceased (e.g., detecting a finger-up event or a break in contact). Contact / motion module 230 receives contact data from the touch-sensitive surface. Determining movement of the point of contact, which is represented by a series of contact data, optionally includes determining speed (magnitude), velocity (magnitude and direction), and / or acceleration (a change in magnitude and / or direction) of the point of contact. These operations are, optionally, applied to single contact events (e.g., one finger events) or to multiple simultaneous contact events (e.g., "multitouch" events).

[0076] In some embodiments, contact / motion module 230 uses a set of one or more intensity thresholds to determine whether operations have been performed (e.g., whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by activation thresholds of specific physical actuators, and can be adjusted without changing the physical hardware of device 200). For example, without changing touchpad or touch screen display hardware, a mouse "click" threshold of a touchpad or touch screen can be set to any one threshold in a large range of predefined thresholds. Additionally, in some implementations, a user of the device is provided with software settings to adjust one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once with a system-wide click on an "intensity" parameter).

[0077] Contact / motion module 230 optionally detects contact input by a user. Different gestures on the touch-sensitive surface have different motion patterns (e.g., different patterns of detected contact, movement, and / or intensity of the contact). Accordingly, a gesture is, optionally, detected by detecting a particular motion pattern. For example, detecting a finger tap gesture includes detecting a finger- down event, followed by a finger-up (lift off) event at the same location (or substantially the same location) as the finger-down event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on the touch- sensitive surface includes detecting a finger-down event, followed by one or more finger- drag events, and then detecting a finger-up (lift off) event.

[0078] Graphics module 232 includes various known software components for rendering and displaying graphics on touch screen 212 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual property) of graphics that are displayed. As used herein, the term "graphics" includes any object that can be displayed to a user, including, without limitation, text, web pages, icons (such as user-interface objects including soft keys), digital images, videos, animations, and the like.

[0079] In some embodiments, graphics module 232 stores data representing graphics to be used. Each graphic is, optionally, assigned a corresponding code. Graphics module 232 receives, from applications etc., one or more codes specifying graphics to be displayed, and then generates screen image data for a

[0080] Haptic feedback module 233 includes various software components for generating instructions used by haptic feedback mechanism(s) 267 to produce tactile outputs at one or more locations on device 200 in response to user interactions with device 200.

[0081] Text input module 234, which in some embodiments is part of graphics module 232, provides soft keyboards for entering text in various applications (e.g., contacts 237, e-mail 240, IM 241, browser 247, and any other application that needs text input).

[0082] GPS module 235 determines the location of the device and provides this information for use in various applications (e.g., to telephone 238 for use in

[0083] The digital assistant client module 229 includes various client-side digital assistant instructions to provide client-side functionality of the digital assistant. For example, the digital assistant client module 229 can accept voice input (e.g., speech input), textual input, touch input, and / or gesture input through various user interfaces of the portable multifunction device 200 (e.g., microphone 213, one or more accelerometers 268, touch-sensitive display system 212, one or more optical sensors 264, other input controls 216, etc.). The digital assistant client module 229 can also provide output in the form of audio (e.g., speech output), visual output, and / or tactile output through various output interfaces of the portable multifunction device 200 (e.g., speaker 211, touch-sensitive display system 212, one or more tactile output generators 267, etc.). For example, output is provided as speech, sound, reminders, text messages, menus, graphics, videos, animations, vibrations, and / or combinations of two or more of the above. During operation, the digital assistant client module 229 communicates with the DA server 106 using the RF circuitry 208.

[0084] The user data and models 231 include various data associated with the user (e.g., user-specific vocabulary data, user preference data, user-specified name pronunciations, data from the user's electronic address book, to-do items, shopping lists, etc.) to provide client-side functionality of the digital assistant. In addition, the user data and models 231 include various models for processing user input and determining user intent (e.g., speech recognition models, statistical language models, natural language processing models, ontologies, task flow models, service models, etc.).

[0085] In some examples, the digital assistant client module 229 utilizes various sensors, subsystems, and peripherals of the portable multifunction device 200 to gather additional information from the surrounding environment of the portable multifunction device 200 to establish a context associated with the user, the current user interaction, and / or the current user input. In some examples, the digital assistant client module 229 provides the context information, or a subset thereof, to the DA server 106 along with the user input to help infer the user intent. In some examples, the digital assistant also uses the context information to determine how to prepare and deliver output to the user. The context information is referred to as context data.

[0086] In some examples, the contextual information that accompanies user input includes sensor information, such as lighting, ambient noise, ambient temperature, images or video of the surrounding environment, etc. In some examples, the contextual information can also include physical state of the device, such as device orientation, device location, device temperature, power level, velocity, acceleration, motion patterns, cellular signal strength, etc. In some examples, information related to the software state of the DA server 106, such as the running processes of the portable multifunctional device 200, installed programs, past and current network activity, background services, error logs, resource usage, etc., are provided to the DA server 106 as contextual information associated with user input.

[0087] In some examples, the digital assistant client module 229 selectively provides information stored on the portable multifunctional device 200 (e.g., user data 231) in response to requests from the DA server 106. In some examples, the digital assistant client module 229 also elicits additional input from the user via natural language dialog or other user interfaces upon request by the DA server 106. The digital assistant client module 229 transmits this additional input to the DA server 106 to assist the DA server 106 in intent inference and / or in fulfilling the user intent expressed in the user request.

[0088] Reference is made below to FIG. 7A to FIG. 7C The digital assistant is described in more detail. It should be appreciated that the digital assistant client module 229 can include any number of sub-modules of the digital assistant module 726 described below.

[0089] The applications 236 include the following modules (or sets of instructions) or a subset or superset thereof:

[0090] A contacts module 237 (sometimes referred to as an address book or contact list);

[0091] A telephony module 238;

[0092] A video conferencing module 239;

[0093] An email client module 240;

[0094] An instant messaging (IM) module 241;

[0095] A fitness support module 242;

[0096] A camera module 243 for still and / or video images;

[0097] An image management module 244;

[0098] A video player module;

[0099] A music player module;

[0100] a browser module 247;

[0101] a calendar module 248;

[0102] a widget module 249, which in some examples includes one or more of the following:

[0103] a weather widget 249-1, a stocks widget 249-2, a calculator widget 249-3, a

[0104] a widget creator module 250 for forming the user-created widgets 249-6;

[0105] a search module 251;

[0106] a video and music player module 252, which merges video player module and music player module;

[0107] a notes module 253;

[0108] a maps module 254; and / or

[0109] an online video module 255.

[0110] Examples of other applications 236 stored on storage 202 include other word processing applications, other image editing applications, a drawing application, presentation application, JAVA-enabled applications, an encryption application, a digital rights management application, a voice recognition and / or voice replication application.

[0111] In conjunction with touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, the contacts module 237 are used for managing a

[0112] In conjunction with RF circuitry 208, audio circuitry 210, speaker 211, microphone 213, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, telephone module 238 are used to place, conduct, and terminate a telephone call or to access other call-related services.

[0113] In conjunction with RF circuitry 208, audio circuitry 210, speaker 211, microphone 213, touch screen 212, display controller 256, optical sensor 264, optical sensor controller 258, contact / motion module 230, graphics module 232, text input module 234, contact module 237, and telephone module 238, video conference module 239 includes executable instructions to initiate, conduct, and terminate a video conference between a user and one or more other participants in accordance with user instructions.

[0114] In conjunction with RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, e-mail client module 240 includes executable instructions to create, send, receive, and manage e-mail in response to user instructions. In conjunction with image management module 244, e-mail client module 240 makes it very easy to create and send e-mails with still or video images taken with camera module 243.

[0115] In conjunction with RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, the instant messaging module 241 includes executable instructions to enter a sequence of characters corresponding to a instant message, modify previously entered characters, transmit a respective instant message (for example, using a Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for a phone-based instant message or using XMPP, SIMPLE, or IMPS for an Internet-based instant message), receive instant messages, and view received instant messages. In some embodiments, an instant message that has been sent to or received by the device includes a graphical component, photo component, audio component, video component, and / or other attachment component, such as in a MMS and / or an Enhanced Messaging Service (EMS). As used herein, "instant message" refers to both a phone-based message (e.g., a message sent using SMS or MMS) and an Internet-based message (e.g., a message sent using XMPP, SIMPLE, or IMPS).

[0116] In conjunction with RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, GPS module 235, map module 254, and music player module, workout support module 242 includes instructions to create workouts (e.g., with time, distance, and / or calorie burning goals); communicate with workout sensors (sports devices); receive workout sensor data; calibrate sensors used for a workout; select and play music for a workout; and display, store, and transmit workout data.

[0117] In conjunction with touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and image management module 244, camera module 243 includes instructions to capture still images or video (including a video stream) and store them into memory 202, modify characteristics of a still image or video, or delete a still image or video from memory 202.

[0118] In conjunction with touch screen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, and camera module 243, image management module 244 includes instructions to arrange, modify (e.g., edit), or otherwise manipulate, label, delete, present (e.g., in a digital slide show or album), and store still and / or video images.

[0119] In conjunction with RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, e-mail client module 240 includes instructions to create, send, receive, and manage e-mail

[0120] In conjunction with RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, e-mail client module 240, and browser module 247, calendar module 248 includes instructions to create and display calendars and data associated with calendars (e.g., calendar entries, to-do lists, etc.), receive data

[0121] In conjunction with RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, and browser module 247, the widget module 249 is used by a user to create widgets (e.g., widgets 249-1 and 249-2, which respectively include weather and stock information) or to retrieve widgets for

[0122] In conjunction with RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, and browser module 247, the widget creator module 250 is used by a user to create widgets (e.g., to select the user- specified portion of a web page and to select the widget type, such as weather, stocks, or photos, in which the web page portion is provided).

[0123] In conjunction with touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, the search module 251 includes executable instructions to search for text, music, sound, image, video, and / or other files in memory 202 that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0124] In conjunction with touch screen 212, display controller 256, contact / motion module 230, graphics module 232, audio circuitry 210, speaker 211, RF circuitry 208, and browser module 247, the video and music player module 252 includes executable instructions that allow the user to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and executable instructions to display, present or otherwise play back videos (e.g., on the touch screen 212, or on an external, connected display via external port 224). In some embodiments, device 200 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).

[0125] In conjunction with touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, the notes module 253 includes executable instructions to create and manage notes, to-do lists, and the like in accordance with user instructions.

[0126] In conjunction with RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, GPS module 235, and browser module 247, map module 254 can be used to receive, display, modify, and store maps and data associated with maps (e.g., driving directions, data on stores and other points of interest at or near a particular location, and other location-based data) in accordance with user instructions.

[0127] In conjunction with touch screen 212, display controller 256, contact / motion module 230, graphics module 232, audio circuitry 210, speaker 211, RF circuitry 208, text input module 234, e-mail client module 240, and browser module 247, online video module 255 includes instructions that allow the user to access, browse, receive (e.g., by streaming and / or download), play back (e.g., on the touch screen or on an external, connected display via external port 224), send an e-mail with a link to a particular online video, and otherwise manage online videos in one or more file formats, such as H.264. In some embodiments, instant messaging module 241, rather than e-mail client module 240, is used to send a link to a particular online video. Additional descriptions of online video application can be found in U.S. Provisional Patent Application No. 60 / 936,562, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed June 20, 2007, and U.S. Patent Application No. 11 / 968,067, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed December 31, 2007, the contents of which are hereby incorporated by reference in their entirety.

[0128] Each of the above-identified modules and applications corresponds to a set of executable instructions for performing one or more functions described above and the methods described in this application (e.g., the computer-implemented methods described herein and other information processing methods). These modules (e.g., sets of instructions) need not be implemented as separate software programs, procedures or modules, and thus various subsets of these modules can be combined or otherwise rearranged in various embodiments. For example, video FIG. 2AThe memory 202 includes computer- readable instructions that, when executed by the processor 204, enable the device 200 to perform various functions (e.g., functions related to video and music player module 252, etc.). In some embodiments, the memory 202 stores a subset of the instructions described above. Alternatively, the memory 202 stores additional or different instructions.

[0129] In some embodiments, the device 200 is a device for which a predefined set of functions of the device are performed exclusively through touchscreens and / or touchpads. By using a touchscreen and / or touchpad as the primary input control device for operation of the device 200, the number of physical input control devices (such as push buttons, dials, etc.) on the device 200 is reduced.

[0130] The predefined set of functions performed exclusively through touchscreens and / or touchpads optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates the device 200 from any user interface displayed on the device 200 to a main menu, home menu, or root menu. In such embodiments, the touchpad is used to implement a "menu button." In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0131] FIG. 2B FIG. 27 is a block diagram illustrating example components of an event handler, in accordance with some embodiments. In some embodiments, the memory 202 (e.g., within operating system 226) or the memory 470 (e.g., within operating system 470) includes an event classifier 270 and respective applications 236-1 (e.g., any of the aforementioned applications 237-251, 255, 480-490). FIG. 2A ) or the memory 470 ( FIG. 4 ) includes an event classifier 270 (e.g., within operating system 226) and respective applications 236-1 (e.g., any of the aforementioned applications 237-251, 255, 480-490).

[0132] The event classifier 270 receives event information and determines an application 236-1 and an application view 291 of the application 236-1 to which to deliver the event information. The event classifier 270 includes an event monitor 271 and an event dispatcher module 274. In some embodiments, the application 236-1 includes an application internal state 292 that indicates a current application view that is displayed on the touch-sensitive display 212 when the application is active or executing. In some embodiments, the device / global internal state 257 is used by the event classifier 270 to determine which application(s) are currently active, and the application internal state 292 is used by the event classifier 270 to determine an application view 291 to which to deliver event information.

[0133] In some embodiments, the application internal state 292 includes additional information such as one or more of: resume information to be used when the application 236-1 resumes execution, user interface state information indicating that information is being displayed or is ready for display by the application 236-1, a state queue for enabling the user to return to a previous state or view of the application 236-1, and a repeat / undo queue of previous actions taken by the user.

[0134] The event monitor 271 receives event information from the peripherals interface 218. The event information includes information about sub-events (e.g., user touches on touch- sensitive display 212, motion sensor events such as free fall, wireless signal events such as Wi-Fi and Bluetooth beacons, and the like). The peripherals interface 218 communicates event information as it happens, through the input-output subsystem 206 or sensors such as the proximity sensor 266, the one or more accelerometers 268, and / or the microphone 213 (through the audio circuitry 210). The peripherals interface 218 communicates event information from I / O subsystem 206 including information from the touch-sensitive display 212 or touch-sensitive surfaces.

[0135] In some embodiments, the event monitor 271 sends a request to the peripherals interface 218 at predetermined intervals. In response, the peripherals interface 218 transmits event information. In other embodiments, the peripherals interface 218 transmits event information only when there is significant event (e.g., input that is higher than a predetermined noise threshold and / or input that exceeds a predetermined duration).

[0136] In some embodiments, the event classifier 270 also includes a hit view determination module 272 and / or an active event recognizer determination module 273.

[0137] When the touch-sensitive display 212 displays more than one view, the hit view determination module 272 provides a software process for determining where within one or more views a sub-event has occurred. A view is made up of controls and other elements that the user is able to see on the display.

[0138] Another aspect of a user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the respective application) in which a touch is detected corresponds to a programmed level within the application's programmatic hierarchy or view hierarchy. For example, the lowest level view in which a touch is detected is referred to as the hit view, and the set of events that are correctly entered are determined based at least in part on the hit view of the initial touch that begins the touch-based gesture.

[0139] Hit view determination module 272 receives information related to touch-based gesture sub-events. When an application has multiple views organized in a hierarchy, hit view determination module 272 identifies the hit view as the lowest view in the hierarchy that should handle the sub-event. In most cases, the hit view is the lowest level view in which the sub-event (e.g., the first sub-event in a sequence of sub-events forming an event or potential event) originated. Once the hit view is identified by hit view determination module 272, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0140] Active event recognizer determination module 273 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 273 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 273 determines that all views that include the physical location of the sub-events are actively engaged views, and thus determines that all actively engaged views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, higher views in the hierarchy will remain actively engaged views.

[0141] Event dispatcher module 274 dispatches event information to event recognizers (e.g., event recognizers 280). In embodiments that include active event recognizer determination module 273, event dispatcher module 274 delivers event information to the event recognizers determined by active event recognizer determination module 273. In some embodiments, event dispatcher module 274 stores event information in an event queue that is retrieved by respective event receivers 282.

[0142] In some embodiments, operating system 226 includes event classifier 270. Alternatively, application 236-1 includes event classifier 270. In yet another embodiment, event classifier 270 is a standalone module, or part of another module stored in memory 202, such as contact / motion module 230.

[0143] In some embodiments, application 236-1 includes a plurality of event handlers 290 and one or more application views 291, each of which includes instructions for handling touch events that occur within a respective view of the user interface of the application. Each application view 291 of application 236-1 includes one or more event recognizers 280. Typically, a respective application view 291 includes multiple event recognizers 280. In other embodiments, one or more of event recognizers 280 are part of a separate module, such as a user interface toolkit (not shown) or a higher level object from which the application 236-1 inherits methods and other attributes. In some embodiments, a respective event handler 290 includes one or more of a data updater 276, an object updater 277, a GUI updater 278, and / or event data 279 received from event sorter 270. Event handler 290 utilizes or invokes the data updater 276, object updater 277, and GUI updater 278 in order to update the application internal state 292. Alternatively, one or more of the event handler 290, data updater 276, object updater 277, and GUI updater 278 can be included in one or more application views 291.

[0144] A respective event recognizer 280 receives event information (e.g., event data 279) from event sorter 270 and identifies an event from the event information. Event recognizer 280 includes an event receiver 282 and an event comparator 284. In some embodiments, event recognizer 280 also includes at least a subset of metadata 283 and event delivery instructions 288 (which includes sub-event delivery instructions).

[0145] Event receiver 282 receives event information from event sorter 270. The event information includes information about a sub-event, such as a touch or touch movement. Additionally, the event information also includes information about other touches such as one or more touches that occur while the sub-event is occurring. In some embodiments, event information includes information about a device (e.g., device 100) from which the sub-event originates. In some embodiments, event information includes information generated in accordance with previously occurring events (e.g., events that occurred a predetermined amount of time before the sub-event was received).

[0146] Event comparator 284 compares event information to predefined event or sub-event definitions, and based on the comparison, determines an event or sub-event, or determines or updates a state of an event or sub-event. In some embodiments, event comparator 284 includes event definitions 286. Event definitions 286 contain definitions of events (e.g., predefined sequences of sub-events), such as event 1 (287-1), event 2 (287-2), and other events. In some embodiments, sub-events in an event (287) include, for example, touch begin, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (287-1) is a double tap on a displayed object. For example, a double tap includes a first touch (touch begin) on a displayed object for a predetermined duration, a first lift off (touch end) for a predetermined duration, a second touch (touch begin) on a displayed object for a predetermined duration, and a second lift off (touch end) for a predetermined duration. In another example, the definition of event 2 (287-2) is a drag on a displayed object. For example, a drag includes a touch (or contact) on a displayed object for a predetermined duration, movement of the touch on the touch-sensitive display 212, and a lift off of the touch (touch end). In some embodiments, an event also includes information for one or more associated event handlers 290.

[0147] In some embodiments, event definitions 287 include definitions of events for respective user interface objects. In some embodiments, event comparator 284 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view that displays three user interface objects on touch-sensitive display 212, when a touch is detected on touch-sensitive display 212, event comparator 284 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a respective event handler 290, event comparator uses the results of the hit test to determine which event handler 290 should be activated. For example, event comparator 284 selects the event handler that is associated with the sub-event and the object that triggered the hit test.

[0148] In some embodiments, the definition of a respective event (287) also includes a deferred action that delays delivery of event information until it has been determined that a sequence of sub-events does or does not correspond to an event type of the event recognizer.

[0149] When a respective event recognizer 280 determines that a sub-event sequence does not match any event in the event definitions 286, the respective event recognizer 280 enters an event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0150] In some embodiments, the respective event recognizers 280 include metadata 283 having configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery to the actively engaged event recognizer. In some embodiments, the metadata 283 includes configurable properties, flags, and / or lists that indicate how the event recognizers interact or are able to interact with each other. In some embodiments, the metadata 283 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0151] In some embodiments, when one or more particular sub-events of an event are recognized, the respective event recognizer 280 activates an event handler 290 associated with the event. In some embodiments, the respective event recognizer 280 delivers event information associated with the event to the event handler 290. Activating an event handler 290 is distinct from sending (and deferring sending) sub-events to a respective hit view. In some embodiments, the event recognizer 280 throws a token associated with the recognized event, and an event handler 290 associated with the token picks up the token and performs a pre-defined process.

[0152] In some embodiments, the event delivery instructions 288 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions deliver the event information to an event handler associated with the sub-event sequence or to the actively engaged view. The event handler associated with the sub-event sequence or with the actively engaged view receives the event information and performs a pre-determined process.

[0153] In some embodiments, data updater 276 creates and updates data used in application 236-1. For example, data updater 276 updates the telephone number used in contacts module 237, or stores a video file used in a video player module. In some embodiments, object updater 277 creates and updates objects used in application 236-1. For example, object updater 277 creates a new user interface object or updates the position of a user interface object. GUI updater 278 updates the GUI. For example, GUI updater 278 prepares display information and sends it to graphics module 232 for display on a touch-sensitive display.

[0154] In some embodiments, event handler 290 includes data updater 276, object updater 277, and GUI updater 278, or has access to them and utilizes them to generate updates. In some embodiments, data updater 276, object updater 277, and GUI updater 278 are included in a single module of application 236-1 or application view 291. In other embodiments, they are included in two or more software modules.

[0155] It should be understood that the above discussion regarding event handling of user touches on touch-sensitive displays also applies to other forms of user inputs to the multi-function device 200 using input devices 2910 and 2912. Not all user inputs may be touching on a touch screen. Other user input means are optionally used, such as mouse-based or cursor-based input, in conjunction with a single or multiple keyboard presses or holds, touchpad contact movements such as taps, drags, scrolls, etc., stylus input, device movement, spoken commands, detected eye movements, biometric inputs, and / or any combination thereof.

[0156] FIG. 3The portable multifunction device 200 with touch screen 212 is, optionally, a handheld device. Touch screen 212 optionally uses LCD (liquid crystal display) technology or OLED (organic light emitting diode) technology, although other technologies can be used. The touch screen 212 and the touch screen controller 256 optionally each include a plurality of pixels. The touch screen 212 optionally displays one or more graphics from the graphics memory 262 in accordance with various embodiments. Some embodiments include a touch screen controller 256 coupled with the touch screen 212. In some embodiments, the touch screen 212 displays one or more graphical items of a graphical user interface (GUI), various embodiments of which are described herein.

[0157] Device 200 also includes one or more physical buttons, such as "home" or menu button 304. As described previously, the menu button 304 is used to navigate to any application 236 in a set of applications on device 200. Alternatively, the menu button is implemented as a soft key in a GUI displayed by

[0158] In some embodiments, device 200 includes touch screen 212, menu button 304, push button 306 for powering the device on / off and placing device 200 in a sleep state, one or more volume adjustment buttons 308, a user identity module (SIM) card slot 310, a headphone jack 312, and a dock / charging external port 224. Push button 306 is, optionally, used to

[0159] FIG. 4This is a block diagram of an exemplary multi-functional device having a display and a touch-sensitive surface according to some embodiments. Device 400 need not be portable. In some embodiments, device 400 is a laptop computer, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a children's learning toy), gaming system, or control device (e.g., a home controller or industrial controller). Device 400 typically includes one or more processing units (CPUs) 410, one or more network or other communication interfaces 460, memory 470, and one or more communication buses 420 for interconnecting these components. Communication bus 420 optionally includes circuitry (sometimes referred to as a chipset) that interconnects system components and controls communication between system components. Device 400 includes an input / output (I / O) interface 430 having a display 440, which is typically a touchscreen display. I / O interface 430 also optionally includes a keyboard and / or mouse (or other pointing device) 450 and a touchpad 455, and a haptic output generator 457 for generating haptic output on device 400 (e.g., similar to the reference above). FIG. 2A The one or more tactile output generators 267 and sensors 459 (e.g., optical sensors, accelerometers, proximity sensors, touch sensors, and / or contact intensity sensors similar to those mentioned above) FIG. 2A The one or more contact strength sensors 265 mentioned above). Memory 470 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 470 optionally includes one or more storage devices located remotely from CPU 410. In some embodiments, memory 470 stores data with portable multifunction device 200 (…). FIG. 2A The memory 470 stores programs, modules, and data structures similar to those in the memory 202 of the portable multifunction device 200, or subsets thereof. Additionally, the memory 470 optionally stores additional programs, modules, and data structures not present in the memory 202 of the portable multifunction device 200. For example, the memory 470 of the device 400 optionally stores a drawing module 480, a rendering module 482, a word processing module 484, a website creation module 486, a disk editing module 488, and / or a spreadsheet module 490, while the portable multifunction device 200 ( FIG. 2A The memory 202 optionally does not store these modules.

[0160] FIG. 4Each of the above identified elements in the memory 470 can store data used by various applications run by the device 400. For example, the memory 470 can store software elements 472, 474, 476, 478, 480 corresponding to the operating system 464, the contact module 266, the telephone module 238, the videos and music player module 252, the IM module 241, and other modules (not shown) for the device 400. In some embodiments, the software elements 472, 474, 476, 478, 480 can also include instructions for implementing the functionality of the device 400, including the functionality of the user interface 500.

[0161] Attention is now directed towards embodiments of user interfaces that can be implemented on portable multifunction devices 200.

[0162] FIG. 5A An exemplary user interface that includes an application menu on a portable multifunction device 200 is shown. Similar user interfaces can be implemented on device 400. In some embodiments, user interface 500 includes the following elements, or a subset or superset thereof:

[0163] Signal strength indicators 502 for wireless communications such as cellular signals and Wi-Fi signals;

[0164] Time 504;

[0165] Bluetooth indicator 505;

[0166] Battery status indicator 506;

[0167] Dock 508 with icons for frequently used applications, such as:

[0168] • an icon 516 for telephone module 238 labeled "Phone," which optionally includes an indicator 514 of the number of missed calls or voice mails;

[0169] • an icon 518 for email client module 240 labeled "Mail," which optionally includes an indicator 510 of the number of unread emails;

[0170] • an icon 520 for browser module 247 labeled "Browser," and

[0171] • an icon 522 for videos and music player module 252 (also referred to as iPod (Apple Inc.'s trademark) module 252) labeled "iPod," and

[0172] • icons for other applications, such as:

[0173] • an icon 524 for IM module 241 labeled "Messages," and

[0174] o an icon 526 for the calendar module 248, labeled "Calendar";

[0175] o an icon 528 for the image management module 244, labeled "Photos";

[0176] o an icon 530 for the camera module 243, labeled "Camera";

[0177] o an icon 532 for the online video module 255, labeled "Online Videos";

[0178] o an icon 534 for the stock widget 249-2, labeled "Stocks";

[0179] o an icon 536 for the maps module 254, labeled "Maps";

[0180] o an icon 538 for the weather widget 249-1, labeled "Weather";

[0181] o an icon 540 for the alarm clock widget 249-4, labeled "Clock";

[0182] o an icon 542 for the fitness support module 242, labeled "Fitness Support";

[0183] o an icon 544 for the notes module 253, labeled "Notes"; and

[0184] o an icon 546 for setting the applications or modules, labeled "Settings", which provides access to settings for the device 200 and its various applications 236.

[0185] It should be noted that FIG. 5A The icon labels shown are merely examples. For example, the icon 522 for the video and music player module 252 is optionally labeled "Music" or "Music Player." Other labels for various application icons are optionally used in some embodiments. In some embodiments, the label for a particular application icon includes the name of the application that the particular application icon corresponds to. In some embodiments, the label for a particular application icon is different than the name of the application that the particular application icon corresponds to.

[0186] FIG. 5B A device (e.g., device 500) is shown with a touch-sensitive surface 551 (e.g., a touchpad or touch screen 455) that is separate from the display 550 (e.g., FIG. 4 in FIG. 5B). The device 500 optionally includes one or more sensors 560. FIG. 5B shows a process 570 for detecting a gesture on the touch-sensitive surface 551. FIG. 4An exemplary user interface on device 400. Device 400 also optionally includes one or more contact intensity sensors (e.g., one or more sensors in sensor 459) for detecting the intensity of contact on tactile surface 551 and / or one or more tactile output generators 457 for generating tactile output for the user of device 400.

[0187] While some examples of input on a reference touchscreen display 212 (which combines a touch-sensitive surface and a display) are given in the following examples, in some implementations, the device detects input on a touch-sensitive surface separate from the display, such as... FIG. 5B As shown in the diagram. In some embodiments, the touch-sensitive surface (e.g., FIG. 5B 551) has a spindle (e.g., on the display (e.g., 550) that is aligned with the main axis on the display (e.g., FIG. 5B The spindle corresponding to 553 in the figure (e.g., FIG. 5B (552 in the example). According to these embodiments, the device detects the position corresponding to the corresponding position on the display (e.g., in the example). FIG. 5B In the middle, 560 corresponds to 568 and 562 corresponds to 570) the contact with the touch-sensitive surface 551 at the location (e.g., FIG. 5B (560 and 562 in the example). Thus, on touch-sensitive surfaces (e.g., FIG. 5B 551 in the middle) and the display of a multi-functional device (e.g., FIG. 5B When 550 is separated from 560, user input detected by the device on the touch-sensitive surface (e.g., contact with 560 and 562 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods may be optionally used for other user interfaces described herein.

[0188] Additionally, while the examples below are given primarily with reference to finger input (e.g., finger touch, single-finger tap gesture, finger swipe gesture), it should be understood that in some implementations, one or more of these finger inputs may be replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture may optionally be replaced by a mouse click (e.g., instead of a touch), followed by movement of the cursor along the swipe path (e.g., instead of movement of the touch). Similarly, a tap gesture may optionally be replaced by a mouse click while the cursor is over the location of the tap gesture (e.g., instead of detection of touch, followed by cessation of touch detection). Likewise, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice may optionally be used simultaneously, or mouse and finger touch may optionally be used simultaneously.

[0189] FIG. 6AAn example personal electronic device 600 is shown. Device 600 includes a body 602. In some embodiments, device 600 includes some or all of the features described with respect to devices 200 and 400 (e.g., FIG. 2A-FIG. 4 In some embodiments, device 600 has a touch-sensitive display 604, referred to hereafter as touch screen 604. Alternatively, device 600 has a display and a touch-sensitive surface. As with device 200 and 400, in some embodiments, touch screen 604 (or the touch- sensitive surface) has one or more intensity sensors to detect intensity of contacts (e.g., touches) on the touch screen 604. One or more intensity sensors of touch screen 604 (or the touch- sensitive surface) provide output data that represents the intensity of a touch. The user interface of device 600 responds to touches of different intensities by different user interface operations, which means that different intensities of touches can invoke different user interface operations on device 600.

[0190] Techniques for detecting and processing touch intensity can be found, for example, in related applications: International Patent Application Serial No. PCT / US2013 / 040061, entitled "Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application," filed May 8, 2013, and International Patent Application Serial No. PCT / US2013 / 069483, entitled "Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships," filed November 11, 2013, each of which is hereby incorporated by reference in its entirety.

[0191] In some embodiments, device 600 has one or more input mechanisms 606 and 608. Input mechanisms 606 and 608, if included, are physical. Examples of physical input mechanisms include a depressible button and a rotatable mechanism. In some embodiments, device 600 has one or more attachment mechanisms. Such attachment mechanisms, if included, can allow device 600 to be attached to, for example, a hat, glasses, earrings, a necklace, a shirt, a jacket, a bracelet, a watchband, a necklace, pants, a belt, shoes, a purse, a backpack, and the like. These attachment mechanisms allow a user to wear device 600.

[0192] FIG. 6BAn example personal electronic device 600 is shown. In some embodiments, device 600 includes some or all of the components described with respect to FIG. 2A 、 FIG. 2B and FIG. 4 . Device 600 has bus 612 that operatively couples I / O section 614 with one or more computer processors 616 and memory 618. I / O section 614 is connected to display 604, which can have touch-sensitive component 622, and optionally also touch intensity sensor component 624. Additionally, I / O section 614 is connected with communication unit 630 for receiving application and operating system data, such as data used in

[0193] In some examples, input mechanism 608 is a microphone. Personal electronic device 600 includes various sensors, such as GPS sensor 632, accelerometer 634, directional sensor 640 (e.g., compass), gyroscope 636, motion sensor 638, and / or a combination thereof, all of which can be operatively connected to I / O section 614.

[0194] Memory 618 of personal electronic device 600 is a non-transitory computer-readable storage medium for storing computer-executable instructions, for example, that, when executed by one or more computer processors 616, for example, cause the computer processors to perform the techniques and processes described above. The computer-executable instructions are also, for example, stored and / or transported within any non-transitory computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch instructions from the instruction execution system, apparatus, or device and execute instructions. The personal electronic device 600 is not limited to FIG. 6B the components and configurations of the components described above, but can include other or additional components in multiple configurations.

[0195] As used herein, the term "affordance" refers to a graphical user interface object that is capable of being FIG. 2A 、 FIG. 4 、 FIG. 6A to FIG. 6B and FIG. 8A to FIG. 8P displayed on a display screen of a device 200, 400, 600, and / or 800, for example, and that is user-interactive. For example, an image (e.g., an icon), a button, and text (e.g., a hyperlink) each constitute an affordance.

[0196] As used herein, the term “focus selector” refers to an input element that indicates a current portion of a user interface that is actively being edited by a user. In some implementations that include a cursor or other location marker, the cursor acts as a “focus selector,” so that when an input (e.g., a press input) is detected on a touchpad 455 in a computing device 500 or a touch-sensitive surface 551 in a computing device 500

[0199] , the particular user interface element (e.g., a button, window, slider, or other user interface element) that is under the cursor when the input (e.g., a press input) is detected is adjusted according to the detected input. In some implementations that include a touch screen display (e.g., a touch-sensitive display system 212 in a computing device 500 or a touch screen 212 in a computing device 500

[0200] that enables direct interaction with user interface elements on the touch screen display, the location where contact is detected on the touch screen displays acts as a “focus selector,” so that when an input (e.g., a press input by a contact) is detected on the touch screen display at a location that corresponds to a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations, focus is moved from one region of a user interface to another region of the user interface without corresponding movement of a cursor or movement of a contact on a touch screen display (e.g., by using a tab key or arrow keys to move focus from one button to another button); in these implementations, the focus selector moves according to movement of focus between different regions of a user interface. Not all user interface elements are focusable. Not all user interface elements that are focusable are integrated into a single scrolling region. In some implementations, a focus selector is moved according to movement of a contact on a touch screen display. In some implementations, a focus selector is moved according to movement of a cursor on a display. In some implementations, a focus selector is moved according to movement of a cursor on a display and movement of a contact on a touch screen display. In some implementations, a focus selector is moved according to movement of a cursor on a display and movement of a contact on a touch screen display, and movement of a contact on a touch screen display takes precedence over movement of a cursor on a display.

[0201] As used in the specification and claims, the term "feature strength" of a contact refers to a characteristic of the contact that is based on one or more strengths of the contact. In some embodiments, the feature strength is based on a plurality of strength samples. The feature strength is optionally based on a predefined number of strength samples or a set of strength samples that are acquired during a predetermined period of time (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) with respect to a predefined event (e.g., after detecting the contact, before detecting liftoff of the contact, before or after detecting the contact starting to move, before detecting the end of the contact, before detecting an increase in strength of the contact, before detecting a decrease in strength of the contact, and / or the like). The feature strength of the contact is optionally based on one or more of: a maximum value of the contact strength, a mean value of the contact strength, a median value of the contact strength, a value at the 10th percentile of the contact strength, a half-maximum value of the contact strength, a 90th maximum value of the contact strength, and / or the like. In some embodiments, the duration of the contact is used in determining the feature strength (e.g., when the feature strength is an average of the strength of the contact over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds includes a first strength threshold and a second strength threshold. In this example, a contact with a feature strength that does not exceed the first threshold results in a first operation, a contact with a feature strength that exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact with a feature strength that exceeds the second threshold results in a third operation. In some embodiments, the comparison between the feature strength and the one or more thresholds is used to determine whether one or more operations are to be performed (e.g., whether a respective operation is performed or forgoen), rather than to determine whether a first operation or a second operation is performed.

[0198] In some embodiments, a portion of a gesture is identified for use in determining a feature strength. For example, a touch-sensitive surface receives a continuous swipe contact that transitions from a starting location and reaches an ending location at which the strength of the contact increases. In this example, the feature strength of the contact at the ending location is based only on a portion of the continuous swipe contact, rather than the entire swipe contact (e.g., the portion of the swipe contact that is at the ending location only). In some embodiments, a smoothing algorithm is applied to the strength of the swipe contact prior to determining the feature strength of the contact. For example, the smoothing algorithm optionally includes one or more of: an unweighted sliding average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow spikes or dips in the strength of the swipe contact for purposes of determining the feature strength.

[0199] The intensity of contact on the touch-sensitive surface is characterized relative to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, a light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or trackpad. In some embodiments, a deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from that typically associated with clicking a button of a physical mouse or trackpad. In some embodiments, when a contact of feature intensity below the light press intensity threshold (e.g., and above the nominal contact detection intensity threshold, below which contacts are no longer detected) is detected, the device will move a focus selector in accordance with movement of the contact on the touch-sensitive surface without performing an operation associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise stated, these intensity thresholds are consistent between different sets of user interface figures.

[0200] An increase in contact feature intensity from an intensity below the light press intensity threshold to an intensity between the light press intensity threshold and the deep press intensity threshold is sometimes referred to as a "light press" input. An increase in contact feature intensity from an intensity below the deep press intensity threshold to an intensity above the deep press intensity threshold is sometimes referred to as a "deep press" input. An increase in contact feature intensity from an intensity below the contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light press intensity threshold is sometimes referred to as detecting a contact on the touch surface. A decrease in contact feature intensity from an intensity above the contact detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting liftoff of a contact from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0201] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a respective press input or in response to detecting a respective press input performed with a respective contact (or multiple contacts), where the respective press input is detected based at least in part on detecting an increase in intensity of the contact (or multiple contacts) above a press input intensity threshold. In some embodiments, a respective operation is performed in response to detecting an increase in intensity of the respective contact above the press input intensity threshold (e.g., a "downstroke" of the respective press input). In some embodiments, a press input includes an increase in intensity of a respective contact above a press input intensity threshold and a subsequent decrease in intensity of the contact below the press input intensity threshold, and a respective operation is performed in response to detecting the subsequent decrease in intensity of the respective contact below the press input threshold (e.g., an "upstroke" of the respective press input).

[0202] In some embodiments, the device employs intensity hysteresis to avoid an unintended input sometimes referred to as "bouncing," in which the device defines or selects a hysteresis intensity threshold that has a predefined relationship to the press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in intensity of a respective contact above the press input intensity threshold and a subsequent decrease in intensity of the contact below a hysteresis intensity threshold corresponding to the press input intensity threshold, and a respective operation is performed in response to detecting the subsequent decrease in intensity of the respective contact below the hysteresis intensity threshold (e.g., an "upstroke" of the respective press input). Similarly, in some embodiments, a press input is detected only when the device detects an increase in contact intensity from an intensity at or below the hysteresis intensity threshold to an intensity at or above the press input intensity threshold, and optionally a subsequent decrease in contact intensity to an intensity at or below the hysteresis intensity, and a respective operation is performed in response to detecting the press input (e.g., an increase in contact intensity or a decrease in contact intensity depending on the circumstances).

[0203] For ease of explanation, optionally, descriptions of operations performed in response to a press input associated with a press input intensity threshold or in response to a gesture that includes a press input are triggered in response to detecting any of a variety of conditions: an increase in contact intensity above the press input intensity threshold, an increase in contact intensity from an intensity below the hysteresis intensity threshold to an intensity above the press input intensity threshold, a decrease in contact intensity below the press input intensity threshold, and / or a decrease in contact intensity below a hysteresis intensity threshold corresponding to the press input intensity threshold. Additionally, in examples in which an operation is described as being performed in response to detecting a decrease in intensity of a contact below a press input intensity threshold, optionally the operation is performed in response to detecting a decrease in intensity of the contact below a hysteresis intensity threshold corresponding to and less than the press input intensity threshold.

[0204] 3. Digital assistant system

[0205] FIG. 7A A block diagram of a digital assistant system 700 is shown in accordance with various examples. In some examples, the digital assistant system 700 is implemented on a standalone computer system. In some examples, the digital assistant system 700 is distributed across multiple computers. In some examples, some of the modules and functionality of the digital assistant are divided into a server portion and a client portion, where the client portion is located on one or more user devices (e.g., devices 104, 122, 200, 400, 600, or 800) and communicates with the server portion (e.g., server system 108) over one or more networks, e.g., as shown in FIG. 1 In some examples, the digital assistant system 700 is a FIG. 1The specific implementation of the server system 108 (and / or DA server 106) shown is illustrated. It should be noted that the digital assistant system 700 is merely an example of a digital assistant system, and the digital assistant system 700 may have more or fewer components than shown, combine two or more components, or have different configurations or layouts of components. FIG. 7A The various components shown are implemented in hardware, software instructions for execution by one or more processors, firmware (including one or more signal processing integrated circuits and / or application-specific integrated circuits), or a combination thereof.

[0206] The digital assistant system 700 includes a memory 702, an input / output (I / O) interface 706, a network communication interface 708, and one or more processors 704. These components can communicate with each other via one or more communication buses or signal lines 710.

[0207] In some examples, memory 702 includes non-transitory computer-readable media, such as high-speed random access memory and / or non-volatile computer-readable storage media (e.g., one or more disk storage devices, flash memory devices or other non-volatile solid-state memory devices).

[0208] In some examples, I / O interface 706 couples input / output devices 716 of digital assistant system 700, such as a display, keyboard, touchscreen, and microphone, to user interface module 722. I / O interface 706, together with user interface module 722, receives user input (e.g., voice input, keyboard input, touch input, etc.) and processes this input accordingly. In some examples, such as when the digital assistant is implemented on a standalone user device, digital assistant system 700 includes a user interface module 722. FIG. 2A , FIG. 4 , FIG. 6A to FIG. 6B as well as FIG. 8A to FIG. 8P The components and I / O communication interfaces described in devices 200, 400, 600, or 800, respectively. In some examples, digital assistant system 700 represents the server portion of a digital assistant implementation and can interact with the user through a client-side portion located on a user device (e.g., device 104, 200, 400, 600, or 800).

[0209] In some examples, the network communication interface 708 includes one or more wired communication ports 712 and / or wireless transmission and reception circuitry 714. The one or more wired communication ports receive and send communication signals via one or more wired interfaces, such as Ethernet, Universal Serial Bus (USB), FIREWIRE, etc. The wireless circuitry 714 receives and sends RF signals and / or optical signals from / to communication networks and other communication devices. The wireless communication uses any of a plurality of communications standards, protocols, and technologies, such as GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol. The network communication interface 708 enables communications between the digital assistant system 700 and other devices over a network, such as the Internet, an intranet, and / or a wireless network, such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN).

[0210] In some examples, the memory 702 or the computer-readable storage medium of the memory 702 stores programs, modules, instructions, and data structures, including all or a subset of the following: an operating system 718, a communication module 720, a user interface module 722, one or more applications 724, and a digital assistant module 726. In particular, the memory 702 or the computer-readable storage medium of the memory 702 stores instructions for performing the processes described above. The one or more processors 704 execute these programs, modules, and instructions, and read data from and write data to the data structures.

[0211] The operating system 718 (e.g., Darwin, RTXC, LINUX, UNIX, iOS, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates wireless communications between various hardware and software components.

[0212] The communication module 720 facilitates communication between the digital assistant system 700 and other devices over the network communication interface 708. For example, the communication module 720 communicates with the RF circuitry 208 of the electronic devices (such as the devices 200, 400, or 600 shown in FIGS. 2, 4, and 6, respectively). FIG. 2A , FIG. 4 , FIG. 6A to FIG. 6B The communication module 720 also includes various components for processing data received by the wireless circuitry 714 and / or the wired communication ports 712.

[0213] The user interface module 722 receives commands and / or inputs from a user (e.g., from a keyboard, touch screen, pointing device, controller, and / or microphone) via the I / O interface 706 and generates user interface objects on a display. The user interface module 722 also prepares and transmits output (e.g., speech, sound, animation, text, icons, vibrations, haptic feedback, illumination, etc.) to the user via the I / O interface 706 (e.g., through a display, audio channel, speaker, touchpad, etc.).

[0214] The applications 724 include programs and / or modules configured for execution by the one or more processors 704. For example, if the digital assistant system is implemented on a standalone user device, the applications 724 include user applications such as games, calendar applications, navigation applications, or email applications. If the digital assistant system 700 is implemented on a server, the applications 724 include, for example, resource management applications, diagnostic applications, or scheduling applications.

[0215] The memory 702 also stores a digital assistant module 726 (or server portion of a digital assistant). In some examples, the digital assistant module 726 includes the following sub-modules or a subset or superset thereof: an input / output processing module 728, a speech-to-text (STT) processing module 730, a natural language processing module 732, a dialog flow processing module 734, a task flow processing module 736, a service processing module 738, and a speech synthesis processing module 740. Each of these modules has access to one or more of the following systems or data and models of the digital assistant module 726 or a subset or superset thereof: a knowledge ontology 760, a vocabulary index 744, user data 748, task flow models 754, service models 756, and an ASR system 758.

[0216] In some examples, using the processing modules, data, and models implemented in the digital assistant module 726, the digital assistant can perform at least some of the following: convert speech input into text; identify a user intent expressed in natural language input received from a user; proactively elicit and obtain information needed to fully infer the user intent (e.g., disambiguate words, games, intents, etc.); determine a task flow for satisfying the inferred intent; and execute the task flow to satisfy the inferred intent.

[0217] In some examples, as shown in FIG. 7B the I / O processing module 728 can interact with a user through the I / O devices 716 in FIG. 7A or through the I / O devices 716 in FIG. 7AThe network communication interface 708 in the user device (e.g., device 104, device 200, device 400, or device 600) interacts with the user device to obtain user input (e.g., speech input) and provide responses to the user input (e.g., as speech output). The I / O processing module 728 optionally obtains contextual information associated with the user input from the user device along with or shortly after receiving the user input. The contextual information includes user-specific data, vocabulary, and / or preferences related to the user input. In some examples, the contextual information also includes the software state and hardware state of the user device at the time the user request is received, and / or information related to the user's surrounding environment at the time the user request is received. In some examples, the I / O processing module 728 also sends follow-up questions to the user related to the user request and receives answers from the user. When a user request is received by the I / O processing module 728 and the user request includes speech input, the I / O processing module 728 forwards the speech input to the STT processing module 730 (or speech recognizer) for speech-to-text conversion.

[0218] The STT processing module 730 includes one or more ASR systems 758. The one or more ASR systems 758 can process speech input received through the I / O processing module 728 to produce recognition results. Each ASR system 758 includes a front-end speech preprocessor. The front-end speech preprocessor extracts representative features from the speech input. For example, the front-end speech preprocessor performs a Fourier transform on the speech input to extract spectral features characterizing the speech input as a sequence of representative multi-dimensional vectors. In addition, each ASR system 758 includes one or more speech recognition models (e.g., acoustic models and / or language models) and implements one or more speech recognition engines. Examples of speech recognition models include hidden Markov models, Gaussian mixture models, deep neural network models, n-gram language models, and other statistical models. Examples of speech recognition engines include engines based on dynamic time warping and engines based on weighted finite-state transducers (WFSTs). The extracted representative features of the front-end speech preprocessor are processed using the one or more speech recognition models and the one or more speech recognition engines to produce intermediate recognition results (e.g., phonemes, phoneme strings, and sub-words) and ultimately text recognition results (e.g., words, word strings, or symbol sequences). In some examples, the speech input is processed at least in part by a third-party service or on a user's device (e.g., device 104, device 200, device 400, or device 600) to produce the recognition results. Once the STT processing module 730 produces a recognition result containing a text string (e.g., a word, or a sequence of words, or a sequence of symbols), the recognition result is passed to the natural language processing module 732 for intent inference. In some examples, the STT processing module 730 produces multiple candidate text representations of the speech input. Each candidate text representation is a sequence of words or symbols corresponding to the speech input. In some examples, each candidate text representation is associated with a speech recognition confidence score. Based on the speech recognition confidence scores, the STT processing module 730 ranks the candidate text representations and provides the n-best (e.g., the n-highest ranked) candidate text representations to the natural language processing module 732 for intent inference, where n is a predetermined integer greater than zero. For example, in one example, only the highest-ranked (n = 1) candidate text representation is delivered to the natural language processing module 732 for intent inference. As another example, the five highest-ranked (n = 5) candidate text representations are passed to the natural language processing module 732 for intent inference.

[0219] Further details regarding speech-to-text processing are described in U.S. Utility Patent Application Serial No. 13 / 236,942, entitled "Consolidating Speech Recognition Results," filed September 20, 2011, the entire disclosure of which is incorporated by reference herein.

[0220] In some examples, the STT processing module 730 includes a vocabulary of recognizable words and / or accesses that vocabulary via the speech-to-letter conversion module 731. Each vocabulary word is associated with one or more candidate pronunciations of a word represented in a speech recognition speech alphabet. Specifically, the vocabulary of recognizable words includes words associated with multiple candidate pronunciations. For example, the vocabulary includes words associated with... and The candidate pronunciation of / is associated with the word "tomato". Additionally, lexical words are associated with custom candidate pronunciations based on previous speech input from the user. These custom candidate pronunciations are stored in the STT processing module 730 and associated with a specific user via a user profile on the device. In some examples, candidate pronunciations are determined based on the spelling of the word and one or more linguistic and / or phonetic rules. In some examples, candidate pronunciations are generated manually, for example, based on known standard pronunciations.

[0221] In some examples, candidate pronunciations are ranked based on their prevalence. For example, candidate pronunciations... The ranking is higher than This is because the former is a more commonly used pronunciation (e.g., among all users, for users in a specific geographic region, or for any other suitable subset of users). In some examples, candidate pronunciations are ranked based on whether they are custom candidate pronunciations associated with a user. For example, custom candidate pronunciations rank higher than standard candidate pronunciations. This can be used to identify proper nouns with unique pronunciations that deviate from the canonical pronunciation. In some examples, candidate pronunciations are associated with one or more speech features such as geographic origin, country, or ethnicity. For example, candidate pronunciations... Associated with the United States, and candidate pronunciation The candidate pronunciations are associated with the United Kingdom. Furthermore, the ranking of candidate pronunciations is based on one or more characteristics of the user (e.g., geographic origin, country, ethnicity, etc.) stored in the user profile on the device. For example, it can be determined from the user profile that the user is associated with the United States. Based on the user's association with the United States, candidate pronunciations... (US-related) Comparable candidate pronunciations (Related to the UK) It ranks higher. In some examples, one of the ranked candidate pronunciations can be selected as the predicted pronunciation (e.g., the most likely pronunciation).

[0222] Upon receiving speech input, the STT processing module 730 is used (e.g., using an acoustic model) to determine the phonemes corresponding to the speech input, and then attempts (e.g., using a language model) to determine the words that match those phonemes. For example, if the STT processing module 730 first identifies a sequence of phonemes corresponding to a portion of the speech input... It can then subsequently determine that the sequence corresponds to the word "tomato" based on the vocabulary index 744.

[0223] In some examples, the STT processing module 730 uses fuzzy matching techniques to determine the words in the utterance. Thus, for example, the STT processing module 730 determines that the phoneme sequence corresponds to the word "tomato," even though that particular phoneme sequence is not a candidate phoneme sequence for that word.

[0224] The natural language processing module 732 of the digital assistant ("natural language processor") takes the n-best candidate text representations ("word sequences" or "symbol sequences") generated by the STT processing module 730 and attempts to associate each candidate text representation with one or more "actionable intents" recognized by the digital assistant. An "actionable intent" (or "user intent") represents a task that can be performed by the digital assistant and that can have an associated task flow implemented in the task flow model 754. The associated task flow is a series of programmed actions and steps that the digital assistant takes in order to perform the task. The range of capabilities of the digital assistant depends on the number and variety of task flows that have been implemented and stored in the task flow model 754, or in other words, on the number and variety of "actionable intents" recognized by the digital assistant. However, the effectiveness of the digital assistant also depends on the ability of the assistant to infer the correct "one or more actionable intents" from user requests expressed in natural language.

[0225] In some examples, in addition to the sequence of words or symbols obtained from the STT processing module 730, the natural language processing module 732 also receives, e.g., from the I / O processing module 728, contextual information associated with the user request. The natural language processing module 732 optionally uses the contextual information to disambiguate, supplement, and / or further qualify the information contained in the candidate text representations received from the STT processing module 730. The contextual information includes, e.g., user preferences, hardware and / or software states of the user device, sensor information collected prior to, during, or shortly after the user request, prior interactions (e.g., conversations) between the digital assistant and the user, etc. As described herein, in some examples, the contextual information is dynamic and varies with the time, location, content, and other factors of the conversation.

[0226] In some examples, natural language processing is based on, for example, a knowledge ontology 760. The knowledge ontology 760 is a hierarchical structure containing a number of nodes, each of which represents a "actionable intent" or an "attribute" related to one or more of an "actionable intent" or another "attribute." As described above, an "actionable intent" represents a task that a digital assistant is capable of performing, i.e., the task is "actionable" or can be performed. An "attribute" represents a parameter associated with a sub- aspect of an actionable intent or another attribute. Connections between actionable intent nodes and attribute nodes in the knowledge ontology 760 define how the parameters represented by the attribute nodes pertain to the tasks represented by the actionable intent nodes.

[0227] In some examples, the knowledge ontology 760 is composed of actionable intent nodes and attribute nodes. Within the knowledge ontology 760, each actionable intent node is connected directly to or through one or more intermediate attribute nodes to one or more attribute nodes. Similarly, each attribute node is connected directly to or through one or more intermediate attribute nodes to one or more actionable intent nodes. For example, as shown in FIG. 7C the knowledge ontology 760 includes a "restaurant reservation" node (i.e., an actionable intent node). The attribute nodes "cuisine," "price range," "phone number," and "location" are all sub-nodes of the attribute node "restaurant" and are each linked to the "restaurant reservation" node (i.e., the actionable intent node) through the intermediate attribute node "restaurant."

[0228] Further, the attribute nodes "cuisine," "price range," "phone number," and "location" are sub-nodes of the attribute node "restaurant" and are each linked to the "restaurant reservation" node (i.e., the actionable intent node) through the intermediate attribute node "restaurant." As another example, as shown in FIG. 7C the knowledge ontology 760 also includes a "set reminder" node (i.e., another actionable intent node). The attribute nodes "date / time" (for setting a reminder) and "topic" (for the reminder) are each connected to the "set reminder" node. Because the attribute "date / time" is related to both the task of making a restaurant reservation and the task of setting a reminder, the attribute node "date / time" is connected to both the "restaurant reservation" node and the "set reminder" node in the knowledge ontology 760.

[0229] An actionable intent node, along with its linked attribute nodes, is described as a "domain." In this discussion, each domain is associated with a respective actionable intent and refers to a set of nodes (and relationships between these nodes) associated with a particular actionable intent. For example, FIG. 7CThe knowledge base 760 shown in FIG. 8 includes an example of a restaurant reservation domain 762 and an example of a reminder domain 764 within the knowledge base 760. The restaurant reservation domain includes an executable intent node "restaurant reservation," attribute nodes "restaurant," "date / time," and "party size," and sub-attribute nodes "cuisine," "price range," "phone number," and "location." The reminder domain 764 includes an executable intent node "set a reminder" and attribute nodes "subject" and "date / time." In some examples, the knowledge base 760 is composed of multiple domains. Each domain shares one or more attribute nodes with one or more other domains. For example, in addition to the restaurant reservation domain 762 and the reminder domain 764, the "date / time" attribute node is also associated with many different domains (e.g., a travel scheduling domain, a travel reservation domain, a movie ticket domain, etc.).

[0230] Although FIG. 7C While two example domains are shown within the knowledge base 760, other domains include, for example, "find a movie," "initiate a phone call," "find directions," "schedule a meeting," "send a message," and "provide answers to questions," "read a list," "provide navigation instructions," "provide instructions for a task," etc. The "send a message" domain is associated with a "send a message" executable intent node and further includes attribute nodes such as "one or more recipients," "message type," and "message body." The attribute node "recipient" is further defined, for example, by sub-attribute nodes such as "recipient name" and "message address."

[0231] In some examples, the knowledge base 760 includes all domains (and thus executable intents) that the digital assistant is capable of understanding and acting upon. In some examples, the knowledge base 760 is modified, such as by adding or removing entire domains or nodes, or by modifying relationships between nodes within the knowledge base 760.

[0232] In some examples, nodes associated with multiple related executable intents are clustered under a "super domain" in the knowledge base 760. For example, a "travel" super domain includes a cluster of attribute nodes and executable intent nodes related to travel. The executable intent nodes related to travel include "flight reservation," "hotel reservation," "car rental," "route planning," "find points of interest," etc. The executable intent nodes under the same super domain (e.g., the "travel" super domain) have multiple attribute nodes in common. For example, the executable intent nodes for "flight reservation," "hotel reservation," "car rental," "get directions," and "find points of interest" share one or more of the attribute nodes "start location," "destination," "departure date / time," "arrival date / time," and "party size."

[0233] In some examples, each node in the knowledge ontology 760 is associated with a set of words and / or phrases related to the attribute or actionable intent represented by the node. The respective set of words and / or phrases associated with each node is a so-called "vocabulary" associated with the node. The respective set of words and / or phrases associated with each node is stored in the vocabulary index 744 associated with the attribute or actionable intent represented by the node. For example, returning to the example above FIG. 7B The vocabulary associated with the node for the "restaurant" attribute includes words such as "food," "drinks," "cuisine," "hungry," "eat," "pizza," "fast food," "meal," and the like. As another example, the vocabulary associated with the node for the "initiate a phone call" actionable intent includes words and phrases such as "call," "make a call," "dial," "talk to," "call that number," "call," and the like. The vocabulary index 744 optionally includes words and phrases in different languages.

[0234] The natural language processing module 732 receives the candidate text representations (e.g., one or more text strings or one or more symbol sequences) from the STT processing module 730 and, for each candidate representation, determines which nodes in the knowledge ontology 760 the words in the candidate text representation refer to. In some examples, if a word or phrase in the candidate text representation is found to be associated (via the vocabulary index 744) with one or more nodes in the knowledge ontology 760, the word or phrase "triggers" or "activates" those nodes. Based on the number and / or relative importance of the activated nodes, the natural language processing module 732 selects one of the actionable intents as the task the user intends for the digital assistant to perform. In some examples, the domain with the most "triggered" nodes is selected. In some examples, the domain with the highest confidence (e.g., based on the relative importance of its individual triggered nodes) is selected. In some examples, the domain is selected based on a combination of the number and importance of the triggered nodes. In some examples, additional factors are also considered in the process of selecting the node, such as whether the digital assistant has previously correctly interpreted similar requests from the user.

[0235] The user data 748 includes information specific to the user, such as user-specific vocabulary, user preferences, user address, the user's default second language, the user's contact list, and other short-term or long-term information for each user. In some examples, the natural language processing module 732 uses the user-specific information to supplement the information contained in the user input to further qualify the user's intent. For example, for the user request "invite my friends to my birthday party," the natural language processing module 732 can access the user data 748 to determine which "friends" and when and where the "birthday party" will take place, without requiring the user to explicitly provide such information in their request.

[0236] It is recognized that, in some examples, the natural language processing module 732 is implemented with one or more machine learning mechanisms (e.g., neural networks). In particular, the one or more machine learning mechanisms are configured to receive a candidate text representation and context information associated with the candidate text representation. Based on the candidate text representation and the associated context information, the one or more machine learning mechanisms are configured to determine an intent confidence score based on a set of candidate actionable intents. The natural language processing module 732 can select one or more candidate actionable intents from the set of candidate actionable intents based on the determined intent confidence score. In some examples, the one or more candidate actionable intents are also selected from the set of candidate actionable intents with a knowledge ontology (e.g., the knowledge ontology 760).

[0237] Further details of searching a knowledge ontology based on a symbol string are described in U.S. Utility Patent Application Serial No. 12 / 341,743, entitled "Method and Apparatus for Searching Using An Active Ontology," filed December 22, 2008, the entire disclosure of which is incorporated by reference herein.

[0238] In some examples, once the natural language processing module 732 identifies an actionable intent (or domain) based on a user request, the natural language processing module 732 generates a structured query to represent the identified actionable intent. In some examples, the structured query includes parameters for one or more nodes within the domain of the actionable intent, and at least some of the parameters are populated with specific information and requirements specified in the user request. For example, a user says "Help me make a reservation for a sushi restaurant at 7 p.m. tonight." In this case, the natural language processing module 732 is able to correctly identify the actionable intent as "restaurant reservation" based on the user input. From the knowledge ontology, the structured query for the "restaurant reservation" domain includes parameters such as {cuisine}, {time}, {date}, {party size}, etc. In some examples, based on the spoken input and the text derived from the spoken input using the STT processing module 730, the natural language processing module 732 generates a partially structured query for the restaurant reservation domain, where the partially structured query includes the parameters {cuisine = "sushi"} and {time = "7 p.m."}. However, in this example, the user utterance contains insufficient information to complete the structured query associated with the domain. Thus, based on the currently available information, other necessary parameters such as {party size} and {date} are not specified in the structured query. In some examples, the natural language processing module 732 populates some of the parameters of the structured query with received context information. For example, in some examples, if the user requests a "nearby" sushi restaurant, the natural language processing module 732 populates the {location} parameter in the structured query with GPS coordinates from the user device.

[0239] In some examples, the natural language processing module 732 identifies a plurality of candidate actionable intents for each candidate text representation received from the STT processing module 730. Additionally, in some examples, a respective structured query is generated (partially or entirely) for each identified candidate actionable intent. The natural language processing module 732 determines an intent confidence score for each candidate actionable intent and ranks the candidate actionable intents based on the intent confidence scores. In some examples, the natural language processing module 732 communicates the generated structured query(s) (including any completed parameters) to the task flow processing module 736 ("task flow processor"). In some examples, one or more structured queries are provided to the task flow processing module 736 for the m best (e.g., m highest ranked) candidate actionable intents, where m is a predetermined integer greater than zero. In some examples, the one or more structured queries for the m best candidate actionable intents are provided to the task flow processing module 736 along with the corresponding candidate text representation(s).

[0240] Other details of inferring user intent based on a plurality of candidate actionable intents determined from a plurality of candidate text representations of speech inputs are described in U.S. Utility Patent Application Serial No. 14 / 298,725, filed June 6, 2014, entitled "System and Method for Inferring User Intent From Speech Inputs," the entire disclosure of which is incorporated by reference herein.

[0241] The task flow processing module 736 is configured to receive one or more structured queries from the natural language processing module 732, complete the structured query(s) (as necessary), and perform the actions necessary to "fulfill" the user's ultimate request. In some examples, various processes necessary to complete these tasks are provided in a task flow model 754. In some examples, the task flow model 754 includes processes for obtaining additional information from the user, as well as task flows for performing the actions associated with the actionable intent.

[0242] As described above, to complete the structured query, the task flow processing module 736 needs to initiate additional dialog with the user in order to obtain additional information and / or disambiguate potentially ambiguous utterances. When such interaction is necessary, the task flow processing module 736 invokes the dialog flow processing module 734 to engage in a dialog with the user. In some examples, the dialog flow processing module 734 determines how (and / or when) to request additional information from the user, and receives and processes the user's response. The questions are provided to the user and answers are received from the user through the I / O processing module 728. In some examples, the dialog flow processing module 734 presents dialog output to the user via audible output and / or visual output, and receives input from the user via spoken or physical (e.g., click) responses. Continuing the example described above, when the task flow processing module 736 invokes the dialog flow processing module 734 to determine "party size" and "date" information for the structured query associated with the domain "restaurant reservation," the dialog flow processing module 734 generates questions such as "How many in your party?" and "Which day for the reservation?" to pass to the user. Once answers are received from the user, the dialog flow processing module 734 fills in the structured query with the missing information, or passes the information to the task flow processing module 736 to complete the missing information from the structured query.

[0243] Once the task flow processing module 736 has completed the structured query for the executable intent, the task flow processing module 736 begins executing the final task associated with the executable intent. Thus, the task flow processing module 736 executes the steps and instructions in the task flow model according to the particular parameters contained in the structured query. For example, the task flow model for the executable intent "restaurant reservation" includes steps and instructions for contacting the restaurant and actually requesting a reservation for a particular party size at a particular time. For example, using a structured query such as: {restaurant reservation, restaurant = ABC Cafe, date = 3 / 12 / 2012, time = 7pm, party size = 5}, the task flow processing module 736 can perform the following steps: (1) log into the server for ABC Cafe or a restaurant reservation system such as (2) enter the date, time, and party size information in the form on the website, (3) submit the form, and (4) create a calendar entry for the reservation in the user's calendar.

[0244] In some examples, the task flow processing module 736, with the assistance of the service processing module 738 ("service processing module"), fulfills a task requested in the user input or provides an informational answer requested in the user input. For example, the service processing module 738 initiates a telephone call on behalf of the task flow processing module 736, sets a calendar entry, invokes a map search, invokes or interacts with other user applications installed on the user device, and invokes or interacts with third-party services (e.g., a restaurant reservation portal, a social networking site, a banking portal, etc.). In some examples, the protocols and application programming interfaces (APIs) required for each service are specified by a corresponding service model in the service models 756. The service processing module 738 accesses the appropriate service model for a service and generates a request for the service according to the protocols and APIs required for the service in accordance with the service model.

[0245] For example, if a restaurant has enabled an online reservation service, the restaurant submits a service model that specifies the necessary parameters for making a reservation and an API for transmitting values of the necessary parameters to the online reservation service. Upon request by the task flow processing module 736, the service processing module 738 can use the web address stored in the service model to establish a network connection with the online reservation service and transmit the necessary parameters for the reservation (e.g., time, date, number of party) to the online reservation interface in a format according to the API of the online reservation service.

[0246] In some examples, the natural language processing module 732, the dialog flow processing module 734, and the task flow processing module 736 are used collectively and iteratively to infer and qualify a user's intent, obtain information to further clarify and refine the user's intent, and ultimately generate a response (i.e., output to the user, or fulfill a task) to satisfy the user's intent. The generated response is a dialog response to the speech input that satisfies the user's intent, at least in part. Additionally, in some examples, the generated response is output as speech output. In these examples, the generated response is sent to a speech synthesis processing module 740 (e.g., a speech synthesizer), where the generated response can be processed to synthesize the dialog response in speech form. In other examples, the generated response is data content related to satisfying the user's request in the speech input.

[0247] In examples where the task flow processing module 736 receives multiple structured queries from the natural language processing module 732, the task flow processing module 736 first processes a first structured query of the received structured queries to attempt to fulfill the first structured query and / or execute one or more tasks or actions represented by the first structured query. In some examples, the first structured query corresponds to the highest ranked executable intent. In other examples, the first structured query is selected from the received structured queries based on a combination of the corresponding speech recognition confidence scores and the corresponding intent confidence scores. In some examples, if the task flow processing module 736 encounters an error during processing of the first structured query (e.g., due to an inability to determine a necessary parameter), the task flow processing module 736 can proceed to select and process a second structured query of the received structured queries that corresponds to a lower ranked executable intent. The second structured query is selected, for example, based on a speech recognition confidence score of the corresponding candidate text representation, an intent confidence score of the corresponding candidate executable intent, a missing necessary parameter in the first structured query, or any combination thereof.

[0248] The speech synthesis processing module 740 is configured to synthesize speech output for presentation to the user. The speech synthesis processing module 740 synthesizes speech output based on text provided by the digital assistant. For example, the generated dialog response is in the form of a text string. The speech synthesis processing module 740 converts the text string into audible speech output. The speech synthesis processing module 740 uses any appropriate speech synthesis technique in order to generate speech output from text, including but not limited to: concatenative synthesis, unit selection synthesis, diphthong synthesis, domain-specific synthesis, formant synthesis, articulatory synthesis, Hidden Markov Model (HMM)-based synthesis, and sinusoidal synthesis. In some examples, the speech synthesis processing module 740 is configured to synthesize individual words based on a phoneme string corresponding to the words. For example, a phoneme string is associated with a word in the generated dialog response. The phoneme string is stored in metadata associated with the word. The speech synthesis processing module 740 is configured to directly process the phoneme string in the metadata to synthesize the word in speech form.

[0249] In some examples, instead of (or in addition to) using the speech synthesis processing module 740, speech synthesis is performed on a remote device (e.g., the server system 108) and the synthesized speech is sent to the user device for output to the user. For example, this can occur in some implementations where the output of the digital assistant is generated at the server system. Also, since the server system typically has more processing power or more resources than the user device, it is possible to obtain higher quality speech output than would be achieved with client-side synthesis.

[0250] Additional details regarding digital assistants can be found in U.S. Utility Patent Application No. 12 / 987,982, entitled "Intelligent Automated Assistant," filed on January 10, 2011, and U.S. Utility Patent Application No. 13 / 251,088, entitled "Generating and Processing Task Items That Represent Tasks to Perform," filed on September 30, 2011, the entire disclosures of which are incorporated by reference herein.

[0251] 4. User interfaces and processes for providing search functionality

[0252] FIG. 8A to FIG. 8P Example user interfaces for providing search functionality are shown in accordance with some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes in FIG. 9 and FIG. 10 .

[0253] FIG. 8A A computer system 800 (e.g., a smart phone) with a touch-sensitive display 802 is shown. In some embodiments, the computer system 800 is a tablet, a smart watch, a laptop, a desktop, and / or a camera. In some embodiments, the inputs described below can be replaced with alternative inputs, such as tap inputs, swipe inputs, press inputs, etc.

[0254] In FIG. 8A , the computer system 800 displays a user interface 801 indicating a locked state on the display 802. In some examples, the computer system 800 displays the user interface 801 after transitioning from a locked and / or sleep state to an unlocked and / or active state. In some examples, the computer system 800 displays the user interface 801 after verifying a user's identity. In some examples, verifying a user's identity and / or transitioning from a low power state to a high power state includes detecting an input with one or more sensors of the computer system 800, including a camera, an accelerometer, etc. While displaying the user interface 801, the computer system 800 detects a user input 803 (e.g., a swipe input), and in response to detecting the user input 803, the computer system shows a home screen user interface 804 as shown in FIG. 8B .

[0255] In FIG. 8BIn some examples, the computer system 800 displays the home screen user interface 804 on the display 802 upon receiving a request to display a user interface. In some examples, the request to display the home screen user interface 804 includes detecting a user input, such as a swipe gesture, a tap gesture, and / or a press gesture on a lock screen or other user interface of the computer system 800. In some examples, the computer system 800 displays the home screen user interface 804 upon transitioning from a low-power state to a high-power or full-power state.

[0256] The home screen user interface 804 includes an affordance 804a that indicates the number of pages included in the home screen user interface 804 and that allows selection of different pages of the home screen user interface 804. In particular, the affordance 804a includes three large dots that indicate the location of the current page relative to additional pages that can be accessed (e.g., via a swipe gesture) as part of the home screen user interface 804. Additionally, in some examples, if the home screen user interface 804 includes more than 3 pages, additional small dots shown to the right of the affordance 804a are displayed to indicate additional pages (e.g., pages 4, 5, 6, etc.). In some examples, if the home screen user interface 804 is showing a page other than the first page of the home screen user interface 804 (e.g., page 2, 3, 4, 5, 6, etc.), in addition to (or instead of) the small dots on the right of the affordance 804a, there can be small dots shown on the left of the affordance 804a that are displayed to indicate additional previous pages.

[0257] While displaying the home screen user interface 804, the computer system 800 detects the occurrence of a predetermined condition, and in response to detecting the predetermined condition, updates (e.g., displays) the affordance 804a to include an indication of a search function, as shown in FIG. 8C In some examples, the predetermined condition includes a predetermined amount of time elapsing while displaying the home screen user interface 804. For example, the computer system 800 causes the affordance 804a to include an indication of a search function after displaying the home screen user interface 804 for 5, 10, 15, or 20 seconds.

[0258] In some examples, the predetermined condition includes displaying a particular page of the home screen user interface 804, such as the first page or the second page. In some examples, the predetermined condition includes transitioning from one page of the home screen user interface 804 to another page of the home screen user interface 804. In some examples, the predetermined condition includes transitioning from one page of the home screen user interface 804 to another page of the home screen user interface 804 and a predetermined amount of time elapsing after transitioning from one page of the home screen user interface 804 to another page.

[0259] The updated affordance 804a includes changing the indication of the number of pages of the home screen user interface 804 to include an indication of a search function, such as the word "Search," and an illustration associated with the search (e.g., a magnifying lens), as shown in FIG. 8C In some examples, one of the indications of the pages is replaced with the illustration of the search function, while the other indications of the pages continue to be displayed in the affordance 804a, as shown in FIG. 8M Thus, the affordance 804a can include both an indication of the number of pages and an indication of a search function, such as the illustration.

[0260] While displaying the home screen user interface 804 with the updated affordance 804a including an indication of a search function, the computer system 800 detects (e.g., receives) a swipe input 806 from the right side of the display 802 to the left side of the display 802, as shown in FIG. 8C In response to detecting the swipe input 806, the computer system 800 displays the second page of the home screen user interface 804, as shown in FIG. 8D

[0261] In some examples, the computer system 800 displays the second page of the home screen user interface 804 in response to detecting a user input, such as a tap input on a portion of the home screen user interface 804 indicating the second page (e.g., the second dot). For example, in response to detecting a tap input 808 on the second portion (e.g., dot) of the affordance 804a, as shown in FIG. 8B the computer system 800 displays the second page of the home screen user interface 804, as shown in FIG. 8D

[0262] In FIG. 8D the second page of the home screen user interface 804 includes an affordance 804a that does not have an indication of a search function but has an indication of the number of pages. Specifically, the affordance 804a includes the same number of page indications (e.g., 3), but different page indications are highlighted when compared to the affordance 804a, as shown in FIG. 8B Because the home screen user interface 804 includes the first page in FIG. 8B the first circle or dot of the affordance 804a is highlighted, while in FIG. 8D the second page of the home screen user interface 804 is included, and thus the second circle or dot of the affordance 804a is highlighted. Additionally, in examples that include more than 3 available pages, if the user continues to navigate through the pages, a small dot on the right and / or a small dot on the left can be displayed to indicate additional pages.

[0263] ​​Additionally, because the location of affordance 804a is predetermined, affordance 804a is displayed in the same location in FIG. 8B , FIG. 8C and FIG. 8D . In some examples, the location of affordance 804a is dynamic and based on factors including the number of applications, widgets, and / or other affordances displayed in home screen user interface 804. For example, affordance 804a can be displayed below the last row of application icons or affordances, and thus can be in a first location below the third row on the first page and in a second location below the second row on the second page.

[0264] Affordance 804a also has a predetermined size in FIG. 8B , FIG. 8C and FIG. 8D . In some examples, the predetermined size is based on the number of pages of home screen user interface 804. For example, affordance 804a can have a first size when there are three pages of home screen user interface 804 and a second size (e.g., a larger size) when there are five pages of home screen user interface 804. In other examples, the predetermined size is not based on the number of pages of home screen user interface 804. For example, affordance 804a can have the same size whether there are three pages or five pages of home screen user interface 804.

[0265] In some examples, affordance 804a includes an indication that additional pages exist, as shown in FIG. 8D . In some examples, the indication of additional pages includes an indication that is a different size (e.g., larger or smaller) than other indications of different pages. In some examples, the number and / or size of the indications of pages changes when computer system 800 displays different pages of home screen user interface 804. For example, when computer system 800 displays the third page of home screen user interface 804, the indication representing the fourth page is displayed in the same size as the indications representing the second and third pages, while the indication representing the first page is displayed in a smaller size.

[0266] In some examples, additional indications of pages are displayed in affordance 804a after detecting an input to change the page of home screen user interface 804. For example, after displaying the second page of home screen user interface 804, computer system 800 updates affordance 804a to provide an indication to the user that there are more pages than the original indication of three pages. Similarly, when computer system 800 displays the third page of home screen user interface 804, affordance 804a can be updated to include a fifth indication of a page.

[0267] In some examples, the power indicator 804c associated with the search function and the power indicator 804b associated with the number of pages in the main screen user interface are displayed concurrently (e.g., simultaneously), such as FIG. 8O As shown. Therefore, when a user input of a swipe gesture is detected (840) and the second page of the main screen user interface is displayed, as... FIG. 8P As shown, the highlighted indicator 804b corresponding to the currently displayed page is changed, but the indicator 804c associated with the search continues to be displayed. In this example, the currently displayed page is highlighted not by changing its color as discussed above, but by making the indicator associated with the current page larger.

[0268] After a predetermined condition is detected when the main screen user interface 804 is displayed (e.g., in response to detecting the predetermined condition), such as FIG. 8D As shown, computer system 800 displays (e.g., updates) a capability display 804a to include, for example, the following: FIG. 8E The instructions for the search function are shown and discussed above.

[0269] When displaying an indicator 804a that includes instructions for the search function, such as FIG. 8E As shown, computer system 800 detects one or more inputs to display a search user interface. For example, computer system 800 detects tap input 810 on a display 804a that includes an indication of search functionality. As another example, computer system 800 detects a swipe gesture 812 from the top of display 802 in a direction towards the bottom of display 802. In another example, computer system 800 detects upward touches and drags (e.g., swipe inputs) on display 804a.

[0270] In some examples, computer system 800 detects verbal input instructing the display of a search user interface. For example, computer system 800 may detect (e.g., receive) verbal input such as “I want to perform a search” or “Who is Charles Appleseed?” instructing the user to perform a search. In some examples, verbal input is received while the main screen user interface 804 is displayed. Therefore, computer system 800 can use the above-mentioned... FIG. 6A to FIG. 6B and FIG. 7A to FIG. 7C The described components and processes determine the user intent to perform the search and display the search user interface 804a in response to these verbal inputs (e.g., as...). FIG. 8E (As shown).

[0271] In response to detecting one or more inputs that would otherwise display the search user interface, computer system 800 displays search user interface 814, such as... FIG. 8GAs shown. In some examples, displaying the search user interface 814 includes displaying an animation transitioning from the main screen user interface 804 to the search user interface 814, such as... FIG. 8F As shown. The animation showing the transition from the main screen user interface 804 to the search user interface 814 includes displaying the power indicator 804a from its first position (e.g., ...). FIG. 8E (As shown) Move to the second position of the indicator 814a (as shown) FIG. 8G The animation (as shown). In some examples, user interface 814 is larger (e.g., wider) than user interface 804 (e.g., expands to a larger size), however, in other examples, user interface 814 and user interface 804 are the same size. In some examples, as further described below, the size of display representation 804a changes (e.g., grows) from the size in user interface 804 to a second size as shown in search user interface 814. Furthermore, the animated transition from home screen user interface 804 to search user interface 814 includes obscuring home screen user interface 804 (e.g., the content displayed on home screen user interface 814) when home screen user interface 804 is replaced by search user interface 814 (e.g., the content displayed on search user interface 804).

[0272] The animation displaying the transition from the main screen user interface 804 to the search user interface 814 also includes an animation of the power representation 804a changing to the power representation 814a associated with the search (e.g., a user interface element associated with the search). The animation of the power representation 804a changing to the power representation 814a includes the power representation 804a stretching (e.g., growing longer) and merging with the virtual keyboard 814b, as... FIG. 8G As shown. In addition, the transition from power indicator 804a to power indicator 814a includes displaying cursor 814c in power indicator 814a and canceling power indicator 814d in power indicator 814a.

[0273] like FIG. 8G As shown, after the computer system 800 completes the animation transitioning from the main screen user interface 804 to the search user interface 814, the computer system 800 displays the search user interface 814. The search user interface 814 includes a power display 814a and a virtual keyboard 814b, as discussed above, as well as user interface elements 814e and 814f associated with different topics of interest.

[0274] In some examples, the topic of interest is based on a context of the computer system 800. The context of the computer system 800 can include a location of the computer system 800, a recent location of the computer system 800, a type of network to which the computer system 800 is connected, previously accessed applications, a usage history of the computer system 800 and / or a digital assistant of the computer system 800, and other data available to the computer system 800, including contacts, messages, webpages, browsing history, etc.

[0275] In some examples, the content displayed at the edge of the search user interface 814 is obscured such that the general color or other general attributes of the content are visible to the user, but the specific content is hidden or presented in a way that appears blurred. In some examples, the affordances 814a, virtual keyboard 814b, and other content of the search user interface 814 are displayed over the obscured content.

[0276] While displaying the search user interface 814, as shown in FIG. 8G the computer system 800 detects (e.g., receives) input 816 of the word “coffee” in the affordance (e.g., user interface element) 814a using the virtual keyboard 814b and a tap input 817 on “go.” In response to detecting the set of user inputs 816 and 818, the computer system 800 displays the search results user interface 818, as shown in FIG. 8H .

[0277] The search results user interface 818 includes the affordance 814a (e.g., user interface element 814a) in a second docked position at the bottom of the search results user interface 818. The search results user interface 818 also includes a plurality of user interface elements 818b associated with search results and user interface elements 818c and 818d associated with a coffee shop application and a delivery application, respectively. The search results user interface 818 also does not display the virtual keyboard 814b.

[0278] In some examples, displaying the search results user interface 818 includes displaying an animation that transitions from the search user interface 814 to the search results user interface 818. The animation that transitions from the search user interface 814 to the search results user interface 818 includes an animation that moves the affordance 814a from a first position displayed over the virtual keyboard 814b, as shown in FIG. 8G , to a second position docked at the bottom of the screen, as shown in FIG. 8H . The animation that transitions from the search user interface 814 to the search results user interface 818 also includes an animation that fades out and stops displaying the virtual keyboard 814b.

[0279] User interface elements 818B, 818C, and 818D are all determined based on the "coffee" user input search, and thus are all related to the topic of coffee. User interface element 818B is associated with search results related to coffee, such as coffee shops nearby, best coffee near the location of computer system 800, coffee distributors, and coffee in general. User interface elements 818C and 818D are associated with applications related to the search topic of coffee that the user can find interesting or useful due to their search for coffee. In this way, search results user interface 818 provides topics and applications that the user can find useful and related to the topic that the user has searched for.

[0280] In some examples, the content displayed at the edge of search results user interface 818 is obscured such that the general color or other general attribute of the content is visible to the user, but the specific content is hidden or presented in a way that appears blurred. In some examples, affordance 814a and other content of search results user interface 814 is displayed over the obscured content.

[0281] When displaying search results user interface 818 as shown in FIG. 8H Computer system 800 detects selection of affordance 814d with tap input 820 and user input of a new search topic "Charles Appleseed" while search results user interface 818 is displayed. In response to detecting the set of inputs including tap input 820 and the new topic, computer system 800 displays an updated search results user interface 818 as shown in FIG. 8I

[0282] Search results user interface 818 includes affordance 814a with search topic "Charles Appleseed" and user interface element 818B associated with a music artist named "Charles Appleseed." In some examples, user interface element 818B is not associated with performing a search for the associated music artist (or other topic). In some examples, user interface element 818B is associated with providing information (e.g., knowledge) about the topic of the search from a database, without providing search results. For example, as discussed further below, selection of user interface element 818B can cause computer system 800 to provide information about "Charles Appleseed" that the user is likely to find interesting, culled from a database, without needing to search the internet or other databases.

[0283] ​While displaying the search results user interface 818, the computer system 800 detects a user input 822 (e.g., a tap, press, or other selection) on the user interface element 818B associated with the music artist "Charles Appleseed." In response to detecting the user input 822 (e.g., after detecting the user input), the computer system 800 displays an updated search results user interface 818 that includes a topic page 824 about Charles Appleseed, as shown. FIG. 8J

[0284] The topic page 824 includes information about a specified topic (in this case, the music artist Charles Appleseed). As discussed above, the topic page 824 provides information related to a selected topic in an organized and efficient interface without requiring interaction between the user and the computer system 800 that provided the search results. Thus, rather than the computer system 800 providing search results that include a link to a web page about Charles Appleseed, the computer system 800 detecting a user input selecting the web page, and then the computer system 800 providing one or more web pages that can or can not include information that the user will find relevant, the computer system 800 provides the topic page 824 that includes relevant information about the topic that the user is interested in.

[0285] In some examples, the topic page 824 is generated and / or provided by a digital assistant of the computer system 800. For example, upon receiving a request to display information related to the topic "Charles Appleseed," the digital assistant of the computer system 800 can access one or more databases, the internet, and / or other available resources to gather data related to "Charles Appleseed." The digital assistant then compiles and provides the information in a topic page, such as the topic page 824.

[0286] The updated search results user interface 818 includes the affordance 814a that has been updated to reflect that the computer system 800 is displaying the topic page 824. In particular, the affordance 814a includes the topic of the topic page 824, "Charles Appleseed," as well as an illustration that indicates the topic. For example, because "Charles Appleseed" is a music artist, the illustration is a microphone. Other example illustrations are a film strip for a movie, comedy and tragedy masks for a play, a baseball for a baseball team or game, a football for a football team or game, etc.

[0287] ​In some examples, the search results user interface 818 is displayed such that the affordance 814a is proximate to the bottom of the display 802, but appears to float in or over the topic page 824 or the topic page 828, as shown. Thus, rather than docking the affordance 814a to its own portion of the search results user interface 818, the affordance 814a appears to be dynamic and the information of the displayed topic page flows underneath the affordance 814a. FIG. 8N

[0288] While displaying the topic page 824 in the search results user interface 818, the computer system 800 detects (e.g., receives) a user input 826 on a portion of the topic page 824 that includes the topic “Chicago.” In response to detecting the user input 826, the computer system 800 displays the topic page 828 about the topic “Chicago” in the search results user interface 818, as shown. FIG. 8K

[0289] In addition to updating the search results user interface 818 with the new topic page 828, the computer system 800 also updates the affordance 814a to include the title or name of the topic (e.g., Chicago) and an icon associated with the topic. The affordance 814a includes an icon of a city to indicate that the topic is a point of interest as a city. In some examples, the icon corresponds to different types of points of interest, such as a mountain for a mountain, a river for a river, a tree for a park, or other suitable icons for different types of points of interest.

[0290] The affordance 814a also includes a back affordance 818e. In some examples, the computer system 800 detects a user input 830 (e.g., a tap or press input) on the back affordance 818e and, in response to detecting the user input 830, displays the search results user interface 818 including the topic page 824, as shown. FIG. 8J In other examples, the computer system 800 detects a user input 832 (e.g., a swipe input) from the left side of the display 802 and, in response to detecting the user input 832, displays the search results user interface 818 including the topic page 824, as shown. FIG. 8J In this way, a user can request to back up in a series of topics or search results that have been accessed, and the computer system 800 displays, in response, a previously accessed or provided topic or search result.

[0291] In some examples, the computer system 800 detects a user input 834 (e.g., a tap or press input) about the topic “Chicago” in the affordance 814a. In response to detecting the user input 834, the computer system 800 displays the search results user interface 818 including the topic page 828 about the topic “Chicago,” as shown. FIG. 8L ​​The search results user interface 818 shown includes a virtual keyboard 814b, a search related to "Chicago," and topics for the city of Chicago and the band Chicago. As discussed above, the user can interact with the computer system 800 and the search results user interface 818 to display a search or a topic page from the displayed topics based on user input.

[0292] In particular, the affordance 814a is updated to show a search for "Chicago" instead of a search for a specific topic for the city of Chicago, and thus causes the search results user interface 818 to be updated with search results related to "Chicago" instead of the topic page 828 related to the city of Chicago. From this interface, the user can choose to return to the topic page 828 by selecting the topic for the city of Chicago, select a search result to explore a certain topic related to "Chicago," or provide a new search altogether.

[0293] In some examples, displaying the search results user interface 818 includes displaying an animation transitioning from the search results user interface 818 with the topic page 828 to the search results user interface 818, as shown in FIG. 8E. FIG. 8L In some examples, the animation transitioning from the search results user interface 818 with the topic page 828 to the search results user interface 818 as shown in FIG. 8E includes an animation of the affordance 814a changing from a docked position as shown in FIG. 8D to a second position as shown in FIG. 8E. FIG. 8L In some examples, the animation transitioning from the search results user interface 818 with the topic page 828 to the search results user interface 818 as shown in FIG. 8E includes an animation of the virtual keyboard 814b displaying under the affordance 814a in the second position. FIG. 8K In some examples, the animation transitioning from the search results user interface 818 with the topic page 828 to the search results user interface 818 as shown in FIG. 8E includes an animation of the virtual keyboard 814b displaying under the affordance 814a in the second position. FIG. 8L In some examples, the animation transitioning from the search results user interface 818 with the topic page 828 to the search results user interface 818 as shown in FIG. 8E includes an animation of the virtual keyboard 814b displaying under the affordance 814a in the second position. FIG. 8L In some examples, the animation transitioning from the search results user interface 818 with the topic page 828 to the search results user interface 818 as shown in FIG. 8E includes an animation of the virtual keyboard 814b displaying under the affordance 814a in the second position.

[0294] In some examples, while displaying the search results user interface 818 as shown in FIG. 8E, the computer system 800 detects user input indicating that a different user interface should be displayed. For example, the computer system 800 can detect user input transitioning the computer system 800 from an unlocked state to a locked state, such as a button press. As another example, the computer system 800 can detect a swipe user input from the bottom or top of the display 802 indicating that a different user interface should be displayed. FIG. 8L In response to detecting these user inputs, the computer system 800 displays a different user interface, such as the search results user interface 818 as shown in FIG. 8E.

[0295] FIG. 8B ​the home screen user interface 804. In some examples, displaying the different user interface includes displaying an animation that transitions from the search results user interface 818 to the different user interface. For example, when transitioning from the search results user interface 818 to the home screen user interface 804, the computer system 800 can display an animation similar to the animation shown and discussed above, but in reverse. Thus, the affordance 814a can shrink to the size of the affordance 804a and move from a position above the virtual keyboard to a position on the home screen user interface 804, while the word "search" and the icon are replaced with an indication of the number of pages of the home screen user interface 804. Additionally, the search results user interface 818 can fade out and be replaced by the home screen user interface 804 and the various applications and other features included in the home screen user interface 804. FIG. 8F the animation shown and discussed above, but in reverse. Thus, the affordance 814a can shrink to the size of the affordance 804a and move from a position above the virtual keyboard to a position on the home screen user interface 804, while the word "search" and the icon are replaced with an indication of the number of pages of the home screen user interface 804. Additionally, the search results user interface 818 can fade out and be replaced by the home screen user interface 804 and the various applications and other features included in the home screen user interface 804.

[0296] Similarly, once the different user interface is displayed, such as the home screen user interface 804 shown FIG. 8B the search user interface 814, the computer system 800 can detect another user input that indicates to display the search user interface 814 and display a corresponding animation that transitions from the home screen user interface 804 to the search user interface 814, as discussed above with respect to FIG. 8F and FIG. 8G discussed above.

[0297] FIG. 9 A process 900 for accessing a search user interface is shown in accordance with various examples. For example, the process 900 is performed using one or more electronic devices and / or computer systems. In some examples, the process 900 is performed using an electronic device and / or computer system that implements a digital assistant. In some examples, the process 900 is performed using a client-server system (e.g., the system 100), and the blocks of the process 900 are divided between a server (e.g., the DA server 106) and a client device in any manner. In other examples, the blocks of the process 900 are divided between a server and multiple client devices (e.g., a mobile phone and a smart watch). Thus, while portions of the process 900 are described herein as being performed by particular devices of a client-server system, it should be understood that the process 900 is not limited thereto. In other examples, the process 900 is performed using a client device only (e.g., the user device 104) or multiple client devices only. In the process 900, some blocks are optionally combined, the order of some blocks is optionally changed, and some blocks are optionally omitted. In some examples, additional steps can be performed in connection with the process 900.

[0298] At block 902, at a computer system (e.g., 800) (e.g., a user device, a smartphone, a laptop, a tablet, etc.) that includes a touch-sensitive display (e.g., 802) (e.g., a display controller and / or a display generation component) and one or more input devices (e.g., 802) (e.g., buttons, speakers, cameras, motion detectors (e.g., accelerometers and / or gyroscopes), or touch-sensitive surfaces), a request to display a user interface (e.g., 801, 804) is received. In some examples, receiving the request to display the user interface includes detecting a swipe gesture (e.g., 803) in a predetermined direction on a touch-sensitive surface. In some examples, receiving the request to display the user interface includes detecting a user gaze directed at a predetermined portion of the user interface. In some examples, receiving the request to display the user interface includes detecting a user gaze at an affordance (e.g., a search affordance) displayed in the user interface.

[0299] At block 904, in response to receiving the request to display the user interface (e.g., 804), the user interface including an affordance (e.g., 804a) is displayed via the touch- sensitive display (e.g., 802), the affordance including an indication of a number of pages of the user interface. In some examples, the affordance has a predetermined size that is unaffected by the number of pages. In some examples, the affordance has a predetermined location on a first page (e.g., as illustrated in 804) of the user interface and a second page (e.g., as illustrated in 804) of the user interface. FIG. 8B FIG. 8D

[0300] In some examples, in accordance with a determination that the number of pages of the user interface (e.g., 804) is greater than a predetermined number, the affordance (e.g., 804a) is displayed with a first indication (e.g., as illustrated in 804a) that the number of pages is greater than the predetermined number; and in accordance with a determination that the number of pages is not greater than the predetermined number, the affordance is displayed with a second indication (e.g., as illustrated in 804a) that the number of pages is not greater than the predetermined number. FIG. 8B FIG. 8D

[0301] In some examples, an input (e.g., 806, 808) corresponding to a selection of a page of the user interface (e.g., 804) is detected, and in response to detecting the input corresponding to the selection of the page of the user interface, the selected page of the user interface is displayed (e.g., as illustrated in 804a). FIG. 8C

[0302] ​​​​​At block 906, while displaying the user interface (e.g., 804): detecting an occurrence of a predetermined condition (e.g., a predetermined amount of time elapsing, a change to a different page of the user interface, a change to a different page of the user interface and a predetermined amount of time elapsing).

[0303] At block 908, while displaying the user interface (e.g., 804), in response to detecting the predetermined condition, updating the affordance (e.g., user interface element) (e.g., 804a) to include an indication of a search function (e.g., as shown in 804a). FIG. 8E and FIG. 8C At block 908, while displaying the user interface (e.g., 804), in response to detecting the predetermined condition, updating the affordance (e.g., user interface element) (e.g., 804a) to include an indication of a search function (e.g., as shown in 804a).

[0304] In some examples, while displaying the affordance with the indication of the search function (e.g., 804a), detecting an input to display a second page of the user interface (e.g., 806), and in response to detecting the input to display the second page of the user interface, displaying the second page of the user interface that includes an indication of a number of pages of the user interface (e.g., 804 as shown in FIG. 8E and FIG. 8D In some examples, a first portion of the indication of the number of pages is emphasized when displayed on the first page of the user interface (e.g., 804a as shown in FIG. 8B and a second portion of the indication of the number of pages is emphasized when displayed on the second page of the user interface (e.g., 804a as shown in FIG. 8D FIG. 8D

[0305] In some examples, while displaying the second page of the user interface (e.g., 804a as shown in FIG. 8E In response to detecting a second occurrence of the predetermined condition, updating the affordance to include the indication of the search function (e.g., 804a as shown in FIG. 8F

[0306] In some examples, detecting an input to select the indication of the search function (e.g., 804a) (e.g., 810, 812), and in response to detecting the input to select the indication of the search function, displaying a search user interface (e.g., 814). In some examples, displaying the search user interface further includes concurrently displaying a user interface element (e.g., affordance) associated with the search (e.g., 814a) and a user interface element associated with a topic of interest (e.g., 814e, 814f). In some examples, the topic of interest is based on a context of the computer system.

[0307] ​​​In some examples, animations are shown that include a transition from the user interface (e.g., 804) to the search user interface (e.g., 814). FIG. 9 In some examples, the animation showing the transition from the user interface to the search user interface also includes: an animation showing a power representation (e.g., a user interface element) (804a) transforming into a user interface element (e.g., a power representation) associated with the search (e.g., 814a); and an animation showing the power representation moving from a first position in the user interface to a second position in the search user interface. In some examples, the animations of the power representation transforming into a user interface element associated with the search and the animations of the power representation moving from a first position in the user interface to a second position in the search user interface are displayed concurrently.

[0308] The above reference FIG. 1 to FIG. 4 The described operation is optionally determined by FIG. 6A to FIG. 6B , FIG. 7A to FIG. 7C , FIG. 8A to FIG. 8P as well as FIG. 1 to FIG. 4 The components depicted in the diagram are used for implementation. For example, the operation of process 900 can be implemented by computer system 800. Those skilled in the art will clearly understand how to implement this based on the components depicted in the diagram. FIG. 6A to FIG. 6B , FIG. 7A to FIG. 7C , FIG. 8A to FIG. 8P as well as FIG. 10 The components depicted in the diagram are used to perform other processes.

[0309] FIG. 8H A process 1000 for updating a search user interface is illustrated according to various examples. For example, process 1000 may be performed using one or more electronic devices and / or computer systems. In some examples, process 1000 may be performed using electronic devices and / or computer systems implementing digital assistants. In some examples, process 1000 may be performed using a client-server system (e.g., system 100), and the boxes of process 1000 may be divided in any way between the server (e.g., DA server 106) and the client devices. In other examples, the boxes of process 1000 may be divided between the server and multiple client devices (e.g., mobile phones and smartwatches). Thus, while parts of process 1000 are described herein as being performed by a specific device of the client-server system, it should be understood that process 1000 is not limited thereto. In other examples, process 1000 may be performed using only client devices (e.g., user device 104) or only multiple client devices. In process 1000, some boxes may be optionally combined, some boxes may be optionally changed in order, and some boxes may be optionally omitted. In some examples, process 1000 can be combined to perform additional steps.

[0310] At block 1002, at a computer system (e.g., 800) (e.g., a user device, a smartphone, a laptop, a tablet, etc.) that includes a display generation component (e.g., 802) (e.g., a display controller and / or a touch-sensitive display system) and one or more input devices (e.g., 802) (e.g., buttons, speakers, cameras, motion detectors (e.g., accelerometers and / or gyroscopes), or touch-sensitive surfaces), user input (e.g., 816, 817) on a first portion of a search user interface (e.g., 814) is detected via the one or more input devices.

[0311] At block 1004, in response to detecting user input (e.g., 816, 817) on the first portion of the search user interface (e.g., 814), a search results user interface (e.g., 818) is displayed via the display generation component (e.g., 802), the search results user interface including user interface elements (1006) associated with search results (e.g., 818a, 818b, 818c) and a user interface element associated with a search operation (e.g., 814a), where the user interface element associated with the search operation is displayed in a predetermined location (e.g., as illustrated by 814a) and includes a textual indication (1008) of a target of the search operation. FIG. 8G

[0312] In some examples, prior to displaying the search results user interface (e.g., 818), an animation is displayed that transitions from the search user interface (e.g., 814) to the search results user interface. In some examples, displaying the search user interface includes displaying a virtual keyboard (e.g., 814b). In some examples, displaying the search results user interface includes ceasing to display the virtual keyboard. In some examples, the animation includes a transition from displaying the virtual keyboard to ceasing to display the virtual keyboard. In some examples, the user interface element associated with the search operation (e.g., 814a) is displayed in a first location prior to detecting the user input (e.g., as illustrated by 814a) and is displayed in a second location when displayed in the predetermined location (e.g., as illustrated by 814a). In some examples, the animation that transitions from the search user interface to the search results user interface includes an animation that moves the user interface element associated with the search operation from the first location to the second location. FIG. 8H FIG. 8L

[0313] ​​​In some examples, displaying the search results user interface (e.g., 818) further includes displaying a user interface element (e.g., 814a) associated with the search operation, the user interface element including an affordance (e.g., 818b) to provide previous search results. In some examples, detecting user input (e.g., 830) on the affordance to provide previous search results, and in response to detecting the user input on the affordance to provide previous search results, displaying a search results user interface including: user interface elements (e.g., 818a, 818b, 818c, 824, 828) associated with previous search results; and the user interface element (e.g., 814a) associated with the search operation in the predetermined location.

[0314] In some examples, while displaying the search results user interface (e.g., 818): detecting user input (e.g., 822, 826) that selects a user interface element (e.g., 818a, 818b, 818c, 824, 828) associated with a search result, and in response: data (e.g., 824, 828) related to the search result is displayed in the search results user interface; and a title of the search result is displayed in the user interface element (e.g., 814a) associated with the search operation. In some examples, the title of the search result is an entity associated with the search result. In some examples, the user interface element associated with the search operation further includes an icon that indicates a type of the entity.

[0315] In some examples, detecting user input (e.g., 834) that selects the user interface element (e.g., 814a) associated with the search operation, and in response to detecting the user input that selects the user interface element associated with the search operation: displaying a search user interface including a virtual keyboard (e.g., 814b) and the user interface element (e.g., 814a) associated with the search operation in the first location (e.g., as shown in FIG. 8B). FIG. 10

[0316] In some examples, content displayed at an edge of the search results user interface (e.g., 818) is partially obscured. In some examples, the user interface element (e.g., 814a) associated with the search operation is displayed over the partially obscured content.

[0317] ​In some examples, a predetermined event is detected (e.g., a user input, the computer system entering a low power mode, the computer system exiting a low power mode, etc.) and, in response to detecting the predetermined event, the display of the search results user interface (e.g., 818) is stopped. In some examples, stopping the display of the search results user interface further includes displaying an animation that transitions from the search results user interface (e.g., 818) to a different user interface (e.g., the home screen user interface 804, the lock screen user interface 801, etc.). In some examples, displaying the animation that transitions from the search results user interface to the different user interface includes displaying an animation that changes a user interface element associated with the search (e.g., 814a) to an affordance associated with the number of pages (e.g., 804a) and displaying an animation that moves the user interface element associated with the search from a first location in the user interface to a second location in the different user interface. In some examples, a swipe user input is detected from the top of the different user interface and, in response to detecting the swipe user input from the top of the different user interface, a search user interface is displayed that includes the user interface element associated with the search in the predetermined location.

[0318] The above description references FIG. 1 to FIG. 4 The described operations are optionally implemented by FIG. 6A to FIG. 6B , FIG. 7A to FIG. 7C , FIG. 8A to FIG. 8P and FIG. 1 to FIG. 4 depicted in FIGS. 10-13. For example, operations of process 1000 can be implemented by computer system 800. One of ordinary skill in the art will readily recognize how to implement other processes based on the components depicted in FIGS. 10-13. FIG. 6A to FIG. 6B , FIG. 7A to FIG. 7C , FIG. 8A to FIG. 8P and FIG. 10 FIG. 8H FIG. 8G FIG. 8H FIG. 8L FIG. 10 FIG. 1 to FIG. 4 FIG. 6A to FIG. 6B FIG. 7A to FIG. 7C FIG. 8A to FIG. 8P depicted in FIGS. 10-13.

[0319] According to some implementations, a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) storing one or more programs for execution by one or more processors of an electronic device is provided, the one or more programs including instructions for performing any of the methods or processes described herein.

[0320] According to some implementations, an electronic device (e.g., a portable electronic device) is provided that includes means for performing any of the methods or processes described herein.

[0321] According to some implementations, an electronic device (e.g., a portable electronic device) is provided that includes a processing unit configured to perform any of the methods and processes described herein.

[0322] According to some implementations, an electronic device (e.g., a portable electronic device) is provided that includes one or more processors and memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions for performing any of the methods or processes described herein.

[0323] The foregoing description, for purposes of explanation, describes specific embodiments in order to provide a thorough understanding of the application. However, the illustrative discussions above are not intended to be exhaustive or to limit the application to the precise forms described. Many modifications and variations are possible in view of the above teachings. The embodiments are chosen and described in order to best explain the principles of the techniques and their practical applications, and to thereby enable others skilled in the art to best utilize the techniques and various embodiments with various modifications as are suited to the particular uses contemplated.

[0324] Although the present disclosure and examples have been fully described with reference to the accompanying drawings, it is to be noted that various changes and modifications will become apparent to those skilled in the art. It is therefore to be understood that all these changes and modifications are believed to fall within the scope of the present disclosure and examples as defined by the appended claims.

[0325] As described above, one aspect of the present technology is to gather and use data from various sources to improve the delivery of search functionality services. The present disclosure contemplates that, in some instances, this gathered data can include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data can include demographic data, location-based data, telephone numbers, email addresses, twitter ID's, home addresses, data or records pertaining to a user's health or fitness level, date of birth, or any other identifying or personal information.

[0326] The present disclosure recognizes that the use of such personal information data, in the present technology, can be used to the benefit of users. For example, the personal information data can be used to deliver targeted content that is of greater interest to the user. Accordingly, use of such personal information data enables a user to make planned decisions about the content that is delivered. Further, other uses for personal information data that benefit the user are also contemplated by the present disclosure. For example, health and fitness data can be used to provide insights into a user's general wellness, or can be used as positive feedback to individuals using the technology to pursue a healthy goal.

[0327] The present disclosure contemplates that the entities responsible for the collection, analysis, disclosure, transfer, storage, or other use of such personal information data will comply with established privacy policies and / or privacy practices. In particular, such entities should implement and consistently use privacy policies and practices that are generally recognized as industry best practices. In addition, such entities should adhere to specialized fields' privacy policies and practices, which are generally recognized as industry best practices. However, those skilled in the art will appreciate that the application might be practiced without such dedicated privacy policies and practices. Specifically, such entities should implement and consistently use privacy policies and practices that are generally recognized as industry best practices. In addition, such entities should adhere to specialized field privacy policies and practices which are generally recognized as industry best practices. However, those skilled in the art will appreciate that the application might be practiced without such dedicated privacy policies and practices.

[0328] Regardless of the foregoing, the present disclosure also contemplates embodiments in which users selectively block the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and / or software elements might be provided to prevent or block access to such personal information data. For example, in the case of search functionality services, the present technology can be configured to allow users to select to "opt in" or "opt out" of participation in the collection of personal information data during registration for services or anytime thereafter. In another example, users can select not to provide data to a search functionality service. In addition to providing "opt in" and "opt out" options, the present disclosure contemplates providing notifications relating to the access or use of personal information. For instance, a user can be notified upon download of an application that their personal information data will be accessed. Additionally, the user can be notified when their personal information data is accessed.

[0329] Moreover, it is the intent of the present disclosure that personal information data should be managed and processed in a manner that minimizes risks of unintentional or unauthorized access or use of personal information data. Risk can be minimized, for example, by limiting the collection of data and deleting data once it is no longer needed. In addition, and when applicable, data de-identification can be used to protect the privacy of users. When appropriate, de-identification can be facilitated, for example, by removing specific identifiers (e.g., date of birth, etc.). Additionally or alternatively, de-identification can be facilitated, for example, by controlling the amount or characteristics of data that is stored, e.g., aggregated data at a city level rather than at a street level, or by controlling data storage, e.g., user-level data is aggregated in a database.

[0330] Thus, although the present disclosure broadly covers use of personal information data to implement one or more various disclosed embodiments, the present disclosure also contemplates that the various embodiments can also be implemented without the need for accessing such personal information data. That is, the various embodiments of the present technology are not rendered inoperable due to the lack of, or inaccessibility of such personal information data. For example, content can be selected and delivered to users without access to personal information data about those users, such as by inferring preferences based on non-personal information data or a minimal amount of personal information, such as the content requested by a user's device, other non-personal information available to the search functionality service, or publicly available information.

Claims

1. A computer system, the computer system comprising: One or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for the following operations: Receive a request to display a user interface via one or more input devices; In response to receiving the request to display the user interface, the user interface, including a power indicator, is displayed via a touch-sensitive display, the power indicator including an indication of the number of pages in the user interface; When displaying the user interface: Detect the occurrence of predetermined conditions; as well as In response to the detection of the occurrence of the predetermined condition, the indication of the number of pages in the user interface is updated to an indication of the search function; When displaying the indication having the search function: Detect input on the second page displaying the user interface; as well as In response to detecting the input that displays the second page of the user interface, the second page of the user interface, including the indication of the number of pages of the user interface, is displayed.

2. The computer system of claim 1, wherein receiving the request to display the user interface includes detecting a swipe gesture along a predetermined direction on the touch-sensitive surface.

3. The computer system of claim 1, wherein the power indicator has a predetermined size that is unaffected by the number of pages.

4. The computer system of claim 1, wherein the power indicator has a predetermined position on the first page of the user interface and the second page of the user interface.

5. The computer system of claim 1, wherein the one or more programs further include instructions for performing the following operations: Based on the determination that the number of pages in the user interface is greater than a predetermined number, the display shows a first indication that the number of pages is greater than the predetermined number; and Based on the determination that the number of pages is not greater than the predetermined number, the power indication is displayed with a second indication that the number of pages is not greater than the predetermined number.

6. The computer system of claim 1, wherein the one or more programs further include instructions for performing the following operations: Detect input corresponding to selecting a page in the user interface; and In response to detecting the input corresponding to selecting the page of the user interface, the selected page of the user interface is displayed.

7. The computer system of claim 1, wherein when displayed on a first page of the user interface, a first portion of the indication of the number of pages is emphasized, and when displayed on a second page of the user interface, a second portion of the indication of the number of pages is emphasized.

8. The computer system of claim 1, wherein the one or more programs further include instructions for performing the following operations: When the second page of the user interface is displayed: Detecting a second occurrence of the predetermined condition; and In response to the detection of the second occurrence of the predetermined condition, the capability representation is updated to include the indication of the search function.

9. The computer system of claim 1, wherein the one or more programs further include instructions for performing the following operations: Detect the input of the instruction to select the search function; and In response to the input indicating that the search function has been selected, a search user interface is displayed.

10. The computer system of claim 9, wherein displaying the search user interface further comprises: Concurrent display: User interface elements associated with the search; as well as User interface elements associated with topics of interest.

11. The computer system of claim 10, wherein the subject of interest is based on the context of the computer system.

12. The computer system of claim 10, wherein the one or more programs further include instructions for performing the following operations: The display includes an animation showing the transition from the user interface to the search user interface.

13. The computer system of claim 12, wherein displaying the animation including the transition from the user interface to the search user interface further comprises: The display shows an animation of the user interface element that is associated with the search; as well as The display shows an animation of moving from a first position in the user interface to a second position in the search user interface.

14. The computer system of claim 13, wherein the animation of the power indicator becoming the user interface element associated with the search and the animation of the power indicator moving from the first position in the user interface to the second position in the search user interface are displayed concurrently.

15. A method, the method comprising: In a computer system that includes a touch-sensitive display and one or more input devices: Receive a request to display a user interface via the one or more input devices; In response to receiving the request to display the user interface, the user interface, including a power indicator, is displayed via the touch-sensitive display, the power indicator including an indication of the number of pages in the user interface; When displaying the user interface: Detect the occurrence of predetermined conditions; as well as In response to the detection of the occurrence of the predetermined condition, the indication of the number of pages in the user interface is updated to an indication of the search function; When displaying the indication having the search function: Detect input on the second page displaying the user interface; as well as In response to detecting the input that displays the second page of the user interface, the second page of the user interface, including the indication of the number of pages of the user interface, is displayed.

16. The method of claim 15, wherein receiving the request to display the user interface comprises detecting a swipe gesture along a predetermined direction on the touch-sensitive surface.

17. The method of claim 15, wherein the power representation has a predetermined size that is unaffected by the number of pages.

18. The method of claim 15, wherein the power indicator has a predetermined position on the first page and the second page of the user interface.

19. The method according to claim 15, further comprising: Based on the determination that the number of pages in the user interface is greater than a predetermined number, the display shows a first indication that the number of pages is greater than the predetermined number; as well as Based on the determination that the number of pages is not greater than the predetermined number, the power indication is displayed with a second indication that the number of pages is not greater than the predetermined number.

20. The method of claim 15, further comprising: Detect input corresponding to the page selected in the user interface; as well as In response to detecting the input corresponding to selecting the page of the user interface, the selected page of the user interface is displayed.

21. The method of claim 15, wherein when displayed on a first page of the user interface, a first portion of the indication of the number of pages is emphasized, and when displayed on a second page of the user interface, a second portion of the indication of the number of pages is emphasized.

22. The method according to claim 15, further comprising: When the second page of the user interface is displayed: Detect the second occurrence of the predetermined condition; as well as In response to the detection of the second occurrence of the predetermined condition, the capability representation is updated to include the indication of the search function.

23. The method according to claim 15, further comprising: Detect the input of the instruction to select the search function; as well as In response to the input indicating that the search function has been selected, a search user interface is displayed.

24. The method of claim 23, wherein displaying the search user interface further comprises: Concurrent display: User interface elements associated with the search; as well as User interface elements associated with topics of interest.

25. The method of claim 24, wherein the subject of interest is based on the context of the computer system.

26. The method according to claim 24, further comprising: The display includes an animation showing the transition from the user interface to the search user interface.

27. The method of claim 26, wherein displaying the animation including the transition from the user interface to the search user interface further comprises: The display shows an animation of the user interface element that is associated with the search; as well as The display shows an animation of moving from a first position in the user interface to a second position in the search user interface.

28. The method of claim 27, wherein the animation of the power indicator becoming the user interface element associated with the search and the animation of the power indicator moving from the first position in the user interface to the second position in the search user interface are displayed concurrently.

29. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system, the one or more programs including instructions for: Receive a request to display a user interface via one or more input devices; In response to receiving the request to display the user interface, the user interface, including a power indicator, is displayed via a touch-sensitive display, the power indicator including an indication of the number of pages in the user interface; When displaying the user interface: Detect the occurrence of predetermined conditions; as well as In response to the detection of the occurrence of the predetermined condition, the indication of the number of pages in the user interface is updated to an indication of the search function; When displaying the indication having the search function: Detect input on the second page displaying the user interface; as well as In response to detecting the input that displays the second page of the user interface, the second page of the user interface, including the indication of the number of pages of the user interface, is displayed.

30. The non-transitory computer-readable storage medium of claim 29, wherein receiving the request to display the user interface includes detecting a swipe gesture along a predetermined direction on the touch-sensitive surface.

31. The non-transitory computer-readable storage medium of claim 29, wherein the power indication has a predetermined size unaffected by the number of pages.

32. The non-transitory computer-readable storage medium of claim 29, wherein the power indication has a predetermined position on the first page of the user interface and the second page of the user interface.

33. The non-transitory computer-readable storage medium of claim 29, wherein the one or more programs further include instructions for performing the following operations: Based on the determination that the number of pages in the user interface is greater than a predetermined number, the display shows a first indication that the number of pages is greater than the predetermined number; and Based on the determination that the number of pages is not greater than the predetermined number, the power indication is displayed with a second indication that the number of pages is not greater than the predetermined number.

34. The non-transitory computer-readable storage medium of claim 29, wherein the one or more programs further include instructions for performing the following operations: Detect input corresponding to selecting a page in the user interface; and In response to detecting the input corresponding to selecting the page of the user interface, the selected page of the user interface is displayed.

35. The non-transitory computer-readable storage medium of claim 29, wherein when displayed on a first page of the user interface, a first portion of the indication of the number of pages is emphasized, and when displayed on a second page of the user interface, a second portion of the indication of the number of pages is emphasized.

36. The non-transitory computer-readable storage medium of claim 29, wherein the one or more programs further include instructions for performing the following operations: When the second page of the user interface is displayed: Detecting a second occurrence of the predetermined condition; and In response to the detection of the second occurrence of the predetermined condition, the capability representation is updated to include the indication of the search function.

37. The non-transitory computer-readable storage medium of claim 29, wherein the one or more programs further include instructions for performing the following operations: Detect the input of the instruction to select the search function; and In response to the input indicating that the search function has been selected, a search user interface is displayed.

38. The non-transitory computer-readable storage medium of claim 37, wherein displaying the search user interface further comprises: Concurrent display: User interface elements associated with the search; as well as User interface elements associated with topics of interest.

39. The non-transitory computer-readable storage medium of claim 38, wherein the subject matter of interest is based on the context of the computer system.

40. The non-transitory computer-readable storage medium of claim 38, wherein the one or more programs further include instructions for performing the following operations: The display includes an animation showing the transition from the user interface to the search user interface.

41. The non-transitory computer-readable storage medium of claim 40, wherein displaying the animation including the transition from the user interface to the search user interface further comprises: The display shows an animation of the user interface element that is associated with the search; as well as The display shows an animation of moving from a first position in the user interface to a second position in the search user interface.

42. The non-transitory computer-readable storage medium of claim 41, wherein the animation of the power indicator becoming a user interface element associated with a search and the animation of the power indicator moving from the first position in the user interface to the second position in the search user interface are displayed concurrently.

Images