Contextual response proposals using secondary electronic devices

Abstract

This system and process provides context message suggestion. [Solution] A system and process for context message suggestion are provided. In some embodiments, a first electronic device receives a message and determines a response message based on the received message. In some embodiments, the first electronic device causes a second electronic device to provide a prompt including the received message and instructions for the response message. In some embodiments, upon determination that an input in response to the prompt has been received from the second electronic device, the first electronic device transmits a response message upon determination that the received input corresponds to acceptance of the prompt. In some embodiments, upon determination that an input has been received, the first electronic device cancels the transmission of a response message upon determination that the received input corresponds to rejection of the prompt.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims priority to U.S. Patent Application No. 18 / 373,211, titled "CONTEXTUAL RESPONSE SUGGESTIONS USING SECONDARY ELECTRONIC DEVICE", filed on September 26, 2023, and U.S. Provisional Patent Application No. 63 / 462,956, titled "CONTEXTUAL RESPONSE SUGGESTIONS USING SECONDARY ELECTRONIC DEVICE", filed on April 28, 2023. The entire content of each of these applications is hereby incorporated by reference into this specification for all purposes.

[0002] This application generally relates to communication sessions, and more particularly to contextual response suggestions.

Background Art

[0003] Electronic devices provide many beneficial features, such as the ability to communicate with other devices via various modalities. In particular, a user can utilize wearable devices such as smartwatches or earphones to quickly send messages to others. For example, earphones wirelessly connected to a device such as a smartphone can provide a convenient way for a user to listen to messages for another user. However, conventional devices do not provide intelligent and contextually relevant response suggestions based on currently available information. For example, in response to a question from another user, a typical messaging interface usually provides default or generalized response suggestions that often lack detail or context recognition. Thus, a user may choose to instead input a custom message, which takes additional time and thus does not provide an optimized user experience. Therefore, an improved system for contextual response suggestions is desired.

Summary of the Invention

[0004] A system and process for context message suggestion are provided. In some embodiments, a first electronic device receives a message and determines a response message based on the received message. In some embodiments, the first electronic device causes a second electronic device to provide a prompt including the received message and instructions for the response message. In some embodiments, upon determination that an input in response to the prompt has been received from the second electronic device, the first electronic device transmits a response message, upon determination that the received input corresponds to acceptance of the prompt. In some embodiments, upon determination that an input has been received, the first electronic device cancels the transmission of a response message, upon determination that the received input corresponds to rejection of the prompt. [Brief explanation of the drawing]

[0005] [Figure 1] This block diagram shows systems and environments for implementing digital assistants in various embodiments.

[0006] [Figure 2A] This block diagram shows a portable multifunctional device implementing the client-side portion of a digital assistant, with various implementations.

[0007] [Figure 2B] This is a block diagram showing exemplary components for event processing in various embodiments.

[0008] [Figure 3] This document illustrates portable, multifunctional devices that implement the client-side portion of a digital assistant through various embodiments.

[0009] [Figure 4] This is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface, according to various embodiments.

[0010] [Figure 5A] A diagram showing an exemplary user interface related to the menu of an application on a portable multi-functional device according to various embodiments.

[0011] [Figure 5B] A diagram showing an exemplary user interface related to a multi-functional device having a touch-sensitive surface separate from the display according to various embodiments.

[0012] [Figure 6A] A diagram showing a personal electronic device according to various embodiments.

[0013] [Figure 6B] A block diagram showing a personal electronic device according to various embodiments.

[0014] [Figure 7A] A block diagram showing a digital assistant system or its server part according to various embodiments.

[0015] [Figure 7B] A diagram showing the functions of the digital assistant shown in FIG. 7A according to various embodiments.

[0016] [Figure 7C] A diagram showing a part of an ontology according to various embodiments.

[0017] [Figure 8] A system for context-responsive suggestions using earphones is shown according to various embodiments.

[0018] [Figure 9A] A system for context-responsive suggestions using a smartwatch device is shown according to various embodiments. [Figure 9B]Systems for context - responsive suggestions using a smartwatch device according to various embodiments are shown. [Figure 9C] Systems for context - responsive suggestions using a smartwatch device according to various embodiments are shown.

[0019] [Figure 10] Systems for context - responsive suggestions using a smartwatch device according to various embodiments are shown

[0020] [Figure 11] Processes for context - responsive suggestions according to various embodiments are shown.

[0021] [Figure 12] Processes for context - responsive suggestions according to various embodiments are shown.

Best Mode for Carrying Out the Invention

[0022] In the following description of the embodiments, reference is made to the accompanying drawings, which show specific embodiments that are practicable as examples. It should be understood that other embodiments can be used and structural changes can be made without departing from the scope of the various embodiments.

[0023] Generally, the conventional technology related to context - responsive suggestions needs improvement. Conventional messaging systems with two devices, such as a primary smartphone and communicatively coupled earphones, do not provide an efficient mechanism for recommending response suggestions to the user. For example, existing systems typically provide default or standard response suggestions that are not usually based on the context information and current information associated with the user. Therefore, the user tends to either use such suggestions even though they are inaccurate or spend time creating custom responses. Thus, an improved system for context - responsive suggestions is desired.

[0024] In the following description, terms such as "first," "second," etc., are used to describe various elements, but these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, without deviating from the scope of the various embodiments described, it is possible to call the first input the second input, and similarly, the second input the first input. Both the first input and the second input are inputs, and in some cases are distinct and different inputs.

[0025] The terminology used in the descriptions of the various embodiments described herein is intended solely to describe specific embodiments and not to limit them. In the descriptions of the various embodiments and in the appended claims, the singular forms “a,” “an,” and “the” are intended to also include the plural form unless the context explicitly indicates otherwise. Furthermore, the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the enumerated items relating to the description. It will be further understood that the terms “includes,” “comprises,” and / or “comprising,” as used herein, specify the presence of the described features, integers, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof.

[0026] The term "if" can be interpreted, depending on the context, as meaning "when," "upon," "in response to determining," or "in response to detecting." Similarly, the phrases "if it is determined" or "if [a stated condition or event] is detected" can be interpreted, depending on the context, as meaning "upon determining," "in response to determining," "upon detecting [the stated condition or event]," or "in response to detecting [the stated condition or event]." 1. System and Environment

[0027] Figure 1 shows block diagrams of System 100 in various embodiments. In some embodiments, System 100 implements a digital assistant. The terms “digital assistant,” “virtual assistant,” “intelligent automated assistant,” or “automated digital assistant” refer to any information processing system that infers user intent by interpreting natural language input in oral and / or textual form and performs actions based on that inferred user intent. For example, in order to act based on inferred user intent, the system performs one or more of the following: identifying a task flow having steps and parameters designed to perform the inferred user intent; inputting specific requirements from the inferred user intent into the task flow; executing the task flow by calling a program, method, service, API, etc.; and generating an output response to the user in audible (e.g., voice) and / or visual form.

[0028] Specifically, a digital assistant can accept user requests, at least partially, in the form of natural language commands, requests, opinions, discourses, and / or inquiries. Typically, a user request asks the digital assistant to either provide information or perform a task. A satisfactory response to a user request includes providing the requested information, performing the requested task, or a combination of both. For example, a user might ask the digital assistant, "Where am I right now?" Based on the user's current location, the digital assistant might respond, "You are near the West Gate of Central Park." The user might also request the performance of a task, such as, "Please invite my friends to my girlfriend's birthday party next week." In response, the digital assistant could affirm the request by saying, "Yes, right now," and then send appropriate calendar invitations on behalf of the user to each of the user's friends listed in the user's electronic address book. While performing a requested task, a digital assistant may interact with the user in a continuous conversation involving multiple information exchanges over extended periods. Numerous other ways exist to interact with a digital assistant to request information or to request the performance of various tasks. In addition to providing verbal responses and taking programmed actions, digital assistants may also provide responses in other visual or audio formats, such as text, alarms, music, videos, and animations.

[0029] As shown in Figure 1, in some embodiments, the digital assistant is implemented according to a client-server model. The digital assistant includes a client-side portion 102 (hereinafter, "DA client 102") that runs on a user device 104, and a server-side portion 106 (hereinafter, "DA server 106") that runs on a server system 108. The DA client 102 communicates with the DA server 106 through one or more networks 110. The DA client 102 provides client-side functions such as user-responsive input and output processing, and communication with the DA server 106. The DA server 106 provides server-side functions to any number of DA clients 102, each residing on an individual user device 104.

[0030] In some embodiments, the DA server 106 includes a client-responsive I / O interface 112, one or more processing modules 114, data and models 116, and an I / O interface 118 to external services. The client-responsive I / O interface 112 facilitates client-responsive input and output processing of the DA server 106. One or more processing modules 114 utilize the data and models 116 to process voice input and determine user intent based on natural language input. Furthermore, one or more processing modules 114 perform task execution based on the inferred user intent. In some embodiments, the DA server 106 communicates with external services 120 via a network(s) 110 to complete tasks or retrieve information. The I / O interface 118 to external services facilitates such communication.

[0031] The user device 104 can be any suitable electronic device. In some embodiments, the user device 104 is a portable multifunction device (e.g., device 200 described below in relation to Figure 2A), a multifunction device (e.g., device 400 described below in relation to Figure 4), or a personal electronic device (e.g., device 600 described below in relation to Figures 6A and 6B). A portable multifunction device is, for example, a mobile phone that also includes other functions such as a PDA and / or music player function. Specific examples of portable multifunction devices include Apple Watch®, iPhone®, iPod Touch®, and iPad® devices manufactured by Apple Inc. (Cupertino, California). Other examples of portable multifunction devices include, but are not limited to, earphones / headphones, speakers, and laptop computers or tablet computers. Furthermore, in some embodiments, the user device 104 is a non-portable multifunction device. Specifically, the user device 104 is a desktop computer, a game console, a speaker, a television, or a television set-top box. In some embodiments, the user device 104 includes a touch-sensitive surface (e.g., a touchscreen display and / or touchpad). Furthermore, the user device 104 optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick. Various embodiments of electronic devices, such as multifunction devices, are described in further detail below.

[0032] Examples of communication networks (one or more) 110 include local area networks (LANs) and wide area networks (WANs), such as the Internet. Communication networks (one or more) 110 are implemented using any known network protocol, including a variety of wired or wireless protocols, such as Ethernet, Universal Serial Bus (USB), FireWire®, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth®, Wi-Fi®, Voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol.

[0033] The server system 108 is implemented on one or more standalone data processing devices or a distributed computer network. In some embodiments, the server system 108 also employs various virtual devices and / or services from third-party service providers (e.g., third-party cloud service providers) to provide the basic computing and / or infrastructure resources of the server system 108.

[0034] In some embodiments, user device 104 communicates with DA server 106 via a second user device 122. The second user device 122 is similar to or identical to user device 104. For example, the second user device 122 is similar to devices 200, 400, or 600 described below in relation to Figures 2A, 4, and 6A and 6B. User device 104 is configured to be communicatively coupled to the second user device 122 via a direct communication connection such as Bluetooth, NFC, or BTLE, or via a wired or wireless network such as a local Wi-Fi network. In some embodiments, the second user device 122 is configured to act as a proxy between user device 104 and DA server 106. For example, the DA client 102 of user device 104 is configured to transmit information (e.g., user requests received by user device 104) to the DA server 106 via the second user device 122. The DA server 106 processes the information and returns the relevant data (e.g., data content responding to a user request) to the user device 104 via the second user device 122.

[0035] In some embodiments, user device 104 is configured to communicate a shortened request for data to a second user device 122 in order to reduce the amount of information transmitted from user device 104. The second user device 122 is configured to determine supplementary information to be added to the shortened request in order to generate a complete request to send to DA server 106. This system architecture allows user device 104 (e.g., a wristwatch or similar small electronic device) with limited communication capabilities and / or limited battery power to advantageously access services provided by DA server 106 by using a second user device 122 (e.g., a mobile phone, laptop computer, tablet computer, etc.) with higher communication capabilities and / or battery power as a proxy to DA server 106. Although only two user devices 104 and 122 are shown in Figure 1, it should be understood that system 100 includes any number and types of user devices configured to communicate with DA server system 106 in this proxy configuration in some embodiments.

[0036] The digital assistant shown in Figure 1 includes both a client-side portion (e.g., DA client 102) and a server-side portion (e.g., DA server 106), however, in some embodiments, the functions of the digital assistant are implemented as a standalone application installed on the user device. Furthermore, the division of functions between the client and server portions of the digital assistant may vary depending on the implementation. For example, in some embodiments, the DA client is a thin client that provides only user-responsive input and output processing functions, and delegates all other functions of the digital assistant to the backend server. 2. Electronic devices

[0037] Here, we focus on embodiments of electronic devices for implementing the client-side portion of a digital assistant. Figure 2A is a block diagram of a portable multifunction device 200 comprising a touch-sensitive display system 212, according to some embodiments. The touch-sensitive display 212 may be conveniently referred to as a “touchscreen” and may be known or referred to as a “touch-sensitive display system.” Device 200 includes memory 202 (optionally including one or more computer-readable storage media), a memory controller 222, one or more processing units (CPUs) 220, a peripheral interface 218, an RF circuit 208, an audio circuit 210, a speaker 211, a microphone 213, an input / output (I / O) subsystem 206, other input control devices 216, and an external port 224. Device 200 optionally includes one or more optical sensors 264. Device 200 optionally includes one or more contact intensity sensors 265 (e.g., touch-sensitive surfaces such as the touch-sensitive display system 212 of device 200) for detecting the intensity of contact on device 200. Device 200 optionally includes one or more tactile output generators 267 for generating tactile output on device 200 (e.g., generating tactile output on touch-sensitive surfaces such as the touch-sensitive display system 212 of device 200 or the touchpad 455 of device 400). These components optionally communicate via one or more communication buses or signal lines 203.

[0038] As used herein and in the claims, the term “strength” of contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of contact on the touch-sensitive surface (e.g., finger contact), or a proxy for the force or pressure of contact on the touch-sensitive surface. The strength of contact has a range of values, including at least four distinct values, and more typically, including several hundred (e.g., at least 256) distinct values. The strength of contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted averaged) to determine the estimated force of contact. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size and / or change in the contact area detected on the touch-sensitive surface, the capacitance and / or change in the touch-sensitive surface adjacent to the contact, and / or the resistance and / or change in the resistance of the touch-sensitive surface adjacent to the contact may optionally be used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some implementations, the substitute measurement for the force or pressure of the contact is used directly to determine whether it exceeds an intensity threshold (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some implementations, the substitute measurement for the contact force or pressure is converted into an estimate of the force or pressure, and this estimate is used to determine whether it exceeds an intensity threshold (e.g., the intensity threshold is a pressure threshold measured in units of pressure).By using the intensity of contact as an attribute of user input, it becomes possible for users to access (e.g., on a touch-sensitive display) and / or receive user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical control such as a knob or button) additional device functions that might otherwise be inaccessible to the user, on reduced-size devices where the implementation area for displaying affordances is limited.

[0039] As used herein and in the claims, the term “tactile output” means a physical displacement of the device relative to its previous position, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by the user through the user’s sense of touch. For example, in a situation where the device or a component of the device is in contact with the touch-sensitive surface of the user (e.g., fingers, palm, or other part of the user’s hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical properties of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) may be optionally interpreted by the user as a “down-click” or “up-click” of a physical actuator button. In some cases, the user may perceive a tactile sensation such as a “down-click” or “up-click” even when there is no movement of a physical actuator button associated with a touch-sensitive surface that has been physically pressed (e.g., displaced) by the user’s action. In another embodiment, movement of a touch-sensitive surface may be optionally interpreted or perceived by the user as "roughness" of the touch-sensitive surface, even when there is no change in the smoothness of the touch-sensitive surface. Such interpretations of touch by the user depend on the user's personal sensory perception, but there are many touch sensory perceptions common to the majority of users. Therefore, when a tactile output is described as corresponding to a user's specific sensory perception (e.g., "up-click," "down-click," "roughness"), unless otherwise stated, the generated tactile output corresponds to the physical displacement of the device or its components that produce the described sensory perception of a typical (or average) user.

[0040] It should be understood that device 200 is merely an example of a portable multifunction device, and that device 200 may optionally have more or fewer components than those shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of those components. The various components shown in Figure 2A may be implemented as hardware, software, or a combination of both hardware and software, including one or more signal processing circuits and / or application-specific integrated circuits.

[0041] Memory 202 includes one or more computer-readable storage media. These computer-readable storage media are, for example, tangible and non-temporary. Memory 202 includes high-speed random-access memory and also includes non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. The memory controller 222 controls access to memory 202 by other components of device 200.

[0042] In some embodiments, the non-temporary computer-readable storage medium of memory 202 is used to store instructions (for example, for performing the actions of the actions described below) for use by or in connection with instruction execution systems, devices, or other systems capable of fetching instructions from such instruction execution systems, devices, or devices and executing those instructions, such as computer-based systems or systems including processors. In other embodiments, the instructions (for example, for performing the actions of the actions described below) are stored in the non-temporary computer-readable storage medium (not shown) of the server system 108, or are shared between the non-temporary computer-readable storage medium of memory 202 and the non-temporary computer-readable storage medium of the server system 108.

[0043] The peripheral interface 218 is used to connect the input and output peripherals of this device to the CPU 220 and memory 202. One or more processors 220 execute or implement various software programs and / or instruction sets stored in memory 202 to perform various functions for the device 200 and process data. In some embodiments, the peripheral interface 218, CPU 220, and memory controller 222 are implemented on a single chip, such as chip 204. In some other embodiments, they are implemented on separate chips.

[0044] The RF (radio frequency) circuit 208 transmits and receives RF signals, also known as electromagnetic signals. The RF circuit 208 converts electrical signals to electromagnetic signals or electromagnetic signals to electrical signals and communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 208 optionally includes, but is not limited to, well-known circuits for performing these functions, such as antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, CODEC chipsets, subscriber identity module (SIM) cards, and memory. The RF circuit 208 optionally communicates wirelessly with networks such as the Internet, also known as the World Wide Web (WWW), intranets, and / or wireless networks such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs), as well as with other devices. The RF circuit 208 optionally includes well-known circuits for detecting near-field communication (NFC) fields by short-range communication radios, etc. The wireless communication optionally includes Global System for Mobile Communications (GSM), Extended Data GSM Environment (EDGE), High-Speed ​​Downlink Packet Access (HSDPA), High-Speed ​​Uplink Packet Access (HSUPA), Evolution Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), Long-Term Evolution (LTE), Near-Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), and Code Division Multiple Access (codeDivision Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE®), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX®, Email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), Instant Messaging (e.g., Extensible Messaging and Presence Protocol (XMPP)), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions) Using any of several communication standards, protocols, and technologies, including, but not limited to, Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS), and / or Short Message Service (SMS), or any other suitable communication protocol, including a communication protocol not yet developed as of the filing date of this specification.

[0045] The audio circuit 210, speaker 211, and microphone 213 provide an audio interface between the user and the device 200. The audio circuit 210 receives audio data from the peripheral interface 218, converts this audio data into an electrical signal, and transmits this electrical signal to the speaker 211. The speaker 211 converts the electrical signal into human audible sound waves. The audio circuit 210 also receives the electrical signal converted from the sound waves by the microphone 213. The audio circuit 210 converts the electrical signal into audio data and transmits this audio data to the peripheral interface 218 for processing. The audio data is retrieved by the peripheral interface 218 from memory 202 and / or RF circuit 208, and / or transmitted to memory 202 and / or RF circuit 208. In some embodiments, the audio circuit 210 also includes a headset jack (e.g., 312 in Figure 3). The headset jack provides an interface between the audio circuit 210 and detachable audio input / output peripherals such as output-only headphones or headsets that have both output (e.g., headphones for one or both ears) and input (e.g., a microphone).

[0046] The I / O subsystem 206 connects input / output peripherals on device 200, such as the touchscreen 212 and other input control devices 216, to the peripheral interface 218. The I / O subsystem 206 optionally includes a display controller 256, an optical sensor controller 258, an intensity sensor controller 259, a haptic feedback controller 261, and one or more input controllers 260 for other input or control devices. One or more input controllers 260 receive electrical signals from / transmit electrical signals to other input control devices 216. The other input control devices 216 optionally include physical buttons (e.g., push buttons, rocker buttons), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, one or more input controllers 260 are optionally connected to (or not connected to) any of the following: a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. One or more buttons (e.g., 308 in Figure 3) optionally include up / down buttons for volume control of speaker 211 and / or microphone 213. One or more buttons optionally include push buttons (e.g., 306 in Figure 3).

[0047] As described in U.S. Patent Application No. 11 / 322,549, filed December 23, 2005, “Unlocking a Device by Performing Gestures on an Unlock Image,” which is incorporated herein by reference in its entirety, a quick press of a push button unlocks the touchscreen 212 or initiates the process of using gestures on the touchscreen to unlock the device. A long press of a push button (e.g., 306) turns the device 200 on or off. The user can customize the function of one or more buttons. The touchscreen 212 is used to implement virtual or soft buttons and one or more soft keyboards.

[0048] The touch-sensitive display 212 provides input and output interfaces between the device and the user. The display controller 256 receives electrical signals from and / or transmits electrical signals to the touchscreen 212. The touchscreen 212 displays visual output to the user. Visual output includes graphics, text, icons, videos, and any combination thereof (collectively referred to as “graphics”). In some embodiments, some or all of the visual output corresponds to user interface objects.

[0049] The touchscreen 212 has a touch-sensing surface, sensor, or set of sensors that accept user input based on touch and / or tactile contact. The touchscreen 212 and display controller 256 (together with any associated modules and / or instruction sets in memory 202) detect contact (and any movement or interruption of contact) on the touchscreen 212 and translate the detected contact into interaction with user interface objects displayed on the touchscreen 212 (e.g., one or more soft keys, icons, web pages, or images). In an exemplary embodiment, the point of contact between the touchscreen 212 and the user corresponds to the user's finger.

[0050] The touchscreen 212 uses LCD (liquid crystal display) technology, LPD (polymer light-emitting display) technology, or LED (light-emitting diode) technology, but other display technologies may be used in other embodiments. The touchscreen 212 and display controller 256 detect contact and any movement or disconnection thereof using any of several currently known or future-developed touch sensing technologies, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, and other proximity sensor arrays or other elements for determining one or more contact points using the touchscreen 212. In exemplary embodiments, projected mutual capacitive sensing technology is used, such as that found in the iPhone® and iPod Touch® manufactured by Apple Inc. (Cupertino, California).

[0051] The touch-sensitive displays in some embodiments of the touchscreen 212 are similar to the multi-touch-sensitive touchpads described in U.S. Patents 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.) and / or 6,677,932 (Westerman) and / or U.S. Patent Publication 2002 / 0015024 (A1), each of which is incorporated herein by reference in whole. However, while the touchscreen 212 displays visual output from device 200, the touch-sensitive touchpads do not provide visual output.

[0052] The touch-sensitive display in some embodiments of the touchscreen 212 is based on the following applications: (1) U.S. Patent Application No. 11 / 381,313, filed May 2, 2006, "Multipoint Touch Surface Controller"; (2) U.S. Patent Application No. 10 / 840,862, filed May 6, 2004, "Multipoint Touchscreen"; (3) U.S. Patent Application No. 10 / 903,964, filed July 30, 2004, "Gestures For Touch Sensitive Input Devices"; (4) U.S. Patent Application No. 11 / 048,264, filed January 31, 2005, "Gestures For Touch Sensitive Input Devices"; (5) U.S. Patent Application No. 11 / 038,590, filed January 18, 2005, "Mode-Based Graphical User Interfaces For Touch Sensitive Input Such devices are described in (6) U.S. Patent Application No. 11 / 228,758, filed September 16, 2005, "Virtual Input Device Placement On A Touch Screen User Interface," (7) U.S. Patent Application No. 11 / 228,700, filed September 16, 2005, "Operation Of A Computer With A Touch Screen Interface," (8) U.S. Patent Application No. 11 / 228,737, filed September 16, 2005, "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard," and (9) U.S. Patent Application No. 11 / 367,749, filed March 3, 2006, "Multi-Functional Hand-Held Device." All of these applications are incorporated herein by reference in their entirety.

[0053] The touchscreen 212 has a video resolution of, for example, more than 100 dpi. In some embodiments, the touchscreen has a video resolution of about 160 dpi. The user touches the touchscreen 212 using a suitable object or attachment such as a stylus or finger. In some embodiments, the user interface is designed to operate primarily using finger touch and gestures, which may be less precise than stylus input due to the larger contact area of ​​the finger on the touchscreen. In some embodiments, the device translates coarse finger input into a precise pointer / cursor position or command to perform an action desired by the user.

[0054] In some embodiments, in addition to the touchscreen, the device 200 includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touchscreen, does not display a visual output. The touchpad is either a separate touch-sensitive surface from the touchscreen 212 or an extension of the touch-sensitive surface formed by the touchscreen.

[0055] Device 200 also includes a power system 262 for supplying power to various components. The power system 262 includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharge system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in the portable device.

[0056] Device 200 also includes one or more optical sensors 264. Figure 2A shows the optical sensors coupled to an optical sensor controller 258 in the I / O subsystem 206. The optical sensors 264 include charge-coupled devices (CCDs) or complementary metal-oxide semiconductor (CMOS) phototransistors. The optical sensors 264 receive light from the environment projected through one or more lenses and convert that light into data representing an image. The optical sensors 264 work in conjunction with an imaging module 243 (also called a camera module) to capture still images or video. In some embodiments, the optical sensors are located on the back of Device 200, opposite the touchscreen display 212 on the front of the device, so that the touchscreen display is used as a viewfinder for acquiring still and / or video images. In some embodiments, the optical sensors are located on the front of the device so that an image of the user for video conferencing is acquired while the user is viewing other video conferencing participants on the touchscreen display. In some embodiments, the position of the optical sensor 264 can be changed by the user (for example, by rotating the lens and sensor within the device housing), so that a single optical sensor 264 can be used in conjunction with the touchscreen display for both video conferencing and the acquisition of still and / or video images.

[0057] Device 200 also optionally includes one or more contact intensity sensors 265. Figure 2A shows a contact intensity sensor coupled to an intensity sensor controller 259 in the I / O subsystem 206. The contact intensity sensor 265 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electric force sensors, pressure-power sensors, optical force sensors, capacitive touch-sensing surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of contact on a touch-sensing surface). The contact intensity sensor 265 receives contact intensity information (e.g., pressure information, or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is positioned juxtaposed with or adjacent to a touch-sensing surface (e.g., a touch-sensing display system 212). In some embodiments, at least one contact intensity sensor is located on the back of Device 200, opposite the touchscreen display 212 located on the front of Device 200.

[0058] Device 200 also includes one or more proximity sensors 266. Figure 2A shows a proximity sensor 266 coupled to a peripheral interface 218. Alternatively, the proximity sensor 266 is coupled to an input controller 260 in an I / O subsystem 206. The proximity sensor 266 functions as described in U.S. Patent Applications 11 / 241,839, “Proximity Detector In Handheld Device,” 11 / 240,788, “Proximity Detector In Handheld Device,” 11 / 620,702, “Using Ambient Light Sensor To Augment Proximity Sensor Output,” 11 / 586,862, “Automated Response To And Sensing Of User Activity In Portable Devices,” and 11 / 638,251, “Methods And Systems For Automatic Configuration Of Peripherals,” which are all incorporated herein by reference. In some embodiments, when the multifunction device is placed near the user's ear (for example, when the user is making a phone call), the proximity sensor turns off and disables the touchscreen 212.

[0059] Device 200 also optionally includes one or more tactile output generators 267. Figure 2A shows a tactile output generator coupled to a tactile feedback controller 261 in the I / O subsystem 206. The tactile output generator 267 optionally includes one or more electroacoustic devices such as a speaker or other audio component, and / or electromechanical devices that convert energy into linear motion, such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component that converts an electrical signal into a tactile output on the device). The contact intensity sensor 265 receives a tactile feedback generation command from the tactile feedback module 233 and generates a tactile output on device 200 that can be sensed by the user of device 200. In some embodiments, at least one tactile output generator is positioned alongside or adjacent to a touch-sensing surface (e.g., a touch-sensing display system 212) and optionally generates a tactile output by moving the touch-sensing surface vertically (e.g., inward / outward from the surface of the device 200) or horizontally (e.g., forward / backward in the same plane as the surface of the device 200). In some embodiments, at least one tactile output generator sensor is located on the back of the device 200, opposite the touchscreen display 212 which is located on the front of the device 200.

[0060] Device 200 also includes one or more accelerometers 268. Figure 2A shows an accelerometer 268 coupled to a peripheral interface 218. Alternatively, the accelerometer 268 is coupled to an input controller 260 in an I / O subsystem 206. The accelerometer 268 operates as described, for example, in U.S. Patent Publication No. 20050190059, “Acceleration-based Theft Detection System for Portable Electronic Devices,” and U.S. Patent Publication No. 20060017692, “Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer,” both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed on a touchscreen display in portrait or landscape view based on an analysis of data received from one or more accelerometers. Device 200 optionally includes, in addition to one or more accelerometers 268, a magnetometer (not shown), and a GPS (or GLONASS or other global navigation system) receiver (not shown) for acquiring information regarding the position and orientation of Device 200 (e.g., portrait or landscape).

[0061] In some embodiments, the software components stored in memory 202 include an operating system 226, a communications module (or instruction set) 228, a contact / motion module (or instruction set) 230, a graphics module (or instruction set) 232, a text input module (or instruction set) 234, a Global Positioning System (GPS) module (or instruction set) 235, a digital assistant client module 229, and an application (or instruction set) 236. Furthermore, memory 202 stores data and models, such as user data and models 231. In addition, in some embodiments, as shown in Figures 2A and 4, memory 202 (Figure 2A) or memory 470 (Figure 4) stores device / global internal state 257. The device / global internal state 257 includes one or more of the following: an active application state indicating which application is currently active, if there is an active application; a display state indicating which applications, views, or other information occupy different areas of the touchscreen display 212; a sensor state including information obtained from various sensors and input control devices 216 of the device; and position information relating to the device's position and / or orientation.

[0062] An operating system 226 (for example, embedded operating systems such as Darwin, RTXC, LINUX, UNIX®, OS X, iOS, WINDOWS®, or VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware components and software components.

[0063] The communication module 228 facilitates communication with other devices via one or more external ports 224 and also includes various software components for processing data received by the RF circuit 208 and / or the external ports 224. The external ports 224 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted to connect to other devices directly or indirectly via a network (e.g., the Internet, Wi-Fi, etc.). In some embodiments, the external ports are multi-pin (e.g., 30-pin) connectors that are the same as and / or compatible with the 30-pin connector used on iPod® (a trademark of Apple Inc.) devices.

[0064] The contact / motion module 230 optionally detects contact with the touchscreen 212 and other touch-sensitive devices (e.g., touchpad or physical click wheel) (in cooperation with the display controller 256). The contact / motion module 230 includes various software components for performing various contact detection operations, such as determining whether contact has occurred (e.g., detecting a finger down event), determining the intensity of the contact (e.g., the force or pressure of the contact, or something equivalent to the force or pressure of the contact), determining whether there is movement of contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more events of a finger dragging), and determining whether contact has been terminated (e.g., detecting a finger up event or interruption of contact). The contact / motion module 230 receives contact data from the touch-sensitive surface. Determining the movement of the contact point, represented by a series of contact data, optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These actions can be optionally applied to a single contact (e.g., a single finger contact) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contacts). In some embodiments, the contact / motion module 230 and the display controller 256 detect contact on the touchpad.

[0065] In some embodiments, the contact / motion module 230 uses a set of one or more intensity thresholds to determine whether an action has been performed by a user (for example, to determine whether a user has "clicked" on an icon). In some embodiments, at least a subset of the intensity thresholds is determined according to software parameters (for example, the intensity thresholds can be adjusted without modifying the physical hardware of the device 200, rather than being determined by the activation threshold of a particular physical actuator). For example, the mouse "click" threshold for a trackpad or touchscreen display can be set to one of a wide range of default thresholds without modifying the trackpad or touchscreen display hardware. In addition, some implementations provide the user of the device with software settings to adjust one or more of the set of intensity thresholds (for example, by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once using a system-level click "intensity" parameter).

[0066] The contact / motion module 230 optionally detects gesture input from the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motion, timing, and / or intensity of the detected contact). Therefore, gestures are optionally detected by detecting a specific contact pattern. For example, detecting a finger tap gesture involves detecting a finger down event, followed by a finger up (lift-off) event at the same position (or substantially the same position) as the finger down event (e.g., the position of the icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface involves detecting a finger down event, followed by one or more finger drag events, and then a finger up (lift-off) event.

[0067] The graphics module 232 includes various known software components for rendering and displaying graphics on the touchscreen 212 or other display, including components that modify the visual effects of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual properties). As used herein, the term “graphics” includes any object that can be displayed to a user, and such objects include, but are not limited to, text, web pages, icons (including user interface objects such as soft keys), digital images, videos, and animations.

[0068] In some embodiments, the graphics module 232 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 232 receives one or more codes from an application or the like, specifying the graphics to be displayed, along with coordinate data and other graphic property data as needed, and then generates screen image data to output to the display controller 256.

[0069] The haptic feedback module 233 includes various software components for generating commands used by a haptic output generator(s) 267 to generate haptic outputs at one or more locations on the device 200 in response to user interaction with the device 200.

[0070] In some embodiments, a text input module 234, which is a component of the graphics module 232, provides a soft keyboard for entering text in various applications (e.g., contacts 237, email 240, IM 241, browser 247, and any other applications that require text input).

[0071] The GPS module 235 determines the location of the device and provides this information for use in various applications (for example, to the telephone 238 for use in location-based dialing, to the camera 243 as picture / video metadata, and to applications that provide location-based services such as weather widgets, local yellow pages widgets, and map / navigation widgets).

[0072] The digital assistant client module 229 includes various client-side digital assistant commands to provide client-side functions of the digital assistant. For example, the digital assistant client module 229 can accept voice input (e.g., speech input), text input, touch input, and / or gesture input through various user interfaces of the portable multifunction device 200 (e.g., microphone 213, accelerometer(s) 268, touch-sensitive display system 212, optical sensor(s) 264, other input control devices 216, etc.). The digital assistant client module 229 can also provide audio (e.g., voice output), visual, and / or tactile output, etc., through various output interfaces of the portable multifunction device 200 (e.g., speaker 211, touch-sensitive display system 212, tactile output generator(s) 267, etc.). For example, the output is provided as voice, sound, alarm, text message, menu, graphic, video, animation, vibration, and / or a combination of two or more of the above. During operation, the digital assistant client module 229 communicates with the DA server 106 using the RF circuit 208.

[0073] User data and model 231 include various user-related data (e.g., user-specific vocabulary data, user preference data, pronunciation of user-specified names, data from the user's electronic address book, to-do lists, shopping lists, etc.) to provide client-side functionality for the digital assistant. Furthermore, user data and model 231 include various models (e.g., speech recognition models, statistical language models, natural language processing models, ontologs, task flow models, service models, etc.) to process user input and determine user intent.

[0074] In some embodiments, the digital assistant client module 229 utilizes various sensors, subsystems, and peripherals of the portable multifunction device 200 to collect additional information from the surrounding environment of the portable multifunction device 200 in order to establish a context associated with the user, the current user interaction, and / or the current user input. In some embodiments, the digital assistant client module 229 provides the DA server 106 with context information, or a subset thereof, along with the user input, to help infer the user's intent. In some embodiments, the digital assistant also uses the context information to determine how to prepare and deliver output to the user. Context information is referred to as context data.

[0075] In some embodiments, contextual information associated with user input includes sensor information, such as lighting, ambient noise, ambient temperature, and images or videos of the surrounding environment. In some embodiments, contextual information may also include the physical state of the device, such as the orientation of the device, the location of the device, the temperature of the device, power level, speed, acceleration, motion pattern, and cellular signal strength. In some embodiments, the software state of the DA server 106, such as information on processes being performed, installed programs, past and present network activity, background services, error logs, and resource usage, as well as information on the software state of the portable multifunction device 200, are provided to the DA server 106 as contextual information associated with user input.

[0076] In some embodiments, the digital assistant client module 229 selectively provides information (e.g., user data 231) stored on the portable multifunction device 200 in response to a request from the DA server 106. In some embodiments, the digital assistant client module 229 also elicits additional input from the user via a natural language dialog or other user interface in response to a request from the DA server 106. The digital assistant client module 229 passes its additional input to the DA server 106 to assist the DA server 106 in intent inference and / or intent performance of the user's intent expressed in the user request.

[0077] A more detailed description of the digital assistant is provided below with reference to Figures 7A to 7C. It should be noted that the digital assistant client module 229 may contain any number of submodules of the digital assistant module 726, as described below.

[0078] Application 236 includes the following modules (or instruction sets), or subsets or supersets thereof: ● Contact module 237 (sometimes called the address book or contact list), ●Telephone module 238, ●Video conferencing module 239, ● Email client module 240, ● Instant messaging (IM) module 241, ●Training support module 242, ● Camera module 243 for still images and / or video images, ●Image management module 244, ●Video player module, ● Music player module, ● Browser module 247, ●Calendar module 248, ● A widget module 249, which in some embodiments includes one or more of the following: a weather widget 249-1, a stock widget 249-2, a calculator widget 249-3, an alarm clock widget 249-4, a dictionary widget 249-5, and other widgets obtained by the user, as well as a user-created widget 249-6. ●Widget creator module 250 for creating user-created widget 249-6, ● Search module 251, ●Video and music player module 252, which integrates a video player module and a music player module. ●Memo Module 253, ●Map module 254 and / or, ● Online video module 255.

[0079] Other applications 236 stored in memory 202 include other word processing applications, other image editing applications, drawing applications, presentation applications, Java®-enabled applications, encryption, digital rights management, speech recognition, and speech duplication.

[0080] The contact module 237 is used in conjunction with the touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234 to manage an address book or contact list (for example, stored in the application internal state 292 of the contact module 237 in memory 202 or memory 470), which includes adding names(s) to the address book, removing names(s) from the address book, associating telephone numbers(s) to names, email addresses(s) to names, physical addresses(s) to names, or other information, associating images to names, categorizing and sorting names, and providing telephone numbers or email addresses to initiate and / or facilitate communication via telephone 238, video conferencing module 239, email 240, or IM241.

[0081] The telephone module 238 works in conjunction with the RF circuit 208, audio circuit 210, speaker 211, microphone 213, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234 to input strings corresponding to telephone numbers, access one or more telephone numbers in the contact module 237, modify the entered telephone number, dial individual telephone numbers, conduct conversations, and disconnect or hang up when the conversation is complete. As described above, wireless communication uses one of several communication standards, protocols, and technologies.

[0082] The video conferencing module 239 works in conjunction with an RF circuit 208, an audio circuit 210, a speaker 211, a microphone 213, a touchscreen 212, a display controller 256, an optical sensor 264, an optical sensor controller 258, a contact / motion module 230, a graphics module 232, a text input module 234, a contact module 237, and a telephone module 238 to include executable commands for starting, running, and ending video conferences between the user and one or more other participants in accordance with the user's commands.

[0083] The email client module 240, in conjunction with the RF circuit 208, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, includes executable commands for creating, sending, receiving, and managing emails in response to user commands. The email client module 240, in conjunction with the image management module 244, makes it extremely easy to create and send emails containing still or video images captured by the camera module 243.

[0084] The instant messaging module 241, in conjunction with the RF circuit 208, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, includes executable instructions for inputting strings corresponding to instant messages, modifying previously entered characters, sending individual instant messages (e.g., using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for telephone-based instant messaging, or XMPP, SIMPLE, or IMPS for internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, sent and / or received instant messages include graphics, photographs, audio files, video files, and / or other attachments supported by MMS and / or Enhanced Messaging Services (EMS). As used herein, “instant messaging” refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0085] The training support module 242 works in conjunction with the RF circuit 208, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, GPS module 235, map module 254, and music player module to create training (e.g., with time, distance, and / or calorie burn goals), communicate with training sensors (sports devices), receive training sensor data, calibrate sensors used to monitor training, select and play music for training, and includes executable commands for displaying, storing, and transmitting training data.

[0086] The camera module 243, in cooperation with the touchscreen 212, display controller 256, optical sensor(s) 264, optical sensor controller 258, contact / motion module 230, graphics module 232, and image management module 244, includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 202, modifying the characteristics of the still images or videos, or deleting still images or videos from memory 202.

[0087] The image management module 244, in conjunction with the touchscreen 212, display controller 256, touch / motion module 230, graphics module 232, text input module 234, and camera module 243, includes executable commands for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing still and / or video images.

[0088] The browser module 247, in conjunction with the RF circuit 208, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, includes executable commands for browsing the internet in accordance with user commands, including searching, linking, receiving, and displaying web pages or parts thereof, as well as attachments and other files linked to web pages.

[0089] The calendar module 248 works in conjunction with the RF circuit 208, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, email client module 240, and browser module 247 to include executable commands for creating, displaying, modifying, and storing a calendar and data associated with the calendar (e.g., calendar entries, to-do lists, etc.) according to user commands.

[0090] The widget module 249 is a mini-application that can be downloaded and used by the user (e.g., weather widget 249-1, stock widget 249-2, calculator widget 249-3, alarm clock widget 249-4, and dictionary widget 249-5) or created by the user (e.g., user-created widget 249-6), in conjunction with the RF circuit 208, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, and browser module 247. In some embodiments, the widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! Widgets®).

[0091] The widget creation module 250 works in conjunction with the RF circuit 208, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, and browser module 247 to be used by the user to create widgets (for example, to convert user-specified portions of a web page into widgets).

[0092] The search module 251 works in conjunction with the touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234 to include executable instructions for searching for text, music, sounds, images, videos, and / or other files in memory 202 that match one or more search criteria (e.g., one or more user-specified search terms) according to user commands.

[0093] The video and music player module 252 works in conjunction with the touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, audio circuit 210, speaker 211, RF circuit 208, and browser module 247 to include executable instructions that allow the user to download and play recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, as well as executable instructions for displaying, presenting, or otherwise playing video (for example, on the touchscreen 212 or on an external display connected via the external port 224). In some embodiments, the device 200 optionally includes the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0094] The memo module 253, in conjunction with the touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, includes executable commands for creating and managing memos, to-do lists, etc., according to user commands.

[0095] The map module 254 works in conjunction with the RF circuit 208, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, GPS module 235, and browser module 247 to receive, display, modify, and store maps and map-related data (e.g., driving instructions, data about stores and other places of interest in a specific location or nearby, and other location-based data) according to user commands.

[0096] The online video module 255, in conjunction with the touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, audio circuit 210, speaker 211, RF circuit 208, text input module 234, email client module 240, and browser module 247, includes instructions that enable a user to access a specific online video, browse a specific online video, receive it (e.g., by streaming and / or downloading), play it (e.g., on the touchscreen or on an external display connected via external port 224), send emails containing links to a specific online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, an instant messaging module 241 is used instead of the email client module 240 to send links to a specific online video. Further descriptions of online video applications can be found in U.S. Provisional Patent Application No. 60 / 936,562, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed June 20, 2007, and U.S. Patent Application No. 11 / 968,067, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed December 31, 2007, the entire contents of which are incorporated herein by reference.

[0097] Each of the modules and applications identified above corresponds to a set of executable instructions for performing one or more of the functions described above and the methods described in this application (e.g., the computer-implemented methods and other information processing methods described herein). Since these modules (e.g., instruction sets) do not need to be implemented as separate software programs, procedures, or modules, various subsets of these modules can be combined or otherwise reconfigured in various embodiments. For example, a video player module can be combined with a music player module to form a single module (e.g., video and music player module 252, Figure 2A). In some embodiments, memory 202 stores a subset of the modules and data structures identified above. Furthermore, memory 202 stores additional modules and data structures not described above.

[0098] In some embodiments, device 200 is a device in which the operation of a default set of functions in that device is performed solely through a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for the operation of device 200, the number of physical input control devices (push buttons, dials, etc.) on device 200 is reduced.

[0099] A default set of functions performed solely through the touchscreen and / or touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 200 from any user interface displayed on the device 200 to the main menu, home menu, or root menu. In such embodiments, a “menu button” is implemented using the touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0100] Figure 2B is a block diagram showing exemplary components for event processing according to some embodiments. In some embodiments, memory 202 (Figure 2A) or memory 470 (Figure 4) includes an event sorter 270 (e.g., within the operating system 226) and individual applications 236-1 (e.g., any of the aforementioned applications 237-251, 255, 480-490).

[0101] The event sorter 270 receives event information and determines which application 236-1 and which application view 291 of application 236-1 should receive the event information. The event sorter 270 includes an event monitor 271 and an event dispatcher module 274. In some embodiments, application 236-1 includes an application internal state 292 that indicates the current application view(s) displayed on the touch-sensitive display 212 when the application is active or running. In some embodiments, a device / global internal state 257 is used by the event sorter 270 to determine which application(s) are currently active, and the application internal state 292 is used by the event sorter 270 to determine which application view(s) should receive the event information.

[0102] In some embodiments, the application internal state 292 includes additional information such as resume information to be used when the application 236-1 resumes execution, user interface state information indicating information that is displayed or ready to be displayed by the application 236-1, a state queue to allow the user to return to the previous state or view of the application 236-1, and a redo / undo queue for previous actions performed by the user.

[0103] The event monitor 271 receives event information from the peripheral interface 218. The event information includes information about sub-events (for example, user touch as part of a multi-touch gesture on the touch-sensitive display 212). The peripheral interface 218 transmits information received from the I / O subsystem 206, or from sensors such as the proximity sensor 266, one or more accelerometers 268, and / or the microphone 213 (via the audio circuit 210). The information received by the peripheral interface 218 from the I / O subsystem 206 includes information from the touch-sensitive display 212 or the touch-sensitive surface.

[0104] In some embodiments, the event monitor 271 sends requests to the peripheral interface 218 at predetermined intervals. In response, the peripheral interface 218 transmits event information. In other embodiments, the peripheral interface 218 transmits event information only when there is a significant event (e.g., reception of an input exceeding a predetermined noise threshold and / or an input exceeding a predetermined duration).

[0105] In some embodiments, the event sorter 270 also includes a hit view determination module 272 and / or an active event recognition determination module 273.

[0106] The hit view determination module 272 provides software procedures for determining where a sub-event occurred within one or more views when the touch-sensitive display 212 is displaying one or more views. A view consists of controls and other elements that the user can see on the display.

[0107] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch gestures are performed. The application view (of an individual application) in which a touch is detected corresponds to the program level within the application's program hierarchy or view hierarchy. For example, the lowest-level view in which a touch is detected is called the hit view, and the set of events recognized as a valid input is determined at least in part based on the hit view of the initial touch that initiates the touch gesture.

[0108] The hit view determination module 272 receives information related to sub-events of touch gestures. When an application has multiple views arranged in a hierarchy, the hit view determination module 272 identifies the hit view as the lowest-level view in the hierarchy from which the sub-events should be processed. In most situations, the hit view is the lowest-level view from which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or a potential event) occurs. Once a hit view is identified by the hit view determination module 272, the hit view typically receives all sub-events related to the same touch or input source that identified it as the hit view.

[0109] The active event recognition determination module 273 determines which view(s) in the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognition determination module 273 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognition determination module 273 determines that all views, including the physical location of the sub-event, are actively involved views, and therefore all actively involved views should receive a particular sequence of sub-events. In other embodiments, even if the touch sub-event is entirely confined to an area associated with one particular view, higher-level views in the hierarchy remain actively involved views.

[0110] The event dispatcher module 274 dispatches event information to an event recognition unit (e.g., an event recognition unit 280). In embodiments including an active event recognition unit determination module 273, the event dispatcher module 274 distributes the event information to the event recognition unit determined by the active event recognition unit determination module 273. In some embodiments, the event dispatcher module 274 stores event information retrieved by individual event receiving units 282 in an event queue.

[0111] In some embodiments, the operating system 226 includes an event sorter 270. Alternatively, application 236-1 includes an event sorter 270. In yet other embodiments, the event sorter 270 is a standalone module or part of another module stored in memory 202, such as a contact / motion module 230.

[0112] In some embodiments, application 236-1 includes a plurality of event handlers 290 and one or more application views 291, each of which includes instructions for handling touch events occurring within a separate view of the application's user interface. Each application view 291 of application 236-1 includes one or more event recognition units 280. Typically, a separate application view 291 includes a plurality of event recognition units 280. In other embodiments, one or more of the event recognition units 280 are part of a separate module, such as a user interface kit (not shown) or a higher-level object from which application 236-1 inherits methods and other characteristics. In some embodiments, a separate event handler 290 includes one or more of the following: a data updater 276, an object updater 277, a GUI updater 278, and / or event data 279 received from an event sorter 270. The event handler 290 utilizes or calls the data updater 276, object updater 277, or GUI updater 278 to update the application's internal state 292. Alternatively, one or more application views 291 include one or more respective event handlers 290. Also, in some embodiments, one or more of the data updater 276, object updater 277, and GUI updater 278 are included in a separate application view 291.

[0113] Individual event recognition units 280 receive event information (e.g., event data 279) from the event sorter 270 and identify events from the event information. The event recognition unit 280 includes an event receiving unit 282 and an event comparison unit 284. In some embodiments, the event recognition unit 280 also includes at least a subset of metadata 283 and event distribution commands 288 (including sub-event distribution commands).

[0114] The event receiver 282 receives event information from the event sorter 270. The event information includes information about sub-events, such as touches or the movement of touches. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. If the sub-event relates to the movement of a touch, the event information also includes the velocity and direction of the sub-event. In some embodiments, an event includes a rotation of the device from one orientation to another (for example, from portrait orientation to landscape orientation, or vice versa), and the event information includes corresponding information about the device's current orientation (also called the device's orientation).

[0115] The event comparison unit 284 compares the event information with a default event or sub-event definition and, based on the comparison, determines the event or sub-event, or determines or updates the state of the event or sub-event. In some embodiments, the event comparison unit 284 includes an event definition 286. The event definition 286 includes definitions of events (e.g., a default sequence of sub-events), such as event 1 (287-1) and event 2 (287-2). In some embodiments, sub-events within event (287) include, for example, touch start, touch end, touch movement, touch cancellation, and multiple touches. In one embodiment, the definition for event 1 (287-1) is a double tap on a displayed object. A double tap includes, for example, a first touch on the displayed object for a predetermined stage (touch start), a first lift-off for the predetermined stage (touch end), a second touch on the displayed object for the predetermined stage (touch start), and a second lift-off for the predetermined stage (touch end). In another embodiment, the definition of event 2(287-2) is a drag on a displayed object. A drag includes, for example, a touch (or contact) on the displayed object to a predetermined stage, movement of the touch across the touch-sensitive display 212, and lift-off of the touch (end of touch). In some embodiments, the event also includes information about one or more associated event handlers 290.

[0116] In some embodiments, the event definition 287 includes event definitions for individual user interface objects. In some embodiments, the event comparison unit 284 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view where three user interface objects are displayed on the touch-sensitive display 212, when a touch is detected on the touch-sensitive display 212, the event comparison unit 284 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with an individual event handler 290, the event comparison unit uses the results of the hit test to determine which event handler 290 should be activated. For example, the event comparison unit 284 selects the sub-event and the event handler associated with the object that triggers the hit test.

[0117] In some embodiments, the definition of an individual event 287 also includes a delay action that delays the delivery of event information until it is determined whether the sequence of sub-events corresponds to the event type of the event recognition unit.

[0118] If an individual event recognition unit 280 determines that a series of sub-events does not match any of the events in the event definition 286, the individual event recognition unit 280 enters an event impossible, event failed, or event terminated state, and thereafter ignores subsequent sub-events of the touch gesture. In this situation, if there are other event recognition units that remain active for the hit view, those event recognition units continue to track and process the sub-events of the ongoing touch gesture.

[0119] In some embodiments, an individual event recognition unit 280 includes metadata 283 having configurable properties, flags, and / or lists that indicate to the actively involved event recognition unit how the event distribution system should perform sub-event distribution. In some embodiments, the metadata 283 includes configurable properties, flags, and / or lists that indicate how the event recognition units interact with each other, or how they can interact with each other. In some embodiments, the metadata 283 includes configurable properties, flags, and / or lists that indicate whether sub-events are distributed to various levels in the view hierarchy or program hierarchy.

[0120] In some embodiments, an individual event recognition unit 280 activates an event handler 290 associated with an event when one or more specific sub-events of an event are recognized. In some embodiments, the individual event recognition unit 280 delivers event information associated with the event to the event handler 290. Activating the event handler 290 is separate from sending (and delaying sending) sub-events to individual hit views. In some embodiments, the event recognition unit 280 sets a flag associated with the recognized event, and the event handler 290 associated with that flag catches the flag and executes a default process.

[0121] In some embodiments, the event distribution command 288 includes a sub-event distribution command that distributes event information about a sub-event without activating an event handler. Instead, the sub-event distribution command distributes event information to an event handler associated with a set of sub-events, or to a view that is actively involved. The event handler associated with the set of sub-events or the view that is actively involved receives the event information and executes a predetermined process.

[0122] In some embodiments, the data updater 276 creates and updates data used in application 236-1. For example, the data updater 276 updates telephone numbers used in contact module 237 or stores video files used in video player module. In some embodiments, the object updater 277 creates and updates objects used in application 236-1. For example, the object updater 277 creates new user interface objects or updates the positions of user interface objects. The GUI updater 278 updates the GUI. For example, the GUI updater 278 prepares display information and sends it to graphics module 232 for display on touch-sensitive display.

[0123] In some embodiments, the event handler(s) 290 includes, or has access to, a data updater 276, an object updater 277, and a GUI updater 278. In some embodiments, the data updater 276, the object updater 277, and the GUI updater 278 are contained in a single module of a separate application 236-1 or application view 291. In other embodiments, they are contained in two or more software modules.

[0124] The above description of handling user touch events on a touch-sensitive display also applies to other forms of user input for operating the multifunction device 200 using input devices, but it should be understood that not all of them originate on the touchscreen. For example, mouse movements and mouse button presses, touch movements such as taps, drags, and scrolls on a touchpad, pen stylus input, device movement, verbal commands, detected eye movements, biometric input, and / or any combination thereof may be optionally used as inputs corresponding to sub-events that define the events to be recognized.

[0125] Figure 3 shows a portable multifunction device 200 having a touchscreen 212 according to some embodiments. The touchscreen optionally displays one or more graphics within a user interface (UI) 300. In this embodiment, and in other embodiments described below, the user can select one or more of the graphics by making gestures on the graphics using, for example, one or more fingers 302 (not shown in the figure to an exact scale) or one or more styluses 303 (not shown in the figure to an exact scale). In some embodiments, the selection of one or more graphics occurs when the user interrupts contact with that one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, right to left, upward and / or downward) and / or rolling (from right to left, left to right, upward and / or downward) with a finger in contact with the device 200. In some implementations or situations, an unintended contact with a graphic does not necessarily select that graphic. For example, when the gesture corresponding to selection is a tap, a swipe gesture sweeping over an application icon will not, by choice, select the corresponding application.

[0126] Device 200 also includes one or more physical buttons, such as a "Home" or menu button 304. As previously stated, the menu button 304 is used to navigate to any application 236 within the set of applications running on device 200. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touchscreen 212.

[0127] In one embodiment, the device 200 includes a touchscreen 212, a menu button 304, a push button 306 for turning the device on / off and locking the device, one or more volume control buttons 308, a subscriber identification module (SIM) card slot 310, a headset jack 312, and an external docking / charging port 224. The push button 306 is optionally used to turn the device on / off by pressing and holding the button for a predetermined period of time, to lock the device by pressing and releasing the button before a predetermined period of time has elapsed, and / or to unlock the device or initiate an unlocking process. In an alternative embodiment, the device 200 also accepts verbal input via a microphone 213 to activate or deactivate certain functions. Device 200 also optionally includes one or more contact intensity sensors 265 for detecting the intensity of contact on the touchscreen 212, and / or one or more tactile output generators 267 for generating tactile output to the user of device 200.

[0128] Figure 4 is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface, according to some embodiments. Device 400 does not need to be portable. In some embodiments, device 400 is a laptop computer, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a children's learning toy), game system, or control device (e.g., a home or commercial controller). Device 400 typically includes one or more processing units (CPUs) 410, one or more network or other communication interfaces 460, memory 470, and one or more communication buses 420 for interconnecting these components. The communication buses 420 optionally include circuitry (sometimes called a chipset) that interconnects and controls communication between system components. Device 400 includes an input / output (I / O) interface 430, which includes a display 440, and the display 440 is typically a touchscreen display. The I / O interface 430 also optionally includes a keyboard and / or mouse (or other pointing device) 450, as well as a touchpad 455, a tactile output generator 457 (similar to the tactile output generator 267(single or multiple) described above with reference to Figure 2A) for generating tactile output on device 400, and a sensor 459 (e.g., an optical sensor, an accelerometer, a proximity sensor, a touch sensor, and / or a contact intensity sensor (similar to the contact intensity sensor(single or multiple) 265 described above with reference to Figure 2A). The memory 470 includes high-speed random-access memory such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices, and optionally includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. The memory 470 optionally includes one or more storage devices located remotely from the CPU(single or multiple) 410.In some embodiments, memory 470 stores programs, modules, and data structures, or subsets thereof, that are similar to those stored in memory 202 of the portable multifunction device 200 (Figure 2A). Furthermore, memory 470 optionally stores additional programs, modules, and data structures that are not present in memory 202 of the portable multifunction device 200. For example, memory 470 of device 400 optionally stores a drawing module 480, a presentation module 482, a word processing module 484, a website creation module 486, a disk authoring module 488, and / or a spreadsheet module 490, while memory 202 of the portable multifunction device 200 (Figure 2A) optionally does not store these modules.

[0129] Each element of Figure 4 identified above is stored in any one or more of the aforementioned memory devices in some embodiments. Each module identified above corresponds to an instruction set for performing the function described above. The modules or programs (e.g., instruction sets) identified above do not need to be implemented as separate software programs, procedures, or modules; therefore, various subsets of these modules are combined or otherwise reconfigured in various embodiments. In some embodiments, memory 470 stores a subset of the modules and data structures identified above. Furthermore, memory 470 stores additional modules and data structures not described above.

[0130] Here, we focus on an embodiment of a user interface that can be implemented, for example, in a portable multi-functional device 200.

[0131] Figure 5A shows an exemplary user interface for an application menu on a portable multifunction device 200, according to one embodiment. A similar user interface is implemented on device 400. In some embodiments, the user interface 500 includes the following elements, or subsets or supersets thereof:

[0132] Signal strength indicators (single or multiple) for wireless communication (single or multiple) such as cellular and Wi-Fi signals 502, ●Time 504, ●Bluetooth indicator 505, ●Battery status indicator 506, ●Tray 508 containing icons for frequently used applications, as shown below. ○Optionally including an indicator 514 for the number of missed calls or voicemail messages, an icon 516 labeled "Telephone" for the telephone module 238, ○Optionally includes an indicator 510 for the number of unread emails, an icon 518 labeled "Mail" for the email client module 240. ○ Icon 520 for browser module 247, labeled "Browser", and ○ Icon 522 for the video and music player module 252, also known as the iPod (trademark of Apple Inc.) module 252, labeled as "iPod," and ● Icons for other applications, such as the following: ○ Icon 524 for IM module 241, labeled "Message" ○ Icon 526 for calendar module 248, labeled "Calendar" ○ Icon 528 for image management module 244, labeled "Photo" ○ Icon 530 for camera module 243, labeled "Camera" ○ Icon 532 for online video module 255, labeled "online video" ○ Icon 534 for stock widget 249-2, labeled "Stocks" ○ Icon 536 for map module 254, labeled "Map" ○ Icon 538 for weather widget 249-1, labeled "Weather" ○ Icon 540 for alarm clock widget 249-4, labeled "Clock" ○ Icon 542 for training support module 242, labeled "Training Support" ○ Icon 544 for memo module 253, labeled as "Memo", and ○ An icon 546 labeled "Settings" for a settings application or module, providing access to the settings of device 200 and its various applications 236.

[0133] Note that the icon labels shown in Figure 5A are for illustrative purposes only. For example, the icon 522 for the video and music player module 252 may optionally be labeled "Music" or "Music Player." Other labels may optionally be used for various application icons. In some embodiments, the label for an individual application icon may include the name of the application to which that individual application icon corresponds. In some embodiments, the label for a particular application icon may differ from the name of the application to which that particular application icon corresponds.

[0134] Figure 5B shows an exemplary user interface on a device (e.g., device 400 in Figure 4) having a touch-sensitive surface 551 (e.g., tablet or touchpad 455 in Figure 4) separate from the display 550 (e.g., touchscreen display 212). Device 400 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 459) for detecting the intensity of contact on the touch-sensitive surface 551, and / or one or more tactile output generators 457 for generating tactile output for the user of device 400.

[0135] Some of the following embodiments are described with reference to input on a touchscreen display 212 (when the touch-sensing surface and the display are combined), but in some embodiments, the device detects input on a touch-sensing surface separate from the display, as shown in Figure 5B. In some embodiments, this touch-sensing surface (e.g., 551 in Figure 5B) has a principal axis (e.g., 552 in Figure 5B) corresponding to a principal axis (e.g., 553 in Figure 5B) on the display (e.g., 550). According to these embodiments, the device detects contact with the touch-sensing surface 551 at positions corresponding to each of the positions on the display (e.g., 560 and 562 in Figure 5B) (e.g., in Figure 5B, 560 corresponds to 568 and 562 corresponds to 570). In this method, if the touch-sensitive surface is separate from the display, user input detected by the device on the touch-sensitive surface (e.g., 551 in Figure 5B) (e.g., touches 560 and 562, and their movement) is used by the device to operate the user interface on the display of the multifunction device (e.g., 550 in Figure 5B). It should be understood that a similar method may be optionally used for other user interfaces described herein.

[0136] In addition, while the following examples are given primarily in relation to finger input (e.g., finger touch, finger tap gesture, finger swipe gesture), it should be understood that in some embodiments, one or more of the finger inputs may be replaced by input from another input device (e.g., mouse input or stylus input). For example, a swipe gesture may optionally be replaced with a mouse click (e.g., instead of touch) followed by cursor movement along the swipe path (e.g., instead of touch movement). As another example, a tap gesture may optionally be replaced with a mouse click (e.g., instead of detecting touch and then ceasing touch detection) while the cursor is located over the tap gesture. Similarly, it should be understood that when multiple user inputs are detected simultaneously, multiple computer mice may optionally be used simultaneously, or a mouse and finger touch may optionally be used simultaneously.

[0137] Figure 6A shows an exemplary personal electronic device 600. Device 600 includes a body 602. In some embodiments, device 600 includes some or all of the features described in relation to devices 200 and 400 (e.g., Figures 2A to 4). In some embodiments, device 600 has a touch-sensitive display screen 604, hereafter referred to as touchscreen 604. Alternatively, or in addition to touchscreen 604, device 600 has a display and a touch-sensitive surface. Together with devices 200 and 400, in some embodiments, touchscreen 604 (or touch-sensitive surface) has one or more intensity sensors for detecting the intensity of the applied contact (e.g., touch). One or more intensity sensors on touchscreen 604 (or touch-sensitive surface) provide output data representing the intensity of the touch. The user interface of device 600 responds to touches based on the intensity of the touch, meaning that touches of different intensity can invoke different user interface behaviors on device 600.

[0138] Techniques for detecting and processing touch intensity are found, for example, in the related applications, International Patent Application No. PCT / US2013 / 040061, filed on 8 May 2013, “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” and International Patent Application No. PCT / US2013 / 069483, filed on 11 November 2013, “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” each of which is incorporated herein by reference in whole.

[0139] In some embodiments, the device 600 has one or more input mechanisms 606 and 608. The input mechanisms 606 and 608 are physical, if included. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, the device 600 has one or more attachment mechanisms. Such attachment mechanisms, if included, can allow the device 600 to be attached to, for example, hats, eyeglasses, earrings, necklaces, shirts, jackets, bracelets, watch bands, chains, trousers, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow the user to wear the device 600.

[0140] Figure 6B shows an exemplary personal electronic device 600. In some embodiments, the device 600 includes some or all of the components described in relation to Figures 2A, 2B, and 4. The device 600 has a bus 612 that operably connects an I / O section 614 to one or more computer processors 616 and memory 618. The I / O section 614 is connected to a display 604 which may have a touch-sensitive component 622 and optionally a touch intensity-sensitive component 624. In addition, the I / O section 614 is connected to a communication unit 630 for receiving application and operating system data using Wi-Fi, Bluetooth, near-field communication (NFC), cellular, and / or other wireless communication technologies. The device 600 includes input mechanisms 606 and / or 608. The input mechanism 606 is, for example, a rotatable input device or a pressable and rotatable input device. The input mechanism 608 is, in some embodiments, a button.

[0141] The input mechanism 608 is a microphone in some embodiments. The personal electronic device 600 includes various sensors such as a GPS sensor 632, an accelerometer 634, a direction sensor 640 (e.g., a compass), a gyroscope 636, a motion sensor 638, and / or a combination thereof, all of which are operably connected to the I / O unit 614.

[0142] The memory 618 of the personal electronic device 600 is a non-temporary computer-readable storage medium for storing computer executable instructions, which, when executed by, for example, one or more computer processors 616, cause the computer processors to perform the following techniques and processes. These computer executable instructions are also stored and / or transmitted in any non-temporary computer-readable storage medium for use by, or in connection with, instruction execution systems, devices, or other systems capable of fetching instructions from computer-based systems, systems including processors, or other systems capable of executing those instructions. The personal electronic device 600 is not limited to the components and configurations shown in Figure 6B and may include other or additional components in multiple configurations.

[0143] As used herein, the term "affordance" refers to, for example, user-interactive graphical user interface objects displayed on the display screens of devices 200, 400, and / or 600 (Figures 2A, 4, and 6A-6B). For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) each constitute affordances.

[0144] As used herein, the term “focus selector” refers to an input element that indicates the current portion of the user interface with which the user is interacting. In some implementations, including a cursor or other position marker, the cursor acts as a “focus selector” so that when input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 455 in Figure 4, or touch-sensitive surface 551 in Figure 5B) while the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), that particular user interface element is adjusted according to the detected input. In some implementations, including a touchscreen display that enables direct interaction with user interface elements on the touchscreen display (e.g., touch-sensitive display system 212 in Figure 2A, or touchscreen 212 in Figure 5A), contact detected on the touchscreen acts as a “focus selector” so that when input (e.g., a press input by touch) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, that particular user interface element is adjusted according to the detected input. In some implementations, focus is moved from one area of ​​the user interface to another without corresponding cursor movement or touch movement on the touchscreen display (for example, by using the tab key or arrow keys to move focus from one button to another), and in these implementations, the focus selector moves in accordance with the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is generally a user interface element (or touch on the touchscreen display) controlled by the user to communicate the user's intended interaction with the user interface (for example, by pointing the device to the user interface element that the user intends to interact with).For example, the position of a focus selector (e.g., cursor, touch, or selection box) over an individual button while pressure input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen) indicates that the user intends to activate that individual button (rather than other user interface elements displayed on the device's display).

[0145] As used herein and in the claims, the term “characteristic intensity” of a contact refers to the characteristics of that contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a set of intensity samples collected over a predetermined time (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) for a predetermined number of intensity samples, i.e., after a predetermined event (e.g., after detecting contact, before detecting lift-off of contact, before or after detecting the start of movement of contact, before detecting the end of contact, before or after detecting an increase in contact intensity, and / or before or after detecting a decrease in contact intensity). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the contact intensity, the median value of the contact intensity, the average value of the contact intensity, the top 10 percentile value of the contact intensity, half the maximum value of the contact intensity, the 90 percent value of the maximum value of the contact intensity, and so on. In some embodiments, the duration of contact is used when determining characteristic intensity (for example, when characteristic intensity is the average intensity of contact over time). In some embodiments, characteristic intensity is compared to a set of one or more intensity thresholds to determine whether an action has been performed by the user. For example, the set of one or more intensity thresholds includes a first intensity threshold and a second intensity threshold. In this embodiment, contact with a characteristic intensity not exceeding the first threshold results in a first action, contact with a characteristic intensity above the first intensity threshold but not exceeding the second intensity threshold results in a second action, and contact with a characteristic intensity above the second threshold results in a third action. In some embodiments, the comparison of characteristic intensity to one or more thresholds is not used to determine whether to perform the first or second action, but rather to determine whether to perform one or more actions (for example, whether to perform individual actions or to refrain from performing individual actions).

[0146] In some embodiments, a portion of the gesture is identified for the purpose of determining characteristic intensity. For example, a touch-sensitive surface receives a continuous swipe contact transitioning from a start position to an end position, with the contact intensity increasing at the end position. In this example, the characteristic intensity of the contact at the end position is based only on a portion of the continuous swipe contact (e.g., only a portion of the swipe contact at the end position), rather than the entire swipe contact. In some embodiments, a smoothing algorithm is applied to the intensity of the swipe contact before determining the characteristic intensity of the contact. For example, the smoothing algorithm may optionally include one or more of the following: an unweighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some situations, these smoothing algorithms eliminate fine spikes or dips in the swipe contact intensity for the purpose of determining characteristic intensity.

[0147] The intensity of contact on a touch-sensitive surface is characterized with respect to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to the intensity at which the device performs an action typically associated with clicking a physical mouse button or trackpad. In some embodiments, the deep press intensity threshold corresponds to the intensity at which the device performs an action different from the action typically associated with clicking a physical mouse button or trackpad. In some embodiments, when a contact with a characteristic intensity below the light press intensity threshold (for example, above a nominal contact detection intensity threshold below which contact is no longer detected) is detected, the device moves the focus selector in accordance with the movement of the contact on the touch-sensitive surface without performing an action associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise specified, these intensity thresholds are consistent across different sets of user interface values.

[0148] An increase in the characteristic intensity of contact from an intensity below a light pressure intensity threshold to an intensity between the light and deep pressure intensity thresholds is sometimes referred to as a "light pressure" input. An increase in the characteristic intensity of contact from an intensity below a deep pressure intensity threshold to an intensity above a deep pressure intensity threshold is sometimes referred to as a "deep pressure" input. An increase in the characteristic intensity of contact from an intensity below a contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light pressure intensity threshold is sometimes referred to as detection of contact on the touch surface. A decrease in the characteristic intensity of contact from an intensity above a contact detection intensity threshold to an intensity below a contact detection intensity threshold is sometimes referred to as detection of contact lift-off from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0149] In some embodiments described herein, one or more actions are performed in response to the detection of a gesture including an individual press input, or in response to the detection of an individual press input performed by an individual contact (or multiple contacts), the individual press input being detected at least in part on the detection of an increase in the intensity of the contact (or multiple contacts) above a press input intensity threshold. In some embodiments, the individual action is performed in response to the detection of an increase in the intensity of the individual contact above a press input intensity threshold (e.g., a "downstroke" of the individual press input). In some embodiments, the press input includes an increase in the intensity of the individual contact above a press input intensity threshold, followed by a decrease in the intensity of the contact below the press input intensity threshold, and the individual action is performed in response to the detection of a subsequent decrease in the intensity of the individual contact below the press input threshold (e.g., an "upstroke" of the individual press input).

[0150] In some embodiments, the device employs intensity hysteresis to avoid accidental inputs, which may be referred to as “jitter,” and the device defines or selects a hysteresis intensity threshold that has a predetermined relationship with a press input intensity threshold (for example, the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable percentage of the press input intensity threshold). Thus, in some embodiments, the press input includes an increase in the intensity of an individual contact above the press input intensity threshold, followed by a decrease in the intensity of the contact below the hysteresis intensity threshold corresponding to the press input intensity threshold, and the individual action is performed in response to the detection of a subsequent decrease in the intensity of an individual contact below the hysteresis intensity threshold (for example, an “upstroke” of the individual press input). Similarly, in some embodiments, a press input is detected only when the device detects an increase in contact intensity from an intensity below a hysteresis intensity threshold to an intensity above a press input intensity threshold, and optionally a decrease in contact intensity to an intensity below the hysteresis intensity, and individual actions are performed in response to the detection of a press input (e.g., an increase in contact intensity or a decrease in contact intensity, depending on the situation).

[0151] For the sake of clarity, the description of an action performed in response to a press input associated with a press input intensity threshold, or in response to a gesture including a press input, is optionally triggered in response to the detection of any of the following: an increase in contact intensity above the press input intensity threshold, an increase in contact intensity from below the hysteresis intensity threshold to above the press input intensity threshold, a decrease in contact intensity below the press input intensity threshold, and / or a decrease in contact intensity below the hysteresis intensity threshold corresponding to the press input intensity threshold. Furthermore, in an example where an action is described to be performed in response to the detection of a decrease in contact intensity below the press input intensity threshold, the action is optionally performed in response to the detection of a decrease in contact intensity below a hysteresis intensity threshold corresponding to the press input intensity threshold and lower than that threshold. 3. Digital Assistant System

[0152] Figure 7A shows block diagrams of the digital assistant system 700 according to various embodiments. In some embodiments, the digital assistant system 700 is implemented on a standalone computer system. In some embodiments, the digital assistant system 700 is distributed across multiple computers. In some embodiments, some of the modules and functions of the digital assistant are divided into a server portion and a client portion, the client portion residing on one or more user devices (e.g., devices 104, 122, 200, 400, or 600), as shown in Figure 1, for example, and communicating with the server portion (e.g., server system 108) through one or more networks. In some embodiments, the digital assistant system 700 is an implementation of the server system 108 (and / or DA server 106) shown in Figure 1. It should be noted that the digital assistant system 700 is only one embodiment of the digital assistant system, and the digital assistant system 700 may have more or fewer components than shown, may combine two or more components, or may have different configurations or arrangements of those components. The various components shown in Figure 7A are implemented as hardware including one or more signal processing circuits and / or application-specific integrated circuits, software instructions executed by one or more processors, firmware, or a combination thereof.

[0153] The digital assistant system 700 includes a memory 702, one or more processors 704, an input / output (I / O) interface 706, and a network communication interface 708. These components can communicate with each other via one or more communication buses or signal lines 710.

[0154] In some embodiments, the memory 702 includes a non-transient computer-readable medium, such as a high-speed random-access memory and / or a non-volatile computer-readable storage medium (e.g., one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices).

[0155] In some embodiments, the I / O interface 706 connects the input / output devices 716 of the digital assistant system 700, such as a display, keyboard, touchscreen, and microphone, to the user interface module 722. The I / O interface 706 works in conjunction with the user interface module 722 to receive and process user input (e.g., voice input, keyboard input, touch input, etc.). In some embodiments, for example, when the digital assistant is implemented on a standalone user device, the digital assistant system 700 includes any of the components and I / O communication interfaces described in relation to devices 200, 400, or 600 in Figures 2A, 4, and 6A-6B, respectively. In some embodiments, the digital assistant system 700 can represent the server portion of the digital assistant implementation and can interact with the user through a client portion residing on a user device (e.g., devices 104, 200, 400, or 600).

[0156] In some embodiments, the network communication interface 708 includes one or more wired communication ports 712 and / or a wireless transceiver circuit 714. The one or more wired communication ports transmit and receive communication signals via one or more wired interfaces, such as Ethernet, Universal Serial Bus (USB), FireWire, etc. The wireless circuit 714 transmits and receives RF signals and / or optical signals to and from the communication network and other communication devices. Wireless communication uses one of several communication standards, communication protocols, and communication technologies, such as GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol. The network communication interface 708 enables communication between the digital assistant system 700 and networks such as the Internet, intranets, and / or wireless networks such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs), and between the digital assistant system 700 and other devices.

[0157] In some embodiments, memory 702, or the computer-readable storage medium of memory 702, stores programs, modules, instructions, and data structures, including all or a subset of the operating system 718, communication module 720, user interface module 722, one or more applications 724, and digital assistant module 726. In particular, memory 702, or the computer-readable storage medium of memory 702, stores instructions for executing the processes described below. One or more processors 704 execute these programs, modules, and instructions, and read from / write to data structures.

[0158] Operating System 718 (e.g., embedded operating systems such as Darwin, RTXC, LINUX, UNIX®, iOS, OS X, WINDOWS®, or VxWorks) includes various software components and / or drivers for controlling and managing common system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware, firmware, and software components.

[0159] The communication module 720 facilitates communication between the digital assistant system 700 and other devices via the network communication interface 708. For example, the communication module 720 communicates with the RF circuit 208 of electronic devices such as devices 200, 400, and 600 shown in Figures 2A, 4, 6A, and 6B, respectively. The communication module 720 also includes various components for processing data received by the wireless circuit 714 and / or the wired communication port 712.

[0160] The user interface module 722 receives commands and / or input from the user via the I / O interface 706 (e.g., from a keyboard, touchscreen, pointing device, controller, and / or microphone) and generates user interface objects on the display. The user interface module 722 also prepares and delivers outputs to the user (e.g., voice, sound, animation, text, icons, vibration, haptic feedback, light, etc.) via the I / O interface 706 (e.g., through the display, audio channel, speaker, touchpad, etc.).

[0161] Application 724 includes programs and / or modules configured to run by one or more processors 704. For example, if the digital assistant system is implemented on a standalone user device, application 724 includes user applications such as games, calendar applications, navigation applications, or email applications. If the digital assistant system 700 is implemented on a server, application 724 includes, for example, resource management applications, diagnostic applications, or scheduling applications.

[0162] Memory 702 also stores the digital assistant module 726 (or the server portion of the digital assistant). In some embodiments, the digital assistant module 726 includes the following submodules, or subsets or supersets thereof: input / output processing module 728, speech-to-text (STT) processing module 730, natural language processing module 732, dialogue flow processing module 734, task flow processing module 736, service processing module 738, and speech synthesis processing module 740. Each of these modules has access to one or more of the following systems or data and models of the digital assistant module 726, or subsets or supersets thereof: ontology 760, vocabulary index 744, user data 748, task flow model 754, service model 756, and ASR system 758.

[0163] In some embodiments, by using processing modules, data, and models implemented in the digital assistant module 726, the digital assistant can perform at least some of the following: converting voice input to text; identifying the user's intent expressed in the natural language input received from the user; actively extracting and obtaining the information necessary to fully infer the user's intent (e.g., by clarifying words, games, intentions, etc.); determining a task flow to satisfy the inferred intent; and executing that task flow to satisfy the inferred intent.

[0164] In some embodiments, as shown in Figure 7B, the I / O processing module 728 interacts with the user through the I / O device 716 in Figure 7A, or with a user device (e.g., device 104, 200, 400, or device 600) through the network communication interface 708 in Figure 7A, in order to acquire user input (e.g., voice input) and to provide a response to the user input (e.g., as voice output). The I / O processing module 728 optionally acquires context information associated with the user input from the user device, either with or immediately after receiving the user input. The context information includes user-specific data, vocabulary, and / or preferences related to the user input. In some embodiments, the context information also includes the software and hardware state of the user device at the time the user request is received, and / or information about the user's surrounding environment at the time the user request is received. In some embodiments, the I / O processing module 728 also sends supplementary questions to the user regarding the user request and receives answers from the user. When a user request is received by the I / O processing module 728 and the user request includes voice input, the I / O processing module 728 forwards the voice input to the STT processing module 730 (or a voice recognition device) for speech-to-text conversion.

[0165] The STT processing module 730 includes one or more ASR systems 758. These one or more ASR systems 758 can process the audio input received via the I / O processing module 728 to generate recognition results. Each ASR system 758 includes a front-end audio preprocessor. This front-end audio preprocessor extracts representative features from the audio input. For example, the front-end audio preprocessor can perform a Fourier transform on the audio input to extract spectral features that characterize the audio input as a sequence of representative multidimensional vectors. Furthermore, each ASR system 758 includes one or more speech recognition models (e.g., acoustic models and / or language models) and implements one or more speech recognition engines. Examples of speech recognition models include hidden Markov models, Gaussian mixture models, deep neural network models, n-gram language models, and other statistical models. Examples of speech recognition engines include dynamic time stretching-based engines and weighted finite-state transducer (WFST)-based engines. One or more speech recognition models and one or more speech recognition engines are used to process the extracted representative features of a front-end speech preprocessor to generate intermediate recognition results (e.g., phonemes, phoneme sequences, subwords) and finally text recognition results (words, word sequences, token sequences). In some embodiments, the speech input is processed at least partially by a third-party service or on the user's device (e.g., device 104, 200, 400, or device 600) to generate the recognition results. Once the STT processing module 730 generates a recognition result containing a text string (e.g., words, word sequences, or token sequences), the recognition result is passed to the natural language processing module 732 for intent inference. In some embodiments, the STT processing module 730 generates a number of text representation candidates for the speech input. Each text representation candidate is a word sequence or token sequence corresponding to the speech input. In some embodiments, each text representation candidate is associated with a speech recognition confidence score.Based on this speech recognition reliability score, the STT processing module 730 ranks the text expression candidates and provides the n best (e.g., n highest-ranked) text expression candidates (one or more) to the natural language processing module 732 for intent inference, where n is a predetermined integer greater than zero. For example, in one embodiment, only the highest-ranked (n=1) text expression candidate is passed to the natural language processing module 732 for intent inference. In another embodiment, five highest-ranked (n=5) text expression candidates are passed to the natural language processing module 732 for intent inference.

[0166] Further details regarding the speech-to-text processing are described in U.S. Utility Patent Application No. 13 / 236,942, filed September 20, 2011, for "Consolidating Speech Recognition Results," the entire disclosure of which is incorporated herein by reference.

[0167] In some embodiments, the STT processing module 730 includes a vocabulary of recognizable words and / or accesses that vocabulary via the phonetic-to-text conversion module 731. Each vocabulary word is associated with one or more pronunciation candidates for that word, represented by speech-recognition phonetic characters. Specifically, the vocabulary of recognizable words includes words associated with multiple pronunciation candidates. For example, the vocabulary may include:

number

number

[0168] In some embodiments, pronunciation candidates are ranked based on their generality. For example, pronunciation candidates

number

number

number

number

number

number

[0169] When speech input is received, the STT processing module 730 is used to determine the phonemes corresponding to the speech input (for example, using an acoustic model), and then attempts to determine the words that match those phonemes (for example, using a language model). For example, the STT processing module 730 first determines the phoneme sequence corresponding to a portion of the speech input.

number

[0170] In some embodiments, the STT processing module 730 uses approximate matching techniques to determine words in a utterance. For example, the STT processing module 730 uses phoneme sequences.

number

[0171] The digital assistant's natural language processing module 732 ("natural language processor") retrieves n best text representation candidates (singular or plural) ("word sequences (singular or plural)" or "token sequences (singular or plural)") generated by the STT processing module 730 and attempts to associate each of these text representation candidates with one or more "actionable intentions" recognized by the digital assistant. An "actionable intention" (or "user intention") represents a task that the digital assistant can perform and may have an associated task flow implemented within the task flow model 754. This associated task flow is a series of programmed actions and steps that the digital assistant takes to perform that task. The scope of the digital assistant's capabilities is determined by the number and variety of task flows implemented and stored within the task flow model 754, or in other words, by the number and variety of "actionable intentions" recognized by the digital assistant. However, the effectiveness of a digital assistant also depends on its ability to infer the correct "feasible intent(s)" from user requests expressed in natural language.

[0172] In some embodiments, in addition to the word or token sequence obtained from the STT processing module 730, the natural language processing module 732 also receives contextual information associated with the user request, for example, from the I / O processing module 728. The natural language processing module 732 optionally uses this contextual information to clarify, complement, and / or further define the information contained within the text representation candidates received from the STT processing module 730. Contextual information includes, for example, user preferences, the hardware and / or software state of the user device, sensor information collected before, during, or immediately after the user request, and previous interactions (e.g., dialogues) between the digital assistant and the user. As described herein, the contextual information is dynamic in some embodiments and changes with time, location, dialogue content, and other factors.

[0173] In some embodiments, natural language processing is based on, for example, Ontology 760. Ontology 760 is a hierarchical structure containing numerous nodes, each representing either an "executable intent" or an "attribute" or other "attribute" associated with one or more "executable intents." As described above, an "executable intent" represents a task that a digital assistant can perform, i.e., a task that is "executable" or can be performed. An "attribute" represents a parameter associated with an executable intent, or associated with a sub-part of another attribute. Links between executable intent nodes and attribute nodes within Ontology 760 define how the parameter represented by the attribute node relates to the task represented by the executable intent node.

[0174] In some embodiments, the ontology 760 consists of actionable intent nodes and attribute nodes. Within the ontology 760, each actionable intent node is linked to one or more attribute nodes, either directly or via one or more intermediate attribute nodes. Similarly, each attribute node is linked to one or more actionable intent nodes, either directly or via one or more intermediate attribute nodes. For example, as shown in Figure 7C, the ontology 760 includes a “Restaurant Reservation” node (i.e., an actionable intent node). The attribute nodes “Restaurant,” “Date / Time” (for reservation), and “Number of Participants” are each directly linked to the actionable intent node (i.e., the “Restaurant Reservation” node).

[0175] Furthermore, the attribute nodes "Cuisine," "Price Range," "Phone Number," and "Location" are subnodes of the attribute node "Restaurant," each linked to the "Restaurant Reservation" node (i.e., an actionable intent node) via the intermediate attribute node "Restaurant." As another example, as shown in Figure 7C, ontology 760 also includes a "Reminder Settings" node (i.e., another actionable intent node). The attribute nodes "Date / Time" (for reminder settings) and "Theme" (for reminders) are linked to the "Reminder Settings" node, respectively. Since the attribute "Date / Time" is relevant to both the task of making a restaurant reservation and the task of setting a reminder, the attribute node "Date / Time" is linked to both the "Restaurant Reservation" node and the "Reminder Settings" node within ontology 760.

[0176] An actionable intent node, along with its linked attribute nodes, is described as a “domain.” In this paper, each domain is associated with a specific actionable intent and refers to a group of nodes (and the relationships between those nodes) associated with that particular actionable intent. For example, ontology 760 shown in Figure 7C contains an example of a restaurant reservation domain 762 and an example of a reminder domain 764 within ontology 760. The restaurant reservation domain includes the actionable intent node “Restaurant Reservation,” attribute nodes “Restaurant,” “Date / Time,” and “Number of Participants,” as well as subordinate attribute nodes “Cuisine,” “Price Range,” “Phone Number,” and “Location.” The reminder domain 764 includes the actionable intent node “Reminder Settings,” as well as attribute nodes “Theme” and “Date / Time.” In some embodiments, ontology 760 consists of multiple domains. Each domain shares one or more attribute nodes with one or more other domains. For example, the "Date / Time" attribute node is associated with numerous different domains (e.g., scheduling domains, travel booking domains, movie ticket domains, etc.) in addition to the restaurant reservation domain 762 and the reminder domain 764.

[0177] Figure 7C shows two exemplary domains within Ontology 760, but other domains include, for example, “Find a movie,” “Start a phone call,” “Find directions,” “Schedule a meeting,” “Send a message,” “Provide answers to questions,” “Read a list aloud,” “Provide navigation instructions,” and “Provide instructions about a task.” The “Send a message” domain is associated with the “Send a message” actionable intent node and further includes attribute nodes such as “Recipients,” “Message type,” and “Message body.” The attribute node “Recipients” is further defined by sub-attribute nodes such as “Recipient name” and “Message address.”

[0178] In some embodiments, the ontology 760 includes all domains (and therefore, all actionable intentions) that the digital assistant can understand and perform. In some embodiments, the ontology 760 is modified by adding or removing entire domains or nodes, or by modifying the relationships between nodes within the ontology 760.

[0179] In some embodiments, nodes associated with multiple related actionable intentions are clustered under a “higher-level domain” within Ontology 760. For example, the “Travel” higher-level domain contains a cluster of attribute nodes and actionable intention nodes related to travel. Actionable intention nodes related to travel include “Flight Booking,” “Hotel Booking,” “Car Rental,” “Get Directions,” and “Find Places of Interest.” Actionable intention nodes under the same higher-level domain (e.g., the “Travel” higher-level domain) share a number of common attribute nodes. For example, actionable intention nodes related to “Flight Booking,” “Hotel Booking,” “Car Rental,” “Get Directions,” and “Find Places of Interest” share one or more of the attribute nodes “Departure Point,” “Destination,” “Departure Date / Time,” “Arrival Date / Time,” and “Number of Participants.”

[0180] In some embodiments, each node in the ontology 760 is associated with a set of words and / or phrases related to the attribute or actionable intention represented by that node. The distinct set of words and / or phrases associated with each node is the so-called "vocabulary" associated with that node. The distinct set of words and / or phrases associated with each node is stored in the vocabulary index 744 in relation to the attribute or actionable intention represented by that node. For example, returning to Figure 7B, the vocabulary associated with the node relating to the attribute of "restaurant" includes words such as "food," "drink," "cooking," "hungry," "eat," "pizza," "fast food," and "meal." As another example, the vocabulary associated with the actionable intention of "make a phone call" includes words and phrases such as "call," "phone," "dial," "ring," "call this number," and "make a call to." The vocabulary index 744 optionally includes words and phrases from different languages.

[0181] The natural language processing module 732 receives text representation candidates (e.g., a text string (single or multiple) or a token sequence (single or multiple)) from the STT processing module 730 and determines, for each representation candidate, which node the word in the text representation candidate is associated with. In some embodiments, if a word or phrase in the text representation candidate is deemed to be associated (via the lexical index 744) with one or more nodes in the ontology 760, the word or phrase “triggers” or “activates” those nodes. Based on the number and / or relative importance of the activated nodes, the natural language processing module 732 selects one of those actionable intentions as the task the user intends the digital assistant to perform. In some embodiments, the domain with the most “triggered” nodes is selected. In some embodiments, the domain with the highest confidence value (e.g., based on the relative importance of the various triggered nodes) is selected. In some embodiments, the domain is selected based on a combination of the number and importance of the triggered nodes. In some embodiments, additional factors are considered when selecting a node, such as whether the digital assistant has previously interpreted a similar request from the user accurately.

[0182] User data 748 includes user-specific information such as user-specific vocabulary, user preferences, user address, user's default and second languages, user's contact list, and other short-term or long-term information specific to the user. In some embodiments, the natural language processing module 732 uses this user-specific information to complement the information contained in the user input and to further define the user's intent. For example, with respect to the user request "invite my friends to my birthday party," the natural language processing module 732 can access user data 748 to determine who the "friends" are and when and where the "birthday party" will take place, without requiring the user to explicitly provide such information in the user request.

[0183] It should be noted that in some embodiments, the natural language processing module 732 is implemented using one or more machine learning mechanisms (e.g., neural networks). Specifically, one or more machine learning mechanisms are configured to receive text representation candidates and contextual information associated with those text representation candidates. Based on these text representation candidates and associated contextual information, one or more machine learning mechanisms are configured to determine an intent confidence score for a set of feasible intent candidates. Based on the determined intent confidence score, the natural language processing module 732 can select one or more feasible intent candidates from the set of feasible intent candidates. In some embodiments, an ontology (e.g., ontology 760) is also used to select one or more feasible intent candidates from the set of feasible intent candidates.

[0184] Further details regarding the token string-based ontology search are described in U.S. Utility Patent Application No. 12 / 341,743, filed December 22, 2008, for "Method and Apparatus for Searching Using an Active Ontology," which is incorporated herein by reference in its entirety.

[0185] In some embodiments, once the natural language processing module 732 identifies an actionable intent (or domain) based on a user request, it generates a structured query to represent that identified actionable intent. In some embodiments, this structured query includes parameters for one or more nodes within the domain relating to the actionable intent, with at least some of these parameters populated with specific information and requirements specified in the user request. For example, the user says, "Make me a dinner reservation at a sushi place at 7." In this case, the natural language processing module 732 can accurately identify the actionable intent as "restaurant reservation" based on the user input. According to the ontology, the structured query for the "restaurant reservation" domain includes parameters such as {cuisine}, {time}, {date}, and {number of people}. In some embodiments, based on voice input and text derived from the voice input using the STT processing module 730, the natural language processing module 732 generates a partially structured query about a restaurant reservation domain, which includes the parameters {cuisine = "sushi"} and {time = "7pm"}. However, in this embodiment, the information contained in the user's utterance is insufficient to complete the structured query associated with that domain. Therefore, other necessary parameters such as {number of people} and {date} are not specified in the structured query based on the currently available information. In some embodiments, the natural language processing module 732 populates received contextual information into some of the parameters of this structured query. For example, in some embodiments, if the user requests a "nearby" sushi restaurant, the natural language processing module 732 populates the GPS coordinates from the user's device into the {location} parameter in the structured query.

[0186] In some embodiments, the natural language processing module 732 identifies multiple feasible intent candidates for each text representation candidate received from the STT processing module 730. Furthermore, in some embodiments, a separate (partially or completely) structured query is generated for each identified feasible intent candidate. The natural language processing module 732 determines an intent confidence score for each feasible intent candidate and ranks them based on that intent confidence score. In some embodiments, the natural language processing module 732 passes the generated structured query(s) containing any input parameters to the task flow processing module 736 ("task flow processor"). In some embodiments, structured queries(s) for the m best (e.g., m highest-ranked) feasible intent candidates are provided to the task flow processing module 736, where m is a predetermined integer greater than zero. In some embodiments, structured queries(s) for the m best feasible intent candidates are provided to the task flow processing module 736 along with the corresponding text representation candidates(s).

[0187] Further details of inferring user intent based on multiple feasible intent candidates determined from multiple textual representation candidates of speech input are described in U.S. Utility Patent Application No. 14 / 298,725, filed June 6, 2014, for "System and Method for Inferring User Intent From Speech Inputs," the entire disclosure of which is incorporated herein by reference.

[0188] The task flow processing module 736 is configured to receive a structured query (one or more) from the natural language processing module 732, complete the structured query as needed, and perform the actions required to "complete" the user's final request. In some embodiments, the various steps required to complete these tasks are provided within the task flow model 754. In some embodiments, the task flow model 754 includes steps for obtaining additional information from the user and a task flow for performing actions associated with actionable intents.

[0189] As described above, in order to complete the structured query, the task flow processing module 736 needs to initiate an additional dialogue with the user to obtain additional information and / or to remove any ambiguity in potentially ambiguous statements. If such a dialogue is necessary, the task flow processing module 736 calls the dialogue flow processing module 734 to conduct the dialogue with the user. In some embodiments, the dialogue flow processing module 734 determines how (and / or when) to request additional information from the user and receives and processes the user's response. Questions are provided to the user and answers are received from the user via the I / O processing module 728. In some embodiments, the dialogue flow processing module 734 presents the dialogue output to the user via audio and / or visual output and receives input from the user via verbal or physical (e.g., click) responses. Continuing with the above embodiment, when the task flow processing module 736 calls the dialog flow processing module 734 to determine the "number of participants" and "date" information for a structured query associated with the domain "restaurant reservation", the dialog flow processing module 734 generates questions such as "For how many people?" and "On which day?" and passes them to the user. Once the user's response is received, the dialog flow processing module 734 then passes that information to the task flow processing module 736 to populate the missing information into the structured query or to complete the missing information from the structured query.

[0190] Once the task flow processing module 736 completes the structured query for an actionable intent, it proceeds to execute the final task associated with that actionable intent. Thus, the task flow processing module 736 executes the steps and instructions in the task flow model according to the specific parameters contained within the structured query. For example, a task flow model for the actionable intent of "restaurant reservation" includes steps and instructions to contact a restaurant and actually request a reservation for a specific number of people at a specific time. For example, using a structured query such as {restaurant reservation, restaurant=ABC Cafe, date=3 / 12 / 2012, time=7pm, number of people=5}, the task flow processing module 736 executes the steps of (1) logging on to the ABC Cafe server or a restaurant reservation system such as OPENTABLE®, (2) entering the date, time, and number of people information into a form on the website, (3) submitting the form, and (4) entering a calendar entry for the reservation into the user's calendar.

[0191] In some embodiments, the task flow processing module 736 employs the assistance of the service processing module 738 ("service processing module") to complete a task requested by user input or to provide a response to information requested by user input. For example, the service processing module 738 performs functions on behalf of the task flow processing module 736, such as making phone calls, setting calendar items, calling map searches, calling or interacting with other user applications installed on the user device, or calling or interacting with third-party services (e.g., restaurant reservation portals, social networking websites, banking portals, etc.). In some embodiments, the protocols and application programming interfaces (APIs) required by each service are specified by individual service models in the service model 756. The service processing module 738 accesses the appropriate service model for the service and generates a request for that service in accordance with the protocols and APIs required by that service, as specified in that service model.

[0192] For example, if a restaurant supports an online reservation service, the restaurant submits a service model that specifies the parameters required to make a reservation and an API for communicating the values ​​of those parameters to the online reservation service. When requested by the task flow processing module 736, the service processing module 738 establishes a network connection with the online reservation service using the web address stored in the service model and sends the required reservation parameters (e.g., time, date, number of participants) to the online reservation interface in the format specified by the online reservation service's API.

[0193] In some embodiments, the natural language processing module 732, the dialogue flow processing module 734, and the task flow processing module 736 are used collectively and iteratively to infer and define the user's intent, obtain information to further clarify and refine that intent, and ultimately generate a response (i.e., output to the user or completion of a task) that satisfies the user's intent. The generated response is a dialogue response to the voice input that satisfies at least partially the user's intent. Furthermore, in some embodiments, the generated response is output as voice output. In these embodiments, the generated response is sent to a speech synthesis processing module 740 (e.g., a speech synthesizer), which can process it to synthesize a dialogue response in voice format. In yet other embodiments, the generated response is data content related to satisfying the user request in the voice input.

[0194] In embodiments where the task flow processing module 736 receives multiple structured queries from the natural language processing module 732, the task flow processing module 736 first processes a first structured query from the received structured queries in order to attempt to complete a first structured query and / or to perform one or more tasks or actions represented by the first structured query. In some embodiments, the first structured query corresponds to the highest-ranked actionable intent. In other embodiments, the first structured query is selected from the received structured queries based on a combination of a corresponding speech recognition confidence score and a corresponding intent confidence score. In some embodiments, if the task flow processing module 736 encounters an error while processing the first structured query (for example, due to an undeterminable required parameter), the task flow processing module 736 may proceed to select and process a second structured query from the received structured queries that corresponds to a lower-ranked actionable intent. This second structured query is selected based, for example, on the speech recognition confidence score of the corresponding text representation candidate, the intent confidence score of the corresponding actionable intent candidate, any missing required parameters in the first structured query, or any combination thereof.

[0195] The speech synthesis processing module 740 is configured to synthesize speech output for presentation to the user. The speech synthesis processing module 740 synthesizes speech output based on text provided by a digital assistant. For example, the generated dialogue response is in the form of a text string. The speech synthesis processing module 740 converts that text string into audible speech output. The speech synthesis processing module 740 uses any suitable speech synthesis technique to generate speech output from text, including but not limited to waveform concatenation synthesis, unit selection synthesis, diphon synthesis, domain-specific synthesis, formant synthesis, articulation synthesis, synthesis based on hidden Markov models (HMMs), and sinusoidal synthesis. In some embodiments, the speech synthesis processing module 740 is configured to synthesize individual words based on phoneme sequences corresponding to words. For example, phoneme sequences are associated with words in the generated dialogue response. The phoneme sequences are stored in metadata associated with the words. The speech synthesis processing module 740 is configured to synthesize words in speech form by directly processing the phoneme sequences in the metadata.

[0196] In some embodiments, instead of using (or in addition to) the speech synthesis processing module 740, speech synthesis is performed on a remote device (e.g., server system 108), and the synthesized speech is sent to the user device for output to the user. For example, this can be done in some implementations where the output to the digital assistant is generated on the server system. Also, since the server system generally has more processing power or resources than the user device, it is possible to obtain higher quality speech output than would be realistic in the case of client-side synthesis.

[0197] Further details regarding the digital assistant can be found in U.S. Utility Application No. 12 / 987,982, filed on 10 January 2011, entitled "Intelligent Automated Assistant," and in U.S. Utility Application No. 13 / 251,088, filed on 30 September 2011, entitled "Generating and Processing Task Items That Represent Tasks to Perform," the entire disclosure of which is incorporated herein by reference. 4. Contextual response proposals using secondary electronic devices

[0198] Figures 8, 9A–9C, and 10 illustrate a system for contextual response proposals. For example, device 804 includes, but is not limited to, devices 104, 200, 400, and 600 (Figures 1, 2A, 4, and 6A–6B), and may include any other device described herein. Thus, it will be understood that the electronic devices associated with Figures 8, 9A–9C, and 10 can correspond to any type of user device, such as laptop computers, tablets, telephones, wearable devices (e.g., headphones, smartwatches, smart glasses, or head-mounted displays), and wired or wireless speakers. Furthermore, the processes described herein may be performed by a server using information delivered to and from the devices, performed on the devices, or a combination of both.

[0199] Referring to Figure 8, a system for contextual response suggestion using a secondary electronic device is shown. Generally, user 802 may utilize a primary device such as a smartphone. User 802 may also wear earphones 804, which are communicatively coupled to the smartphone, to perform various activities such as listening to music, making phone calls, and sending and receiving messages. In some cases, earphones 804 can proactively output notifications to the user, such as alerts associated with incoming messages. The user may also be given the opportunity to respond to the notification via voice input to earphones 804 or to take action in another way. Generally, the incoming message may first be received on the user's smartphone device, so that the smartphone device delivers the alert to earphones 804. In some embodiments, the incoming message may be received on another device, such as a tablet computer, which is communicating with earphones 804. In one embodiment, a message such as "When will your flight land?" received from a contact named "John" may be received on user 802's smartphone. Accordingly, user 802's smartphone can be configured to provide audible alerts to earphone 804, such as "John says, 'When will your flight land?'" The smartphone can also be configured to announce various follow-up options to earphone 804, such as "Do you want to respond?", followed by a period during which follow-up audio is heard in earphone 804.

[0200] In general, to improve the messaging experience, various response messages can be determined based on both the content of the received message and contextual information corresponding to the receiving user. The user can affirm the response message, thereby causing the response message to be sent back to the message sender. For example, in response to receiving a message on user 802's smartphone device, the smartphone may be able to provide a prompt 806 in the earphone 804. The first part of prompt 806 may include a notification of the received message, such as "John says, 'When will your flight land?'" Prompt 806 may further include instructions for a response message that may be sent back to the contact "John" via user 802's smartphone. To generate a prompt, a query and / or intent associated with the received message can be identified. In this embodiment, the query may correspond to a task to retrieve current flight information for a user of a receiving device, such as user 802. Based on the identified query, various data sources can be searched to obtain appropriate results to satisfy the query. For example, data can be obtained from sources such as calendar applications, messaging applications, email applications, note-taking applications, map applications, contacts applications, social media applications, game applications, fitness applications, financial and / or banking applications, music applications, and news applications. The data may be stored locally on the user 802's smartphone device, and / or remotely on a cloud-based server accessible via the user 802's smartphone device (for example, linked to a secure account associated with the user). In addition, data sources may include publicly available sources on the internet, subscription-based sources, and local or remote private servers owned by the user.

[0201] Other data sources, such as environmental context, device state, and past user behavior, can also be used to obtain results for generating response messages. For example, a user may have notification settings or other device states turned on, such as "Focus Mode," "Silent," or "Vibrate Only." Therefore, based on device state information, an appropriate response such as "Busy" or "I'll call you later" can be generated. Various device sensors can indicate that the user is currently on the move (and therefore possibly driving), in a noisy environment, or in a conference call, thereby generating an appropriate response message to inform the message sender that the user may not be able to answer at that time.

[0202] Returning to the embodiment in Figure 8, the identified query can be resolved by retrieving flight information from applications such as email applications, calendar applications, and third-party flight applications. The retrieved flight information can then be entered into a flight tracking program (e.g., via a public website, via a third-party application on user 802's smartphone, etc.). The current flight status information is then used to prepare a response message to be sent back to the message sender "John". Specifically, the flight status information may include "Arriving on time, at 8:20 p.m. today". Therefore, the prepared response message could be written as "My flight will arrive at 8:20 p.m.".

[0203] Generally, the prompt provided to the user 802 via the earphone 804 may be less detailed than the complete response message sent back to the message sender and can be generated in various ways. The response message instructions in the prompt may contain fewer than a threshold number of parameters, based on the prepared response message. For example, the prepared response message may contain eight parameters (e.g., date, time, name, location, etc.), while the response message instructions may contain only one or two parameters (e.g., one name and one date). In some cases, the response message instructions may contain fewer than a threshold number of words (e.g., fewer than seven words, fewer than ten words, etc.) compared to the prepared response message. Thus, as long as the prepared response message contains fewer than a threshold number of words, the response message instructions may contain the entire prepared response message. As long as the prepared response message contains more than a threshold number of words, the prepared response message may be intelligently modified to retain parameters or words that are considered more important than other parameters or words (e.g., by removing unnecessary words based on the confidence level of the parameters). Alternatively, if a prepared response message does not meet the appropriate threshold, a general instruction for the response message, such as "Do you want to send John your flight status?", can be provided within prompt 806. In some embodiments, a full preview of the response message, such as "Do you want to send John the message 'My flight will land tonight at 8:20 p.m.'?", may be included within the prompt without exception (based on, for example, user preference or default settings).

[0204] In some cases, multiple results that may satisfy the query obtained from the received message can be retrieved. For example, retrieved flight information may include multiple flights, such as departure and arrival flights. Various confidence scores can be assigned to the results in a specific manner. For example, flight results occurring closer to the current time (e.g., flights departing soon) may be assigned a higher priority, while flights that have already completed may be assigned a lower priority (e.g., flights from yesterday). In another embodiment, past user behavior may influence the confidence of the results, such as associating higher confidence with results from applications the user frequently uses, and conversely, lower confidence with results from applications the user rarely or never uses. As long as multiple results exist, the result with the highest confidence score can be selected as the result used to generate the response message.

[0205] Prompts may be selectively provided to the user based on various contextual circumstances. Generally, the amount of response prompts provided to the user can be adjusted using factors such as the frequency of prompt use, the frequency of responses to specific senders, and / or other user history. Unless user 802 frequently utilizes or accepts a provided prompt, such as prompt 806, future prompts may be provided less frequently or not at all. For example, user 802 may repeatedly respond with "no" to prompts such as prompt 806, and as a result, prompts may no longer be provided after their usage frequency falls below a threshold frequency. Alternatively, user 802 may utilize prompts with one or more specific contacts (e.g., spouse, best friend, etc.) but not with the remaining contacts (e.g., boss, coworker, etc.). Thus, response prompts may be provided when those specific users send messages to user 802, but not when the remaining users send messages to user 802.

[0206] Once a prompt is provided to the user, the user can respond to the prompt in various ways. For example, user 802 can respond to prompt 806 by voice input to earphone 804, such as an affirmative response accepting the prompt (e.g., "yes," "please respond," "uh-huh") or a negative response rejecting the response (e.g., "no," "no," "I won't respond"). Earphone 804 may also include various sensors, such as an accelerometer and / or gyroscope, to perform head tracking of user 802 for gesture recognition. Using the sensors, the user can affirm by nodding up and down to accept the prompt, or negative by shaking their head back and forth to reject prompt 806. Thus, sensor data can be processed to determine the user response without voice input. Alternatively or additionally, user 802 can invoke one or more tap inputs, such as a single tap or multi-tap, on earphone 804 to respond to prompt 806.

[0207] In some embodiments, the user may be offered multiple response options. For example, the user may receive a message from the contact "Jane" that includes the question, "What is your address?" The data source may indicate that the user frequently uses both a "home" address and a "work" address. Therefore, the user may be given a prompt that includes, "Would you like to answer Jane with your home address or your work address?" The user may then respond via voice to select the appropriate response.

[0208] When a user responds to a provided prompt, a determination is made as to whether the user has accepted the prompt (or selected a response within the prompt). As long as the user accepts the prompt, such as prompt 806, a prepared response message is sent as a reply to the message sender, such as "My flight will land tonight at 8:20 p.m." being sent to contact "John". As long as the user rejects the prompt, no response is sent to contact "John". In some cases, user 802 may not provide any response to prompt 806 for a predetermined period (e.g., 5 seconds, 10 seconds, etc.). If no response is received from the user within the predetermined period, no response is sent to contact "John".

[0209] Referring to Figures 9A to 9C, a system for contextual response suggestion for a smartwatch is shown. In particular, Figure 9A shows a smartwatch 902 including a display. The user of the smartwatch 902 may currently be interacting with a messaging application 904 displayed on the display. The messaging application 904 may further include messages 906 received from another user, such as a user named "Olivia" stored as a contact in a smartphone communicably coupled to the smartwatch 902. In some embodiments, the smartwatch 902 can function as a standalone device (e.g., not generally communicating with a smartphone device), thereby the smartwatch 902 includes a contact database including the contact "Olivia" and is communicably coupled to one or more networks, such as a WiFi network and / or a cellular network.

[0210] The received message 906 may contain a question directed at the user of the smartwatch 902, such as "Are you free?". Based on the received message 906, the query and / or intent associated with the received message is identified. For example, the determined intent may correspond to a task to determine the current availability of the user of the receiving device, such as the user of the smartwatch 902. Based on the identified query, various data sources are searched to obtain results that satisfy the query. In this particular embodiment, data may be obtained from sources such as the user's calendar application, social media applications, and device status information. The data can be stored locally on the user's smartphone device and / or smartwatch 902, and / or remotely on a cloud-based server accessible via the user's device(s). In addition, data sources may include publicly available sources on the internet, subscription-based sources, etc.

[0211] In response to a query to a data source, one or more response values ​​can be determined. Generally, response values ​​are determined in an attempt to satisfy the query and / or intent identified from the received message. In this embodiment, the satisfaction of a particular query (determining the user's current availability) may include values ​​such as (i) whether the user is currently available, (ii) if the user is currently available, how long the user will be available, and (iii) if the user is not currently available, the next available time (and optionally, for how long). For example, a user's calendar application might indicate that the user is currently in a meeting and has several meetings scheduled until 3:00 p.m. Therefore, the specific response values ​​determined for this embodiment may include "currently unavailable" and "available at 3:00 p.m. today."

[0212] Referring to Figure 9B, a prompt 908 can be displayed on the smartwatch 902 based on the determined response value. The prompt 908 may include an expression of at least one of one or more response values, such as a list of suggestions based on the response value. In this embodiment, the prompt 908 includes a list of suggestions with the attribute "Suggestions from Calendar" and shows a list of response suggestions based on the user's availability determined from the user's calendar accessible by the smartwatch 902. For example, the first suggestion in the prompt 902 includes the response suggestion "No, my meeting ends at 3pm," which provides a response to a query about the user's current availability, along with instructions on when the user will next be available. Referring to Figure 9C, the user of the smartwatch 902 can scroll down to reveal additional response suggestions in the list, such as a second response suggestion that includes "I'm free after 3pm." Response suggestions may also include varying levels of specificity. In particular, the second response suggestion implies that the user is not available and further indicates that the user will be available after 3pm. A third response suggestion may include "Sorry, I'm busy right now." Here, the third response suggestion is inherently more general than the first and second response suggestions, indicating that the user is not currently available but providing no additional information about the user's future availability. Therefore, in response to user input in response to prompt 908, such as the user selecting a response suggestion from the list of response suggestions, one or more response values ​​are sent to the message sender via a response message indicating that the user is currently unavailable and / or may be available in the future.

[0213] Referring to Figure 10, a smartwatch 1002 is shown. In the context of response suggestions to received messages, various additional types of response values ​​can be used. For example, the user of smartwatch 1002 may receive a message from a contact named "John" that includes a request for payment (e.g., "Can you send me the money from last night's dinner?"). In this embodiment, a data source can be queried to determine a response value corresponding to a payment from the referenced event. In some embodiments, if it is not possible to determine a response value that exceeds a confidence threshold, a default response value can be provided along with options for adjusting the response value. Specifically, an initial amount 1004, such as 20 US dollars, can be displayed on the smartwatch. Options 1006 and 1008 corresponding to options for increasing or decreasing the displayed amount 1004 can also be displayed. In particular, the user of smartwatch 1002 can increase or decrease the amount 1004 by tapping options 1006 and 1008 or otherwise interacting with them. A single tap on option 1006 can result in an increase of a predetermined value such as 1, 5, or 10. A long press on option 1006 can similarly result in a continuous increase of a predetermined value, such as 1 per second, 5 per second, or 10 per second. A single tap on option 1008 can result in a decrease of a predetermined value, such as 1, 5, or 10. A long press on option 1008 can result in a continuous decrease of a predetermined value, such as 1 per second, 5 per second, or 10 per second. Once the user is satisfied with the amount, the user can tap the amount 1004, or optionally, an acceptance affordance (not shown), to initiate a transaction to the contact "John" using one or more applications, such as on the smartwatch 1002 or on the corresponding smartphone.

[0214] In some embodiments, various responses may require authentication before the response is sent, such as transmitting a certain amount of US dollars to another user or providing personal information (e.g., a social security number, a medical ID number, etc.). Therefore, if accepting a prompt requires authentication before sending a response value, an authentication procedure is performed to authenticate the user. The authentication procedure may involve various types of authentication on another device, such as the smartwatch 1002 or a smartphone that is communicably coupled. For example, passcode authentication, facial recognition authentication, fingerprint authentication, or other types of biometric authentication (e.g., determining that the same user's wrist has been in contact with the smartwatch 1002 since the last passcode entry) may be performed before initiating any individual task that requires authentication.

[0215] Various other types of response values ​​can be incorporated into message response suggestions via a smartwatch. For example, in response to an received message such as "Where are you listening?", a graphical representation including an image of an artist or album cover can be displayed on the smartwatch as a response suggestion. The artist or album cover can correspond to the song the user is currently listening to (or, alternatively, the song they have recently listened to). In some cases, multiple images can be provided, including a list of recently played media. When the user selects an image, a link to the corresponding media item can be sent to the message sender as a response. In another embodiment, in response to an received message such as "Where are you?" or "What is your address?", an image including a thumbnail of a location may be displayed as a response suggestion. The thumbnail of a location can correspond to the user's current location (determined, for example, by GPS), a saved "home" location, a saved "work" location, etc.

[0216] In some embodiments, the prompt includes an option to present a customized response. Therefore, unless the user wishes to use any of the responses suggested based on a determined response value, the user can accept the option to present a customized response via the smartwatch 1002 and enter input accordingly. In this case, the determined response value is not sent to the message sender's device. Input can be provided via various modalities, such as voice input (which is further converted to text input) or one or more gesture inputs representing one or more characters.

[0217] Generally, the process for contextual response suggestions can be performed on a variety of different devices, including but not limited to mobile phones, tablet computers, desktop and / or laptop computers, automotive display systems, and head-mounted displays. For example, while a user is interacting with their mobile phone, they may receive a message from another user containing the text, "How long will it take you to get here?" Following the process described above, suggestions, including response suggestions obtained from data sources such as a map application, can be displayed to the user. Specifically, the response suggestion may be determined based on a route calculation from the user's current location to the location corresponding to the message sender, and may be displayed on the mobile phone (and / or announced audibly), including "I'll be there in 20 minutes." Similarly, a user may be using an instant messaging application on a laptop computer and receive a message such as, "What time is our meeting tomorrow?" Using a data source such as a calendar application, the response suggestion may be determined based on a calendar appointment with contact information associated with the message sender. For example, a response suggestion such as "Our meeting is tomorrow from 3pm to 4pm" may be displayed (and / or announced audibly). In another embodiment, a user may connect their mobile phone to an intelligent automotive display system in a communicative manner, allowing the user to perform various phone functions using an interactive display installed in the vehicle. The user may receive messages on their mobile phone such as, "What are you listening to?" Information can then be retrieved from data sources, such as the media currently being played from a media player application. Thus, response suggestions such as, "You are listening to Symphony No. 5 by The Grand Orchestra," can be provided via the vehicle's interactive display (e.g., by displaying and / or announcing).

[0218] Figures 11 and 12 illustrate processes 1100 and 1200 for contextual response suggestion in various embodiments. Processes 1100 and 1200 are performed using, for example, one or more electronic devices implementing a digital assistant. In some embodiments, processes 1100 and 1200 are performed using a client-server system (e.g., system 100), and blocks of processes 1100 and 1200 are divided in any manner between a server (e.g., DA server 106) and client devices. In other embodiments, blocks of processes 1100 and 1200 are divided between a server and multiple client devices (e.g., mobile phones and smartwatches). Thus, although parts of processes 1100 and 1200 are described herein as being performed by specific devices in a client-server system, it will be understood that processes 1100 and 1200 are not limited in this way. In other embodiments, processes 1100 and 1200 are performed using only a client device (e.g., user device 104) or using only multiple client devices. In processes 1100 and 1200, some blocks are optionally combined, the order of some blocks is optionally changed, and some blocks are optionally omitted. In some embodiments, additional steps can be performed in combination with processes 1100 and 1200.

[0219] Referring to Figure 11, in block 1102, in some embodiments, the first electronic device receives a message. In some embodiments, the received message includes a message received on the first electronic device as part of an instant messaging conversation. In block 1104, in some embodiments, the first electronic device determines a response message based on the received message. In some embodiments, determining a response message includes identifying a query associated with the received message and retrieving results from at least one data source based on the identified query, and the response message is generated based on the results. In some embodiments, the at least one data source includes at least one of a calendar application, a messaging application, an email application, a memo application, a map application, and a contacts application. In some embodiments, determining a response message includes retrieving multiple results based on the received message and identifying the result with the highest confidence score from the multiple results, and the response message is generated based on the identified result. In some embodiments, determining a response message includes determining a response message using at least one parameter associated with the state of the electronic device, according to a determination that the state of the first electronic device corresponds to a predetermined state.

[0220] By leveraging a diverse set of data sources such as calendars, messages, notes, and maps, the system enhances device functionality by increasing the available context for generating response suggestions. This increased context makes the device more efficient by improving the quality of contextual suggestions. Consequently, these features reduce power consumption and improve the device's battery life, enabling users to use the device more quickly and efficiently.

[0221] In block 1106, in some embodiments, the first electronic device causes the second electronic device to provide a prompt including instructions for the received message and response message. In some embodiments, the first electronic device identifies the frequency of use associated with previous prompt acceptances from the user, and according to the determination that the frequency of use meets or exceeds a threshold frequency, the first electronic device causes the second electronic device to provide a prompt, and according to the determination that the frequency of use does not exceed a threshold frequency, the first electronic device discontinues causing the second electronic device to provide a prompt. In some embodiments, the first electronic device identifies the response frequency associated with the message sender of the received message, and according to the determination that the response frequency meets or exceeds a threshold frequency, the first electronic device causes the second electronic device to provide a prompt, and according to the determination that the response frequency does not exceed a threshold frequency, the first electronic device discontinues causing the second electronic device to provide a prompt. In some embodiments, the second electronic device corresponds to headphones communicably coupled to the first electronic device. In some embodiments, the prompt provided by the second electronic device is an audible prompt. In some embodiments, the instruction for the response message includes fewer than a threshold number of parameters from the determined response message. In some embodiments, according to the determination that the determined response message contains fewer than a threshold number of words, the first electronic device provides the determined response message as the instruction for the response message. In some embodiments, according to the determination that the determined response message contains more than a threshold number of words, the first electronic device modifies the determined response message by shortening its length and further provides the modified response message as the instruction for the response message. In some embodiments, in response to receiving a message, multiple response messages are determined based on the received message, and the instruction for the response message includes multiple response options.

[0222] By shortening the length of response suggestions, the system enhances device functionality by limiting the output provided to the user. Limiting the output length makes the device more efficient by providing information to the user in a timely manner. Therefore, these features reduce power consumption and improve the device's battery life by enabling the user to use the device more quickly and efficiently.

[0223] In block 1108, in some embodiments, the first electronic device transmits a response message according to a determination that the received input corresponds to an acceptance of the prompt, based on a determination that a prompt-responding input has been received from the second electronic device. In some embodiments, the first electronic device refrains from transmitting a response message according to a determination that a prompt-responding input has not been received from the second electronic device. In some embodiments, the determination that a prompt-responding input has not been received from the second electronic device includes a determination that a prompt-responding input has not been received within a predetermined period. In some embodiments, the prompt-responding input includes one of voice input, head gesture, and touch input. In some embodiments, the first electronic device receives a user-selected response option from a plurality of response options as a prompt-responding input, identifies a response message from a plurality of response messages based on the prompt-responding input, and the transmitted response message corresponds to the identified response message. In block 1110, upon determining that an input in response to a prompt has been received from the second electronic device, the first electronic device, upon determining that the received input corresponds to a rejection of the prompt, refrains from sending a response message.

[0224] By considering various input modalities such as voice input or head gestures, the system enhances device functionality by providing users with multiple ways to affirm or reject response suggestions. Providing multiple response options makes the device more efficient by increasing the user's ability to respond to suggestions. Consequently, these features reduce power consumption and improve the device's battery life by enabling users to use the device more quickly and efficiently.

[0225] Referring to Figure 12, in block 1202, in some embodiments, the message is received by the first electronic device. In block 1204, in some embodiments, the first electronic device queries a data source associated with the user of the first electronic device based on the received message. In some embodiments, at least one data source includes at least one of the following: a calendar application, a messaging application, an email application, a memo application, a map application, and a contacts application. In block 1206, in some embodiments, in response to the query, the first electronic device determines one or more response values.

[0226] By leveraging a diverse set of data sources such as calendars, messages, notes, and maps, the system enhances device functionality by increasing the available context for generating response suggestions. This increased context makes the device more efficient by improving the quality of contextual suggestions. Consequently, these features reduce power consumption and improve the device's battery life, enabling users to use the device more quickly and efficiently.

[0227] In block 1208, in some embodiments, the first electronic device causes the second electronic device to display a prompt containing one or more response value representations. In some embodiments, the one or more response value representations include a list of response suggestions. In some embodiments, the first response suggestion in the list of response suggestions includes a specific response suggestion containing an individual response value, and the second response suggestion in the list of response suggestions includes a general response suggestion that does not contain an individual parameter, and the general response suggestion is based on an individual response value. In some embodiments, the one or more response value representations include a response value, an option to increase the response value, and an option to decrease the response value. In some embodiments, in response to detecting a first user selection of the option to increase the response value, the first electronic device displays an increase in the displayed representation of one or more response values, and in response to detecting a second user selection of the option to decrease the response value, the first electronic device displays a decrease in the displayed representation of one or more response values.

[0228] By providing options to increase or decrease the values ​​of response options, the system enhances device functionality by increasing user customization options for responses. Increased customization makes the device more efficient by improving the options available to the user in responding to messages. Thus, these features reduce power consumption and improve the device's battery life by enabling the user to use the device more quickly and efficiently.

[0229] In block 1210, upon determination that an input in response to a prompt has been received from the second electronic device, the first electronic device transmits at least one of one or more response values, upon determination that the received input corresponds to acceptance of the prompt. In some embodiments, upon determination that the received input corresponds to acceptance of the prompt, the first electronic device performs an authentication procedure to authenticate the user, upon determination that acceptance of the prompt requires authentication before transmitting at least one of one or more response values. In some embodiments, the authentication procedure is performed by the first electronic device. In some embodiments, the authentication procedure is performed by the second electronic device. In some embodiments, the authentication procedure includes verifying that periodic biometric authentication is valid.

[0230] By performing authentication for corresponding response types, the system enhances device functionality by increasing security associated with tasks involving personal information. This improved security makes the device more efficient by reducing the need to manually protect information. Consequently, these features reduce power consumption and improve the device's battery life by enabling users to use the device more quickly and efficiently.

[0231] In block 1212, in some embodiments, upon determination that an input in response to a prompt has been received from the second electronic device, the first electronic device refrains from transmitting at least one of one or more response values, upon determination that the received input corresponds to a rejection of the prompt. In some embodiments, the prompt includes an option for presenting a customized response. In some embodiments, the input provided by the user in the second electronic device includes a voice input and one or more characters based on one or more respective gesture inputs. In some embodiments, the input in response to a prompt includes acceptance of the option for presenting a customized response and the input provided by the user in the second electronic device. In some embodiments, upon determination that the input in response to a prompt includes acceptance of the option for presenting a customized response, the first electronic device refrains from transmitting at least one of one or more response values ​​and, based on the input provided by the user in the second electronic device, sends a message to the sender of the received message. In some embodiments, the representation of one or more response values ​​includes a graphical representation. In some embodiments, the graphical representation corresponds to a media item, which is either a media item currently being played on the first electronic device or the most recent media item played on the first electronic device. In some embodiments, the graphical representation includes a location on a map. In some embodiments, the location includes either the current location of the first electronic device or a location associated with the user of the first electronic device.

[0232] By providing response types that involve location or media, the system enhances device functionality by providing an efficient means for sharing dynamic information. Sharing dynamic information makes the device more efficient by reducing the input that users typically have to make when sending messages. Consequently, these features reduce power consumption and improve the device's battery life by enabling users to use the device more quickly and efficiently.

[0233] The operations described above, with reference to Figures 11 and 12, are optionally implemented by the components shown in Figures 1-4, 6A and 6B, and 7A-7C. For example, the operation of process 900 can be implemented by one or more of the following: operating system 718, application module 724, I / O processing module 728, STT processing module 730, natural language processing module 732, vocabulary index 744, task flow processing module 736, service processing module 738, media service(s) 120-1, or processor(s) 220, 410, and 704. How other processes are implemented based on the components shown in Figures 1-4, 6A and 6B, and 7A-7C will be obvious to those skilled in the art.

[0234] In some implementations, a computer-readable storage medium (e.g., a non-temporary computer-readable storage medium) is provided, which stores one or more programs executed by one or more processors of an electronic device, and these one or more programs include instructions for executing any of the methods or processes described herein.

[0235] In some implementations, an electronic device (e.g., a portable electronic device) is provided that includes means for performing any of the methods or processes described herein.

[0236] In some implementations, an electronic device (e.g., a portable electronic device) is provided, comprising a processing unit configured to perform one of the methods or processes described herein.

[0237] In some implementations, an electronic device (e.g., a portable electronic device) is provided, comprising one or more processors and a memory storing one or more programs to be executed by the one or more processors, wherein the one or more programs include instructions for executing any of the methods or processes described herein.

[0238] The above is written with reference to specific embodiments for illustrative purposes. However, the above exemplary discussion is not intended to be exhaustive or to limit the invention to the exact form disclosed. Many modifications and variations are possible in light of the above teachings. Embodiments have been selected and described to best illustrate the principles of the Art and their practical applications. This will enable other persons skilled in the art to best utilize the Art and its various embodiments with various modifications suitable for their intended specific applications.

[0239] While the present disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. Such changes and modifications should be understood to fall within the scope of the present disclosure and examples as defined by the claims.

[0240] As described above, one aspect of the technology involves collecting and using data available from various sources to improve contextual response suggestions. This disclosure intends that, in some cases, such collected data may include personal information data that uniquely identifies a particular person, or personal information data that can be used to contact a particular person or locate them. Such personal information data may include demographic data, location-based data, telephone numbers, email addresses, Twitter IDs, home addresses, data or records relating to a user's health or fitness level (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0241] This disclosure acknowledges that such use of personal data in the technology may be for the benefit of the user. For example, personal information such as contact information can be used to provide contextual response suggestions. Furthermore, other uses of personal data that benefit the user are also intended by this disclosure. For example, health data and fitness data can be used to provide insights into the user's overall wellness, or as positive feedback to individuals using the technology to pursue wellness goals.

[0242] This disclosure is intended to ensure that entities involved in the collection, analysis, disclosure, transmission, storage, or other use of such personal data comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or government requirements for keeping personal data confidential and secure. Such policies should be readily accessible to users and should be updated as data collection and / or use changes. Personal data from users should be collected for the lawful and legitimate use of the entity and should not be shared or sold except for such lawful use. Furthermore, such collection / sharing should only occur after informing and obtaining the user's consent. In addition, such entities should consider taking all necessary steps to protect and secure access to such personal data and to ensure that others with access to personal data faithfully adhere to those privacy policies and procedures. Furthermore, such entities may undergo third-party evaluations to demonstrate their compliance with widely accepted privacy policies and practices. Furthermore, policies and practices should be adapted to the specific types of personal data collected and / or accessed, and should comply with applicable laws and standards, including jurisdiction-specific considerations. For example, in the United States, the collection or access to certain health data may be subject to federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA). Health data in other countries, on the other hand, may be subject to other regulations and policies and should be addressed accordingly. Therefore, different privacy practices should be maintained in each country with respect to different types of personal data.

[0243] Notwithstanding the foregoing, this disclosure is also intended to provide embodiments that allow users to selectively prevent the use of or access to personal data. Specifically, this disclosure is intended to provide hardware and / or software elements to prevent or prevent access to such personal data. For example, the technology could be configured to allow a user to choose to “opt in” or “opt out” of participating in the collection of personal data during or at any time thereafter when registering for the service. In another example, the user could choose not to provide contact information data. In yet another example, the user could choose to limit the details provided regarding contact information, messaging habits, etc. In addition to providing “opt-in” and “opt-out” options, this disclosure is intended to provide notices regarding access to or use of personal data. For example, the user may be notified when downloading an app that will access the user’s personal data, and then again immediately before the app accesses the personal data.

[0244] Furthermore, the intent of this disclosure is that personal data should be managed and handled in a manner that minimizes the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data when it is no longer needed. In addition, where applicable in certain health-related applications, data anonymization can be used to protect user privacy. Anonymization can be facilitated by removing certain identifiers (e.g., date of birth), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data across users), and / or by other means, where necessary.

[0245] Therefore, while this disclosure broadly covers the use of personal data to implement one or more different disclosed embodiments, it is also intended that these embodiments can be implemented without requiring access to such personal data. In other words, the various embodiments of the Technology are not rendered inoperable by the absence of all or part of such personal data. For example, contextual response suggestions can be achieved based on minimal personal information, such as non-personal data, or de-identified contact data, other non-personal information available in the messaging system, or publicly available information.

Claims

1. A method performed by a computer, In a first electronic device having one or more processors and memory, Receiving a message and, Determining a response message based on the received message, The second electronic device provides a prompt including the instructions for the received message and the response message, In accordance with the determination that an input in response to the prompt has been received from the second electronic device, Sending the response message in accordance with the determination that the received input corresponds to acceptance of the prompt, In accordance with the determination that the received input corresponds to a rejection of the prompt, the sending of the response message is to be stopped. Methods that include...

2. The method according to claim 1, wherein the received message includes a message received on the first electronic device as part of an instant messaging conversation.

3. Determining the aforementioned response message Identifying the query associated with the received message, Based on the identified query, obtain results from at least one data source, Includes, The response message is generated based on the result. The method according to claim 1 or 2.

4. The method according to claim 3, wherein the at least one data source includes at least one of a calendar application, a messaging application, an email application, a memo application, a map application, and a contacts application.

5. Determining the aforementioned response message Obtaining multiple results based on the received message, From the above multiple results, identify the result with the highest reliability score, Includes, The response message is generated based on the identified result. The method according to any one of claims 1 to 4.

6. Determining the aforementioned response message The process includes determining the response message using at least one parameter associated with the state of the electronic device, in accordance with the determination that the state of the first electronic device corresponds to a predetermined state, The method according to any one of claims 1 to 5.

7. To identify the frequency of use associated with previous prompt acceptances from the aforementioned user, In accordance with the determination that the aforementioned usage frequency meets or exceeds a threshold usage frequency, the prompt is provided by the second electronic device. In accordance with the determination that the aforementioned usage frequency does not exceed the aforementioned threshold usage frequency, the provision of the prompt by the second electronic device is discontinued. The method according to any one of claims 1 to 6, including the method described in any one of claims 1 to 6.

8. Identifying the response frequency associated with the message sender of the received message, The second electronic device is made to provide the prompt in accordance with the determination that the response frequency satisfies or exceeds a threshold usage frequency. In accordance with the determination that the response frequency does not exceed the threshold usage frequency, the provision of the prompt by the second electronic device is discontinued. The method according to any one of claims 1 to 7, including the method described in any one of claims 1 to 7.

9. The method according to any one of claims 1 to 8, wherein the second electronic device corresponds to headphones that are communicatively coupled to the first electronic device.

10. The method according to any one of claims 1 to 9, wherein the prompt provided in the second electronic device is an audible prompt.

11. The method according to any one of claims 1 to 10, wherein the input in response to the prompt includes one of voice input, head gesture, and touch input.

12. The method according to any one of claims 1 to 11, wherein the instruction of the response message includes a number of parameters less than a threshold number from the determined response message.

13. The determined response message includes providing the determined response message as the instruction for the response message, in accordance with the determination that the determined response message contains fewer than a threshold number of words. The method according to any one of claims 1 to 12.

14. In accordance with the determination that the determined response message contains more words than the threshold number, Modifying the determined response message by shortening the length of the determined response message, To provide the modified response message as the instruction of the response message, The method according to any one of claims 1 to 13, including the method described in any one of claims 1 to 13.

15. The process includes, in determining that no input in response to the prompt is received from the second electronic device, ceasing to send the response message, The method according to any one of claims 1 to 14.

16. The method according to claim 15, wherein the determination that no input in response to the prompt is received from the second electronic device includes the determination that no input in response to the prompt is received within a predetermined period of time.

17. The process includes determining a plurality of response messages based on the received message in response to the receipt of the message, wherein the instructions in the response messages include a plurality of response options. The method according to any one of claims 1 to 16.

18. The input in response to the prompt is to receive the response option selected by the user from the plurality of response options, Based on the input in response to the prompt, identify a response message from the plurality of response messages, The transmitted response message includes, and the transmitted response message corresponds to the identified response message, The method according to claim 17.

19. One or more processors, Memory and One or more programs, A first electronic device comprising, wherein one or more programs are stored in the memory and executed by one or more processors, Received a message, Based on the received message, a response message is determined. The second electronic device provides a prompt including the instructions for the received message and the response message. In accordance with the determination that an input in response to the prompt has been received from the second electronic device, The response message is sent in accordance with the determination that the received input corresponds to acceptance of the prompt. In accordance with the determination that the received input corresponds to a rejection of the prompt, the sending of the response message is canceled. Includes instructions for, The first electronic device.

20. A non-temporary computer-readable storage medium that stores one or more programs configured to be executed by one or more processors of a first electronic device, wherein the one or more programs are Received a message, Based on the received message, a response message is determined. The second electronic device provides a prompt including the instructions for the received message and the response message. In accordance with the determination that an input in response to the prompt has been received from the second electronic device, The response message is sent in accordance with the determination that the received input corresponds to acceptance of the prompt. In accordance with the determination that the received input corresponds to a rejection of the prompt, the sending of the response message is canceled. Includes instructions for, Non-temporary computer-readable storage medium.

21. A means of receiving a message, Means for determining a response message based on the received message, Means for causing a second electronic device to provide a prompt including instructions for the received message and the response message, In accordance with the determination that an input in response to the prompt has been received from the second electronic device, Means for sending the response message in accordance with the determination that the received input corresponds to acceptance of the prompt, Means for ceasing to send the response message in accordance with the determination that the received input corresponds to a rejection of the prompt, A first electronic device comprising the following:

22. One or more processors, Memory and One or more programs, A first electronic device comprising, wherein one or more programs are stored in the memory and are configured to be executed by one or more processors, and include instructions for performing the method according to any one of claims 1 to 18, The first electronic device.

23. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of an electronic device, the electronic device causes the electronic device to execute the method according to any one of claims 1 to 18.

24. The means for carrying out the method described in any one of claims 1 to 18 Electronic devices.

25. A method performed by a computer, In a first electronic device having one or more processors and memory, Receiving a message and, Based on the received message, query the data source associated with the user of the first electronic device, In response to the aforementioned query, one or more response values ​​are determined, A prompt containing one or more representations of the aforementioned response values ​​is displayed on a second electronic device, In accordance with the determination that an input in response to the prompt has been received from the second electronic device, In accordance with the determination that the received input corresponds to acceptance of the prompt, at least one of the one or more response values ​​is transmitted. In accordance with the determination that the received input corresponds to a rejection of the prompt, the transmission of at least one of the one or more response values ​​is stopped. Methods that include...

26. The method according to claim 25, wherein the at least one data source includes at least one of a calendar application, a messaging application, an email application, a memo application, a map application, and a contacts application.

27. The method according to claim 25 or 26, wherein the representation of one or more response values ​​includes a list of response suggestions.

28. The first response proposal in the aforementioned list of response proposals includes a specific response proposal that includes individual response values, The second response proposal in the list of response proposals includes a general response proposal that does not include the individual parameters, and the general response proposal is based on the individual response values, The method according to claim 27.

29. The method according to any one of claims 25 to 28, wherein the representation of one or more response values ​​includes a response value, an option to increase the response value, and an option to decrease the response value.

30. In response to detecting a first user selection of the option that increases the response value, an increase is displayed within the displayed representation of one or more response values. In response to detecting a second user selection of the option that reduces the response value, the reduction is displayed within the displayed representation of one or more response values. The method according to claim 29, including the method described in claim 29.

31. The method according to any one of claims 25 to 30, wherein the representation of one or more response values ​​includes a graphical representation.

32. The method according to claim 31, wherein the graphical representation corresponds to a media item, and the media item is one of the media items currently being played on the first electronic device or the most recent media item played on the first electronic device.

33. The method according to claim 31, wherein the graphical representation includes a location on a map.

34. The method according to claim 33, wherein the position includes one of the current position of the first electronic device or the position of the first electronic device associated with the user.

35. The method according to any one of claims 25 to 34, wherein the prompt includes an option for providing a customized response.

36. The input in response to the prompt is, Acceptance of options for presenting a customized response, The input provided by the user in the second electronic device, including, The method according to any one of claims 25 to 35.

37. The method according to claim 36, wherein the input provided by the user in the second electronic device includes one of voice input and one or more characters based on one or more gesture inputs.

38. In accordance with the determination that the input in response to the prompt includes the acceptance of the option for presenting the customized response, the transmission of at least one of the one or more response values ​​is to be stopped. The second electronic device transmits a message to the sender of the received message based on the input provided by the user, The method according to claim 36 or 37, including the method described in claim 36 or 37.

39. In accordance with the determination that the received input corresponds to acceptance of the prompt, The process includes performing an authentication procedure to authenticate the user, in accordance with the determination that the acceptance of the prompt requires authentication before the transmission of at least one of the one or more response values. The method according to any one of claims 25 to 38.

40. The method according to claim 39, wherein the authentication procedure is performed in the first electronic device.

41. The method according to claim 39, wherein the authentication procedure is performed in the second electronic device.

42. The method according to claim 41, wherein the authentication procedure includes verifying that periodic biometric authentication is valid.

43. One or more processors, Memory and One or more programs, A first electronic device comprising, wherein one or more programs are stored in the memory and executed by one or more processors, Received a message, Based on the received message, a response message is determined. The second electronic device provides a prompt including the instructions for the received message and the response message. In accordance with the determination that an input in response to the prompt has been received from the second electronic device, The response message is sent in accordance with the determination that the received input corresponds to acceptance of the prompt. In accordance with the determination that the received input corresponds to a rejection of the prompt, the sending of the response message is canceled. Includes instructions for, The first electronic device.

44. A non-temporary computer-readable storage medium that stores one or more programs configured to be executed by one or more processors of a first electronic device, wherein the one or more programs are Received a message, Based on the received message, a response message is determined. The second electronic device provides a prompt including the instructions for the received message and the response message. In accordance with the determination that an input in response to the prompt has been received from the second electronic device, The response message is sent in accordance with the determination that the received input corresponds to acceptance of the prompt. In accordance with the determination that the received input corresponds to a rejection of the prompt, the sending of the response message is canceled. Includes instructions for, Non-temporary computer-readable storage medium.

45. A means of receiving a message, Means for determining a response message based on the received message, Means for causing a second electronic device to provide a prompt including instructions for the received message and the response message, In accordance with the determination that an input in response to the prompt has been received from the second electronic device, A means for sending the response message in accordance with the determination that the received input corresponds to acceptance of the prompt, Means for ceasing to send the response message in accordance with the determination that the received input corresponds to a rejection of the prompt, A first electronic device comprising the following:

46. One or more processors, Memory and One or more programs, A first electronic device comprising, wherein one or more programs are configured to be stored in the memory and executed by one or more processors, and include instructions for performing the method according to any one of claims 25 to 42, The first electronic device.

47. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of an electronic device, the electronic device causes the electronic device to execute the method according to any one of claims 25 to 42.

48. An electronic device comprising means for performing the method described in any one of claims 25 to 42.

Images