Device control using gaze information

Abstract

Regarding computer user interfaces, in particular, this provides techniques for controlling electronic devices using gaze information. [Solution] In some embodiments, the electronic device uses gaze information to activate a digital assistant. In some embodiments, the electronic device uses gaze information to identify an external device to interact with. In some embodiments, the electronic device provides instructions that distinguish between different speakers.

Description

Cross - reference to related applications

[0001] This application claims priority to U.S. Provisional Patent Application No. 62 / 739,087, filed on September 28, 2018, with the title "DEVICE CONTROL USING GAZE INFORMATION", the entire content of which is incorporated herein by reference.

Technical Field

[0002] This disclosure generally relates to computer user interfaces, and more particularly to techniques for controlling an electronic device using gaze information.

Background Art

[0003] Users often provide inputs such as key presses and voice inputs to control an electronic device. For example, a user may activate a device button or utter a trigger phrase to start an application on the device. Such inputs often require the user to be within reach of their hand or within the range of a microphone.

[0004] Intelligent automated assistants (or digital assistants) can provide a useful interface between human users and electronic devices. Such assistants may enable users to interact with the device or system using natural language in utterance and / or text form. For example, a user can provide utterance input containing user requests to a digital assistant running on an electronic device. The digital assistant can interpret the user's intent from the utterance input, translate that intent into a task, and execute that task. In some systems, executing a task in this manner may be constrained by how the task is perceived. However, in some cases, users may be limited to a specific set of commands, and it may not be easy for the user to instruct the digital assistant to perform a task using natural language utterance input. Furthermore, in many cases, digital assistants are unable to adapt based on previous user behavior and similarly lack the desired optimization of the user experience. [Overview of the project]

[0005] However, some techniques for controlling electronic devices are generally cumbersome and inefficient. For example, some existing techniques use complex and time-consuming user interfaces that may involve multiple key presses or keystrokes. In another embodiment, some existing techniques require the user to be within arm's length to activate a device button. Existing techniques take more time than necessary, wasting both the user's time and the device's energy. The latter problem is particularly serious in battery-powered devices.

[0006] Therefore, this technique provides electronic devices with faster and more efficient methods and interfaces for controlling them. Such methods and interfaces optionally complement or replace other methods for controlling electronic devices. Such methods and interfaces reduce the cognitive burden on the user and create a more efficient human-machine interface. In the case of battery-operated computing devices, such methods and interfaces conserve power and extend the interval between battery charges. Such techniques also enable users to interact with electronic devices more efficiently when the user is not within reach of the electronic device and / or when the user is in a noisy environment (e.g., noise based on sounds generated by the electronic device).

[0007] According to several embodiments, a method is provided. This method is performed in an electronic device. The method involves acquiring first gaze information using one or more camera sensors while the digital assistant of the electronic device is not activated, The process includes activating a digital assistant on an electronic device in accordance with a determination that a first gaze information satisfies a set of one or more activation criteria, and providing an instruction that a set of one or more activation criteria has been met.

[0008] According to some embodiments, a non-temporary computer-readable storage medium is provided. The medium stores one or more programs configured to be executed by one or more processors of an electronic device. The one or more programs include: instructions for acquiring first gaze information using one or more camera sensors while the digital assistant of the electronic device is not activated; instructions for activating the digital assistant of the electronic device according to a determination that the first gaze information satisfies a set of one or more activation criteria; and instructions for providing an indication that the set of one or more activation criteria is satisfied.

[0009] According to some embodiments, a temporary computer-readable storage medium is provided. The medium stores one or more programs configured to be executed by one or more processors of an electronic device. The one or more programs include instructions for acquiring first gaze information using one or more camera sensors while the digital assistant of the electronic device is not activated; instructions for activating the digital assistant of the electronic device according to a determination that the first gaze information satisfies a set of one or more activation criteria; and instructions for providing an indication that the set of one or more activation criteria is satisfied.

[0010] According to several embodiments, an electronic device is provided. The electronic device comprises one or more processors and a memory for storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for acquiring first gaze information using one or more camera sensors while the digital assistant of the electronic device is not activated, instructions for activating the digital assistant of the electronic device according to a determination that the first gaze information satisfies a set of one or more activation criteria, and instructions for providing an indication that the set of one or more activation criteria has been satisfied.

[0011] According to several embodiments, an electronic device is provided. The electronic device includes means for acquiring first gaze information using one or more camera sensors while the digital assistant of the electronic device is not activated, and means for activating the digital assistant of the electronic device and providing an indication that the set of one or more activation criteria is met, based on a determination that the first gaze information satisfies a set of one or more activation criteria.

[0012] According to several embodiments, a method is provided. The method is performed in an electronic device. The method includes receiving a voice user input request for executing a first command while a first external device is in a first state; acquiring first gaze information using one or more camera sensors; and, using the first gaze information, sending a command to transition the first external device from a first state to a second state based on a first command, in accordance with a determination that one or more sets of gaze criteria are met for the first external device.

[0013] According to some embodiments, a non-temporary computer-readable storage medium is provided. The medium stores one or more programs configured to be executed by one or more processors of an electronic device. The one or more programs include instructions for receiving a voice user input request to execute a first command while a first external device is in a first state; instructions for acquiring first gaze information using one or more camera sensors; and instructions for transmitting instructions, based on a first command, for transitioning the first external device from a first state to a second state, according to a determination that one or more sets of gaze criteria are met for the first external device using the first gaze information.

[0014] According to some embodiments, a temporary computer-readable storage medium is provided. The medium stores one or more programs configured to be executed by one or more processors of an electronic device. The one or more programs include instructions for receiving a voice user input request to execute a first command while a first external device is in a first state; instructions for acquiring first gaze information using one or more camera sensors; and instructions for transmitting instructions, based on a first command, for transitioning the first external device from a first state to a second state, according to a determination that one or more sets of gaze criteria have been met for the first external device using the first gaze information.

[0015] According to several embodiments, an electronic device is provided. The electronic device comprises one or more processors and a memory for storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for receiving a voice user input request to execute a first command while a first external device is in a first state, instructions for acquiring first gaze information using one or more camera sensors, and instructions for transmitting instructions, based on a first command, for transitioning the first external device from a first state to a second state, according to a determination that one or more sets of gaze criteria have been met for the first external device using the first gaze information.

[0016] According to several embodiments, an electronic device is provided. The electronic device includes means for receiving a voice user input request for executing a first command while a first external device is in a first state, and for acquiring first gaze information using one or more camera sensors, and means for transmitting a command to transition the first external device from a first state to a second state based on a first command, according to a determination that one or more sets of gaze criteria are met for the first external device using the first gaze information.

[0017] According to several embodiments, a method is provided. The method is performed on an electronic device. The method includes receiving a voice user input request; updating the value of an indicator characteristic to a first value corresponding to the first user, according to a determination that the voice user input request corresponds to a first user; updating the value of an indicator characteristic to a second value corresponding to the second user, different from the first value, according to a determination that the voice user input request corresponds to a second user different from the first user; and responding to the voice user input request using an indicator that includes the updated value of the characteristic.

[0018] According to some embodiments, a non-temporary computer-readable storage medium is provided. The medium stores one or more programs configured to be executed by one or more processors of an electronic device. The one or more programs include: an instruction to receive a voice user input request; an instruction to update the value of an indicator characteristic to a first value corresponding to the first user, according to a determination that the voice user input request corresponds to a first user; an instruction to update the value of an indicator characteristic to a second value different from the first value, corresponding to the second user, according to a determination that the voice user input request corresponds to a second user different from the first user; and an instruction to respond to the voice user input request using an indicator containing the updated value of the characteristic.

[0019] According to some embodiments, a temporary computer-readable storage medium is provided. The medium stores one or more programs configured to be executed by one or more processors of an electronic device. The one or more programs include: an instruction to receive a voice user input request; an instruction to update the value of an indicator characteristic to a first value corresponding to the first user, according to a determination that the voice user input request corresponds to a first user; an instruction to update the value of an indicator characteristic to a second value different from the first value, corresponding to the second user, according to a determination that the voice user input request corresponds to a second user different from the first user; and an instruction to respond to the voice user input request using an indicator containing the updated value of the characteristic.

[0020] According to several embodiments, an electronic device is provided. The electronic device comprises one or more processors and a memory for storing one or more programs configured to be executed by the one or more processors, the one or more programs including: an instruction for receiving a voice user input request; an instruction for updating the value of an indicator characteristic to a first value corresponding to the first user, according to a determination that the voice user input request corresponds to a first user; an instruction for updating the value of an indicator characteristic to a second value different from the first value, corresponding to the second user, according to a determination that the voice user input request corresponds to a second user different from the first user; and an instruction for responding to a voice user input request using an indicator including the updated value of the characteristic.

[0021] According to several embodiments, an electronic device is provided. The electronic device includes means for receiving a voice user input request; means for updating the value of an indicator characteristic to a first value corresponding to the first user, according to a determination that the voice user input request corresponds to a first user; means for updating the value of an indicator characteristic to a second value different from the first value, according to a determination that the voice user input request corresponds to a second user different from the first user; and means for responding to a voice user input request using an indicator that includes the updated value of the characteristic.

[0022] The executable instructions that perform these functions are optionally contained within a non-temporary computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0023] For this reason, the device has a faster and more efficient method and interface for controlling an electronic device, thereby increasing the effectiveness, efficiency, and user satisfaction of such a device. Such a method and interface can complement or replace other methods for controlling an electronic device.

Brief Description of the Drawings

[0024] To better understand the various embodiments described below, reference should be made to the following "Modes for Carrying Out the Invention" in conjunction with the following drawings, and like reference numerals refer to corresponding parts throughout the following figures.

[0025] [Figure 1A] FIG. 1 is a block diagram showing a portable multifunctional device having a touch-sensitive display according to some embodiments.

[0026] [Figure 1B] FIG. 2 is a block diagram showing exemplary components for event processing according to some embodiments.

[0027] [Figure 1C] FIG. 3 is a block diagram showing a system and environment for implementing a digital assistant according to some embodiments.

[0028] [Figure 2] FIG. 4 is a diagram showing a portable multifunctional device having a touch screen according to some embodiments.

[0029] [Figure 3] FIG. 5 is a block diagram showing an exemplary multifunctional device having a display and a touch-sensitive surface according to some embodiments.

[0030] [Figure 4A] FIG. 6 is a diagram showing an exemplary user interface of a menu of an application on a portable multifunctional device according to some embodiments.

[0031] [Figure 4B] This figure shows an exemplary user interface for a multifunctional device having a touch-sensitive surface separate from the display, according to several embodiments.

[0032] [Figure 5A] This figure shows a personal electronic device according to several embodiments.

[0033] [Figure 5B] This is a block diagram showing a personal electronic device according to several embodiments.

[0034] [Figure 6A] This demonstrates an exemplary technique for activating a digital assistant using gaze information. [Figure 6B] This demonstrates an exemplary technique for activating a digital assistant using gaze information. [Figure 6C] This demonstrates an exemplary technique for activating a digital assistant using gaze information. [Figure 6D] This demonstrates an exemplary technique for activating a digital assistant using gaze information. [Figure 6E] This demonstrates an exemplary technique for activating a digital assistant using gaze information. [Figure 6F] This demonstrates an exemplary technique for activating a digital assistant using gaze information. [Figure 6G] This demonstrates an exemplary technique for activating a digital assistant using gaze information. [Figure 6H] This demonstrates an exemplary technique for activating a digital assistant using gaze information. [Figure 6I] This demonstrates an exemplary technique for activating a digital assistant using gaze information. [Figure 6J]This demonstrates an exemplary technique for activating a digital assistant using gaze information. [Figure 6K] This demonstrates an exemplary technique for activating a digital assistant using gaze information. [Figure 6L] This demonstrates an exemplary technique for activating a digital assistant using gaze information. [Figure 6M] This demonstrates an exemplary technique for activating a digital assistant using gaze information. [Figure 6N] This demonstrates an exemplary technique for activating a digital assistant using gaze information. [Figure 6O] This demonstrates an exemplary technique for activating a digital assistant using gaze information.

[0035] [Figure 7A] This flowchart shows how to activate a digital assistant using gaze information. [Figure 7B] This flowchart shows how to activate a digital assistant using gaze information.

[0036] [Figure 8A] This demonstrates an exemplary technique of using gaze information to provide context to commands. [Figure 8B] This demonstrates an exemplary technique of using gaze information to provide context to commands. [Figure 8C] This demonstrates an exemplary technique of using gaze information to provide context to commands. [Figure 8D] This demonstrates an exemplary technique of using gaze information to provide context to commands. [Figure 8E] This demonstrates an exemplary technique of using gaze information to provide context to commands. [Figure 8F] This demonstrates an exemplary technique of using gaze information to provide context to commands. [Figure 8G] This demonstrates an exemplary technique of using gaze information to provide context to commands. [Figure 8H] This demonstrates an exemplary technique of using gaze information to provide context to commands. [Figure 8I] This demonstrates an exemplary technique of using gaze information to provide context to commands. [Figure 8J] This demonstrates an exemplary technique of using gaze information to provide context to commands. [Figure 8K] This demonstrates an exemplary technique of using gaze information to provide context to commands. [Figure 8L] This demonstrates an exemplary technique of using gaze information to provide context to commands.

[0037] [Figure 9A] This flowchart illustrates how gaze information can be used to provide context to commands. [Figure 9B] This flowchart illustrates how gaze information can be used to provide context to commands.

[0038] [Figure 10A] This section presents an exemplary technique for providing instructions to distinguish between different speakers. [Figure 10B] This section presents an exemplary technique for providing instructions to distinguish between different speakers. [Figure 10C] This section presents an exemplary technique for providing instructions to distinguish between different speakers. [Figure 10D] This section presents an exemplary technique for providing instructions to distinguish between different speakers.

[0039] [Figure 11A] This is a flowchart illustrating a method for providing instructions to distinguish between different speakers. [Figure 11B] This is a flowchart illustrating a method for providing instructions to distinguish between different speakers. [Modes for carrying out the invention]

[0040] The following description includes exemplary methods, parameters, etc. However, it should be noted that such descriptions are not intended to limit the scope of this disclosure, but rather are provided to describe exemplary embodiments.

[0041] There is a need for electronic devices that provide efficient methods and interfaces for controlling them. For example, by eliminating or reducing the need for the user to provide tactile or linguistic input to activate a digital assistant, the user can control the electronic device more effectively. In another embodiment, by identifying an external device through the user's gaze, the need to provide context and complex, time-consuming user input to identify such a device is eliminated or reduced. Such techniques can reduce the cognitive burden on the user controlling the electronic device, thereby increasing productivity. Furthermore, such techniques can reduce the power of the processor and battery that would normally be wasted on redundant user input.

[0042] Figures 1A-1C, 2, 3, 4A-4B, and 5A-5B below provide a description of exemplary devices for performing techniques for controlling electronic devices. Figures 6A-6O show exemplary user interfaces for activating a digital assistant using gaze information. Figures 7A-7B are flowcharts illustrating how to activate a digital assistant using gaze information according to several embodiments. The user interfaces in Figures 6A-6O are used to illustrate a later process, including the process in Figures 7A-7B. Figures 8A-8L show exemplary user interfaces for providing context to a command using gaze information. Figures 9A-9B are flowcharts illustrating how to provide context to a command using gaze information according to several embodiments. The user interfaces in Figures 8A-8L are used to illustrate a later process, including the process in Figures 9A-9B. Figures 10A-10D show exemplary user interfaces for providing instructions that distinguish between different utterances. Figures 11A to 11B are flowcharts illustrating methods for providing instructions that distinguish between different utterances, according to several embodiments. The user interfaces in Figures 10A to 10D are used to illustrate a process described later, which includes the process in Figures 11A to 11B.

[0043] In the following description, terms such as “first,” “second,” etc., are used to describe various elements, but these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, without departing from the scope of the various embodiments described, the first touch may be called the second touch, and similarly, the second touch may be called the first touch. Both the first touch and the second touch are touches, but they are not the same touch.

[0044] The terminology used in the descriptions of the various embodiments described herein is intended solely to describe specific embodiments and is not intended to be limiting. When used in the descriptions of the various embodiments and in the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless otherwise clearly indicated in the context. Furthermore, when used herein, the term “and / or” should be understood to refer to and include any and all possible combinations of one or more of the enumerated items relating to the description. When used herein, the terms “includes,” “comprises,” and / or “comprising” specify the presence of the described features, integers, steps, actions, elements, and / or components, but it will be further understood that they do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof.

[0045] The phrase "if" can be interpreted, at will, depending on the context, as "when" or "upon," or "in response to determining" or "in response to detecting." Similarly, the phrases "if it is determined" or "if [a stated condition or event] is detected" can be interpreted, at will, depending on the context, as "upon determining" or "in response to determining," or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]."

[0046] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, which also includes other functions such as PDA functionality and / or music player functionality. Exemplary embodiments of portable multifunction devices include, but are not limited to, the iPhone®, iPod Touch®, and iPad® devices from Apple Inc., Cupertino, California. Optionally, other portable electronic devices such as laptops or tablet computers having touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads) are also used. It should also be understood that in some embodiments, the device is not a portable communication device but a desktop computer having touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0047] The following discussion describes electronic devices including displays and touch-sensitive surfaces. However, it should be understood that electronic devices optionally include one or more other physical user interface devices such as physical keyboards, mice, and / or joysticks.

[0048] The device typically supports a variety of applications, including drawing applications, presentation applications, word processing applications, website creation applications, disk authoring applications, spreadsheet applications, game applications, telephone applications, video conferencing applications, email applications, instant messaging applications, training support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0049] Various applications running on this device optionally utilize at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface, as well as the corresponding information displayed on the device, are optionally adjusted and / or modified on an application-by-application basis and / or within each application. In this way, the device's common physical architecture (such as the touch-sensitive surface) optionally supports a variety of applications, including user interfaces that are intuitive and transparent to the user.

[0050] Here, we turn our attention to embodiments of portable devices having a touch-sensitive display. Figure 1A is a block diagram of a portable multifunction device 100 having a touch-sensitive display system 112 according to several embodiments. The touch-sensitive display 112 may be conveniently referred to as a “touchscreen” and may be known or referred to as a “touch-sensitive display system”. Device 100 includes a memory 102 (optionally including one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and an external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more contact intensity sensors 165 (e.g., touch-sensitive surfaces such as the touch-sensitive display system 112 of Device 100) that detect the intensity of contact on Device 100. Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on Device 100 (for example, generating tactile output on a touch-sensitive surface such as the touch-sensitive display system 112 of Device 100 or the touchpad 355 of Device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0051] As used herein and in the claims, the term “strength” of contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of contact on the touch-sensitive surface (e.g., finger contact), or a proxy for the force or pressure of contact on the touch-sensitive surface. The strength of contact has a range of values, including at least four distinct values, and more typically, including several hundred (e.g., at least 256) distinct values. The strength of contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure forces at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted averaged) to determine an estimated force of contact. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size and / or change in the contact area detected on the touch-sensing surface, the capacitance and / or change in the touch-sensing surface adjacent to the contact, and / or the resistance and / or change in the touch-sensing surface adjacent to the contact may optionally be used as a substitute for the force or pressure of the contact on the touch-sensing surface. In some implementations, the substitute measurement of the contact force or pressure is used directly to determine whether an intensity threshold is exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some implementations, the substitute measurement of the contact force or pressure is converted into an estimate of the force or pressure, and this estimate is used to determine whether an intensity threshold is exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). By using the intensity of contact as an attribute of user input, it becomes possible for users to access additional device functions that may otherwise be inaccessible on reduced-size devices with limited mounting area, to display affordances (e.g., on a touch-sensitive display), and / or to receive user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical control unit such as a knob or button).

[0052] As used herein and in the claims, the term “tactile output” means a physical displacement of the device relative to its previous position, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by the user through the user’s sense of touch. For example, in a situation where the device or a component of the device is in contact with a touch-sensitive surface of the user (e.g., the user’s fingers, palm, or other part of their hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical properties of the device or a component of the device. For example, the movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) may be optionally interpreted by the user as a “down-click” or “up-click” of a physical actuator button. In some cases, the user may feel a tactile sensation such as a “down-click” or “up-click” even when there is no movement of a physical actuator button associated with a touch-sensitive surface that is physically pressed (e.g., displaced) by the user’s action. As another example, movement of a touch-sensitive surface may be interpreted or perceived by the user as "roughness" of that surface, even if there is no change in the smoothness of the touch-sensitive surface. Such user interpretations of touch depend on the user's personal sensory perception, but there are many touch sensory perceptions common to the majority of users. Therefore, when a tactile output is described as corresponding to a user's specific sensory perception (e.g., "up-click," "down-click," "roughness"), unless otherwise stated, the generated tactile output corresponds to the physical displacement of the device or its components that produce the described sensory perception for a typical (or average) user.

[0053] It should be understood that device 100 is merely an example of a portable multifunction device, and that device 100 may optionally have more or fewer components than those shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of those components. The various components shown in Figure 1A are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing integrated circuits and / or application-specific integrated circuits.

[0054] Memory 102 optionally includes high-speed random-access memory and optionally also includes non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls access to memory 102 by other components of device 100.

[0055] The peripheral interface 118 can be used to connect the device's input and output peripherals to the CPU 120 and memory 102. One or more processors 120 operate or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data. In some embodiments, the peripheral interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0056] The RF (radio frequency) circuit 108 transmits and receives RF signals, also known as electromagnetic signals. The RF circuit 108 converts electrical signals to electromagnetic signals, or electromagnetic signals to electrical signals, and communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 108 optionally includes well-known circuits for performing these functions, which include, but are not limited to, antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, CODEC chipsets, subscriber identity module (SIM) cards, and memory. The RF circuit 108 optionally communicates wirelessly with networks such as the Internet, also known as the World Wide Web (WWW), intranets, and / or wireless networks such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs), as well as with other devices. The RF circuit 108 optionally includes a well-known circuit for detecting a near-field communication (NFC) field using a short-range communication radio. Wireless communication is not limited to this, but optionally includes Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), and long-term evolution.Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, Email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), Instant messaging (e.g., Extensible Messaging and Presence Protocol) Using any of several communication standards, protocols, and technologies, including the XMPP protocol, the Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), the Instant Messaging and Presence Service (IMPS), and / or the Short Message Service (SMS), or any other suitable communication protocol, including a communication protocol not yet developed as of the filing date of this specification.

[0057] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral interface 118, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 111. The speaker 111 converts the electrical signal into human audible sound waves. The audio circuit 110 also receives the electrical signal converted from the sound waves by the microphone 113. The audio circuit 110 converts the electrical signal into audio data and transmits the audio data to the peripheral interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or RF circuit 108 by the peripheral interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., 212 in Figure 2). The headset jack provides an interface between the audio circuit 110 and detachable audio input / output peripherals such as output-only headphones or headsets that have both output (e.g., headphones for one or both ears) and input (e.g., a microphone).

[0058] The I / O subsystem 106 connects input / output peripherals on device 100, such as the touchscreen 112 and other input control devices 116, to the peripheral interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, a depth camera controller 169, an intensity sensor controller 159, a haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. One or more input controllers 160 receive electrical signals from / transmit electrical signals to other input control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, the input controllers 160 are optionally connected to (or not connected to) one of the following: a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. One or more buttons (e.g., 208 in Figure 2) optionally include up / down buttons for volume control of speaker 111 and / or microphone 113. One or more buttons optionally include push buttons (e.g., 206 in Figure 2).

[0059] As described in U.S. Patent Application No. 11 / 322,549, “Unlocking a Device by Performing Gestures on an Unlock Image,” filed December 23, 2005, U.S. Patent No. 7,657,849, which is incorporated herein by reference in its entirety, a quick press of a push button optionally unlocks the touchscreen 112, or optionally initiates a process to unlock the device using gestures on the touchscreen. A longer press of a push button (e.g., 206) optionally turns power on or off the device 100. The functionality of one or more of the buttons is optionally customizable by the user. The touchscreen 112 is used to implement virtual or soft buttons and one or more soft keyboards.

[0060] The touch-sensitive display 112 provides input and output interfaces between the device and the user. The display controller 156 receives electrical signals from and / or transmits electrical signals to the touchscreen 112. The touchscreen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively, “graphics”). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0061] The touchscreen 112 has a touch-sensing surface, sensor, or set of sensors that accept user input based on touch and / or tactile contact. The touchscreen 112 and the display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of contact) on the touchscreen 112 and translate the detected contact into interaction with user interface objects displayed on the touchscreen 112 (e.g., one or more soft keys, icons, web pages, or images). In an exemplary embodiment, the point of contact between the touchscreen 112 and the user corresponds to the user's finger.

[0062] The touchscreen 112 optionally uses LCD (liquid crystal display) technology, LPD (polymer light-emitting display) technology, or LED (light-emitting diode) technology, but other display technologies may also be used in other embodiments. The touchscreen 112 and the display controller 156 optionally, but not limited to, use any of several currently known or future-developed touch sensing technologies, including capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements that determine one or more points of contact with the touchscreen 112, to detect contact and any movement or interruption thereof. In exemplary embodiments, projected mutual capacitive sensing technology is used, such as that found in the iPhone® and iPod Touch® from Apple Inc. of Cupertino, California.

[0063] The touch-sensitive displays in some embodiments of the touchscreen 112 are optionally similar to the multi-touch-sensitive touchpads described in U.S. Patent No. 6,323,846 (Westerman et al.), No. 6,570,557 (Westerman et al.), and / or No. 6,677,932 (Westerman), and / or U.S. Patent Publication No. 2002 / 0015024(A1), which are each incorporated herein in whole by reference. However, the touchscreen 112 displays visual output from device 100, whereas the touch-sensitive touchpad does not provide visual output.

[0064] Touch-sensitive displays in some embodiments of the touchscreen 112 are described in the following application. (1) U.S. Patent Application No. 11 / 381,313, filed May 2, 2006, "Multipoint Touch Surface Controller", (2) U.S. Patent Application No. 10 / 840,862, filed May 6, 2004, "Multipoint Touchscreen", (3) U.S. Patent Application No. 10 / 903,964, filed July 30, 2004, "Gestures For Touch Sensitive Input Devices", (4) U.S. Patent Application No. 11 / 048,264, filed January 31, 2005, "Gestures For Touch Sensitive Input Devices", (5) U.S. Patent Application No. 11 / 038,590, filed January 18, 2005, "Mode-Based Graphical User Interfaces For Touch Sensitive Input (6) U.S. Patent Application No. 11 / 228,758, filed September 16, 2005, "Virtual Input Device Placement On A Touch Screen User Interface", (7) U.S. Patent Application No. 11 / 228,700, filed September 16, 2005, "Operation Of A Computer With A Touch Screen Interface", (8) U.S. Patent Application No. 11 / 228,737, filed September 16, 2005, "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard", and (9) U.S. Patent Application No. 11 / 367,749, filed March 3, 2006, "Multi-Functional Hand-Held Device". All of these applications are incorporated herein by reference in their entirety.

[0065] The touchscreen 112 optionally has a video resolution greater than 100 dpi. In some embodiments, the touchscreen has a video resolution of approximately 160 dpi. The user optionally touches the touchscreen 112 using any suitable object or attachment such as a stylus or finger. In some embodiments, the user interface is designed to operate primarily using finger-based touch and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​the finger on the touchscreen. In some embodiments, the device translates coarse finger input into a precise pointer / cursor position or command to perform an action desired by the user.

[0066] In some embodiments, in addition to the touchscreen, the device 100 optionally includes a touchpad for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensing area of ​​the device that, unlike the touchscreen, does not display a visual output. The touchpad is optionally a touch-sensing surface separate from the touchscreen 112 or an extension of the touch-sensing surface formed by the touchscreen.

[0067] Device 100 also includes a power system 162 that supplies power to various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., a battery, alternating current (AC)), a recharge system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)), and any other components associated with the generation, management, and distribution of power within the portable device.

[0068] The device 100 also optionally includes one or more optical sensors 164. Figure 1A shows optical sensors coupled to an optical sensor controller 158 in the I / O subsystem 106. The optical sensors 164 optionally include a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. The optical sensors 164 receive light from the environment projected through one or more lenses and convert that light into data representing an image. The optical sensors 164 work in conjunction with an imaging module 143 (also called a camera module) to optionally capture still images or video. In some embodiments, the optical sensors are located on the back of the device 100, opposite the touchscreen display 112 which is on the front of the device, so that the touchscreen display can be used as a viewfinder for acquiring still images and / or video. In some embodiments, the optical sensors are located on the front of the device so that the user's image is optionally acquired for video conferencing while the user is viewing other video conference participants on the touchscreen display. In some embodiments, the position of the light sensor 164 can be changed by the user (for example, by rotating the lens and sensor within the device housing), so that a single light sensor 164 can be used for both video conferencing and acquiring still images and / or videos, together with the touchscreen display.

[0069] Device 100 optionally also includes one or more depth camera sensors 175. Figure 1A shows a depth camera sensor coupled to a depth camera controller 169 in the I / O subsystem 106. The depth camera sensor 175 receives data from the environment to create a three-dimensional model of an object in the scene (e.g., a face) from a viewpoint (e.g., the depth camera sensor). In some embodiments, in conjunction with an imaging module 143 (also called a camera module), the depth camera sensor 175 is optionally used to determine depth maps of various parts of an image captured by the imaging module 143. In some embodiments, the depth camera sensor is positioned on the front of Device 100 to optionally acquire an image of the user with depth information for video conferencing while the user views other video conference participants on a touchscreen display, and also to capture a selfie image with depth map data. In some embodiments, the depth camera sensor 175 is positioned on the back of the device, or on both the back and front of Device 100. In some embodiments, the position of the depth camera sensor 175 can be changed by the user (for example, by rotating the lens and sensor within the device housing), so that the depth camera sensor 175, together with the touchscreen display, can be used for both video conferencing and the acquisition of still images and / or videos.

[0070] In some embodiments, the depth map (e.g., depth map image) includes information (e.g., values) about the distance of objects in the scene from a viewpoint (e.g., camera, light sensor, depth camera sensor). In one embodiment of the depth map, each depth pixel defines the position on the Z-axis of the viewpoint where its corresponding 2D pixel is located. In some embodiments, the depth map consists of pixels, each defined by a value (e.g., 0 to 255). For example, a value of "0" represents the furthest pixel in the "3D" scene, and a value of "255" represents the closest pixel to the viewpoint (e.g., camera, light sensor, depth camera sensor) in that "3D" scene. In other embodiments, the depth map represents the distance between objects in the scene and the plane of the viewpoint. In some embodiments, the depth map includes information about the relative depth of various features of the object of interest as seen from the depth camera (e.g., the relative depth of the eyes, nose, mouth, and ears on the user's face). In some embodiments, the depth map includes information that enables the device to determine the contour of the object of interest in the z-direction.

[0071] Device 100 also optionally includes one or more contact intensity sensors 165. Figure 1A shows a contact intensity sensor coupled to an intensity sensor controller 159 in the I / O subsystem 106. The contact intensity sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electric force sensors, pressure-power sensors, optical force sensors, capacitive touch-sensing surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of contact on a touch-sensing surface). The contact intensity sensor 165 receives contact intensity information (e.g., pressure information, or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is positioned juxtaposed with or adjacent to a touch-sensing surface (e.g., a touch-sensing display system 112). In some embodiments, at least one contact intensity sensor is positioned on the back of Device 100, opposite the touchscreen display 112 located on the front of Device 100.

[0072] Device 100 optionally also includes one or more proximity sensors 166. Figure 1A shows a proximity sensor 166 coupled to a peripheral interface 118. Alternatively, the proximity sensor 166 is optionally coupled to an input controller 160 in an I / O subsystem 106. The proximity sensor 166 optionally functions as described in U.S. Patent Applications 11 / 241,839, “Proximity Detector In Handheld Device”, 11 / 240,788, “Proximity Detector In Handheld Device”, 11 / 620,702, “Using Ambient Light Sensor To Augment Proximity Sensor Output”, 11 / 586,862, “Automated Response To And Sensing Of User Activity In Portable Devices”, and 11 / 638,251, “Methods And Systems For Automatic Configuration Of Peripherals”, which are all incorporated herein by reference. In some embodiments, if the multifunction device is placed near the user's ear (for example, when the user is making a phone call), the proximity sensor turns off and disables the touchscreen 112.

[0073] Device 100 also optionally includes one or more tactile output generators 167. Figure 1A shows a tactile output generator coupled to a tactile feedback controller 161 in the I / O subsystem 106. The tactile output generator 167 optionally includes one or more electroacoustic devices such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components that convert electrical signals into tactile output on the device). The contact intensity sensor 165 receives a tactile feedback generation command from the tactile feedback module 133 and generates a tactile output on device 100 that can be sensed by the user of device 100. In some embodiments, at least one tactile output generator is positioned alongside or adjacent to a touch-sensing surface (e.g., a touch-sensing display system 112) and optionally generates a tactile output by moving the touch-sensing surface vertically (e.g., inward / outward from the surface of device 100) or horizontally (e.g., back and forth in the same plane as the surface of device 100). In some embodiments, at least one tactile output generator sensor is positioned on the back of device 100, opposite the touchscreen display 112 which is positioned on the front of device 100.

[0074] The device 100 also optionally includes one or more accelerometers 168. Figure 1A shows an accelerometer 168 coupled to a peripheral interface 118. Alternatively, the accelerometer 168 is optionally coupled to an input controller 160 in the I / O subsystem 106. The accelerometer 168 optionally functions as described in U.S. Patent Publication No. 20050190059, “Acceleration-based Theft Detection System for Portable Electronic Devices,” and U.S. Patent Publication No. 20060017692, “Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer,” both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed on a touchscreen display in portrait or landscape orientation based on an analysis of data received from one or more accelerometers. In addition to the accelerometer 168, device 100 optionally includes a magnetometer and a GPS (or GLONASS or other global navigation system) receiver for acquiring information about the location and orientation of device 100 (e.g., longitudinal or transverse).

[0075] In some embodiments, the software components stored in memory 102 include an operating system 126, a communications module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application (or instruction set) 136. Furthermore, in some embodiments, memory 102 (Figure 1A) or 370 (Figure 3) stores a device / global internal state 157, as shown in Figures 1A and 3. The device / global internal state 157 includes one or more of the following: an active application state indicating which application is active, if there is an application currently active; a display state indicating which applications, views, or other information occupy different areas of the touchscreen display 112; a sensor state including information obtained from various sensors and input control devices 116 of the device; and location information relating to the device's position and / or orientation.

[0076] An operating system 126 (for example, an embedded operating system such as Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or VxWorks) includes various software components and / or drivers that control and manage general system tasks (e.g., memory management, storage control, power management, etc.) and facilitate communication between various hardware components and software components.

[0077] The communication module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted for direct connection with other devices or indirect connection via a network (e.g., the Internet, Wi-Fi, etc.). In some embodiments, the external ports are multi-pin (e.g., 30-pin) connectors that are the same as and / or adapted to the 30-pin connector used on iPod® (a trademark of Apple Inc.) devices.

[0078] The contact / motion module 130 optionally detects contact with the touchscreen 112 and other touch-sensitive devices (e.g., a touchpad or physical click wheel) (in cooperation with the display controller 156). The contact / motion module 130 includes various software components for performing various contact detection operations, such as determining whether contact has occurred (e.g., detecting a finger down event), determining the intensity of the contact (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of contact and tracking movement across the touch-sensitive surface (e.g., detecting one or more events of a finger dragging), and determining whether contact has stopped (e.g., detecting a finger up event or interruption of contact). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point, represented by a series of contact data, optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These actions can be optionally applied to a single contact (e.g., a single finger contact) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contacts). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.

[0079] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by a user (for example, to determine whether a user has "clicked" on an icon). In some embodiments, at least one subset of the intensity thresholds is determined according to a software parameter (for example, the intensity thresholds can be adjusted without modifying the physical hardware of device 100, rather than being determined by the activation threshold of a particular physical actuator). For example, the mouse "click" threshold for a trackpad or touchscreen display can be set to one of a range of default thresholds without modifying the trackpad or touchscreen display hardware. In addition, in some implementations, the user of the device is provided with software settings to adjust one or more of the set of intensity thresholds (for example, by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once using a system-level click "intensity" parameter).

[0080] The contact / motion module 130 optionally detects gesture input from the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different movement, timing, and / or intensity of the detected contact). Therefore, gestures are optionally detected by detecting a specific contact pattern. For example, detecting a finger tap gesture involves detecting a finger down event, followed by a finger up (lift-off) event at the same location (or substantially the same location) as the finger down event (e.g., the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface involves detecting a finger down event, followed by one or more finger drag events, and then a finger up (lift-off) event.

[0081] The graphics module 132 includes various known software components for rendering and displaying graphics on the touchscreen 112 or other display, including components for changing the visual effects of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual properties). In this specification, the term “graphics” includes, but is not limited to, any object that can be displayed to the user, including characters, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0082] In some embodiments, the graphics module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from an application or the like that specify a graphic to be displayed, along with coordinate data and other graphic characteristic data as needed, and then generates screen image data to output to the display controller 156.

[0083] The haptic feedback module 133 includes various software components for generating commands used by the haptic output generator 167, which generates haptic outputs at one or more locations on the device 100 in response to the user's interaction with the device 100.

[0084] The text input module 134 is optionally a component of the graphics module 132 and provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other applications that require text input).

[0085] The GPS module 135 determines the location of the device and provides this information for use in various applications (for example, to the telephone 138 for use in location-based dialing, to the camera 143 as metadata for photos / videos, and to applications that provide location-based services such as weather widgets, local occupational phonebook widgets, and map / navigation widgets).

[0086] Application 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof: ● Contact module 137 (sometimes called the address book or contact list), ●Telephone module 138, ●Video conferencing module 139, ● Email client module 140, ● Instant messaging (IM) module 141, ●Training support module 142, ● Camera module 143 for still images and / or video, ●Image management module 144, ●Video player module, ● Music player module, ● Browser module 147, ● Calendar module 148, ●Optionally, a widget module 149 may include one or more of the following: weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets obtained by the user, as well as user-created widgets 149-6. ●Widget creator module 150 for creating user-created widget 149-6, ●Search module 151, ●Video and music player module 152, which integrates a video player module and a music player module. ●Memo Module 153, ●Map module 154, and / or, ● Online video module 155.

[0087] Examples of other applications 136 that may be optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, Java-enabled applications, encryption, digital rights management, speech recognition, and speech duplication.

[0088] The contact module 137 works in conjunction with the touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134 to optionally manage an address book or contact list (for example, by storing it in the application internal state 192 of the contact module 137 in memory 102 or memory 370). Management by the contact module 137 includes: This includes adding names to an address book, removing names (single or multiple) from an address book, associating names with telephone numbers (single or multiple), email addresses (single or multiple), physical addresses (single or multiple), or other information, associating images with names, categorizing and sorting names, and providing telephone numbers or email addresses to initiate and / or facilitate communication by telephone 138, video conferencing module 139, email 140, or IM 141.

[0089] The telephone module 138 works in conjunction with the RF circuit 108, audio circuit 110, speaker 111, microphone 113, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134 to optionally input character sequences corresponding to telephone numbers, access one or more telephone numbers in the contact module 137, modify entered telephone numbers, dial each telephone number, make a call, and disconnect and terminate a call at the end of the call. As previously mentioned, wireless communication may optionally use any of several communication standards, protocols, and technologies.

[0090] The video conferencing module 139 works in conjunction with an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, a touchscreen 112, a display controller 156, a light sensor 164, a light sensor controller 158, a contact / motion module 130, a graphics module 132, a text input module 134, a contact module 137, and a telephone module 138 to include executable commands for starting, running, and ending video conferences between the user and one or more other participants, according to the user's instructions.

[0091] The email client module 140, in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, includes executable commands for creating, sending, receiving, and managing emails in response to user instructions. The email client module 140 also works with the image management module 144 to greatly facilitate the creation and sending of emails containing still images or videos captured by the camera module 143.

[0092] The instant messaging module 141, in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, includes executable commands for inputting character sequences corresponding to instant messages, modifying previously entered characters, sending each instant message (e.g., using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for telephone-based instant messaging, or XMPP, SIMPLE, or IMPS for internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photographs, audio files, video files, and / or other attachments, such as those supported by MMS and / or Enhanced Messaging Service (EMS). In this specification, “instant messaging” refers to both telephone-based messaging (e.g., messages sent using SMS or MMS) and internet-based messaging (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0093] The training support module 142 works in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module to create training (e.g., with time, distance, and / or calorie burn goals), communicate with training sensors (sports devices), receive training sensor data, calibrate sensors used to monitor training, select and play music for training, and includes executable commands for displaying, storing, and transmitting training data.

[0094] The camera module 143 works in conjunction with the touchscreen 112, display controller 156, light sensor 164, light sensor controller 158, contact / motion module 130, graphics module 132, and image management module 144 to include executable commands for capturing still images or videos (including video streams) and storing them in memory 102, modifying the characteristics of still images or videos, or deleting still images or videos from memory 102.

[0095] The image management module 144 works in conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143 to include executable commands for arranging, modifying (e.g., editing), or otherwise manipulating still images and / or videos, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing them.

[0096] The browser module 147, in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, includes executable commands for browsing the internet according to user instructions, including searching, linking, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.

[0097] The calendar module 148 works in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, email client module 140, and browser module 147 to include executable commands for creating, displaying, modifying, and storing calendars and data associated with the calendar (e.g., calendar items, to-do lists, etc.) according to user instructions.

[0098] The widget module 149 works in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, and browser module 147 to optionally download and use mini-applications (e.g., weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or mini-applications created by the user (e.g., user-created widget 149-6). In some embodiments, the widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! widget).

[0099] The widget creator module 150 works in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, and browser module 147 to be used by the user to optionally create widgets (for example, to turn a user-specified portion of a web page into a widget).

[0100] The search module 151 works in conjunction with the touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134 to include executable commands for searching for characters, music, sounds, images, videos, and / or other files in memory 102 that match one or more search criteria (e.g., one or more user-specified search terms) according to user instructions.

[0101] The video and music player module 152 works in conjunction with the touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, and browser module 147 to include executable commands that allow the user to download and play recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, as well as executable commands for displaying, presenting, or otherwise playing videos (for example, on the touchscreen 112 or on an external display connected via the external port 124). In some embodiments, the device 100 optionally includes the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0102] The memo module 153 works in conjunction with the touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134 to include executable commands for creating and managing memos, to-do lists, etc., according to user instructions.

[0103] The map module 154 works in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147 to optionally receive, display, modify, and store maps and map-related data (e.g., driving directions, data on shops and other points of interest in or near a specific location, and other location-based data) according to user instructions.

[0104] The online video module 155, in conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, includes instructions that enable the user to access a specific online video, browse a specific online video, receive it (e.g., by streaming and / or downloading), play it (e.g., on the touchscreen or on an external display connected via external port 124), send an email with a link to a specific online video, and perform other management of online videos in one or more file formats such as H.264. In some embodiments, an instant messaging module 141 is used instead of the email client module 140 to send a link to a specific online video. For further information regarding online video applications, please refer to U.S. Provisional Patent Application No. 60 / 936,562, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed June 20, 2007, and U.S. Patent Application No. 11 / 968,067, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed December 31, 2007, the entire contents of which are incorporated herein by reference.

[0105] Each of the modules and applications identified above corresponds to a set of executable instructions that perform one or more of the functions described above and the methods described in this application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) do not need to be implemented as separate software programs, procedures, or modules; therefore, in various embodiments, various subsets of these modules can be optionally combined or otherwise reconfigured. For example, a video player module can optionally be combined with a music player module to form a single module (e.g., the video and music player module 152 in Figure 1A). In some embodiments, memory 102 optionally stores a subset of the modules and data structures identified above. Furthermore, memory 102 optionally stores additional modules and data structures not described above.

[0106] In some embodiments, device 100 is a device in which the operation of a default set of functions in the device is performed solely via a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for device 100 to operate, the number of physical input control devices (push buttons, dials, etc.) on device 100 is optionally reduced.

[0107] A default set of functions, performed only through the touchscreen and / or touchpad, optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to the main, home, or root menu. In such embodiments, a “menu button” is implemented using the touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0108] Figure 1B is a block diagram showing exemplary components for event processing according to several embodiments. In some embodiments, memory 102 (Figure 1A) or 370 (Figure 3) includes an event sorter 170 (e.g., within the operating system 126) and each application 136-1 (e.g., any of the aforementioned applications 137-151, 155, 380-390).

[0109] The event sorter 170 receives event information and determines application 136-1 and the application view 191 of application 136-1 to which the event information is to be transmitted. The event sorter 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes an application internal state 192 that indicates the current application view displayed on the touch-sensitive display 112 when the application is active or running. In some embodiments, a device / global internal state 157 is used by the event sorter 170 to determine which application(s) are currently active, and the application internal state 192 is used by the event sorter 170 to determine the application view 191 to which the event information is to be transmitted.

[0110] In some embodiments, the application internal state 192 includes additional information such as resume information to be used when the application 136-1 resumes execution, user interface state information indicating information displayed by the application 136-1 or information ready to be displayed, a state queue that allows the user to return to a previous state or view of the application 136-1, and one or more redo / undo queues of previous actions performed by the user.

[0111] The event monitor 171 receives event information from the peripheral interface 118. The event information includes information about sub-events (for example, user touch as part of a multi-touch gesture on the touch-sensitive display 112). The peripheral interface 118 transmits information received from the I / O subsystem 106, or information received from sensors such as the proximity sensor 166, one or more accelerometers 168, and / or the microphone 113 (via the audio circuit 110). The information received by the peripheral interface 118 from the I / O subsystem 106 includes information from the touch-sensitive display 112 or the touch-sensitive surface.

[0112] In some embodiments, the event monitor 171 sends requests to the peripheral interface 118 at predetermined intervals. In response, the peripheral interface 118 transmits event information. In other embodiments, the peripheral interface 118 transmits event information only when there is a significant event (e.g., reception of input exceeding a predetermined noise threshold and / or exceeding a predetermined duration).

[0113] In some embodiments, the event sorter 170 also includes a hit view determination module 172 and / or an active event recognition determination module 173.

[0114] The hit view determination module 172 provides a software procedure for determining where in one or more views a sub-event occurred when the touch-sensitive display 112 is displaying two or more views. A view consists of a control unit and other elements that the user can see on the display.

[0115] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of each application) in which a touch is detected optionally corresponds to a program level within the application's program hierarchy or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as appropriate input is optionally determined at least in part based on the hit view of the initial touch that initiates a touch gesture.

[0116] The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views arranged in a hierarchy, the hit view determination module 172 identifies the hit view as the lowest-level view in the hierarchy from which the sub-events should be processed. In most situations, the hit view is the lowest-level view from which the initiating sub-event (e.g., the first sub-event in a sub-event sequence that forms an event or potential event) occurs. Once the hit view is identified by the hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source that was identified as the hit view.

[0117] The active event recognition determination module 173 determines which view(s) in the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognition determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognition determination module 173 determines that all views, including the physical location of the sub-event, are actively involved views, and therefore all actively involved views should receive a particular sequence of sub-events. In other embodiments, even if the touch sub-event is entirely confined to an area associated with one particular view, higher-level views in the hierarchy still remain actively involved views.

[0118] The event dispatcher module 174 sends event information to an event recognition unit (e.g., an event recognition unit 180). In embodiments including an active event recognition unit determination module 173, the event dispatcher module 174 transmits the event information to the event recognition unit determined by the active event recognition unit determination module 173. In some embodiments, the event dispatcher module 174 stores the event information acquired by each event receiving unit 182 in an event queue.

[0119] In some embodiments, the operating system 126 includes an event sorter 170. Alternatively, application 136-1 includes an event sorter 170. In yet another embodiment, the event sorter 170 is a standalone module or part of another module stored in memory 102, such as a contact / motion module 130.

[0120] In some embodiments, application 136-1 includes a plurality of event processing units 190 and one or more application views 191, each containing instructions for handling touch events occurring within each view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognition units 180. Typically, each application view 191 includes a plurality of event recognition units 180. In other embodiments, one or more of the event recognition units 180 are part of a separate module, such as a user interface kit or a higher-level object from which application 136-1 inherits methods and other attributes. In some embodiments, each event processing unit 190 includes one or more of the following: a data update unit 176, an object update unit 177, a GUI update unit 178, and / or event data 179 received from an event sorter 170. The event processing unit 190 optionally uses or calls the data update unit 176, the object update unit 177, or the GUI update unit 178 to update the application's internal state 192. Alternatively, one or more of the application views 191 include one or more event processing units 190. In some embodiments, one or more of the data update unit 176, object update unit 177, and GUI update unit 178 are included in each application view 191.

[0121] Each event recognition unit 180 receives event information (e.g., event data 179) from the event sorter 170 and identifies an event from the event information. The event recognition unit 180 includes an event receiving unit 182 and an event comparison unit 184. In some embodiments, the event recognition unit 180 also includes metadata 183 and at least a subset of event transmission commands 188 (optionally including sub-event transmission commands).

[0122] The event receiving unit 182 receives event information from the event sorter 170. The event information includes information about sub-events, such as touches or the movement of touches. Depending on the sub-event, the event information also includes additional information, such as the position of the sub-event. When the sub-event involves the movement of a touch, the event information optionally also includes the speed and direction of the sub-event. In some embodiments, an event includes the rotation of the device from one orientation to another (e.g., from portrait to landscape, or vice versa), and the event information includes corresponding information about the current orientation of the device (also called the device's orientation).

[0123] The event comparison unit 184 compares event information with a definition of a default event or sub-event, and based on the comparison, determines an event or sub-event, or determines or updates the state of an event or sub-event. In some embodiments, the event comparison unit 184 includes an event definition 186. The event definition 186 includes definitions of events (e.g., a sequence of default sub-events), such as event 1 (187-1) and event 2 (187-2). In some embodiments, sub-events within event (187) include, for example, touch start, touch end, touch movement, touch cancel, and multiple touches. In one embodiment, the definition for event 1 (187-1) is a double tap on a displayed object. A double tap includes, for example, a first touch on the displayed object for a predetermined stage (touch start), a first lift-off for the predetermined stage (touch end), a second touch on the displayed object for the predetermined stage (touch start), and a second lift-off for the predetermined stage (touch end). In another embodiment, event 2(187-2) is defined as a drag on a displayed object. The drag includes, for example, a touch (or contact) on the displayed object to a predetermined stage, movement of the touch across the touch-sensitive display 112, and lift-off of the touch (end of touch). In some embodiments, the event also includes information about one or more associated event processing units 190.

[0124] In some embodiments, the event definition 187 includes an event definition for each user interface object. In some embodiments, the event comparison unit 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view where three user interface objects are displayed on the touch-sensitive display 112, when a touch is detected on the touch-sensitive display 112, the event comparison unit 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with its respective event processing unit 190, the event comparison unit uses the results of the hit test to determine which event processing unit 190 should be activated. For example, the event comparison unit 184 selects the sub-event and the event processing unit associated with the object that triggers the hit test.

[0125] In some embodiments, the definition of each event 187 also includes a delay action that delays the transmission of event information until it is determined whether the sequence of sub-events corresponds to the event type of the event recognition unit.

[0126] If each event recognition unit 180 determines that a series of sub-events does not match any of the events in the event definition 186, each event recognition unit 180 enters an event impossible, event failed, or event terminated state and thereafter ignores subsequent sub-events of the touch-based gesture. In this situation, if there are other event recognition units that remain active for the hit view, those event recognition units continue to track and process the sub-events of the ongoing touch-based gesture.

[0127] In some embodiments, each event recognition unit 180 includes metadata 183, which has configurable properties, flags, and / or lists indicating to the actively involved event recognition units how the event transmission system should perform sub-event transmission. In some embodiments, the metadata 183 includes configurable properties, flags, and / or lists indicating how the event recognition units interact with each other or how they are made capable of interacting with each other. In some embodiments, the metadata 183 includes configurable properties, flags, and / or lists indicating whether sub-events are transmitted to various levels in the view hierarchy or program hierarchy.

[0128] In some embodiments, each event recognition unit 180 activates an event processing unit 190 associated with an event when one or more specific sub-events of an event are recognized. In some embodiments, each event recognition unit 180 transmits event information associated with the event to the event processing unit 190. Activating the event processing unit 190 is separate from sending (and delaying the transmission of) sub-events to the respective hit view. In some embodiments, the event recognition unit 180 sets a flag associated with the recognized event, and the event processing unit 190 associated with that flag catches the flag and executes a default process.

[0129] In some embodiments, the event transmission command 188 includes a sub-event transmission command that transmits event information about a sub-event without activating an event processing unit. Instead, the sub-event transmission command transmits the event information to an event processing unit associated with a set of sub-events, or to an actively involved view. The event processing unit associated with the set of sub-events or the actively involved view receives the event information and performs predetermined processing.

[0130] In some embodiments, the data update unit 176 creates and updates data used in application 136-1. For example, the data update unit 176 updates telephone numbers used in contact module 137 or stores video files used in video player module. In some embodiments, the object update unit 177 creates and updates objects used in application 136-1. For example, the object update unit 177 creates new user interface objects or updates the position of user interface objects. The GUI update unit 178 updates the GUI. For example, the GUI update unit 178 prepares display information and sends it to graphics module 132 for display on touch-sensitive display.

[0131] In some embodiments, the event processing unit(s) 190 includes or has access to a data update unit 176, an object update unit 177, and a GUI update unit 178. In some embodiments, the data update unit 176, the object update unit 177, and the GUI update unit 178 are included in a single module of their respective applications 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0132] The foregoing description regarding the handling of user touch events on a touch-sensitive display also applies to other forms of user input for operating the multifunction device 100 using input devices, but it should be understood that not all of these begin on the touchscreen. For example, mouse movement and mouse button presses, touch movements such as taps, drags, and scrolls on a touchpad, pen stylus input, device movement, verbal commands, detected eye movements, biometric input, and / or any combination thereof may be optionally used as inputs corresponding to sub-events that define the recognized event.

[0133] Figure 1C shows block diagrams of System 2100 in various embodiments. In some embodiments, System 2100 implements a digital assistant. The terms “digital assistant,” “virtual assistant,” “intelligent automated assistant,” or “automated digital assistant” refer to any information processing system that infers user intent by interpreting natural language input in utterance form and / or text form, and performs actions based on that inferred user intent. For example, in order to act based on the inferred user intent, the system identifies a task flow having steps and parameters designed to realize the inferred user intent, and performs one or more of the following: inputting specific requirements from the inferred user intent into the task flow, executing the task flow by calling a program, method, service, or API, etc., and generating an output response to the user in audible (e.g., utterance) form and / or visual form.

[0134] Specifically, a digital assistant can accept user requests, at least partially, in the form of natural language commands, requests, opinions, discourses, and / or inquiries. Typically, a user request asks the digital assistant to either provide information or perform a task. A satisfactory response to a user request includes providing the requested information, performing the requested task, or a combination of the two. For example, a user might ask the digital assistant, "Where am I right now?" Based on the user's current location, the digital assistant might respond, "You are in Central Park near the West Gate." The user might also request the performance of a task, for example, "Please invite my friends to my girlfriend's birthday party next week." In response, the digital assistant could affirm the request by saying, "Yes, right away," and then, on behalf of the user, send appropriate calendar invitations to each of the user's friends listed in the user's electronic address book. While performing a requested task, the digital assistant may interact with the user in a continuous conversation involving multiple information exchanges over an extended period. Numerous other ways exist to interact with a digital assistant to request information or to perform various tasks. In addition to providing verbal responses and taking programmed actions, digital assistants can also provide responses in other visual or auditory formats, such as text, alarms, music, videos, and animations.

[0135] As shown in Figure 1C, in some embodiments, the digital assistant is implemented according to a client-server model. The digital assistant includes a client-side portion 2102 (hereinafter, "DA client 2102") that runs on a user device 104, and a server-side portion 2106 (hereinafter, "DA server 2106") that runs on a server system 2108. The DA client 2102 communicates with the DA server 2106 through one or more networks 2110. The DA client 2102 provides client-side functions such as user-responsive input and output processing, and communication with the DA server 2106. The DA server 2106 provides server-side functions to any number of DA clients 2102, each residing on a user device 2104.

[0136] In some embodiments, the DA server 2106 includes a client-responsive I / O interface 2112, one or more processing modules 2114, data and models 2116, and an I / O interface 2118 to external services. The client-responsive I / O interface 2112 facilitates client-responsive input and output processing by the DA server 2106. One or more processing modules 2114 use the data and models 2116 to process voice input and determine user intent based on natural language input. Furthermore, one or more processing modules 2114 perform task execution based on the inferred user intent. In some embodiments, the DA server 2106 communicates with an external service 120 via a network(s) 2110 to complete a task or retrieve information. The I / O interface 2118 to external services facilitates such communication.

[0137] The user device 2104 can be any suitable electronic device. In some embodiments, the user device 2104 is a portable multifunction device (e.g., device 100 as described above with reference to Figure 1A), a multifunction device, or a portable multifunction device (e.g., devices 600, 800, 1000). A portable multifunction device is, for example, a mobile phone that also includes other functions such as a PDA and / or music player function. Specific examples of portable multifunction devices include the Apple Watch®, iPhone®, iPod Touch®, and iPad® devices by Apple Inc. (Cupertino, California). Other examples of portable multifunction devices include, but are not limited to, earphones / headphones, speakers, and laptop computers or tablet computers. Furthermore, in some embodiments, the user device 2104 is a non-portable multifunction device. Specifically, the user device 2104 is a desktop computer, a game console, a speaker, a television, or a television set-top box. In some embodiments, the user device 2104 includes a touch-sensitive surface (e.g., a touchscreen display and / or touchpad). Furthermore, the user device 2104 optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick. Various embodiments of electronic devices, such as multifunction devices, are described in further detail below.

[0138] Examples of communication networks (one or more) 2110 include local area networks (LANs) and wide area networks (WANs), such as the Internet. Communication networks (one or more) 2110 are implemented using any known network protocol, including a variety of wired or wireless protocols, such as Ethernet, Universal Serial Bus (USB), FireWire, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wi-Fi, Voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol.

[0139] The server system 2108 is implemented on one or more standalone data processing devices or on a distributed computer network. In some embodiments, the server system 2108 also employs various virtual devices and / or services from third-party service providers (e.g., third-party cloud service providers) to provide the basic computing and / or infrastructure resources of the server system 2108.

[0140] In some embodiments, user device 2104 communicates with DA server 2106 via second user device 2122, which is similar to or identical to user device 2104. User device 2104 is configured to be communicatively coupled to second user device 2122 via a direct communication connection such as Bluetooth, NFC, or BTLE, or via a wired or wireless network such as a local Wi-Fi network. In some embodiments, second user device 2122 is configured to act as a proxy between user device 2104 and DA server 2106. For example, the DA client 2102 of user device 2104 is configured to send information (e.g., user requests received by user device 2104) to DA server 2106 via second user device 2122. DA server 2106 processes the information and returns the relevant data (e.g., data content responding to the user request) to user device 2104 via second user device 2122.

[0141] In some embodiments, user device 2104 is configured to reduce the amount of information transmitted from user device 2104 by transmitting a shortened request for data to a second user device 2122. The second user device 2122 is configured to determine any supplementary information to be added to the shortened request and generate a complete request for transmission to the DA server 2106. This system architecture can favorably allow a user device 2104 with limited communication capabilities and / or limited battery power (e.g., a wristwatch or similar small electronic device) to access services provided by the DA server 2106 by using a second user device 2122 with higher communication capabilities and / or battery power (e.g., a mobile phone, laptop computer, tablet computer, etc.) as a proxy to the DA server 2106. Although only two user devices 2104 and 2122 are shown in Figure 1C, it should be understood that system 2100 includes any number and types of user devices configured to communicate with the DA server system 2106 in this proxy configuration in some embodiments.

[0142] The digital assistant shown in Figure 1C includes both a client-side portion (e.g., DA client 2102) and a server-side portion (e.g., DA server 2106), although in some examples, the digital assistant's functions are implemented as independent applications installed on the user device. Furthermore, the allocation of functions between the client and server portions of the digital assistant may vary depending on the implementation. For example, in some embodiments, the DA client is a thin client that provides only user-responsive input and output processing functions, delegating all other functions of the digital assistant to the backend server.

[0143] Figure 2 shows a portable multifunction device 100 having a touchscreen 112 according to several embodiments. The touchscreen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment, and in other embodiments described below, the user can select one or more of the graphics by performing gestures on the graphics using, for example, one or more fingers 202 (not shown in the figure to an exact scale) or one or more styluses 203 (not shown in the figure to an exact scale). In some embodiments, the selection of one or more graphics is performed when the user interrupts contact with that one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, right to left, upward and / or downward) and / or rolling (from right to left, left to right, upward and / or downward) with a finger in contact with the device 100. In some implementations or situations, accidental contact with a graphic does not result in the selection of that graphic. For example, a swipe gesture, which optionally sweeps over an application icon, will not select the corresponding application if the corresponding gesture for selection is a tap.

[0144] Device 100 also optionally includes one or more physical buttons, such as a "Home" button or a menu button 204. As previously mentioned, the menu button 204 is optionally used to navigate to any application 136 within a set of applications running on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touchscreen 112.

[0145] In some embodiments, device 100 includes a touchscreen 112, a menu button 204, a push button 206 for turning the device on / off and locking the device, volume control buttons 208, a subscriber identification module (SIM) card slot 210, a headset jack 212, and an external port 124 for docking / charging. The push button 206 is optionally used to turn the device on / off by pressing down and holding the button down for a predetermined time interval, to lock the device by pressing down and releasing the button before the predetermined time interval has elapsed, and / or to unlock the device or initiate an unlocking process. In alternative embodiments, device 100 also accepts verbal input via a microphone 113 to activate or deactivate certain functions. Device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on the touchscreen 112, and / or one or more tactile output generators 167 for generating tactile output to the user of device 100.

[0146] Figure 3 is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface according to several embodiments. The device 300 does not need to be portable. In some embodiments, the device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a game system, or a control device (e.g., a home or commercial controller). The device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 that interconnect these components. The communication buses 320 optionally include circuitry (sometimes called a chipset) that interconnects and controls communication between system components. The device 300 includes an input / output (I / O) interface 330 including a display 340, the display 340 is typically a touchscreen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 (similar to the tactile output generator 167 mentioned above with reference to Figure 1A) that generates tactile output on device 300, and a sensor 359 (e.g., light, acceleration, proximity, touch sensing, and / or a contact intensity sensor similar to the contact intensity sensor 165 mentioned above with reference to Figure 1A). The memory 370 includes high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and optionally includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located away from the CPU(s) 310.In some embodiments, memory 370 stores programs, modules, and data structures similar to, or subsets thereof, that are stored in memory 102 of the portable multifunction device 100 (Figure 1A). Furthermore, memory 370 optionally stores additional programs, modules, and data structures that are not present in memory 102 of the portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk authoring module 388, and / or a spreadsheet module 390, whereas memory 102 of the portable multifunction device 100 (Figure 1A) optionally does not store these modules.

[0147] Each of the elements identified above in Figure 3 is optionally stored in one or more of the memory devices described above. Each of the modules identified above corresponds to an instruction set that performs the function described above. The modules or programs (e.g., instruction sets) identified above do not need to be implemented as separate software programs, procedures, or modules, and therefore in various embodiments, various subsets of these modules are optionally combined or otherwise reconfigured. In some embodiments, memory 370 optionally stores subsets of the modules and data structures identified above. Furthermore, memory 370 optionally stores additional modules and data structures not described above.

[0148] Next, we optionally turn our attention to an embodiment of a user interface implemented in, for example, a portable multi-functional device 100.

[0149] Figure 4A shows an exemplary user interface for an application menu on a portable multifunction device 100 according to several embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, the user interface 400 includes the following elements, or subsets or supersets thereof. ● Signal strength indicators (single or multiple) for wireless communication (single or multiple) such as cellular signals and Wi-Fi signals 402, ●Time 404, ●Bluetooth indicator 405, ●Battery status indicator 406, ●Tray 408 containing icons for frequently used applications, as shown below. ○Optionally including an indicator 414 for the number of missed calls or voicemail messages, an icon 416 of the telephone module 138 labeled "Telephone", ○Optionally including an indicator 410 for the number of unread emails, an icon 418 of the email client module 140 labeled "Mail", ○ Icon 420 of browser module 147, labeled "Browser", and ○ Icon 422 for the video and music player module 152, also known as the iPod (trademark of Apple Inc.) module 152, which is labeled "iPod", and ● Icons of other applications, such as the following: ○ Icon 424 of IM module 141, labeled "Message", ○ Icon 426 of calendar module 148, labeled "Calendar", ○ Icon 428 of image management module 144, labeled "Photo" ○ Icon 430 of camera module 143, labeled "Camera" ○ Icon 432 of online video module 155, labeled "online video" ○ Icon 434 of stock widget 149-2, labeled "Stock Price" ○ Icon 436 of map module 154, labeled "Map" ○ Icon 438 of weather widget 149-1, labeled "Weather" ○ Icon 440 of the alarm clock widget 149-4, labeled "Clock" ○ Icon 442 of training support module 142, labeled "Training Support" ○ Icon 444 of memo module 153, labeled as "Memo", and ○ An icon 446 labeled "Settings," which provides access to the settings of device 100 and its various applications 136, for a settings application or module.

[0150] Please note that the icon labels shown in Figure 4A are for illustrative purposes only. For example, the icon 422 for the video and music player module 152 is labeled "Music" or "Music Player," and other labels are optionally used for various application icons. In some embodiments, the label for each application icon includes the name of the application corresponding to that application icon. In some embodiments, the label for a particular application icon is different from the name of the application corresponding to that particular application icon.

[0151] Figure 4B shows an exemplary user interface on a device (e.g., device 300 in Figure 3) having a touch-sensitive surface 451 (e.g., tablet or touchpad 355 in Figure 3) separate from the display 450 (e.g., touchscreen display 112). Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting the intensity of contact on the touch-sensitive surface 451, and / or one or more tactile output generators 357 for generating tactile output to the user of device 300.

[0152] Some of the following examples are given by referring to input on a touchscreen display 112 (a combination of a touch-sensing surface and a display), but in some embodiments, the device detects input on a touch-sensing surface separate from the display, as shown in Figure 4B. In some embodiments, the touch-sensing surface (e.g., 451 in Figure 4B) has a primary axis (e.g., 452 in Figure 4B) corresponding to a primary axis (e.g., 453 in Figure 4B) on the display (e.g., 450). According to these embodiments, the device detects contact with the touch-sensing surface 451 (e.g., 460 and 462 in Figure 4B) at positions corresponding to each of the positions on the display (e.g., 460 corresponds to 468 and 462 corresponds to 470 in Figure 4B). In this way, user input (e.g., touches 460 and 462, and their movements) detected by the device on a touch-sensitive surface (e.g., 451 in Figure 4B) is used by the device to operate the user interface on the display of the multifunction device (e.g., 450 in Figure 4B) when the touch-sensitive surface is separate from the display. It should be understood that a similar method may be optionally used for other user interfaces described herein.

[0153] In addition, while the following examples are given primarily with reference to finger input (e.g., finger touch, finger tap gesture, finger swipe gesture), it should be understood that in some embodiments, one or more of the finger inputs may be replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture may optionally be replaced by a mouse click (e.g., instead of touch), followed by a mouse click with cursor movement along the swipe path (e.g., instead of touch movement). As another example, a tap gesture may optionally be replaced by a mouse click (e.g., instead of touch detection and subsequent cessation of touch detection) while the cursor is located over the tap gesture position. Similarly, it should be understood that when multiple user inputs are detected simultaneously, multiple computer mice may optionally be used simultaneously, or mouse and finger touch may optionally be used simultaneously.

[0154] Figure 5A shows an exemplary personal electronic device 500. Device 500 includes a body 502. In some embodiments, device 500 may include some or all of the features described with respect to devices 100 and 300 (e.g., Figures 1A to 4B). In some embodiments, device 500 has a touch-sensitive display screen 504, hereafter referred to as touchscreen 504. Alternatively, in addition to touchscreen 504, device 500 may have a display and a touch-sensitive surface. Similar to devices 100 and 300, in some embodiments, touchscreen 504 (or touch-sensitive surface) optionally includes one or more intensity sensors that detect the intensity of the applied contact (e.g., touch). One or more intensity sensors on touchscreen 504 (or touch-sensitive surface) may provide output data representing the intensity of the touch. The user interface of device 500 may respond to touches based on their intensity, meaning that touches of different intensity may invoke different user interface behaviors on device 500.

[0155] For example techniques for detecting and processing touch intensity, see, for instance, International Patent Application PCT / US2013 / 040061, “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” filed 8 May 2013, published as International Patent WO / 2013 / 169849, and International Patent Application PCT / US2013 / 069483, “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” filed 11 November 2013, published as International Patent WO / 2014 / 105276.

[0156] In some embodiments, the device 500 has one or more input mechanisms 506 and 508. The input mechanisms 506 and 508 may be physical, if included. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, the device 500 has one or more attachment mechanisms. Such attachment mechanisms, if included, can allow the device 500 to be attached to, for example, hats, eyeglasses, earrings, necklaces, shirts, jackets, bracelets, watch bands, chains, trousers, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow the user to wear the device 500.

[0157] Figure 5B shows an exemplary personal electronic device 500. In some embodiments, the device 500 may include some or all of the components described with respect to Figures 1A, 1B, and 3. The device 500 has a bus 512 that operably connects an I / O section 514 to one or more computer processors 516 and memory 518. The I / O section 514 may be connected to a display 504, which may have a touch-sensing component 522 and optionally an intensity sensor 524 (e.g., a contact intensity sensor). In addition, the I / O section 514 may be connected to a communication unit 530 that receives application and operating system data using Wi-Fi, Bluetooth, near-field communication (NFC), cellular, and / or other wireless communication techniques. The device 500 may include input mechanisms 506 and / or 508. The input mechanism 506 may optionally be, for example, a rotatable input device or a pressable and rotatable input device. In some examples, the input mechanism 508 may optionally be a button.

[0158] In some examples, the input mechanism 508 is optionally a microphone. The personal electronic device 500 optionally includes various sensors such as a GPS sensor 532, an accelerometer 534, a direction sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or a combination thereof, all of which can be operably connected to the I / O section 514.

[0159] The memory 518 of the personal electronic device 500 may include one or more non-temporary computer-readable storage media for storing computer-executable instructions, which, when executed by one or more computer processors 516, can cause the computer processors to execute techniques described below, including processes 700, 900, and 1100 (Figures 7A-7B, 9A-9B, and 11A-11B). The computer-readable storage media may be any medium capable of tangibly containing or storing computer-executable instructions used by or in connection with an instruction execution system, apparatus, or device. In some embodiments, the storage medium is a temporary computer-readable storage medium. In some embodiments, the storage medium is a non-temporary computer-readable storage medium. The non-temporary computer-readable storage medium may include, but is not limited to, magnetic, optical, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, CDs, DVDs, or optical disks based on Blu-ray technology, as well as resident solid-state memory such as flash and solid-state drives. The personal electronic device 500 is not limited to the components and configurations shown in Figure 5B, and may include other or additional components in multiple configurations.

[0160] As used herein, the term “affordance” optionally refers to user interaction graphical user interface objects displayed on the display screens of devices 100, 300, and / or 500 (Figures 1A, 3, and 5A-5B). For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) each optionally constitute an affordance.

[0161] As used herein, the term “focus selector” refers to an input element that indicates the current part of the user interface with which the user is interacting. In some implementations, including a cursor or other position marker, the cursor acts as a “focus selector,” and therefore, when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 355 in Figure 3 or touch-sensitive surface 451 in Figure 4B) while the cursor is positioned over a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations, including a touchscreen display that enables direct interaction with user interface elements on the touchscreen display (e.g., touch-sensitive display system 112 in Figure 1A or touchscreen 112 in Figure 4A), a detected contact on the touchscreen acts as a “focus selector,” and therefore, when an input (e.g., a press input by touch) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, the particular user interface element is adjusted according to the detected input. In some implementations, focus is moved from one area of ​​the user interface to another without corresponding cursor movement or touch movement on the touchscreen display (for example, by using the tab key or arrow keys to move focus from one button to another), and in these implementations, the focus selector moves in accordance with the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is generally a user interface element (or touch on the touchscreen display) controlled by the user to communicate about the user's intended interaction with the user interface (for example, by pointing to the device the user interface element through which the user intends to interact).For example, while a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of the focus selector (e.g., a cursor, touch, or selection box) above each button indicates that the user intends to activate that button (rather than other user interface elements displayed on the device's display).

[0162] As used herein and in the claims, the term “characteristic intensity” of a contact refers to the characteristics of that contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a set of intensity samples collected over a predetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) associated with a predetermined event (e.g., after detection of contact, before detection of lift-off of contact, before or after detection of the start of movement of contact, before detection of the end of contact, before or after detection of an increase in contact intensity, and / or before or after detection of a decrease in contact intensity). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the top 10 percentile value of the contact intensity, the maximum half value of the contact intensity, the maximum 90 percent value of the contact intensity, and so on. In some embodiments, the duration of contact is used when determining characteristic intensity (for example, when characteristic intensity is the average intensity of contact over time). In some embodiments, characteristic intensity is compared to a set of one or more intensity thresholds to determine whether an action has been performed by the user. For example, the set of one or more intensity thresholds optionally includes a first intensity threshold and a second intensity threshold. In this example, contact with a characteristic intensity not exceeding the first threshold results in a first action, contact with a characteristic intensity exceeding the first intensity threshold but not exceeding the second intensity threshold results in a second action, and contact with a characteristic intensity exceeding the second threshold results in a third action. In some embodiments, the comparison between characteristic intensity and one or more thresholds is not used to determine whether a first action should be performed or a second action should be performed, but rather to determine whether one or more actions should be performed at all (for example, whether each action should be performed or whether each action should be postponed).

[0163] In some embodiments, a portion of the gesture is identified for the purpose of determining characteristic intensity. For example, the touch-sensitive surface optionally receives a series of swipe contacts that transition from a starting position to an ending position, where the intensity of contact increases. In this example, the characteristic intensity of the contact at the ending position is optionally based only on a portion of the series of swipe contacts (e.g., only the portion of the swipe contact at the ending position) rather than the entire swipe contact. In some embodiments, optionally, a smoothing algorithm is applied to the intensity of the swipe contact before determining the characteristic intensity of the contact. For example, the smoothing algorithm optionally includes one or more of the following: an unweighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some situations, these smoothing algorithms eliminate narrow spikes or drops in the swipe contact intensity for the purpose of determining characteristic intensity.

[0164] The intensity of contact on a touch-sensitive surface is optionally characterized with respect to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to the intensity at which the device performs an action typically associated with clicking a physical mouse button or trackpad. In some embodiments, the deep press intensity threshold corresponds to the intensity at which the device performs an action different from the action typically associated with clicking a physical mouse button or trackpad. In some embodiments, when a contact with a characteristic intensity below the light press intensity threshold (for example, above a nominal contact detection intensity threshold below which contact is no longer detected) is detected, the device moves the focus selector in accordance with the movement of the contact on the touch-sensitive surface without performing an action associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise specified, these intensity thresholds are consistent across various sets of user interface values.

[0165] An increase in the characteristic intensity of contact from an intensity below a light press intensity threshold to an intensity between the light press intensity threshold and the deep press intensity threshold is sometimes called a "light press" input. An increase in the characteristic intensity of contact from an intensity below a deep press intensity threshold to an intensity above a deep press intensity threshold is sometimes called a "deep press" input. An increase in the characteristic intensity of contact from an intensity below a contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light press intensity threshold is sometimes called detection of contact on the touch surface. A decrease in the characteristic intensity of contact from an intensity above a contact detection intensity threshold to an intensity below a contact detection intensity threshold is sometimes called detection of contact lift-off from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0166] In some embodiments described herein, one or more actions are performed in response to the detection of a gesture including each press input, or in response to the detection of each press input performed by each (or more) contact, and each press input is detected at least in part on the detection of an increase in the intensity of the contact (or more) above a press input intensity threshold. In some embodiments, each action is performed in response to the detection of an increase in the intensity of each contact above a press input intensity threshold (e.g., a "downstroke" of each press input). In some embodiments, a press input includes an increase in the intensity of each contact above a press input intensity threshold, followed by a decrease in the intensity of the contact below the press input intensity threshold, and each action is performed in response to the detection of a decrease in the intensity of each contact below the press input threshold (e.g., an "upstroke" of each press input).

[0167] In some embodiments, the device employs intensity hysteresis to avoid accidental inputs, which may be referred to as “jitter,” and the device defines or selects a hysteresis intensity threshold that has a predetermined relationship with a press input intensity threshold (for example, the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable percentage of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of each contact above the press input intensity threshold, followed by a decrease in the intensity of the contact below the hysteresis intensity threshold corresponding to the press input intensity threshold, and each action is performed in response to the detection of a subsequent decrease in the intensity of each contact below the hysteresis intensity threshold (for example, an “upstroke” of each press input). Similarly, in some embodiments, a press input is detected only when the device detects an increase in contact intensity from below a hysteresis intensity threshold to above a press input intensity threshold, and optionally a decrease in contact intensity to below the hysteresis intensity, and each action is performed in response to the detection of a press input (e.g., depending on the situation, an increase in contact intensity or a decrease in contact intensity).

[0168] For the sake of clarity, the description of an action performed in response to a press input associated with a press input intensity threshold, or a gesture including a press input, is optionally triggered in response to the detection of any of the following: an increase in contact intensity above the press input intensity threshold, an increase in contact intensity from below the hysteresis intensity threshold to above the press input intensity threshold, a decrease in contact intensity below the press input intensity threshold, and / or a decrease in contact intensity below the hysteresis intensity threshold corresponding to the press input intensity threshold. Furthermore, in examples where an action is described to be performed in response to the detection of a decrease in contact intensity below the press input intensity threshold, the action is optionally performed in response to the detection of a decrease in contact intensity below a hysteresis intensity threshold corresponding to and lower than the press input intensity threshold.

[0169] Next, we will focus on embodiments of user interfaces ("UI") and related processes implemented on electronic devices such as portable multifunction device 100, device 300, or device 500.

[0170] Figures 6A to 6O show exemplary user interfaces for activating and interacting with a digital assistant using glance commands and voice commands, according to several embodiments. These figures are used to illustrate the processes described later, including the processes shown in Figures 7A to 7B.

[0171] Figures 6A to 6D show that user 620 activates an exemplary digital assistant on the electronic device 600 by glancing at the electronic device 600. Once the digital assistant is activated, user 620 turns on the table lamp by uttering commands 624A to 624B to the digital assistant.

[0172] Figure 6A shows a digital assistant on electronic device 600 in an inactive state, indicated by the display 602 being off. In some embodiments, the digital assistant is inactive when electronic device 600 does not detect user gaze, i.e., that user 620 is looking at electronic device 600. In some embodiments, the digital assistant is inactive before electronic device 600 detects a button press (or other activation command) and / or before another device (e.g., a wristwatch 630) receives an activation command and sends an activation instruction to electronic device 600. User 620 can trigger the activation of the digital assistant by glancing at electronic device 600 when gaze information based on user 620's gaze satisfies one or more activation criteria. Optionally, electronic device 600 acquires gaze information for user 620's gaze using one or more camera sensors, such as camera 616B, which are mounted on an external device 616A and / or integrated with electronic device 600. The camera sensor 616A may be wirelessly or wiredly connected to the electronic device 600. One or more camera sensors include an infrared camera sensor and / or a visible light sensor that measures gaze information as soon as a gaze is detected. The gaze information optionally includes one or more of the following: the location of the user's head position, the time or duration of the gaze, the direction of the gaze, the field of view 606 of the gaze, and whether an object (e.g., an electronic device 600 or an external device such as a table lamp 618) is within the field of view 606 of the gaze.

[0173] The electronic device 600 determines whether gaze information acquired from one or more camera sensors 616A-616B satisfies one or more activation criteria. In other words, the electronic device 600 determines whether user 620 intended to look at the electronic device 600 in order to activate the digital assistant. The set of one or more activation criteria optionally includes a direction criterion that is satisfied when the user's gaze is directed toward the electronic device 600. The set of one or more activation criteria also optionally includes a resting time criterion that is satisfied when the electronic device 600 detects a resting time or duration of gaze toward the electronic device 600 that exceeds a predetermined non-zero time (e.g., several seconds). If the resting time or duration of gaze is less than the threshold time, this may indicate that the user did not intend to trigger the digital assistant. Therefore, a resting time less than the threshold time does not satisfy the activation criteria, and the digital assistant is not activated.

[0174] A set of one or more activation criteria may also optionally include a field of view criterion. As shown in Figure 6A, user 620 is not looking at the electronic device 600, and it is determined that the electronic device 600 is not within user 620's field of view 606. As will be further explained below, the field of view criterion is satisfied if it is determined that the electronic device 600 is within user 620's field of view 606, but not if it is determined that the electronic device 600 is not within user 620's field of view 600. By using the field of view 606 to determine whether user 602 is looking at the target electronic device (e.g., device 600), the gaze detection rate is increased because a gaze can be detected within the field of view when the user is not directly looking at the target electronic device but is still attempting to identify (or activate) the target electronic device.

[0175] The field of view 606 is the range of degrees above and below the user's line of sight 612, as indicated by the threshold offset angle 604. In some embodiments, the field of view criterion is met when the electronic device 600 is directly within the user 620's line of sight 612, i.e., when the determined offset degree is zero. In some embodiments, the field of view criterion is met when the determined offset angle 622 of the gaze is less than the threshold offset angle 604 (e.g., the maximum acceptable deviation from the user 620's line of sight 612 that is directly viewing the electronic device, as shown in Figure 6B). (In this example) the determined offset angle 622 and the threshold offset angle 604 are greater than zero degrees (e.g., 30 degrees). The determined offset angle 622 is calculated based on the angle formed between the line of sight 612 from the user 620's gaze (e.g., where the user is actually looking) and the calculated line of sight 608 from the user 620 to the electronic device 600 (e.g., the ideal path for the user to directly view the electronic device 600). When the offset angle 622 is greater than the threshold offset angle 604, the field of view criterion is not met because the electronic device 600 is outside the field of view 606, as shown in Figure 6A, and as a result, the digital assistant remains unactivated. In some embodiments, if the electronic device 600 is determined to be located beyond a non-zero threshold degree (e.g., beyond 30 degrees) outside the user's field of view 606, the electronic device 600 is not within the user's gaze, and the digital assistant is not activated. In some embodiments, the line of sight 612 is measured based on the user's head position. In some embodiments, the user's head position is based on the tilt or rotation of the user's head (e.g., yaw, pitch, and / or roll).

[0176] In Figure 6A, as described above, the gaze information acquired from the camera sensors (e.g., 616A, 61B) does not satisfy one or more activation criteria, so the digital assistant on the electronic device 600 is not activated. The electronic device 600 determines that user 620 is not looking at the electronic device 600 if the user's gaze duration is less than a threshold duration, or if the electronic device 600 is not within user 620's field of view. In some embodiments, the field of view criterion is not met when the determined offset angle 622 between the user's line of sight 612 and the calculated line of sight 608 is greater than a threshold offset angle 604. If the digital assistant is not activated and the gaze information does not satisfy one or more activation criteria, the electronic device 600 refrains from activating the digital assistant. When the digital assistant is not activated, the electronic device 600 optionally does not turn on one or more of the microphone or speaker, does not record voice, and / or does not enable processing of recorded voice for the purpose of executing utterance commands (for example, the electronic device 600 can still process voice to detect trigger words, but does not process voice for the execution of commands other than activation). In some embodiments, the electronic device 600 does not provide any instructions using visual or voice indicators when the digital assistant is not activated (for example, the state of the electronic device 600 remains unchanged). In some embodiments, the electronic device 600 remains in the same inactive state as the state prior to the determination that the gaze information did not meet one or more activation criteria.

[0177] In Figure 6A, user 620 begins uttering part 624A "Turn..." of command 624A-624B of "Turn on the table lamp" before the digital assistant is activated. In some embodiments, as will be discussed further below, if the digital assistant is not subsequently activated (e.g., within a set (non-zero) duration), part 624A of command 624A-624B is not processed by the digital assistant, is canceled, and / or ignored. In some embodiments, as will be discussed further below, if the digital assistant is activated (e.g., by user 620 looking at electronic device 600 and meeting one or more activation criteria) before the command 624A-624B is completed (or within a set duration), part 624A of command 624A-624B received by electronic device 600 before the digital assistant was activated is processed.

[0178] As shown in Figure 6B, user 620 is looking at electronic device 600, and electronic device 600 determines that it is within user 620's field of view 606. The gaze information obtained from camera sensors (e.g., 616A, 616B) satisfies one or more sets of activation criteria.

[0179] In Figure 6C, the electronic device 600 determines that gaze information acquired from camera sensors (e.g., 616A, 616B) satisfies one or more activation criteria and activates the digital assistant accordingly. Activating the digital assistant optionally includes one or more of the following: turning on a microphone or speaker, recording speech, and enabling processing of the recorded speech for the purpose of executing utterance commands (rather than simply detecting a trigger phrase). The electronic device 600 optionally provides an indication (e.g., via indicators 610A, 610B, or 610C) that one or more activation criteria are met and / or that the digital assistant is activated. The indicators optionally include a visual indication 610A, an utterance 610B, a confirmation 610C uttered by the digital assistant, or any combination of different indicators. Providing a visual indicator 610A optionally includes turning on the display 602 of the electronic device 600, turning on the LEDs of the electronic device 600, displaying light of a specific color (e.g., blue or a color other than the default color when the digital assistant is not activated), a light pattern, or a visual output on the display 602 of the electronic device 600. In some embodiments, a visual indicator 610A indicating that the digital assistant is activated is different from a visual indicator 610A indicating that the digital assistant has executed a command or is in an inactive state. For example, the electronic device displays a first color of the visual indicator 610A in response to the fulfillment of one or more activation conditions, a second color of the visual indicator 610A in response to the activation of the digital assistant, a third color of the visual indicator 610A in response to the determination that the digital assistant has executed a command, and / or a fourth color of the visual indicator 610A in response to the change that the digital assistant has not been activated.The voice instruction 610B optionally includes the playback of one or more sounds, phrases, words, or tones. In some embodiments, the voice instruction 610B indicating that the digital assistant is activated (or that a set of one or more activation criteria has been met) is different from the voice instruction 610B indicating that the digital assistant has executed a command (e.g., 624A) or is in an inactive state. The digital assistant optionally provides a digital assistant voice confirmation 610C (e.g., "How can I help you?") indicating that the digital assistant is activated or that a set of one or more activation criteria has been met. In some embodiments, as will be described in more detail with respect to Figures 10A–10D, the electronic device outputs a voice confirmation using a specific voice (e.g., accent, male or female voice) based on the detected user.

[0180] In Figure 6C, following the activation of the digital assistant and the provision of instructions using the display 602, the electronic device detects that user 620 has completed uttering the remaining part 624B of command 624C, "Turn on the table lamp," "...on the table lamp." In some embodiments, the electronic device 600 detects that user 620 has uttered the entire command 624C, "turn on the table lamp," after the digital assistant has been activated. Optionally, the electronic device 600 provides instruction 610C according to a determination that one or more activation criteria were met before it began receiving commands 624A-624B, and does not provide instruction 610C according to a determination that one or more activation criteria were met after it began receiving commands 624A-624B (but not before).

[0181] In some embodiments, the electronic device 600 generates an audio output, such as playing music, at a first volume before receiving the remaining portion 624B of the commands 624A-624B. In some embodiments, upon receiving the first portion 624A of the commands 624A-624B, the electronic device 600 ducks the first volume of the audio output down to a second volume lower than the first volume. This second volume is optionally based on the distance between the user 620 and the electronic device 600. The first volume of the audio output is lowered to reduce interference from background noise, allowing the electronic device 600 to detect words more effectively. In some embodiments, the first volume is lowered in response to a determination that one or more sets of activation criteria are met. In some embodiments, the volume reduction is based on the distance between user 620 and electronic device 600 when the first portion 624A (but not the second portion 624B) of command 624A-624B is received. Electronic device 600 uses an external camera sensor 616A and / or a camera sensor 616B on electronic device 600 to determine the distance between user 620 and electronic device 600. In some embodiments, electronic device 600 variably reduces the volume of the audio output by muting or lowering the volume of the audio output while detecting that user 620 is speaking. After detecting the end of command 624B, electronic device 600 restores the volume of the audio output to the first volume. The volume change is not based on the content of the request. In some embodiments, even if the electronic device 600 detects that user 620 has uttered the first part 624A of commands 624A-624B before activating the digital assistant, the digital assistant can still process the command (including 624A) if one or more sets of execution criteria are met. If one or more sets of execution criteria are met, the digital assistant executes the command based on the content of the request. One or more sets of execution criteria optionally include one or more of the command completion criteria, the enforceability criteria, and the activation criteria.The command completion criterion is met when the electronic device 600 detects the end of a received voice command (e.g., a voice user input request). The end of commands 624A-24B is determined by the electronic device 600 detecting a pause in the received voice (i.e., no user voice has been received) that exceeds a predetermined non-zero period. In some embodiments, the detected pause is less than a predetermined non-zero period (in which case the command completion criterion is not met, e.g., the user has not finished uttering the command). The enforceability criterion is met when the electronic device 600 determines that the received command is enforceable (e.g., the device has the authority and / or ability to execute the command). In some embodiments, if the command is to access information from an account, and the electronic device has not received authorization to access the account, the command is not enforceable because the device does not have the authority to access the account. In some embodiments, the enforceability criterion is not met when the electronic device 600 cannot process the received request into a command that the digital assistant can execute or send (e.g., the command is "turn on the TV," but there is no TV in the room to which the corresponding command may be sent). The activation criteria are met when the digital assistant is activated.

[0182] Figure 6D illustrates that the digital assistant executes a command when one or more execution criteria are met while the digital assistant is activated. If the digital assistant successfully processes the command and determines that the execution criteria have been met, it sends a command to change the state of the appropriate device. In this example, a command is sent to turn on a table lamp 618 (external device). In some embodiments, the determination of which external device to activate is made as described in Figures 8A-8C, as will be described in more detail below. After the digital assistant has executed a command, the electronic device 600 optionally provides an indicator (e.g., indicators 610A, 610B, 610C) to indicate whether commands 624A-624B were successfully executed. In some embodiments, the indicator is a visual indicator 610A that displays a specific color of light (e.g., blue, or a color other than the default color when the digital assistant is not activated) or a light pattern to indicate that the digital assistant has executed commands 624A-624B. In some embodiments, the indicator is an audio indicator 610B that plays a sound, phrase, word, or tone. In some embodiments, as described with respect to Figures 10A to 10D, the indicator is a confirmation 610C uttered by the digital assistant, which can be uttered in a specific voice (e.g., accent, male or female voice) when the digital assistant executes commands 624A to 624B. In some embodiments, the user 620 can customize the visual indicator, voice indicator, or digital assistant indicator to include values ​​of characteristics associated with the user's profile (e.g., username, bright colors, patterns, sounds, voice, specific user preferences for accent), as described later with respect to Figures 10A to 10D. For example, in Figure 6D, the digital assistant may provide the confirmation, "Jack, the table lamp has been turned on."The name of user 620 is obtained from the user profile provided by user 620 during registration and is determined as described later with respect to Figures 10A to 10D. The registration process is described in detail below with respect to Figures 10A to 10D and Method 1100. The digital assistant can further customize the indicators to reflect which user (e.g., Jack) issued the command, using the settings from user 620's profile.

[0183] If one or more execution criteria are not met, the electronic device 600 refrains from executing the command based on the content of the command. In some embodiments, if the enablement criterion is not met, one or more execution criteria are not met. If the enablement criterion is not met, the electronic device 600 optionally generates a headshake instruction, as shown in Figure 6O, to indicate that the command cannot be executed because the digital assistant was unable to process the command, lacked authorization, or was incapable of executing the command. The headshake instruction in Figure 6O includes light or light pattern 602A on the first side of the display 602 of the electronic device 600 (as shown in 6O-A), followed by a light pattern 602B on the second side of the display 602 (as shown in 6O-B), followed by a light pattern 602A on the first side of the display 602 (as shown in 6O-C), which simulates a person shaking their head. In some embodiments, when the execution criteria are not met, a specific indicator (e.g., a pattern of light, a color, or a specific sound) is generated by the electronic device 600. The specific indicator may correspond to a specific error code indicating why the digital assistant was unable to execute the command (e.g., blue light indicates that the command was not understood, and purple indicates a lack of authorization). In some embodiments, the indicator generated when the digital assistant is unable to execute the command is different from the indicator generated when the digital assistant is able to execute the command.

[0184] Figures 6E to 6G illustrate another embodiment in which a user 620 activates a digital assistant on a wearable electronic device or mobile electronic device 630. In Figure 6E, the digital assistant on the electronic device 630 is activated when the electronic device 630 detects user input or a gesture such as raising the wrist. In some embodiments, the user 620 may also optionally activate the digital assistant on the electronic device 630 by pressing a button, turning a crown, or providing voice user input. Once the digital assistant is activated, the electronic device 630 optionally provides a visual indicator 610A (e.g., light), a voice indicator 610B (e.g., sound, tone, word), or voice confirmation 610C of the digital assistant to indicate that the digital assistant has been activated or that one or more sets of activation criteria have been met. Various types of indicators are described above with respect to Figure 6B.

[0185] In Figure 6F, the electronic device 630 detects an utterance command from user 620, including the command 624C, "Turn on the table lamp." After receiving the utterance command, the digital assistant determines whether one or more execution criteria (e.g., a feasibility criterion, a command completion criterion, and an activation criterion) have been met. As described above, if one or more execution criteria are met, the digital assistant executes the command based on the content of the voice input request. As shown in Figure 6F, the electronic device also optionally determines whether the user's gaze meets one or more gaze criteria to identify an external device to act upon. The set of one or more gaze criteria is described in more detail below.

[0186] Figure 6G shows that the electronic device 630 provides indicators (e.g., 610A, 610B, 610C) indicating that the command was successfully executed when turning the table lamp from the off state to the on state. The indicators that show that the command was successfully executed are described above with respect to Figure 6E.

[0187] Figures 6H to 6J illustrate that the digital assistant does not execute the commands associated with the utterance commands 624A to 624B because one or more sets of execution criteria are not met due to the user 620 interrupting their gaze while uttering commands 642A to 624B. The set of one or more execution criteria includes a continuous gaze criterion that is met when the electronic device 600 determines that second gaze information (e.g., acquired by the electronic device 600 using one or more camera sensors 616A, 616B (while the user 620 is uttering) indicates that the user 620 did not interrupt their gaze toward the electronic device 600 for more than a threshold duration between the activation of the digital assistant and the completion of the utterance commands 624A to 624B. In some embodiments, the threshold duration is zero, meaning that the user cannot interrupt their gaze while uttering. In some embodiments, the threshold duration is a non-zero period, which allows the user to temporarily look away without canceling the command.

[0188] In Figures 6H to 6I, user 602 utters the commands 624A to 624B, "Turn on the table lamp," while the electronic device 600 is being activated, interrupting their gaze towards the electronic device 600 during this time. As shown in Figure 6H, while the electronic device 600 receives the first part 624A of the commands 624A to 624B, "turn...", the electronic device 600 determines that user 620 is looking at the electronic device. In Figure 6I, the electronic device 600 detects the interruption of the user's gaze when user 620 turns 180 degrees away from the electronic device 600 and looks away before the user completes uttering the remaining 624B of the commands 624A to 624B, "...on the table lamp." The user's gaze is interrupted from the electronic device 600 when the electronic device 600 determines that the field of view criteria are not met for a non-zero threshold duration (e.g., 3 seconds) of the user's gaze.

[0189] Figure 6J shows that even if one or more activation criteria are met (e.g., Figure 6H), the electronic device 600 refrains from executing the "turn on that light" commands 624A-624B. As a result, the table lamp 618 remains off because the digital assistant has not sent a command to turn it on. The electronic device 600 optionally generates an indicator (e.g., 610A, 610B, 610C) indicating that no command has been executed. In some embodiments, the indicator may be a visual indicator 610A (e.g., a colored light or light pattern), an audio indicator 610B (e.g., a sound, utterance, or specific voice), or a digital voice confirmation 620C, as described above. When commands 624A to 624B are not executed, the indicators provided by the electronic device 600 (e.g., 610A, 610B, 610C) are optionally different from the indicators provided by the electronic device 600 when commands 624A to 624B are successfully executed (e.g., 610A, 610B, 610C).

[0190] Figures 6K to 6N show that even if the electronic device 600 detects that user 620 has interrupted gazing at the electronic device 600 while the electronic device 600 is receiving commands 624D to 624F (when the electronic device 600 is activated), the continuous gazing criterion is still met if the interruption of gazing is less than the threshold duration. Figure 6K shows that the electronic device 600 has activated the digital assistant and is receiving part 624D "turn..." of the command 624D to 624F "turn on the table lamp" from user 620. In Figure 6L, the electronic device 600 continues to receive the second part 624E "...on the..." of the command 624D to 624F from user 620, but the electronic device 600 detects the interruption of the user's gazing when the user looks away from the electronic device 600.

[0191] In Figure 6M, the electronic device 600 detects that user 620 has returned the gaze to the electronic device 600 within a threshold duration (e.g., 3 seconds). The electronic device continues to receive the last part 624F of the commands 624D-624F, "...table lamp". Since the gaze interruption is less than the threshold duration, the continuous gaze criterion is still met. In Figure 6N, once the continuous gaze criterion is met, the digital assistant executes the commands 624D-624F, "Turn on the table lamp", and provides instructions (e.g., visual instruction 610A, voice instruction 610B, and digital assistant confirmation instruction 610C) when the table lamp 618 is turned on.

[0192] Figures 7A and 7B are flowcharts illustrating Method 700 for using electronic devices according to several embodiments. Method 700 is performed on electronic devices (e.g., 100, 300, 500, 600, 630, 800, 1000). Some operations of Method 700 are combined optionally, the order of some operations is changed optionally, and some operations are omitted optionally.

[0193] As described below, Method 700 provides an intuitive method for activating a digital assistant on an electronic device by simply glancing at the electronic device. This method reduces the cognitive burden on the user when providing user input (e.g., voice input by utterance or touch input via button press) to activate the digital assistant. In some situations, the user may not be able to effectively use voice activation (e.g., in a noisy room) or provide tactile user input (e.g., by pressing a button). Therefore, by enabling the digital assistant to be activated by looking at the electronic device, the user can interact with the digital assistant using a more efficient user-device interface.

[0194] While the digital assistant of the electronic device (e.g., 600) is not activated (702), the electronic device (e.g., 600) uses one or more camera sensors (e.g., cameras 616A, 616B) to acquire first gaze information (e.g., of user 620) (704). For example, one or more camera sensors are infrared camera sensors (one or more) and / or visible light sensors (one or more). For example, one or more camera sensors are located outside the electronic device (e.g., 600), for example, by being wirelessly connected to the electronic device.

[0195] If, for example, while the digital assistant is not activated, the first gaze information satisfies a set of one or more activation criteria (e.g., the angle of the gaze is less than a threshold degree relative to the line of sight, the gaze is in a particular direction, the duration of the gaze is longer than a predetermined period of time) (706), the electronic device (e.g., 600) activates the digital assistant of the electronic device (e.g., 600) (e.g., by turning on the microphone and enabling the processing of recorded audio) (708).

[0196] Using gaze detection to activate a digital assistant when one or more activation criteria are met provides the user with a more efficient user interface for activating the digital assistant, particularly in contexts where the user cannot press a button or use voice control due to a noisy room. By providing an improved user interface for activating the digital assistant that does not require haptic user input (e.g., button press) or voice input, it enhances the usability of the digital assistant and makes the user interface of the digital assistant interface more efficient. In addition, using gaze detection reduces the number of user inputs required to activate the digital assistant and execute commands, improving the usability of the electronic device.

[0197] In some embodiments, a set of one or more activation criteria includes a field of view criterion that is satisfied when a determined offset angle (e.g., 622) is less than a threshold offset angle (e.g., 604) (e.g., a non-zero degree value, the maximum acceptable deviation from the user directly viewing the device), where the determined offset angle (e.g., 622) is the angle formed between the line of sight (e.g., 612) of the user's (e.g., 620) gaze (e.g., determined from gaze information) and the calculated line of sight (e.g., 608) from the user (e.g., 620) to the electronic device (e.g., 600).

[0198] By using a field of view criterion as the activation criterion, it becomes possible to detect the user's gaze even if the user (e.g., 620) is not directly looking at the electronic device (e.g., 600). As long as the electronic device (e.g., 600) is within the user's field of view (e.g., 606), the user's gaze satisfies the activation criterion, and the digital assistant can be activated. Using a field of view criterion allows for more successful activation of the digital assistant via gaze, thereby improving the usability of the digital assistant and making gaze detection more efficient (for example, the digital assistant has a higher activation rate because the electronic device (e.g., 600) does not need to be within the user's (e.g., 620) direct line of sight (e.g., 612)).

[0199] In some embodiments, if an electronic device (e.g., 600) is determined to be located beyond a threshold (e.g., outside the gaze field 606), the electronic device (e.g., 600) is not within the user's gaze, and the digital assistant is not activated. In some embodiments, the digital assistant is activated when the electronic device (e.g., 600) is within the gaze field (e.g., 606) by being within the threshold range.

[0200] In some embodiments, the gaze line (e.g., 612) of the user (e.g., 620) is based on the position of the user's (e.g., 620) head. In some embodiments, the position of the user's head may be influenced by tilt or rotation (e.g., yaw, pitch, and roll of the user's head).

[0201] In some embodiments, one or more activation criteria include a static time criterion that is met when the static time of a gaze (e.g., duration) determined from first gaze information exceeds a threshold time (e.g., a non-zero threshold time). In some embodiments, if the static time of a gaze is less than the threshold time, the digital assistant is not activated.

[0202] Furthermore, in accordance with the determination that the first gaze information satisfies one or more activation criteria (706), the electronic device (e.g., 600) provides an indication that one or more activation criteria have been met and / or that the digital assistant of the electronic device is transitioning or has transitioned and is being activated (710).

[0203] Feedback on the current state of the digital assistant is provided to the user by providing an indication (e.g., 610A, 610B, 610C) that the digital assistant is activated when one or more activation criteria are met. The indication (e.g., 610A, 610B, 610C) may be a visual confirmation, an audible confirmation, or an audible confirmation from the digital assistant, providing customizable feedback for various contexts. For example, in a context where a sound indicator is not audible (e.g., a noisy room), providing visual feedback (e.g., 610A) allows the user (e.g., 620) to know whether the digital assistant is activated and capable of processing commands. Providing customized feedback on the state of the digital device improves the usability of the digital assistant and makes the user-device interface more efficient by informing the user (e.g., 620) when the digital assistant is available to process commands (e.g., activated) and when it is unavailable (e.g., inactive) (e.g., if the digital assistant is inactive, the user (e.g., 620) does not issue commands). Furthermore, feedback on the digital assistant's status allows users (e.g., 620) to use the digital assistant more efficiently.

[0204] In some embodiments, instruction (712) is a visual instruction (e.g., 610A) indicating that the digital assistant is activated (e.g., electronic device 600 turns on the display, LED, or light of electronic device 600). In some embodiments, instruction (714) is an audio instruction (e.g., 610B) indicating that the digital assistant is now activated (e.g., electronic device 600 plays a sound through the speaker of electronic device 600).

[0205] In some embodiments, the electronic device (e.g., 600) refrains from activating the digital assistant upon determination (716) that the first gaze information (e.g., while the digital assistant is not activated) does not satisfy one or more activation criteria (e.g., the electronic device (e.g., 600) is not within the user's field of view (e.g., 606), the gaze is in the wrong direction, the duration of the gaze is shorter than a predetermined period). In some embodiments, refraining from activating the digital assistant includes leaving the digital assistant unactivated or the electronic device (e.g., 600) not activating the microphone or processing recorded audio. In some embodiments, if the first set of gaze information does not satisfy one or more activation criteria, the electronic device (e.g., 600) also refrains from providing an indicator (e.g., 610A, 610B, 610C) that one or more activation criteria have been met.

[0206] In some embodiments, an electronic device (e.g., 600) receives a voice-activated user input request to execute a command (e.g., 624A-624B, 624C) (718). In response to the electronic device (e.g., 600) receiving a voice-activated user input request to execute a command (e.g., 624A-624B, 624C) (720), (or in accordance with the interpretation of the command (e.g., 624A-624B, 624C) by a digital assistant, and in accordance with the determination that one or more execution criteria (e.g., enablement criterion, command completion criterion, activation criterion) have been met, the electronic device (e.g., 600) executes the command (e.g., 624A-624B, 624C) based on the content of the voice-activated user input request (722). In some embodiments, upon receiving a voice input request and failing to interpret the command (e.g., 624A-624B, 624C), the electronic device (e.g., 600) suspends execution of the command (e.g., 624A-624B, 624C) and provides an indication (e.g., 610A, 610B, 610C) (e.g., display of a visual indication such as an indication by head shake) that the command (e.g., 624A-624B, 624C) was not interpreted. Thus, the electronic device (e.g., 600) provides an indication (e.g., 610A, 610B, 610C) if the digital assistant does not understand the voice input. In some embodiments, if the digital assistant is not activated, the digital assistant suspends execution of the received command (e.g., 624A-624B, 624C).

[0207] In some embodiments, in response to receiving a voice-activated user input request for an execution command (e.g., 624A-624B, 624C) from an electronic device (e.g., 600), and in accordance with a determination that one or more execution criteria (e.g., enable criteria, command completion criteria, activation criteria) are not met, the electronic device (e.g., 600) provides a headshake instruction. The headshake instruction includes the electronic device (e.g., 600) displaying a first light pattern on a first side of the electronic device's (e.g., 600) display (e.g., 602). Following the display of the first light pattern, the electronic device (e.g., 600) displays a second light pattern on a second side of the electronic device's (e.g., 600) display (e.g., 602). Following the display of the second light pattern, the electronic device (e.g., 600) displays the first light pattern on the first side of the electronic device's (e.g., 600) display (e.g., 602).

[0208] By providing visual instructions (e.g., 610A) (e.g., head shake instructions (e.g., 602A-602B)) indicating that one or more execution criteria are not met, the user is given feedback that a command issued by the user cannot be processed. By providing customizable instructions, such as head shake instructions (e.g., 602A-602B) indicating when a command (e.g., 624A-624B, 624C) cannot be processed, the usability of the digital assistant is improved, and the user-device interface becomes more efficient by notifying the user (e.g., 602) that a command (e.g., 624A-624B, 624C) will not be executed. Furthermore, customized instructions (e.g., 610A, 610B, 610C) corresponding to error codes provide the user with improved feedback that explains why the command (e.g., 624A-624B, 624C) was not executed, thereby making the user-device interface more efficient.

[0209] In some embodiments, a set of one or more execution criteria includes a completion criterion that is satisfied when an electronic device (e.g., 600) detects the end of an received voice command (e.g., the end of a voice user input request, a pause in voice beyond a predetermined period). In some embodiments, a set of one or more execution criteria includes an enforceability criterion that is satisfied when an electronic device (e.g., 600) determines that an received command is enforceable (e.g., the digital assistant has the authority and / or ability to execute the command).

[0210] In some embodiments, the electronic device (e.g., 600) begins receiving voice user input requests while the digital assistant is not activated. In some embodiments, the digital assistant is inactive because gaze information does not yet satisfy a set of one or more activation conditions. This can occur, for example, when the user (e.g., 620) begins uttering a command (e.g., 624A-624B, 624C) before looking at the digital assistant. In some embodiments, the first gaze information satisfies a set of one or more activation criteria, and accordingly, the digital assistant is activated before the electronic device (e.g., 600) terminates receiving voice user input requests to execute a command.

[0211] By being able to receive voice user input requests before the digital assistant is activated, the user (e.g., 602) can begin speaking even if they have not yet activated the digital assistant. Detecting voice user input requests when the set of activation conditions has not yet been met improves the usability of the digital assistant and makes the device-user interface more efficient, as activation and command utterances (e.g., 624A-624B, 624C) do not need to be performed in a specific order.

[0212] In some embodiments, in response to an electronic device (e.g., 600) receiving a voice user input request to execute a command (e.g., 124C), and in accordance with a determination that one or more sets of execution criteria are not met (e.g., various criteria that should be met for the electronic device (e.g., 600) to execute a command (e.g., 624A-624B, 624C)), the electronic device (e.g., 600) refrains from executing the command (e.g., 624A-624B, 624C) based on the content of the voice user input request (724), the set of one or more execution criteria includes a continuous gaze criterion that the electronic device (e.g., 600) determines is met if second gaze information (e.g., acquired by the electronic device using one or more camera sensors) indicates that the user (e.g., 602) does not interrupt gazing toward the electronic device (e.g., 600) for more than a threshold duration (e.g., a non-zero duration) between the activation of the digital assistant and the completion of the voice user input request. In some embodiments, the digital assistant will not execute a command (e.g., 624A-624B, 624C) if the user (e.g., 602) interrupts looking at the device (e.g., the user (e.g., 602) looks away before the command is completed, or the user (e.g., 602) looks away for more than one second at a time). In some embodiments, the threshold duration is 0 seconds. In some embodiments, the threshold duration is a non-zero duration (e.g., 0.5 seconds).

[0213] In some embodiments, an electronic device (e.g., 600) generates an audio output (e.g., the electronic device (e.g., 600) is playing music), and the audio output is generated at a first volume before receiving a voice user input request. In response to initiating reception of a voice user input request, the electronic device (e.g., 600) reduces the volume of the audio output to a second volume lower than the first volume, and the second volume is based on the distance between the user (e.g., the user providing the voice user input request) and the electronic device (e.g., 600). In some embodiments, the electronic device (e.g., 600) uses an external sensor (e.g., 616A) or a sensor on the electronic device (e.g., 600) (e.g., 616B) to determine the distance between the user (e.g., 602) and the electronic device (e.g., 600) and determines whether to mute or reduce the volume of the music (e.g., variably lower the audio output to tuck one's head) while the user (e.g., 602) is speaking. In some embodiments, the electronic device may return the volume to a first volume level after detecting the end of a command (e.g., 624C). In some embodiments, the volume change is not based on the content of the request.

[0214] When an electronic device receives a voice user input request, reducing the voice output volume or muting the voice output makes it easier to process the voice user input request. As a result, ambient noise is reduced, increasing the likelihood that the command derived from the voice user input request (e.g., 624C) will be understood by the electronic device. By eliminating ambient noise or sounds that may interfere with the command (e.g., 624C) processed by the digital assistant, the usability of the digital assistant is improved, and the accuracy of executing the correct command is increased, making the user-device interface more efficient. This improves the user experience because the user (e.g., 602) does not need to repeat the command when the digital assistant cannot process the command (e.g., 624C).

[0215] It should be noted that the details of the process described above with respect to Method 700 (for example, Figures 7A to 7B) are also applicable in a similar manner to the methods described below. For example, Method 700 may include one or more characteristics of the various methods described below with reference to the processes in Figures 9A, 9B, 11A, and 11B. For the sake of brevity, these details will not be repeated below.

[0216] Please understand that the specific order in which operations are described in Figures 9A-9B and 11A-11B is illustrative and not intended to indicate that the described order is the only possible order in which the operations can be performed. Those skilled in the art will recognize various ways of rearranging the order of operations described herein, as well as the exclusion of certain operations. For the sake of brevity, these details will not be repeated here. Furthermore, please note that the methods and modes of processing described throughout this description may be integrated with one another.

[0217] Figures 8A–8L illustrate exemplary user interfaces, in some embodiments, for an electronic device to use gaze information to determine the context (e.g., the device to which the command is directed) for executing a command after the digital assistant has been activated. The techniques shown in Figures 8A–8L optionally work in conjunction with, or include, the techniques shown in Figures 6A–6O, which disclose how a user may activate a digital assistant by looking at an electronic device and issuing a command. Accordingly, the techniques described below include one or more characteristics of the various techniques described above with reference to Figures 6A–6O and Method 700. The user interfaces in these figures are used to illustrate the processes described below, including the processes in Figures 9A and 9B.

[0218] Figures 8A–8C illustrate that the electronic device 800 receives a voice user input request from user 820 to execute a first set of commands 824A–824B when the digital assistant of the electronic device 800 is activated. The digital assistant is activated based on a trigger word, gaze (e.g., as described above with respect to Figures 6A–6O), button press, the user lifting their wrist while wearing a wearable electronic device (e.g., a wristwatch) (Figures 6E–6G), and / or detection of an external device (e.g., a telephone) pointed to by the electronic device 800. The electronic device (e.g., 800) optionally provides visual or voice instructions 810A–810B to indicate that the digital assistant is activated.

[0219] In Figure 8A, user 820 is standing in a room containing a table lamp 818 and a floor lamp 814. The digital assistant of electronic device 800 is activated based on the determination by electronic device 800 that one or more sets of activation criteria are met as described above with respect to method 700 (for example, device 800 determines that user 820 is looking at device 800). Once the digital assistant is activated, the electronic device detects that user 820 utters part 824A "turn..." of the full command 824A-824B. In some embodiments, the complete command 824A-824B is applicable to multiple external devices that may be present in the room (e.g., table lamp 818, floor lamp 814). In some embodiments, the user's command 824A-824B does not contain enough detail for the digital assistant to determine which external device (e.g., table lamp 818, floor lamp 814) the command 824A-824B is directed to. Therefore, the electronic device 800 cannot determine whether “that light” refers to the table lamp 818 or the floor lamp 814 based solely on commands 824A to 824B.

[0220] In Figure 8B, user 820 indicates which external device (e.g., a light) he wants to activate by looking at the table lamp 818, and finishes saying the remaining part of the command 824A-824B "turn on that light" (824B "...on that light"). The electronic device receives gaze information based on the user's gaze while user 820 is speaking, and uses it to determine that the table lamp 818 is the target device when the user's gaze satisfies one or more sets of gaze criteria. Electronic device 800 acquires information about the user's gaze via one or more camera sensors, which may be located on an external device 816A and / or integrated into electronic device 800 with a camera 816B. Camera 816A is wirelessly connected to electronic device 600, but may be wired instead. One or more camera sensors include an infrared camera sensor and / or a visible light sensor, which measure gaze information as soon as a gaze is detected. The gaze information optionally includes one or more of the following: the user's head position, the time or duration of the gaze, the direction of the gaze, the field of view 806 of the gaze, the direction of the gaze, and whether an object (e.g., an electronic device 800 or an external device (e.g., a lamp 814, 818)) is within the field of view 806 of the gaze (e.g., the external device is within a threshold offset angle 804 from the user's line of sight 812).

[0221] In some embodiments, the electronic device 800 determines that one or more sets of gaze criteria are met when a direction criterion, a static time criterion, and / or a field of view criterion are met. The set of one or more gaze criteria optionally includes a direction criterion that is met when the gaze is directed toward the table lamp 818.

[0222] A set of one or more gaze criteria includes a quiescent time criterion that is met when the electronic device 800 detects a quiescent or duration of gazing toward the table lamp 818 for a predetermined period of time (e.g., 2 seconds, threshold time) that is not zero. In some embodiments, the electronic device 800 uses the quiescent time to determine whether user 820 is looking at the table lamp 818 in order to determine which external object to act upon (e.g., to send a command). If the quiescent or duration of gazing is too short or less than the threshold time, this indicates that the user does not intend a particular external device (e.g., the table lamp 818) to be the target device of commands 824A-824B. In some embodiments, a quiescent time less than the threshold time does not satisfy the gaze criterion, and as a result, commands 824A-824B are not sent to the particular external device identified by the electronic device 800. In some embodiments, a quiescent time shorter than the threshold time results in the electronic device 800 not identifying any external device.

[0223] A set of one or more gaze criteria may optionally include a field of view criterion. A field of view criterion is met for a particular external device when it is determined that the external device, such as a table lamp 818, is within the user 820's field of view 806. The table lamp 818 is determined to be within the user 820's field of view 806 when it is within a threshold offset angle 804 from the user's line of sight 812. By using the field of view to determine whether the user is looking at the external device in question (e.g., the table lamp 818, floor lamp 814), the gaze detection rate is increased because a gaze can be detected within the field of view even if the user is not directly looking at the device in question. In some embodiments, a field of view criterion is met for the table lamp 818 when it is directly within the user 820's line of sight 812. In some embodiments, a field of view criterion is met for a particular external device when the determined offset angle of the gaze is less than a threshold offset angle 804 for the external device (e.g., the maximum allowable deviation from the user 820's line of sight 812 when the table lamp 818 is directly looking at it). The threshold offset angle 804 is greater than zero degrees (e.g., 30 degrees). The determined offset angle is calculated based on the angle formed between the line of sight 812 from the user 820's gaze and the calculated line of sight from the user 820 to the electronic device 800. In the example in Figure 8B, the user 820 is looking directly at the table lamp 818. In some embodiments, the field of view criterion is met for the table lamp 818 when it is within the field of view of the gaze by being within a threshold degree range from the user's direct line of sight, and the electronic device 800 optionally sends a command to the table lamp 818. When the offset angle is greater than the threshold offset angle 804, the field of view criterion is not met, and as a result, the digital assistant does not send a command to the table lamp 818 which is outside the user 820's field of view 806. In some embodiments, if it is determined that the table lamp 818 is located beyond a threshold degree outside the user 820's field of view 808, the table lamp 818 is determined not to be within the user's line of sight, and the electronic device 800 does not send a command to the table lamp 818.In some embodiments, the line of sight 812 is measured based on the user's head position. In some embodiments, the user's head position is based on the tilt or rotation of the user's head (e.g., yaw, pitch, and / or roll). These aspects of the technique are illustrated and described above in more detail with respect to Figures 6B-6C.

[0224] The electronic device 800 optionally provides instructions (e.g., indicators 810A, 810B, 810C) when one or more sets of gaze criteria are met for the table lamp 818. The instructions optionally indicate that commands 824A-824B from an received voice-activated user input request are associated with or directed towards the table lamp 818. The instructions may be a visual 810A indicator or a voice 810B indicator indicating that the digital assistant has identified the table lamp 818 based on the user's gaze. In some embodiments, the indicator is a visual indicator 810A that displays a specific color of light (e.g., blue or a color other than the default color when the table lamp 818 is identified) or a light pattern to indicate that the digital assistant has identified the table lamp 818 based on the user's gaze. In some embodiments, the indicator is a voice indicator 810B that plays a sound, word, tone, or speaks in a specific voice (e.g., accent, male or female voice) when the digital assistant has identified the table lamp 818. In some embodiments, the digital assistant provides voice confirmation ("I'll turn on the table lamp") indicating that the electronic device 800 has identified an external device that it should act on (for example, turn on).

[0225] In Figure 8C, when the electronic device determines that one or more sets of gaze criteria have been met for the table lamp 818, the digital assistant sends a command that acts on the command. For example, the electronic device determines that the command is to perform a "turn on" function and that this function should be directed to a specific device (table lamp 818). Therefore, the electronic device sends a command (for example, to the table lamp 818 or to another device that controls the table lamp 818) to turn the table light from the off state to the on state. In some embodiments, the digital assistant sends a command that causes an external device to change a task as a result of a transition from a first state to a second state. For example, the digital assistant may send a command to an external device (for example, a computer or telephone) that causes the external device to change music, transition from a first state to a second state, transition from the execution of one task to the execution of another task (for example, from music playback to weather forecasting), or transition from an inactive state to an active state, or vice versa.

[0226] As shown in Figure 8C, if there are multiple external devices (e.g., a table lamp 818, a floor lamp 814) in the room (or accessible by the electronic device 800), the electronic device 800 can distinguish between the multiple external devices and send commands only to the intended external device (e.g., the table lamp 818, the floor lamp 814). The electronic device 800 identifies the intended external device (e.g., the table lamp 818, the floor lamp 814) by determining whether one or more sets of gaze criteria are met for that particular external device (e.g., the table lamp 818, the floor lamp 814). In other words, when one or more sets of gaze criteria are met, the electronic device 800 uses the gaze information to determine which external device (e.g., the table lamp 818, the floor lamp 814) the user is looking at. In some embodiments, when electronic device 800 sends a command (e.g., "turn on") to a identified first external device (table lamp 818), a second external device in the room (floor lamp 814) is not affected because it is not an external device identified by gaze information. The floor lamp 814 (second external device) may be in an on state or an off state. For example, in Figure 8C, a command is sent to the table lamp 818 to turn it on, but no command is sent to the floor lamp 814, which remains off (the floor lamp 814 was previously in an off state). Therefore, the transmitted command does not change the state of the floor lamp 814. In some embodiments, when user 820 glances at the floor lamp 814 instead of the table lamp 818, one or more sets of gaze criteria are met by the floor lamp 814 instead of the table lamp 818. Therefore, the digital assistant sends commands to the floor lamp 814 while the table lamp 818 remains unaffected (for example, the table lamp 818 remains on if it was previously on, and remains off if it was previously off).

[0227] When the digital assistant sends a command to a corresponding external device, the electronic device 800 optionally provides an indicator (e.g., indicators 810A, 810B, 810C) to indicate that the table lamp 818 has transitioned from a first state to a second state. In some embodiments, the indicator is a visual indicator 810A that displays light of a specific color (e.g., blue or a color other than the default color) or a light pattern to indicate that the table lamp 818 has been turned on from the off state as a result of receiving the command. In some embodiments, the indicator is a voice indicator 810B, which is a sound, one or more words, or a tone. In some embodiments, the indicator is a voice confirmation 810C from the digital assistant, such as "Ok, Jack, the table lamp is on." The user can customize the indicator to include characteristics associated with the user's profile (e.g., username, specific user preferences for light color, pattern, sound, voice, and accent) that the user 820 provided during registration. The registration process is described in detail below in relation to Figures 10A-10D and Method 1100. The digital assistant customizes the indicators to reflect which user (e.g., Jack) is recognized, as will be described later with respect to Figures 10A to 10D and Method 1100. In some embodiments, the electronic device 800 provides indicators (e.g., 810A, 810B, 810C) when the digital assistant successfully sends a command to the table lamp 818, regardless of whether the table lamp 818 is actually turned on. In some embodiments, the electronic device 800 provides indicators (e.g., 810A, 810B, 810C) in response to a determination that the table lamp 818 is turned on. Optionally, when the table lamp 818 is unable to execute the "turn on" command and remains off, the electronic device 800 provides different indicators (e.g., 810A, 810B, 810C).In some embodiments, the electronic device 800 may provide different indicators (e.g., 810A, 810B, 810C) to indicate a specific error code when commands 824A to 824B fail to execute.

[0228] The indicators (e.g., 810A, 810B, 810C) are optionally domain-specific (e.g., music). In some embodiments, the electronic device 800 determines that a command refers to an action within a particular domain (e.g., the "Love that song" command within the music domain). According to the determination that the command is in a first domain (e.g., music), the instruction is a first type of instruction (e.g., a specific set of sound or light indicators). According to the determination that the command is in a second domain different from the first domain (e.g., email), the instruction is a second type of instruction different from the first type of instruction (e.g., a different set of sound or light indicators). By having different indicators (e.g., 810A, 810B, 810C) for different domains, the user can easily recognize when a command has been successfully executed and when it has failed to execute.

[0229] Figure 8D shows that when the digital assistant sends a command to turn on the table lamp 818, user 820 can further change the settings of a second external device (e.g., the brightness of the lamp) by providing input to the external device (e.g., by rotating the crown 832 of the wristwatch 830 worn by user 820). The wristwatch 830 does not have access to contextual information about which external device should be controlled. Instead, the user simply uses the crown 832 of the wristwatch 830 as an extended input mechanism to control the lamp. The wristwatch 830 detects the user input and sends a user input instruction to the electronic device 800. The electronic device 800 receives the user input instruction and determines that the table lamp 818 is the intended external device (e.g., table lamp 818, floor lamp 814) that should receive the command. In Figure 8D, user 820 rotates the crown 842 to increase the brightness of the table lamp 818 to brightness 80. In some embodiments, the electronic device 800 determines that the table lamp 818 is the intended external device based on its relevance, specifically that the electronic device 800 has sent a command to the table lamp 818 based on user 820's actions. In some embodiments, the electronic device 800 sends a command (based on user input on the wristwatch 830) to the nearest device that the electronic device 800 has commanded (e.g., the table lamp 818). For example, if the electronic device 800 just commands the computer to play music, the user turning the crown 842 on the wristwatch 840 will cause the electronic device 800 to send a command to the computer to increase (or decrease) the computer's volume. In some embodiments, the electronic device 800 is paired with the table lamp 818. In some embodiments, the electronic device 800 detects that user 820 is wearing the wristwatch 840 while the wristwatch 840 is paired with the electronic device 800. In some embodiments, the wristwatch 840 is (likewise) paired with the table lamp 818.When the electronic device 800 sends a command to the table lamp 818, the electronic device 800 provides an indicator (e.g., 810A, 810B, 810C) to indicate that the command was successfully sent. In some embodiments, the electronic device 800 provides different indicators (e.g., 810A, 810B, 810C) to indicate that the command was not successfully sent to the table lamp 818.

[0230] Figure 8E shows that the electronic device 800 receives additional user input from the wristwatch 830 to dim the light of the table lamp 818 until it reaches a low brightness corresponding to the amount of rotation of the crown 832 by the user 820. In response to receiving the input from the wristwatch 830, the electronic device 800 sends a command to the table lamp 818 to dim the light from brightness 80 to a reduced brightness 20, based on its relevance to having commanded the table lamp 818 to be turned on. In some embodiments, the user 820 directly controls the table lamp 818 at the point when it is turned on using the wristwatch 830 without having to provide additional commands to the digital assistant. Once the electronic device 800 has sent a command to the table lamp 818, the electronic device 800 provides an indicator (e.g., 810A, 810B, 810C) to indicate that the command has been successfully sent. In some embodiments, the electronic device 800 provides different indicators (e.g., 810A, 810B, 810C) to indicate that the command was not successfully transmitted to the table lamp 818.

[0231] In some embodiments, as shown in Figures 8F to 8H, if multiple external devices (e.g., floor lamp 814, table lamp 818) are present in the room, the electronic device 800 uses the user's gaze information to identify a particular external device as the intended external device based on the user's gaze. The electronic device 800 identifies the intended external device by determining, based on the gaze information, whether one or more sets of gaze criteria are met for that particular external device. If one or more sets of gaze criteria are not met for any external device, the electronic device 800 cannot determine the external device from the gaze information and therefore refrains from sending a command. As a result, the external devices in the room remain in their original state.

[0232] Figure 8F shows that the electronic device 800 detects gaze information from user 820 and, in accordance with the determination that one or more sets of activation criteria are met (for example, as described above with respect to Figures 6A to 6D), the electronic device activates the digital assistant. The electronic device may optionally provide a visual indicator 810A (e.g., turning on the display 802), an audio indicator 810B, or a digital voice confirmation 810C to indicate that the digital assistant is activated.

[0233] Figure 8G shows a table lamp 818 that is initially on and a floor lamp 814 that is initially off. The electronic device 800 detects that user 820 utters a second command 824C, “Turn on that light.” However, user 820 is not looking at the floor lamp 814, which is the intended external device for command 826C. The electronic device 800 acquires second gaze information via one or more camera sensors 816A that may be located on the external device and / or integrated with the electronic device 800 by camera 816B. The second gaze information optionally includes one or more of the following: the position of the user’s head, the time or duration of the gaze, the direction of the gaze, the field of view 806 of the gaze, the direction of the gaze, and whether the floor lamp 814 is within the field of view 806 of the gaze.

[0234] In Figure 8H, the electronic device 800 determines that the second gaze information acquired from one or more camera sensors 816 does not satisfy one or more sets of gaze criteria. As described above, one or more sets of gaze criteria optionally include a direction criterion that is satisfied for the floor lamp 814 when the gaze is directed toward the floor lamp 814. One or more sets of gaze criteria optionally include a static time criterion that is satisfied for the floor lamp 814 when the electronic device 800 detects a static time or duration of gaze toward the floor lamp 814 that exceeds a predetermined period (e.g., a non-zero threshold, 2 seconds). One or more sets of gaze criteria optionally include a field of view criterion. The field of view criterion is satisfied for the floor lamp 814 when it is determined that the floor lamp 814 is within the user 820's field of view 808. Figures 8G to 8H show that the floor lamp 814 and table lamp 818 are not determined to be within the user 820's field of view 808 because the determined offset angle is greater than the threshold offset angle from the user 820's line of sight 812. The threshold offset angle 804 is greater than zero degrees (e.g., 30 degrees). If one or more of the duration criterion, the stationary time criterion, and / or field of view criterion are not met for any external device, and therefore one or more sets of gaze criteria are not met, the electronic device 800 cannot determine which external device (floor lamp 814, table lamp 818) the user 820 intends to act on (e.g., turn on).

[0235] Figure 8H shows that because one or more sets of gaze criteria were not met for any external device, the electronic device 800 refrains from sending a “turn on its light” command to transition the external device, floor lamp 814, or table lamp 818, from a first state (e.g., off) to a second state (e.g., on). As a result, floor lamp 814 remains off and table lamp 818 remains off, which were the initial states of the external devices before the electronic device 800 detected that user 820 uttered command 824C. For example, table lamp 818 remains on because no command has been sent to table lamp 818 to change its initial on state.

[0236] The electronic device 800 optionally generates certain indicators (e.g., 810A, 810B, 810C) (e.g., a series of lights, such as a head shake indicator (e.g., Figure 6O or a sound)) indicating that the electronic device 800 did not send a command because it could not identify the floor lamp 814 (or any other device) as the intended external device, based on the gaze information of the user 820. In some embodiments, the certain indicators (e.g., patterns of light, color, or a particular sound) may correspond to a specific error code indicating why the digital assistant could not determine the external device associated with command 824C. Indicators generated when the digital assistant cannot send a command are optionally different from indicators generated when the digital assistant can send a command. In some embodiments, if the digital assistant does not send the command associated with command 824C, no indicators are generated.

[0237] Figure 8I shows another embodiment in which multiple external devices (e.g., a floor lamp 814 and a table lamp 818) are present in the room and paired with the electronic device 800. In this embodiment, the digital assistant is activated by a glance by user 820. In other embodiments, the digital assistant is activated by a word, a glance, or a button press, as described above. As shown in Figure 8I, before the user utters commands 824A-824B in Figures 8J-8L, the table lamp 818 is already on and the floor lamp 814 is off.

[0238] In Figure 8J, the electronic device 800 detects that user 820 is looking at the floor lamp 814 and that user 820 utters part 824A "Turn..." of the second command 824A-824B "turn on that light". In Figure 8J, the electronic device 800 acquires second gaze information based on the user glancing at the floor lamp 814 via one or more camera sensors 816A, which may be located on an external device and / or integrated with the electronic device 800 in camera 816B. The second gaze information optionally includes one or more of the following: the position of the user's head, the time or duration of the gaze, the direction of the gaze, the field of view of the gaze, the direction of the gaze, and whether the floor lamp 814 is within the field of view of the gaze based on an offset angle.

[0239] The electronic device 800 determines whether the second gaze information acquired from one or more camera sensors 816A to 816B satisfies one or more sets of gaze criteria. If one or more sets of gaze criteria are satisfied for the floor lamp 814, the electronic device 800 can determine that the user 820 intended to apply the second commands 824A to 824B to the floor lamp 814. As described above, the set of one or more gaze criteria optionally includes a direction criterion that is satisfied for the floor lamp 814 when the gaze is directed toward the floor lamp 814. The set of one or more gaze criteria optionally includes a static time criterion that is satisfied when the electronic device 800 detects a static time or duration of gaze in the direction of the floor lamp 814 that exceeds a predetermined non-zero time (e.g., 2 seconds). The set of one or more gaze criteria optionally includes a field of view criterion. The field of view criterion is satisfied when the external device determines that the floor lamp 814 is within the user 820's field of view 808. The floor lamp 814 is determined to be within the user 820's field of view 808 when it is within a threshold offset angle 804 from the user's line of sight 812. In some embodiments, the field of view criterion is met when the floor lamp 814 is directly within the user 820's line of sight 812, as shown in Figure 8J. In some embodiments, the field of view criterion is met when the determined offset angle of the gaze is less than the threshold offset angle (e.g., the maximum allowable deviation from the user 820's line of sight 812, which is directly viewing the floor lamp 814). The threshold offset angle 804 is optionally greater than 0 degrees (e.g., 30 degrees). The determined offset angle is calculated based on the angle formed between the user 820's line of sight 812 and the calculated line of sight 808 from the user 820 to the electronic device 800. In some embodiments, the field of view criterion is met for the floor lamp 814 when it is within the gaze's field of view by being within a threshold degree range from the user's direct line of sight 812.

[0240] In some embodiments, as shown in Figure 8K, user 820 interrupts gazing at the floor lamp 814 while uttering the remaining portion 824B of the second commands 824A-824B. Figure 8K shows that the electronic device 800 detects that user 820 interrupts their gaze at the floor lamp 814 by averting their eyes from it, while uttering the second portion 824B of the second commands 824A-824B. In some embodiments, if the electronic device 800 has already determined that the floor lamp 814 is an intended external device before user 820 averts their eyes, then even if user 820 interrupts their gaze before completing the second commands 824A-824B, one or more sets of gazing criteria are still met. In some embodiments, if the electronic device 800 detects that the user 820 has returned their gaze to the floor lamp 814 within a threshold duration before completing commands 824A to 824B (for example, Figures 6E to 6G), the electronic device 800 still identifies the floor lamp 814 as an external device.

[0241] When one or more gaze criteria are met, the electronic device sends a command to the floor lamp 814, thereby transitioning it from the off state to the on state, as shown in Figure 8L. Since the table lamp 818 does not meet one or more gaze criteria, the table lamp 818 remains in its previous on state, and therefore the electronic device does not send a command to change the state of the table lamp 818. In some embodiments, if the electronic device 800 determines that command 824C received is directed to the floor lamp 814 and includes "turn off that light", then when one or more gaze criteria are met for the floor lamp 814, the electronic device 800 sends a command to turn off the floor lamp 814, while the table lamp 818, which was already on, remains on.

[0242] Figure 8L shows that when electronic device 800 sends a command to floor lamp 814, electronic device 800 optionally provides an indicator (e.g., indicators 810A, 810B, 810C) to indicate that floor lamp 814 has transitioned from an off state to an on state. In some embodiments, the indicator is a visual indicator 810A that displays light or a light pattern of a specific color (e.g., blue or a color other than the default color) to indicate that the table lamp 818 has been turned on from an off state as a result of receiving the command. In some embodiments, the indicator is a voice indicator 810B that plays a sound, one or more words, or a tone. In some embodiments, the digital assistant utters a confirmation 810C. In some embodiments, the user 820 can customize the voice and visual indicators to include characteristics associated with the user's user profile (e.g., username, specific user preferences for light color, pattern, sound, voice, and accent). In Figure 8K, the digital assistant provides a voice indicator saying, "Jack, the floor lamp is on." The voice indicator provides confirmation that the command has been executed. The voice indicator also includes the name of user 820, obtained from user 820's user profile, which user 820 provided during registration. The registration process is described in detail below with respect to Figures 10A to 10D. The digital assistant can further customize the indicator to reflect which user (e.g., Jack) uttered the command by utilizing settings from user 820's profile. In some embodiments, the electronic device 800 provides an indicator when the electronic device successfully sends a “turn on” command to the floor lamp 814, regardless of whether the floor lamp 814 is actually on. Optionally, the electronic device 800 provides a different indicator when the floor lamp 814 fails to execute the “turn on” command and remains off. In some embodiments, the indicator when command 824C is executed unsuccessfully includes different indicators for different error codes.

[0243] Figures 9A and 9B are flowcharts illustrating a method 900 using an electronic device according to several embodiments. The method 900 is performed on an electronic device having a display device (e.g., 100, 300, 500, 600, 640, 800, 1000). Some operations of the method 900 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0244] As will be described later, method 900 provides an intuitive way for a user (e.g., 820) to provide context regarding a voice user input request to a digital assistant on an electronic device (e.g., 800) by glancing at an external device (e.g., 814, 818) on which the voice user input request is intended to act. This method reduces the cognitive burden on the user (e.g., 820) when providing a voice user input request to the digital assistant (e.g., remembering to mention the external device in the command). Thus, the digital assistant can determine the external device (e.g., 814, 818) for a command by monitoring the user's gaze (e.g., 820).

[0245] While the first external device (e.g., 818) is in a first state (e.g., the first external device (e.g., 818) is on or off, or the first external device (e.g., 818) is playing a particular media item) (902), the electronic device (e.g., 800) receives a user input request (e.g., a user utterance) (904) for executing a first command (e.g., 824A-824B) (e.g., for transitioning the external device to a second state and telling the digital assistant to "turn on that light"). In some embodiments, the electronic device receives a voice user input request (904) while the digital assistant on the electronic device (e.g., 800) is active (e.g., based on a trigger word, gaze, and / or button press).

[0246] While the first external device (e.g., 818) is in a first state (902), the electronic device uses one or more camera sensors (e.g., 816A, 816B) (e.g., one or more infrared camera sensors, one or more visible light sensors) to acquire first gaze information from the electronic device (e.g., 800) and / or from external cameras (one or more) of the electronic device (e.g., 800), such as wirelessly connected to a part of the electronic device (e.g., 800) (906) (e.g., the electronic device (e.g., 800) detects gaze information directed towards the first external device (e.g., 818)). In some embodiments, the electronic device (e.g., 800) uses the first gaze information to determine whether one or more sets of gaze criteria are met for a particular external device.

[0247] Using the first gaze information, upon determination that one or more sets of gaze criteria are met for a first external device (e.g., 818) (e.g., equipment such as a lamp, television, or computer) (908) (and optionally, in response to an electronic device (e.g., 800) receiving a voice user input request), the electronic device (e.g., 800) sends a command (e.g., to the first external device (e.g., 818)) to transition the first external device (e.g., 818) from a first state to a second state (910) (e.g., a digital assistant performs an action on the external device (e.g., 818) by transitioning it from an on state to an off state or from an off state to an on state).

[0248] By using gaze detection to determine which external devices (e.g., 814, 818) are associated with a command (e.g., 824A-824B) when a set of one or more gaze criteria is met, the user is provided with a more efficient user interface that provides context to the digital assistant simply by looking at the external devices (e.g., 814, 818) to clarify which external devices (e.g., 814, 818) are intended, without having to answer follow-up questions from the digital assistant. The user interface of the digital assistant becomes more efficient by providing the digital assistant with an improved user interface that provides context to commands (e.g., 824A-824B) without the digital assistant having to interact with the user to obtain more information. In addition, by using gaze detection, the number of inputs required to activate the digital assistant and execute commands (e.g., 824A-824B) is reduced, thereby improving the usability of the digital assistant.

[0249] In some embodiments, according to a determination that one or more sets of gaze criteria are met for a first external device (e.g., 818) (e.g., angle, duration, and direction of gaze), the electronic device (e.g., 800) provides an instruction (e.g., 810A, 810B, 810c) (e.g., a visual instruction or an audible instruction) (912) indicating that one or more sets of gaze criteria are met (e.g., an instruction indicating that a first command (e.g., 824A-824B) is associated with the first external device (e.g., 818).

[0250] One or more sets of fixation criteria are met, and by providing an indication that the digital assistant has successfully sent a command (e.g., 824A - 824B) to an external device (e.g., 800), feedback is provided to the user regarding whether the command has been executed. The indication (e.g., 810A, 810B, 810C) can be a visual confirmation, an audio confirmation, or an audio confirmation from the digital assistant that provides customizable feedback regarding various contexts. For example, in a context where sound cannot be heard (e.g., a noisy room), by providing visual feedback, the user can know whether the digital assistant has sent a command (e.g., 824C). By providing customized feedback on the state of the digital device, the operability of the digital assistant is improved, and by notifying the user when the digital assistant has successfully sent a command to an external device, the user - device interface becomes more efficient (e.g., the user does not need to repeat the command because they do not know whether the command has been executed by the digital assistant). Further, feedback on the state of the digital assistant enables the user to use the device quickly and efficiently.

[0251] In some embodiments, the indication (e.g., 810A, 810B, 810C) (e.g., visual or audible indication) is domain - specific (914) (e.g., music domain, weather domain, messaging domain). In some examples, an electronic device (e.g., 800) determines that a command (e.g., 824C) is in a particular domain. In accordance with the determination that the command (e.g., 824C) is in a first domain, the indication (e.g., 810A, 810B, 810C) is of a first type. In accordance with the determination that the command (e.g., 824C) is in a second domain different from the first domain, the indication (e.g., 810A, 810B, 810C) is of a second type different from the first type of indication.

[0252] By providing domain-specific instructions, the digital assistant quickly notifies the user (e.g., 820) whether the command (e.g., 824C) was successfully sent. By associating a specific visual indicator, auditory indicator, or digital voice indicator with a specific domain, an improved user interface is quickly provided to the user that enables the user to quickly determine whether further action is required if the command (e.g., 824C) was not successfully executed. For example, if the user quickly says "love that song", the user (e.g., 820) can quickly know that the song has been added to the favorites list by a specific tone associated with the command, and the user does not need to check if it was added. This reduces the user's cognitive burden quickly because the association with a tone indicating that a specific action in a specific domain was successful provides an immediate confirmation that the task was executed. Further, feedback on the state of the digital assistant enables the user (e.g., 820) to use the digital assistant more quickly and efficiently.

[0253] In some embodiments, the electronic device (e.g., 800) is paired (916) with a first external device (e.g., 814, 818) (e.g., a paired lamp, television, computer, phone, or watch).

[0254] In some embodiments, using the first fixation information, in accordance with a determination that one or more sets of fixation criteria are not met for the first electronic device (e.g., 814, 818) (e.g., the user interrupts fixation, the user does not look at the external device for a threshold time), the electronic device (e.g., 800) withholds sending an instruction to transition the first external device (e.g., 818) from the first state to the second state. In some examples, the digital assistant does not execute a command (e.g., 824C) when one or more sets of fixation criteria are not met.

[0255] In some embodiments, a set of one or more activation criteria includes a field of view criterion that is satisfied when a determined offset angle (e.g., 810) is less than a threshold offset angle (e.g., 804) (e.g., the maximum acceptable deviation from the user directly viewing the device, a non-zero threshold offset angle, or a threshold offset angle greater than zero degrees), where the determined offset angle (e.g., 810) is the angle formed between the line of sight (e.g., 812) of the user's gaze (e.g., determined from gaze information) and the calculated line of sight (e.g., 808) from the user (e.g., 820) to the electronic device (e.g., 818). In some embodiments, if it is determined that the first external device (e.g., 818) is located beyond a threshold degree (e.g., outside the field of view of the gaze (e.g., 806)), the first external device (e.g., 818) is not within the user's line of sight and no command is sent to the first external device (e.g., 818). In some embodiments, the digital assistant sends a command to the first external device (e.g., 818) when the first external device (e.g., 818) is within the field of view of the gaze (e.g., 806) by being within a threshold range.

[0256] By using the field of view criterion as the gaze criterion, it becomes possible to detect the user's gaze without the user needing to directly look at an external device (e.g., 818). Instead, as long as the external device (e.g., 818) is within the user's field of view, the user's gaze satisfies the gaze criterion, thereby allowing the electronic device (e.g., 800) to determine which external device (e.g., 814, 818) is intended. Using the field of view criterion allows for the identification of external devices (e.g., 818) without requiring further interaction with the user, thereby improving the usability of the digital assistant and making gaze detection by the electronic device (e.g., 800) more efficient (e.g., the external device (e.g., 818) is more easily identified because it does not need to be within the user's (e.g., 820) direct line of sight).

[0257] In some embodiments, the gaze line (e.g., 812) of a user (e.g., 820) is based on the head position of the user (e.g., 820) (e.g., the same user who received an audio user input request from that user). In some embodiments, the user's head position may be influenced by the tilt or rotation (e.g., yaw, pitch, and roll) of the user's head. In some embodiments, head position tracking is used to determine whether a first external device (e.g., 818) is within the field of view (e.g., 806).

[0258] In some embodiments, a set of one or more gaze criteria includes a resting time criterion that is satisfied for a first external device (e.g., 818) when the resting time of a gaze (e.g., duration) determined from first gaze information exceeds a threshold time (e.g., a non-zero time). In some embodiments, when the resting time of a user's (e.g., 820) gaze is less than the threshold time.

[0259] In some embodiments, following the transmission of a command (e.g., to the first external device (e.g., 818)) by an electronic device (e.g., 800) to a first external device (e.g., 818) to transition it from a first state to a second state, the electronic device (e.g., 800) provides an indication (e.g., 810A, 810B, 810C) (e.g., audible or visual) that the first external device (e.g., 818) is in the second state (e.g., the electronic device (e.g., 800) provides a visual indication (e.g., 810A) that the first external device (e.g., 818) has been turned on). In some embodiments, the electronic device (e.g., 800) provides a different indication (e.g., 810A, 810B, 810C) when the command (e.g., 824C) could not be executed (e.g., the first external device (e.g., 818) is still in the first state).

[0260] In some embodiments, if a first external device (e.g., 818) is in a first state and a second external device is in a third state, and a set of one or more gaze criteria is met for the second external device (e.g., equipment such as a lamp, television, or computer), then according to a determination using first gaze information (and optionally in response to receiving a voice user input request), the electronic device (e.g., 800) sends a command (e.g., to the first external device (e.g., 818)) to transition the second external device (e.g., 814) from the third state to the fourth state based on a first command (e.g., 824C) (for example, the electronic device (e.g., 800) performs an action on the second external device (e.g., 814) by transitioning the second external device from an ON state to an OFF state, or from an OFF state to an ON state).

[0261] In accordance with a determination using the first gaze information that one or more sets of gaze criteria are not met for a second external device (e.g., 814) (e.g., equipment such as a lamp, television, or computer) (and optionally in response to receiving a voice user input request), the electronic device (e.g., 800) refrains from sending a command (e.g., to the first external device) to transition the second external device (e.g., 814) from a third state to a fourth state based on the first command (e.g., 824C) (for example, the electronic device (e.g., 800) refrains from performing any action on the second external device, such as refraining from transitioning the second external device from an on state to an off state, or from an off state to an on state).

[0262] In some embodiments, while a first external device (e.g., 818) is in a second state (e.g., the first external device 9, e.g., 818) may also be in a first state), and while a second external device (e.g., 814) is in a third state (918), an electronic device (e.g., 800) receives a second user input request (e.g., user utterance input) including a second command (e.g., 826C) (920).

[0263] An electronic device (e.g., 800) uses one or more camera sensors (e.g., 816A, 816B) to acquire second gaze information (922) (for example, to tell a digital assistant to turn on "that" light and cause a second external device (e.g., 814) to transition to a second state).

[0264] Based on a determination using second gaze information that one or more gaze criteria are met for the second external device (e.g., 814), the electronic device (e.g., 800) transmits a command (e.g., 924) that causes the second external device (e.g., 814) to transition from a third state to a fourth state (e.g., change the state of the second external device (e.g., 814), i.e., turn on the floor lamp), based on the second command (e.g., 824C). Meanwhile, the first external device (e.g., 818) remains in the second state (e.g., the table lamp remains on, without changing the state of the first external device).

[0265] In some embodiments, while a second external device (e.g., 814) is in a third state, and according to a determination using second gaze information that one or more sets of gaze criteria are not met for the second external device (e.g., 814) (e.g., the second external device is not identified), the electronic device (e.g., 800) refrains from sending a command to transition the second external device (e.g., 814) from the third state to a fourth state (926) (e.g., the second external device (e.g., 814) remains in the same state as before).

[0266] In some embodiments, following the transmission (e.g., to the second external device (e.g., 814)) of a command to transition the second external device (e.g., 814) from a third state to a fourth state, the electronic device (e.g., 800) provides an instruction (e.g., 810A, 810B, 810C) (e.g., a visual or audible instruction) indicating that the second external device (e.g., 814) is in the fourth state (e.g., providing a visual (e.g., 810A) instruction or an audible (e.g., 810B) instruction to indicate that the state of the second external device (e.g., 814) has changed). In some embodiments, the electronic device (e.g., 800) provides a different instruction (e.g., 810A, 810B, 810C) when the second command (e.g., 824C) could not be executed (e.g., the second external device (e.g., 813) is still in the third state).

[0267] In some embodiments, following the transmission of a command (e.g., to the first external device (e.g., 818)) to transition the first external device (e.g., 818) from a first state to a second state, the electronic device (e.g., 800) receives an input instruction (e.g., rotation of the crown (e.g., 832) on the third external device) from the third external device (e.g., a wristwatch or telephone). The electronic device (e.g., 800) transmits a second command to the first external device (e.g., 818) based on the recency of the electronic device (e.g., 800) commanding the first external device (e.g., 818) to transition the first external device (e.g., 818) (e.g., dimming the brightness of the table lamp after it has been turned on). In some embodiments, a third external device (e.g., 830) does not know which external device (e.g., 814, 818) the third external device (e.g., 840) is interacting with. The third external device (e.g., 830) only receives input and transmits user input to an electronic device (e.g., 800), and the electronic device (e.g., 800) knows which external device (e.g., 814, 818) to send a command to (e.g., dim the lights).

[0268] An additional user interface with further control over the external device (e.g., 818) is provided to the user (e.g., 820) by using a third external device (e.g., 830) to further control the nearest external device (e.g., 818) from which the electronic device (e.g., 800) sent a command.

[0269] By providing the user (e.g., 820) with additional user interfaces and mechanisms to control the device without the need for a digital assistant, a more efficient user interface is provided, reducing the number of interactions with the digital assistant required to execute commands. Without needing to interact with the digital assistant to process additional commands, the user (e.g., 820) can simply use an external device (e.g., 830) to provide precise input, which may sometimes be difficult to describe in words. By receiving additional user input based on input received from a third external device (e.g., 840), the operability of the digital assistant is improved, and the user interface for controlling the external device (e.g., 818) becomes more efficient, as no additional commands are required.

[0270] It should be noted that the process details described above with respect to Method 900 (for example, Figures 9A-9B) are also applicable in a similar manner to the methods described below / above. For example, Method 900 may optionally include one or more of the characteristics of the various methods described above with respect to Method 700 and described below with respect to Method 1100. For example, Method 900 may include one or more of the characteristics of the various methods described above and described below with reference to the processes in Figures 7A, 7B, 11A, and 11B. For brevity, these details will not be repeated below.

[0271] It should be understood that the specific order in which the operations are described in Figures 7A–7B and 11A–11B is illustrative, and is not intended to indicate that the described order is the only possible order in which those operations can be performed. Those skilled in the art will recognize various ways of rearranging the order of the operations described herein, as well as the exclusion of certain operations. For the sake of brevity, these details will not be repeated here. Furthermore, it should be noted that the aspects of methods and processes described throughout this description may be integrated with one another.

[0272] Figures 10A to 10D show exemplary user interfaces for providing various different indicators to show that the digital assistant recognizes various users (e.g., 1020, 1030) uttering commands in the room at the time the digital assistant is activated, according to some embodiments. The digital assistant is activated based on trigger words, gazing as described above with respect to method 700, button presses, the user raising their wrist while wearing a wearable electronic device (e.g., a wristwatch) (e.g., Figures 6E to 6G), and / or the user pointing an external device (e.g., a telephone) over the electronic device 1000. In some embodiments, in response to determining that a user (e.g., 1020, 1030) is looking at the electronic device 1000 (e.g., according to the determination that one or more sets of activation criteria are met using gaze information as described above), the electronic device 1000 activates the digital assistant, and once the digital assistant is activated, it receives commands. The techniques shown in Figures 10A to 10D also optionally include the techniques described with respect to aspects of Method 900 and Figures 8A to 8L, which illustrate an electronic device that uses the user's gaze to determine a specific external object (e.g., table lamp 1018, floor lamp) associated with a command. The techniques in these figures are used to illustrate the processes described later, including the processes in Figures 11A to 11B.

[0273] As shown in Figures 10A to 10D, users Jane 1030 and Jack 1020 are in the same room as the electronic device 1000. In Figure 10A, the digital assistant of the electronic device is activated by using an embodiment of the technique described above. The electronic device 1000 receives utterance commands (e.g., 1024A, 1024B) from one or more users (e.g., Jane 1030 and Jack 1020) and controls various external devices in the room, such as a floor lamp 1014 or a table lamp 1018. In response to the electronic device 1000 receiving a command (e.g., 1024A, 1024B), the electronic device 1000 determines the user identification information (e.g., 1020, 1030) of the specific user who uttered the command (e.g., 1024A, 1024B). In some embodiments, an external device (e.g., an external camera 1016A, a telephone, or an integrated camera 1016B) determines user identification information corresponding to a command and transmits the user identification information to the electronic device 1000. In some embodiments, the electronic device 1000 acquires information about the speaking user using an external or internal device (e.g., an external camera 1016A, a telephone, or an integrated camera 1016B). In some embodiments, the electronic device 1000 uses the acquired information to perform face recognition, voice recognition, or calculate the distance to the speaking user to determine user identification information.

[0274] In some embodiments, before the electronic device activates the digital assistant, the users of Jane 1030 and Jack 1020 optionally register a user profile stored in the electronic device 1000. In some embodiments, the user profile is stored on a server and is accessible by the electronic device 1000. The user profile includes settings and preferences (e.g., the type of accent of the digital assistant's voice, the gender of the digital assistant's voice, the tone of the indicator, the color of the LED light) that the electronic device can use to customize the indicators 1010A, 1010B, 1010C used by the electronic device to indicate whether the digital assistant is activated, or whether commands 1024A, 1024B have been successfully or unsuccessfully executed. For example, the user can customize the digital indicator settings by providing values for each characteristic of the indicator. In some examples, the characteristics of the indicator include type (e.g., visual, audio, voice confirmation, or any combination of types), the color of the light for visual indicators, the tone of the voice indicator, and customizations related to digital voice (e.g., language, accent, and type of gender). The user profile also optionally stores information about the user, such as the user's name, birthday, gender, and / or address, which is provided by the users Jane 1030 and Jack 1020 during registration with the user's consent. The user profile settings and preferences may be updated at any time. The techniques for user registration on the electronic device 1000 described herein may be used in combination with methods 700 and 900, referring to the processes of FIGS. 7A, 7B, 9A, and 9B above. For the sake of brevity, these details are not repeated here.

[0275] In Figure 10A, when the digital assistant is activated, the electronic device 1000 receives a command 1024A from Jane 1030, "Turn on the table lamp." When the electronic device 100 receives the command 1024A spoken by Jane 1030 (or after receiving the command 1024A spoken by Jane 1030), the electronic device 1000 determines that the speaker is Jane 1030. For example, using an external camera 1016B, an integrated camera 1016B, and / or a microphone, the device 1000 determines that the received command 1024A is coming from the direction in which Jane 1030 is sitting. Therefore, the electronic device 1000 displays a light pattern 1010D (e.g., three illumination dots) on the display 1002 at a position corresponding to the direction in which Jane 1030 is sitting. In some embodiments, the electronic device 1000 displays an animated light pattern 1010D on a display 1002, the light pattern being animated to point in the direction in which Jane 1030 is sitting.

[0276] In Figure 10A, when the digital assistant determines that the utterancer of command 1024A is Jane 1030, the electronic device 1000 updates the values ​​of the characteristics of the indicator (e.g., color of light, specific sound, word, or voice of another digital assistant) to values ​​corresponding to Jane 1030 (e.g., purple light, two-tone tone, Jane's name, British accent, female voice). In some embodiments, the characteristic values ​​are derived from Jane's user profile. For example, the electronic device sets the color of the light pattern 1010D (e.g., three illumination dots) on the display 1002 to a color corresponding to Jane 1030 (e.g., purple). As a result, the electronic device 1000 indicates to Jane 1030 and Jack 1020 that the device 1000 has recognized the speaker as Jane 1030 and that one or more of Jane 1030's preferences will be used.

[0277] In contrast, in Figure 10C, when the digital assistant determines that the utterer of command 1024B is Jack 1020, a different user from Jane 1030, the electronic device 1000 updates the values ​​of the indicator characteristics (e.g., color of light, sound, or digital assistant voice) to values ​​corresponding to Jack 1020 (e.g., blue light, tone of one pitch, Jack's name, Australian accent, male voice), and the values ​​of the characteristics associated with Jack 1020 are different from the values ​​of the characteristics associated with Jane 1030. For example, the electronic device sets the color of the light pattern 1010E (e.g., three illumination dots) on the display 1002 to a color corresponding to Jack 1020 (such as blue). As a result, the electronic device 1000 indicates to Jane 1030 and Jack 1020 that the device 1000 recognizes the utterer as Jack 1020 and that one or more of Jack 1020's preferences are being used.

[0278] In some embodiments, Jack 1020 and Jane 1030 may be associated with each person and have the same characteristic values. For example, both Jack 1020 and Jane 1030 can select the voice of an American female digital assistant. In some embodiments, the characteristic values ​​of the indicators associated with each user are unique. For example, Jane 1030 and Jack 1020 may have different light colors (e.g., purple for Jane and blue for Jack) and different sounds (e.g., a two-tone sound versus a one-tone sound) associated with their respective users.

[0279] In Figure 10B, after the electronic device 1000 identifies that Jane 1030 has issued command 1024A, the electronic device 1000 determines whether one or more sets of execution criteria are met, as described above with respect to method 900. One or more sets of execution criteria include one or more of the completion criteria, the implementability criteria, and the activation criteria. If one or more sets of execution criteria are met, the electronic device 1000 takes an action, such as sending a command to turn on the floor lamp 1014.

[0280] When electronic device 1000 executes command 1024A, electronic device 1000 optionally provides an indicator (e.g., indicators 1010A, 1010B, 1010C) using Jane's 1030 preferences to indicate that the command was successfully executed. The types of indicators (e.g., visual confirmation, voice confirmation, and digital voice confirmation) and their characteristics are described above with respect to methods 700 and 900. In Figure 10B, electronic device 1000 determines that the speaker is Jane 1030 and, based on that determination, provides confirmation using a female digital assistant voice confirmation in an American accent, "Jane, the floor lamp has been turned on." The name "Jane" is obtained from Jane 1030's user profile provided by Jane 1030 during registration. Electronic device 1000 uses a female American accent based on previous preferences stored as part of Jane's 1030 user profile. A purple light pattern may be displayed on the electronic device 1000 as a visual indicator to show that the digital assistant has processed Jane's command 1024A. The tone associated with Jane 1030 may also be played as an audio indicator on the electronic device 1000 to show that the digital assistant has processed Jane's command 1024A. As a result, both Jane 1030 and Jack 1020 recognize that Jane has been identified as the speaker.

[0281] Similarly, Figure 10C shows that Jack 1020 is uttering the command 1024B, “Turn on the floor lamp.” The electronic device 1000, using, for example, an external camera 1016A, an integrated camera 1016B, and / or a microphone, determines that the speaker is Jack 1020 and that the command 1024B is coming from the direction in which Jack 1030 is sitting. In response, the electronic device 1000 displays a light pattern 1010E (e.g., three illumination dots) on the display 1002 at a position corresponding to the direction in which Jack 1020 is standing. In some embodiments, the electronic device 1000 displays an animated light pattern 1010E on the display 1002, and the light pattern is animated to point in the direction in which Jack 1020 is standing. For example, the electronic device sets the color of the light pattern 1010E (e.g., three illumination dots) on the display 1002 to a color corresponding to Jack 1020 (e.g., blue). As a result, the electronic device 1000 indicates to Jane 1030 and Jack 1020 that the device 1000 recognizes the speaker as Jack 1020 and that one or more of Jack 1020's preferences will be used.

[0282] In Figure 10D, after the electronic device 1000 identifies that the jack 1020 has issued command 1024B, the electronic device 1000 determines whether one or more sets of execution criteria are met, as described above with respect to method 900. One or more sets of execution criteria include one or more of the completion criteria, the implementability criteria, and the activation criteria. If one or more sets of execution criteria are met, the electronic device 1000 performs an action, such as sending a command to turn on the floor lamp 1014.

[0283] When electronic device 1000 executes command 1024B, electronic device 1000 optionally provides indicators (e.g., indicators 1010A, 1010B, 1010C) using Jack's preferences to indicate that the command was successfully executed. The types of indicators (e.g., visual confirmation, voice confirmation, and digital voice confirmation) and their characteristics are described above with respect to methods 700 and 900. In Figure 10D, electronic device 1000 determines that the speaker is Jack 1020 and, based on that determination, provides confirmation using a male digital assistant voice with an Australian accent, saying, "Jack, the table lamp has been turned on." The name "Jack" is derived from Jack's 1030 user profile provided by Jack 1020 during registration. Electronic device 1000 uses a male Australian accent based on previous preferences stored as part of Jack's 1020 user profile. A blue light pattern may be displayed as a visual indicator on the electronic device 1000 to indicate that the digital assistant has processed Jack's command 1024B. The electronic device may optionally play a custom tone associated with Jack 1020 as an audio indicator to indicate that the digital assistant has processed Jack's command 1024B.

[0284] Figures 11A and 11B are flowcharts illustrating method 1100 using an electronic device according to several embodiments. Method 1100 is performed using an electronic device (e.g., 100, 300, 500, 600, 640, 800, 1000). Some operations of method 1100 are arbitrarily combined, the order of some operations is arbitrarily changed, and some operations are arbitrarily omitted.

[0285] As described below, Method 1100 provides an intuitive way for a digital assistant to indicate the recognition of different users (e.g., 1020, 1030) by providing indicators unique to each person. This method reduces the cognitive burden on users (e.g., 1020, 1030) because users can confirm that the digital assistant has processed a command for a specific user when multiple users are uttering commands in the room. Thus, the digital assistant can provide immediate feedback to indicate which user's command has been processed.

[0286] After a digital assistant on an electronic device (e.g., 1000) is activated based on a trigger word, gaze, and / or button press, the electronic device 1000 receives a voice user input request (e.g., 1024A, 1024B) (1106). In some embodiments, upon receiving the voice user input request (e.g., 1024A, 1024B), the electronic device (e.g., 1000) determines user identification information (e.g., a specific user (e.g., 1020, 1030)) corresponding to the voice user input request (e.g., 1024A, 1024B). In some embodiments, a remote device (e.g., camera 1016) determines the user identification information corresponding to the voice user input request (e.g., 1024A, 1024B), and the electronic device (e.g., 1000) receives the result of the determination.

[0287] In some embodiments, before an electronic device (e.g., 1000) receives a voice user input request (e.g., 1024A, 1024B), the electronic device (e.g., 1000) receives registration information (e.g., user profile information) (1102). In some embodiments, the electronic device (e.g., 1000) uses the registration information to associate a first value of a characteristic (e.g., light, sound, digital assistant voice) of an indicator (e.g., 1010A, 1010B) of the electronic device (e.g., 1000) with a first user (e.g., 1020) (and not with a second user), and to associate a second value of a characteristic of an indicator of the electronic device (e.g., 1000) with a second user (e.g., 1030) (and not with a second user). In some embodiments, during the registration process, an electronic device (e.g., 1000) receives the user's name and then associates that name with the user's (e.g., 1020, 1030) voice. In some embodiments, during the registration process, an electronic device (e.g., 1000) associates the color of a visual indicator (e.g., LED color) with the user (e.g., 1020, 1030).

[0288] By associating the characteristic values ​​of indicators (e.g., 1010A, 1010B, 1010C) with preferences within the user's user profile, the digital assistant can provide feedback to confirm that it has identified a specific user (e.g., 1020, 1030) from whom it is issuing a command. Providing customized feedback to confirm the user (e.g., 1020, 1030) improves the usability of the digital assistant, and providing the user (e.g., 1020, 1030) with confirmation using the user's preferences in the indicators (e.g., 1010A, 1010B, 1010C) makes the user-device interface more efficient (e.g., the user (e.g., 1020, 1030) knows that the digital assistant has processed the user's command (1024A, 1024B)). This reduces the cognitive burden on the user (e.g., 1020, 1030) by providing immediate confirmation that a task has been completed by associating a success indicator with a specific user's settings, and also reduces confusion when multiple users are speaking simultaneously or when other speakers are present in the background. Furthermore, feedback on the digital assistant's status enables the user (e.g., 1020, 1030) to use the digital assistant more quickly and efficiently.

[0289] The electronic device receives a voice-based user input request (1106).

[0290] In some embodiments, the voice user input request (e.g., 1024A, 1024B) is user utterance input (e.g., a statement, a command), and the update of the value of a characteristic (e.g., light, sound, voice of a digital assistant) (1108) is not based on the transcribed content of the voice user input request (e.g., 1024A, 1024B) (e.g., the value of the characteristic is not based on the words spoken by the user, but rather on the user's identifying information).

[0291] In accordance with the determination that a voice user input request corresponds to a first user (e.g., 1020) (and optionally, in response to having received a voice user input request), the electronic device (e.g., 1000) updates the values ​​of the characteristics (e.g., light color, sound, digital assistant voice) of the indicator (e.g., 1010A, 1010B, 1010C) (e.g., visual indicator, voice indicator, or digital assistant voice confirmation) to first values ​​corresponding to the first user (e.g., 1020) (1110) (e.g., the characteristic values ​​are associated with the identification information of the first user).

[0292] In accordance with the determination that a voice user input request corresponds to a second user (e.g., 1030) different from the first user (and optionally, in response to having received a voice user input request), the electronic device (1000) updates the values ​​of the characteristics (e.g., light color, sound, digital assistant voice) of the indicator (e.g., 1010A, 1010B, 1010C) (e.g., visual indicator, voice indicator, or voice confirmation of a digital assistant) to second values ​​corresponding to the second user (e.g., 1030) (1112) (e.g., the characteristics values ​​are associated with the identification information of the second user), the second values ​​being different from the first values.

[0293] An electronic device (e.g., 1000) responds to voice user input requests (e.g., 1024A, 1024B) using indicators (e.g., 1010A, 1010B, 1010C) (1114) (for example, the electronic device (e.g., 1000) displays light and plays sound using a specific digital assistant voice or the user's name in the response). The indicators (e.g., 1010A, 1010B, 1010C) include updated values ​​for characteristics (e.g., light color, sound, or digital assistant voice).

[0294] In some embodiments, an electronic device (e.g., 1000) that responds to voice user input requests (e.g., 1024A, 1024B) using indicators (e.g., 1010A, 1010B, 1010C) includes an electronic device (e.g., 1000) that displays a visual indicator (e.g., 1010A) (e.g., light) using values ​​of user-corresponding characteristics (e.g., color of light, position of displayed light) associated with the voice user input requests (e.g., 1024A, 1024B) (1116).

[0295] In some embodiments, an electronic device (e.g., 1000) that uses an indicator to respond to a voice user input request (e.g., 1024A, 1024B) includes an electronic device (e.g., 1000) that provides a voice indicator (e.g., 1010B) (e.g., voice, sound) using a characteristic value (the characteristic value may be accent, gender, pitch, voice, sound, or a specific digital assistant voice, such as the voice of an American female digital assistant) associated with the voice user input request (e.g., 1024A, 1024B) (1118). In some instances, the user selected the voice of an American woman digital assistant within the digital assistant voice settings, as indicated in the user's user profile (e.g., 1020, 1030).

[0296] In some embodiments, the indicator is a visual indicator (e.g., 1010A), and in response to the electronic device (e.g., 1000) receiving a voice user input request (e.g., 1024A, 1024B), and before determining that the voice user input request (e.g., 1024A, 1024B) corresponds to a specific user (e.g., 1020, 1030), the electronic device (e.g., 1000) displays a visual indicator (e.g., 1010A) having a default value for the characteristic (e.g., white light) (e.g., displaying a first light color when the digital assistant is activated). In some embodiments, the default value for the characteristic is different from the first and second values.

[0297] In some embodiments, the characteristics of a voice indicator (e.g., 1010B) (e.g., a digital assistant providing confirmation using a word) are the terms (e.g., the word, name) corresponding to the user (e.g., providing it) associated with the voice user input request (e.g., 1024A, 1024B).

[0298] By using the user's name (e.g., 1020, 1030) within the digital assistant's voice confirmation indicator (e.g., 1010C), the digital assistant can provide feedback to confirm that it has identified the specific user (e.g., 1020, 1030) who is issuing the command. Providing customized feedback to confirm the user (e.g., 1020, 1030) improves the usability of the digital assistant and makes the user-device interface more efficient (e.g., a specific user (e.g., 1020, 1030) knows that the digital assistant has processed their command (e.g., 1024A, 1024B)). This reduces the cognitive burden on the user (e.g., 1020, 1030) by providing immediate confirmation that the task has been completed, and reduces confusion when multiple users (e.g., 1020, 1030) are speaking simultaneously or when other speakers are present in the background. Furthermore, feedback on the digital assistant's status allows users (e.g., 1020, 1030) to use the digital assistant more quickly and efficiently.

[0299] In some embodiments, an electronic device (e.g., 1000) determines the user associated with a voice user input request based on one or more of the following: speech recognition, facial recognition, and the direction of the voice user input request.

[0300] The above is written with reference to specific embodiments for illustrative purposes. However, the above exemplary discussion is not intended to be exhaustive or to limit the invention to the exact form disclosed. Many modifications and variations are possible in light of the above teachings. Embodiments have been selected and described to best illustrate the principles of the Art and their practical applications. This will enable other persons skilled in the art to best utilize the Art and its various embodiments with various modifications suitable for their intended specific use.

[0301] While the present disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. Such changes and modifications should be understood to fall within the scope of the present disclosure and examples as defined by the claims.

[0302] As described above, one aspect of this technology involves collecting and using data available from various sources to better enable a user to control their electronic device. This disclosure considers that, in some examples, such collected data may include personal information data that uniquely identifies a particular person, or personal information data that can be used to contact a particular person or locate them. Such personal information data may include demographic data, location-based data, telephone numbers, email addresses, Twitter IDs, home addresses, data or records relating to a user's health or fitness level (e.g., vital signs measurements, medication information, exercise information), birth dates, or any other identifying or personal information.

[0303] This disclosure acknowledges that the use of such personal data in the technology may be beneficial to the user. For example, personal data can be used to personalize device interactions. Thus, the use of such personal data allows users to have greater control over electronic devices. Furthermore, other uses of personal data that benefit users are also considered in this disclosure. For example, health and fitness data can be used to provide insights into a user's overall wellness, or as positive feedback to individuals using the technology to pursue wellness goals.

[0304] This disclosure considers that entities involved in the collection, analysis, disclosure, transfer, storage, or other use of such personal data should adhere to a robust privacy policy and / or privacy practice. Specifically, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or government requirements for the strict confidentiality of personal data. Such policies should be readily accessible to users and should be updated as data collection and / or use changes. Personal data from users should be collected for the lawful and legitimate use of the entity and should not be shared or sold for any other purpose. Furthermore, such collection / sharing should be carried out only after informing and obtaining the user's consent. Furthermore, such entities should consider taking all necessary steps to protect and secure access to such personal data and to ensure that others who have access to such personal data comply with those privacy policies and procedures. Furthermore, such entities may undergo third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. Furthermore, policies and practices should be adapted to the specific types of personal data being collected and / or accessed, and should comply with applicable laws and standards, including jurisdiction-specific considerations. For example, in the United States, the collection or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to and should be addressed accordingly. Therefore, different privacy practices should be maintained for different types of personal data in each country.

[0305] Notwithstanding the foregoing, the Disclosure also considers embodiments that allow a user to selectively prevent the use of or access to personal data. Specifically, the Disclosure considers that hardware and / or software elements can be provided to prevent or prevent access to such personal data. For example, in the case of a digital assistant, user gaze, and user profile, the technology could be configured so that a user can choose to “opt in” or “opt out” of participating in the collection of personal data during or at any time thereafter. In another embodiment, the user can choose not to provide personal data. In yet another embodiment, the user can choose to limit the period for which personal data is retained or to completely prohibit the deployment of a baseline profile. In addition to providing “opt-in” and “opt-out” options, the Disclosure considers providing notices regarding access to or use of personal data. For example, the user may be notified when downloading an app that accesses the user’s personal data (e.g., a digital assistant app), and then the user may be reminded again immediately before the app accesses the personal data.

[0306] Furthermore, the intent of this disclosure is that personal data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. This risk can be minimized by limiting data collection and deleting data when it is no longer needed. Additionally, where applicable, data de-identification can be used in certain health-related applications to protect user privacy. De-identification can be facilitated, where appropriate, by removing specific identifiers (e.g., date of birth), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data across users), and / or by other means.

[0307] Therefore, while this disclosure broadly covers the use of personal data to implement one or more of the disclosed embodiments, the disclosure also considers that it is possible to implement those embodiments without requiring access to such personal data. In other words, the various embodiments of the technology are not rendered inoperable by the absence of all or part of such personal data. For example, a user may interact with an electronic device based on non-personal data or a minimal amount of personal data.

Claims

1. It is a method, In electronic devices, While the digital assistant of the aforementioned electronic device is not activated, Using one or more camera sensors to acquire first gaze information, According to the determination that the first gaze information satisfies one or more activation criteria, Activating the digital assistant of the electronic device, To provide an indication that one or more of the aforementioned activation criteria are met, Methods that include...

2. In accordance with the determination that the first gaze information does not satisfy the set of one or more activation criteria, The activation of the aforementioned digital assistant will be postponed, The method according to claim 1, further comprising:

3. Receiving voice-based user input requests to execute commands, In response to receiving the voice user input request in order to execute the command, In accordance with the determination that one or more sets of execution criteria are met, the command is executed based on the content of the voice-based user input request, The method according to claim 1 or 2, further comprising:

4. The method according to claim 3, wherein the electronic device begins receiving the voice user input request while the digital assistant is not activated.

5. Before receiving the aforementioned voice-based user input request, an audio output is generated at a first volume level. Upon initiating the reception of the aforementioned voice-based user input request, To reduce the volume of the audio output to a second volume that is lower than the first volume and is based on the distance between the user and the electronic device, The method according to claim 3 or 4, further comprising:

6. Upon receiving the voice user input request for executing the command, The execution of the command based on the content of the voice user input request is withheld in accordance with the determination that the set of one or more execution criteria, which includes a continuous gaze criterion, is not met when the electronic device determines that the second gaze information indicates that the user will not interrupt gazing toward the electronic device for a threshold duration between the activation of the digital assistant and the termination of the voice user input request. The method according to any one of claims 3 to 5, further comprising:

7. Upon receiving the voice user input request for executing the command, In accordance with the determination that one or more of the aforementioned sets of execution criteria are not met, instructions are given by head shake, Displaying a first light pattern on the first side of the display of the electronic device, Following the display of the first light pattern, a second light pattern is displayed on the second side of the display of the electronic device, Following the display of the second light pattern, the first light pattern is displayed on the first side of the display of the electronic device, The method according to any one of claims 3 to 6, which provides instructions by head shaking, including the head shake.

8. The method according to any one of claims 1 to 7, wherein the instruction is a visual instruction indicating that the digital assistant is activated.

9. The method according to any one of claims 1 to 8, wherein the instruction is an audio instruction indicating that the digital assistant is activated.

10. The set of one or more activation criteria includes a field of view criterion that is satisfied when the determined offset angle is less than a threshold offset angle, and the determined offset angle is The user's gaze and, The calculated line of sight from the user to the electronic device and The method according to any one of claims 1 to 9, wherein the angle is formed between them.

11. The method according to claim 10, wherein the line of sight of the user's gaze is based on the position of the user's head.

12. The method according to any one of claims 1 to 11, wherein the set of one or more activation criteria includes a resting time criterion that is satisfied when the resting time of the gaze determined from the first gaze information exceeds a threshold time.

13. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of an electronic device, wherein the one or more programs include instructions for performing the method according to any one of claims 1 to 12.

14. One or more processors, A memory that stores one or more programs configured to be executed by one or more processors, An electronic device comprising, wherein one or more programs include instructions for performing the method according to any one of claims 1 to 12.

15. An electronic device comprising means for performing the method described in any one of claims 1 to 12.

16. A non-temporary computer-readable storage medium that stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the one or more programs are While the digital assistant of the aforementioned electronic device is not activated, A command to acquire first gaze information using one or more camera sensors, According to the determination that the first gaze information satisfies one or more activation criteria, A command to activate the digital assistant of the electronic device, A non-temporary computer-readable storage medium comprising: an instruction that provides an indication that one or more of the aforementioned sets of activation criteria are met;

17. One or more processors, An electronic device comprising: a memory that stores one or more programs configured to be executed by one or more processors, wherein the one or more programs are While the digital assistant of the aforementioned electronic device is not activated, A command to acquire first gaze information using one or more camera sensors, According to the determination that the first gaze information satisfies one or more activation criteria, A command to activate the digital assistant of the electronic device, An electronic device comprising: an instruction that provides an indication that one or more of the aforementioned sets of activation criteria are met;

18. While the digital assistant of the aforementioned electronic device is not activated, A means for acquiring first gaze information using one or more camera sensors, According to the determination that the first gaze information satisfies one or more activation criteria, The digital assistant of the electronic device is activated, Means for providing an indication that one or more of the aforementioned sets of activation criteria are met, An electronic device equipped with the following features.

19. It is a method, In electronic devices, While the first external device is in the first state, Receiving a voice user input request to execute the first command, Using one or more camera sensors to acquire first gaze information, According to the determination that one or more sets of gaze criteria are met for the first external device using the first gaze information, Based on the first command, a command is transmitted to transition the first external device from the first state to the second state, Methods that include...

20. Based on the determination that one or more sets of gaze criteria are not met for the first external device, using the first gaze information, To refrain from transmitting the command that causes the first external device to transition from the first state to the second state, The method according to claim 19, further comprising:

21. The set of one or more gaze criteria includes a field of view criterion that is satisfied when the determined offset angle is less than a threshold offset angle, and the determined offset angle is The user's gaze and, The method according to claim 19 or 20, wherein the angle is formed between the user and the calculated line of sight to the first external device.

22. The method according to claim 21, wherein the line of sight of the user's gaze is based on the position of the user's head.

23. The method according to any one of claims 19 to 22, wherein the set of one or more gaze criteria includes a resting time criterion that is satisfied for the first external device when the resting time of the gaze determined from the first gaze information exceeds a threshold time.

24. In accordance with the determination that the set of one or more gaze criteria is satisfied for the first external device, an indication that the set of one or more gaze criteria is satisfied is provided. The method according to any one of claims 19 to 23, further comprising:

25. Following the transmission of the command to transition the first external device from the first state to the second state, an instruction is provided indicating that the first external device is in the second state. The method according to any one of claims 19 to 24, further comprising:

26. The method according to claim 24 or 25, wherein the instructions are domain-specific.

27. The method according to any one of claims 19 to 26, wherein the electronic device is paired with the first external device.

28. While the first external device is in the first state and the second external device is in the third state, Based on the determination that the set of one or more gaze criteria is met for the second external device using the first gaze information, Based on the first command, a command is transmitted to transition the second external device from the third state to the fourth state, Based on the determination that one or more sets of gaze criteria are not met for the second external device, using the first gaze information, Based on the first command, the transmission of a command to transition the second external device from the third state to the fourth state is withheld, The method according to any one of claims 19 to 27, further comprising:

29. While the first external device is in the second state and the second external device is in the third state, Receiving a second user input request that includes a second command, Using one or more camera sensors to acquire second gaze information, Based on the determination that the set of one or more gaze criteria is met for the second external device using the second gaze information, Based on the second command, while the first external device remains in the second state, a command is transmitted to transition the second external device from the third state to the fourth state, The method according to any one of claims 19 to 28, further comprising:

30. While the second external device is in the third state, In response to a determination that one or more sets of gaze criteria, using the second gaze information, are not met for the second external device, The transmission of the command to cause the second external device to transition from the third state to the fourth state is withheld, The method according to claim 29, further comprising:

31. The transmission of the command to cause the second external device to transition from the third state to the fourth state, followed by providing an instruction indicating that the second external device is in the fourth state, The method according to any one of claims 28 to 30, further comprising:

32. Following the transmission of the command to transition the first external device from the first state to the second state, Receiving input instructions from the third external device, Based on the latest status of the electronic device commanded to the first external device, a second command is transmitted to the first external device, The method according to any one of claims 19 to 31, further comprising:

33. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of an electronic device, wherein the one or more programs include instructions for performing the method according to any one of claims 19 to 32.

34. One or more processors, An electronic device comprising: a memory that stores one or more programs configured to be executed by one or more processors, wherein the one or more programs are instructions for performing the method according to any one of claims 19 to 32. Electronic devices, including those mentioned above.

35. Means for carrying out the method described in any one of claims 19 to 32 An electronic device equipped with the following features.

36. A non-temporary computer-readable storage medium that stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the one or more programs are While the first external device is in the first state, A command to receive a voice user input request for executing the first command, A command to acquire first gaze information using one or more camera sensors, Based on the determination that one or more sets of gaze criteria are met for the first external device using the first gaze information, A non-temporary computer-readable storage medium, comprising: an instruction that transmits an instruction to transition the first external device from a first state to a second state based on the first command;

37. One or more processors, An electronic device comprising: a memory that stores one or more programs configured to be executed by one or more processors, wherein the one or more programs are While the first external device is in the first state, A command to receive a voice user input request for executing the first command, A command to acquire first gaze information using one or more camera sensors, Based on the determination that one or more sets of gaze criteria are met for the first external device using the first gaze information, An electronic device comprising: an instruction that transmits an instruction to transition the first external device from a first state to a second state based on the first command.

38. While the first external device is in the first state, Upon receiving a voice-based user input request to execute the first command, A means for acquiring first gaze information using one or more camera sensors, According to the determination that one or more sets of gaze criteria are met for the first external device using the first gaze information, A means for transmitting a command to transition the first external device from the first state to the second state based on the first command, Electronic devices, including those mentioned above.

39. It is a method, In electronic devices, Receiving voice-based user input requests, In accordance with the determination that the voice-based user input request corresponds to a first user, the value of the indicator's characteristic is updated to a first value corresponding to the first user. In accordance with the determination that the voice-based user input request corresponds to a second user different from the first user, the value of the characteristic of the indicator is updated to a second value different from the first value, which corresponds to the second user. Responding to a voice user input request using the indicator which includes the updated value of the characteristic, Methods that include...

40. The method according to claim 39, wherein the voice-based user input request is a user utterance, and updating the value of the characteristic is not based on the content of the voice-based user input request.

41. The aforementioned indicator is a visual indicator, In response to receiving the voice user input request, and before determining that the voice user input request corresponds to a specific user, the visual indicator is displayed with a default value for the characteristic that is different from the first and second values. The method according to claim 39 or 40, further comprising:

42. Before receiving the voice-based user input request, the system receives registration information, and uses the registration information to associate the first value of the characteristic with the first user and the second value of the characteristic with the second user. The method according to any one of claims 39 to 41, further comprising:

43. Determining the user associated with a voice-based user input request based on one or more of the following: speech recognition, facial recognition, and the direction of the voice-based user input request. The method according to any one of claims 39 to 42, further comprising:

44. The method according to any one of claims 39 to 43, wherein responding to the voice user input request using the indicator includes displaying a visual indicator using the value of the characteristic corresponding to the user associated with the voice user input request.

45. The method according to any one of claims 39 to 44, wherein responding to a voice user input request using the indicator includes providing a voice indicator using the value of the characteristic corresponding to the user associated with the voice user input request.

46. The method of claim 45, wherein the characteristic of the voice indicator is a term corresponding to the user associated with the voice user input request.

47. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of an electronic device, wherein the one or more programs include instructions for performing the method according to any one of claims 39 to 46.

48. One or more processors, An electronic device comprising: a memory for storing one or more programs configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the method according to any one of claims 39 to 46.

49. Means for carrying out the method described in any one of claims 39 to 46 An electronic device equipped with the following features.

50. A non-temporary computer-readable storage medium that stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the one or more programs are A command to receive a voice-based user input request, A command to update the value of the indicator characteristic to a first value corresponding to the first user, in accordance with the determination that the voice-based user input request corresponds to a first user, A command to update the value of the characteristic of the indicator to a second value corresponding to the second user, which is different from the first value, in accordance with the determination that the voice user input request corresponds to a second user different from the first user, A non-temporary computer-readable storage medium comprising: an indicator, which includes an instruction to respond to a voice user input request using the indicator, which includes the updated value of the characteristic;

51. One or more processors, An electronic device comprising: a memory that stores one or more programs configured to be executed by one or more processors, wherein the one or more programs are A command to receive a voice-based user input request, A command to update the value of the indicator characteristic to a first value corresponding to the first user, in accordance with the determination that the voice-based user input request corresponds to a first user, A command to update the value of the characteristic of the indicator to a second value corresponding to the second user, which is different from the first value, in accordance with the determination that the voice user input request corresponds to a second user different from the first user, An electronic device comprising: an indicator, which includes an instruction to respond to a voice user input request using the indicator, which includes the updated value of the characteristic.

52. A means for receiving user input requests by voice, A means for updating the value of the indicator's characteristic to a first value corresponding to the first user, in accordance with the determination that the voice-based user input request corresponds to a first user, A means for updating the value of the characteristic of the indicator to a second value corresponding to the second user, which is different from the first value, in accordance with the determination that the voice-based user input request corresponds to a second user different from the first user, The indicator includes means for responding to a voice user input request using the indicator, which includes the updated value of the characteristic, An electronic device equipped with the following features.

Images