User interface for managing controllable external devices

Abstract

To provide a faster and more efficient method, computer program and electronic device for managing a controllable external device.SOLUTION: A method includes in an electronic device: displaying a user interface corresponding to an automation profile; detecting a user input set; displaying a map view including at least one indication of a designated location-based standard; associating the designated location-based standard with an automation standard of an automation profile associated with a first location in response to a detection of the user input set; associating a time-based standard with an automation standard for controlling the controllable external device in response to the detection of the user input set; and sending an instruction to set the controllable external device in a designated state in accordance with a determination that the automation standard is satisfied.SELECTED DRAWING: Figure 19

Description

Technical Field

[0001] Cross-reference to Related Applications This application claims priority to U.S. Provisional Patent Application No. 62 / 349,057, filed Jun. 12, 2016, entitled "USER INTERFACE FOR MANAGING CONTROLLABLE EXTERNAL DEVICES", the entire contents of which are incorporated herein by reference. This application claims priority to U.S. Non-Provisional Patent Application No. 15 / 427,516, filed Feb. 8, 2017, entitled "USER INTERFACE FOR MANAGING CONTROLLABLE EXTERNAL DEVICES", the entire contents of which are incorporated herein by reference. This application claims priority to Danish Patent Application No. PA 2016 70601, filed Aug. 9, 2016, entitled "USER INTERFACE FOR MANAGING CONTROLLABLE EXTERNAL DEVICES", the entire contents of which are incorporated herein by reference. This application claims priority to Danish Patent Application No. PA 2016 70602, filed Aug. 9, 2016, entitled "USER INTERFACE FOR MANAGING CONTROLLABLE EXTERNAL DEVICES", the entire contents of which are incorporated herein by reference. This application claims priority to Danish Patent Application No. PA 2016 70603, filed Aug. 9, 2016, entitled "USER INTERFACE FOR MANAGING CONTROLLABLE EXTERNAL DEVICES", the entire contents of which are incorporated herein by reference. This application claims priority to Danish Patent Application No. PA 2016 70604, filed Aug. 9, 2016, entitled "USER INTERFACE FOR MANAGING CONTROLLABLE EXTERNAL DEVICES", the entire contents of which are incorporated herein by reference.This application claims priority to Danish Patent Application No. PA 2016 70605, filed on Aug. 9, 2016, entitled "USER INTERFACE FOR MANAGING CONTROLLABLE EXTERNAL DEVICES", the entire content of which is incorporated herein by reference.

[0002] This application is related to U.S. Patent Application No. 14 / 725,912, filed on May 28, 2015, entitled "Accessory Management System Using Environment Model", the entire content of which is incorporated herein by reference. This application is also related to U.S. Patent Application No. (Attorney Docket No. P30795USP1), filed on Jun. 12, 2016, entitled "Devices and Methods for Accessing Prevalent Device Functions", the content of which is included as Appendix A to this application and is incorporated herein by reference in its entirety. [Technical Field]

[0003] The present disclosure generally relates to computer user interfaces, and more particularly to techniques for managing controllable external devices. [Background Art]

[0004] Electronic devices are becoming increasingly prevalent in various applications. Mobile phones, tablet computers, home entertainment systems, etc. are just a few of the electronic devices with which users typically interact.

[0005] In addition, various electronically controllable devices, such as thermostats, lighting fixtures, and home appliances, are also becoming more prevalent. [Summary of the Invention]

[0006] However, some techniques for managing controllable external devices using electronic devices are generally cumbersome and inefficient. For example, in some existing techniques, complex and time-consuming user interfaces that may involve multiple key presses or keystrokes are used. The existing techniques require more time than necessary, wasting the user's time and the device's power. The latter problem is particularly significant in battery-operated devices.

[0007] Accordingly, the present technology provides an electronic device with a faster and more efficient method and interface for managing controllable external devices. Such methods and interfaces optionally complement or replace other methods for managing controllable external devices. Such methods and interfaces reduce the user's cognitive burden and implement a more efficient human-machine interface. In the case of battery-operated computing devices, such methods and interfaces save power and increase the intervals between battery charges.

[0008] The method is for an electronic device comprising a display, a touch-sensitive surface, one or more processors, and a memory, and includes displaying an indication of a location on a first user interface screen, displaying a representation of a controllable external device at the location on the first user interface screen, detecting a first user input corresponding to a selection of the controllable external device, in response to detecting the first user input, displaying a user interface object indicating a plurality of possible states of the controllable external device, while the user interface object indicating the plurality of possible states of the controllable external device is being displayed, detecting a second user input corresponding to a selection of a specified state of the controllable external device, and in response to detecting the second user input, sending an instruction to set the controllable external device to the specified state.

[0009] An electronic device includes a display, a touch-sensing surface, one or more processors, a memory, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs display an indication of a location on a first user interface screen, display a representation of a controllable external device on the first user interface screen, detect a first user input corresponding to a selection of the controllable external device, in response to detecting the first user input, display a user interface object indicating a plurality of possible states of the controllable external device, while displaying the user interface object indicating the plurality of possible states of the controllable external device, detect a second user input corresponding to a selection of a specified state of the controllable external device, and in response to detecting the second user input, send an instruction to set the controllable external device to the specified state, including instructions.

[0010] A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device including a display and a touch-sensing surface, cause the device to display an indication of a location on a first user interface screen, display a representation of a controllable external device on the first user interface screen, detect a first user input corresponding to a selection of the controllable external device, in response to detecting the first user input, display a user interface object indicating a plurality of possible states of the controllable external device, while displaying the user interface object indicating the plurality of possible states of the controllable external device, detect a second user input corresponding to a selection of a specified state of the controllable external device, and in response to detecting the second user input, send an instruction to set the controllable external device to the specified state.

[0011] A temporary computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions that, when executed by one or more processors of an electronic device comprising a display and a touch-sensitive surface, cause the device to display an indication of a location on a first user interface screen, display a representation of a controllable external device on the first user interface screen, detect a first user input corresponding to a selection of the controllable external device, in response to detecting the first user input, display a user interface object indicating a plurality of possible states of the controllable external device, while the user interface object indicating the plurality of possible states of the controllable external device is being displayed, detect a second user input corresponding to a selection of a specified state of the controllable external device, and in response to detecting the second user input, transmit an instruction to set the controllable external device to the specified state.

[0012] The electronic device comprises a display, a touch-sensitive surface, means for displaying an indication of a location on a first user interface screen, means for displaying a representation of a controllable external device on the first user interface screen, means for detecting a first user input corresponding to a selection of the controllable external device, means for displaying, in response to detecting the first user input, a user interface object indicating a plurality of possible states of the controllable external device, means for detecting a second user input corresponding to a selection of a specified state of the controllable external device while the user interface object indicating the plurality of possible states of the controllable external device is being displayed, and means for transmitting, in response to detecting the second user input, an instruction to set the controllable external device to the specified state.

[0013] The electronic device includes a display unit configured to display a graphic user interface, a touch sensing surface unit configured to receive contact, and a processing unit coupled to the display unit and the touch sensing surface unit. The processing unit enables the display of an indication of a location on a first user interface screen, enables the display of a representation of a controllable external device on the first user interface screen, detects a first user input corresponding to the selection of the controllable external device, in response to detecting the first user input, enables the display of a user interface object indicating a plurality of possible states of the controllable external device, and while the user interface object indicating the plurality of possible states of the controllable external device is being displayed, detects a second user input corresponding to the selection of a specified state of the controllable external device, and in response to detecting the second user input, causes the transmission of an instruction to set the controllable external device to the specified state.

[0014] The method includes, in an electronic device comprising a display, a touch sensing surface, one or more processors, and a memory, displaying a first representation of a first controllable external device located at a location, detecting a first user input corresponding to the selection of the first controllable external device, and after detecting the first user input, including in a scene profile data identifying the first controllable external device and a first state of the first controllable external device.

[0015] The electronic device includes a display, a touch sensing surface, one or more processors, a memory, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs include instructions to display a first representation of a first controllable external device located at a location, detect a first user input corresponding to a selection of the first controllable external device, and after detecting the first user input, include data identifying the first controllable external device and a first state of the first controllable external device in a scene profile.

[0016] A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device including a display and a touch sensing surface, cause the device to display a first representation of a first controllable external device located at a location, detect a first user input corresponding to a selection of the first controllable external device, and after detecting the first user input, cause the data identifying the first controllable external device and a first state of the first controllable external device to be included in a scene profile.

[0017] A transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device including a display and a touch sensing surface, cause the device to display a first representation of a first controllable external device located at a location, detect a first user input corresponding to a selection of the first controllable external device, and after detecting the first user input, cause the data identifying the first controllable external device and a first state of the first controllable external device to be included in a scene profile.

[0018] An electronic device includes a display, a touch sensing surface, means for displaying a first representation of a first controllable external device located at a location, means for detecting a first user input corresponding to a selection of the first controllable external device, and means for including, in a scene profile, data for identifying the first controllable external device and a first state of the first controllable external device after detecting the first user input.

[0019] The electronic device includes a display unit configured to display a graphical user interface, a touch sensing surface unit configured to receive contact, and a processing unit coupled to the display unit and the touch sensing surface unit, the processing unit being configured to enable display of a first representation of a first controllable external device located at a location, detect a first user input corresponding to a selection of the first controllable external device, and include, in a scene profile, data for identifying the first controllable external device and a first state of the first controllable external device after detecting the first user input.

[0020] A method is implemented on an electronic device including a display, a touch sensing surface, one or more processors, and a memory, and includes detecting a first user input corresponding to a selection of a first criterion, associating the first criterion, in response to detecting the first user input, with an automation criterion including at least the first criterion and an automation profile including data representing at least one specified state for at least one controllable external device located at a location, detecting a second user input corresponding to a selection of a first controllable external device located at the location, adding data indicating a specified state of the first controllable external device to the automation profile, determining whether the automation criterion is satisfied, and transmitting an instruction to set the first controllable external device to the specified state according to a determination that the automation criterion is satisfied.

[0021] The electronic device includes a display, a touch sensing surface, one or more processors, a memory, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs are configured to detect a first user input corresponding to the selection of a first criterion, and in response to detecting the first user input, associate the first criterion with an automation criterion including at least the first criterion and an automation profile including data representing at least one specified state for at least one controllable external device located at a location. Detect a second user input corresponding to the selection of a first controllable external device located at the location, add data indicating the specified state of the first controllable external device to the automation profile, determine whether the automation criterion is satisfied, and in accordance with the determination that the automation criterion is satisfied, send an instruction to set the first controllable external device to the specified state, including instructions.

[0022] A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device including a display and a touch sensing surface, cause the device to detect a first user input corresponding to the selection of a first criterion, and in response to detecting the first user input, associate the first criterion with an automation criterion including at least the first criterion and an automation profile including data representing at least one specified state for at least one controllable external device located at a location, detect a second user input corresponding to the selection of a first controllable external device located at the location, add data indicating the specified state of the first controllable external device to the automation profile, determine whether the automation criterion is satisfied, and in accordance with the determination that the automation criterion is satisfied, send an instruction to set the first controllable external device to the specified state.

[0023] A temporary computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device comprising a display and a touch-sensing surface, cause the device to detect a first user input corresponding to a selection of a first criterion, associate the first criterion with an automation profile including at least the first criterion and data representing at least one specified state for at least one controllable external device located at a location in response to detecting the first user input, detect a second user input corresponding to a selection of a first controllable external device located at the location, add data indicating the specified state of the first controllable external device to the automation profile, determine whether the automation criterion is satisfied, and transmit an instruction to set the first controllable external device to the specified state according to the determination that the automation criterion is satisfied.

[0024] The electronic device comprises means for detecting a first user input corresponding to a selection of a first criterion, means for associating the first criterion with an automation profile including at least the first criterion and data representing at least one specified state for at least one controllable external device located at a location in response to detecting the first user input, means for detecting a second user input corresponding to a selection of a first controllable external device located at the location, means for adding data indicating the specified state of the first controllable external device to the automation profile, means for determining whether the automation criterion is satisfied, and means for transmitting an instruction to set the first controllable external device to the specified state according to the determination that the automation criterion is satisfied.

[0025] An electronic device includes a display unit configured to display a graphical user interface, a touch sensing surface unit configured to receive contact, and a processing unit coupled to the display unit and the touch sensing surface unit. The processing unit detects a first user input corresponding to the selection of a first criterion, and in response to detecting the first user input, associates the first criterion with an automation criterion including at least the first criterion and an automation profile including data representing at least one specified state for at least one controllable external device located at a location. The processing unit detects a second user input corresponding to the selection of a first controllable external device located at the location, adds data indicating the specified state of the first controllable external device to the automation profile, determines whether the automation criterion is satisfied, and causes the transmission of an instruction to set the first controllable external device to the specified state according to the determination that the automation criterion is satisfied.

[0026] A method is implemented on an electronic device including a display, a touch sensing surface, a camera, one or more processors, and a memory. The method includes using the camera to capture an image of a pattern, determining that the pattern corresponds to a controllable external device configured to operate in two or more states and remotely controllable between the two or more states, associating the controllable external device with a location having at least one specified room in response to the determination that the pattern corresponds to the controllable external device, detecting a first user input representing the selection of a room from the at least one specified room, and associating the selected room with the controllable external device in response to detecting the first user input.

[0027] The electronic device includes a display, a touch sensing surface, a camera, one or more processors, a memory, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs use the camera to capture an image of a pattern, determine that the pattern corresponds to a controllable external device configured to operate in two or more states and be remotely controllable between the two or more states, associate the controllable external device with a location having at least one designated room in response to the determination that the pattern corresponds to the controllable external device, detect a first user input representing a selection of a room from the at least one designated room, and associate the selected room with the controllable external device in response to detecting the first user input, including instructions.

[0028] A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device including a display, a touch sensing surface, and a camera, cause the device to capture an image of a pattern using the camera, determine that the pattern corresponds to a controllable external device configured to operate in two or more states and be remotely controllable between the two or more states, associate the controllable external device with a location having at least one designated room in response to the determination that the pattern corresponds to the controllable external device, detect a first user input representing a selection of a room from the at least one designated room, and associate the selected room with the controllable external device in response to detecting the first user input.

[0029] A temporary computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device comprising a display, a touch-sensing surface, and a camera, cause the device to use the camera to capture an image of a pattern, the pattern being configured to operate in two or more states and being remotely controllable between the two or more states, determine that the pattern corresponds to a controllable external device, in response to the determination that the pattern corresponds to a controllable external device, associate the controllable external device with a location having at least one designated room, detect a first user input representing a selection of a room from the at least one designated room, and in response to detecting the first user input, associate the selected room with the controllable external device.

[0030] The electronic device comprises a display, a touch-sensing surface, a camera, means for using the camera to capture an image of a pattern, means for determining that the pattern, which is configured to operate in two or more states and is remotely controllable between the two or more states, corresponds to a controllable external device, means for associating the controllable external device with a location having at least one designated room in response to the determination that the pattern corresponds to a controllable external device, means for detecting a first user input representing a selection of a room from the at least one designated room, and means for associating the selected room with the controllable external device in response to detecting the first user input.

[0031] The electronic device includes a display unit configured to display a graphical user interface, a touch sensing surface unit configured to receive contact, a camera unit configured to capture an image, and a processing unit coupled to the display unit and the touch sensing surface unit. The processing unit uses the camera to cause the capture of an image of a pattern, and the pattern is configured to operate in two or more states and is remotely controllable between the two or more states. The processing unit determines that the pattern corresponds to a controllable external device. In response to the determination that the pattern corresponds to a controllable external device, the processing unit associates the controllable external device with a location having at least one designated room, detects a first user input representing the selection of a room from the at least one designated room, and in response to detecting the first user input, associates the selected room with the controllable external device.

[0032] The method includes detecting, by an electronic device comprising a display, a touch sensing surface, one or more processors, and memory, a user input corresponding to a specification of a location-based criterion that is at least partially based on the location of the electronic device relative to a first location having at least one controllable external device; adding, in response to detecting the user input corresponding to the specification of the location-based criterion, the location-based criterion to an automation profile associated with the location, the automation profile including the location-based criterion for controlling at least one controllable external device and data representing a specified state of the at least one controllable external device; determining the location of the electronic device; determining whether an automation criterion for controlling at least one controllable external device is satisfied based at least partially on the determined location of the electronic device; and transmitting an instruction to set the at least one controllable external device to the specified state in accordance with the determination that the automation criterion for controlling the at least one controllable external device is satisfied.

[0033] An electronic device includes a display, a touch sensing surface, one or more processors, a memory, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs detect a user input corresponding to a specification of a criterion based at least in part on a position of the electronic device relative to a first location having at least one controllable external device. In response to detecting the user input corresponding to the specification of the criterion, the position-based criterion is added to an automation profile associated with the location, the automation profile including the position-based criterion for controlling at least one controllable external device and data representing a specified state of the at least one controllable external device. The position of the electronic device is determined, and it is determined whether an automation criterion for controlling at least one controllable external device is satisfied based at least in part on the determined position of the electronic device. In accordance with the determination that the automation criterion for controlling at least one controllable external device is satisfied, an instruction is sent to set the at least one controllable external device to the specified state, including the instruction.

[0034] A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device comprising a display and a touch-sensitive surface, cause the device to detect a user input corresponding to a specification of a reference based at least in part on a position of the electronic device relative to a first location having at least one controllable external device, in response to detecting the user input corresponding to the specification of the reference, add the position-based reference to an automation profile associated with the location, the automation profile including an automation criterion based on the position for controlling at least one controllable external device and data representing a specified state of the at least one controllable external device, determine a position of the electronic device, determine whether an automation criterion for controlling at least one controllable external device is satisfied based at least in part on the determined position of the electronic device, and transmit an instruction to set the at least one controllable external device to the specified state in accordance with the determination that the automation criterion for controlling the at least one controllable external device is satisfied.

[0035] A temporary computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device comprising a display and a touch-sensing surface, cause the device to detect a user input corresponding to a specification of a reference based at least in part on a position of the electronic device relative to a first location having at least one controllable external device, in response to detecting the user input corresponding to the specification of the reference, add the position-based reference to an automation profile associated with the location, the automation profile including an automation criterion based on the position for controlling at least one controllable external device and data representing a specified state of the at least one controllable external device, determine the position of the electronic device, determine whether an automation criterion for controlling at least one controllable external device is satisfied based at least in part on the determined position of the electronic device, and transmit an instruction to set the at least one controllable external device to the specified state in accordance with the determination that the automation criterion for controlling at least one controllable external device is satisfied.

[0036] The electronic device includes means for detecting a user input corresponding to the specification of a criterion based at least in part on the position of the electronic device relative to a first location having a display, a touch-sensitive surface, and at least one controllable external device; means for adding, in response to detecting the user input corresponding to the specification of the criterion, the position-based criterion to an automation profile associated with the location, the automation profile including an automation criterion based on the position for controlling at least one controllable external device and data representing a specified state of the at least one controllable external device; means for determining the position of the electronic device; means for determining whether an automation criterion for controlling at least one controllable external device is satisfied based at least in part on the determined position of the electronic device; and means for transmitting an instruction to set the at least one controllable external device to the specified state in accordance with the determination that the automation criterion for controlling the at least one controllable external device is satisfied.

[0037] The electronic device comprises a display unit configured to display a graphical user interface, a touch sensing surface unit configured to receive contact, and a processing unit coupled to the display unit and the touch sensing surface unit, wherein the processing unit detects a user input corresponding to a specification of a reference based at least in part on the position of the electronic device relative to a first location having at least one controllable external device, and in response to detecting the user input corresponding to the specification of the reference, adds the position-based reference to an automation profile associated with the location, the automation profile including a position-based reference for controlling at least one controllable external device and data representing a specified state of the at least one controllable external device, determines the position of the electronic device, determines whether a position-based reference for controlling at least one controllable external device is satisfied based at least in part on the determined position of the electronic device, and causes transmission of an instruction to set the at least one controllable external device to the specified state in accordance with the determination that the position-based reference for controlling at least one controllable external device is satisfied.

[0038] The method is implemented on an electronic device comprising a display, a touch sensing surface, one or more processors, and memory, and includes determining whether criteria of an automation profile including criteria for controlling the state of at least one controllable external device and data representing a specified state for the at least one controllable external device are satisfied, displaying a notification including an indication identifying the automation profile and an indication confirming implementation of the automation profile in accordance with the determination that the criteria of the automation profile are satisfied, detecting a user input, and in response to detecting the user input, transmitting an instruction to implement the specified state of the automation profile in accordance with the user input corresponding to confirmation of implementation of the automation profile, and transmitting an instruction not to implement the specified state of the automation profile in accordance with the user input corresponding to non-confirmation of implementation of the automation profile.

[0039] The electronic device includes a display, a touch sensing surface, one or more processors, a memory, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs determine whether criteria of an automation profile are met, the criteria including criteria for controlling the state of at least one controllable external device and data representing a specified state for the at least one controllable external device. According to the determination that the criteria of the automation profile are met, the one or more programs display a notification including an indication for identifying the automation profile and an indication for confirming the implementation of the automation profile, detect user input, and in response to detecting the user input, send an instruction to implement the specified state of the automation profile according to user input corresponding to confirmation of the implementation of the automation profile, and send an instruction not to implement the specified state of the automation profile according to user input corresponding to non-confirmation of the implementation of the automation profile.

[0040] A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device including a display and a touch sensing surface, cause the device to determine whether criteria of an automation profile are met, the criteria including criteria for controlling the state of at least one controllable external device and data representing a specified state for the at least one controllable external device, display a notification including an indication for identifying the automation profile and an indication for confirming the implementation of the automation profile according to the determination that the criteria of the automation profile are met, detect user input, and in response to detecting the user input, send an instruction to implement the specified state of the automation profile according to user input corresponding to confirmation of the implementation of the automation profile, and send an instruction not to implement the specified state of the automation profile according to user input corresponding to non-confirmation of the implementation of the automation profile.

[0041] A temporary computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions that, when executed by one or more processors of an electronic device having a display and a touch-sensitive surface, cause the device to determine whether criteria of an automation profile are met, the criteria including criteria for controlling the state of at least one controllable external device and data representing a specified state for the at least one controllable external device; display a notification including an indication identifying the automation profile and an indication confirming the implementation of the automation profile in accordance with a determination that the criteria of the automation profile are met; detect user input; and in response to detecting the user input, transmit instructions to implement the specified state of the automation profile in accordance with user input corresponding to confirmation of implementation of the automation profile, or transmit instructions not to implement the specified state of the automation profile in accordance with user input corresponding to non-confirmation of implementation of the automation profile.

[0042] The electronic device includes means for determining whether criteria of an automation profile are met, the criteria including criteria for controlling the state of at least one controllable external device and data representing a specified state for the at least one controllable external device; means for displaying a notification including an indication identifying the automation profile and an indication confirming the implementation of the automation profile in accordance with a determination that the criteria of the automation profile are met; means for detecting user input; and means for transmitting instructions to implement the specified state of the automation profile in accordance with user input corresponding to confirmation of implementation of the automation profile, or transmitting instructions not to implement the specified state of the automation profile in accordance with user input corresponding to non-confirmation of implementation of the automation profile, in response to detecting the user input.

[0043] An electronic device includes a display unit configured to display a graphical user interface, a touch sensing surface unit configured to receive contact, and a processing unit coupled to the display unit and the touch sensing surface unit. The processing unit determines whether criteria of an automation profile are met, the criteria including criteria for controlling the state of at least one controllable external device and data representing a specified state for the at least one controllable external device. In accordance with a determination that the criteria of the automation profile are met, the processing unit enables display of a notification including an indication identifying the automation profile and an indication confirming implementation of the automation profile. The processing unit detects user input and, in response to detecting the user input, causes transmission of a command to implement the specified state of the automation profile in accordance with user input corresponding to confirmation of implementation of the automation profile, and causes transmission of a command not to implement the specified state of the automation profile in accordance with user input corresponding to non-confirmation of implementation of the automation profile.

[0044] A method is performed by an electronic device including a display, a touch sensing surface, one or more processors, and memory, the electronic device being configured to operate in a plurality of states. The method includes displaying a representation of a controllable external device in a first state of the plurality of states, detecting a first user input corresponding to selection of the controllable external device, determining whether the first user input meets input criteria, transmitting a command to set the state of the controllable external device to a second state of the plurality of states in accordance with a determination that the first user input does not meet the input criteria, displaying affordances indicating the plurality of states of the controllable external device in accordance with a determination that the first user input meets the input criteria, detecting a second user input on an affordance corresponding to selection of a third state of the plurality of states while the affordances indicating the plurality of states of the controllable external device are being displayed, and transmitting a command to set the state of the controllable external device to the third state in response to detecting the second user input.

[0045] The electronic device includes a display, a touch sensing surface, one or more processors, a memory, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs are configured to operate in multiple states, and display a representation of a controllable external device in a first state of the multiple states, detect a first user input corresponding to a selection of the controllable external device, determine whether the first user input meets an input criterion, send an instruction to set the state of the controllable external device to a second state of the multiple states according to a determination that the first user input does not meet the input criterion, display affordances indicating the multiple states of the controllable external device according to a determination that the first user input meets the input criterion, detect a second user input on an affordance corresponding to a selection of a third state of the multiple states while displaying the affordances indicating the multiple states of the controllable external device, and send an instruction to set the state of the controllable external device to the third state in response to detecting the second user input.

[0046] A non-transitory computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions that, when executed by one or more processors of an electronic device comprising a display and a touch-sensitive surface, cause the device to be configured to operate in a plurality of states, and to display a representation of a controllable external device in a first state of the plurality of states, detect a first user input corresponding to a selection of the controllable external device, determine whether the first user input meets an input criterion, transmit an instruction to set the state of the controllable external device to a second state of the plurality of states according to a determination that the first user input does not meet the input criterion, display affordances indicating the plurality of states of the controllable external device according to a determination that the first user input meets the input criterion, detect a second user input on an affordance corresponding to a selection of a third state of the plurality of states while the affordances indicating the plurality of states of the controllable external device are being displayed, and transmit an instruction to set the state of the controllable external device to the third state in response to detecting the second user input.

[0047] A temporary computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions that, when executed by one or more processors of an electronic device comprising a display and a touch-sensitive surface, cause the device to be configured to operate in a plurality of states, and to display a representation of a controllable external device in a first state of the plurality of states, detect a first user input corresponding to a selection of the controllable external device, determine whether the first user input meets an input criterion, send an instruction to set the state of the controllable external device to a second state of the plurality of states according to a determination that the first user input does not meet the input criterion, display affordances indicating the plurality of states of the controllable external device according to a determination that the first user input meets the input criterion, detect a second user input on an affordance corresponding to a selection of a third state of the plurality of states while the affordances indicating the plurality of states of the controllable external device are being displayed, and send an instruction to set the state of the controllable external device to the third state in response to detecting the second user input.

[0048] The electronic device comprises means for displaying a representation of a controllable external device that is configured to operate in a plurality of states and is in a first state of the plurality of states, means for detecting a first user input corresponding to a selection of the controllable external device, means for determining whether the first user input meets an input criterion, means for sending an instruction to set the state of the controllable external device to a second state of the plurality of states according to a determination that the first user input does not meet the input criterion, means for displaying affordances indicating the plurality of states of the controllable external device according to a determination that the first user input meets the input criterion, means for detecting a second user input on an affordance corresponding to a selection of a third state of the plurality of states while the affordances indicating the plurality of states of the controllable external device are being displayed, and means for sending an instruction to set the state of the controllable external device to the third state in response to detecting the second user input.

[0049] The electronic device includes a display unit configured to display a graphical user interface, a touch sensing surface unit configured to receive contacts, and a processing unit coupled to the display unit and the touch sensing surface unit, wherein the processing unit is configured to operate in a plurality of states, and enable the display of a representation of a controllable external device in a first state of the plurality of states, detect a first user input corresponding to a selection of the controllable external device, determine whether the first user input meets an input criterion, and cause the transmission of an instruction to set the state of the controllable external device to a second state of the plurality of states according to a determination that the first user input does not meet the input criterion, and enable the display of affordances indicating the plurality of states of the controllable external device according to a determination that the first user input meets the input criterion, and while displaying the affordances indicating the plurality of states of the controllable external device, detect a second user input on an affordance corresponding to a selection of a third state of the plurality of states, and cause the transmission of an instruction to set the state of the controllable external device to the third state in response to detecting the second user input, and is configured as such.

[0050] The method includes determining, in an electronic device comprising a display, one or more processors, and a memory, a first position of the electronic device within a location including at least one designated room, determining that the first position corresponds to a first room of the at least one designated room, and displaying a representation of a first controllable external device associated with the first room in response to determining that the first position corresponds to the first room.

[0051] The electronic device includes a display, a touch sensing surface, one or more processors, a memory, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs include instructions to determine a first position of the electronic device within a location that includes at least one designated room, determine that the first position corresponds to a first room of the at least one designated room, and in response to determining that the first position corresponds to the first room, display a representation of a first controllable external device associated with the first room.

[0052] A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device including a display, cause the device to determine a first position of the electronic device within a location that includes at least one designated room, determine that the first position corresponds to a first room of the at least one designated room, and in response to determining that the first position corresponds to the first room, display a representation of a first controllable external device associated with the first room.

[0053] A transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device including a display, cause the device to determine a first position of the electronic device within a location that includes at least one designated room, determine that the first position corresponds to a first room of the at least one designated room, and in response to determining that the first position corresponds to the first room, display a representation of a first controllable external device associated with the first room.

[0054] An electronic device comprises a display, means for determining a first position of the electronic device within a location including at least one designated room, means for determining that the first position corresponds to a first room of the at least one designated room, and means for displaying a representation of a first controllable external device associated with the first room in response to determining that the first position corresponds to the first room.

[0055] The electronic device comprises a display unit configured to display a graphical user interface, and a processing unit coupled to the display unit and a touch sensing surface unit, the processing unit being configured to determine a first position of the electronic device within a location including at least one designated room, determine that the first position corresponds to a first room of the at least one designated room, and enable the display of a representation of a first controllable external device associated with the first room in response to determining that the first position corresponds to the first room.

[0056] A method in an electronic device comprising a display, a touch sensing surface, one or more processors, and a memory includes displaying a first affordance representing a first personal contact, detecting a first user input corresponding to a selection of the first affordance representing the first personal contact, displaying a second affordance representing an access parameter for determining access to a location profile including information about controllable external devices of the location associated with the location, detecting a second user input corresponding to a selection of the second affordance, permitting access to an external device associated with the first personal contact based on the access parameter to the location profile, and transmitting data including an invitation to access the location profile to an external device associated with the first personal contact.

[0057] An electronic device includes a display, a touch sensing surface, one or more processors, a memory, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs display a first affordance representing a first personal contact, detect a first user input corresponding to a selection of the first affordance representing the first personal contact, display a second affordance representing an access parameter for determining access to a location profile including information about a controllable external device related to the location, detect a second user input corresponding to a selection of the second affordance, permit access to the location profile for an external device associated with the first personal contact based on the access parameter, and send data including an invitation to access the location profile to the external device associated with the first personal contact.

[0058] A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device including a display and a touch sensing surface, cause the device to display a first affordance representing a first personal contact, detect a first user input corresponding to a selection of the first affordance representing the first personal contact, display a second affordance representing an access parameter for determining access to a location profile including information about a controllable external device related to the location, detect a second user input corresponding to a selection of the second affordance, permit access to the location profile for an external device associated with the first personal contact based on the access parameter, and send data including an invitation to access the location profile to the external device associated with the first personal contact.

[0059] A temporary computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device comprising a display and a touch-sensing surface, cause the device to display a first affordance representing a first personal contact, detect a first user input corresponding to a selection of the first affordance representing the first personal contact, display a second affordance representing an access parameter for determining access to a location profile including information about a controllable external device related to the location, detect a second user input corresponding to a selection of the second affordance, permit access to the location profile for an external device associated with the first personal contact based on the access parameter, and transmit data including an invitation to access the location profile to the external device associated with the first personal contact.

[0060] The electronic device comprises: means for displaying a first affordance representing a first personal contact; means for detecting a first user input corresponding to a selection of the first affordance representing the first personal contact; means for displaying a second affordance representing an access parameter for determining access to a location profile including information about a controllable external device related to the location; means for detecting a second user input corresponding to a selection of the second affordance; means for permitting access to the location profile for an external device associated with the first personal contact based on the access parameter; and means for transmitting data including an invitation to access the location profile to the external device associated with the first personal contact.

[0061] The electronic device comprises a display unit configured to display a graphical user interface, a touch sensing surface unit configured to receive contacts, and a processing unit coupled to the display unit and the touch sensing surface unit, wherein the processing unit enables the display of a first affordance representing a first personal contact, detects a first user input corresponding to the selection of the first affordance representing the first personal contact, enables the display of a second affordance representing an access parameter for determining access to a location profile including information regarding a controllable external device associated with the location, detects a second user input corresponding to the selection of the second affordance, permits access to the location profile to an external device associated with the first personal contact based on the access parameter, and causes transmission of data including an invitation to access the location profile to the external device associated with the first personal contact.

[0062] The method includes receiving, by an electronic device comprising a display, a touch sensing surface, one or more processors, and a memory, data including information regarding a location and information regarding a first controllable external device associated with the location; displaying, in accordance with receiving the data, an affordance corresponding to the first controllable external device; detecting a first user input corresponding to a selection of the affordance; and displaying, in response to detecting the first user input, a user interface screen including an indication of the location and a representation of the first controllable external device.

[0063] An electronic device includes a display, a touch sensing surface, one or more processors, a memory, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs include instructions to receive data including information about a location and information about a first controllable external device at the location, display an affordance corresponding to the first controllable external device in accordance with receiving the data, detect a first user input corresponding to a selection of the affordance, and display a user interface screen including an indication of the location and a representation of the first controllable external device in response to detecting the first user input.

[0064] A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device including a display and a touch sensing surface, cause the device to receive data including information about a location and information about a first controllable external device at the location, display an affordance corresponding to the first controllable external device in accordance with receiving the data, detect a first user input corresponding to a selection of the affordance, and display a user interface screen including an indication of the location and a representation of the first controllable external device in response to detecting the first user input.

[0065] A temporary computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions that, when executed by one or more processors of an electronic device comprising a display and a touch-sensitive surface, cause the device to receive data including information about a location and information about a first controllable external device at the location, in accordance with receiving the data, cause an affordance corresponding to the first controllable external device to be displayed, detect a first user input corresponding to a selection of the affordance, and in response to detecting the first user input, cause a user interface screen including an indication of the location and a representation of the first controllable external device to be displayed.

[0066] The electronic device comprises means for receiving data including information about a location and information about a first controllable external device at the location, means for displaying, in accordance with receiving the data, an affordance corresponding to the first controllable external device, means for detecting a first user input corresponding to a selection of the affordance, and means for displaying, in response to detecting the first user input, a user interface screen including an indication of the location and a representation of the first controllable external device.

[0067] The electronic device comprises a display unit configured to display a graphical user interface, a touch-sensitive surface unit configured to receive contacts, and a processing unit coupled to the display unit and the touch-sensitive surface unit, wherein the processing unit is configured to receive data including information about a location and information about a first controllable external device at the location, display, in accordance with receiving the data, an affordance corresponding to the first controllable external device, detect a first user input corresponding to a selection of the affordance, and display, in response to detecting the first user input, a user interface screen including an indication of the location and a representation of the first controllable external device.

[0068] The executable instructions for performing these functions are optionally included within a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors. The executable instructions for performing these functions are optionally included within a transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0069] Accordingly, a faster and more efficient method and interface for managing a controllable external device are provided to the device, thereby increasing the effectiveness, efficiency, and user satisfaction of such a device. Such a method and interface can complement or replace other methods for managing a controllable external device.

Brief Description of the Drawings

[0070] To better understand the various embodiments described, the following "Modes for Carrying Out the Invention" should be referred to in conjunction with the following drawings. Here, like reference numerals refer to corresponding parts throughout all of those figures.

[0071]

Figure 1A

[0072]

Figure 1B

[0073]

Figure 2

[0074]

Figure 3

[0075]

Figure 4A

[0076]

Figure 4B

[0077]

Figure 5A

[0078]

Figure 5B

[0079]

Figure 5C

Figure 5D

[0080]

Figure 5E

Figure 5F

Figure 5G

Figure 5H

[0081]

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 6E

Figure 6F

Figure 6G

Figure 6H

Figure 6I

Figure 6J

Figure 6K

Figure 6L

Figure 6M

Figure 6N

Figure 6O

Figure 6P

Figure 6Q

Figure 6R

Figure 6S

[0082]

Figure 7A

Figure 7B

Figure 7C

[0083]

Figure 8

[0084]

Figure 9A

Figure 9B

Figure 9C

Figure 9D

Figure 9E

Figure 9F

[0085]

Figure 10

[0086]

Figure 11

[0087]

Figure 12A

Figure 12B

Figure 12C

[0088]

Figure 13

[0089]

Figure 14

[0090]

Figure 15A

Figure 15B

[0091]

Figure 16

[0092]

Figure 17

[0093]

Figure 18A

Figure 18B

Figure 18C

[0094]

Figure 19

[0095]

Figure 20

[0096]

Figure 21

[0097]

Figure 22

[0098]

Figure 23

[0099]

Figure 24

[0100]

Figure 25

[0101]

Figure 26

[0102]

Figure 27

[0103]

Figure 28A

Figure 28B

Figure 28C

Figure 28D

Figure 28E

[0104]

Figure 29

[0105]

Figure 30

[0106]

Figure 31A

Figure 31B

[0107]

Figure 32

[0108]

Figure 33

[0109]

Figure 34A

Figure 34B

Figure 34C

Figure 34D

Figure 34E

Figure 34F

Figure 34G

Figure 34H

Figure 34I

Figure 34J

Figure 34K

Mode for Carrying Out the Invention

[0110] The following description describes exemplary methods, parameters, etc. However, such a description is not intended as a limitation on the scope of the present disclosure, but should instead be recognized as a description of exemplary embodiments.

[0111] There is a need for an efficient method and interface for managing controllable external devices. In some embodiments, an electronic device provides access to a specified (e.g., favorite) controllable device on a home screen for managing a system of controllable devices. In some embodiments, a user can coarsely control a device by toggling its state between two states (e.g., on / off) using an input having one type of characteristic, and obtain more detailed control of the device using inputs having different characteristics. In some embodiments, a set of devices defined by a user is controlled according to conditions defined by the user. Various other techniques are also described. Such techniques can reduce the cognitive burden on the user for managing controllable external devices, thereby increasing productivity. Further, such techniques can reduce processor and battery power that would otherwise be wasted on redundant user input.

[0112] Hereinafter, FIGS. 1A and 1B, FIG. 2, FIG. 3, FIGS. 4A-4B, and FIGS. 5A-5H provide an illustration of an exemplary device for implementing techniques for managing controllable external devices. FIGS. 6A-6S, FIGS. 9A-9F, FIGS. 12A-12C, FIGS. 15A-15B, FIGS. 18A-18C, FIG. 25, FIGS. 28A-28E, FIGS. 31A-31B, and FIGS. 34A-34K show exemplary user interfaces for managing controllable external devices. FIGS. 7A-7C, FIG. 10, FIG. 13, FIG. 16, FIG. 19, FIG. 21, FIG. 23, FIG. 26, FIG. 29, and FIG. 32 are flow diagrams showing methods for managing controllable external devices according to some embodiments. FIGS. 6A-6S, FIGS. 9A-9F, FIGS. 12A-12C, FIGS. 15A-15B, FIGS. 18A-18C, FIG. 25, FIGS. 28A-28E, FIGS. 31A-31B, and FIGS. 34A-34K are used to show the processes described below, including the processes of FIGS. 7A-7C, FIG. 10, FIG. 13, FIG. 16, FIG. 19, FIG. 21, FIG. 23, FIG. 26, FIG. 29, and FIG. 32.

[0113] In the following description, terms such as "first" and "second" are used to describe various elements, but these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various embodiments described, the first touch can be called the second touch, and similarly, the second touch can be called the first touch. The first touch and the second touch are both touches, but they are not the same touch.

[0114] The terms used in the description of the various embodiments described herein are for the purpose of describing only specific embodiments and are not intended to be limiting. Within the description of the various embodiments described and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural as well, unless the context clearly dictates otherwise. Also, as used herein, the term "and / or" refers to any and all combinations of one or more of the associated listed items and is to be understood to include the same. The terms "includes", "including", "comprises", and / or "comprising", when used herein, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0115] The term "if" is optionally interpreted, depending on the context, to mean "when" or "upon", or "in response to determining" or "in response to detecting". Similarly, the phrases "if it is determined" or "if [a stated condition or event] is detected" are optionally interpreted, depending on the context, to mean "upon determining" or "in response to determining", or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]".

[0116] Embodiments of an electronic device, a user interface for such a device, and related processes for using such a device are described. In some embodiments, the device is a portable communication device such as a mobile phone that also includes other functions such as PDA functionality and / or music player functionality. Exemplary embodiments of the portable multifunctional device include, but are not limited to, devices such as the iPhone®, iPod Touch®, and iPad® from Apple Inc. of Cupertino, California. Optionally, other portable electronic devices such as a laptop or tablet computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad) may also be used. Also, in some embodiments, it should be understood that the device is not a portable communication device but a desktop computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0117] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices such as a physical keyboard, a mouse, and / or a joystick.

[0118] This device generally supports various applications such as one or more of a drawing application, a presentation application, a word processing application, a website creation application, a disk authoring application, a spreadsheet application, a game application, a telephone application, a video conferencing application, an email application, an instant messaging application, a training support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0119] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface, as well as the corresponding information displayed on the device, are optionally adjusted and / or changed for each application and / or within each respective application. In this way, the common physical architecture of the device (such as a touch-sensitive surface) optionally supports various applications with a user interface that is intuitive and transparent to the user.

[0120] Attention is now directed to an embodiment of a portable device with a touch-sensitive display. FIG. 1A is a block diagram showing a portable multifunctional device 100 having a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display 112 may be referred to as a "touch screen" for convenience and is known as or may be referred to as a "touch-sensitive display system". The device 100 includes a memory 102 (optionally including one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and an external port 124. The device 100 optionally includes one or more light sensors 164. The device 100 optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on the device 100 (such as a touch-sensitive surface such as the touch-sensitive display system 112 of the device 100). The device 100 optionally includes one or more haptic output generators 167 for generating haptic output on the device 100 (such as generating haptic output on a touch-sensitive surface such as the touch-sensitive display system 112 of the device 100 or the touch pad 355 of the device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0121] As used in this specification and the claims, the term "intensity" of a contact on a touch-sensing surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensing surface, or an alternative (substitute) for the force or pressure of a contact on the touch-sensing surface. The intensity of the contact has a range of values that includes at least four different values and, more typically, hundreds (e.g., at least 256) of different values. The intensity of the contact is optionally determined (or measured) using a variety of techniques and a variety of sensors or combinations of sensors. For example, one or more force sensors under or adjacent to the touch-sensing surface are optionally used to measure the force at various points on the touch-sensing surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted average) to determine the estimated force of the contact. Similarly, a pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensing surface. Alternatively, the size and / or change thereof of the contact area detected on the touch-sensing surface, the capacitance and / or change thereof of the touch-sensing surface proximate to the contact, and / or the resistance and / or change thereof of the touch-sensing surface proximate to the contact are optionally used as an alternative to the force or pressure of a contact on the touch-sensing surface. In some implementations, the alternative measurements of the force or pressure of the contact are used directly to determine whether they exceed an intensity threshold (e.g., the intensity threshold is described in units corresponding to the alternative measurements). In some implementations, these alternative measurements of the force or pressure of the contact are converted to an estimated force or pressure, and that estimated force or pressure is used to determine whether the intensity threshold is exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). By using the intensity of the contact as an attribute of a user input, a user can access additional device functions that would normally be inaccessible to the user on a device of reduced size with a limited area for displaying affordances (e.g., on a touch-sensing display) and / or receiving user input (e.g., via a touch-sensing display, a touch-sensing surface, or a physical / mechanical control such as a knob or button).

[0122] As used in this specification and the claims, the term "haptic output" refers to a physical displacement of the device relative to its previous position, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., the housing), or a displacement of a component relative to the center of mass of the device that is to be detected by the user's sense of touch. For example, in a situation where the device or a component of the device is in contact with a touch-sensitive surface of the user (e.g., the finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the device or the component of the device. For example, the movement of a touch-sensitive surface (e.g., a touch-sensitive display or a trackpad) can optionally be interpreted by the user as a "down click" or "up click" of a physical actuator button. In some cases, even when there is no movement of a physical actuator button associated with a touch-sensitive surface physically pushed (e.g., displaced) by the user's action, the user can feel a tactile sensation such as a "down click" or "up click". As another example, the movement of a touch-sensitive surface can optionally be interpreted or perceived by the user as the "roughness" of the touch-sensitive surface even when there is no change in the smoothness of the touch-sensitive surface. Such an interpretation of the touch by the user depends on the user's individual sensory perception, but there are many sensory perceptions of touch that are common to the majority of users. Therefore, when the haptic output is described as corresponding to a particular sensory perception of the user (e.g., "up click", "down click", "roughness"), unless otherwise stated, the generated haptic output corresponds to a physical displacement of the device or a component of the device that produces the described sensory perception of a typical (or average) user.

[0123] Device 100 is merely an example of a portable multifunctional device, and it should be understood that device 100 may optionally have more or fewer components than those shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of those components. The various components shown in FIG. 1A are implemented in a combination of hardware, software, or both hardware and software, including one or more signal processing circuits and / or application-specific integrated circuits.

[0124] Memory 102 optionally includes high-speed random access memory and also optionally includes non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls access to memory 102 by other components of device 100.

[0125] Peripheral interface 118 can be used to couple the input and output peripheral devices of the device to CPU 120 and memory 102. One or more processors 120 operate or execute various software programs and / or instruction sets stored in memory 102 to perform various functions for device 100 and process data. In some embodiments, peripheral interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0126] The RF (radio frequency) circuit 108 transmits and receives RF signals, also called electromagnetic signals. The RF circuit 108 converts electrical signals into electromagnetic signals or vice versa and communicates with a communication network and other communication devices via electromagnetic signals. The RF circuit 108 optionally includes well-known circuits for performing these functions, such as, but not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, and the like. The RF circuit 108 optionally communicates wirelessly with a network such as the Internet, also called the World Wide Web (WWW), an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN), as well as with other devices. The RF circuit 108 optionally includes well-known circuits for detecting a near field communication (NFC) field, such as by near-field communication wireless. The wireless communication optionally uses any of a plurality of communication standards, communication protocols, and communication technologies, such as the Global System for Mobile Communications (GSM (registered trademark)) for mobile communications, Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPAHSPA, DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth (registered trademark), Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (registered trademark) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol including communication protocols not yet developed as of the filing date of this document, but not limited thereto.

[0127] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts this audio data into an electrical signal, and transmits this electrical signal to the speaker 111. The speaker 111 converts the electrical signal into human audible sound waves. Also, the audio circuit 110 receives the electrical signal converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signal into audio data and transmits this audio data to the peripheral device interface 118 for processing. The audio data is optionally obtained from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., 212 of FIG. 2). The headset jack provides an interface between the audio circuit 110 and a detachable audio input / output peripheral device such as an output-only headset or a headset having both outputs (e.g., mono or stereo headphones) and inputs (e.g., a microphone).

[0128] The I / O subsystem 106 couples input / output peripheral devices on the device 100, such as the touch screen 112 and other input control devices 116, to the peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, a light sensor controller 158, an intensity sensor controller 159, a haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / transmit electrical signals to / from the other input control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, the input controller(s) 160 are optionally coupled to (or not coupled to) any of a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. One or more buttons (e.g., 208 in FIG. 2) optionally include up / down buttons for adjusting the volume of the speaker 111 and / or the microphone 113. One or more buttons optionally include push buttons (e.g., 206 in FIG. 2).

[0129] As described in U.S. Patent Application No. 11 / 322,549, filed December 23, 2005, and entitled "Unlocking a Device by Performing Gestures on an Unlock Image," which is hereby incorporated by reference in its entirety, a quick press of this push button optionally unlocks the touch screen 112 or, optionally, initiates a process of unlocking the device using gestures on the touch screen. By pressing the push button (e.g., 206) for a longer time, the power of the device 100 is optionally turned on or off. The functionality of one or more of the buttons is optionally customizable by the user. The touch screen 112 is used to implement virtual buttons or soft buttons and one or more soft keyboards.

[0130] The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives and / or transmits electrical signals to and from the touch screen 112. The touch screen 112 displays visual output to the user. This visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of these visual outputs optionally correspond to user interface objects.

[0131] The touch screen 112 has a touch sensing surface, sensor, or set of sensors that accepts input from a user based on tactile and / or haptic contact. The touch screen 112 and the display controller 156 (along with any associated modules and / or instruction sets within the memory 102) detect contact (and any movement or interruption of the contact) on the touch screen 112 and convert the detected contact into an interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.

[0132] The touch screen 112 optionally uses LCD (liquid crystal display), LPD (light emitting polymer display), or LED (light emitting diode) technology, although in other embodiments other display technologies are used. The touch screen 112 and the display controller 156 optionally use any of a plurality of touch sensing technologies currently known or later developed to detect contact and any movement or interruption of that contact, and those touch sensing technologies include, but are not limited to, capacitance technology, resistive technology, infrared technology, and surface acoustic wave technology, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 112. In an exemplary embodiment, projected mutual capacitance sensing technology, as seen in the iPhone (registered trademark) and iPod Touch (registered trademark) by Apple Inc. (Cupertino, California), is used.

[0133] In some embodiments of touch screen 112, the touch-sensing display optionally is similar to a multi-touch sensing touch pad as described in U.S. Patent Nos. 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman), and / or U.S. Patent Application Publication No. 2002 / 0015024 (A1), each of these documents being incorporated herein by reference in its entirety. However, while touch screen 112 displays visual output from device 100, the touch-sensing touch pad does not provide visual output.

[0134] The touch sensing display in some embodiments of the touch screen 112 is described in the following applications: (1) U.S. Patent Application No. 11 / 381,313, "Multipoint Touch Surface Controller," filed May 2, 2006; (2) U.S. Patent Application No. 10 / 840,862, "Multipoint Touchscreen," filed May 6, 2004; (3) U.S. Patent Application No. 10 / 903,964, "Gestures For Touch Sensitive Input Devices," filed Jul. 30, 2004; (4) U.S. Patent Application No. 11 / 048,264, "Gestures For Touch Sensitive Input Devices," filed Jan. 31, 2005; (5) U.S. Patent Application No. 11 / 038,590, "Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices," filed Jan. 18, 2005; (6) U.S. Patent Application No. 11 / 228,758, "Virtual Input Device Placement On A Touch Screen User Interface," filed Sep. 16, 2005; (7) U.S. Patent Application No. 11 / 228,700, "Operation Of A Computer With A Touch Screen Interface," filed Sep. 16, 2005; (8) U.S. Patent Application No. 11 / 228,737, "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard," filed Sep. 16, 2005; and (9) U.S. Patent Application No. 11 / 367,749, "Multi-Functional Hand-Held Device," filed Mar. 3, 2006. All of these applications are hereby incorporated by reference in their entirety.

[0135] The touch screen 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, this touch screen has a video resolution of about 160 dpi. The user optionally uses any suitable object or appendage such as a stylus, finger, etc. to contact the touch screen 112. In some embodiments, the user interface is designed to function primarily based on finger-based contact and gestures. However, finger-based contact and gestures may be less accurate than stylus-based input due to the larger contact area of the finger on the touch screen. In some embodiments, the device converts rough finger-based input into an accurate pointer / cursor position or command for performing the action desired by the user.

[0136] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touch pad (not shown) for activating or deactivating certain functions. In some embodiments, the touch pad, unlike the touch screen, is a touch-sensitive area of the device that does not display visual output. The touch pad is optionally a separate touch-sensitive surface from the touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.

[0137] The device 100 also includes a power system 162 that supplies power to various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharge system, a power outage detection circuit, a power converter or inverter, a power status indicator (e.g., light-emitting diode (LED)), and any other components associated with the generation, management, and distribution of power within a portable device.

[0138] Device 100 also optionally includes one or more light sensors 164. FIG. 1A shows a light sensor coupled to a light sensor controller 158 within I / O subsystem 106. The light sensor 164 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. The light sensor 164 receives light from the environment projected through one or more lenses and converts the light into data representing an image. The light sensor 164, in conjunction with imaging module 143 (also referred to as a camera module), optionally captures a still image or video. In some embodiments, the light sensor is disposed on the back of device 100, opposite touch screen display 112 on the front of the device, such that the touch screen display is enabled for use as a viewfinder for the acquisition of still and / or video images. In some embodiments, the light sensor is disposed on the front of the device such that the user's image is optionally acquired for a video conference while the user is viewing other video conference participants on the touch screen display. In some embodiments, the position of the light sensor 164 can be changed by the user (e.g., by rotating the lens and sensor within the device housing), whereby a single light sensor 164 is used for both video conferencing and the acquisition of still and / or video images, in conjunction with the touch screen display.

[0139] In addition, device 100 optionally includes one or more contact intensity sensors 165. FIG. 1A shows a contact intensity sensor coupled to an intensity sensor controller 159 within I / O subsystem 106. Contact intensity sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrokinetic force sensors, piezoelectric force sensors, optical force sensors, capacitive touch sensing surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of contact on a touch sensing surface). Contact intensity sensor 165 receives contact intensity information (e.g., pressure information or a surrogate for pressure information) from the surrounding environment. In some embodiments, at least one contact intensity sensor is juxtaposed with or proximate to a touch sensing surface (e.g., touch sensing display system 112). In some embodiments, at least one contact intensity sensor is disposed on the back of device 100, opposite a touch screen display 112 disposed on the front of device 100.

[0140] Device 100 also optionally includes one or more proximity sensors 166. FIG. 1A shows a proximity sensor 166 coupled to the peripheral device interface 118. Alternatively, the proximity sensor 166 is optionally coupled to an input controller 160 within the I / O subsystem 106. The proximity sensor 166 functions as described in U.S. Patent Application Nos. 11 / 241,839, "Proximity Detector In Handheld Device", 11 / 240,788, "Proximity Detector In Handheld Device", 11 / 620,702, "Using Ambient Light Sensor To Augment Proximity Sensor Output", 11 / 586,862, "Automated Response To And Sensing Of User Activity In Portable Devices", and 11 / 638,251, "Methods And Systems For Automatic Configuration Of Peripherals", which are hereby incorporated by reference in their entirety. In some embodiments, the proximity sensor turns off and disables the touch screen 112 when the multifunctional device is placed near the user's ear (e.g., when the user is on a phone call).

[0141] Device 100 also optionally includes one or more haptic output generators 167. FIG. 1A shows a haptic output generator coupled to a haptic feedback controller 161 within I / O subsystem 106. The haptic output generator 167 optionally includes one or more electroacoustic devices such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components that convert an electrical signal into a haptic output on the device). The contact intensity sensor 165 receives haptic feedback generation instructions from the haptic feedback module 133 and generates a haptic output on device 100 that can be sensed by a user of device 100. In some embodiments, at least one haptic output generator is disposed on or proximate to a touch sensing surface (e.g., touch sensing display system 112) and optionally generates a haptic output by moving the touch sensing surface in a vertical direction (e.g., in / out of the surface of device 100) or in a horizontal direction (e.g., back and forth within the same plane as the surface of device 100). In some embodiments, at least one haptic output generator sensor is disposed on the back of device 100, on the opposite side of touch screen display 112 disposed on the front of device 100.

[0142] In addition, device 100 optionally includes one or more accelerometers 168. FIG. 1A shows an accelerometer 168 coupled to the peripheral device interface 118. Alternatively, the accelerometer 168 is optionally coupled to an input controller 160 within the I / O subsystem 106. The accelerometer 168 functions as described, for example, in U.S. Patent Application Publication Nos. 20050190059, "Acceleration-based Theft Detection System for Portable Electronic Devices," and 20060017692, "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer," both of which are hereby incorporated by reference in their entirety. In some embodiments, information is displayed on the touch screen display in a portrait or landscape orientation based on analysis of data received from one or more accelerometers. Device 100 optionally includes, in addition to the accelerometer(s) 168, a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information regarding the position and orientation (e.g., portrait or landscape) of device 100.

[0143] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a touch / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application (or instruction set) 136. Further, in some embodiments, memory 102 (FIG. 1A) or 370 (FIG. 3) stores a device / global internal state 157 as shown in FIGS. 1A and 3. The device / global internal state 157 includes an active application state indicating which application is active if there is a currently active application, a display state indicating which application, view, or other information occupies various regions of the touch screen display 112, a sensor state including information obtained from various sensors and input control devices 116 of the device, and one or more of position information regarding the position and / or orientation of the device.

[0144] The operating system 126 (e.g., an embedded operating system such as Darwin (registered trademark), RTXC (registered trademark), LINUX (registered trademark), UNIX (registered trademark), OS X (registered trademark), iOS (registered trademark), WINDOWS (registered trademark), or VxWorks (registered trademark)) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware components and software components.

[0145] The communication module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external ports 124 (e.g., Universal Serial Bus (USB), FIREWIRE (registered trademark), etc.) are adapted to connect to other devices either directly or indirectly via a network (e.g., the Internet, a wireless LAN, etc.). In some embodiments, the external port is the same or similar multi-pin (e.g., 30-pin) connector used on an iPod (registered trademark) (a trademark of Apple Inc.) device and / or a compatible multi-pin (e.g., 30-pin) connector.

[0146] The contact / motion module 130 optionally detects contact with the touch screen 112 and other touch-sensing devices (e.g., a touch pad or a physical click wheel) (in cooperation with the display controller 156). The contact / motion module 130 includes various software components for performing various operations related to the detection of contact, such as determining whether contact has occurred (e.g., detecting an event of a finger being lowered), determining the intensity of the contact (e.g., the force or pressure of the contact, or an alternative to the force or pressure of the contact), determining whether there is movement of the contact, tracking movement across the touch-sensing surface (e.g., detecting one or more events of a finger being dragged), and determining whether the contact has stopped (e.g., detecting an event of a finger being raised or an interruption of the contact). The contact / motion module 130 receives contact data from the touch-sensing surface. Determining the movement of the contact point, represented by a series of contact data, optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These operations are optionally applied to a single contact (e.g., contact with one finger) or multiple simultaneous contacts (e.g., "multi-touch" / contact with multiple fingers). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touch pad.

[0147] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by the user (e.g., to determine whether the user has "clicked" on an icon). In some embodiments, at least a subset of the intensity thresholds are determined according to software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of specific physical actuators and can be adjusted without changing the physical hardware of the device 100). For example, the mouse "click" threshold of a trackpad or touch screen display can be set to any of a wide range of default thresholds without changing the hardware of that trackpad or touch screen display. Further, in some implementations, the user of the device is provided with software settings for adjusting one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by using system-level click "intensity" parameters to adjust multiple intensity thresholds at once).

[0148] The contact / motion module 130 optionally detects gesture inputs by the user. Different gestures on the touch-sensing surface have different contact patterns (e.g., the detected movement, timing, and / or intensity of the contact is different). Thus, gestures are optionally detected by detecting a specific contact pattern. For example, detecting a finger tap gesture includes detecting an event of lowering a finger (e.g., at the location of an icon) and subsequently detecting an event of raising (lifting off) the finger at the same location (or substantially the same location) as the event of lowering the finger. As another example, detecting a finger swipe gesture on the touch-sensing surface includes detecting an event of lowering a finger, subsequently detecting one or more events of dragging the finger, and then subsequently detecting an event of raising (lifting off) the finger.

[0149] The graphic module 132 includes various known software components for rendering and displaying graphics on the touch screen 112 or other display, including components for changing the visual effects of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual attributes). As used herein, the term "graphic" includes any object that can be displayed to the user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, and the like.

[0150] In some embodiments, the graphic module 132 stores data representing the graphics used. Each graphic is optionally assigned a corresponding code. The graphic module 132 receives one or more codes specifying the graphics to be displayed, along with coordinate data and other graphic characteristic data as needed, from an application or the like, and then generates the image data of the screen to be output to the display controller 156.

[0151] The tactile feedback module 133 includes various software components for generating instructions to be used by the tactile output generator(s) 167 to generate tactile output at one or more locations on the device 100 in response to user interaction with the device 100.

[0152] The text input module 134 is optionally a component of the graphic module 132 and provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application that requires text input).

[0153] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for location-based dialing, to the camera 143 as metadata for photos / videos, and to applications that provide location-based services such as weather widgets, local yellow page widgets, and map / navigation widgets).

[0154] The application 136 optionally includes the following modules (or sets of instructions) or subsets or supersets thereof. ● Contact module 137 (sometimes referred to as an address book or contact list), ● Phone module 138, ● Video conferencing module 139, ● Email client module 140, ● Instant messaging (IM) module 141, ● Training support module 142, ● Camera module 143 for still and / or video images, ● Image management module 144, ● Video player module, ● Music player module, ● Browser module 147, ● Calendar module 148, The widget module 149 optionally includes one or more of the weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets obtained by the user, as well as the user-created widget 149-6. ● Widget creation module 150 for creating the user-created widget 149-6, ● Search module 151, ● A video and music player module 152 that integrates a video player module and a music player module. ● A memo module 153. ● A map module 154, and / or ● An online video module 155.

[0155] Examples of other applications 136 that are optionally stored in the memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA (registered trademark)-compatible applications, encryption, digital rights management, voice recognition, and voice replication.

[0156] In cooperation with the touch screen 112, the display controller 156, the contact / motion module 130, the graphic module 132, and the text input module 134, the contact module 137 is optionally used to manage an address book or contact list (e.g., stored in the application internal state 192 of the contact module 137 in the memory 102 or the memory 370). This management includes adding a name(s) to the address book, deleting a name(s) from the address book, associating a phone number(s), an email address(es), an address(es), or other information with a name, associating an image with a name, classifying and sorting names, providing a phone number or email address to initiate and / or facilitate communication by the phone 138, the video conferencing module 139, the email 140, or the IM 141, etc.

[0157] In cooperation with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the contact / motion module 130, the graphic module 132, and the text input module 134, the phone module 138 inputs a string corresponding to a phone number, accesses one or more phone numbers in the contact module 137, modifies the input phone number, dials the corresponding phone number, conducts a conversation, and optionally disconnects or hangs up the phone when the conversation is completed. As described above, wireless communication optionally uses any of a plurality of communication standards, communication protocols, and communication technologies.

[0158] In cooperation with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the optical sensor 164, the optical sensor controller 158, the contact / motion module 130, the graphic module 132, the text input module 134, the contact module 137, and the phone module 138, the video conferencing module 139 includes executable instructions to start, conduct, and end a video conference between the user and one or more other participants according to the user's instructions.

[0159] In cooperation with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphic module 132, and the text input module 134, the email client module 140 includes executable instructions to create, send, receive, and manage emails in response to the user's instructions. In relation to the image management module 144, the email client module 140 makes it very easy to create and send emails with still or video images captured by the camera module 143.

[0160] In cooperation with the RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, and text input module 134, the instant messaging module 141 includes executable instructions for inputting a string corresponding to an instant message, modifying the input characters, transmitting the corresponding instant message (e.g., using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for phone-based instant messages, or using XMPP, SIMPLE, or IMPS for Internet-based instant messages), receiving an instant message, and viewing the received instant message. In some embodiments, the instant messages transmitted and / or received optionally include graphics, photos, audio files, video files, and / or other attached files, such as those supported by MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages transmitted using SMS or MMS) and Internet-based messages (e.g., messages transmitted using XMPP, SIMPLE, or IMPS).

[0161] In cooperation with the RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, text input module 134, GPS module 135, map module 154, and music player module, the training support module 142 includes executable instructions for creating a training (e.g., having time, distance, and / or calorie consumption goals), communicating with a training sensor (sports device), receiving training sensor data, calibrating sensors used to monitor the training, selecting and playing music for the training, displaying, storing, and transmitting training data.

[0162] In conjunction with touch screen 112, display controller 156, light sensor(s) 164, light sensor controller 158, contact / motion module 130, graphic module 132, and image management module 144, camera module 143 includes executable instructions to capture still images or video (including video streams), store them in memory 102, modify the characteristics of the still images or video, or delete the still images or video from memory 102.

[0163] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphic module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions to arrange, modify (e.g., edit) or otherwise manipulate, label, delete, present (e.g., as a digital slide show or album), and store still images and / or video images.

[0164] In conjunction with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, and text input module 134, browser module 147 includes executable instructions to browse the Internet according to user instructions, including searching for, linking to, receiving, and displaying web pages or portions thereof, as well as attached files and other files linked to the web pages.

[0165] In cooperation with the RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, text input module 134, email client module 140, and browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying, and storing a calendar and data associated with the calendar (e.g., calendar items, to-do lists, etc.) according to user commands.

[0166] In cooperation with the RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, text input module 134, and browser module 147, the widget module 149 is a mini-application (e.g., weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) that is optionally downloaded and used by the user, or a mini-application created by the user (e.g., user-created widget 149-6). In some embodiments, the widget includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript (registered trademark) files. In some embodiments, the widget includes XML (Extensible Markup Language) files and JavaScript files (e.g., Yahoo! widget).

[0167] In cooperation with the RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, text input module 134, and browser module 147, the widget creation module 150 is optionally used by the user to create a widget (e.g., change a user-specified location on a web page to a widget).

[0168]

[0169]

[0168]

[0170]

[0171] In cooperation with the RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, text input module 134, GPS module 135, and browser module 147, the map module 154 is optionally used to receive, display, modify, and store maps and data associated with the maps (e.g., driving directions, data about stores and other points of interest near a specific location or in its vicinity, and other location-based data) according to a user's command.

[0172] In cooperation with touch screen 112, display controller 156, contact / motion module 130, graphic module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 enables a user to access, view, receive (e.g., by streaming and / or downloading), play (e.g., on the touch screen or on an external display connected via external port 124), send an email including a link to the particular online video, and otherwise manage the online video in one or more file formats such as H.264. In some embodiments, instead of email client module 140, instant messaging module 141 is used to send a link to the particular online video. Further description of the online video application can be found in U.S. Provisional Patent Application No. 60 / 936,562, filed Jun. 20, 2007, "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos", and U.S. Patent Application No. 11 / 968,067, filed Dec. 31, 2007, "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos", the contents of which are hereby incorporated by reference in their entirety.

[0173] Each of the modules and applications identified above corresponds to a set of executable instructions that perform one or more of the functions described above and the methods described in this application (e.g., methods implemented by a computer and other information processing methods described in this specification). These modules (e.g., instruction sets) need not be implemented as separate software program procedures or modules, and thus, in various embodiments, various subsets of these modules are optionally combined or rearranged in other ways. For example, the video player module may optionally be combined with the music player module to form a single module (e.g., the video and music player module 152 of FIG. 1A). In some embodiments, the memory 102 optionally stores a subset of the modules and data structures identified above. Further, the memory 102 optionally stores additional modules and data structures not described above.

[0174] In some embodiments, the device 100 is a device in which the operation of a set of predetermined functions in the device is performed only through the touch screen and / or touch pad. By using the touch screen and / or touch pad as the main input control device for the device 100 to operate, the number of physical input control devices (push buttons, dials, etc.) on the device 100 is optionally reduced.

[0175] The set of predetermined functions performed only through the touch screen and / or touch pad optionally includes navigation between user interfaces. In some embodiments, the touch pad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to the main menu, home menu, or root menu. In such embodiments, the "menu button" is implemented using the touch pad. In some other embodiments, the menu button is a physical push button or other physical input control device rather than the touch pad.

[0176] FIG. 1B is a block diagram showing exemplary components for event processing according to some embodiments. In some embodiments, memory 102 (FIG. 1A) or 370 (FIG. 3) includes an event sorting unit 170 (e.g., within operating system 126) and respective applications 136-1 (e.g., any of the aforementioned applications 137-151, 155, 380-390).

[0177] The event sorting unit 170 receives event information and determines the application 136-1 to which the event information is to be delivered and the application view 191 of the application 136-1. The event sorting unit 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, the application 136-1 includes an application internal state 192 indicating the current application view(s) displayed on the touch-sensitive display 112 when the application is active or running. In some embodiments, the device / global internal state 157 is used by the event sorting unit 170 to determine which application(s) is / are currently active, and the application internal state 192 is used by the event sorting unit 170 to determine the application view 191 to which the event information is to be delivered.

[0178] In some embodiments, the application internal state 192 includes additional information such as resume information used when the application 136-1 resumes execution, user interface state information indicating information being displayed by or ready to be displayed by the application 136-1, a state queue for waiting to allow the user to return to a previous state or view of the application 136-1, and a redo / undo queue of previous actions performed by the user.

[0179] The event monitor 171 receives event information from the peripheral device interface 118. The event information includes information on sub-events (e.g., a user touch on the touch-sensitive display 112 as part of a multi-touch gesture). The peripheral device interface 118 transmits information received from the I / O subsystem 106, or sensors such as the proximity sensor 166, the accelerometer(s) 168, and / or the microphone 113 (via the audio circuit 110). The information that the peripheral device interface 118 receives from the I / O subsystem 106 includes information from the touch-sensitive display 112 or the touch-sensitive surface.

[0180] In some embodiments, the event monitor 171 transmits requests to the peripheral device interface 118 at predetermined intervals. In response, the peripheral device interface 118 transmits event information. In other embodiments, the peripheral device interface 118 transmits event information only when there is an important event (e.g., receiving an input that exceeds a predetermined noise threshold and / or is longer than a predetermined duration).

[0181] In some embodiments, the event sorter 170 also includes a hit view determination module 172 and / or an active event recognition unit determination module 173.

[0182] The hit view determination module 172 provides a software procedure for determining where within one or more views a sub-event has occurred when the touch-sensitive display 112 displays more than one view. The views are composed of controls and other elements that a user can view on the display.

[0183] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, within which information is displayed and touch-based gestures occur. The application views (for each respective application) where touches are detected optionally correspond to a program level within the program hierarchy or view hierarchy of the application. For example, the lowest-level view where a touch is detected is optionally referred to as the hit view, and the set of events recognized as appropriate input is optionally determined based at least in part on the hit view of the initial touch that initiates a touch-based gesture.

[0184] The hit view determination module 172 receives information related to sub-events of a touch-based gesture. When the application has a plurality of hierarchically structured views, the hit view determination module 172 identifies the hit view as the lowest-level view within the hierarchy in which the sub-event is to be processed. In most situations, this hit view is the lowest-level view where the start sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once the hit view is identified by the hit view determination module 172, that hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0185] The active event recognition unit determination module 173 determines which view(s) within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognition unit determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognition unit determination module 173 determines that all views including the physical location of the sub-event are views that are actively involved, and thus determines that all views that are actively involved should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely limited to an area related to one particular view, the upper-level views within the hierarchy continue to be views that are actively involved.

[0186] The event dispatcher module 174 dispatches event information to the event recognition unit (e.g., event recognition unit 180). In embodiments including the active event recognition unit determination module 173, the event dispatcher module 174 dispatches event information to the event recognition unit determined by the active event recognition unit determination module 173. In some embodiments, the event dispatcher module 174 stores the event information obtained by the corresponding event receiver 182 in the event queue.

[0187] In some embodiments, the operating system 126 includes the event sorter 170. Alternatively, the application 136-1 includes the event sorter 170. In still other embodiments, the event sorter 170 is an independent module or part of another module stored in the memory 102, such as the touch / motion module 130.

[0188] In some embodiments, application 136-1 includes a plurality of event processing units 190 and one or more application views 191, each including instructions for processing touch events that occur within respective views of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognition units 180. Typically, each application view 191 includes a plurality of event recognition units 180. In other embodiments, one or more of the event recognition units 180 are part of a separate module, such as a user interface kit (not shown) or a higher-level object from which application 136-1 inherits methods and other characteristics. In some embodiments, the corresponding event processing unit 190 includes one or more of event data 179 received from data update unit 176, object update unit 177, GUI update unit 178, and / or event sorting unit 170. The event processing unit 190 optionally utilizes or calls data update unit 176, object update unit 177, or GUI update unit 178 to update the internal state 192 of the application. Alternatively, one or more of the application views 191 include one or more corresponding event processing units 190. Also, in some embodiments, one or more of data update unit 176, object update unit 177, and GUI update unit 178 are included in the corresponding application view 191.

[0189] Each event recognition unit 180 receives event information (e.g., event data 179) from event sorting unit 170 and identifies an event from the event information. The event recognition unit 180 includes an event receiving unit 182 and an event comparing unit 184. In some embodiments, the event recognition unit 180 also includes at least a subset of metadata 183 and event distribution instructions 188 (optionally including sub-event distribution instructions).

[0190] The event receiving unit 182 receives event information from the event sorting unit 170. The event information includes information about sub-events, for example, information about a touch or a movement of a touch. Depending on the sub-event, the event information also includes additional information such as the position of the sub-event. When the sub-event is related to the movement of a touch, the event information also optionally includes the speed and direction of the sub-event. In some embodiments, the event includes a rotation of the device from one orientation to another (e.g., from portrait to landscape, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the posture of the device).

[0191] The event comparison unit 184 compares the event information with the definitions of predefined events or sub-events and, based on the comparison, determines an event or sub-event, or determines or updates the state of an event or sub-event. In some embodiments, the event comparison unit 184 includes an event definition 186. The event definition 186 includes definitions of events (such as a sequence of predetermined sub-events), such as event 1 (187-1) and event 2 (187-2). In some embodiments, the sub-events within an event (187) include, for example, the start of a touch, the end of a touch, the movement of a touch, the cancellation of a touch, and multiple touches. In one example, the definition of event 1 (187-1) is a double-tap on a displayed object. This double-tap includes, for example, a first touch (start of touch) on the display object for a predetermined stage, a first lift-off (end of touch) for a predetermined stage, a second touch (start of touch) on the display object for a predetermined stage, and a second lift-off (end of touch) for a predetermined stage. In another example, the definition of event 2 (187-2) is a drag on a displayed object. This drag operation includes, for example, a touch (or contact) on the display object for a predetermined stage, a movement of the touch across the touch-sensitive display 112, and a lift-off of the touch (end of touch). In some embodiments, an event also includes information regarding one or more associated event processing units 190.

[0192] In some embodiments, event definition 187 includes the definition of events for each user interface object. In some embodiments, event comparison unit 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view where three user interface objects are displayed on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparison unit 184 performs a hit test to determine which of the three user interface objects is associated with that touch (sub-event). If each displayed object is associated with a corresponding event processing unit 190, the event comparison unit uses the result of the hit test to determine which event processing unit 190 should be activated. For example, event comparison unit 184 selects the event processing unit associated with the sub-event and object that triggered the hit test.

[0193] In some embodiments, the definition for each event (187) also includes a delay action that delays the delivery of event information until it is determined whether the sequence of sub-events corresponds to the event type of the event recognition unit.

[0194] If each event recognition unit 180 determines that a sequence of sub-events does not match any of the events in event definition 186, each event recognition unit 180 enters a state of event impossible, event failure, or event end, and then ignores subsequent sub-events of the touch gesture. In this situation, if there are other event recognition units that remain active for the hit view, that event recognition unit continues to track and process the sub-events of the ongoing touch gesture.

[0195] In some embodiments, the corresponding event recognition unit 180 includes metadata 183 having configurable properties, flags, and / or lists indicating how the event distribution system should actively participate in executing sub - event distribution. In some embodiments, the metadata 183 includes configurable properties, flags, and / or lists indicating how event recognition units interact with each other or how they can interact with each other. In some embodiments, the metadata 183 includes configurable properties, flags, and / or lists indicating whether sub - events are distributed to various levels in the view hierarchy or program hierarchy.

[0196] In some embodiments, each event recognition unit 180 activates an event processing unit 190 associated with the event when one or more specific sub - events of the event are recognized. In some embodiments, each event recognition unit 180 distributes event information associated with the event to the event processing unit 190. Activating the event processing unit 190 is separate from sending (and deferring sending) sub - events to each hit view. In some embodiments, the event recognition unit 180 sets a flag associated with the recognized event, and the event processing unit 190 associated with that flag catches the flag and executes a predefined process.

[0197] In some embodiments, the event distribution command 188 includes a sub - event distribution command that distributes event information about sub - events without activating the event processing unit. Instead, the sub - event distribution command distributes event information to an event processing unit associated with a series of sub - events or to a view actively involved. The event processing unit associated with a series of sub - events or an actively involved view receives the event information and executes a predetermined process.

[0198] In some embodiments, the data update unit 176 creates and updates the data used in the application 136-1. For example, the data update unit 176 updates the phone numbers used in the contact module 137 or stores the video files used in the video player module. In some embodiments, the object update unit 177 creates and updates the objects used in the application 136-1. For example, the object update unit 177 creates a new user interface object or updates the position of the user interface object. The GUI update unit 178 updates the GUI. For example, the GUI update unit 178 prepares the display information and sends the display information to the graphic module 132 for display on the touch-sensitive display.

[0199] In some embodiments, the event processing unit(s) 190 includes or has access to the data update unit 176, the object update unit 177, and the GUI update unit 178. In some embodiments, the data update unit 176, the object update unit 177, and the GUI update unit 178 are included in a single module of their respective application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0200] It should be understood that the above description regarding the event processing of the user's touch on the touch-sensitive display is applicable to other forms of user input for operating the multifunctional device 100 with an input device, not all of which are necessarily initiated on the touch screen. For example, the movement of the mouse and the pressing of the mouse button, contact movements such as taps, drags, and scrolls on the touchpad, pen stylus input, movement of the device, verbal commands, detected eye movements, biometric input, and / or any combination thereof, optionally, are used as inputs corresponding to sub-events that define events to be recognized.

[0201] Figure 2 shows a portable multifunctional device 100 having a touch screen 112, according to some embodiments. The touch screen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment, as well as in other embodiments described below, the user can select one or more of those graphics by performing a gesture on the graphics, for example, using one or more fingers 202 (not drawn to scale in the figure) or one or more styli 203 (not drawn to scale in the figure). In some embodiments, the selection of one or more graphics is performed when the user interrupts contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, upward, and / or downward), and / or rolling of a finger in contact with the device 100 (from right to left, from left to right, upward, and / or downward). In some implementations or situations, an accidental contact with a graphic does not select the graphic. For example, if the gesture corresponding to the selection is a tap, a swipe gesture across an application icon does not optionally select the corresponding application.

[0202] Also, the device 100 optionally includes one or more physical buttons, such as a "home" or menu button 204. As described above, the menu button 204 is optionally used to navigate to any application 136 within a set of applications optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key within a GUI displayed on the touch screen 112.

[0203] In some embodiments, device 100 includes a touch screen 112, a menu button 204, a push button 206 for turning the device on / off and locking the device, volume adjustment button(s) 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and holding it in the depressed state for a predetermined period, to lock the device by pressing the button and releasing it before a predetermined period has elapsed, and / or to unlock the device or initiate an unlock process. In an alternative embodiment, device 100 also accepts verbal input via a microphone 113 to activate or deactivate some functions. Device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on the touch screen 112 and / or one or more haptic output generators 167 for generating haptic output to the user of device 100.

[0204] Figure 3 is a block diagram of an exemplary multifunctional device having a display and a touch sensing surface, according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia playback device, a navigation device, an educational device (such as a child's learning toy), a gaming system, or a control device (e.g., a home or industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more networks or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. Communication bus 320 optionally includes circuitry (sometimes called a chipset) for interconnecting and controlling communications between system components. Device 300 includes an input / output (I / O) interface 330 having a display 340, typically a touch screen display. I / O interface 330 also optionally includes a keyboard and / or a mouse (or other pointing device) 350, a touch pad 355, a haptic output generator 357 (such as haptic output generator 167 described above with reference to FIG. 1A) for generating haptic output on device 300, and sensors 359 (e.g., light sensors, acceleration sensors, proximity sensors, touch sensing sensors, and / or haptic intensity sensors similar to haptic intensity sensor 165 described above with reference to FIG. 1A). Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices, and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 370 optionally includes one or more storage devices remotely located from CPU 310. In some embodiments, memory 370 stores programs, modules, and data structures similar to, or a subset of, the programs, modules, and data structures stored in memory 102 of portable multifunctional device 100 (FIG. 1A).Furthermore, memory 370 optionally stores additional programs, modules, and data structures that do not exist in memory 102 of the portable multifunctional device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, a word processing module 384, a website creation module 386, a disk authoring module 388, and / or a spreadsheet module 390, while memory 102 of the portable multifunctional device 100 (FIG. 1A) optionally does not store these modules.

[0205] Each of the elements of FIG. 3 identified above is optionally stored in one or more of the aforementioned memory devices. Each of the modules identified above corresponds to an instruction set for performing the above-described functions. The modules or programs (e.g., instruction sets) identified above need not be implemented as separate software programs, procedures, or modules, and thus in various embodiments, various subsets of these modules are optionally combined or otherwise reconfigured. In some embodiments, memory 370 optionally stores a subset of the modules and data structures identified above. Further, memory 370 optionally stores additional modules and data structures not described above.

[0206] Attention is now directed to embodiments of the user interface, which are optionally implemented, for example, on the portable multifunctional device 100.

[0207] FIG. 4A shows an exemplary user interface related to an application menu on the portable multifunctional device 100 according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, the user interface 400 includes the following elements, or subsets or supersets thereof. ● Signal strength indicator(s) 402 related to wireless communication(s) such as cellular signal and Wi-Fi signal, ● Time 404, ● Bluetooth (R) indicator 405, ● Battery status indicator 406, ● Tray 408 including icons for frequently used applications such as the following, ○ Icon 416 for phone module 138 labeled "Phone", optionally including indicator 414 of the number of missed calls or voice mail messages, ○ Icon 418 for email client module 140 labeled "Mail", optionally including indicator 410 of the number of unread emails, ○ Icon 420 for browser module 147 labeled "Browser", and ○ Icon 422 for video and music player module 152, also called iPod (trademark of Apple Inc.) module 152, labeled "iPod", and ● Icons for other applications such as the following, ○ Icon 424 for IM module 141 labeled "Message", ○ Icon 426 for calendar module 148 labeled "Calendar", ○ Icon 428 for image management module 144 labeled "Photo", ○ Icon 430 for camera module 143 labeled "Camera", ○ Icon 432 for online video module 155 labeled "Online Video", ○ Icon 434 for stock price widget 149-2 labeled "Stock Price", ○ Icon 436 for map module 154 labeled "Map", ○ Icon 438 for weather widget 149-1 labeled "Weather", ○ An icon 440 for the alarm clock widget 149-4 labeled "Clock", ○ An icon 442 for the training support module 142 labeled "Training Support", ○ An icon 444 for the memo module 153 labeled "Memo", and ○ An icon 446 for the settings application or module labeled "Settings" that provides access to the settings of the device 100 and its various applications 136.

[0208] Note that the icon labels shown in FIG. 4A are merely illustrative. For example, other labels such as "Music" or "Music Player" for the icon 422 for the video and music player module 152 may optionally be used for the various application icons. In some embodiments, the label for each application icon includes the name of the application corresponding to that application icon. In some embodiments, the label for a particular application icon is different from the name of the application corresponding to that particular application icon.

[0209] FIG. 4B shows an exemplary user interface on a device (e.g., device 300 of FIG. 3) having a touch sensing surface 451 (e.g., the tablet or touch pad 355 of FIG. 3) separate from the display 450 (e.g., touch screen display 112). The device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of the sensors 359) for detecting the intensity of contact on the touch sensing surface 451 and / or one or more haptic output generators 357 for generating haptic output to the user of the device 300.

[0210] Some of the following examples are described with reference to input on a touch screen display 112 (when the touch sensing surface and the display are combined), but in some embodiments, the device detects input on a touch sensing surface separate from the display, as shown in FIG. 4B. In some embodiments, this touch sensing surface (e.g., 451 in FIG. 4B) has a major axis (e.g., 452 in FIG. 4B) corresponding to the major axis (e.g., 453 in FIG. 4B) on the display (e.g., 450). According to these embodiments, the device detects contact with the touch sensing surface 451 (e.g., 460 and 462 in FIG. 4B) at positions corresponding to respective positions on the display (e.g., in FIG. 4B, 460 corresponds to 468 and 462 corresponds to 470). Thus, when the touch sensing surface is separate from the display, user input detected by the device on the touch sensing surface (e.g., 451 in FIG. 4B) (e.g., contacts 460 and 462 and their movement) is used by the device to operate the user interface on the display of the multifunctional device (e.g., 450 in FIG. 4B). It should be understood that a similar method is optionally used for other user interfaces described herein.

[0211] Furthermore, although the following examples are mainly described with reference to finger inputs (e.g., finger contact, finger tap gesture, finger swipe gesture), it should be understood that in some embodiments, one or more of those finger inputs may be replaced by inputs from another input device (e.g., mouse-based input, or stylus input). For example, a swipe gesture may optionally be replaced by the movement of a cursor along a swipe path followed by a mouse click (e.g., instead of contact movement). As another example, a tap gesture may optionally be replaced by a mouse click while the cursor is positioned over the location of the tap gesture (e.g., instead of detecting contact and subsequently stopping the detection of the contact). Similarly, it should be understood that when multiple user inputs are detected simultaneously, multiple computer mice may optionally be used simultaneously, or a mouse and finger contact may optionally be used simultaneously.

[0212] FIG. 5A shows an exemplary personal electronic device 500. The device 500 includes a body 502. In some embodiments, the device 500 can include some or all of the features described with respect to devices 100 and 300 (e.g., FIGS. 1A - 4B). In some embodiments, the device 500 has a touch-sensing display screen 504, which is hereinafter the touch screen 504. Alternatively, or in addition to the touch screen 504, the device 500 has a display and a touch-sensing surface. Similar to devices 100 and 300, in some embodiments, the touch screen 504 (or the touch-sensing surface) optionally includes one or more intensity sensors that detect the intensity of an applied contact (e.g., a touch). One or more intensity sensors of the touch screen 504 (or the touch-sensing surface) can provide output data representing the intensity of the touch. The user interface of the device 500 can respond to a touch based on the intensity of the touch, which means that touches of different intensities can call different user interface operations on the device 500.

[0213] Exemplary techniques for detecting and processing touch intensity are found, for example, in the related applications International Patent Application No. PCT / US2013 / 040061, filed May 8, 2013, entitled "Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application" (published as International Publication No. WO 2013 / 169849), and International Patent Application No. PCT / US2013 / 069483, filed Nov. 11, 2013, entitled "Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships" (published as International Publication No. WO 2014 / 105276), each of which is hereby incorporated by reference in its entirety.

[0214] In some embodiments, device 500 has one or more input mechanisms 506 and input mechanism 508. Input mechanisms 506 and 508, if included, can be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, device 500 has one or more attachment mechanisms. Such attachment mechanisms, if included, can enable device 500 to be attached to, for example, hats, eyewear, earrings, necklaces, shirts, jackets, bracelets, watch bands, chains, pants, belts, shoes, wallets, backpacks, etc. These attachment mechanisms enable device 500 to be worn by a user.

[0215] FIG. 5B shows an exemplary personal electronic device 500. In some embodiments, device 500 can include some or all of the components described with respect to FIGS. 1A, 1B, and 3. Device 500 has a bus 512 that operably couples an I / O section 514 to one or more computer processors 516 and a memory 518. The I / O section 514 can be connected to a display 504, which can have a touch sensing component 522 and optionally an intensity sensor 524 (e.g., a contact intensity sensor). Additionally, the I / O section 514 can be connected to a communication unit 530 that receives application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular, and / or other wireless communication techniques. Device 500 can include an input mechanism 506 and / or 508. For example, input mechanism 506 can optionally be a rotatable input device, or a depressible and rotatable input device, or a depressible and rotatable input device. In some embodiments, input mechanism 508 can optionally be a button.

[0216] In some embodiments, input mechanism 508 can optionally be a microphone. The personal electronic device 500 can optionally include various sensors such as a GPS sensor 532, an accelerometer 534, a direction sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or combinations thereof, all of which can be operably connected to the I / O section 514.

[0217] The memory 518 of the personal electronic device 500 can include one or more non-transitory computer-readable storage media that store computer-executable instructions, which, when executed by one or more computer processors 516, can cause the computer processor to execute, for example, the techniques described below including processes 700, 1000, 1300, 1600, 1900, 2100, 2300, 2600, 2900, and 3200 (FIGS. 7A-7C, 10, 13, 16, 19, 21, 23, 26, 29, and 32). The computer-readable storage media can be any media that can tangibly contain or store computer-executable instructions for use by or in connection with an instruction execution system, apparatus, or device. In some examples, the storage media is a transitory computer-readable storage media. In some embodiments, the storage media is a non-transitory computer-readable storage media. Non-transitory computer-readable storage media can include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, CDs, DVDs, or optical disks based on Blu-ray technology, as well as persistent solid-state memories such as flash, solid-state drives. The personal electronic device 500 is not limited to the components and configurations of FIG. 5B and can include other or additional components in a plurality of configurations.

[0218] As used herein, the term "affordance" refers to a user-interactive graphical user interface object that is optionally displayed on the display screen of devices 100, 300, and / or 500 (FIGS. 1A, 3, and 5A-5B). For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) each optionally constitute an affordance.

[0219] As used herein, the term "focus selector" refers to an input element that indicates the current part of the user interface with which the user is interacting. In some implementations that include a cursor or other location marker, the cursor acts as the "focus selector", and thus while the cursor is positioned over a particular user interface element (e.g., a button, window, slider, or other user interface element), when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., the touchpad 355 of FIG. 3 or the touch-sensitive surface 451 of FIG. 4B), the particular user interface element is adjusted according to the detected input. In some implementations that include a touch screen display (e.g., the touch-sensitive display system 112 of FIG. 1A or the touch screen 112 of FIG. 4A) that enables direct interaction with user interface elements on the touch screen display, since the contact detected on the touch screen serves as the "focus selector", when an input (e.g., a press input by contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touch screen display, that particular user interface element is adjusted according to the detected input. In some implementations, the focus is moved from one region of the user interface to another region of the user interface without accompanying movement of the corresponding cursor or movement of the contact on the touch screen display (e.g., by using the tab key or arrow keys to move the focus from one button to another button), and in these implementations, the focus selector moves according to the movement of the focus between different regions of the user interface. Regardless of the specific form presented by the focus selector, the focus selector is generally a user interface element (or a contact on the touch screen display) that is controlled by the user to convey (e.g., by indicating to the device the element of the user interface with which the user intends to interact) the user's intended interaction with the user interface.For example, while a press input is detected on a touch sensing surface (e.g., a touch pad or a touch screen), the position of a focus selector (e.g., a cursor, a contact, or a selection box) over the corresponding button indicates that the user intends to activate that corresponding button (as opposed to other user interface elements shown on the display of the device).

[0220] As used in this specification and the claims, the term "characteristic strength" of a contact refers to the characteristics of that contact based on one or more strengths of the contact. In some embodiments, the characteristic strength is based on a plurality of strength samples. The characteristic strength is optionally based on a set of strength samples collected during a predetermined period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) of a predetermined number of strength samples, i.e., related to a predetermined event (e.g., after detecting the contact, before detecting the lift-off of the contact, before or after detecting the start of movement of the contact, before detecting the end of the contact, before or after detecting an increase in the strength of the contact, and / or before or after detecting a decrease in the strength of the contact). The characteristic strength of a contact is optionally based on one or more of the maximum value of the strength of the contact, the mean value of the strength of the contact, the average value of the strength of the contact, the top 10 percentile value of the strength of the contact, a value that is half of the maximum value of the strength of the contact, a value that is 90 percent of the maximum value of the strength of the contact, etc. In some embodiments, the duration of the contact is used to determine the characteristic strength (e.g., when the characteristic strength is the average of the strength of the contact over time). In some embodiments, the characteristic strength is compared to a set of one or more strength thresholds to determine whether an operation has been performed by a user. For example, the set of one or more strength thresholds optionally includes a first strength threshold and a second strength threshold. In this example, a contact having a characteristic strength that does not exceed the first threshold results in a first action, a contact having a characteristic strength that exceeds the first threshold and does not exceed the second threshold results in a second action, and a contact having a characteristic strength that exceeds the second threshold results in a third action. In some embodiments, the comparison of the characteristic strength to one or more thresholds is used not to determine which of the first or second actions to perform, but to determine whether to perform one or more actions (e.g., to perform each action or to cancel the execution of each action).

[0221] FIG. 5C shows detecting a plurality of contacts 552A-552E on a touch-sensing display screen 504 by a plurality of intensity sensors 524A-524D. FIG. 5C additionally includes an intensity diagram showing current intensity measurements of the intensity sensors 524A-524D in intensity units. In this example, the intensity measurements of intensity sensors 524A and 524D are each 9 intensity units, and the intensity measurements of intensity sensors 524B and 524C are each 7 intensity units. In some implementations, the total intensity is the sum of the intensity measurements of these plurality of intensity sensors 524A-524D, which in this example is 32 intensity units. In some implementations, each contact is assigned a respective intensity that is a portion of this total intensity. FIG. 5D shows the assignment of the total intensity to these contacts 552A-552E based on the distances of the contacts 552A-552E from the center of force 554. In this example, each of contact 552A, contact 552B, and contact 552E is assigned a contact intensity of 8 intensity units of the total intensity, and each of contact 552C and contact 552D is assigned a contact intensity of 4 intensity units of the total intensity. More generally, in some implementations, each contact j is assigned a respective intensity Ij that is a portion of the total intensity A according to a predetermined mathematical function Ij = A·(Dj / ΣDi), where Dj is the distance from the center of force to each respective contact j, and ΣDi is the sum of the distances from the center of force to all respective contacts (e.g., from i = 1 to the last). The operations described with reference to FIGS. 5C-5D can be performed using an electronic device similar or identical to device 100, 300, or 500. In some implementations, the characteristic intensity of a contact is based on one or more intensities of that contact. In some implementations, intensity sensors are used to determine a single characteristic intensity (e.g., a single characteristic intensity of a single contact). Note that the intensity diagrams are not part of the user interface being presented but are included in FIGS. 5C-5D for the reader's sake.

[0222] In some embodiments, for the purpose of determining characteristic intensity, a portion of the gesture is identified. For example, the touch sensing surface optionally receives a continuous swipe contact that moves from a starting position to reach an ending position, at which point the intensity of the contact increases. In this example, the characteristic intensity of the contact at the ending position is optionally based on only a portion of that continuous swipe contact (e.g., only the portion of the swipe contact at the ending position), rather than the entire swipe contact. In some embodiments, a smoothing algorithm is optionally applied to the intensity of the swipe contact before determining the characteristic intensity of the contact. For example, this smoothing algorithm optionally includes one or more of a non-weighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some situations, these smoothing algorithms eliminate minor increases or decreases in the intensity of the swipe contact for the purpose of determining the characteristic intensity.

[0223] The intensity of a contact on the touch sensing surface is optionally characterized in relation to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to the intensity at which the device will perform an operation typically associated with clicking a button or trackpad of a physical mouse. In some embodiments, the deep press intensity threshold corresponds to the intensity at which the device will perform an operation different from an operation typically associated with clicking a button or trackpad of a physical mouse. In some embodiments, if a contact is detected with a characteristic intensity that is below the light press intensity threshold (e.g., and above a weak contact detection intensity threshold below which the contact is no longer detected), the device will move a focus selector in accordance with the movement of the contact on the touch sensing surface without performing an operation associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise stated, these intensity thresholds are consistent across different sets of user interface figures.

[0224] An increase in the characteristic strength of a contact from a strength below a light pressing strength threshold to a strength between the light pressing strength threshold and a deep pressing strength threshold may be referred to as a "light press" input. An increase in the characteristic strength of a contact from a strength below a deep pressing strength threshold to a strength above the deep pressing strength threshold may be referred to as a "deep press" input. An increase in the characteristic strength of a contact from a strength below a contact detection strength threshold to a strength between the contact detection strength threshold and the light pressing strength threshold may be referred to as the detection of a contact on the touch surface. A decrease in the characteristic strength of a contact from a strength above the contact detection strength threshold to a strength below the contact detection strength threshold may be referred to as the detection of a lift-off of the contact from the touch surface. In some embodiments, the contact detection strength threshold is zero. In some embodiments, the contact detection strength threshold is greater than zero.

[0225] In some embodiments described herein, one or more operations are performed in response to the detection of a gesture that includes a corresponding pressing input, or in response to the detection of a corresponding pressing input performed with a corresponding contact (or contacts), and those corresponding pressing inputs are detected based at least in part on the detection of an increase in the strength of a contact (or contacts) that exceeds a pressing input strength threshold. In some embodiments, the corresponding operation is performed in response to the detection of an increase in the strength of a corresponding contact (e.g., the "downstroke" of a corresponding pressing input) that exceeds the pressing input strength threshold. In some embodiments, the pressing input includes an increase in the strength of a corresponding contact that exceeds the pressing input strength threshold and a subsequent decrease in the strength of the contact that is below the pressing input strength threshold, and the corresponding operation is performed in response to the detection of a subsequent decrease in the strength of the corresponding contact (e.g., the "upstroke" of a corresponding pressing input) that is below the pressing input threshold.

[0226] Figures 5E-5H show, from a strength below the light pressing strength threshold of Figure 5E (e.g., "IT L "), to the deep pressing strength threshold of Figure 5H (e.g., "IT DIt shows the detection of a gesture, including a pressing input corresponding to an increase in the intensity of contact 562 that reaches an intensity exceeding 」). While the cursor 576 is displayed on the application icon 572B corresponding to App 2 on the displayed user interface 570, which includes the application icons 572A - 572D displayed within the predefined area 574, a gesture executed by the contact 562 is detected on the touch - sensing surface 560. In some embodiments, this gesture is detected on the touch - sensing display 504. An intensity sensor detects the intensity of the contact on the touch - sensing surface 560. This device determines that the intensity of the contact 562 has reached a peak exceeding a deep - press intensity threshold (e.g., 「IT D 」). The contact 562 is maintained on the touch - sensing surface 560. In response to the detection of the gesture, in accordance with the contact 562 having an intensity exceeding a deep - press intensity threshold (e.g., 「IT D 」) during the gesture, as shown in FIGS. 5F - 5H, reduced representations 578A - 578C (e.g., thumbnails) of the most recently opened documents for App 2 are displayed. In some embodiments, the intensity compared to one or more intensity thresholds is the characteristic intensity of the contact. Note that the intensity diagram of the contact 562 is not part of the displayed user interface but is included in FIGS. 5E - 5H for the reader's reference.

[0227] In some embodiments, the display of the representations 578A - 578C includes an animation. For example, the representation 578A is initially displayed near the application icon 572B as shown in FIG. 5F. As the animation progresses, as shown in FIG. 5G, the representation 578A moves upward and the representation 578B is displayed near the application icon 572B. Then, as shown in FIG. 5H, the representation 578A moves upward, the representation 578B moves upward towards the representation 578A, and the representation 578C is displayed near the application icon 572B. The representations 578A - 578C form an array above the icon 572B. In some embodiments, the animation progresses in accordance with the intensity of the contact 562 as shown in FIGS. 5F - 5G, where the intensity of the contact 562 is at a deep - press intensity threshold (e.g., 「ITD As it increases towards ") ", expressions 578A - 578C appear and move upward. In some embodiments, the intensity based on the progression of this animation is the characteristic intensity of the contact. The operations described with reference to FIGS. 5E - 5H can be performed using an electronic device similar or identical to devices 100, 300, or 500.

[0228] In some embodiments, the device employs intensity hysteresis to avoid spurious inputs sometimes referred to as "jitter", and the device defines or selects a hysteresis intensity threshold having a predetermined relationship to the press - input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press - input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable ratio of the press - input intensity threshold). Thus, in some embodiments, a press - input includes an increase in the intensity of the corresponding contact above the press - input intensity threshold and a subsequent decrease in the intensity of the contact below the hysteresis intensity threshold corresponding to the press - input intensity threshold, and the corresponding operation is executed in response to the detection of a subsequent decrease in the intensity of the corresponding contact (e.g., the "up - stroke" of the corresponding press - input) below that hysteresis intensity threshold. Similarly, in some embodiments, a press - input is detected only when the device detects an increase in the intensity of the contact from an intensity below the hysteresis intensity threshold to an intensity above the press - input intensity threshold and optionally a subsequent decrease in the intensity of the contact to an intensity below the hysteresis intensity, and the corresponding operation is executed in response to the detection of that press - input (e.g., depending on the situation, an increase in the intensity of the contact or a decrease in the intensity of the contact).

[0229] For ease of explanation, the description of operations performed in response to a pressing input associated with a pressing input intensity threshold, or in response to a gesture including the pressing input, is based on an increase in the intensity of contact exceeding the pressing input intensity threshold, an increase in the intensity of contact from an intensity below the hysteresis intensity threshold to an intensity exceeding the pressing input intensity threshold, a decrease in the intensity of contact below the pressing input intensity threshold, and / or a decrease in the intensity of contact below the hysteresis intensity threshold corresponding to the pressing input intensity threshold. Further, in an embodiment described as performing an operation in response to detecting a decrease in the intensity of contact below the pressing input intensity threshold, the operation is optionally performed in response to detecting a decrease in the intensity of contact below a hysteresis intensity threshold that corresponds to and is lower than the pressing input intensity threshold.

[0230] Next, attention is directed to embodiments of a user interface ("UI") and related processes implemented on an electronic device such as the portable multifunctional device 100, device 300, or device 500.

[0231] Figures 6A - 6S illustrate exemplary user interfaces for managing a controllable external device according to some embodiments. The user interfaces in these figures are used to illustrate processes described below, including the processes in Figures 7A - 7C and Figure 23.

[0232] Figure 6A shows an electronic device 600 having a touch - sensitive display 602. In some embodiments, device 600 includes some or all of the features of devices 100, 300, and 500 described above. Device 600 is configured to receive data associated with a controllable external device. Device 600 is also configured to send commands to a controllable external device.

[0233] In FIG. 6A, device 600 displays a user interface screen 610-1 for managing controllable external devices associated with a particular location. In some embodiments, the user interface screen 610 is included in an application for managing controllable external devices associated with one or more locations. Examples of locations include, but are not limited to, an address, a place, a building, a company, an environment (e.g., a backyard, a vehicle, a porch), etc. In some embodiments, a location includes one or more designated areas within or associated with a location, such as a room within a location. In some embodiments, a location includes an environmental model having a hierarchical representation of a physical environment (e.g., a home) that can include a lowest level of an object (e.g., a room) as described in paragraph

[0017] of U.S. Patent Application No. 14 / 725,912, filed May 29, 2015, entitled "Accessory management Systems Using Environment Model", which is incorporated by reference above.

[0234] Device 600 is configured to access a location profile for the location shown on user interface screen 610-1. In some embodiments, the location profile is accessed from local memory on device 600 or from a remote server. The location profile includes information associated with the location, and the information can include, for example, a name, an address, people associated with the location, an image associated with the location (e.g., a wallpaper image), a year (e.g., when the location was built), a memo, accessories associated with the location, the current state of the accessories of the location, an automation profile associated with the location, and a scene profile associated with the location. The terms accessories, automation profile, and scene profile are described in more detail below.

[0235] In some embodiments, the user selects a location profile by user input (e.g., by selecting a location from a list of locations). In some embodiments, device 600 determines its location and determines a location profile based on the location of device 600. Thus, device 600 automatically presents information to the user and controls with respect to a location corresponding to the user's current location.

[0236] User interface screen 610-1 is a home screen corresponding to tab 604-1 and can be referred to as "home screen 610-1". Home screen 610-1 includes a location indication 606 ("Ellsworth St."). In the illustrated embodiment, indication 606 includes text. In some embodiments, the location indication includes an image or other identifying information.

[0237] The home screen 610-1 also includes a representation 620-1 of a location or a controllable external device associated with the location. A controllable external device is also referred to as an "accessory" for the purposes of the present disclosure and refers to any item or device configured to be monitored and / or controlled by another device. In some embodiments, the accessory is configured to operate in two or more states and is remotely controllable between the two or more states. Non-limiting examples of accessories include lighting fixtures, thermostats, outlets, ventilation fans, appliances (e.g., televisions, washing machines, dryers, dishwashers, coffee machines), doors (e.g., door locks), automatic door openers (e.g., garage door openers), air conditioners, furnaces, still cameras or video cameras, smoke alarms, carbon monoxide alarms, security systems, and alarms. An accessory can be a dedicated computing device or a general-purpose computing device such as a desktop computer, laptop computer, smartphone, other mobile phone, other handheld or wearable computing device, and can include an electronic device that can be controlled by providing appropriate executable program code or instructions as described in paragraph

[0008] of U.S. Patent Application No. 14 / 725,912, filed May 29, 2015, entitled "Accessory management Systems Using Environment Model", incorporated herein by reference.

[0238] Representation 620-1 corresponds to a thermostat accessory and includes an indication of the current state of the thermostat. Specifically, Representation 620-1 indicates that the current temperature is 70 degrees and that the thermostat is heating to a set temperature of 72 degrees. In some embodiments, the accessory representation includes an indication of the action that the selection of the representation will cause. For example, Representation 620-2 corresponding to a ventilation fan accessory includes the text "Turn on". This indicates that selecting the representation will turn on the ventilation fan and also means that the ventilation fan is currently off.

[0239] Representation 620-1 is placed on the home screen 610-1 under the title "Favorite Accessories". In some embodiments, the user can manually specify an accessory as a favorite accessory. In some embodiments, an accessory is represented as a favorite accessory based on its operation, such as how frequently it is controlled or triggered. In some embodiments, an accessory is represented as a favorite accessory based on its operation with respect to a particular device or user profile, such as how frequently it is controlled by a particular device or a user associated with a particular device.

[0240] In the illustrated embodiment, Representation 620-1 includes an affordance. In FIG. 6A, device 600 detects a user input 630-1 (e.g., a finger tap) on Representation 920-1 corresponding to the selection of the thermostat accessory. At the time of user input 630-1, the thermostat accessory is in a first state heading towards 72 degrees. In some embodiments, device 600 sends a command to set the thermostat accessory to a second state in response to user input 630-1.

[0241] In some embodiments, device 600 determines the duration of user input 630-1 (e.g., the duration of contact on touch-sensitive display 602). In some embodiments, device 600 determines the characteristic intensity of the user input (e.g., the characteristic intensity of contact on touch-sensitive display 602). Device 600 determines whether user input 630-1 meets the input criteria. In some embodiments, the input criteria includes whether the duration of the user input exceeds a predetermined time and / or whether the characteristic intensity of the user input exceeds a predetermined intensity.

[0242] In accordance with the determination that user input 630-1 does not meet the input criteria, device 600 sends an instruction to set the state of the thermostat accessory to a second state. In some embodiments, the representation of the accessory includes an indication of the state in which the accessory will be set in response to a user input that does not meet the input criteria. For example, representation 620-2 in FIG. 6A indicates that by selecting representation 620-2, the ventilation fan accessory can be turned on.

[0243] FIG. 6C shows an exemplary response when user input 630-1 does not meet the input criteria. As shown in FIG. 6C, device 600 sends an instruction to turn off the thermostat and updates representation 620-1 to indicate the off state. In some embodiments, device 600 outputs a haptic to indicate that the state has changed.

[0244] Optionally, in accordance with a determination that the user input 630-1 does not meet the input criteria, the device 600 transmits a command to switch the state of the accessory from a first of two predetermined states to a second of the two predetermined states (e.g., on / off, open / closed, locked state / unlocked state). In one embodiment of this technique, in response to another user input that is the same as the user input 630-1, the device 600 transmits a command to return the thermostat to its previous state and set it to on (e.g., heat, set to 72 degrees). In this way, the user can provide a relatively short or light tap to quickly toggle the state of the accessory between two predetermined states. In some embodiments, in accordance with a determination that the user input does not meet the input criteria, the device 600 transmits a command to switch the state to an adjacent state within a predetermined order of states (e.g., off -> low -> high -> off). In this way, the user can easily cycle through a series of states.

[0245] Alternatively, in accordance with a determination that the user input 630-1 meets the input criteria, the device 600 displays a control affordance indicating that the accessory is available for use. FIG. 6D shows an exemplary user interface screen 610-2 having a control affordance 608 indicating the current state of the accessory (set to 72 degrees) and the available state of the accessory. The user interface screen 610-2 also includes an affordance 622 that, when selected, displays information about the corresponding accessory and optionally displays an affordance for changing information associated with the accessory. Information associated with the accessory optionally includes a name, room, type, manufacturer, and representative icon. Optionally, the user interface screen 610-2 is partially translucent so that the home screen 610-1 is partially visible.

[0246] Regarding the control affordance 698, the horizontal bar on the control affordance 608 represents the temperature at which, for example, the thermostat can be set in 1-degree increments. In response to detecting a user input on the control affordance 608 corresponding to the selection of a particular state, the device 600 sends a command to set the state of the accessory to the selected state. In this way, the user can interact with the control affordance 608 to select a state that is not available in response to a user input that does not meet the input criteria, or to directly go to a state that is not adjacent to the current state. In FIG. 6D, the device 600 detects a user input 630-2 (e.g., a tap) and, in response, sets the thermostat to 75 degrees. Optionally, the thermostat can be changed to 75 degrees by a vertical swipe from bar 614 to bar 616 on the control affordance 608.

[0247] Optionally, in response to detecting a user input 631-2 on the control affordance 608 corresponding to the selection of a particular state, the device updates the visual appearance of the control affordance 608 to indicate, for example, that the accessory has been set to the selected state as shown in FIG. 6E. Optionally, the device 600 stops displaying the control affordance 608 in response to the selection of a state (e.g., the device 600 returns to the user interface screen 610-1). In some embodiments, the device 600 stops displaying the control affordance 608 in response to detecting a user input corresponding to the selection of a position (e.g., area 612) on the touch-sensitive display 602 that does not correspond to the control affordance 608.

[0248] In some embodiments, the device 600 updates the appearance of the control affordance 608 to indicate the selected state, but does not send a command to set the state of the accessory until further user input is detected. In some embodiments, the device 600 sends a command to set the accessory to the selected state in response to the selection of another affordance (e.g., a "Settings" or "Done" affordance), or an input on the touch-sensitive display 602 that is located outside of the response area (e.g., area 612) of the control affordance 608.

[0249] The control affordance 608 is an example of a control affordance for an accessory that operates in discrete states (e.g., a digital thermostat that can be adjusted in one-degree increments). FIGS. 6F-6I illustrate other examples of control affordances for accessories that operate in discrete states. More specifically, FIG. 6F shows an exemplary control affordance for a ventilation fan accessory that can operate in four discrete states. FIG. 6G shows an exemplary control affordance for a home security system that can operate in various discrete alarm modes, and FIGS. 6H-6I show exemplary control affordances for a door or other lockable accessory that has two discrete states. In some embodiments, the device 600 displays the time at which the current state of the accessory was set, as shown in the example of FIG. 6H (e.g., unlocked at 5:00 PM).

[0250] In some embodiments, the available states of the accessory are continuous. FIG. 6J shows an exemplary control affordance 621 for a dimmable lighting switch that can operate over a continuous range of brightness. In some embodiments, a particular state is selected by either tapping on the corresponding position on the affordance 621 (as shown by element 630-3), or by vertically dragging the upper portion of bar 623 until the desired state is obtained (as shown by arrow 630-4).

[0251] In some embodiments, the accessory is a color - adjustable light bulb configured to operate in different color states. The left side of FIG. 6K shows a control affordance 625 for a color - adjustable light bulb similar to the control affordance 621 of FIG. 6J. However, an affordance 627 is also shown. Selection of the affordance 627 provides a control affordance 629 for selecting the color in which the color - adjustable light bulb operates, as shown on the right side of FIG. 6K. A marker 618 indicates the current color of the light bulb. In some embodiments, different colors are selected by touching a position on the control affordance 629.

[0252] In some embodiments, there is a relationship between two or more accessories such that when one accessory is controlled or operated, another related accessory is controlled. In some embodiments, in response to the device 600 sending a command to change the state of a first accessory, in addition to the first accessory, a related second accessory is automatically controlled. For example, a command to operate a camera on the front porch of a location automatically turns on a microphone and / or speaker on the porch. In some embodiments, in response to the first accessory being activated, a related second accessory is automatically controlled. For example, when a visitor rings the doorbell at the front door, the front door lock, the porch camera, and the intercom system associated with the front porch automatically operate (e.g., are set to a specified state).

[0253] In some embodiments, when an accessory is activated, device 600 displays a notification indicating that the accessory has been activated. In some embodiments, the notification includes information about the accessory related to the activated accessory and / or affordances for controlling the accessory related to the activated accessory. In one example, when a doorbell rings, device 600 displays a notification that includes an image captured by a camera on the front porch and affordances for activating the intercom system at the front door, unlocking the front door, and turning on the porch lighting. Exemplary embodiments of the notification corresponding to this example are described in more detail below with reference to FIGS. 34I - 34K.

[0254] In some embodiments, the relationships between accessories are specified by the user (e.g., by user input) to explicitly tie together the functions of two or more devices such that when one is triggered either manually or by an automatic trigger (e.g., time of day, sensor, etc.), each is triggered and functions accordingly. In some embodiments, the relationships are automatically established based on the location of the accessories within a location (e.g., for certain accessories, the relationships are automatically established because they are associated with a common area or room of the location). Similarly, devices associated based on location function together when one is triggered either manually or by an automatic trigger (e.g., time of day, sensor, etc.).

[0255] In some embodiments, the relationships between accessories are established based on the type of accessory. In one example, a television, a soundbar, an Apple TV, and a home theater amplifier are automatically related because they are associated with an entertainment system. In this example, the selection of the affordance to turn off the television optionally causes the other accessories (the soundbar, the Apple TV, and the home theater amplifier) to be powered off. In some embodiments, the device 600 displays an affordance representing a group of accessories of the same type that can be automatically generated by the system. Referring again to the example of the entertainment system, the device 600 optionally displays an affordance labeled "Entertainment System" that controls all of the accessories of the system listed above.

[0256] Referring back to FIG. 6A, the home screen 610-1 also includes a representation 624 of an individual associated with the location (e.g., an individual contact, a friend, a family member). In some embodiments, the device 600 receives data indicating that a remote electronic device (e.g., a smartphone, a tablet) associated with the individual is present at the location. In some embodiments, the representation 624 is displayed on the home screen 610-1 in response to receiving data indicating that the remote electronic device is present at the location. In the illustrated embodiment, the representation 624 includes an image of the individual. In some embodiments, the representation includes the name of the individual, initials, a designated image associated with the individual, or other text, image, video, etc. associated with the individual. In some embodiments, an individual associated with a device that is not present at the location or has an unknown location (e.g., remote or inactive) is not represented on the home screen 610-1. In some embodiments, the home screen 610-1 includes one or more representations of individuals determined to not be present at the location.

[0257] In some embodiments, the home screen 610-1 includes an indication of an individual's status or location. In some embodiments, the status or location is with respect to a location. For example, in some embodiments, the home screen 610-1 includes an indication of whether an individual is determined to be present at or away from a location, or whether the individual's status is unknown or inactive. In some embodiments, the away or unknown status is indicated by shading or graying out the individual's representation. In some embodiments, the individual's status is determined based on the location of a device associated with the individual. The status optionally also indicates whether the individual has been invited to be associated with the location for the purpose of accessing information regarding the location.

[0258] The home screen 610-1 also includes an indication 626 of the status of a location. In some embodiments, the status includes an overall assessment of the location (e.g., "all safe and sound"). In the embodiment shown in FIG. 6A, the indication 626 includes the status of accessories associated with the location (e.g., the garage door is open, the 70-degree thermostat is currently heating to 72 degrees, the kitchen exhaust fan is on). In some embodiments, the status of the accessories is classified (e.g., critical, attention, or not important), and based on the danger status, more dangerous statuses are displayed over less dangerous statuses. Examples of critical statuses can include activation of a security alarm, a smoke detector, a carbon monoxide detector, or an indication that a hub providing the status of the accessories is not operating. Examples of attention statuses can include an open or unlocked door or window, or an accessory not being connected or not being available for use. Examples of not important statuses can include that the lighting is on, and the status of the thermostat.

[0259] The home screen 610-1 also includes a representation 640-1 of a scene profile. The scene profile includes data regarding the specified states of one or more accessories of a location. The scene profile may also simply be referred to as a "scene". In FIG. 6A, the scene profile corresponding to the representation 640-1 is designated as the "favorite scene". In some embodiments, the scene profile is designated based on one or more of the following: (1) user input that explicitly designates the scene profile, (2) the frequency with which the scene profile is applied, (3) how recently the scene profile has been applied, and (4) how frequently or recently the accessories associated with the scene profile have been selected, controlled, or triggered. Similar to the specified accessories described above, in some embodiments, the scene profile is designated based on actions taken by a particular device or a user associated with the device (e.g., which scene profiles are frequently selected by the user).

[0260] In the embodiment of FIG. 6A, the representation 640-1 includes affordances. In some embodiments, in response to detecting a user input corresponding to the selection of the representation 640-1, the device 600 transmits an instruction to apply the specified states of the accessories included in the corresponding scene profile. In some embodiments, in response to detecting a user input corresponding to the selection of the representation 640-1 of the scene profile, the device 600 displays a user interface screen 610-3 shown in FIG. 6L that includes representations of the accessories associated with the scene profile. The representations of the plurality of accessories within the user interface screen 610-3 include an indication of the specified state of each accessory (e.g., the state in which the accessory is set when the scene profile is implemented). In some embodiments, the indication of the specified state includes an action (e.g., turn off) that would be taken when the scene profile is implemented.

[0261] The home screen 610-1 also includes affordance 628. In response to detecting user input on affordance 628, the device 600 displays, for example, an option 630 to create an additional scene profile as shown in FIG. 6M. In some embodiments, the device 600 displays an option to create an additional scene profile in response to other types of user input.

[0262] In some embodiments, the home screen 610-1 includes an image captured by a controllable external camera of the location. In the embodiment shown in FIG. 6N, the home screen 610-1 is scrollable and is scrolled by known techniques to show an area 632 for displaying an image included in a live video feed from a controllable external camera within the kitchen of the location. In some embodiments, the image is from a single point in time. In some embodiments, the image is a live video feed. In some embodiments, the image is a single image that can be selected (e.g., tapped) to display a live video feed from a controllable external camera. In some embodiments, one or more affordances or camera images can be off-screen and can be displayed by input such as a scrolling gesture such as a button or a swipe. In some embodiments, by displaying one or more affordances or camera images that were initially off-screen, one or more affordances are displaced so that they are no longer displayed.

[0263] Device 600 also displays room tab 604-2. In response to detecting a user input (e.g., a tap) on room tab 604-2, device 600 displays a representation of at least one room associated with the location, as shown, for example, on user interface screen 610-4 of FIG. 6O. In the embodiment of FIG. 6O, the representation of the room includes an image associated with the corresponding room. In response to detecting a user input to select a room on user interface 610-4, device 600 displays a representation of an accessory associated with the selected room and, optionally, a representation of a scene profile that includes the accessory associated with the selected room. FIG. 6P shows an exemplary user interface screen 610-5 corresponding to a bedroom having a representation of accessories and a scene profile associated with the bedroom. In some embodiments, the representation of the accessory indicates the current state of the accessory. In some embodiments, the image associated with the room is displayed as a wallpaper on user interface screen 610-5 behind the representation of the accessory. In some embodiments, the representation of the accessory associated with the selected room is an affordance to control the accessory or access a user interface screen for controlling the corresponding accessory, as described above with reference to representations 620-1 and FIGS. 6B-6K.

[0264] The user interface screen 610-1 also includes an automation tab 604-3. In response to detecting user input on the automation tab 604-3, the device 600 displays an automation profile and / or a representation of an option to create an automation profile. An automation profile includes criteria for controlling the state of at least one accessory of a location and data representing a specified state for the at least one accessory. In other words, an automation profile includes the state to which a selected accessory is set when the criteria of the automation profile are met. An “automation profile” may also simply be referred to as “automation”. Setting an accessory associated with an automation profile to a specified state in response to determining that the criteria of the automation profile are met may sometimes be referred to as implementing the automation profile. An automation profile can include data regarding a scene profile, which also includes data regarding the state of an accessory.

[0265] FIG. 6Q shows an exemplary user interface screen 610-6 that is displayed in response to detecting user input on the automation tab 604-3. The user interface screen 610-6 includes a representation 634 of an automation profile named "When I Leave Home" that is associated with a location. The representation 634 also includes indications of scenes and / or accessories associated with the automation profile, "Execute Scene 1" and "Control 3 Accessories". In some embodiments, the name of the automation profile indicates the criteria of the automation profile. In some embodiments, the representation of the automation profile includes an affordance that, when selected, displays a representation of the accessories and / or scene profiles associated with the automation profile. In some embodiments, the representation of the accessories associated with the automation profile includes an affordance that, when selected, provides a user interface for changing the specified state of the accessories for the automation profile or for selecting the specified state of the accessories for the automation profile. In some embodiments, the specified state of the accessories associated with the automation profile is selected by applying the techniques described with respect to FIGS. 6B-6K.

[0266] Device 600 also provides a function to view information associated with different locations and / or create location profiles for additional locations. In the embodiment shown in FIG. 6R, device 600 detects user input on affordance 636 included on home screen 610-1. In response to detecting the selection of affordance 636, device 600 displays a menu 638 that provides an option to view information associated with different locations (e.g., Lisa's house or beach house) and / or add additional locations (e.g., add a location). Device 600 also provides an option (e.g., location details...) to view and / or modify profile information associated with a location, such as the location profile information described above.

[0267] Device 600 also provides a function to add accessories associated with a location. Referring again to FIG. 6M, device 600 detects user input on affordance 628 included on home screen 610-1, and in response, displays option 642 to associate an accessory with a location.

[0268] In some embodiments, device 600 displays people tab 604-4 as shown in FIG. 6S. In response to detecting user input on people tab 604-4, device 600 displays representations of people associated with a location. In the illustrated embodiment, representations of people 644-1, 644-2, and 644-3 are displayed on user interface screen 610-7. In the illustrated embodiment, the representations of people include images of people. In some embodiments, the representation includes the name of an individual, initials, a designated image associated with the individual, or other text, image, video, etc. associated with the individual. In some embodiments, user interface screen 610-7 includes representations of all people associated with a location. In some embodiments, people having a status indicating that they are not present at the location, or that their location is unknown (e.g., away, inactive, etc.) are not displayed (e.g., only representations of people determined to be present at the location are displayed). In some embodiments, one or more representations of an individual determined to not be present at the location are displayed.

[0269] In FIG. 6S, the user interface screen 610-7 includes an indication of people's status or location. In some embodiments, the status or location is with respect to a location. For example, in some embodiments, an indication is displayed as to whether an individual is determined to be present at or away from a location, or whether the individual's status is unknown or inactive. In some embodiments, an individual's status is determined based on the location of a device associated with the individual. In some embodiments, speaker identification using one or more microphones can be used to determine an individual's presence and identification information. The status also indicates whether an individual has been invited to be associated with a location for the purpose of accessing information regarding the location (e.g., invited).

[0270] FIGS. 7A-7C show flow diagrams illustrating a method for managing an external device controllable using an electronic device, according to some embodiments. Method 700 is executed on a device (e.g., 100, 300, 500, or 600) having a display and optionally a touch-sensitive surface. Some of the operations in method 700 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0271] As described below, method 700 provides an intuitive method for managing a controllable external device. This method reduces the cognitive burden on the user for managing a controllable external device, thereby creating a more efficient human-machine interface. In the case of a battery-operated computing device, power is conserved and the battery charging interval is increased by enabling the user to manage a controllable external device faster and more efficiently.

[0272] In block 702, the device displays a user interface screen (e.g., 610-1) that includes an indication of a location and a representation of a controllable external device at the location (e.g., 620-1).

[0273] Optionally, at block 704, the device receives data indicating that a remote electronic device associated with an individual is present at the location, and in response to receiving the data indicating that the remote electronic device is present at the location, displays a representation of the individual (e.g., 624) associated with the remote electronic device on the user interface screen.

[0274] Optionally, at block 706, the device displays an indication of the status of the location (e.g., 626) on the user interface screen.

[0275] Optionally, at block 708, the device displays a representation of a scene profile (e.g., 640-1) including data regarding the specified status of a plurality of controllable external devices at the location on the user interface screen, detects a first user input corresponding to the selection of the scene profile, and in response to detecting the first user input, displays a representation of the plurality of controllable external devices associated with the scene profile (e.g., 910-2).

[0276] Optionally, at block 710, the device displays an image (e.g., 632) captured by a controllable external camera at the location on the user interface screen.

[0277] At block 712, the device detects a second user input (e.g., 630-1) corresponding to the selection of a controllable external device.

[0278] At block 714, in response to detecting the second user input, the device displays a user interface object (e.g., 608) indicating a plurality of possible states of the controllable external device.

[0279] While a user interface object indicating a plurality of possible states of a controllable external device is being displayed at block 716, the device detects a third user input (e.g., 630-2) corresponding to the selection of a specified state of the controllable external device.

[0280] At block 718, in response to detecting the third user input, the device sends a command to set the controllable external device to the specified state.

[0281] Optionally, at block 720, while the device is displaying the user interface screen, the device detects a fourth user input, and in response to detecting the fourth user input, displays a representation of at least one specified room associated with the location (e.g., FIG. 6O), detects a fifth user input corresponding to the selection of a first specified room of the at least one specified room, and in response to detecting the fifth user input, displays a representation of at least one controllable external device associated with the first specified room (e.g., FIG. 6P).

[0282] Optionally, at block 722, while the device is displaying the user interface screen, the device detects a sixth user input, and in response to detecting the sixth user input, displays a representation of an automation profile including criteria for controlling the state of at least one controllable external device of the location and data representing the specified state for the at least one controllable external device, and / or an option to create an automation profile (e.g., FIG. 6Q).

[0283] Optionally, at block 724, while the device is displaying the user interface screen, the device detects a seventh user input, and in response to detecting the seventh user input, displays a representation of people associated with the location (e.g., FIG. 6S).

[0284] Optionally, in block 726, while the device is displaying a user interface screen, the device detects an eighth user input and, in response to detecting the eighth user input, creates an additional scene profile, views and / or adds information associated with a different location, modifies profile information for a location, and associates a second controllable external device with a location, and displays an option to perform one or more of the foregoing (e.g., FIGS. 6M(630) and 6R(638)).

[0285] Note that the details of the processes described above with respect to method 700 (e.g., FIGS. 7A-7C) are also applicable in a similar manner to the methods described hereinafter. For example, methods 1000, 1300, 1600, 1900, 2100, 2300, 2600, 2900, and 3200 optionally include one or more of the various method features described above with reference to method 700.

[0286] According to some embodiments, FIG. 8 shows an exemplary functional block diagram of an electronic device 800 configured in accordance with the principles of the various embodiments described. According to some embodiments, the functional blocks of the electronic device 800 are configured to perform the techniques described above. The functional blocks of device 800 are optionally implemented by hardware, software, or a combination of hardware and software to perform the principles of the various examples described. It will be understood by those skilled in the art that the functional blocks described in FIG. 8 are optionally combined or separated into sub-blocks to perform the principles of the various examples described. Accordingly, the description herein optionally supports any possible combination or separation of, or further definition of, the functional blocks described herein.

[0287] As shown in FIG. 8, the electronic device 800 includes a display unit 802 configured to display a graphic user interface, an optionally touch-sensing surface unit 804 configured to receive contacts, and a processing unit 806 coupled to the display unit 802 and optionally the touch-sensing surface unit 804. In some embodiments, the processing unit 806 includes a display enabling unit 808, a detection unit 810, a causing unit 812, and a receiving unit 814.

[0288] The processing unit 806 enables the display of an indication of a location (e.g., using the display enabling unit 808) on a first user interface screen, enables the display of a representation of a controllable external device (e.g., using the display enabling unit 808) on the first user interface screen, detects a first user input corresponding to the selection of the controllable external device (e.g., using the detection unit 810), enables the display of a user interface object indicating a plurality of possible states of the controllable external device (e.g., using the display enabling unit 808) in response to detecting the first user input, detects a second user input corresponding to the selection of a specified state of the controllable external device (e.g., using the detection unit 810) while the user interface object indicating the plurality of possible states of the controllable external device is being displayed, and causes the transmission of an instruction to set the controllable external device to the specified state (e.g., using the causing unit 812) in response to detecting the second user input.

[0289] In some embodiments, the processing unit 806 receives data indicating that a remote electronic device associated with an individual is present at a location (e.g., using the receiving unit 814), and in response to receiving the data indicating that the remote electronic device is present at the location, further configured to enable the display of a representation of the individual associated with the remote electronic device on a first user interface screen (e.g., using the display enabling unit 808).

[0290] In some embodiments, the processing unit 806 is further configured to enable the display of an indication of the status of the location on a first user interface screen (e.g., using the display enabling unit 810).

[0291] In some embodiments, the processing unit 806 enables the display of a representation of a scene profile including data regarding the specified status of a plurality of controllable external devices at the location on a first user interface screen (e.g., using the display enabling unit 808), detects a third user input corresponding to the selection of the scene profile (e.g., using the detection unit 810), and in response to detecting the third user input, further configured to enable the display of a representation of the plurality of controllable external devices associated with the scene profile (e.g., using the display enabling unit 808). In some embodiments, the processing unit 806 detects a fourth user input (e.g., using the detection unit 810) while the first user interface screen is being displayed, and in response to detecting the fourth user input, further configured to enable the display of an option to create an additional scene profile (e.g., using the display enabling unit 808).

[0292] In some embodiments, the processing unit 806 is further configured to enable the display of an image included in a live video feed from a controllable external camera captured by the controllable external camera of the location on a first user interface screen (e.g., using the display enabling unit 808).

[0293] In some examples, the processing unit 806 detects a fifth user input (e.g., using the detection unit 810) while the first user interface screen is being displayed, and in response to detecting the fifth user input, enables the display of a representation of at least one designated room associated with the location (e.g., using the display enabling unit 808), detects a sixth user input corresponding to the selection of a first designated room of the at least one designated room (e.g., using the detection unit 810), and in response to detecting the sixth user input, enables the display of a representation of at least one controllable external device associated with the first designated room (e.g., using the display enabling unit 808).

[0294] In some embodiments, the processing unit 806 detects a seventh user input (e.g., using the detection unit 810) while the first user interface screen is being displayed, and in response to detecting the seventh user input, enables the display of a representation of an automation profile including criteria for controlling the state of at least one controllable external device of the location and data representing a designated state for the at least one controllable external device, and / or an option to create an automation profile (e.g., using the display enabling unit 808).

[0295] In some embodiments, the processing unit 806 detects an eighth user input (e.g., using the detection unit 810) while the first user interface screen is being displayed, and in response to detecting the eighth user input, further configured to enable the display of options to view information associated with different locations and / or add additional locations (e.g., using the display enabling unit 808).

[0296] In some embodiments, the processing unit 806 detects a ninth user input (e.g., using the detection unit 810) while the first user interface screen is being displayed, and in response to detecting the ninth user input, further configured to enable the display of options to change the profile information of the location (e.g., using the display enabling unit 808).

[0297] In some embodiments, the processing unit 806 detects a tenth user input (e.g., using the detection unit 810) while the first user interface screen is being displayed, and in response to detecting the tenth user input, further configured to enable the display of options to associate a second controllable external device with the location (e.g., using the display enabling unit 808).

[0298] In some embodiments, the processing unit 806 detects an eleventh user input (e.g., using the detection unit 810) while the first user interface screen is being displayed, and in response to detecting the eleventh user input, further configured to enable the display of representations of people associated with the location (e.g., using the display enabling unit 808).

[0299] The operations described above with reference to FIGS. 7A-7C are optionally implemented by the components shown in FIGS. 1A-1B or FIG. 8. For example, the display operations 702, 704, 706, 708, 710, 714, 720, 722, 724, and 726, the detection operations 708, 712, 716, 720, 722, 724, and 726, the transmission operation 718, and the reception operation 704 are optionally executed by the event sorter 170, the event recognizer 180, and the event processor 190. The event monitor 171 of the event sorter 170 detects a contact on the touch-sensitive display 112, and the event dispatcher module 174 distributes the event information to the application 136-1. Each event recognizer 180 of the application 136-1 compares the event information with its respective event definition 186 to determine whether a first contact at a first location on the touch-sensitive surface corresponds to a predefined event or sub-event, such as the activation of an affordance on the user interface. When a corresponding predefined event or sub-event is detected, the event recognizer 180 activates the event processor 190 associated with the detection of that event or sub-event. The event processor 190 optionally uses or invokes the data updater 176 or the object updater 177 to update the application internal state 192. In some embodiments, the event processor 190 accesses the corresponding GUI updater 178 to update what is displayed by the application. Similarly, it will be apparent to those skilled in the art how other processes can be implemented based on the components shown in FIGS. 1A-1B.

[0300] FIGS. 9A-9F show exemplary user interfaces for creating and / or configuring a scene profile for a controllable external device, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the process in FIG. 10.

[0301] FIG. 9A shows a user interface screen 910-1 for creating a new scene profile, which is displayed on device 600. In some embodiments, user interface screen 910-1 is displayed in response to selection of the "New Scene" option shown in FIG. 6M. User interface screen 910-1 prompts the user to create a new scene profile and / or to provide one or more default (e.g., proposed) scene profiles. The default scene profiles include data regarding a predetermined state of a predetermined controllable electronic device at a location. User interface screen 910-1 includes an affordance 904 for creating a new custom scene profile, as well as representations 902-1, 902-2, 902-3, and 902-4 of exemplary default scene profiles named "Good Morning", "Leaving Home", "Arriving Home", and "Good Night".

[0302] FIG. 9B shows a device 600 displaying a user interface screen 910-2 corresponding to a new custom scene profile named Movie Night. In addition to the name of the scene profile, the user interface screen 910-2 also includes an image 905 (e.g., glyph, symbol, photograph) associated with the scene profile. In some embodiments, the user can manually select an image associated with the scene profile. The user interface screen 910-2 also includes an affordance 906 labeled "Add Accessory" for adding an accessory to the scene profile. In response to a user input selecting the affordance 906, the device 600 displays a user interface screen 910-3 (FIG. 9C) that includes representations 908-1 to 908-5 of the accessories located at the location. The representations 908-1 to 908-5 are arranged based on the corresponding rooms to which they are associated. The representations include an indication of whether the corresponding accessory is associated with the scene profile. In the illustrated embodiment, the circle in the upper right corner of each of the representations 908-1 to 908-5 is checked or filled if the corresponding accessory is associated with the scene profile. Otherwise, the circle is blank. Other examples of an indication that an accessory is included in the scene profile include highlighting, making it bold, etc.

[0303] In the embodiment of FIG. 9C, the representations 908-1 to 908-5 are affordances. The device 600 detects a user input 920-1 corresponding to the selection of the first accessory (e.g., a tap on the representation 908-1). After detecting the user input 920-1, the device 600 associates the first accessory with the scene profile. FIG. 9D shows an updated version of the user interface screen 910-3 indicating that the accessory corresponding to the representation 908-1 is associated with the scene profile. The device 600 associates the first accessory with the scene profile by including in the scene profile data identifying the first accessory and the first state of the first accessory. If the scene profile already exists, including the data in the scene profile includes updating the existing profile (e.g., adding the data to the existing scene profile). In some embodiments, if the scene profile does not yet exist when an accessory is selected, including the data in the scene profile includes creating a new scene profile with the data.

[0304] In response to detecting the selection of the affordance 914, the device 600 causes all the accessories associated with the bedroom to be included in the scene profile. The affordance 916 provides a similar function for the bathroom. In some embodiments, the user interface screen 910-3 includes an affordance that, when selected, causes all the accessories on the user interface screen to be included in the scene profile.

[0305] In some embodiments, the representations 908-1 to 908-5 on the user interface screen 910-3 include an indication of the current state of the corresponding accessory. In some embodiments, when an accessory is selected, the device 600 includes the current state of the accessory in the scene profile. In some embodiments, if the accessory is already associated with the scene profile, the representation on the user interface screen 910-3 corresponding to the accessory indicates the state of the accessory included in the scene profile, which is not necessarily the same as the current state of the accessory. In some embodiments, the representations 908-1 to 908-5 include an indication of a state that will be included in the scene profile when the accessory is selected, which can be, for example, the current state of the accessory or a state specified by the user.

[0306] In some embodiments, the device 600 detects a second user input 920-2 (e.g., contact with a position on the touch-sensitive display 602 corresponding to the representation 908-5) corresponding to the selection of a second accessory. The device 600 determines whether the second user input 920-2 meets a threshold criterion. In some embodiments, the threshold criterion includes, for example, a threshold duration of contact or a threshold characteristic intensity of the contact.

[0307] In response to detecting the second user input 920-2 and according to the determination that the threshold criterion is met, the device 600 displays an indication of a first state of the second accessory. In some embodiments, the indication of the first state is displayed as part of a control affordance such as those shown in FIGS. 6C to 6K. In the embodiment shown in FIG. 9C, the accessory corresponding to the selected representation 908-5 is lighting. In some embodiments, the control affordance shown in FIG. 6J is displayed such that the user can select the state of the lighting.

[0308] Device 600 detects an input corresponding to the selection of a first state (e.g., a tap on the control affordance of FIG. 6J). After detecting the selection of the first state, device 600 includes data in the scene profile that identifies a second accessory and a first state of the second accessory. In some embodiments, after detecting the selection of the first state, device 600 displays a representation 908-5 of a second accessory having an indication that the first state (e.g., off) and that the accessory has been added to the scene profile, as shown in FIG. 9D. In some embodiments, when device 600 determines that a threshold criterion is met, it includes data in the scene profile that identifies the first state in direct response to user input 920-2. In some embodiments, device 600 indicates that the specified state of the accessory has been selected for the scene, for example, by outputting a haptic.

[0309] FIG. 9E shows an updated version of user interface screen 910-2 on device 600 having an overview of the scene profile. Device 600 displays user interface screen 910-2 in response to detecting a user input, e.g., the selection of an affordance labeled "Done" on user interface screen 910-3. User interface screen 910-2 includes representations 908-6 and 908-7 of accessories associated with the scene profile.

[0310] The updated user interface screen 910-2 also includes options (e.g., affordances) to add or remove data associated with accessories from a scene profile (e.g., 912-1), test a scene profile (e.g., 912-2), specify a scene profile (e.g., 912-3), and delete a scene profile (e.g., 912-4). In some embodiments, testing a scene profile includes sending instructions to implement (at least temporarily) the specified states of the accessories included in the scene profile. In some embodiments, the specified scene is displayed as a favorite scene on the home screen 610-1.

[0311] Returning now to FIG. 9A, in response to detecting a user input corresponding to the selection of one of the default scene profiles 902-1 through 902-4, the device 600 displays a user interface screen that includes a representation of the location's predetermined accessories. FIG. 9F shows an exemplary user interface screen corresponding to the default scene profile, good night 902-4. As shown in FIG. 9F, the good night scene profile is pre-populated with two predetermined accessories having predetermined states. More specifically, the front door is set to the locked state and the thermostat in the bedroom is set to 68 degrees. Additional exemplary embodiments regarding default scene profiles are described below.

[0312] In some embodiments, a given accessory is a default accessory based on a selected default scene profile that is automatically included in the scene profile without the need for user selection (e.g., a garage door can be pre-determined for the "arrive home" default scene). In some embodiments, a given accessory is determined based on an action previously performed by device 600 for the given accessory (or a user profile associated with device 600). In some embodiments, the accessory is proposed based on, for example, usage frequency, or a correlation between an event or time and the use of the accessory (e.g., bedroom lighting is determined for the "good morning" scene profile when device 600 typically operates the bedroom lighting in the morning, or a dishwasher can be pre-determined to be "start" for the "good night" scene profile when it typically operates after 11:00 p.m.).

[0313] In some embodiments, the representation of a given accessory includes an indication of a given state (e.g., a proposed state) for the given accessory, as shown in the embodiment of FIG. 9F. In some embodiments, the given state is a default state based on a selected default scene profile that is automatically included in the scene profile without the need for user selection (e.g., the proposed state of a garage door for "arrive home" is "open").

[0314] In some embodiments, a given state of a given accessory is determined based on an action previously taken for the given accessory and associated with the user of device 600. In some embodiments, the state of the accessory is proposed based on, for example, frequency of selection, or a correlation between an event or time and the state of the accessory (e.g., bedroom lighting is set to on for the "good morning" scene profile when device 600 typically turns on the bedroom lighting in the morning, or the state of a dishwasher can be pre-determined to be "start" for the "good night" scene profile when it typically starts after 11:00 p.m.).

[0315] Figure 10 is a flowchart showing a method for managing an external device controllable using an electronic device according to some embodiments. Method 1000 is executed on a device (e.g., 100, 300, 500, or 600) having a display and optionally a touch-sensing surface. Some of the operations in method 1000 may optionally be combined, the order of some operations may optionally be changed, and some operations may optionally be omitted.

[0316] As described below, method 1000 provides an intuitive method for managing a controllable external device. This method reduces the user's cognitive burden for managing a controllable external device, thereby creating a more efficient human-machine interface. In the case of a battery-operated computing device, power is conserved and the battery charging interval is increased by enabling the user to manage a controllable external device faster and more efficiently.

[0317] Optionally, at block 1002, the device displays a first user interface screen (e.g., 910-1) including a representation of a default scene profile (e.g., 902-1 to 902-4) including data regarding a predetermined state of a predetermined controllable electronic device at a location.

[0318] Optionally, at block 1004, while the device is displaying the first user interface screen, the device detects a first user input corresponding to the selection of a default scene profile, and in response to detecting the first user input, displays a second user interface screen (e.g., FIG. 9F) that includes a plurality of representations of a predetermined controllable external device at a location. Optionally, the plurality of representations of the predetermined controllable external device at the location includes an indication of a predetermined state of the predetermined controllable external device. Optionally, the predetermined controllable external device is determined based on an action previously performed by the device on the predetermined controllable external device. Optionally, the predetermined state of the predetermined controllable external device is determined based on an action previously taken on the predetermined controllable external device and associated with a user of the electronic device.

[0319] At block 1006, the device displays a first representation of a first controllable external device located at the location (e.g., 908-1 to 908-5 in FIG. 9C). Optionally, the first representation of the controllable external device includes an indication of the current state of the controllable external device.

[0320] At block 1008, the device detects a second user input (e.g., 920-1) corresponding to the selection of the first controllable external device.

[0321] At block 1010, after detecting the second user input, the device includes data in the scene profile that identifies the first controllable external device and the first state of the first controllable external device (e.g., FIG. 9D).

[0322] Optionally, at block 1012, the device detects a third user input and, in response to detecting the third user input, displays a third user interface screen that includes a second representation of a controllable external device (e.g., 908-6). Optionally, the third user interface screen includes an option to perform at least one of removing data associated with the controllable external device from a scene profile (e.g., 912-1), testing the scene profile (e.g., 912-2), specifying the scene profile (e.g., 912-3), and deleting the scene profile (e.g., 912-4).

[0323] Optionally, at block 1014, the device displays a representation of a second controllable external device located at a location (e.g., 908-5), detects a fourth user input corresponding to a selection of the second controllable external device (e.g., 920-2), determines whether the fourth user input meets a threshold criterion, and in response to detecting the fourth user input and in accordance with the determination that the threshold criterion is met, displays an indication of a first state of the second controllable external device (e.g., 608), detects a fifth user input corresponding to a selection of the first state of the second controllable external device (e.g., 630-2), and after detecting the fifth user input, includes in the scene profile data that identifies the second controllable external device and the first state of the second controllable external device.

[0324] Note that the details of the process (FIG. 10) described above with respect to method 1000 are also applicable in a similar manner to the methods described above and below. For example, methods 700, 1300, 1600, 1900, 2100, 2300, 2600, 2900, and 3200 optionally include one or more of the characteristics of the various methods described above with reference to method 1000.

[0325] According to some embodiments, FIG. 11 shows an exemplary functional block diagram of an electronic device 1100 configured according to the principles of the various embodiments described. According to some embodiments, the functional blocks of the electronic device 1100 are configured to perform the above-described techniques. The functional blocks of the device 1100 are optionally implemented by hardware, software, or a combination of hardware and software to perform the principles of the various examples described. It will be understood by those skilled in the art that the functional blocks described in FIG. 11 may optionally be combined or separated into sub-blocks to perform the principles of the various examples described. Accordingly, the description herein optionally supports any possible combination or separation of the functional blocks described herein, or further definitions.

[0326] As shown in FIG. 11, the electronic device 1100 includes a display unit 1102 configured to display a graphical user interface, optionally a touch sensing surface unit 1104 configured to receive contacts, and a processing unit 1106 coupled to the display unit 1102 and optionally the touch sensing surface unit 1104. In some embodiments, the processing unit 1106 includes a display enabling unit 1108, a detection unit 1110, an inclusion unit 1112, and a determination unit 1114.

[0327] The processing unit 1106 is configured to enable the display of a first representation of a first controllable external device located at a location (e.g., using the display enabling unit 1108), detect a first user input corresponding to the selection of the first controllable external device (e.g., using the detection unit 1110), and after detecting the first user input, include data identifying the first controllable external device and the first state of the first controllable external device in a scene profile (e.g., using the inclusion unit 1112). In some embodiments, the first representation of the controllable external device includes an indication of the current state of the controllable external device.

[0328] In some embodiments, the processing unit 1106 enables the display of a representation of a second controllable external device (e.g., using the display enabling unit 1108), detects a second user input corresponding to the selection of the second controllable external device (e.g., using the detection unit 1110), determines whether the second user input meets a threshold criterion (e.g., using the determination unit 1114), in response to detecting the second user input and according to the determination that the threshold criterion is met, enables the display of an indication of a first state of the second controllable external device (e.g., using the display enabling unit 1108), detects a third user input corresponding to the selection of the first state of the second controllable external device (e.g., using the detection unit 1110), and after detecting the third user input, is configured to include in the scene profile data identifying the second controllable external device and the first state of the second controllable external device (e.g., using the inclusion unit 1112).

[0329] In some embodiments, the processing unit 1106 is further configured to detect a fourth user input (e.g., using the detection unit 1110) and, in response to detecting the fourth user input, enable the display of a first user interface screen including a second representation of a controllable external device (e.g., using the display enabling unit 1108). Optionally, the first user interface screen includes options to perform at least one of removing data associated with the controllable external device from the scene profile, testing the scene profile, specifying the scene profile, and deleting the scene profile.

[0330] In some embodiments, the processing unit 1106 is further configured to enable (e.g., using the display enabling unit 1108) the display of a second user interface screen that includes a representation of a default scene profile that includes data regarding a predetermined state of a predetermined controllable electronic device at a location, before a first representation of a controllable external device is displayed.

[0331] In some embodiments, the processing unit 1106 detects (e.g., using the detection unit 1110) a fifth user input corresponding to a selection of a default scene profile while the second user interface screen is being displayed, and in response to detecting the fifth user input, is further configured to enable (e.g., using the display enabling unit 1108) the display of a third user interface screen that includes a plurality of representations of a predetermined controllable external device at the location. In some embodiments, the plurality of representations of the predetermined controllable external device at the location includes an indication of a predetermined state for the predetermined controllable external device.

[0332] In some embodiments, the predetermined controllable external device is determined based on actions previously performed by the electronic device on the predetermined controllable external device. In some embodiments, the predetermined state of the predetermined controllable external device is determined based on actions previously taken on the predetermined controllable external device and associated with a user of the electronic device.

[0333] The operations described above with reference to FIG. 10 are optionally implemented by the components shown in FIGS. 1A-1B or FIG. 11. For example, the operations of displaying 1002, 1004, 1006, and 1014, detecting 1004, 1008, 1012, and 1014, including 1010 and 1014, and determining 1014 are optionally executed by the event sorting unit 170, the event recognition unit 180, and the event processing unit 190. The event monitor 171 of the event sorting unit 170 detects a contact on the touch-sensitive display 112, and the event dispatcher module 174 distributes event information to the application 136-1. Each event recognition unit 180 of the application 136-1 compares the event information with its respective event definition 186 to determine whether a first contact at a first location on the touch-sensitive surface corresponds to a predefined event or sub-event, such as the activation of an affordance on the user interface. When a corresponding predefined event or sub-event is detected, the event recognition unit 180 activates the event processing unit 190 associated with the detection of that event or sub-event. The event processing unit 190 optionally utilizes or invokes the data update unit 176 or the object update unit 177 to update the internal state 192 of the application. In some embodiments, the event processing unit 190 accesses the corresponding GUI update unit 178 to update what is displayed by the application. Similarly, it will be apparent to those skilled in the art how other processes can be implemented based on the components shown in FIGS. 1A-1B.

[0334] FIGS. 12A-12C show exemplary user interfaces for creating and / or configuring an automation profile for a controllable external device, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes in FIGS. 13 and 21.

[0335] FIG. 12A shows a device 600 displaying a user interface screen 1210-1 for defining an automation profile associated with a location. In some embodiments, the user interface screen 1210-1 is displayed in response to a selection of the affordance of FIG. 6Q labeled "New Automation". The user interface screen 1210-1 includes representations 1202-1 to 1202-4 of criteria that can be selected for an automation profile. An automation profile includes data representing at least one accessory, a specified state for each represented accessory, and automation criteria for controlling at least one accessory. In some embodiments, the automation profile includes a scene profile (or a portion thereof) that is executed when the automation criteria of the automation profile are met. The automation criteria represented by items 1202-1 to 1202-4 may also be referred to as "trigger events", which are events in response to which the automation profile will be implemented.

[0336] In some embodiments, the automation criteria are at least partially based on the position of the device 600 relative to a location. Examples of location-based criteria include whether an electronic device is less than a threshold distance from a first location (e.g., "I am at home"), whether an electronic device is farther than a threshold distance from a first location (e.g., "I am away from home"), whether an electronic device has moved from farther than a threshold distance from a first location to less than a threshold distance from the location (e.g., "I have arrived home" 1202-2), and whether an electronic device has moved from less than a threshold distance from a first location to farther than a threshold distance from the location (e.g., "I am leaving home" 1202-1), but are not limited to these. In some embodiments, the threshold distance is the same in all directions from the location (e.g., a circular boundary). In some embodiments, the threshold distance varies depending on the direction from the location.

[0337] In some embodiments, the automation criteria are at least partially based on time (e.g., "it's time" 1202-3). Examples of time-based automation criteria include, but are not limited to, time of day (e.g., 10:30 am) and relative time (e.g., sunrise, or sunset).

[0338] In some embodiments, the automation criteria are at least partially based on the state of an accessory, or a state or event detected by an accessory (e.g., by a sensor). Examples of criteria based on the state of an accessory include, but are not limited to, the accessory being controlled (e.g., "the accessory is being controlled" 1202-4), and the accessory being triggered or its state changing (e.g., alarm, motion-sensing lighting, rain sensor, etc.). Examples of criteria based on a state or event detected by an accessory include, but are not limited to, weather events, notification events (e.g., receiving a notification), the presence or absence of a user at a location, the absence of a predicted event, machine learning, or manual (user) input as described in U.S. Patent Application No. 14 / 725,912, filed May 29, 2015, entitled "Accessory management Systems Using Environment Model", which is incorporated by reference above.

[0339] Returning to FIG. 12A, device 600 detects a user input corresponding to the selection of a first criterion. In the embodiment shown in FIG. 12A, the user input includes a contact 1220-1 on a representation 1202-1 of the first criterion labeled "I'm leaving home". Examples of the "I'm leaving home" criterion were described above. In response to detecting the user input 1220-1, device 600 associates the first criterion with the automation profile by including the first criterion in the automation criteria of the automation profile. As described above, in addition to the automation criteria, the automation profile also includes data representing at least one specified state for at least one accessory located at a location.

[0340] Figure 12A also shows a user input 1220-2 on an affordance labeled "control accessory". In response to detecting the user input 1220-2, the device 600 displays one or more representations of the location's accessories. In some embodiments, the device 600 displays a user interface similar to the user interface 910-3 described above with reference to FIG. 9C. In FIG. 9C, representations of the location's accessories are displayed and can be selected and included in the scene profile. The specified state of the accessory can also be selected for the scene profile. Using a similar user interface and technique, the accessory and the corresponding specified state of the accessory can be selected to add data indicating the specified state of the accessory to the automation profile. Similar to FIG. 9C, in some embodiments, the representation of the accessory includes an indication of the current state of the accessory. Also, similar to the default scenes described above with reference to FIGS. 9A and 9F, in some embodiments, the representation of the accessory includes an indication of the proposed state of the accessory. In some embodiments, the specified state is the current state of the accessory. In some embodiments, the specified state is a state selected by the user by one of the techniques described above with reference to FIGS. 6B-6K.

[0341] In the embodiment shown in FIG. 12A, the user can also choose to execute a scene profile as part of the automation. In response to user input 1220-3 corresponding to the selection of affordance 1204-1, device 600 displays a list of any scene profiles associated with a location, including data regarding the specified states of the accessories associated with the location, as shown, for example, in FIG. 12B. In the embodiment shown in FIG. 12B, the scene profiles are grouped by room. In some embodiments, the scene profiles are listed under the room if it includes the accessories associated with that room. The user can select one or more of the listed scene profiles and add data indicating the specified states of the accessories within the scene profile to the automation profile.

[0342] In some embodiments, after adding data indicating the specified states of the accessories to the automation profile, device 600 displays an overview of the automation profile based on the first accessory. An exemplary overview of the automation profile is shown by element 634 in FIG. 6Q.

[0343] After automation criteria and accessory data are established for the automation profile, device 600 determines whether the automation criteria are met and, in accordance with the determination that the automation criteria are met, sends a command to set the first accessory to the specified state. In some embodiments, device 600 sends the command directly to the accessory. In some embodiments, device 600 sends the command to a controller, server, host, or other external device configured to control the accessory.

[0344] In some embodiments, when the device 600 determines that the automation criteria are met, it automatically sends a command. For example, the device 600 automatically sends a command when the option 1206-1 labeled "automatically" in FIG. 12A is selected. In some embodiments, in accordance with the determination that the automation criteria are met, the device 600 displays a notification or request to identify an automation profile and provides an option (e.g., an affordance) to confirm or reject the implementation of the automation profile. For example, when the option 1206-2 labeled "always ask" in FIG. 12A is selected, a request to confirm the implementation of the automation profile is displayed. FIG. 12C shows an exemplary embodiment of a notification 1208 that includes a confirmation affordance 1212 and a cancellation affordance 1214. In accordance with a user input (e.g., a tap on the affordance 1212) corresponding to the confirmation of the implementation of the automation profile, the device 600 sends a command to implement the specified state of the automation profile. In accordance with a user input (e.g., a tap on the affordance 1214) corresponding to the non-confirmation or rejection of the implementation of the automation profile, the device 600 forgoes sending a command to implement the specified state of the automation profile or sends a command not to implement the specified state of the automation profile.

[0345] FIG. 13 is a flowchart illustrating a method for managing an external device controllable using an electronic device, according to some embodiments. The method 1300 is executed on a device (e.g., 100, 300, 500, or 600) having a display and optionally a touch-sensing surface. Some of the operations in the method 1300 may be optionally combined, the order of some of the operations may be optionally changed, and some of the operations may be optionally omitted.

[0346] As described below, method 1300 provides an intuitive way to manage a controllable external device. This method reduces the user's cognitive burden for managing the controllable external device, thereby creating a more efficient human-machine interface. In the case of a battery-operated computing device, power is conserved and the battery charging interval is increased by enabling the user to manage the controllable external device faster and more efficiently.

[0347] In block 1302, the device detects a first user input (e.g., 1220-1) corresponding to the selection of a first criterion (e.g., 1202-1).

[0348] In block 1304, in response to detecting the first user input, the device associates the first criterion with an automation profile that includes the automation criteria including at least the first criterion and data representing at least one designated state for at least one controllable external device located at the location. Optionally, the automation criteria are at least partially based on the location of the electronic device relative to the location. Optionally, the automation criteria are at least partially based on time. Optionally, the automation criteria are at least partially based on the state of the controllable external device at the location.

[0349] In block 1306, the device detects a second user input (e.g., 920-2) corresponding to the selection of a first controllable external device located at the location. Optionally, the selection of the first controllable external device includes the selection of a scene profile (e.g., FIG. 12B), and the scene profile includes data regarding the designated states of a plurality of controllable external devices at the location including the first controllable external device.

[0350] In block 1308, the device adds data indicating the designated state of the first controllable external device to the automation profile.

[0351] Optionally, at block 1310, the device displays an overview of an automation profile based on a first controllable external device (e.g., FIG. 6Q).

[0352] At block 1312, the device determines whether the automation criteria are met.

[0353] At block 1314, in accordance with the determination that the automation criteria are met, the device sends a command to set the first controllable external device to a specified state.

[0354] Optionally, at block 1316, the device detects a third user input corresponding to the selection of a second controllable external device located at the location, determines whether the third user input meets an input threshold criterion, and in accordance with the determination that the third user input meets the input threshold criterion, displays an indication of a first state of the second controllable external device (e.g., 608 of FIG. 6D), detects a fourth user input corresponding to the selection of the first state of the second controllable external device, and in response to detecting the fourth user input, includes data identifying the second controllable external device and the first state of the second controllable external device in the automation profile.

[0355] Note that the details of the process (FIG. 13) described above with respect to method 1300 are also applicable in a similar manner to the methods described above and below. For example, methods 700, 1000, 1600, 1900, 2100, 2300, 2600, 2900, and 3200 optionally include one or more of the characteristics of the various methods described above with reference to method 1300.

[0356] According to some embodiments, FIG. 14 shows an exemplary functional block diagram of an electronic device 1400 configured in accordance with the principles of the various embodiments described. According to some embodiments, the functional blocks of the electronic device 800 are configured to perform the techniques described above. The functional blocks of the device 1400 may optionally be implemented by hardware, software, or a combination of hardware and software to perform the principles of the various examples described. It will be understood by those skilled in the art that the functional blocks described in FIG. 14 may optionally be combined or separated into sub-blocks to implement the principles of the various examples described. Accordingly, the description herein optionally supports any possible combination or separation of the functional blocks described herein, or further definitions.

[0357] As shown in FIG. 14, the electronic device 1400 includes a display unit 1402 configured to display a graphical user interface, an optionally touch-sensing surface unit 1404 configured to receive contacts, and a processing unit 1406 coupled to the display unit 1402 and optionally the touch-sensing surface unit 1404. In some embodiments, the processing unit 1406 includes a detection unit 1408, an association unit 1410, an addition unit 1412, a determination unit 1414, a triggering unit 1416, a display enabling unit 1418, and an inclusion unit 1420.

[0358] The processing unit 1406 detects a first user input corresponding to the selection of a first criterion (e.g., using the detection unit 1408), and in response to detecting the first user input, associates the first criterion with an automation criterion including at least the first criterion and an automation profile including data representing at least one specified state for at least one controllable external device located at the location (e.g., using the association unit 1410), detects a second user input corresponding to the selection of a first controllable external device located at the location (e.g., using the detection unit 1408), adds data indicating the specified state of the first controllable external device to the automation profile (e.g., using the addition unit 1412), determines whether the automation criterion is satisfied (e.g., using the determination unit 1414), and in accordance with the determination that the automation criterion is satisfied, causes the transmission of an instruction to set the first controllable external device to the specified state (e.g., using the triggering unit 1416), and is configured as such.

[0359] In some embodiments, the selection of the first controllable external device includes the selection of a scene profile, and the scene profile includes data regarding the specified states of a plurality of controllable external devices at the location including the first controllable external device.

[0360] In some embodiments, the automation criterion is at least partially based on the location of the electronic device with respect to the location. In some embodiments, the automation criterion is at least partially based on time. In some embodiments, the automation criterion is at least partially based on the state of the controllable external devices at the location.

[0361] In some embodiments, the processing unit is further configured to enable the display of an overview of the automation profile based on the first controllable external device after adding data indicating the specified state of the first controllable external device to the automation profile (e.g., using the display enabling unit 1418).

[0362] In some embodiments, the processing unit detects a third user input corresponding to the selection of a second controllable external device located at a location (e.g., using detection unit 1408), determines whether the third user input meets a threshold criterion (e.g., using determination unit 1414), enables the display of an indication of a first state of the second controllable external device in accordance with the determination that the third user input meets the input threshold criterion (e.g., using display enabling unit 1418), detects a fourth user input corresponding to the selection of the first state of the second controllable external device (e.g., using detection unit 1408), and is further configured to include in the automation profile data identifying the second controllable external device and the first state of the second controllable external device in response to detecting the fourth user input (e.g., using inclusion unit 1420).

[0363] The operations described above with reference to FIG. 13 are optionally implemented by the components shown in FIGS. 1A-1B or FIG. 14. For example, the detecting operations 1302, 1306, and 1316, the associating operation 1304, the adding operation 1308, the displaying operations 1310 and 1316, the determining operations 1312 and 1316, the transmitting operation 1314, and the including operation 1316 are optionally executed by the event sorting unit 170, the event recognition unit 180, and the event processing unit 190. The event monitor 171 of the event sorting unit 170 detects a contact on the touch-sensitive display 112, and the event dispatcher module 174 distributes the event information to the application 136-1. Each event recognition unit 180 of the application 136-1 compares the event information with its respective event definition 186 to determine whether a first contact at a first location on the touch-sensitive surface corresponds to a predefined event or sub-event, such as the activation of an affordance on the user interface. When a corresponding predefined event or sub-event is detected, the event recognition unit 180 activates the event processing unit 190 associated with the detection of that event or sub-event. The event processing unit 190 optionally utilizes or invokes the data update unit 176 or the object update unit 177 to update the internal state 192 of the application. In some embodiments, the event processing unit 190 accesses the corresponding GUI update unit 178 to update what is displayed by the application. Similarly, it will be apparent to those skilled in the art how other processes can be implemented based on the components shown in FIGS. 1A-1B.

[0364] FIGS. 15A-15B show an exemplary user interface for associating an accessory with a location, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the process in FIG. 16.

[0365] In some embodiments, the accessory is associated with (e.g., added to a location profile) a location using an image that identifies the accessory (e.g., a barcode, a quick response (QR) code, etc.). The image can be included, for example, on the accessory itself, or on a material provided or associated with the accessory (e.g., packaging, instruction manual, etc.).

[0366] FIG. 15A shows a device 600 displaying a user interface screen 1510-1 that includes an instruction to capture an image of a pattern (e.g., "scan the code"). The device 600 displays a live video feed from a camera associated with the device 600 within an area 1502 of the user interface screen 1510-1 to assist the user in aligning the image within the camera's field of view. In some embodiments, the user interface screen 1510-1 is displayed in response to a selection of an affordance shown in FIG. 6M labeled "Add Accessory".

[0367] Using the camera, the device 600 captures an image of the pattern 1504. Next, the device 600 determines whether the pattern corresponds to an accessory. In some embodiments, instead of capturing the image itself, the device 600 obtains the image from an external device. In some embodiments, the device 600 compares the pattern within the image to accessory information stored in a data repository (e.g., local memory or remote memory, a database, etc.). In some embodiments, the device 600 also receives identification data from the accessory (e.g., via Bluetooth communication) and compares it to the pattern within the image to determine whether the pattern corresponds to the accessory. In this way, the device 600 verifies that it has correctly identified the accessory associated with the image.

[0368] In response to determining that the pattern corresponds to an accessory, device 600 associates the accessory with a location. In some embodiments, device 600 creates a profile (an "accessory profile") associated with the accessory that includes data representing the operating state of the accessory. In some embodiments, in response to determining that the pattern corresponds to an accessory, device 600 enables the electronic device to exchange information with the accessory (e.g., via near-field communication, Bluetooth protocol, Internet, local network, etc.) so that device 600 can monitor and / or control the state of the accessory, establishing a pairing relationship with the accessory.

[0369] In some embodiments, device 600 displays a user interface for entering information about the accessory, including assigning the accessory to a room or designated area of a location. FIG. 15B shows a user interface screen 1510-2 for displaying information about the accessory, a representation 1506 of the accessory, and options 1508-1 and 1508-2 for entering information about the accessory. The user interface screen 1510-2 and / or the representation 1506 can be automatically populated based on information obtained, for example, from capturing an image to identify the accessory. In some embodiments, the information is manually obtained based on input from the user. In the embodiment shown in FIG. 15B, the representation of the accessory includes an icon, name, and room associated with the accessory. In some embodiments, the designated name, image, room, or other information is included in the data profile of the accessory.

[0370] To associate an accessory with a room, the user interface screen 1510-2 includes an affordance 1508-2 labeled "Select Room" that, when selected, allows the user to specify a room. In some embodiments, the device displays a list of rooms, a drop-down menu, a text input field, etc. for the user to select a room. In some embodiments, based on the position of the device 600 relative to the location, a list of rooms is displayed and arranged. For example, in some embodiments, the device 600 estimates which room it is in based on its position and presents the estimated room as the first option. The device 600 detects a user input (e.g., a tap on a room within a drop-down menu) representing the selection of a room and associates the selected room with the accessory in response to detecting the user input. In some embodiments, a similar technique is used to input a name for the accessory. In some embodiments, the device 600 updates the user interface screen 1510-2 and / or the representation 1506 to include the room information in the profile associated with the accessory.

[0371] The user interface screen 1510-2 also includes an icon 1512 with an image that can be associated with an accessory in response to the selection of the corresponding icon. The user interface screen 1510-2 also includes an affordance 1514 for designating an accessory as a "favorite". In some embodiments, the device 600 detects a user input on the affordance 1541 and, in response, displays a representation of the accessory, for example, on the home screen 610-1 under the title of the favorite accessory.

[0372] FIG. 16 is a flowchart showing a method for managing an external device controllable using an electronic device, according to some embodiments. Method 1600 is executed on a device (e.g., 100, 300, 500, or 600) having a display, optionally a touch-sensitive surface, and optionally a camera. Some operations in method 1600 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0373] As described below, method 1600 provides an intuitive method for managing a controllable external device. This method reduces the user's cognitive burden for managing a controllable external device, thereby creating a more efficient human-machine interface. In the case of a battery-operated computing device, power is conserved and the battery charging interval is increased by enabling the user to manage a controllable external device more quickly and efficiently.

[0374] Optionally, at block 1602, the device displays a first user interface screen (e.g., 1510-1) that includes an instruction to capture an image of a pattern. Optionally, the first user interface screen is displayed in response to a selection of a first affordance on a second user interface screen (e.g., FIG. 6M).

[0375] At block 1604, the device uses the camera to capture an image of a pattern (e.g., 1504).

[0376] At block 1606, the device determines that the pattern corresponds to a controllable external device that is configured to operate in two or more states and is remotely controllable between the two or more states.

[0377] In block 1608, in response to determining that the pattern corresponds to a controllable external device, the device associates the controllable external device with a location having at least one designated room.

[0378] Optionally, in block 1610, after determining that the pattern corresponds to a controllable external device, the device displays a representation (e.g., 1506) of the controllable external device.

[0379] In block 1612, the device detects a first user input (e.g., 1508-2) representing a selection of a room from at least one designated room.

[0380] In block 1614, in response to detecting the first user input, the device associates the selected room with the controllable external device.

[0381] Optionally, in block 1616, after determining that the pattern corresponds to a controllable external device, the device detects a second user input (e.g., selection of affordance 1514), and in response to detecting the second user input, displays a representation of the controllable external device on a first user interface screen (e.g., 610-1).

[0382] Note that the details of the process (FIG. 16) described above with respect to method 1600 are also applicable in a similar manner to the methods described above and below. For example, methods 700, 1000, 1300, 1900, 2100, 2300, 2600, 2900, and 3200 optionally include one or more of the various method characteristics described above with reference to method 1600.

[0383] According to some embodiments, FIG. 17 shows an exemplary functional block diagram of an electronic device 1700 configured in accordance with the principles of the various embodiments described. According to some embodiments, the functional blocks of the electronic device 1700 are configured to perform the techniques described above. The functional blocks of the device 1700 may optionally be implemented by hardware, software, or a combination of hardware and software to perform the principles of the various examples described. It will be understood by those skilled in the art that the functional blocks described in FIG. 17 may optionally be combined or separated into sub-blocks to implement the principles of the various examples described. Accordingly, the description herein optionally supports any possible combination or separation of the functional blocks described herein, or further definitions.

[0384] As shown in FIG. 17, the electronic device 1700 includes a display unit 1702 configured to display a graphical user interface, optionally a touch sensing surface unit 1704 configured to receive contacts, a camera unit 1706 configured to capture images, and a processing unit 1708 coupled to the display unit 1702, optionally the touch sensing surface unit 1704, and optionally the camera unit 1706. In some embodiments, the processing unit 1708 includes a triggering unit 1710, a determination unit 1712, an association unit 1714, a detection unit 1716, and a display enabling unit 1718.

[0385] The processing unit 1708 causes an image of a pattern to be captured using a camera unit (e.g., using the triggering unit 1710), determines (e.g., using the determination unit 1712) that the pattern corresponds to a controllable external device configured to operate in two or more states and remotely controllable between the two or more states, and in response to the determination that the pattern corresponds to a controllable external device, associates (e.g., using the association unit 1714) the controllable external device with a location having at least one designated room, detects (e.g., using the detection unit 1716) a first user input representing a selection of a room from the at least one designated room, and in response to detecting the first user input, associates the selected room with the controllable external device (e.g., using the association unit 1714).

[0386] In some embodiments, the processing unit 1708 is further configured to enable (e.g., using the display enabling unit 1718) the display of a first user interface screen including an instruction to capture an image of the pattern before the image of the pattern is captured. In some embodiments, the first user interface screen is displayed in response to a selection of a first affordance on a second user interface screen. In some embodiments, after determining that the pattern corresponds to a controllable external device, the processing unit detects (e.g., using the detection unit 1716) a second user input and is further configured to enable (e.g., using the display enabling unit 1718) the display of a representation of the controllable external device on a second user interface screen in response to detecting the second user input.

[0387] In some embodiments, after determining that the pattern corresponds to a controllable external device, the processing unit 1708 is further configured to enable (e.g., using the display enabling unit 1718) the display of a representation of the controllable external device.

[0388] The operations described above with reference to FIG. 16 are optionally implemented by the components shown in FIGS. 1A-1B or FIG. 17. For example, the operations of displaying 1602 and 1610, capturing 1604, determining 1606, associating 1608 and 1614, and detecting 1612 and 1616 are optionally performed by the event sorter 170, the event recognizer 180, and the event processor 190. The event monitor 171 of the event sorter 170 detects a contact on the touch-sensitive display 112, and the event dispatcher module 174 distributes event information to the application 136-1. Each event recognizer 180 of the application 136-1 compares the event information with its respective event definition 186 to determine whether a first contact at a first location on the touch-sensitive surface corresponds to a predefined event or sub-event, such as the activation of an affordance on the user interface. When a corresponding predefined event or sub-event is detected, the event recognizer 180 activates the event processor 190 associated with the detection of that event or sub-event. The event processor 190 optionally uses or calls the data updater 176 or the object updater 177 to update the application internal state 192. In some embodiments, the event processor 190 accesses the corresponding GUI updater 178 to update what is displayed by the application. Similarly, it will be apparent to those skilled in the art how other processes can be implemented based on the components shown in FIGS. 1A-1B.

[0389] FIGS. 18A-18C show an exemplary user interface for setting criteria for an automation profile associated with a location, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the process in FIG. 19.

[0390] FIG. 18A shows a user interface screen 1810-1 displaying a map view 1802. The map view 1802 is substantially centered on a location associated with an automation profile. The map view 1802 also includes an indication 1804 of a location-based criterion, specifically, a circular threshold around the location of the position. The location-based criterion is at least partially based on the position of the device 600 relative to the location. In some embodiments, the location-based criterion includes whether the device 600 is within a threshold distance of the location represented by the radius of the circle. In some embodiments, the threshold distance is the same in all directions from the location (e.g., a circular boundary). In some embodiments, the threshold distance varies depending on the direction from the location.

[0391] In some embodiments, the location-based criterion includes whether the device 600 is less than a threshold distance from the location. In some embodiments, the location-based criterion includes whether the device 600 is farther than a threshold distance from the location. In some embodiments, the location-based criterion includes whether the device 600 has moved from a location farther than a threshold distance from the location to a location less than a threshold distance from the location. This criterion optionally corresponds to the "I'm home" criterion 1202-2 represented in FIG. 12A. In some embodiments, the location-based criterion includes whether the device 600 has moved from a location less than a threshold distance from a first location to a location farther than a threshold distance from the location. This criterion optionally corresponds to the "I'm leaving home" criterion 1202-1 represented in FIG. 12A.

[0392] The device 600 detects a user input (e.g., selection of the criterion 1202-1 displayed in FIG. 12A) corresponding to the specification of the location-based criterion. In response to detecting the user input corresponding to the specification of the criterion, the device 600 adds the location-based criterion to the automation criteria of the automation profile associated with the location.

[0393] User interface screen 1810-1 also includes additional selectable criteria or trigger conditions for the automation profile. User interface objects 1804-1 through 1804-4 represent time-based criteria, "all day," "only during the day," "only at night," and a specified time window (e.g., "from" and "to") for determining when the automation profile can be enabled. User interface screen 1810-1 also includes options 1806-1 through 1806-7 for selecting, in some cases, which days automation is repeated.

[0394] Device 600 also provides an option to further specify the time-based criteria of the selected objects 1804-1 through 1804-4. In some embodiments, in response to selection of the information icon 1808 on user interface screen 1810-1, device 600 displays user interface screen 1810-2 shown in FIG. 18B, which provides additional time-based criteria for the automation profile, including relative start and end times (e.g., sunrise and sunset), a specific start time (e.g., a time zone), a specific end time, a relative end time, a duration, and time-based constraints (e.g., only on Monday). Device 600 also optionally provides an option to further specify time-based criteria for the selected relative start and end times. In some embodiments, in response to selection of the information icon 1812 on user interface screen 1810-2, device 600 displays user interface screen 1810-3 shown in FIG. 18C, which provides an option to add or subtract an offset from a relative time (e.g., sunrise).

[0395] Device 600 detects a user input (e.g., a tap on one or more of the user interface objects shown in FIGS. 18A-18C) corresponding to the specification of a time-based criterion and, in response, adds the selected time-based criterion to the automation criteria of the automation profile for controlling at least one affordance associated with the automation profile.

[0396] In some embodiments, the user interface and time-based criteria described with respect to FIGS. 18A-18C are provided between the criteria and / or configuration of the automation profile (e.g., "I leave home" 1202-1 and "I arrive home" 1202-2) as described above with reference to FIGS. 12A-12B. Conversely, in some embodiments, the automation criteria are based in part on the state of location accessories, as described with reference to the event criteria 1202-4 "accessory is controlled" in FIG. 12A. (For example, the automation criteria are based on whether an accessory is controlled and / or triggered, or whether an accessory detects an event.)

[0397] Once the automation criteria are established, the location of device 600 is determined. In some embodiments, device 600 determines its location via, for example, GPS, WiFi, or other signals. In some embodiments, device 600 determines its location based on information obtained from an external device.

[0398] Device 600 determines whether the automation criteria are met based at least in part on the determined location. In accordance with a determination that the automation criteria are met, device 600 transmits instructions to implement an automation profile, including setting at least one accessory to a specified state. In some embodiments, in accordance with a determination that the automation criteria are not met, device 600 refrains from transmitting instructions to implement the automation profile.

[0399] FIG. 19 is a flowchart showing a method for managing an external device controllable using an electronic device, according to some embodiments. Method 1900 is executed on a device (e.g., 100, 300, 500, or 600) having a display and optionally a touch-sensitive surface. Some operations in method 1900 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0400] As will be described below, method 1900 provides an intuitive way to manage a controllable external device. This method reduces the user's cognitive burden for managing a controllable external device, thereby creating a more efficient human-machine interface. In the case of a battery-operated computing device, power is conserved and the battery charging interval is increased by enabling the user to manage the controllable external device faster and more efficiently.

[0401] In block 1902, the device detects a user input (e.g., 1220-1) corresponding to a specification of a criterion (e.g., 1202-1) based at least in part on the position of the electronic device relative to a first location having at least one controllable external device. Optionally, the device displays a map view (e.g., 1802) having an indication of the location-based criterion (e.g., 1804). Optionally, the location-based criterion includes one or more of whether the electronic device is within a threshold distance from the first location, whether the electronic device is farther than a threshold distance from the first location, whether the electronic device has moved from farther than a threshold distance from the first location to within a threshold distance from the location, and whether the electronic device has moved from within a threshold distance from the first location to farther than a threshold distance from the location.

[0402] In block 1904, in response to detecting the user input corresponding to the specification of the criterion, the device adds the location-based criterion to an automation profile associated with the location that includes the location-based criterion for controlling at least one controllable external device and data representing a specified state of the at least one controllable external device. Optionally, the automation criterion for controlling at least one controllable external device is based at least in part on the state (e.g., 1202-4) of the first controllable external device of the at least one controllable external device.

[0403] In block 1906, the device determines the location of the electronic device.

[0404] In block 1908, the device determines whether an automation criterion for controlling at least one controllable external device is met, based at least in part on the determined location of the electronic device.

[0405] In block 1910, in accordance with the determination that an automation criterion for controlling at least one controllable external device is met, the device sends a command to set at least one controllable external device to a specified state.

[0406] Optionally, in block 1912, in accordance with the determination that an automation criterion for controlling at least one controllable external device is not met, the device refrains from sending a command to set at least one controllable external device to a specified state.

[0407] Optionally, in block 1914, the device detects a user input corresponding to the specification of a time-based criterion (e.g., 1202-3 and 1804-1 to 1804-4), and in response to detecting the user input, adds the time-based criterion to the automation criterion for controlling at least one controllable external device.

[0408] Note that the details of the process (FIG. 19) described above with respect to method 1900 are also applicable in a similar manner to the methods described above and below. For example, methods 700, 1000, 1300, 1600, 2100, 2300, 2600, 2900, and 3200 optionally include one or more of the characteristics of the various methods described above with reference to method 1900.

[0409] According to some embodiments, FIG. 20 shows an exemplary functional block diagram of an electronic device 2000 configured in accordance with the principles of the various embodiments described. According to some embodiments, the functional blocks of the electronic device 2000 are configured to perform the techniques described above. The functional blocks of the device 2000 may optionally be implemented by hardware, software, or a combination of hardware and software to perform the principles of the various examples described. It will be understood by those skilled in the art that the functional blocks described in FIG. 20 may optionally be combined or separated into sub-blocks to perform the principles of the various examples described. Accordingly, the description herein optionally supports any possible combination or separation of the functional blocks described herein, or further definitions.

[0410] As shown in FIG. 20, the electronic device 2000 includes a display unit 2002 configured to display a graphic user interface, optionally, a touch sensing surface unit 2004 configured to receive contacts, and a processing unit 2006 coupled to the display unit 2002 and optionally the touch sensing surface unit 2004. In some embodiments, the processing unit 2006 includes a detection unit 2008, an addition unit 2010, a determination unit 2012, a triggering unit 2014, and a display enabling unit 2016.

[0411] The processing unit 2006 detects a user input corresponding to the specification of a criterion based at least in part on the position of the electronic device relative to a first location having at least one controllable external device (e.g., using the detection unit 2008), and in response to detecting the user input corresponding to the specification of the criterion, adds the position-based criterion to an automation profile associated with the location, the automation profile including the position-based criterion for controlling at least one controllable external device and data representing the specified state of the at least one controllable external device (e.g., using the addition unit 2010), determines the position of the electronic device (e.g., using the determination unit 2012), determines whether an automation criterion for controlling at least one controllable external device is satisfied based at least in part on the determined position of the electronic device (e.g., using the determination unit 2012), and is configured to cause the transmission of an instruction to set the at least one controllable external device to the specified state in accordance with the determination that the automation criterion for controlling the at least one controllable external device is satisfied (e.g., using the triggering unit 2014).

[0412] In some embodiments, the processing unit 2006 is further configured to refrain from transmitting an instruction to set the at least one controllable external device to the specified state in accordance with the determination that the automation criterion for controlling the at least one controllable external device is not satisfied.

[0413] In some embodiments, the processing unit 2006 is further configured to enable the display of a map view having an indication of the position-based criterion (e.g., using the display enabling unit 2016).

[0414] In some embodiments, the location-based criterion includes whether the electronic device is within a threshold distance from a first location. In some embodiments, the location-based criterion includes whether the electronic device is beyond a threshold distance from a first location. In some embodiments, the location-based criterion includes whether the electronic device has moved from beyond a threshold distance from a first location to within a threshold distance from the location. In some embodiments, the location-based criterion includes whether the electronic device has moved from within a threshold distance from a first location to beyond a threshold distance from the location.

[0415] In some embodiments, the processing unit 2006 is further configured to detect a user input corresponding to the specification of a time-based criterion (e.g., using the detection unit 2008) and, in response to detecting the user input, add the time-based criterion to an automation criterion for controlling at least one controllable external device (e.g., using the addition unit 2010).

[0416] In some embodiments, the automation criterion for controlling at least one controllable external device is partially based on the state of a first controllable external device of the at least one controllable external device.

[0417] The operations described above with reference to FIG. 19 are optionally implemented by the components shown in FIGS. 1A-1B or FIG. 20. For example, the detecting operations 1902 and 1914, the adding operations 1904 and 1914, the determining operations 1906 and 1908, the transmitting operation 1910, and the matching operation 1912 are optionally executed by the event sorting unit 170, the event recognition unit 180, and the event processing unit 190. The event monitor 171 of the event sorting unit 170 detects contacts on the touch-sensitive display 112, and the event dispatcher module 174 distributes event information to the application 136-1. Each event recognition unit 180 of the application 136-1 compares the event information with its respective event definition 186 to determine whether a first contact at a first location on the touch-sensitive surface corresponds to a predefined event or sub-event, such as the activation of an affordance on the user interface. When a corresponding predefined event or sub-event is detected, the event recognition unit 180 activates the event processing unit 190 associated with the detection of that event or sub-event. The event processing unit 190 optionally utilizes or invokes the data update unit 176 or the object update unit 177 to update the application internal state 192. In some embodiments, the event processing unit 190 accesses the corresponding GUI update unit 178 to update what is displayed by the application. Similarly, it will be apparent to those skilled in the art how other processes can be implemented based on the components shown in FIGS. 1A-1B.

[0418] FIG. 21 is a flowchart showing a method for managing an external device controllable using an electronic device, according to some embodiments. The method 2100 is executed on a device (e.g., 100, 300, 500, or 600) having a display and optionally a touch-sensitive surface. Some of the operations in the method 2100 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0419] As will be described below, method 2100 provides an intuitive way to manage controllable external devices. This method reduces the user's cognitive burden for managing controllable external devices, thereby creating a more efficient human-machine interface. In the case of a battery-operated computing device, power is conserved and the battery charging interval is increased by enabling the user to manage controllable external devices faster and more efficiently.

[0420] In block 2102, the device determines whether criteria for controlling the state of at least one controllable external device and data representing a specified state for at least one controllable external device are met, such as the criteria of automation profiles 1202-1 to 1202-4 and 1804-1 to 1804-4. Optionally, the criteria of the automation profile are at least partially based on one or more of the position of the electronic device relative to the location associated with at least one controllable external device, the time, and the state of the first controllable external device of at least one controllable external device (e.g., FIGS. 12A and 18A).

[0421] In block 2104, in accordance with the determination that the criteria of the automation profile are met, the device displays a notification (e.g., 1208) that includes an indication identifying the automation profile and an indication confirming the implementation of the automation profile (e.g., 1212 in FIG. 12C).

[0422] In block 2106, the device detects a user input (e.g., selection of affordance 1212 or 1214).

[0423] In block 2108, in response to detecting user input, the device transmits an instruction to implement the specified state of the automation profile according to user input (e.g., selection of affordance 1212) corresponding to confirmation of implementation of the automation profile, and transmits an instruction not to implement the specified state of the automation profile according to user input (e.g., selection of affordance 1214) corresponding to non-confirmation of implementation of the automation profile.

[0424] Note that the details of the process (FIG. 21) described above with respect to method 2100 are also applicable in a similar manner to the methods described above and below. For example, methods 700, 1000, 1300, 1600, 1900, 2300, 2600, 2900, and 3200 optionally include one or more of the characteristics of the various methods described above with reference to method 2100.

[0425] According to some embodiments, FIG. 22 shows an exemplary functional block diagram of an electronic device 2200 configured according to the principles of the various embodiments described. According to some embodiments, the functional blocks of the electronic device 2200 are configured to perform the above-described techniques. The functional blocks of device 2200 are optionally implemented by hardware, software, or a combination of hardware and software to perform the principles of the various examples described. It will be understood by those skilled in the art that the functional blocks described in FIG. 22 are optionally combined or separated into sub-blocks to perform the principles of the various examples described. Thus, the description herein optionally supports any possible combination or separation of the functional blocks described herein, or further definitions.

[0426] As shown in FIG. 22, the electronic device 2200 includes a display unit 2202 configured to display a graphical user interface, optionally a touch sensing surface unit 2204 configured to receive contacts, and a processing unit 2206 coupled to the display unit 2202 and optionally the touch sensing surface unit 2204. In some embodiments, the processing unit 2206 includes a determination unit 2208, a display enabling unit 2210, a detection unit 2212, and a triggering unit 2214.

[0427] The processing unit 2206 determines whether criteria of an automation profile are met, the criteria including criteria for controlling the state of at least one controllable external device and data representing a specified state for the at least one controllable external device (e.g., using the determination unit 2208), enables display of a notification including an indication identifying the automation profile and an indication confirming implementation of the automation profile according to a determination that the criteria of the automation profile are met (e.g., using the display enabling unit 2210), detects user input (e.g., using the detection unit 2212), and in response to detecting the user input, causes transmission of an instruction to implement the specified state of the automation profile according to the user input corresponding to confirmation of implementation of the automation profile (e.g., using the triggering unit 2214), and causes transmission of an instruction not to implement the specified state of the automation profile according to the user input corresponding to non - confirmation of implementation of the automation profile (e.g., using the triggering unit 2214), and is configured as such.

[0428] In some embodiments, the criteria for the automation profile are at least partially based on the position of the electronic device relative to a location associated with at least one controllable external device. In some embodiments, the criteria for the automation profile are at least partially based on time. In some embodiments, the criteria for the automation profile are at least partially based on the state of a first controllable external device of at least one controllable external device.

[0429] The operations described above with reference to FIG. 21 are optionally implemented by the components shown in FIGS. 1A-1B or FIG. 22. For example, the operations of determining 2102, displaying 2104, detecting 2106, and transmitting 2108 are optionally performed by the event sorter 170, the event recognizer 180, and the event processor 190. The event monitor 171 of the event sorter 170 detects a contact on the touch-sensitive display 112, and the event dispatcher module 174 distributes event information to the application 136-1. Each event recognizer 180 of the application 136-1 compares the event information with its respective event definition 186 to determine whether a first contact at a first location on the touch-sensitive surface corresponds to a predefined event or sub-event, such as the activation of an affordance on the user interface. When a corresponding predefined event or sub-event is detected, the event recognizer 180 activates the event processor 190 associated with the detection of that event or sub-event. The event processor 190 optionally utilizes or invokes the data updater 176 or the object updater 177 to update the application internal state 192. In some embodiments, the event processor 190 accesses the corresponding GUI updater 178 to update what is displayed by the application. Similarly, it will be apparent to those skilled in the art how other processes can be implemented based on the components shown in FIGS. 1A-1B.

[0430] FIG. 23 is a flowchart showing a method for managing an external device controllable using an electronic device, according to some embodiments. Method 2300 is executed on a device (e.g., ...

Claims

1. 1. A method for managing at least one controllable external device, comprising: An electronic device comprising a display, one or more processors, and a memory, displaying a user interface corresponding to an automation profile, the automation profile including data representing automation criteria for controlling the at least one controllable external device and a specified state of the at least one controllable external device, the automation criteria including location-based criteria and time-based criteria, the automation criteria for controlling the at least one controllable external device including automation criteria for controlling the at least one controllable external device that are based in part on an operational state of a separate controllable external device distinct from the at least one controllable external device; following displaying the user interface corresponding to the automation profile; detecting a set of user inputs, said set of user inputs comprising: a user input corresponding to the specification of the location-based criteria, the location-based criteria being based at least in part on the location of the electronic device relative to a first location having the at least one controllable external device; and a user input corresponding to the specification of the time-based criteria; and displaying a map view including an indication of at least one of the specified location-based criteria; in response to detecting the set of user inputs including the user input corresponding to a specification of the location-based criteria, associating the specified location-based criteria with the automation criteria of the automation profile associated with the first location; in response to detecting the set of user inputs including the user input corresponding to a specification of the time-based criterion, associating the time-based criterion with the automation criterion for controlling the at least one controllable external device; transmitting an instruction to set the at least one controllable external device to the specified state in response to a determination that the automation criterion for controlling the at least one controllable external device is satisfied; A method comprising:

2. determining the location of the electronic device; determining whether the automation criteria for controlling the at least one controllable external device are satisfied based at least in part on the determined location of the electronic device; and transmitting an instruction to set the at least one controllable external device to the specified state in response to a determination that the automation criterion for controlling the at least one controllable external device is satisfied; The method of claim 1 further comprising:

3. 3. The method of claim 2, further comprising refraining from sending an instruction to set the at least one controllable external device to the specified state in accordance with a determination that the automation criterion for controlling the at least one controllable external device is not satisfied.

4. The method of claim 1 , wherein the specified position-based criteria includes whether the electronic device is less than a threshold distance from the first location.

5. The method of claim 1 , wherein the specified position-based criteria includes whether the electronic device is more than a threshold distance away from the first location.

6. The method of claim 1 , wherein the specified location-based criteria includes whether the electronic device has moved from more than a threshold distance away from the first location to less than the threshold distance away from the first location.

7. The method of claim 1 , wherein the specified location-based criteria includes whether the electronic device moves from less than a threshold distance from the first location to more than the threshold distance from the first location.

8. The method of claim 1 , wherein the automation criteria for controlling the at least one controllable external device is based in part on a state of a first controllable external device of the at least one controllable external device.

9. A computer program product causing a computer to carry out the method according to any one of claims 1 to 8.

10. 1. An electronic device comprising: A memory storing the computer program according to claim 9; one or more processors capable of executing the computer programs stored in the memory; Equipped with The electronic device is configured to communicate with a display. Electronic devices.

11. 1. An electronic device configured to communicate with a display, comprising: Means for carrying out the method according to any one of claims 1 to 8 An electronic device comprising:

Images