Systems, devices, and methods for dynamically providing user interface controls in touch-sensitive secondary display

Abstract

To dynamically update a touch-sensitive secondary display.SOLUTION: A method includes (i) displaying, on a primary display, a plurality of user interface (UI) objects associated with an application in response to a request to open the application, the plurality of UI objects including a first UI object displayed with associated content and other UI objects displayed without associated content; and (ii) displaying, on a touch-sensitive secondary display, a set of affordances representing the plurality of UI objects. The method also includes detecting, via the touch-sensitive secondary display, a swipe gesture in a direction from a first affordance and towards a second affordance, the first affordance representing the first UI object and the second affordance representing a distinct second UI object.SELECTED DRAWING: Figure 1A

Description

Technical Field

[0001] The disclosed embodiments relate to keyboards, and more particularly, to improved techniques for receiving input via a dynamic input / output (I / O) device.

Background Art

[0002] Existing keyboards include any number of physical keys for entering information (e.g., characters) into a computing device. Typically, a user presses a key or otherwise actuates it movably to provide an input corresponding to the key. In addition to providing character input, a keyboard may include keys that are actuated movably and relate to function inputs. For example, a keyboard may include an "escape" or "esc" key so that a user can activate an escape or exit function. On many keyboards, a set of function keys for function inputs is placed on a "function row". Typically, a set of alphanumeric keys is placed in the portion of the keyboard closest to the user, and the function row is placed adjacent to the alphanumeric and in a portion of the keyboard further from the user. A keyboard may also include function keys that are not part of the function row described above.

[0003] With the emergence and popularity of portable computing devices such as laptop computers, the area consumed by a dedicated keyboard may be limited by the corresponding size of the display. Compared to an external keyboard for a desktop computer, a dedicated keyboard that is a component of a portable computing device may have fewer keys, smaller keys, or keys that are closer together in order to enable the smaller overall size of the portable computing device.

[0004] Existing dedicated keyboards are temporally static and fixed despite changes in the display. Further, the functionality of software applications displayed on the screen is typically accessed via toolbars and menus that act bidirectionally when the user uses a mouse. This requires the user to regularly switch modes and move their hand location between the keyboard and the mouse. Alternatively, the functionality of the application is accessed via complex key combinations that require memorization and practice. Accordingly, it is desirable to provide an I / O device (and method for an I / O device) that addresses the shortcomings of existing systems. SUMMARY OF THE INVENTION

[0005] The embodiments described herein address the above disadvantages by providing dynamic and space - efficient I / O devices and methods. Such devices and methods optionally complement or replace existing input devices and methods. Such devices and methods also reduce the amount of mode switching required of the user (e.g., moving one's hand between a keyboard and a mouse, or moving one's eyes from the keyboard to the display), thereby reducing the number of inputs required to activate a desired function (e.g., reducing the number of inputs required to select a menu option, as will be described in more detail below). Such devices and methods also make more information available on a limited screen (e.g., a touch - sensitive secondary display is used to provide more information to the user, and this information is presented efficiently using limited screen space). Such devices and methods also provide an enhanced effect that makes information more distinguishable, for example, on the touch - sensitive secondary display 104, so that the device provides an output based on continuous input from the user directed at either the touch - sensitive secondary display or the primary display, and then that output is used to facilitate further input from the user, providing a sustained two - way action (e.g., providing a color picker that allows the user to quickly preview how information will be rendered on the primary display by providing an input on the touch - sensitive secondary display), and also reducing the amount of two - way action required from the user to achieve the desired result (e.g., allowing the user to log in to their device using a single biometric input), thereby providing an improved man - machine interface. For these reasons and those described below, the devices and methods described herein reduce the power consumption of an electronic device and improve battery life.

[0006] (A1) According to some embodiments, in a computing system (e.g., computing system 100 or system 200, FIGS. 1A - 2D) including one or more processors, a memory, a first housing including a primary display (e.g., housing 110 including display 102 or housing 204 including display 102), and a second housing at least partially including a physical keyboard (e.g., keyboard 106, FIG. 1A) and a touch - sensitive secondary display (e.g., dynamic function row 104, FIG. 1A, also referred to as a "touch - screen display"), a method is performed. In some embodiments, the touch - sensitive secondary display is separate from the physical keyboard (e.g., the touch - sensitive secondary display is included as part of an external input mechanism 222 (i.e., a stand - alone display) or the touch - sensitive display is integrated with another device such as touch pad 108, FIG. 2C). The method includes displaying a first user interface on the primary display, the first user interface comprising one or more user interface elements; identifying an active user interface element having focus on the primary display among the one or more user interface elements; determining whether the active user interface element having focus on the primary display is associated with an application executed by the computing system; and in accordance with the determination that the active user interface element having focus on the primary display is associated with an application executed by the computing system, displaying a second user interface on the touch - screen display, the second user interface including (A) a first set of one or more affordances corresponding to the application, and (B) at least one system - level affordance corresponding to at least one system - level function.

[0007] Displaying application-specific system-level affordances in a touch-sensitive secondary display in response to a focus change made on a primary display provides the user with accessible affordances that are directly available via the touch-sensitive secondary display. Providing the user with accessible affordances that are directly accessible via the touch-sensitive secondary display improves the operability of the computing system (e.g., by helping the user directly access the functions needed through the touch-sensitive secondary display with fewer bi-directional actions and without having to waste time exploring hierarchical menus to find the functions needed), makes the user device interface more efficient, and thereby reduces the power consumption of the device and improves the battery life by enabling the user to more quickly and efficiently access the functions needed. Similarly, the display of application-specific affordances on the touch-sensitive secondary display indicates the internal state of the device by providing affordances associated with the application currently in focus on the primary display.

[0008] (A2) In some embodiments of the method of A1, the computing system further comprises (i) a primary computing device comprising a primary display, a processor, a memory, and a primary computing device communication circuit, and (ii) an input device comprising a housing, a touch screen display, a physical input mechanism, and an input device communication circuit that communicates with the primary computing device communication circuit, the input device being separate and distinct from the primary computing device.

[0009] (A3) In some embodiments of any one of the methods of A1 to A2, the physical input mechanism comprises a plurality of physical keys.

[0010] In some embodiments of any one of the methods of A1 to A3, the physical input mechanism comprises a touch pad.

[0011] (A5)In some embodiments of any one of the methods of A1 to A4, the application is executed by a processor in the foreground of the first user interface.

[0012] (A6)In some embodiments of any one of the methods of A1 to A5, at least one system-level affordance is configured to cause a display of a plurality of system-level affordances corresponding to system-level functions on a touch screen display when selected.

[0013] (A7)In some embodiments of at least one of the methods of A1 to A3, at least one system-level affordance corresponds to one of power control or escape control.

[0014] (A8)In some embodiments of any one of the methods of A1 to A7, at least one of the affordances displayed on the second user interface is a multi-functional affordance.

[0015] (A9)In some embodiments of the method of A8, the method further includes detecting a user touch input selecting the multi-functional affordance, executing a first function associated with the multi-functional affordance according to a determination that the user touch input corresponds to a first type, and implementing a second function associated with the multi-functional affordance according to a determination that the user touch input corresponds to a second type distinct from the first type.

[0016] (A10)In some embodiments of any one of A1 to A9's methods, the method further includes displaying a third user interface on a touch screen display that includes a second set of one or more affordances corresponding to operating system controls of a computing system, the second set of one or more affordances being distinct from the first set of one or more affordances, according to a determination that the active user interface element is not associated with an application executed by the computing system.

[0017] (A11)In some embodiments of the method of A10, the second set of one or more affordances is an extended set of operating system controls that includes at least one system-level affordance corresponding to at least one system-level function (B).

[0018] (A12)In some embodiments of any one of A1 to A11's methods, the method further includes detecting a user touch input that selects one of the first set of affordances, in response to detecting the user touch input, displaying a different set of affordances corresponding to the function of the application, and maintaining the display of at least one system-level affordance.

[0019] (A13)In some embodiments of the method of A12, the method further includes detecting a subsequent user touch input that selects at least one system-level affordance, and in response to detecting the subsequent user touch input, displaying a plurality of system-level affordances corresponding to system-level functions and at least one application-level affordance corresponding to the application.

[0020] (A14)In some embodiments of any one of the methods A1 - A13, after displaying a second user interface on the touch screen display, among one or more user interface elements, identifying a second active user interface element that has focus on the primary display, determining whether the second active user interface element corresponds to different applications executed by the computing device, and according to the determination that the second active user interface element corresponds to different applications, displaying on the touch screen display a fourth user interface including (D) a third set of one or more affordances corresponding to different applications, and (E) at least one system - level affordance corresponding to at least one system - level function.

[0021] (A15)In some embodiments of any one of the methods A1 - A14, after identifying the second active user interface element, determining whether a media item is being played by the computing system, where the media item is not associated with different applications, and according to the determination that the media item is being played by the computing system, further including displaying at least one persistent affordance on the touch screen display to control the media item.

[0022] (A16)In some embodiments of the method of A15, the at least one persistent affordance displays feedback corresponding to the media item.

[0023] (A17)In some embodiments of any one of the methods A1 to A16, the method includes detecting a user input corresponding to an override key, and in response to detecting the user input, stopping displaying at least a first set of one or more affordances of a second user interface on the touch screen display, and displaying a first set of default function keys.

[0024] (A18)In some embodiments of the method of A17, the method further includes, after displaying the first set of default function keys, detecting a swipe gesture on the touch screen display in a direction substantially parallel to the major axis of the touch screen display, and in response to detecting the swipe gesture, displaying a second set of default function keys having at least one distinct function key.

[0025] (A19)In another aspect, a computing system is provided, the computing system including one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch-sensitive secondary display. One or more programs are stored in the memory and configured for execution by the one or more processors, the one or more programs including instructions to perform or cause to be performed any one of the methods A1 to A18.

[0026] (A20)In an additional aspect, a non-transitory computer-readable storage medium storing one or more programs is provided, the one or more programs including instructions to cause a computing system having a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch-sensitive secondary display separate from the primary display to perform or cause to be performed any one of the methods A1 to A18 when executed by one or more processors of the computing system.

[0027] (A21) In a further aspect, a graphical user interface on a computing system having one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a touch-sensitive secondary display separate from the physical input mechanism and the primary display, the graphical user interface comprising a user interface displayed according to any of the methods of claims A1 - A18.

[0028] (A22) In another aspect, a computing device is provided. The computing device includes a first housing including a primary display, a second housing at least partially including a touch-sensitive secondary display separate from the physical keyboard and the primary display, and means for performing or causing the performance of any of the methods of claims A1 - A18.

[0029] (B1)According to some embodiments, an input device is provided. The input device includes a housing that at least partially encloses a plurality of components, the plurality of components including: (i) a plurality of physical keys (e.g., keyboard 106 in FIG. 1A) that include at least a separate key for each letter of the alphabet; (ii) a touch-sensitive secondary display (also referred to as a "touch screen display") disposed adjacent to the plurality of physical keys; and (iii) a short-range communication circuit configured to communicate with a computing device (e.g., computing system 100 or 200) disposed adjacent to the input device. The computing device includes a computing device display, a processor, and a memory. The short-range communication circuit is configured to transmit a key press of any of the plurality of physical keys and a touch input on the touch screen display to the computing device and to receive an instruction to change the display of affordances on the touch screen display based on the current focus on the computing device display. In some embodiments, when an application has focus on the computing device display, the touch screen display is configured to display: (A) one or more affordances corresponding to the application that has focus; and (B) at least one system-level affordance, where the at least one system-level affordance is configured to cause the display of a plurality of affordances corresponding to system-level functions when selected.

[0030] Displaying application-specific system-level affordances in a touch-sensitive secondary display in response to a focus change made on a primary display provides the user with accessible affordances that are directly available via the touch-sensitive secondary display. Providing the user with accessible affordances that are directly accessible via the touch-sensitive secondary display improves the operability of the computing system (e.g., by helping the user directly access the functions needed through the touch-sensitive secondary display with fewer bi-directional actions and without having to waste time exploring hierarchical menus to find the functions needed), makes the user device interface more efficient, and thereby reduces the power consumption of the device and improves battery life by enabling the user to more quickly and efficiently access the functions needed. Further, by dynamically updating the affordances displayed in the touch-sensitive secondary display based on a focus change in the primary display, the touch-sensitive secondary display can make more information available on a limited screen and helps ensure that the user is provided with those options when desired options are needed (thereby reducing power consumption and extending battery life as the user does not have to search through hierarchical menus for these desired options that are presented, wasting power and battery life).

[0031] (B2)In some embodiments of the input device of B1, when an application has focus on the computing device display, the touch screen display is further configured to display at least one of a power control affordance and an escape affordance.

[0032] (B3)In some embodiments of the input device of any one of B1 to B2, the input device is a keyboard.

[0033] (B4) In some embodiments of the input device of B3, the computing device is a laptop computer including a keyboard.

[0034] (B5) In some embodiments of the input device of B3, the computing device is a desktop computer, and the keyboard is separate from the desktop computer.

[0035] (B6) In some embodiments of any one of the input devices of B1 - B5, the input device is integrated in a laptop computer.

[0036] (B7) In some embodiments of any one of the input devices of B1 - B6, a plurality of physical keys comprise a QWERTY keyboard.

[0037] (B8) In some embodiments of any one of the input devices of B1 - B7, the alphabet corresponds to the Latin alphabet.

[0038] (B9) In some embodiments of any one of the input devices of B1 - B8, the input device includes a touchpad.

[0039] (B10) In some embodiments of any one of the input devices of B1 - B9, the input device has a longitudinal dimension with a length of at least 18 inches (45.72 cm).

[0040] (B11) In some embodiments of any one of the input devices of B1 - B10, the short - range communication circuit is configured to communicate with the computing device within less than 15 feet (4.572 m).

[0041] (B12) In some embodiments of any one of the input devices of B1 - B11, the short - range communication circuit corresponds to a wired or wireless connection to the computing device.

[0042] In some embodiments of any one of the input devices (B13) B1 to B12, the input device includes a fingerprint sensor embedded in a touch screen display.

[0043] According to some embodiments, in a computing system (e.g., system 100 or system 200, FIGS. 1A-2D) including one or more processors, a memory, a first housing including a primary display (e.g., housing 110 including display 102 or housing 204 including display 102), and a second housing at least partially including a physical keyboard (e.g., keyboard 106, FIG. 1A) and a touch-sensitive secondary display (e.g., dynamic function row 104, FIG. 1A, also referred to as a "touch screen display"), a method is performed. In some embodiments, the touch-sensitive secondary display is separate from the physical keyboard (e.g., the touch-sensitive secondary display is a stand-alone display 222 or the touch-sensitive display is integrated with another device such as touch pad 108, FIG. 2C).This method includes displaying a first user interface for an application to be executed by a computing system on a primary display, and displaying a second user interface on a touch screen display, where the second user interface includes a first set of one or more affordances corresponding to the application, and the first set of one or more affordances corresponds to a first portion of the application, detecting a swipe gesture on the touch screen display, and in accordance with a determination that the swipe gesture is executed in a first direction, displaying a second set of one or more affordances corresponding to the application on the touch screen display, where at least one affordance in the second set of one or more affordances is distinct from the first set of one or more affordances, and the second set of one or more affordances also corresponds to the first portion of the application, and in accordance with a determination that the swipe gesture is executed in a second direction that is substantially perpendicular to the first direction, displaying a third set of one or more affordances corresponding to the application on the touch screen display, where the third set of one or more affordances is distinct from the second set of one or more affordances, and the third set of one or more affordances corresponds to a second portion of the application that is distinct from the first portion of the application.

[0044] Enabling a user to quickly navigate application-specific affordances in a touch-sensitive secondary display in response to a swipe gesture provides the user with a convenient way to scroll through and quickly find desired functions via the touch-sensitive secondary display. Providing the user with a convenient way to scroll through and quickly find desired functions via the touch-sensitive secondary display improves the operability of the computing system (e.g., by helping the user directly access the necessary functions through the touch-sensitive secondary display with fewer bidirectional actions and without wasting time exploring hierarchical menus to find the necessary functions), makes the user device interface more efficient, and thereby further enables the user to more quickly and efficiently access the necessary functions, reducing the power consumption of the device and improving battery life. Further, by dynamically updating the affordances displayed in the touch-sensitive secondary display in response to a swipe gesture in the secondary display, the secondary display can make more information available on a limited screen and help ensure that the user is provided with those options when desired (thereby reducing power consumption and extending battery life as the user does not need to search through hierarchical menus for these desired options and waste power and battery life).

[0045] (C2)In some embodiments of the method of C1, the second portion is displayed on the primary display in a compact view within the first user interface prior to detecting the swipe gesture, and the method includes displaying the second portion on the primary display in an expanded view within the first user interface according to a determination that the swipe gesture was performed in a second direction that is substantially perpendicular to the first direction.

[0046] (C3)In some embodiments of the method of C1, a first user interface for an application executed by a computing system is displayed on a primary display in full-screen mode, and a first set of one or more affordances displayed on a touch screen display includes controls corresponding to the full-screen mode.

[0047] (C4)In some embodiments of any one of the methods of C1 to C3, a second set of one or more affordances and a third set of one or more affordances include at least one system-level affordance corresponding to at least one system-level function.

[0048] (C5)In some embodiments of any one of the methods of C1 to C4, the method includes detecting user input selecting a first portion on the first user interface after displaying a third set of one or more affordances on the touch screen display, and in response to detecting the user input, aborting the display of the third set of one or more affordances on the touch screen display, wherein the third set of one or more affordances corresponds to a second portion of the application, and displaying a second set of one or more affordances, wherein the second set of one or more affordances corresponds to the first portion of the application.

[0049] (C6)In some embodiments of any one of the methods of C1 to C5, the first direction is substantially parallel to the long dimension of the touch screen display.

[0050] (C7)In some embodiments of any one of the methods of C1 to C5, the first direction is substantially perpendicular to the long dimension of the touch screen display.

[0051] In some embodiments of any one of the methods of C1 - C7, the first portion is one of an application menu, tab, folder, toolset, or toolbar, and the second portion is one of an application menu, tab, folder, toolset, or toolbar.

[0052] (C9) In another aspect, a computing system is provided, the computing system including one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch - sensitive secondary display. One or more programs are stored in the memory and configured for execution by one or more processors, the one or more programs including instructions to perform or cause to be performed any one of the methods of C1 - C8.

[0053] (C10) In an additional aspect, a non - transitory computer - readable storage medium storing one or more programs is provided, the one or more programs including instructions to cause a computing system having a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch - sensitive secondary display separate from the primary display to perform or cause to be performed any one of the methods of C1 - C8 when executed by one or more processors of the computing system.

[0054] (C11) In a further aspect, a graphical user interface on a computing system having one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a physical input mechanism and a touch - sensitive secondary display separate from the primary display, the graphical user interface comprising a user interface displayed according to any one of the methods of claims C1 - C8.

[0055] In another aspect, a computing device is provided. The computing device includes a first housing including a primary display, a second housing at least partially including a physical keyboard and a touch-sensitive secondary display separate from the primary display, and means for performing or causing to be performed any of the methods of claims C1-C8.

[0056] (D1) According to some embodiments, in a computing system (e.g., system 100 or system 200, FIGS. 1A - 2D) including one or more processors, a memory, a first housing including a primary display (e.g., housing 110 including display 102 or housing 204 including display 102), and a second housing at least partially including a physical keyboard (e.g., keyboard 106, FIG. 1A) and a touch - sensitive secondary display (e.g., dynamic function row 104, FIG. 1A, also referred to as a "touch - screen display"), a method for maintaining the functionality of an application while in full - screen mode is performed. In some embodiments, the touch - sensitive secondary display is separate from the physical keyboard (e.g., the touch - sensitive secondary display is a stand - alone display 222 or the touch - sensitive display is integrated with another device such as touch pad 108, FIG. 2C). The method includes displaying, on the primary display in normal mode, a first user interface for an application executed by the computing system, the first user interface comprising a first set of one or more affordances associated with the application; detecting a user input to display at least a portion of the first user interface for the application on the primary display in full - screen mode; in response to detecting the user input, ceasing to display the first set of one or more affordances associated with the application in the first user interface on the primary display; displaying, on the primary display in full - screen mode, a portion of the first user interface for the application; and automatically, without human intervention, displaying, on the touch - screen display, a second set of one or more affordances for controlling the application, the second set of one or more affordances corresponding to the first set of one or more affordances.

[0057] While the application portion is being displayed in full-screen mode on the primary display, providing an affordance to control the application via a touch-sensitive secondary display enables the user to continue accessing features that may no longer be directly displayed on the primary display. Enabling the user to continue accessing features that may no longer be directly displayed on the primary display provides the user with a quick and convenient way to access features that may have become buried on the primary display, thereby (for example, by helping the user directly access the necessary features through the touch-sensitive secondary display with fewer bi-directional actions and without having to waste time exploring hierarchical menus to find the necessary features) improving the operability of the computing system, making the user device interface more efficient, and thereby further reducing the power usage of the device and improving battery life by enabling the user to access the necessary features more quickly and efficiently. Thus, to ensure that content can be presented in full-screen mode (without obstructions), by shifting menu options from the primary display to the touch-sensitive secondary display, the user can continue to have a bi-directional interaction with the device and their workflow is not interrupted when shifting to full-screen mode. Further, menu options that may have become buried behind content on the primary display are presented on the touch-sensitive secondary display for easy and quick access (and without having to exit full-screen mode and then search around for the menu options), so that while browsing full-screen content, fewer bi-directional actions are required to access the menu options, thereby reducing the power usage of the device and improving battery life.

[0058] In some embodiments of the method of (D2) D1, the second set of one or more affordances is the first set of one or more affordances.

[0059] In some embodiments of the method of any one of (D3) D1 - D2, the second set of one or more affordances includes controls corresponding to full - screen mode.

[0060] In some embodiments of the method of any one of (D4) D1 - D3, the method includes detecting a user touch input that selects one of a second set of affordances displayed on a touch - screen display, and in response to detecting the user touch input, modifying a portion of a first user interface for an application being displayed in full - screen mode on a primary display according to the selected one of the second set of affordances.

[0061] In some embodiments of the method of any one of (D5) D1 - D4, the method includes detecting a subsequent user input to exit full - screen mode after displaying a portion of a first user interface for an application in full - screen mode on a primary display, and in response to detecting the subsequent user input, displaying on the primary display in normal mode a first user interface for an application executed by a computing system, the first user interface comprising a first set of one or more affordances associated with the application, and maintaining a display of at least a subset of a second set of one or more affordances for controlling the application on a touch - screen display, the second set of one or more affordances corresponding to the first set of one or more affordances.

[0062] In some embodiments of any one of the methods D1 - D5, user input that displays at least a portion of a first user interface for an application in full - screen mode on a primary display is at least one of a touch input detected on a touch - screen display and at least one of the controls selected within the first user interface on the primary display.

[0063] In some embodiments of any one of the methods D1 - D6, a second set of one or more affordances includes at least one system - level affordance corresponding to at least one system - level function.

[0064] In another aspect, a computing system is provided, the computing system including one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch - sensitive secondary display. One or more programs are stored in the memory and configured for execution by one or more processors, the one or more programs including instructions to perform or cause the performance of any one of the methods D1 - D7.

[0065] In an additional aspect, a non - transitory computer - readable storage medium storing one or more programs is provided, the one or more programs including instructions that, when executed by one or more processors of a computing system having a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch - sensitive secondary display separate from the primary display, cause the computing system to perform or cause the performance of any one of the methods D1 - D7.

[0066] (D10) In a further aspect, a graphical user interface on a computing system having one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a touch-sensitive secondary display separate from the physical input mechanism and the primary display, the graphical user interface comprising a user interface displayed according to any of the methods of claims D1 - D7.

[0067] (D11) In another aspect, a computing device is provided. The computing device includes a first housing including a primary display, a second housing at least partially including a touch-sensitive secondary display separate from the physical keyboard and the primary display, and means for performing or causing the performance of any of the methods of claims D1 - D7.

[0068] (E1) According to some embodiments, in a computing system (e.g., system 100 or system 200, FIGS. 1A - 2D) including one or more processors, a memory, a first housing including a primary display (e.g., housing 110 including display 102 or housing 204 including display 102), and a second housing at least partially including a physical keyboard (e.g., keyboard 106, FIG. 1A) and a touch - sensitive secondary display (e.g., dynamic function row 104, FIG. 1A, also referred to as a "touch - screen display"), a method is performed. In some embodiments, the touch - sensitive secondary display is separate from the physical keyboard (e.g., the touch - sensitive secondary display is a stand - alone display 222 or the touch - sensitive display is integrated with another device such as touch pad 108, FIG. 2C). The method includes displaying a first user interface for an application executed by the computing system on the primary display, displaying a second user interface on the touch - screen display, the second user interface comprising a set of one or more affordances corresponding to the application, detecting a notification, and in response to detecting the notification, simultaneously displaying on the touch - screen display a set of one or more affordances corresponding to the application and at least a portion of the detected notification in the second user interface, wherein the detected notification is not displayed on the primary display.

[0069] Displaying the received notification on the touch-sensitive secondary display enables the user to continue their work on the primary display in an uninterrupted manner and enables them to interact bidirectionally with the received notification via the touch-sensitive secondary display. Enabling the user to continue their work on the primary display in an uninterrupted manner and enabling the user to interact bidirectionally with the received notification via the touch-sensitive secondary display provides the user with a quick and convenient way to consider and interact bidirectionally with the received notification, thereby improving the operability of the computing system (e.g., by helping the user to conveniently access the received notification directly through the touch-sensitive secondary display and without the need to interrupt their workflow to handle the received notification) and making the user device interface more efficient. Further, displaying the received notification on the touch-sensitive secondary display provides an emphasis effect for the received notification in the touch-sensitive secondary display since, in some embodiments, the received notification is displayed so as to be superimposed on other affordances in the touch-sensitive secondary display, thus ensuring that the received notification is visible and easily accessible in the touch-sensitive secondary display.

[0070] (E2) In some embodiments of the method of E1, the method includes detecting user input selecting a notification setting to display the notification on the touch screen display but not on the primary display before detecting the notification.

[0071] (E3) In some embodiments of any one of the methods E1 - E2, the method includes detecting a user touch input on a touch screen display corresponding to a portion of a detected notification, and in accordance with a determination that the user touch input corresponds to a first type, ceasing to display the portion of the detected notification on the touch screen display in a second user interface, and in accordance with a determination that the user touch input corresponds to a second type distinct from the first type, performing an action associated with the detected notification.

[0072] (E4) In some embodiments of any one of the methods E1 - E3, the portion of the notification displayed on the touch screen display prompts a user of the computing system to select one of a plurality of options in response to the detected notification.

[0073] (E5) In some embodiments of any one of the methods E1 - E4, the portion of the notification displayed on the touch screen display includes one or more proposed responses to the detected notification.

[0074] (E6) In some embodiments of any one of the methods E1 - E5, the notification corresponds to at least one of an incoming instant message, SMS, email, voice call, or video call.

[0075] (E6) In some embodiments of any one of the methods E1 - E5, the notification corresponds to a mode alert issued by an application being executed by a processor of the computing system in response to the user input closing an application or performing an action within the application.

[0076] In some embodiments of any one of the methods (E7) E1 - E7, a set of one or more affordances includes at least one system - level affordance corresponding to at least one system - level function, and the notification corresponds to one or more parts of the input mechanism or user input selecting at least one of the system - level affordances.

[0077] (E8) In another aspect, a computing system is provided, the computing system including one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch - sensitive secondary display. One or more programs are stored in the memory and configured for execution by one or more processors, and the one or more programs include instructions to perform or cause the performance of any one of the methods E1 - E7.

[0078] (E9) In an additional aspect, a non - transitory computer - readable storage medium storing one or more programs is provided, the one or more programs including instructions that, when executed by one or more processors of a computing system having a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch - sensitive secondary display separate from the primary display, cause the computing system to perform or cause the performance of any one of the methods E1 - E7.

[0079] (E10) In a further aspect, a graphical user interface on a computing system having one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a physical input mechanism and a touch - sensitive secondary display separate from the primary display, the graphical user interface comprising a user interface displayed according to any one of the methods of claims E1 - E7.

[0080] In another aspect, a computing device is provided. The computing device includes a first housing including a primary display, a second housing at least partially including a physical keyboard and a touch-sensitive secondary display separate from the primary display, and means for performing or causing the performance of any of the methods of claims E1-E7.

[0081] (F1)According to some embodiments, in a computing system (e.g., system 100 or system 200, FIGS. 1A - 2D) including one or more processors, a memory, a first housing including a primary display (e.g., housing 110 including display 102 or housing 204 including display 102), and a second housing at least partially including a physical keyboard (e.g., keyboard 106, FIG. 1A) and a touch - sensitive secondary display (e.g., dynamic function row 104, FIG. 1A), a method of moving a user interface portion is performed. In some embodiments, the touch - sensitive secondary display is separate from the physical keyboard (e.g., the touch - sensitive secondary display is a stand - alone display 222 or the touch - sensitive display is integrated with another device such as touch pad 108, FIG. 2C). The method includes displaying a user interface on the primary display, where the user interface comprises one or more user interface elements; identifying an active user interface element among one or more user interface elements having focus on the primary display, where the active user interface element is associated with an application executed by the computing system; in response to identifying the active user interface element, displaying a set of one or more affordances corresponding to the application on the touch - screen display; detecting user input to move each part of the user interface; in response to detecting the user input and according to a determination that the user input meets a predefined action criterion, stopping displaying each part of the user interface on the primary display; stopping displaying at least a subset of the set of one or more affordances on the touch - screen display; and displaying a representation of each part of the user interface on the touch - screen display.

[0082] Enabling a user to quickly move a user interface portion (e.g., menu, notification, etc.) from a primary display to a touch-sensitive secondary display provides the user with a convenient and customized way to access the user interface portion. Providing the user with a convenient and customized way to access the user interface portion via the touch-sensitive secondary display (e.g., by helping the user directly access the user interface portion through the touch-sensitive secondary display with fewer bidirectional actions and without wasting time searching for previously viewed (and possibly buried) user interface portions) improves the operability of the computing system, makes the user device interface more efficient, and thereby further enables the user to more quickly and efficiently access the required user interface portion, reducing the power consumption of the device and improving the battery life. Further, displaying the user interface portion on the touch-sensitive secondary display in response to user input provides an emphasis effect on the user interface portion in the touch-sensitive secondary display, since in some embodiments each user interface portion is displayed as overlapping other affordances in the touch-sensitive secondary display, thus ensuring that each user interface portion is visible and easily accessible in the touch-sensitive secondary display.

[0083] (F2) In some embodiments of the method of F1, each portion of the user interface is a menu corresponding to an application executed by the computing system.

[0084] (F3) In some embodiments of any one of the methods of F1 to F2, each portion of the user interface is one of a notification and a mode alert.

[0085] (F4)In some embodiments of any one of F1 to F3 methods, the default action criterion is satisfied when the user input is a dragging gesture that drags each part of the user interface to a default location on the primary display.

[0086] (F5)In some embodiments of any one of F1 to F3 methods, the default action criterion is satisfied when the user input is a predetermined input corresponding to moving each part of the user interface to the touch screen display.

[0087] (F6)In some embodiments of any one of F1 to F5 methods, the method includes maintaining the display of each part of the user interface on the primary display in response to detecting the user input and according to the determination that the user input does not meet the default action criterion, and maintaining the display of a set of one or more affordances on the touch screen display.

[0088] (F7)In some embodiments of any one of F1 to F6 methods, the set of one or more affordances includes at least one system-level affordance corresponding to at least one system-level function, and the method includes maintaining the display of at least one system-level affordance on the touch screen display after displaying the representation of each part of the user interface on the touch screen display.

[0089] (F8)In some embodiments of any one of F1 to F7 methods, the representation of each part of the user interface is overlaid on a set of one or more affordances on the touch screen display.

[0090] (F9)In another aspect, a computing system is provided, the computing system including one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch-sensitive secondary display. One or more programs are stored in the memory and configured for execution by the one or more processors, the one or more programs including instructions to perform or cause to be performed any one of the methods of F1 - F8.

[0091] (F10)In an additional aspect, a non-transitory computer-readable storage medium storing one or more programs is provided, the one or more programs including instructions that, when executed by one or more processors of a computing system having a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch-sensitive secondary display separate from the primary display, cause the computing system to perform or cause to be performed any one of the methods of F1 - F8.

[0092] (F11)In a further aspect, a graphical user interface on a computing system having one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a physical input mechanism and a touch-sensitive secondary display separate from the primary display, the graphical user interface comprising a user interface displayed according to any of the methods of claims F1 - F8.

[0093] (F12)In another aspect, a computing device is provided. The computing device includes a first housing including a primary display, a second housing at least partially including a physical keyboard and a touch-sensitive secondary display separate from the primary display, and means for performing or causing to be performed any of the methods of claims F1 - F8.

[0094] According to some embodiments, in a computing system (e.g., system 100 or system 200, FIGS. 1A - 2D) including one or more processors, a memory, a first housing including a primary display (e.g., housing 110 including display 102 or housing 204 including display 102), and a second housing at least partially including a physical keyboard (e.g., keyboard 106, FIG. 1A) and a touch - sensitive secondary display (e.g., dynamic function row 104, FIG. 1A), a method is performed. In some embodiments, the touch - sensitive secondary display is separate from the physical keyboard (e.g., the touch - sensitive secondary display is a stand - alone display 222, or the touch - sensitive display is integrated with another device such as touch pad 108, FIG. 2C). The method includes receiving a request to open an application. In response to receiving the request, the method includes (i) displaying, on the primary display, a plurality of user interface objects associated with an application running on the computing system (e.g., the plurality of user interface objects correspond to tabs in Safari, individual photos in a photo - browsing application, individual frames of a video in a video - editing application, etc.), the plurality of user interface objects including a first user interface object displayed with associated content and other user interface objects displayed without associated content, and (ii) displaying, on the touch - sensitive secondary display, a set of affordances each representing (i.e., corresponding to) one of the plurality of user interface objects. The method also includes detecting a swipe gesture via the touch - sensitive secondary display in a direction from a first affordance of the set of affordances to a second affordance of the set of affordances.In some embodiments, the first affordance represents a first user interface object and the second affordance represents a second user interface object that is distinct from the first user interface object. In response to detecting a swipe gesture, the method includes updating a primary display (e.g., during the swipe gesture) to cease displaying content associated with the first user interface object and to display content associated with the second user interface object.

[0095] Enabling a user to quickly navigate user interface objects on a primary display (e.g., a browser tab) by providing an input on a touch-sensitive secondary display provides the user with a convenient way to quickly navigate user interface objects. Providing the user with a convenient way to quickly navigate user interface objects via a touch-sensitive secondary display (and reducing the number of inputs required to navigate the user interface objects, and thus, reducing the number of bi-directional actions required to navigate the user interface objects) improves the operability of the computing system, makes the user device interface more efficient, and thereby, further enables the user to more quickly and efficiently navigate user interface objects on the primary display, reducing the power consumption of the device and improving the battery life. Additionally, when the user provides an input (e.g., a swipe gesture) on the touch-sensitive display to navigate a user interface object on the primary display, each touch affordance on the touch-sensitive display (corresponding to one of the user interface objects) is visually distinguishable from the other affordances (e.g., each touch affordance may be enlarged and its edges highlighted), thus making the information displayed on the touch-sensitive secondary display more distinguishable to the user.

[0096] (G2) In some embodiments of the method of G1, the method includes detecting a continuous movement of a swipe gesture across a touch-sensitive secondary display that includes a swipe gesture in contact with a third affordance representing a third user interface object. In response to detecting that the swipe gesture has contacted the third affordance, the method includes updating the primary display to display the relevant content of the third user interface object.

[0097] (G3) In some embodiments of any one of the methods of G1 - G2, each affordance in a set of affordances includes a representation of the respective relevant content for each of a plurality of user interface objects.

[0098] (G4) In some embodiments of any one of the methods of G1 - G3, the method includes detecting an initial contact with the touch-sensitive secondary display on a first affordance before detecting the swipe gesture. In response to detecting the initial contact, the method includes increasing the magnification level (or display size) of the first affordance.

[0099] (G5) In some embodiments of any one of the methods of G1 - G4, the application is a web browsing application, and the plurality of user interface objects each correspond to a web browsing tab.

[0100] (G6) In some embodiments of the method of G6, the method includes detecting an input in the URL input portion of a web browsing application on the primary display. In response to detecting the input, the method includes updating the touch-sensitive secondary display to include a representation of a favorite URL.

[0101] In some embodiments of any one of methods G1 - G4, the application is a photo browsing application, and the plurality of user interface objects respectively correspond to individual photos.

[0102] In some embodiments of any one of methods G1 - G4, the application is a video editing application, and the plurality of user interface objects respectively correspond to individual frames in respective videos.

[0103] In another aspect, a computing system is provided, the computing system includes one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch - sensitive secondary display. One or more programs are stored in the memory and configured for execution by one or more processors, and the one or more programs include instructions to perform or cause to perform any one of methods G1 - G8.

[0104] In an additional aspect, a non - transitory computer - readable storage medium storing one or more programs is provided, the one or more programs, when executed by one or more processors of a computing system having a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch - sensitive secondary display separate from the primary display, include instructions to cause the computing system to perform or cause to perform any one of methods G1 - G8.

[0105] (G11) In yet another aspect, a graphical user interface on a computing system having one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a touch-sensitive secondary display separate from the physical input mechanism and the primary display, the graphical user interface comprising a user interface displayed according to any of the methods of claims G1 - G8.

[0106] (G12) In another aspect, a computing device is provided. The computing device includes a first housing including a primary display, a second housing at least partially including a touch-sensitive secondary display separate from the physical keyboard and the primary display, and means for performing or causing the performance of any of the methods of claims G1 - G8.

[0107] (H1)According to some embodiments, in a computing system (e.g., system 100 or system 200, FIGS. 1A - 2D) including one or more processors, a memory, a first housing including a primary display (e.g., housing 110 including display 102 or housing 204 including display 102), and a second housing at least partially including a physical keyboard (e.g., keyboard 106, FIG. 1A) and a touch - sensitive secondary display (e.g., dynamic function row 104, FIG. 1A), a method is performed. In some embodiments, the touch - sensitive secondary display is separate from the physical keyboard (e.g., the touch - sensitive secondary display is a stand - alone display 222 or the touch - sensitive display is integrated with another device such as touch pad 108, FIG. 2C). The method includes receiving a request to search within content displayed on a primary display of a computing device (e.g., the request corresponds to a request to search for text within the displayed web page content). In response to receiving the request, the method includes displaying, on the primary display, a plurality of search results responsive to the search, with the focus being on a first search result among the plurality of search results, and (ii) displaying, on the touch - sensitive secondary display, respective representations corresponding to each of the search results among the plurality of search results. The method also includes detecting, via the touch - sensitive secondary display, a touch input (e.g., a tap or a swipe) selecting a representation among the respective representations, the representation corresponding to a second search result among a plurality of search results separate from the first search result. In response to detecting the input, the method includes changing the focus on the primary display to the second search result.

[0108] Enabling a user to quickly navigate search results on a primary display by providing input on a touch - sensitive secondary display provides the user with a convenient way to quickly navigate search results. Providing the user with a convenient way to quickly navigate search results via a touch - sensitive secondary display (and reducing the number of inputs required to navigate search results, and thus, reducing the amount of two - way interaction from the user required to quickly browse a large number of search results) improves the operability of the computing system (by, for example, requiring a single input or gesture on the touch - sensitive secondary display to navigate a large number of search results on the primary display), makes the user device interface more efficient, and thereby, further enables the user to more quickly and efficiently navigate search results on the primary display, reducing the power consumption of the device and improving battery life. Additionally, when the user provides an input (e.g., a swipe gesture) on the touch - sensitive display to navigate a search on the primary display, each touch affordance on the touch - sensitive display (corresponding to one of the search results) is visually distinguishable from the other affordances (e.g., each touch affordance may be enlarged and its edges highlighted), thus making the information displayed on the touch - sensitive secondary display more distinguishable to the user.

[0109] (H2)In some embodiments of the method of H1, changing the focus includes modifying the visual characteristics of a particular search result (e.g., displaying a particular search result in a larger font size) on the primary display.

[0110] In some embodiments of any one of the methods H1 to H2, the method includes detecting a gesture that moves across at least two of the respective representations on a touch-sensitive secondary display. In response to detecting the gesture, the method includes changing the focus on the primary display to the respective search results corresponding to at least two of the respective representations when the swipe gesture moves across at least two of the respective representations.

[0111] In some embodiments of the method of H3, the method includes changing the focus on the primary display to the respective search results corresponding to at least two of the respective representations, in addition to those corresponding to at least two of the respective representations, according to a determination that the speed of the gesture exceeds a threshold speed (e.g., when exceeding the threshold speed, in addition to what was touched during the swipe, more search results are cycled through).

[0112] In some embodiments of any one of the methods H3 to H4, the gesture is a swipe gesture.

[0113] In some embodiments of any one of the methods H3 to H4, the gesture is a flick gesture.

[0114] In some embodiments of any one of the methods H1 to H6, the representation is a tick mark corresponding to each of the respective search results among the search results.

[0115] In some embodiments of the method of H7, the tick marks are displayed in a row on the touch-sensitive secondary display in an order corresponding to the ordering of the search results on the primary display.

[0116] (H9)In some embodiments of any one of the methods H1 - H8, the request to search within the content is a request to locate the search string within the content, and the plurality of search results each contain at least the search string.

[0117] (H9)In some embodiments of the method H8, displaying the plurality of search results includes highlighting the search string for each of the plurality of search results.

[0118] (H10)In another aspect, a computing system is provided, the computing system including one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch - sensitive secondary display. One or more programs are stored in the memory and configured for execution by one or more processors, the one or more programs including instructions to perform or cause the performance of any one of the methods H1 - H9.

[0119] (H11)In an additional aspect, a non - transitory computer - readable storage medium storing one or more programs is provided, the one or more programs including instructions that, when executed by one or more processors of a computing system having a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch - sensitive secondary display separate from the primary display, cause the computing system to perform or cause the performance of any one of the methods H1 - H9.

[0120] (H12) In yet another aspect, a graphical user interface on a computing system having one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a touch-sensitive secondary display separate from the physical input mechanism and the primary display, the graphical user interface comprising a user interface displayed according to any of the methods of claims H1-H9.

[0121] (H13) In another aspect, a computing device is provided. The computing device includes a first housing including a primary display, a second housing at least partially including a touch-sensitive secondary display separate from the physical keyboard and the primary display, and means for performing or causing the performance of any of the methods of claims H1-H9.

[0122] (I1) According to some embodiments, in a computing system (e.g., system 100 or system 200, FIGS. 1A - 2D) including one or more processors, a memory, a first housing including a primary display (e.g., housing 110 including display 102 or housing 204 including display 102), and a second housing at least partially including a physical keyboard (e.g., keyboard 106, FIG. 1A) and a touch - sensitive secondary display (e.g., dynamic function row 104, FIG. 1A), a method is performed. In some embodiments, the touch - sensitive secondary display is separate from the physical keyboard (e.g., the touch - sensitive secondary display is a stand - alone display 222 or the touch - sensitive display is integrated with another device such as touch pad 108, FIG. 2C). The method includes displaying a calendar application on the primary display. The method also includes receiving a request to display information regarding an event associated with the calendar application (e.g., the request corresponds to the selection of an event displayed within the calendar application on the primary display). In response to receiving the request, the method includes (i) displaying event details of a first event on the primary display, where the event details include the start time and end time of the event, and (ii) displaying an affordance on the touch - sensitive secondary display, where the affordance (e.g., a user interface control) indicates a time range including at least the start time and the end time.

[0123] Enabling a user to quickly and easily edit event details on a touch-sensitive secondary display provides the user with a convenient way to quickly edit event details without having to perform additional input (e.g., without having to jump back and forth between using a keyboard and using a trackpad to modify event details). Providing the user with a convenient way to quickly edit event details via a touch-sensitive secondary display (and reducing the number of inputs required to edit event details, and thus reducing the amount of two-way interaction required to achieve the desired result of editing event details) improves the operability of the computing system (e.g., by requiring a single input or gesture on the touch-sensitive secondary display to quickly edit some event details), makes the user device interface more efficient, and thereby reduces the power consumption of the device and improves battery life by enabling the user to more quickly and efficiently edit event details. Further, by updating the primary display in response to an input on the touch-sensitive secondary display (e.g., to show updated start and end times of an event), the user can provide an input to modify the event and then immediately see those modifications reflected on the primary display, thus enabling the user to then determine whether additional inputs should be provided, making it possible to sustain two-way interaction with the device in an efficient manner.

[0124] (I2) In some embodiments of the method of I1, the method includes detecting an input in a user interface control that modifies a time range via a touch-sensitive secondary display. In response to detecting the input, (i) at least one of a start time and an end time of the event is modified according to the input, and (ii) a modified range of the time of the event is displayed on the primary display according to the input.

[0125] (I3) In some embodiments of the method of I2, the method includes storing an event in the memory of a computing system with a modified start and / or end time.

[0126] (I4) In some embodiments of any one of the methods of I1 - I3, the input for modifying the time range is a press input that maintains contact with an affordance on a touch - sensitive secondary display for a threshold time and then moves at least a portion of the affordance.

[0127] (I5) In some embodiments of any one of the methods of I1 - I3, the input for modifying the time range is a swipe gesture that moves across a touch - sensitive secondary display and causes the computing system to select a new start time and a new end time for an event, where the new start time and end time correspond to a time slot having the same duration as that covered by the start time and end time.

[0128] (I6) In another aspect, a computing system is provided, the computing system including one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch - sensitive secondary display. One or more programs are stored in the memory and configured for execution by one or more processors, and the one or more programs include instructions for performing or causing the performance of any one of the methods of I1 - I5.

[0129] (I7) In an additional aspect, a non-transitory computer-readable storage medium storing one or more programs is provided, the one or more programs including instructions that, when executed by one or more processors of a computing system having a memory, a first housing including a primary display, and a second housing at least partially including a touch-sensitive secondary display separate from a physical keyboard and the primary display, cause the computing to perform or cause the performance of any one of the methods I1 - I5.

[0130] (I8) In a further aspect, a graphical user interface on a computing system having one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a touch-sensitive secondary display separate from a physical input mechanism and the primary display, the graphical user interface comprising a user interface displayed according to any of the methods of claims I1 - I5.

[0131] (I9) In another aspect, a computing device is provided. The computing device includes a first housing including a primary display, a second housing at least partially including a touch-sensitive secondary display separate from a physical keyboard and the primary display, and means for performing or causing the performance of any of the methods of claims I1 - I5.

[0132] (J1)According to some embodiments, in a computing system (e.g., system 100 or system 200, FIGS. 1A-2D) including one or more processors, a memory, a first housing including a primary display (e.g., housing 110 including display 102 or housing 204 including display 102), and a second housing at least partially including a physical keyboard (e.g., keyboard 106, FIG. 1A) and a touch-sensitive secondary display (e.g., dynamic function row 104, FIG. 1A), a method is performed. In some embodiments, the touch-sensitive secondary display is separate from the physical keyboard (e.g., the touch-sensitive secondary display is a stand-alone display 222 or the touch-sensitive display is integrated with another device such as touch pad 108, FIG. 2C). The method includes detecting a new connection between the computing system and an external device separate from the computing system, and (e.g., connecting a new monitor, connecting headphones via Bluetooth® or a headphone jack, etc.). In response to detecting the new connection, the method includes displaying, on the touch-sensitive secondary display, a plurality of affordances corresponding to functions available via the external device.

[0133] Enabling a user to efficiently utilize an external device via a touch-sensitive secondary display provides the user with a convenient way to access features that may otherwise be buried in a menu. Providing the user with a convenient way to access features of an external device that may otherwise be buried in a menu (and thus reducing the number of inputs required to access the features, thereby reducing the amount of two-way interaction required to use the external device) improves the operability of a computing system (e.g., by requiring a single input or gesture on the touch-sensitive secondary display to perform some functions of the external device), makes the user device interface more efficient, and thereby further enables the user to more quickly and efficiently interact bidirectionally with the external device, reducing the power consumption of the device and improving battery life. In this way, the touch-sensitive secondary display also conveys information about the internal state of the device (by reflecting the connection status between the device and the external device and enabling the user to easily interact bidirectionally with the external device).

[0134] (J2) In some embodiments of the method of J1, the method includes receiving, via a touch-sensitive secondary display, a selection of a first affordance corresponding to a first function available via an external device. In response to receiving the selection, the method includes initiating execution of the first function.

[0135] (J3) In some embodiments of any one of the methods of J1-J2, the external device is an additional display separate from the primary display and the touch-sensitive display.

[0136] In some embodiments of the method of (J4) J3, a plurality of affordances include a first affordance that, when selected, causes the computing system to initiate the execution of a display mirroring function via an additional display.

[0137] In another aspect, a computing system is provided, the computing system including one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch-sensitive secondary display. One or more programs are stored in the memory and configured for execution by one or more processors, the one or more programs including instructions to perform or cause the performance of any one of the methods of J1-J4.

[0138] In an additional aspect, a non-transitory computer-readable storage medium storing one or more programs is provided, the one or more programs including instructions that, when executed by one or more processors of a computing system having a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch-sensitive secondary display separate from the primary display, cause the computing system to perform or cause the performance of any one of the methods of J1-J4.

[0139] In a further aspect, a graphical user interface on a computing system having one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a physical input mechanism and a touch-sensitive secondary display separate from the primary display, the graphical user interface comprising a user interface displayed according to any one of the methods of claims J1-J4.

[0140] (J8)In another aspect, a computing device is provided. The computing device includes a first housing including a primary display, a second housing at least partially including a physical keyboard and a touch-sensitive secondary display separate from the primary display, and means for performing or causing the performance of any of the methods of Claims J1-J4.

[0141] (K1)According to some embodiments, in a computing system (e.g., system 100 or system 200, FIGS. 1A-2D) including one or more processors, a memory, a first housing including a primary display (e.g., housing 110 including display 102 or housing 204 including display 102), and a second housing at least partially including a physical keyboard (e.g., keyboard 106, FIG. 1A) and a touch-sensitive secondary display (e.g., dynamic function row 104, FIG. 1A), a method is performed. In some embodiments, the touch-sensitive secondary display is separate from the physical keyboard (e.g., the touch-sensitive secondary display is a stand-alone display 222 or the touch-sensitive display is integrated with another device such as touch pad 108, FIG. 2C). The method includes displaying a user interface for an application running on the computing system on the primary display and detecting a first input at a particular location within the user interface. In response to detecting the first input, the method includes displaying, on the touch-sensitive secondary display, a set of affordances each corresponding to a distinct character.

[0142] By providing intuitive input on a touch-sensitive secondary display, it is possible for a user to quickly and easily preview how text will appear within an application on a primary display, providing the user with a convenient way to quickly preview how text will appear within the application. Providing the user with a convenient way to quickly preview how text will appear within the application improves the operability of the computing system, makes the user device interface more efficient (e.g., by requiring a single input or gesture on the touch-sensitive secondary display to quickly preview how text will appear, thus requiring less bi-directional action needed to preview how these characters will appear on the primary display), thereby further enabling the user to more quickly and efficiently preview text, reducing the power usage of the device and improving battery life. In this way, the user is allowed to continue to preview text and then continue to provide input until the desired text is found, providing the user with an efficient and sustained bi-directional interaction with their device.

[0143] (K2) In some embodiments of the method of K1, the method includes detecting a second input over a first affordance corresponding to a first character of a separate set of characters via a touch-sensitive secondary display. In response to detecting the second input, the method includes displaying a preview of the first character at a particular location on the primary display while the input remains in contact with the first affordance.

[0144] (K3)In some embodiments of the method of K2, the method includes detecting a movement of a second input from a first affordance to a second affordance corresponding to a second character among separate characters via a touch-sensitive secondary display. In response to detecting the movement of the second input from the first affordance to the second affordance, the method includes replacing a preview of the first character with a preview of the second character.

[0145] (K4)In some embodiments of any one of the methods of K1 - K3, the method includes determining an affordance to include in a set of affordances based at least in part on text content included in a user interface.

[0146] (K5)In some embodiments of the method of K4, the determining is performed in response to detecting that the user has modified text content included in the user interface.

[0147] (K6)In some embodiments of any one of the methods of K1 - K5, the method includes detecting a lift-off of a second input while the second input is in contact with a second affordance. In response to detecting the lift-off, the method includes updating the user interface to include a second user interface element.

[0148] (K7)In some embodiments of any one of the methods of K1 - K5, the method includes detecting an additional input while the second input is in contact with a second affordance. In response to detecting the additional input, the method includes updating the user interface to include a second user interface element.

[0149] (K8)In some embodiments of any one of K1 - K7, the method includes displaying a preview of each of the distinct characters such that a corresponding affordance in a set of affordances is contacted by the second input as the second input continues to move across the touch - sensitive secondary display.

[0150] (K9)In some embodiments of any one of K1 - K8, while the input remains in contact with the second affordance, the preview of the second character remains displayed on the primary display.

[0151] (K10)In another aspect, a computing system is provided, the computing system including one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch - sensitive secondary display. One or more programs are stored in the memory and configured for execution by the one or more processors, the one or more programs including instructions to perform or cause to be performed any one of the methods of K1 - K9.

[0152] (K11)In an additional aspect, a non - transitory computer - readable storage medium storing one or more programs is provided, the one or more programs including instructions that, when executed by one or more processors of a computing system having a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch - sensitive secondary display separate from the primary display, cause the computing system to perform or cause to be performed any one of the methods of K1 - K9.

[0153] (K12)In yet another aspect, a graphical user interface on a computing system having one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a touch-sensitive secondary display separate from the physical input mechanism and the primary display, the graphical user interface comprising a user interface displayed according to any of the methods of claims K1-K9.

[0154] (K13)In another aspect, a computing device is provided. The computing device includes a first housing including a primary display, a second housing at least partially including a touch-sensitive secondary display separate from the physical keyboard and the primary display, and means for performing or causing the performance of any of the methods of claims K1-K9.

[0155] (L1)According to some embodiments, in a computing system (e.g., system 100 or system 200, FIGS. 1A - 2D) including one or more processors, a memory, a first housing including a primary display (e.g., housing 110 including display 102 or housing 204 including display 102), and a second housing at least partially including a physical keyboard (e.g., keyboard 106, FIG. 1A) and a touch - sensitive secondary display (e.g., dynamic function row 104, FIG. 1A), a method is performed. In some embodiments, the touch - sensitive secondary display is separate from the physical keyboard (e.g., the touch - sensitive secondary display is a stand - alone display 222 or the touch - sensitive display is integrated with another device such as touch pad 108, FIG. 2C). The method includes receiving a request to open a content - editing application (an application for creating and editing documents, figures, photos, etc.). In response to receiving the request, the method includes (i) displaying the content - editing application on the primary display and (ii) displaying on the touch - sensitive secondary display a user - interface control (e.g., a color picker including a sliding scale of color values used to select the color of the content displayed within the content - editing application) for modifying at least one visual characteristic used to render the content within the content - editing application.

[0156] Accordingly, the user is provided with an intuitive way to modify the visual characteristics used to render content within a content editing application on the primary display by providing input on the touch-sensitive secondary display. Providing the user with such an intuitive way to modify the visual characteristics (e.g., by requiring a single input or gesture on the touch-sensitive secondary display to quickly preview how some visual characteristics will look when used to render content on the primary display) improves the operability of the computing system, makes the user device interface more efficient, and thereby reduces the power usage of the device and improves battery life by enabling the user to more quickly and efficiently preview changes in visual characteristics. In this way, the user is allowed to continue to preview how modifications to the visual characteristics will look on the primary display and then continue to provide input until the desired modification to the visual characteristics is found, so that the user is provided with an efficient and continuous two-way interaction with their device.

[0157] (L2)In some embodiments of the method of L1, the method includes detecting an input in a user interface control that selects a first value of at least one visual characteristic via the touch-sensitive secondary display. After detecting the input, the method includes rendering the content in the content editing application using the first value of the at least one visual characteristic (e.g., using the first value to render all new content added to the content editing application and / or using the first value to render the currently selected content).

[0158] In some embodiments of any one of the methods of L1 - L2, the user interface control includes respective controls each corresponding to a respective value of at least one visual characteristic along a sliding scale of values.

[0159] In some embodiments of the method of L3, the sliding scale of values represents distinct shades of color.

[0160] In some embodiments of the method of L4, the first value corresponds to a first shade of a first color, and the method includes modifying the user interface control on a touch - sensitive secondary display to include an option to select a different shade of the first color, distinct from the first shade of the first color, according to a determination that the input meets a predetermined criterion (remains in contact for longer than a threshold time).

[0161] In some embodiments of any one of the methods of L1 - L5, the method includes receiving a selection of content before rendering the content, and rendering the content includes presenting a preview of the content using a first value of at least one visual characteristic.

[0162] In some embodiments of the method of L6, the preview is presented while the input remains in contact with the touch - sensitive secondary display.

[0163] In some embodiments of the method of L7, the method includes aborting the display of the preview in response to detecting a lift - off of the input.

[0164] In some embodiments of the method of L8, the method includes displaying an editable portion of the content using a modified value of at least one visual characteristic in response to detecting a lift - off of the input.

[0165] (L10)In another aspect, a computing system is provided, the computing system including one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch-sensitive secondary display. One or more programs are stored in the memory and configured for execution by the one or more processors, the one or more programs including instructions to perform or cause to be performed any one of the methods of L1-L9.

[0166] (L11)In an additional aspect, a non-transitory computer-readable storage medium storing one or more programs is provided, the one or more programs including instructions that, when executed by one or more processors of a computing system having a memory, a first housing including a primary display, and a second housing at least partially including a physical keyboard and a touch-sensitive secondary display separate from the primary display, cause the computing system to perform or cause to be performed any one of the methods of L1-L9.

[0167] (L12)In a further aspect, a graphical user interface on a computing system having one or more processors, a memory, a first housing including a primary display, and a second housing at least partially including a physical input mechanism and a touch-sensitive secondary display separate from the primary display, the graphical user interface comprising a user interface displayed in accordance with any of the methods of claims L1-L9.

[0168] (L13)In another aspect, a computing device is provided. The computing device includes a first housing including a primary display, a second housing at least partially including a physical keyboard and a touch-sensitive secondary display separate from the primary display, and means for performing or causing to be performed any one of the methods of claims L1-L9.

[0169] (M1)According to some embodiments, in an electronic device (e.g., system 100 or system 200, FIGS. 1A - 2D) including one or more processors, a memory, a first housing including a primary display (e.g., the housing 110 including the display 102 or the housing 204 including the display 102, also referred to as a "display"), and a second housing at least partially including a physical keyboard (e.g., keyboard 106, FIG. 1A) and a touch - sensitive secondary display (e.g., dynamic function row 104, FIG. 1A), a method is performed. In some embodiments, the touch - sensitive secondary display is separate from the physical keyboard (e.g., the touch - sensitive secondary display is a stand - alone display 222 or the touch - sensitive display is integrated with another device such as the touch pad 108, FIG. 2C). In some embodiments, the electronic device also includes a biometric sensor that is integrated with the touch - sensitive secondary display or is a separate component disposed adjacent to the touch - sensitive secondary display within the second housing. The method includes displaying, while the device is in a locked state (e.g., the locked state is a state in which one or more functions of the device are disabled and access to confidential information or the ability to modify or delete information is prohibited), respective login user interfaces associated with logging into a plurality of user accounts including a first user account and a second user account. While displaying the login user interfaces, the method includes receiving biometric information about the user.In response to receiving biometric information, the method includes: (i) while a first user account does not have an active session on the device, displaying, on a display, a prompt to enter the login credentials of the first user account according to a determination that the biometric information matches the biometric information of the first user account among a plurality of user accounts; and (ii) while a second user account does not have an active session on the device, displaying, on the display, a prompt to enter the login credentials of the second user account according to a determination that the biometric information matches the biometric information of the second user account among a plurality of user accounts.

[0170] Accordingly, the user is provided with an intuitive way to access a user-specific login page by providing a single input at the biometric sensor. Providing the user with such an intuitive way to access the user-specific login page improves the operability of the computing system (e.g., by requiring a single input or gesture at the biometric sensor for quick access to the user-specific login page as appropriate, and thus fewer bidirectional actions are required to reach the user-specific login page), makes the user device interface more efficient, and thereby reduces the power consumption of the device and improves the battery life by enabling the user to access the login page via a single input.

[0171] (M2) In some embodiments of the method of M1, the method includes, in response to receiving biometric information, unlocking the device with respect to a first user account according to a determination that the biometric information matches the biometric information of the first user account among a plurality of user accounts while the first user account has an active session on the device (e.g., without requiring additional user input).

[0172] (M3)In some embodiments of any one of the methods M1 - M2, the method includes unlocking the device with respect to a second user account (e.g., without requiring additional user input) according to a determination that the biometric information matches the biometric information of the second user account among a plurality of user accounts, in response to receiving the biometric information while the second user account has an active session on the device.

[0173] (M4)In some embodiments of any one of the methods M1 - M3, the method includes maintaining the device in a locked state according to a determination that the biometric information does not match the biometric information of any user account of the device, in response to receiving the biometric information.

[0174] (M5)In some embodiments of any one of the methods M1 - M4, the login user interface includes a plurality of selectable affordances corresponding to a plurality of user accounts.

[0175] (M6)In some embodiments of any one of the methods M1 - M5, the method includes receiving an entry of login credentials while displaying a prompt to enter the login credentials of a first user account. In response to receiving the entry of the login credentials, the method includes (i) unlocking the device with respect to the first user account according to a determination that the login credentials match the login credentials of the first user account, and (ii) maintaining the device in a locked state according to a determination that the login credentials do not match the login credentials of the first user account.

[0176] (M7)In some embodiments of any one of the methods M1 to M6, the method includes receiving an entry of login credentials while displaying a prompt to enter the login credentials of a second user account. In response to receiving the entry of the login credentials, (i) unlocking the device with respect to a first user account according to a determination that the login credentials match the login credentials of the second user account, and (ii) maintaining the device in a locked state according to a determination that the login credentials do not match the login credentials of the second user account.

[0177] (M8)In some embodiments of any one of the methods M1 to M7, the login user interface includes an instruction to provide biometric information.

[0178] (M9)In some embodiments of any one of the methods M1 to M8, the device includes a secondary display (e.g., a touch-sensitive secondary display) adjacent to the biometric sensor, and the method includes displaying, on the secondary display, an instruction to provide biometric information via the biometric sensor while the login user interface is displayed on the device's display.

[0179] (M10)In some embodiments of any one of the methods M1 to M9, the biometric sensor is a fingerprint sensor.

[0180] (M11)In some embodiments of any one of the methods M1 to M9, the biometric sensor is a face detection sensor.

[0181] (M12)In some embodiments of any one of the methods M1 to M9, the biometric sensor is a retina scanner.

[0182] (M13) In another aspect, an electronic device is provided, which includes one or more processors, a memory, a display, and a biosensor. One or more programs are stored in the memory and configured for execution by one or more processors, and the one or more programs include instructions to execute or cause the execution of any one of the methods of M1 - M12.

[0183] (M14) In an additional aspect, a non - transitory computer - readable storage medium storing one or more programs is provided, and the one or more programs include instructions to cause a computing system having a display and a biosensor to execute or cause the execution of any one of the methods of M1 - M12 when executed by one or more processors of the computing system.

[0184] (M15) In a further aspect, a graphical user interface is provided on a computing system having one or more processors, a memory, a display, and a biosensor, and the graphical user interface comprises a user interface that is displayed according to any one of the methods of claims M1 - M12.

[0185] (M16) In another aspect, a computing device is provided. The computing device includes a display and a biosensor and means for executing or causing the execution of any one of the methods of claims M1 - M12.

[0186] According to some embodiments, in an electronic device (e.g., system 100 or system 200, FIGS. 1A - 2D) including one or more processors, a memory, a first housing including a primary display (e.g., housing 110 including display 102 or housing 204 including display 102, also referred to as a "display"), and a second housing at least partially including a physical keyboard (e.g., keyboard 106, FIG. 1A) and a touch - sensitive secondary display (e.g., dynamic function row 104, FIG. 1A), a method is performed. In some embodiments, the touch - sensitive secondary display is separate from the physical keyboard (e.g., the touch - sensitive secondary display is a stand - alone display 222 or the touch - sensitive display is integrated with another device such as touch pad 108, FIG. 2C). In some embodiments, the electronic device also includes a biometric sensor that is integrated with the touch - sensitive secondary display or is a separate component disposed adjacent to the touch - sensitive secondary display within the second housing. The method includes displaying a user interface associated with a first user account (e.g., the home screen or desktop of the first user account) while the device is logged in to the first user account. The device is associated with a plurality of user accounts including the first user account and a second user account, and the second user account is associated with biometric information that enables logging in to the second user account. While displaying the user interface associated with the first user account, the method includes receiving an input via an input element having an integrated biometric sensor.In response to receiving an input via an input element having an integrated biometric sensor, the method determines that the input meets a second user switching criterion while a second user account has an active session on the device, the second user switching criterion including the requirement that biometric information detected during the input using the input element matches the biometric information of the second user account among a plurality of user accounts, and in accordance with the determination, (i) unlocking the device with respect to the second user account, (ii) locking the device with respect to the first user account, and (iii) replacing the display of the user interface associated with the first account with the user interface associated with the second user account.

[0187] Accordingly, the user is provided with an intuitive way to access the active user-specific session on the electronic device by providing a single input at the biometric sensor. Providing the user with such an intuitive way to access the active user-specific session (e.g., by requiring a single input or gesture at the biometric sensor to obtain immediate access to the active session, thus reducing the number of bi-directional actions required to switch user accounts and log in to the device) improves the operability of the electronic device, makes the user device interface more efficient, and thereby further reduces the power consumption of the device and improves the battery life by enabling the user to access the active session via a single input.

[0188] In some embodiments of the method of (N2) N1, in response to receiving biometric information, while a second user account does not have an active session on the device, a determination that the input meets a second user switching criterion, the second user switching criterion including the requirement that biometric information detected during input using an input element matches the biometric information of a second user account among a plurality of user accounts, and in accordance with the determination, displaying, on a display, a prompt to enter the login credentials of the second user account.

[0189] In some embodiments of any one of the methods of (N3) N1 - N2, in response to receiving biometric information, while a third user account has an active session on the device, a determination that the input meets a third user switching criterion, the third user switching criterion including the requirement that biometric information detected during input using an input element matches the biometric information of a third user account among a plurality of user accounts, and in accordance with the determination, (i) unlocking the device with respect to the third user account, (ii) locking the device with respect to the first user account, and (iii) replacing the display of the user interface associated with the first account with the user interface associated with the third user account.

[0190] In some embodiments of any one of the methods of (N4) N1 - N3, in response to receiving biometric information, while a third user account does not have an active session on the device, a determination that the input meets a third user switching criterion, the third user switching criterion including the requirement that biometric information detected during input using an input element matches the biometric information of a third user account among a plurality of user accounts, and in accordance with the determination, displaying, on a display, a prompt to enter the login credentials of the third user account.

[0191] In some embodiments of any one of the methods of N1 to N2, the input element is a button, and the input via the input element with the integrated biosensor includes detecting a pressing input via the button.

[0192] In some embodiments of the method of N5, the second user switching criterion includes the criterion that the pressing input lasts shorter than a first threshold time, and the method includes entering the device into a low power mode (e.g., the low power mode corresponds to a suspended state of an electronic device with the display turned off) in response to receiving a pressing input via the input element with the integrated biosensor according to the determination that the pressing input using the button lasts longer than the first threshold time.

[0193] In some embodiments of the method of N5, the second user switching criterion includes the criterion that the button press is shorter than a first threshold time, and the method includes, in response to receiving an input via the input element with the integrated biosensor: (i) entering the device into a low power mode (e.g., the low power mode corresponds to a sleep / suspend state of an electronic device with the display turned off and enters this low power mode at the end of the pressing input) according to the determination that the pressing input using the button lasts longer than the first threshold time and shorter than a second threshold time; and (ii) displaying an optional menu for changing the state of the device (e.g., the optional menu includes shutdown, restart, sleep / suspend options, which, when selected, shut down, restart, or sleep / suspend the device respectively) according to the determination that the pressing input using the button lasts longer than the second threshold time.

[0194] In some embodiments of the method of N8, the method includes restarting the device according to the determination that the pressing input using the button lasts longer than a third threshold time greater than the second threshold time in response to receiving an input via the input element with the integrated biosensor.

[0195] (N9) In some embodiments of any one of the methods N1 - N8, the method includes receiving a second input via an input element having an integrated biosensor while displaying a user interface associated with a second user account after replacing the display of the user interface associated with a first account with the user interface associated with the second user account. In response to receiving the second input via the input element having the integrated biosensor, while the first user account has an active session on the device, a determination that the second input meets a first user switching criterion, the first user switching criterion including the requirement that biometric information detected during input using the input element matches the biometric information of the first user account among a plurality of user accounts, and in accordance with the determination, (i) unlocking the device with respect to the first user account, (ii) locking the device with respect to the second user account, and (iii) replacing the display of the user interface associated with the second account with the user interface associated with the first user account.

[0196] (N10) In another aspect, an electronic device is provided, the electronic device including one or more processors, a memory, a display, and a biosensor. One or more programs are stored in the memory and configured for execution by the one or more processors, the one or more programs including instructions to perform or cause to be performed any one of the methods N1 - N9.

[0197] (N11) In an additional aspect, a non - transitory computer - readable storage medium storing one or more programs is provided, the one or more programs including instructions that, when executed by one or more processors of a computing system having a display and a biosensor, cause the computing system to perform or cause to be performed any one of the methods N1 - N9.

[0198] (N12) In yet another aspect, a graphical user interface is provided on a computing system having one or more processors, a memory, a display, and a biosensor, and the graphical user interface comprises a user interface that is displayed according to any of the methods of claims N1 to N9.

[0199] (N13) In another aspect, a computing device is provided. The computing device includes a display and a biosensor, and means for executing or causing the execution of any of the methods of claims N1 to N9.

[0200] To better understand the various embodiments described, the following "Modes for Carrying Out the Invention" should be referred to in conjunction with the following drawings. Here, like reference numerals refer to corresponding parts throughout all of those figures.

Brief Description of the Drawings

[0201]

Figure 1A

Figure 1B

Figure 2A

Figure 2B

Figure 2C

Figure 2D

Figure 3A

Figure 3B

Figure 3C

Figure 3D

Figure 3E

Figure 4

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 5E

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 6E

Figure 6F

Figure 7A

Figure 7B

Figure 7C

Figure 7D

Figure 8A

Figure 8B

Figure 8C

Figure 8D

Figure 8E

Figure 8F

Figure 8G

Figure 8H

Figure 9

Figure 10A

Figure 10B

Figure 10C

Figure 10D

Figure 10E

Figure 10F

Figure 10G

Figure 10H

Figure 11A

Figure 11B

Figure 11C

Figure 11D

Figure 11E

Figure 11F

Figure 11G

Figure 11H

Figure 12A

Figure 12B

Figure 12C

Figure 12D

Figure 12E

Figure 12F

Figure 12G

Figure 12H

Figure 13A

Figure 13B

Figure 13C

Figure 14A

Figure 14B

Figure 14C

Figure 14D

Figure 14E

Figure 15A

Figure 15B

Figure 15C

Figure 15D

Figure 15E

Figure 15F

Figure 15G

Figure 15H

Figure 16A

Figure 16B

Figure 16C

Figure 16D

Figure 16E

Figure 16F

Figure 16G

Figure 16H

Figure 16I

Figure 16J

Figure 16K

Figure 16L

Figure 16M

Figure 16N

Figure 16O

Figure 16P

Figure 16Q

Figure 16R

Figure 16S

Figure 16T

Figure 17A

Figure 17B

Figure 17C

Figure 17D

Figure 17E

Figure 17F

Figure 17G

Figure 18A

Figure 18B

Figure 18C

Figure 18D

Figure 18E

Figure 18F

Figure 18G

Figure 18H

Figure 18I

Figure 19A

Figure 19B

Figure 19C

Figure 19D

Figure 19E

Figure 19F

Figure 19G

Figure 19H

Figure 19I

Figure 19J

Figure 19K

Figure 20A

Figure 20B

Figure 20C

Figure 20D

Figure 20E

Figure 20F

Figure 20G

Figure 21A

Figure 21B

Figure 21C

Figure 21D

Figure 21E

Figure 21F

Figure 21G

Figure 21H

Figure 21I

Figure 21J

Figure 22A

Figure 22B

Figure 22C

Figure 23A

Figure 23B

Figure 24A

Figure 24B

Figure 24C

Figure 24D

Figure 24E

Figure 24F

Figure 24G

Figure 24H

Figure 24I

Figure 24J

Figure 24K

Figure 24L

Figure 24M

Figure 24N

Figure 25A

Figure 25B

Figure 25C

Figure 26A

Figure 26B

Figure 26C

Figure 26D

Figure 26E

Figure 27A

Figure 27B

Figure 27C

Figure 27D

Figure 27E

Figure 27F

Figure 27G

Figure 27H

Figure 27I

Figure 27J

Figure 27K

Figure 27L

Figure 27M

Figure 27N

Figure 27O

Figure 28A

Figure 28B

Figure 28C

Figure 28D

Figure 28E

Figure 28F

Figure 28G

Figure 28H

Figure 28I

Figure 28J

Figure 28K

Figure 28L

Figure 28M

Figure 28N

Figure 28O

Figure 28P

Figure 29A

Figure 29B

Figure 29C

Figure 29D

Figure 29E

Figure 29F

Figure 29G

Figure 29H

Figure 29I

Figure 29J

Figure 29K

Figure 29L

Figure 29M

Figure 29N

Figure 29O

Figure 29P

Figure 29Q

Figure 29R

Figure 29S

Figure 29T

Figure 30A

Figure 30B

Figure 30C

Figure 30D

Figure 30E

Figure 30F

Figure 31A

Figure 31B

Figure 32A

Figure 32B

Figure 32C

Figure 32D

Figure 32E

Figure 33A

Figure 33B

Figure 33C

Figure 33D

Figure 33E

Figure 33F

Figure 33G

Figure 33H

Figure 33I

Figure 33J

Figure 33K

Figure 34A

Figure 34B

Figure 34C

Figure 35A

Figure 35B

Figure 36A

Figure 36B

Figure 36C

Figure 36D

Figure 36E

Figure 36F

Figure 36G

Figure 36H

Figure 36I

Figure 36J

Figure 36K

Figure 36L

Figure 36M

Figure 36N

Figure 36O

Figure 36P

Figure 36Q

Figure 36R

Figure 36S

Figure 36T

Figure 36U

Figure 36V

Figure 36W

Figure 37A

Figure 37B

Figure 37C

Figure 37D

Figure 37E

Figure 37F

Figure 37G

Figure 37H

Figure 37I

Figure 37J

Figure 37K

Figure 37L

Figure 37M

Figure 38A

Figure 38B

Figure 38C

Figure 38D

Figure 38E

Figure 38F

Figure 38G

Figure 38H

Figure 38I

Figure 38J

Figure 39

Figure 40

Figure 41A

Figure 41B

Figure 41C

Figure 41D

Figure 41E

Figure 42A

Figure 42B

Figure 43A

Figure 43B

Figure 43C

Figure 43D

Figure 44A

Figure 44B

Figure 44C

Figure 44D

Figure 45A

Figure 45B

Figure 45C

Figure 46A

Figure 46B

Figure 47A

Figure 47B

Figure 48A

Figure 48B

Figure 48C

Figure 49

Figure 50

Figure 51

Figure 52

Figure 53

Figure 54

Figure 55

Figure 56

Figure 57

Figure 58

Figure 59

Figure 60

Figure 61

Figure 62

Figure 63

Figure 64

Figure 65

Figure 66

Figure 67

Figure 68

Figure 69

Figure 70A

Figure 70B

Figure 70C

Figure 70D

Figure 71A

Figure 71B

Figure 71C

Figure 71D1

Figure 71D2

Figure 71E

Figure 71F

Figure 71G

Figure 71H

Figure 71I

Figure 71J

Figure 71K

Figure 71L

Figure 71M

Figure 72A

Figure 72B

Figure 72C1

Figure 72C2

Figure 72D

Figure 72E

Figure 72F

Figure 72G

Figure 72H

Figure 72I

Figure 72J

Figure 72K

Figure 73A

Figure 73B

Figure 73C

Figure 73D

Figure 73E

Figure 73F

Figure 73G

Figure 73H

Figure 73I

Figure 73J

Figure 73K

Figure 73L

Figure 73M

Figure 73N

Figure 74A

Figure 74B

Figure 74C

Figure 74D

Figure 74E

Figure 74F

Figure 74G

Figure 74H

Figure 75

[0202] Figures 1A-1B, 2A-2D, 3A-3E, and 4 provide an illustration of an exemplary device. Figures 5A-14E show an exemplary user interface for updating dynamic input and output devices. Figures 44A-44D are flow diagrams of a method 600 for updating dynamic input and output devices. Figures 45A-45C are flow diagrams of a method 700 for updating dynamic input and output devices. Figures 46A-46B are flow diagrams of a method 800 for maintaining application functionality while in full screen mode. Figures 47A-47B are flow diagrams of a method 900 for displaying notifications on a touch screen display. Figures 48A-48C are flow diagrams of a method 1000 for moving user interface portions. The user interfaces of Figures 5A-14E are used to illustrate the methods and / or processes of Figures 44A-44D, 45A-45C, 46A-46B, 47A-47B, and 48A-48C.

[0203] Figures 49-54 and 66-67 include flow diagrams of methods of utilizing a touch-sensitive secondary display to enable efficient interaction in a computing system. The user interfaces of FIGS. 15A-43D are used to illustrate the methods and / or processes of FIGS. 49-54 and 66-67. Exemplary Devices and Systems

[0204] Reference will now be made in detail to embodiments shown in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to one of ordinary skill in the art that the various embodiments described may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0205] In this specification, terms such as first, second, etc. are used in some instances to describe various elements, but it should also be understood that these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various embodiments described, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0206] The terms used in the description of the various embodiments described in this specification are for the purpose of describing particular embodiments only and are not intended to be limiting. As used in the description of the various embodiments described and in the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. As used herein, the term "and / or" refers to and includes any and all possible combinations of one or more of the associated listed items. It is also to be understood that the terms "includes", "including", "comprises", and / or "comprising", when used herein, specify the presence of the stated feature, integer, step, operation, element, and / or component, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0207] As used herein, the term "if (when)" is optionally construed to mean "when", "upon", "in response to determining", or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if (a stated condition or event) is detected" is optionally construed to mean "upon determining" or "in response to determining", or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0208] FIG. 1A is an exemplary diagram of a portable computing system 100 according to some embodiments. The portable computing system 100 may be, for example, a laptop computer such as a MACBOOK® device, or any other portable computing device. The portable computing system 100 includes (A) a display portion 110 (also referred to herein as the first housing 110 or housing 110) having a primary display 102, and (B) a body portion 120 (also referred to herein as the second housing 120 or housing 120) including a dynamic function row 104, a set of physical (i.e., movably actuated) keys 106, and a touchpad 108, which are partially included within the same housing. The display portion 110 is generally mechanically, electrically, and communicatively coupled to the body portion 120 of the portable computing system 100. For example, the portable computing system 100 may include a hinge, which enables the display portion 110 to be rotated relative to the body portion 120. The portable computing system 100 includes one or more processors and a memory storing one or more programs executed by the one or more processors to execute any of the embodiments described herein. In some embodiments, the dynamic function row 104, described in more detail with reference to FIG. 1B, is a touch screen display that uses resistive sensing, acoustic sensing, capacitive sensing, optical sensing, infrared sensing, etc. to detect user touch inputs and selections. In some embodiments, the primary display 102 of the display portion 110 is also a touch screen display.

[0209] FIG. 1B is an exemplary diagram of a main body portion 120 of a portable computing system 100 according to some embodiments. The main body portion 120 includes a set of physical keys 106 (also referred to herein as "physical keys 106" and "keyboard 106"), a dynamic function row 104, and a touchpad 108 that are partially contained within the same housing. In some embodiments, the dynamic function row 104, which is a touch screen, replaces the function row of the set of physical keys 106, which allows for a reduction in the space consumed by the set of physical keys 106, allows for a smaller overall main body portion 120, or allows for other components such as the touchpad 108 to be larger. In some embodiments, the length of the dynamic function row 104 is approximately 18 inches relative to the major dimension of the set of physical keys 106. Although referred to as a "row" for simplicity of explanation, in some other embodiments, the touch screen including the dynamic function row 104 of FIG. 1A may take any other form such as a rectangle, a circle, multiple rows, columns, multiple columns, multiple separate sectors, etc. FIGS. 1A-1B show the dynamic function row 104 replacing the function row of the set of physical keys 106, but in some other embodiments, the dynamic function row 104 may replace, in addition and / or alternatively, the numeric keypad section, the edit / function section, etc. of the set of physical keys 106.

[0210] Each physical key of the physical key set 106 has at least one associated input. The input may be a printable character, a non-printable character, a function, or other input. The input associated with the physical key may be indicated by a character, word, symbol, or other mark (e.g., printed) on the surface of the key in Latin, Arabic, Chinese, or any other character. For example, a specific physical key indicated by 138 is associated with the alphabetic character "z" as indicated by the character z shown on the key. In another example, a physical key labeled with the word "command" may be associated with a command function. For example, the physical key set 106 may be associated with a QWERTY, Dvorak, or other keyboard layout with alphanumeric, numeric, and / or editing / function sections (e.g., standard, extended, or compact) according to ISO / IEC 9995, ANSI-INCITS 154-1988, JIS X 6002-1980, or other similar standards.

[0211] A signal corresponding to the input associated with the physical key may be received by the processor of the portable computing system 100 (or the computing device 202 of FIGS. 2A-2D or the external keyboard 206 of FIGS. 2A-2B) when the key is activated by the user. In an exemplary embodiment, each key of the physical key set 106 includes two plates and a spring. The user can activate the key by pressing it down, which compresses the spring. When the spring is compressed, the two plates can contact each other, which allows current to flow through the connected plates. The input corresponding to the key can be provided to the processor in response to the flow of current through the connected plates. For example, in response to the activation of one of the keys of the key set 106 of the external keyboard 206 of FIG. 2C, the input corresponding to the activated key can be provided to the computing device 202. It will be appreciated that other systems for keys that operate movably may be used.

[0212] In some embodiments, the dynamic function row 104 is a touch screen display (also referred to herein as a touch-sensitive secondary display) that displays one or more user-selectable symbols 142 (which may also be referred to herein as "user interface elements", "user interface components", "affordances", "buttons", or "soft keys"). For example, the dynamic function row 104 replaces the function row keys on a typical keyboard. The user can select a particular one of the one or more user-selectable symbols 142 by touching a location on the touch screen display corresponding to the particular symbol of the one or more user-selectable symbols 142. For example, the user can select the user-selectable symbol indicated by the glasses symbol 144 by tapping the dynamic function row 104 such that the user's finger contacts the dynamic function row 104 at the location of the glasses indicator 214. In some embodiments, a tap contact or tap gesture includes a touch down of the contact and a lift off of the contact within a predetermined time (e.g., 250 ms, etc.). In some embodiments, the touch screen display of the dynamic function row 104 is implemented using resistive sensing, acoustic sensing, capacitive sensing, optical sensing, infrared sensing, etc. to detect user input and selections.

[0213] When the user selects a particular one of the one or more user-selectable symbols 142, a signal corresponding to the particular symbol of the one or more user-selectable symbols 142 is generated by the dynamic function row 104. For example, when the user taps "esc" on the dynamic function row 104, the dynamic function row 104 transmits a signal indicating user input corresponding to the escape function to the processor of the portable computing system 100 (or the processor of the computing device 202 of FIGS. 2A-2D, or the processor of the external keyboard 206 of FIGS. 2A-2B, or the processor of the first input mechanism 212, FIGS. 2C or the external input mechanism 222, FIG. 2D).

[0214] In some embodiments, when a particular symbol among one or more user-selectable symbols 142 is selected, the dynamic function row 104 transmits a signal corresponding to the position on the touch screen display where the particular symbol among the one or more user-selectable symbols 142 is displayed to a processor of the portable computing system 100 (or the computing device 202 of FIGS. 2A-2D, or the processor of the external keyboard 206 of FIGS. 2A-2B, or the processor of the first input mechanism 212, FIG. 2C or the processor of the external input mechanism 222, FIG. 2D). For example, the dynamic function row 104 can transmit a signal including a position value (0-20) that depends on the position on the touch screen display of the particular symbol among the selected one or more user-selectable symbols 142. In the exemplary embodiment of FIG. 1B, the "esc" symbol may have a position value of 0, the binoculars symbol 144 may have a position value of 16, and so on. The processor of the portable computing system 100 (or the computing device 202 of FIGS. 2A-2D, or the processor of the external keyboard 206 of FIGS. 2A-2B, or the processor of the first input mechanism 212, FIG. 2C, or the processor of the external input mechanism 222, FIG. 2D) can receive a signal indicating the position value of the selected user-selectable symbol and interpret the position value using context information such as elements of the graphical user interface displayed on the primary display 102 of the display portion 110 (or the external display device 204, FIGS. 2A-2D) that is currently active or has focus.

[0215] Each of the one or more user-selectable symbols 142 can include an indicator such as a symbol (e.g., the binoculars symbol as shown by 144), an abbreviation (e.g., "esc"), a non-abbreviated word, a character, an image, an animated image, a video, etc. In some embodiments, each symbol of the one or more user-selectable symbols 142 is capable of receiving user input(s).

[0216] The input may be associated with each of one or more user-selectable symbols 142. The input may be a function, a character, a numerical value, or the like. Each symbol of the one or more user-selectable symbols 142 can include an indicator corresponding to the input for each symbol of the one or more user-selectable symbols 142. For example, in FIG. 1B, a user-selectable symbol having the abbreviation "esc" indicates to the user that an escape function is associated with the user-selectable symbol. A function associated with one or more user-selectable symbols 142 may be activated when the user selects the user-selectable symbol. For example, the escape function may be activated when the user selects a user-selectable symbol having the indicator "esc". The activation of the function can have different effects depending on the current state of the portable computing system 100 (or the computing device 202 in FIGS. 2A-2D). For example, when a dialog box is open on the primary display 102 of the display portion 110 (or an external display device 204, FIGS. 2A-2D), the dialog box can be closed by activating the escape function on the dynamic function row 104. In another example, when a game application is being executed by the processor of the portable computing system 100 (or the computing device 202 in FIGS. 2A-2D), the game can be paused by activating the escape function on the dynamic function row 104.

[0217] In some embodiments, a function may be associated with a combination of a key that operates movably and / or user-selectable symbols. For example, the simultaneous activation of a command key and the "c" key (i.e., command + c) may be associated with a "copy" function. In another example, the simultaneous activation of a command key and the selection of a user-selectable symbol having an indicator "esc" (i.e., command + esc) can activate a function to open a specific application such as a media player application. In yet another example, the simultaneous selection of two user-selectable symbols (e.g., a user-selectable symbol having an indicator esc and a user-selectable symbol 144 having a magnifying glass indicator) can result in the activation of a function such as a special search function.

[0218] In some embodiments, a first subset 146 of one or more user-selectable symbols 142 of the dynamic function row 104 may be associated with one group of functions, and a second subset 148 of one or more user-selectable symbols 142 of the dynamic function row 104 may be associated with a second group of functions. For example, the user-selectable symbols in the first subset 146 may be global functions (e.g., system-level functions or affordances), and the user-selectable symbols in the second subset 148 may be application-specific functions. Thus, the user-selectable symbols in the second subset 148 change when the focus shifts from a first element of the graphical user interface (e.g., a first window corresponding to an internet browser application) displayed on the primary display 102 to a second element of the graphical user interface (e.g., a second window corresponding to an email application). In contrast, the user-selectable symbols in the first subset 146 are maintained when the focus shifts from a first element of the graphical user interface to a second element of the graphical user interface.

[0219] In some embodiments, the user-selectable symbols in the second subset 148 are determined based on the display of the active user interface element on the primary display 102 that is in focus. In some embodiments, the term "in focus" refers to the active element of the user interface that is currently in the foreground, operating actively, or controllable by input received from a user of the computing system, such as a key press, mouse click, voice command, gesture movement, etc. (e.g., a window associated with an application, a specific toolbar or menu associated with an application, or an operating system).

[0220] In some embodiments, the first subset 146 of one or more user-selectable symbols 142 corresponding to the global user-selectable symbols occupies the first area of the dynamic function row 104 (e.g., the left half of the dynamic function row 104), and the second subset 148 of one or more user-selectable symbols 142 occupies the second area of the dynamic function row 104 (e.g., the right half of the dynamic function row 104). It will be appreciated that other portions of the dynamic function row 104 may be allocated to the first subset 146 and the second subset 148. In some embodiments, when no application has focus, the second area of the dynamic function row 104 may not contain any user-selectable symbols. In some embodiments, the dynamic function row 104 includes three or more subsets of user-selectable symbols. In some embodiments, the dynamic function row 104 includes a single set of user-selectable symbols that is not divided into subsets. Although a single row of user-selectable symbols is shown in the dynamic function row 104 of FIG. 1B, it will be recognized that the dynamic function row 104 can include multiple rows of user-selectable symbols.

[0221] In some embodiments, changing the focus changes which elements of the graphical user interface displayed on the primary display 102 of the display portion 110 (or the external display device 204, FIGS. 2A-2D) are active and which elements receive user input. The user input may be received from a keyboard, mouse, touchpad, or other user input device. Additionally and / or alternatively, in some embodiments, changing the focus changes the elements presented in the foreground of the graphical user interface displayed on the primary display 102 of the display portion 110 (or the external display device 204, FIGS. 2A-2D).

[0222] In some embodiments, the change in focus occurs in response to user input, e.g., in response to a user selection of an element (e.g., a different window) of the graphical user interface displayed on the primary display 102 of the display portion 110 (or the external display device 204, FIGS. 2A-2D), or in response to a user selection of a user-selectable symbol (e.g., one of the affordances / symbols displayed on the dynamic function row 104). The user selection may be a keystroke, mouse click, mouse over, command+tab input, etc. In some embodiments, the change in focus occurs in response to a determination by the operating system of the portable system 100 (or the computing device 202 of FIGS. 2A-2D). For example, when the user closes an application window that has focus, the operating system can give focus to a different application, such as the application that had focus before the closed application window. In another example, when the user closes an application window that has focus, the operating system can give focus to a dialog box that prompts the user to save changes made to a document via the application.

[0223] In some embodiments, the focus change may be from one element associated with an application to another element associated with the same application (e.g., from the email composition window of an email application to the inbox list window of the email application, or from one tab of an internet browser application to another tab of the internet browser application). In some embodiments, the focus change may be from an element associated with one application to an element associated with another application (e.g., from an internet browser window to an email application window). Further, in some embodiments, the focus change may be from an element associated with an application to an element associated with the operating system, such as a system dialog box, a system settings control (e.g., volume control), a window associated with a file / folder navigation application (e.g., the FINDER application of Apple Inc.). Additionally, the focus may also be directed to a dialog box, a file directory, a settings control (e.g., volume control), or any other element of the graphical user interface where information may be presented to the user and / or user input may be received.

[0224] Figure 2A is an exemplary diagram of a first implementation of a desktop computing system 200 according to some embodiments. The desktop computing system 200 includes a computing device 202, an external display device 204 with a primary display 102, an external keyboard 206, and an external mouse 208. The computing device 202 includes one or more processors and a memory that stores one or more programs executed by the one or more processors. In some embodiments, the external display device 204 may be integrated with the computing device 202, such as an iMAC (registered trademark) device. In some embodiments, the primary display 102 of the external display device 204 is a touch screen display. In Figure 2A, the external display device 204 (also referred to herein as the first housing 204 or housing 204), the external keyboard 206, and the external mouse 208 are communicatively coupled to the computing device 202 via a wired connection such as USB or PS / 2, or via a wireless communication link using a communication protocol such as Bluetooth, Wi-Fi (registered trademark). For example, the external keyboard 206 (also referred to herein as the second housing 206 or housing 206) is at most 15 feet (e.g., about 3 feet away) from the computing device 202. In Figure 2A, the external keyboard 206 includes a dynamic function row 104 and a set of physical keys 106 that are at least partially included within the same housing. In some embodiments, the dynamic function row 104, described in more detail with reference to Figure 1B, is a touch screen display. In some embodiments, the external keyboard 206 includes one or more processors and a memory that stores one or more programs that may be executed by the one or more processors of the external keyboard 206 to execute any of the embodiments described herein.In some embodiments, the external keyboard 206 relays signals indicating user input (e.g., keystrokes and selection of user-selectable symbols / affordances displayed by the dynamic function row 104) to the computing device 202.

[0225] FIG. 2B is an exemplary diagram of a second implementation of the desktop computing system 200 according to some embodiments. In FIG. 2B, the desktop computing system 200 includes a computing device 202, an external display device 204 with a primary display 102, and an external keyboard 206. In FIG. 2B, the external display device 204 and the external keyboard 206 are communicatively coupled to the computing device 202 via a wired connection such as USB or PS / 2, or via a wireless communication link using a communication protocol such as Bluetooth, Wi-Fi. In FIG. 2B, the external keyboard 206 includes a dynamic function row 104, a set of physical keys 106, and a touchpad 108 that are at least partially included within the same housing. In some embodiments, the dynamic function row 104, described in more detail with reference to FIG. 1B, is a touchscreen display. In some embodiments, the external keyboard 206 includes one or more processors and a memory storing one or more programs executable by the one or more processors of the external keyboard 206 to perform any of the embodiments described herein. In some embodiments, the external keyboard 206 relays signals indicating user input (e.g., keystrokes, user interaction with the touchpad 108, and selection of user-selectable symbols / affordances displayed by the dynamic function row 104) to the computing device 202.

[0226] Figure 2C is an exemplary diagram of a third implementation of a desktop computing system 200 according to some embodiments. In Figure 2C, the desktop computing system 200 includes a computing device 202, an external display device 204 with a primary display 102, an external keyboard 206, and a first external input mechanism 212. In Figure 2C, the external display device 204, the external keyboard 206, and the first external input mechanism 212 are communicatively coupled to the computing device 202 via a wired connection such as USB or PS / 2, or via a wireless communication link using a communication protocol such as Bluetooth, Wi-Fi. In Figure 2C, the external keyboard 206 includes a set of physical keys 106, and the first external input mechanism 212 includes a dynamic function row 104 and a touchpad 108 that are at least partially included within the same housing. In some embodiments, the dynamic function row 104, described in more detail with reference to Figure 1B, is a touch screen display. In some embodiments, the first external input mechanism 212 includes one or more processors and a memory storing one or more programs executable by the one or more processors of the first external input mechanism 212 to execute any of the embodiments described herein. In some embodiments, the first external input mechanism 212 relays a signal indicative of user input (e.g., user interaction with the touchpad 108 and user selection of user-selectable symbols / affordances displayed by the dynamic function row 104) to the computing device 202.

[0227] Figure 2D is an exemplary diagram of a fourth implementation of a desktop computing system 200 according to some embodiments. In Figure 2D, the desktop computing system 200 includes a computing device 202, an external display device 204 with a primary display 102, an external keyboard 206, an external mouse 208, and a second external input mechanism 222. In Figure 2D, the external display device 204, the external keyboard 206, the external mouse 208, and the second external input mechanism 222 are communicatively coupled to the computing device 202 via a wired connection such as USB or PS / 2, or via a wireless communication link using a communication protocol such as Bluetooth, Wi-Fi. In Figure 2A, the external keyboard 206 includes a dynamic function row 104 and a set of physical keys 106. In Figure 2D, the external keyboard 206 includes the set of physical keys 106, and the second external input mechanism 222 includes a dynamic function row 104 that is at least partially included within the housing of the second external input mechanism 222. In some embodiments, the dynamic function row 104, described in more detail with reference to Figure 1B, is a touch screen display. In some embodiments, the second external input mechanism 222 includes one or more processors and a memory storing one or more programs executable by the one or more processors of the second external input mechanism 222 to execute any of the embodiments described herein. In some embodiments, the second external input mechanism 222 relays a signal indicating a user input (e.g., a user selection of a user-selectable symbol / affordance displayed by the dynamic function row 104) to the computing device 202.

[0228] FIG. 3A is a block diagram of an electronic device 300 according to some embodiments. In some embodiments, the electronic device 300 is a portable electronic device such as a laptop (e.g., the portable computing system 100, FIG. 1A). In some embodiments, the electronic device 300 is not a portable device but a desktop computer (e.g., the computing device 202 of the desktop computing system 200, FIGS. 2A-2D), which is communicatively coupled to an external display system (e.g., the external display device 204, FIGS. 2A-2D), and optionally, an external touch sensing surface (e.g., the touch pad 108, FIGS. 2B-2C, a touch sensing display such as the external display device 204, FIGS. 2A-2D, and / or the dynamic function row 104, FIGS. 2A-2D).

[0229] The electronic device 300 generally supports various applications such as one or more of a drawing application, a presentation application, a word processing application, a website creation application, a disk authoring application, a spreadsheet application, a game application, a video conferencing application, an email application, an instant messaging application, an image management application, a digital camera application, a digital video camera application, a web browser application, and / or a media player application.

[0230] The various applications executed on the electronic device 300 optionally use at least one common physical user interface device, such as a touch sensing surface. One or more functions of the touch sensing surface, as well as the corresponding information displayed by the electronic device 300, are optionally adjusted and / or changed for each application and / or within an application. Thus, the common physical architecture (such as a touch sensing surface) of the electronic device 300 optionally supports various applications having a user interface that is intuitive and transparent to the user.

[0231] The electronic device 300 includes a memory 302 (optionally including one or more computer-readable storage media), a memory controller 322, one or more processing units (CPUs) 320, a peripheral device interface 318, an RF circuit 308, an audio circuit 310, a speaker 311, a microphone 313, an input / output (I / O) subsystem 306, other input or control devices 316, and an external port 324. The electronic device 300 optionally includes a display system 312 (e.g., the primary display 102 of the display portion 110, FIGS. 1A and / or the dynamic function row 104, FIGS. 1A - 1B), which may be a touch sensing display (also sometimes referred to herein as a "touch screen" or "touch screen display"). The electronic device 300 optionally includes one or more light sensors 364. The electronic device 300 optionally includes one or more intensity sensors 365 that detect the intensity of contact on a touch sensing surface, such as a touch sensing display or a touch pad. The electronic device 300 optionally includes one or more haptic output generators 367 that generate haptic output on a touch sensing surface, such as a touch sensing display or a touch pad (e.g., touch pad 108, FIGS. 1A - 1B). These components optionally communicate via one or more communication buses or signal lines 303.

[0232] As used herein, the term "intensity" of a contact on a touch sensing surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch sensing surface, or a proxy for the force or pressure of the contact on the touch sensing surface. The intensity of the contact has a range of values that includes at least four distinct values and, more generally, hundreds (e.g., at least 256) of distinct values. The intensity of the contact is optionally determined (or measured) using a variety of techniques and a variety of sensors or combinations of sensors. For example, one or more force sensors under or adjacent to the touch sensing surface are optionally used to measure the force at various points on the touch sensing surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted average) to determine the estimated force of the contact. Similarly, a pressure sensing tip of a stylus is optionally used to determine the pressure of the stylus on the touch sensing surface. Alternatively, the size and / or change in size of the contact area detected on the touch sensing surface, the capacitance and / or change in capacitance of the touch sensing surface proximate to the contact, and / or the resistance and / or change in resistance of the touch sensing surface proximate to the contact are optionally used as a proxy for the force or pressure of the contact on the touch sensing surface. In some implementations, the alternative measurement of the force or pressure of the contact is used directly to determine whether it exceeds an intensity threshold (e.g., the intensity threshold is described in units corresponding to the alternative measurement). In some implementations, the alternative measurement of the force or pressure of the contact is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether it exceeds an intensity threshold (e.g., the intensity threshold is a pressure threshold measured in units of pressure).

[0233] As used in this specification and the claims, the term "haptic output" refers to the physical displacement of the device relative to its previous position, the physical displacement of a component of the device (e.g., a touch sensing surface) relative to another component of the device (e.g., the housing), or the displacement of a component relative to the center of mass of the device, that will be detected by the user through the user's sense of touch. For example, in a situation where the device or a component of the device is in contact with a touch-sensitive surface of the user (e.g., the finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the device or the component of the device. For example, the movement of a touch sensing surface (e.g., a touch sensing display or a touch / track pad) can optionally be interpreted by the user as a "down click" or "up click" of a physical actuator button. In some cases, the user may feel a tactile sensation such as a "down click" or "up click" even when there is no movement of the physical actuator button associated with the touch sensing surface that has been physically pressed (e.g., displaced) by the user's action. As another example, the movement of a touch sensing surface can optionally be interpreted or perceived by the user as the "roughness" of the touch sensing surface even when there is no change in the smoothness of the touch sensing surface. Such an interpretation of the touch by the user depends on the user's individual sensory perception, but there are many sensory perceptions of touch that are common to the majority of users. Thus, when a haptic output is described as corresponding to a particular sensory perception of the user (e.g., "up click", "down click", "roughness"), unless otherwise stated, the generated haptic output corresponds to the physical displacement of the device or a component of the device that produces the described sensory perception of an average user.

[0234] The electronic device 300 is merely an example, and it should be understood that the electronic device 300 may optionally have more or fewer components than those shown, or may optionally combine two or more components, or may optionally have different configurations or arrangements of those components. The various components shown in FIG. 3A are implemented in hardware, software, firmware, or a combination thereof, including one or more signal processing circuits and / or application specific integrated circuits.

[0235] The memory 302 may optionally include high-speed random access memory and may also optionally include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state memory devices. Access to the memory 302 by other components of the electronic device 300, such as the CPU(s) 320 and the peripheral interface 318, may optionally be controlled by a memory controller 322. The peripheral interface 318 may be used to couple input and output peripheral devices to the CPU(s) 320 and the memory 302. One or more processing units 320 perform various functions for the electronic device 300 and operate or execute various software programs and / or instruction sets stored in the memory 302 to process data. In some embodiments, the peripheral interface 318, the CPU(s) 320, and the memory controller 322 may optionally be implemented on a single chip, such as chip 304. In some other embodiments, they may optionally be implemented on separate chips.

[0236] The RF (radio frequency) circuit 308 transmits and receives RF signals, also called electromagnetic signals. The RF circuit 308 converts an electrical signal into an electromagnetic signal or an electromagnetic signal into an electrical signal, and communicates with a communication network and other communication devices via the electromagnetic signal. The RF circuit 308 optionally includes well-known circuits that perform these functions, and these circuits include, but are not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, etc. The RF circuit 308 optionally communicates wirelessly with a network such as the Internet, also called the World Wide Web (WWW), an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN) and / or a metropolitan area network (MAN), and other devices. The wireless communication optionally uses any one of a plurality of communication standards, communication protocols and communication technologies, and these communication standards, communication protocols and communication technologies include the Global System for Mobile Communications (GSM (registered trademark)) for mobile communication, Enhanced Data GSM Environment (EDGE (registered trademark)), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), long term evolution (LTE (registered trademark)), near field wireless communication (near fieldcommunications, NFC), wideband code division multiple access (W-CDMA), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), voice over Internet Protocol (VoIP), Wi-MAX (registered trademark), protocols for e-mail (e.g., Internet message access protocol (IMAP) and / or post office protocol (POP)), instant messaging (e.g., extensible messaging and presence protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol including communication protocols not yet developed as of the filing date of the present document, but not limited thereto.

[0237] The audio circuit 310, speaker 311, and microphone 313 provide an audio interface between the user and the electronic device 300. The audio circuit 310 receives audio data from the peripheral device interface 318, converts this audio data into an electrical signal, and transmits this electrical signal to the speaker 311. The speaker 311 converts the electrical signal into human audible sound waves. The audio circuit 310 also receives an electrical signal converted from sound waves by the microphone 313. The audio circuit 310 converts the electrical signal into audio data and transmits this audio data to the peripheral device interface 318 for processing. The audio data is optionally obtained from and / or transmitted to the memory 302 and / or the RF circuit 308 by the peripheral device interface 318. In some embodiments, the audio circuit 310 also includes a headset jack. The headset jack provides an interface between the audio circuit 310 and a removable audio input / output peripheral device such as an output-only headset, or a headset with both output (e.g., headphones for one or both ears) and input (e.g., a microphone).

[0238] The I / O subsystem 306 couples input / output peripheral devices of the electronic device 300, such as the display system 312 and other input or output devices 316, to the peripheral device interface 318. The I / O subsystem 306 optionally includes a display controller 356, a light sensor controller 358, an intensity sensor controller 359, a haptic feedback controller 361, and one or more other input controllers 360 for other input or control devices. The one or more other input controllers 360 receive electrical signals from and transmit electrical signals to other input or control devices 316. The other input or control devices 316 optionally include physical buttons (such as push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, the other input controller(s) 360 are optionally coupled to (or not coupled to any of) a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. One or more physical buttons optionally include up / down buttons for volume control of the speaker 311 and / or the microphone 313.

[0239] The display system 312 (e.g., the primary display 102 of the display portion 110, FIGS. 1A and / or the dynamic function row 104, FIGS. 1A - 1B) provides an output interface (and optionally an input interface when it is a touch-sensitive display) between the electronic device 300 and the user. The display controller 356 receives electrical signals from and / or transmits electrical signals to the display system 312. The display system 312 displays visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output corresponds to user interface objects / elements.

[0240] In some embodiments, the display system 312 (e.g., the primary display 102 of the display portion 110, FIGS. 1A and / or the dynamic function row 104, FIGS. 1A - 1B) is a touch - sensitive display having a touch - sensing surface, sensors, or a set of sensors that receive input from a user based on tactile and / or haptic contact. Thus, the display system 312 and the display controller 356 (along with any associated modules and / or instruction sets within the memory 302) detect contact (and any movement or interruption of the contact) on the display system 312 and convert the detected contact into an interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on the display system 312. In an exemplary embodiment, the point of contact between the display system 312 and the user corresponds to the area under the user's finger.

[0241] The display system 312 (e.g., the primary display 102 of the display portion 110, FIGS. 1A and / or the dynamic function row 104, FIGS. 1A - 1B) optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, LED (light emitting diode) technology, or OLED (organic light emitting diode) technology, although in other embodiments, other display technologies are used. In some embodiments, when the display system 312 is a touch - sensitive display, the display system 312 and the display controller 356 optionally use any of a plurality of touch - sensing technologies currently known or later developed to detect contact and any movement or interruption of that contact, and those touch - sensing technologies include capacitive technology, resistive technology, infrared technology, and surface acoustic wave technology, as well as other proximity sensor arrays or other elements for determining one or more contact points with the display system 312, but are not limited thereto. In an exemplary embodiment, a projected mutual capacitance sensing technology as seen in iPHONE (registered trademark), iPOD TOUCH (registered trademark), and iPAD (registered trademark) from Apple Inc., Cupertino, California, is used.

[0242] The display system 312 (e.g., the primary display 102 of the display portion 110, FIGS. 1A and / or the dynamic function row 104, FIGS. 1A - 1B) optionally has a video resolution exceeding 400 dpi (e.g., 500 dpi, 800 dpi, or more). In some embodiments, the display system 312 is a touch - sensitive display where the user can optionally make contact using a stylus, finger, etc. In some embodiments, the user interface is designed to primarily operate with finger contact and gestures. In some embodiments, the electronic device 300 converts rough input by a finger into a command to execute an accurate pointer / cursor position or an action desired by the user.

[0243] In some embodiments, in addition to the display system 312, the electronic device 300 optionally includes a touch pad (e.g., touch pad 108, FIGS. 1A - 1B) for activating or de - activating specific functions. In some embodiments, the touch pad is a touch - sensitive area of the electronic device 300 that, unlike the display system 312, does not display visual output. In some embodiments, when the display system 312 is a touch - sensitive display, the touch pad is optionally a touch - sensitive surface separate from the display system 312 or an extension of the touch - sensitive surface formed by the display system 312.

[0244] The electronic device 300 also includes a power system 362 for supplying power to various components. The power system 362 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC), etc.), a recharge system, a power - outage detection circuit, a power converter or inverter, a power - state indicator (e.g., light - emitting diode (LED)), and any other components associated with power generation, management, and distribution within a portable device.

[0245] The electronic device 300 also optionally includes one or more optical sensors 364 coupled to an optical sensor controller 358 within the I / O subsystem 306. The optical sensor(s) 364 optionally include a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. The optical sensor(s) 364 receive light from the environment projected through one or more lenses and convert the light into data representing an image. In cooperation with the imaging module 343, the optical sensor(s) 364 optionally capture a still image or video. In some embodiments, the optical sensor is disposed on the front of the electronic device 300 so as to optionally obtain an image of the user for a video conference while the user is viewing other video conference participants on the display system 312.

[0246] The electronic device 300 also optionally includes one or more contact intensity sensors 365 coupled to an intensity sensor controller 359 within the I / O subsystem 306. The contact intensity sensor(s) 365 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrokinetic sensors, piezoelectric sensors, optomechanical sensors, capacitive touch sensing surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of contact on a touch sensing surface). The contact intensity sensor(s) 365 receive contact intensity information (e.g., pressure information or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is juxtaposed with or proximate to a touch sensing surface (e.g., the touch pad 108, FIGS. 1A-1B, or the display system 312 when it is a touch sensing display).

[0247] The electronic device 300 also optionally includes one or more haptic output generators 367 coupled to a haptic feedback controller 361 within the I / O subsystem 306. The haptic output generator(s) 367 optionally includes one or more electroacoustic devices, such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components that convert an electrical signal into a haptic output) on the device. The contact intensity sensor(s) 365 receives haptic feedback generation instructions from the haptic feedback module 333 and generates a haptic output that can be sensed by a user of the electronic device 300. In some embodiments, at least one haptic output generator is juxtaposed with or proximate to a touch sensing surface (e.g., touch pad 108, FIGS. 1A-1B, or display system 312 when it is a touch sensing display), and optionally generates a haptic output by moving the touch sensing surface in a vertical direction (e.g., in / out of the surface of the electronic device 300) or a horizontal direction (e.g., back and forth within the same plane as the surface of the electronic device 300).

[0248] The electronic device 300 also optionally includes one or more proximity sensors 366 coupled to the peripheral device interface 318. Alternatively, the proximity sensor(s) 366 is coupled to other input controller(s) 360 within the I / O subsystem 306. The electronic device 300 also optionally includes one or more accelerometers 368 coupled to the peripheral device interface 318. Alternatively, the accelerometer(s) 368 is coupled to other input controller(s) 360 within the I / O subsystem 306.

[0249] In some embodiments, the software components stored in memory 302 include an operating system 326, a communication module 328 (or instruction set), a touch / motion module 330 (or instruction set), a graphics module 332 (or instruction set), an application 340 (or instruction set), and a dynamic function row module 350 (or instruction set). Further, in some embodiments, as shown in FIG. 3A, memory 302 stores a device / global internal state 357 (or instruction set). The device / global internal state 357 includes, if any, an active application state indicating which application is currently active and / or in focus, a display state indicating which application, view, or other information occupies various regions of the display system 312 (e.g., the primary display 102 of the display portion 110, FIGS. 1A and / or the dynamic function row 104, FIGS. 1A - 1B) and / or an external display system (e.g., the primary display 102 of the external display device 204, FIGS. 2A - 2D and / or the dynamic function row 104, FIGS. 2A - 2D), a sensor state including information obtained from various sensors and input or control devices 316 of the electronic device 300, and one or more of location information regarding the location and / or orientation of the electronic device 300.

[0250] The operating system 326 (e.g., an embedded operating system such as DARWIN®, RTXC®, LINUX®, UNIX®, OS X®, WINDOWS®, or VXWorks®) includes various software components and / or drivers that control and manage general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitate communication between various hardware components and software components.

[0251] The communication module 328 includes various software components that facilitate communication with one or more other devices (e.g., computing device 202, FIGS. 2A-2D, external mouse 208, FIGS. 2A and 2D, external keyboard 206, FIGS. 2A-2B, first external input mechanism 212, FIG. 2C, and / or second external input mechanism 222, FIG. 2D) via one or more external ports 324 and / or RF circuit 308, and transmits / receives data via RF circuit 308 and / or external port 324. The external port 324 (e.g., Universal Serial Bus (USB), FIREWIRE (registered trademark), etc.) is adapted to couple to other devices either directly or indirectly via a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port 324 is a multi-pin (e.g., 30-pin) connector that is the same as, similar to, and / or compatible with the 30-pin connector used on iPod (registered trademark) devices.

[0252] The contact / motion module 330 optionally detects contact with the display system 312 when it is a touch-sensitive display (in cooperation with the display controller 356) and other touch-sensitive devices (e.g., a touch pad or a physical click wheel). The contact / motion module 330 performs various software components related to the detection of contact, such as determining whether contact has occurred (e.g., detecting an event of placing a finger down), determining the intensity of the contact (e.g., the force or pressure of the contact, or an alternative to the force or pressure of the contact), determining whether there is movement of the contact, tracking movement across the touch-sensitive surface (e.g., detecting one or more events of dragging a finger), and determining whether the contact has stopped (e.g., detecting an event of lifting a finger or an interruption of the contact). The contact / motion module 330 receives contact data from the touch-sensitive surface. Determining the movement of the contact point, represented by a series of contact data, optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These operations are optionally applied to a single contact (e.g., contact with one finger) or multiple simultaneous contacts (e.g., "multi-touch" / contact with multiple fingers). In some embodiments, the contact / motion module 330 also detects contact on a touch pad (e.g., touch pad 108, FIGS. 1A-1B).

[0253] In some embodiments, the contact / motion module 330 uses a set of one or more intensity thresholds for determining whether an action has been performed by a user (e.g., for determining whether the user has selected or “clicked” on an affordance). In some embodiments, at least a subset of the intensity thresholds are determined according to software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of the electronic device 300). For example, the mouse “click” threshold of a trackpad or touch screen display can be set to any of a wide range of default thresholds without changing the hardware of that trackpad or touch screen display. Additionally, in some implementations, the user of the device is provided with software settings for adjusting one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once via a system-level click “intensity” parameter).

[0254] The contact / motion module 330 optionally detects gesture inputs by the user. Different gestures on the touch-sensing surface have different contact patterns (e.g., different motions, timings, and / or intensities of detected contacts). Thus, gestures are optionally detected by detecting a particular contact pattern. For example, detecting a finger tap contact includes detecting an event of the finger touching down and subsequently detecting an event of the finger lifting off (lifting off) at the same position (or substantially the same position) as the event of the finger touching down (e.g., at the position of an icon). As another example, detecting a finger swipe gesture on the touch-sensing surface includes detecting an event of the finger touching down, subsequently detecting one or more events of the finger being dragged, and in some embodiments, also subsequently detecting an event of the finger lifting off (lifting off).

[0255] The graphic module 332 includes various known software components that render graphics on the primary display 102 (e.g., the primary display 102 of the display portion 110, FIG. 1A, or the primary display 102 of the external display device 204, FIGS. 2A - 2D) or other displays, and cause their display, including components that change the visual effects of the displayed graphics (e.g., brightness, transparency, chroma, contrast, or other visual characteristics). As used herein, the term "graphic" includes any object that can be displayed to a user, such as text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc., but is not limited thereto. In some embodiments, the graphic module 332 stores data representing the graphics used. Optionally, a corresponding code is assigned to each graphic. The graphic module 332 receives, from an application or the like, one or more codes that specify the graphics to be displayed, along with coordinate data and other graphic characteristic data as needed, and then generates the image data of the screen to be output to the display controller 356.

[0256] The tactile feedback module 333 includes various software components that generate instructions used by the tactile output generator(s) 367 to generate tactile output at one or more locations on the electronic device 300 in response to user interaction with the electronic device 300.

[0257] The application 340 optionally includes the following modules (or sets of instructions), or subsets or supersets thereof. · An email client module 341 that receives, sends, configures, and views emails (sometimes referred to herein as an "email app" or "electronic mail app"). · An imaging module 342 that captures still images and / or video images, · An image management module 343 that edits and views still images and / or video images (sometimes referred to as a "photo app" in this specification), · A media player module 344 that plays audio and / or video (sometimes referred to as a "media player app" in this specification), and · A web browsing module 345 that connects to and browses the Internet (sometimes referred to as a "web browser" in this specification).

[0258] Examples of other applications 340 that are optionally stored in the memory 302 include messaging and communication applications, word processing applications, other image editing applications, drawing applications, presentation applications, JAVA (registered trademark)-compatible applications, encryption applications, digital rights management applications, voice recognition applications, and voice replication applications.

[0259] In cooperation with one or more of the RF circuit 308, the display system 312 (e.g., the primary display 102 of the display portion 110, FIGS. 1A and / or the dynamic function row 104, FIGS. 1A - 1B), the display controller 356, and the touch module 330, the graphic module 332, the email client module 341 includes executable instructions for creating, sending, receiving, and managing emails in response to user commands. In cooperation with the image management module 343, the email client module 341 makes it very easy to create and send emails with still images or video images captured by the imaging module 342.

[0260] In cooperation with one or more of display system 312 (e.g., primary display 102 of display portion 110, FIGS. 1A and / or dynamic function row 104, FIGS. 1A - 1B), display controller 356, light sensor(s) 364, light sensor controller 358, contact module 330, graphic module 332, and image management module 343, imaging module 342 includes executable instructions for capturing a still image or video (including a video stream), storing them in memory 302, changing the characteristics of the still image or video, or deleting the still image or video from memory 302.

[0261] In cooperation with one or more of display system 312 (e.g., primary display 102 of display portion 110, FIGS. 1A and / or dynamic function row 104, FIGS. 1A - 1B), display controller 356, contact module 330, graphic module 332, and imaging module 342, image management module 343 includes executable instructions for arranging, changing (e.g., editing), or otherwise operating on still images and / or video images, labeling, deleting, presenting (e.g., within a digital slide show or album), and storing them.

[0262] In cooperation with one or more of display system 312 (e.g., primary display 102 of display portion 110, FIGS. 1A and / or dynamic function row 104, FIGS. 1A - 1B), display controller 356, contact module 330, graphic module 332, audio circuit 310, speaker 311, RF circuit 308, and web browsing module 345, media player module 344 includes executable instructions that enable a user to download and play recorded music or other sound files stored in one or more file formats such as MP3 or AAC files, and executable instructions for displaying, presenting, or otherwise playing video (e.g., on primary display 102 of display portion 110, FIGS. 1A or primary display 102 of external display device 2014 connected via external port 324, FIGS. 2A - 2B).

[0263] In cooperation with one or more of RF circuit 308, display system 312 (e.g., primary display 102 of display portion 110, FIGS. 1A and / or dynamic function row 104, FIGS. 1A - 1B), display controller 356, contact module 330, and graphic module 332, web browsing module 345 includes executable instructions for browsing the Internet according to user instructions, including searching for, linking to, receiving, and displaying web pages or portions thereof, and attached files and other files linked to the web page.

[0264] The dynamic function row (DFR) module 350 includes a focus determination module 351, a DFR determination module 352, and a DFR presentation module 353. In some embodiments, the focus determination module 351 is configured to determine an active user interface element that has focus on a graphical user interface displayed by a display system 312 (e.g., the primary display 102 of the display portion 110, FIG. 1A) or an external display system (e.g., the external display device 204, FIGS. 2A - 2D). In some embodiments, the DFR determination module 352 is configured to determine a graphic (e.g., a set of one or more affordances) based on the active user interface element that has focus. In some embodiments, the DFR presentation module 353 is configured to render the graphic determined by the DFR determination module 352 on the display system 312 (e.g., the dynamic function row 104, FIGS. 1A - 1B). The DFR presentation module 353 includes components that change the visual effects (e.g., brightness, transparency, chroma, contrast, or other visual characteristics) of the displayed graphic, and includes various known software components that cause its display. As used herein, the term "graphic" includes any object that can be displayed to a user, and examples of such objects include, but are not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.In some embodiments, the DFR module 350 includes other modules, such as a module that adjusts the sensitivity of the dynamic function row 104, a module that adjusts the audible and / or tactile feedback provided by the dynamic function row 104, a module that adjusts the settings of affordances and information (e.g., size, brightness, font, language, etc.) displayed by the dynamic function row 104, a module that adjusts the current power mode (e.g., normal mode and low power mode) of the dynamic function row 104, and the like.

[0265] In some embodiments, the dynamic function row module 350 interfaces with components that enable it to provide predicted / pre-posed / proposed content items (including, for example, predicted recipients, proposed text end strings, pre-proposed applications, etc.). Pre-posing content items is described in more detail in U.S. Application No. 15 / 167,713, which is hereby incorporated by reference in its entirety.

[0266] Each of the modules and applications identified above corresponds to a set of executable instructions for performing one or more of the functions described above and the methods described in this application (e.g., computer-executed methods and other information processing methods described herein). These modules (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus, in various embodiments, various subsets of these modules may be optionally combined or otherwise rearranged. In some embodiments, the memory 302 optionally stores a subset of the modules and data structures identified above. Further, the memory 302 optionally stores additional modules and data structures not described above.

[0267] Figure 3B is a block diagram of components for event processing according to some embodiments. In some embodiments, memory 302 (Figure 3A) includes an event sorter 370 (e.g., within operating system 326) and an application 340-1 (e.g., any of applications 341, 342, 343, 344, or 345 described above).

[0268] Event sorter 370 receives event information and determines the application 340-1 to which the event information should be delivered and the application view 391 of application 340-1. Event sorter 370 includes an event monitor 371 and an event dispatcher module 374. In some embodiments, application 340-1 includes an application internal state 392, which indicates the current application view(s) to be displayed on display system 312 (e.g., primary display 102 of display portion 110, Figures 1A and / or dynamic function row 104, Figures 1A - 1B) when the application is active or running. In some embodiments, device / global internal state 357 is used by event sorter 370 to determine which application(s) is / are currently active or in focus, and application internal state 392 is used by event sorter 370 to determine the application view 391 to which the event information should be delivered.

[0269] In some embodiments, application internal state 392 includes additional information such as resume information used when application 340-1 resumes execution, user interface state information indicating information being displayed or ready to be displayed by application 340-1, a state waiting queue that enables the user to return to a previous state or view of application 340-1, and a redo / undo waiting queue of previous actions performed by the user.

[0270] The event monitor 371 receives event information from the peripheral device interface 318. The event information includes information about sub-events (e.g., a user touch on the display system 312 when it is a touch-sensitive display as part of a multi-touch gesture). The peripheral device interface 318 transmits information received from sensors such as the I / O subsystem 306, or proximity sensor(s) 366, accelerometer(s) 368, and / or microphone 313 (via the audio circuit 310). The information that the peripheral device interface 318 receives from the I / O subsystem 306 includes information from the display system 312 when it is a touch-sensitive display or another touch-sensitive surface (e.g., touch pad 108, FIGS. 1A - 1B).

[0271] In some embodiments, the event monitor 371 transmits requests to the peripheral device interface 318 at predetermined intervals. In response, the peripheral device interface 318 transmits event information. In other embodiments, the peripheral device interface 318 transmits event information only when there is a significant event (e.g., receiving an input that exceeds a predetermined noise threshold and / or is longer than a predetermined duration).

[0272] In some embodiments, the event sorter 370 also includes a hit view determination module 372 and / or an active event recognition unit determination module 373.

[0273] The hit view determination module 372 provides software procedures for determining where a view is configured from where within one or more views a sub-event occurs, when the display system 312 displays two or more views, and control and other elements viewable by a user on the display.

[0274] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as an application view or user interface window, in which information is displayed and touch gestures occur. The application view (of the application) in which a touch is detected optionally corresponds to a program level within the program hierarchy or view hierarchy of the application. For example, the lowest level view in which a touch is detected is optionally referred to as the hit view, and the set of events recognized as appropriate input is optionally determined at least in part based on the hit view of the initial touch that initiates the touch gesture.

[0275] The hit view determination module 372 receives information related to sub-events of a touch gesture. When the application has a plurality of views configured in a hierarchy, the hit view determination module 372 identifies the hit view as the lowest level view within the hierarchy in which the sub-event is to be processed. In most situations, the hit view is the lowest level view in which the starting sub-event (i.e., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once the hit view is identified by the hit view determination module, the hit view generally receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0276] The active event recognition unit determination module 373 determines which view(s) within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognition unit determination module 373 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognition unit determination module 373 determines that all views including the physical location of the sub-event are views that are actively involved, and thus determines that all views that are actively involved should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely limited to an area associated with one particular view, the upper-level views within the hierarchy still remain views that are actively involved.

[0277] The event dispatcher module 374 dispatches event information to an event recognition unit (e.g., event recognition unit 380). In embodiments including the active event recognition unit determination module 373, the event dispatcher module 374 dispatches event information to the event recognition unit determined by the active event recognition unit determination module 373. In some embodiments, the event dispatcher module 374 stores the event information obtained by each event receiver 382 in an event queue.

[0278] In some embodiments, the operating system 326 includes an event sorter 370. Alternatively, the application 340-1 includes the event sorter 370. In still other embodiments, the event sorter 370 is an independent module or part of another module stored in the memory 302 such as the touch / motion module 330.

[0279] In some embodiments, application 340-1 includes a plurality of event processing units 390 and one or more application views 391, each including instructions for processing touch events that occur within respective views of the user interface of the application. Each application view 391 of application 340-1 includes one or more event recognition units 380. Generally, an application view 391 includes a plurality of event recognition units 380. In other embodiments, one or more of the event recognition units 380 are part of a separate module, such as a user interface kit (not shown) or a higher-level object from which application 340-1 inherits methods and other characteristics. In some embodiments, each event processing unit 390 includes one or more of event data 379 received from data update unit 376, object update unit 377, GUI update unit 378, and / or event sorter 370. The event processing unit 390 optionally utilizes or calls data update unit 376, object update unit 377, or GUI update unit 378 to update the internal state 392 of the application. Alternatively, one or more of the application views 391 include one or more respective event processing units 390. Also, in some embodiments, one or more of data update unit 376, object update unit 377, and GUI update unit 378 are included in application view 391.

[0280] Each event recognition unit 380 receives event information (e.g., event data 379) from event sorter 370 and identifies an event from the event information. The event recognition unit 380 includes an event receiving unit 382 and an event comparing unit 384. In some embodiments, the event recognition unit 380 also includes at least a subset of metadata 383 and event distribution instructions 388 (optionally including sub-event distribution instructions).

[0281] The event receiving unit 382 receives event information from the event sorter 370. The event information includes information about sub-events, for example, information about a touch or a movement of a touch. Depending on the sub-event, the event information also includes additional information such as the location of the sub-event. When the sub-event relates to the movement of a touch, the event information also optionally includes the speed and direction of the sub-event. In some embodiments, the event includes a rotation of the device from one orientation to another (e.g., from portrait to landscape, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the posture of the device).

[0282] The event comparison unit 384 compares event information with the definitions of predefined events or sub-events and, based on the comparison, determines an event or sub-event, or determines or updates the state of an event or sub-event. In some embodiments, the event comparison unit 384 includes an event definition 386. The event definition 386 includes definitions of events (e.g., a predefined sequence of sub-events), such as event 1 (387-1), event 2 (387-2), etc. In some embodiments, the sub-events in event 387 include, for example, the start of a touch, the end of a touch, the movement of a touch, the cancellation of a touch, and multiple touches. In one example, the definition of event 1 (387-1) is a double-tap on a displayed object. A double-tap includes, for example, a first touch (start of touch) at a predefined stage on the displayed object, a first lift-off (end of touch) at a predefined stage, a second touch (start of touch) at a predefined stage on the displayed object, and a second lift-off (end of touch) at a predefined stage. In another example, the definition of event 2 (387-2) is a drag on a displayed object. A drag includes, for example, a touch (or contact) at a predefined stage on the displayed object, movement of the touch across the display system 312 when it is a touch-sensitive display, and lift-off of the touch (end of touch). In some embodiments, an event also includes information regarding one or more associated event processing units 390.

[0283] In some embodiments, event definition 387 includes definitions of events for each user interface object. In some embodiments, event comparison unit 384 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view where three user interface objects are displayed on display system 312, when a touch is detected on display system 312 when it is a touch-sensitive display, event comparison unit 384 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a respective event processing unit 390, the event comparison unit uses the results of the hit test to determine which event processing unit 390 should be activated. For example, event comparison unit 384 selects the sub-event that triggers the hit test and the event processing unit associated with the object.

[0284] In some embodiments, the definition for each event 387 also includes a delay action that delays the delivery of event information until it is determined whether the sequence of sub-events corresponds to the event type of the event recognition unit.

[0285] When each event recognition unit 380 determines that a series of sub-events does not match any of the events in event definition 386, each event recognition unit 380 enters a state of event impossible, event failure, or event end, and then ignores subsequent sub-events of the touch gesture. In this situation, if there is another event recognition unit that remains active for the hit view, that event recognition unit continues to track and process the sub-events of the ongoing touch gesture.

[0286] In some embodiments, each event recognition unit 380 includes metadata 383 having configurable properties, flags, and / or lists indicating how the event delivery system should actively participate in event recognition units that perform sub-event delivery. In some embodiments, metadata 383 includes configurable properties, flags, and / or lists indicating how event recognition units interact with each other or are capable of interacting with each other. In some embodiments, metadata 383 includes configurable properties, flags, and / or lists indicating whether sub-events are delivered to various levels in the view hierarchy or program hierarchy.

[0287] In some embodiments, each event recognition unit 380 activates an event processing unit 390 associated with the event when one or more specific sub-events of the event are recognized. In some embodiments, each event recognition unit 380 delivers event information associated with the event to the event processing unit 390. Activating the event processing unit 390 is separate from sending (and deferring sending) sub-events to each hit view. In some embodiments, the event recognition unit 380 sets a flag associated with the recognized event, and the event processing unit 390 associated with the flag captures the flag and executes a predefined process.

[0288] In some embodiments, the event delivery command 388 includes a sub-event delivery command that delivers event information about sub-events without activating the event processing unit. Instead, the sub-event delivery command delivers event information to an event processing unit associated with a series of sub-events or to a view actively involved, and the event processing unit associated with the series of sub-events or the actively involved view receives the event information and executes a predetermined process.

[0289] In some embodiments, the data update unit 376 creates and updates data used in the application 340-1. For example, the data update unit 376 stores video files used by the media player module 344. In some embodiments, the object update unit 377 creates and updates objects used by the application 340-1. For example, the object update unit 376 creates a new user interface object or updates the position of a user interface object. The GUI update unit 378 updates the GUI. For example, the GUI update unit 378 prepares display information and sends it to the graphic module 332 for display on the display system 312 (e.g., the primary display 102 of the display portion 110, FIGS. 1A and / or the dynamic function row 104, FIGS. 1A-1B).

[0290] In some embodiments, the event processing unit(s) 390 includes or has access to the data update unit 376, the object update unit 377, and the GUI update unit 378. In some embodiments, the data update unit 376, the object update unit 377, and the GUI update unit 378 are included in a single module of the application 340-1 or the application view 391. In other embodiments, they are included in two or more software modules.

[0291] The foregoing discussion regarding event handling of user touches on a touch-sensing display applies to other forms of user input for operating an electronic device 300 using an input device, but it should be understood that not all of them are initiated on a touch screen. For example, movement of a mouse and pressing of a mouse button, contact movement such as tapping, dragging, scrolling on a touch pad, pen stylus input, movement of a device, spoken commands, detected eye movement, biometric input, and / or any combination thereof, optionally in association with single or multiple presses or holds of a keyboard, are used as input corresponding to sub-events that define events to be recognized optionally.

[0292] FIG. 4 is a block diagram of an external electronic device 400 according to some embodiments. In some embodiments, the external electronic device 400 is an external input and output device that at least partially includes a dynamic function row 104, a physical input mechanism such as a set of physical keys (e.g., set of physical keys 106, FIGS. 2A-2B), and / or a touch pad (e.g., touch pad 108, FIGS. 2B-2C) within the same housing. Examples of the external electronic device 400 include an external keyboard (e.g., external keyboard 206, FIGS. 2A-2B), an external touch-sensing surface (e.g., first external input mechanism 212, FIG. 2C), or other external input mechanisms (e.g., second external input mechanism 222, FIG. 2D). The external electronic device 400 is communicatively coupled to a computing device 202 (FIGS. 2A-2D). For example, the external electronic device 400 is communicatively coupled to the computing device 202 via a wired connection such as USB or PS / 2, or via a wireless communication link using a communication protocol such as Bluetooth, Wi-Fi. The external electronic device 400 may rely on some of the components or procedures in the electronic device 300 (FIG. 3A), or some of these components or procedures may be completed, located, or stored therein by the external electronic device 400 instead of the electronic device 300.

[0293] In some embodiments, the external electronic device 400 includes one or more of a memory 402 (optionally including one or more computer-readable storage media), a memory controller 422, one or more processing units (CPUs) 420, a peripheral device interface 418, an RF circuit 408, an audio circuit 410, a speaker 411, a microphone 413, an input / output (I / O) subsystem 406, other input or control devices 416, and an external port 424. The external electronic device 400 includes a touch-sensing display system 412 (which may also be referred to herein as a "touch-sensitive display", "touch screen", or "touch screen display") (e.g., dynamic function row 104, FIGS. 2A-2D).

[0294] The external electronic device 400 optionally includes one or more intensity sensors 465 that detect the intensity of contact on a touch-sensing surface, such as the touch-sensing display system 412 or a touch pad (e.g., touch pad 108, FIGS. 2B-2C). The external electronic device 400 optionally includes one or more haptic output generators 467 that generate haptic output on a touch-sensing surface, such as the touch-sensing display system 412 or a touch pad (e.g., touch pad 108, FIGS. 2B-2C). These components optionally communicate via one or more communication buses or signal lines 403.

[0295] Memory 402 optionally includes high-speed random access memory and also optionally includes non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state memory devices. Access to memory 402 by other components of the external electronic device 400, such as CPU(s) 420 and peripheral interface 418, is optionally controlled by memory controller 422. Peripheral interface 418 may be used to couple CPU(s) 420 and memory 402 to I / O subsystem 406 and other circuitry. One or more processing units 420 perform various functions for the external electronic device 400 and operate or execute various software programs and / or instruction sets stored in memory 402 to process data. In some embodiments, peripheral interface 418, CPU(s) 420, and memory controller 422 are optionally implemented on a single chip such as chip 404. In some other embodiments, they are optionally implemented on separate chips.

[0296] The RF (radio frequency) circuit 408 transmits and receives RF signals, also called electromagnetic signals. The RF circuit 408 converts an electrical signal into an electromagnetic signal or an electromagnetic signal into an electrical signal, and communicates with a communication network and other communication devices via the electromagnetic signal. The RF circuit 408 optionally includes well-known circuits that perform these functions, such as an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, a memory, etc., but is not limited thereto. The wireless communication optionally uses any one of a plurality of communication standards, communication protocols, and communication technologies, such as near field communication (NFC), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Wi-MAX, or any other suitable communication protocol including communication protocols not yet developed as of the filing date of this document, but is not limited thereto.

[0297] Optional audio circuit 410, speaker 411, and microphone 413 provide an audio interface between the user and the external electronic device 400. The audio circuit 410 receives audio data from the peripheral device interface 418, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 411. The speaker 411 converts the electrical signal into human audible sound waves. The audio circuit 410 also receives an electrical signal converted from sound waves by the microphone 413. The audio circuit 410 converts the electrical signal into audio data and transmits the audio data to the peripheral device interface 418 for processing. The audio data is optionally obtained from and / or transmitted to the memory 402 and / or the RF circuit 408 by the peripheral device interface 418. In some embodiments, the audio circuit 410 also includes a headset jack. The headset jack provides an interface between the audio circuit 410 and a removable audio input / output peripheral device such as an output-only headset, or a headset with both output (e.g., headphones for one or both ears) and input (e.g., a microphone).

[0298] The I / O subsystem 406 couples input / output peripheral devices of the external electronic device 400, such as the touch sensing display system 412 (e.g., the dynamic function row 104, FIGS. 2A-2D), to the peripheral device interface 418. The I / O subsystem 406 optionally includes one or more input controllers 460 for the display controller 456, the intensity sensor controller 459, the tactile feedback controller 461, and other input or control devices 416. The one or more other input controllers 460 receive electrical signals from and transmit electrical signals to the other input or control devices 416. The other input or control devices 416 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, a set of physical keys, a touch pad, and the like.

[0299] A touch-sensing display system 412 (e.g., dynamic function row 104, FIGS. 2A-2D) provides an input / output interface between an external electronic device 400 and a user. A touch-sensitive display (TSD) controller 456 receives electrical signals from and / or transmits electrical signals to the touch-sensing display system 412. The touch-sensing display system 412 displays a visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively referred to as “graphics”). In some embodiments, some or all of the visual output corresponds to user interface objects / elements.

[0300] The touch-sensing display system 412 (e.g., dynamic function row 104, FIGS. 2A-2D) includes a touch-sensing surface, sensors, or a set of sensors that receive input from the user based on tactile and / or haptic contact. Thus, the touch-sensing display system 412 and the TSD controller 456 (along with any associated modules and / or instruction sets in the memory 402) detect contact (and any movement or interruption of the contact) on the touch-sensing display system 412 and convert the detected contact into a signal that is used to select or control user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensing display system 412. In an exemplary embodiment, the point of contact between the touch-sensing display system 412 and the user corresponds to the area of the touch-sensing display system 412 that is in contact with the user's finger.

[0301] The touch sensing display system 412 (e.g., dynamic function row 104, FIGS. 2A-2D) optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, LED (light emitting diode) technology, or OLED (organic light emitting diode) technology, although in other embodiments other display technologies are used. The touch sensing display system 412 and the TSD controller 456 optionally use any of a plurality of touch sensing technologies currently known or developed in the future to detect contact and any movement or interruption of that contact, and those touch sensing technologies include capacitive technology, resistive technology, infrared technology, and surface acoustic wave technology, as well as other proximity sensor arrays or other elements that determine one or more contact points with the touch sensing display system 412, but are not limited thereto. In an exemplary embodiment, a projected mutual capacitance sensing technology as seen in the iPHONE, iPOD TOUCH, and iPAD from Apple Inc. of Cupertino, California is used.

[0302] The touch sensing display system 412 (e.g., dynamic function row 104, FIGS. 2A-2D) optionally has a video resolution exceeding 400 dpi (e.g., 500 dpi, 800 dpi, or more). In some embodiments, the user makes contact with the touch sensing display system 412 using a stylus, finger, etc. In some embodiments, the user interface is designed to primarily operate with finger contact and gestures.

[0303] In some embodiments, in addition to the touch sensing display system 412, the external electronic device 400 optionally includes a touch pad (e.g., touch pad 108, FIGS. 2B-2C). In some embodiments, the touch pad, unlike the touch sensing display system 412, is a touch sensing area of the external electronic device 400 that does not display visual output. In some embodiments, the touch pad is optionally a touch sensing surface separate from the touch sensing display system 412, or an extension of the touch sensing surface formed by the touch sensing display system 412.

[0304] The external electronic device 400 also includes a power system 462 that supplies power to various components. The power system 462 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC), etc.), a recharge system, a power outage detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power within a portable device.

[0305] The external electronic device 400 also optionally includes one or more contact intensity sensors 465 coupled to an intensity sensor controller 459 within the I / O subsystem 406. The contact intensity sensor(s) 465 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrokinetic sensors, piezoelectric sensors, optomechanical sensors, capacitive touch sensing surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of contact on a touch sensing surface). The contact intensity sensor(s) 465 receive contact intensity information (e.g., pressure information or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is juxtaposed with or in proximity to a touch sensing surface (e.g., touch sensing display system 412 and / or touch pad 108, FIGS. 2B-2C).

[0306] The external electronic device 400 also optionally includes one or more haptic output generators 467 coupled to a haptic feedback controller 461 within the I / O subsystem 406. The haptic output generator(s) 467 optionally include one or more electroacoustic devices, such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components that convert an electrical signal into a haptic output) on the device. The contact intensity sensor(s) 465 receive haptic feedback generation commands from the haptic feedback module 433 and generate haptic output that can be sensed by a user of the external electronic device 400. In some embodiments, at least one haptic output generator is juxtaposed with or proximate to a touch sensing surface (e.g., the touch sensing display system 412 and / or the touch pad 108, FIGS. 2B-2C) and optionally generates haptic output by moving the touch sensing surface in a vertical direction (e.g., in / out of the surface of the external electronic device 400) or in a horizontal direction (e.g., back and forth within the same plane as the surface of the external electronic device 400).

[0307] In some embodiments, the software components stored in the memory 402 include an operating system 426, a communication module 428 (or instruction set), a contact / motion module 430 (or instruction set), and a dynamic function row module 450 (or instruction set). Additionally, in some embodiments, the memory 402 stores a device state 457 that includes a display state indicating what view or other information occupies various regions of the touch sensing display system 412 (e.g., the dynamic function row 104, FIGS. 2A-2D).

[0308] The operating system 426 includes various software components and / or drivers that control and manage common system tasks (e.g., memory management, storage device control, power management, etc.), and facilitate communication between various hardware components and software components.

[0309] The communication module 428 facilitates communication with other devices (e.g., computing device 202, FIGS. 2A - 2D) via one or more external ports 424 and / or RF circuit 408, and includes various software components that transmit / receive data via RF circuit 408 and / or external port 424. The external port 424 (e.g., Universal Serial Bus (USB), FIREWIRE, etc.) is adapted to couple directly or indirectly via a network (e.g., the Internet, wireless LAN, etc.) to other devices.

[0310] The contact / motion module 430 optionally detects contact with a touch sensing display system 412 and other touch sensing devices (e.g., a touch pad or a physical click wheel). The contact / motion module 430 performs various software components related to the detection of contact, such as determining whether contact has occurred (e.g., detecting an event of a finger being lowered), determining the intensity of the contact (e.g., the force or pressure of the contact, or an alternative to the force or pressure of the contact), determining whether there is movement of the contact, tracking movement across the touch sensing surface (e.g., detecting one or more events of a finger being dragged), and determining whether the contact has stopped (e.g., detecting an event of a finger being raised or an interruption of the contact). The contact / motion module 430 receives contact data from the touch sensing surface. Determining the movement of the contact point, represented by a series of contact data, optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These operations are optionally applied to a single contact (e.g., contact with one finger) or multiple simultaneous contacts (e.g., "multi-touch" / contact with multiple fingers). In some embodiments, the contact / motion module 430 also detects contact on a touch pad (e.g., touch pad 108, FIGS. 2B-2C).

[0311] In some embodiments, the contact / motion module 430 uses a set of one or more intensity thresholds for determining whether an action has been performed by the user (e.g., for determining whether the user has selected or "clicked" on an affordance). In some embodiments, at least a subset of the intensity thresholds are determined according to software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of the external electronic device 400). For example, the mouse "click" threshold of a trackpad or touch screen display can be set to any of a wide range of default thresholds without changing the hardware of that trackpad or touch screen display. Additionally, in some implementations, the user of the device is provided with software settings for adjusting one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once via a system-level click "intensity" parameter).

[0312] The contact / motion module 430 optionally detects gesture inputs by the user. Different gestures on the touch-sensing surface have different contact patterns (e.g., different movements, timings, and / or intensities of detected contacts). Thus, gestures are optionally detected by detecting a particular contact pattern. For example, detecting a finger tap contact includes detecting an event of the finger touching down and subsequently detecting an event of the finger lifting off (lifting off) at the same position (or substantially the same position) as the event of the finger touching down (e.g., at the position of an icon). As another example, detecting a finger swipe gesture on the touch-sensing surface includes detecting an event of the finger touching down, subsequently detecting one or more events of the finger being dragged, and in some embodiments, also subsequently detecting an event of the finger lifting off (lifting off).

[0313] The tactile feedback module 433 includes various software components that generate instructions for use by the tactile output generator(s) 467 to generate tactile output at one or more locations on the external electronic device 400 in response to a user interaction with the external electronic device 400.

[0314] The Dynamic Function Row (DFR) module 450 includes a focus acquisition module 451, a DFR determination module 452, and a DFR presentation module 453. In some embodiments, the focus acquisition module 451 is configured to obtain an indication of an active user interface element that is the current focus of the graphical user interface displayed on the primary display 102 of the external display device 204 (FIGS. 2A-2D) from the computing device 202 (FIGS. 2A-2D). In some embodiments, the DFR determination module 452 is configured to determine a graphic (e.g., a set of one or more affordances) based on the active user interface element that is the current focus. Alternatively, in some embodiments, the computing device 202 (FIGS. 2A-2D) determines a graphic (e.g., a set of one or more affordances) based on the active user interface element that the focus is on, and provides the graphic to the external electronic device 400 or its components (e.g., the DFR module 450) for display on the touch-sensing display system 412 (e.g., the dynamic function row 104, FIGS. 2A-2D). In some embodiments, the DFR presentation module 453 is configured to render the graphic determined by the DFR determination module 452 (or provided by the computing device 202) on the touch-sensing display system 412 (e.g., the dynamic function row 104, FIGS. 2A-2D). The DFR presentation module 453 renders the graphic on the touch-sensing display system 412, including components that change the visual effects (e.g., brightness, transparency, chroma, contrast, or other visual characteristics) of the displayed graphic, and includes various known software components that cause its display.In some embodiments, the DFR module 450 includes other modules such as a module that adjusts the sensitivity of the dynamic function row 104, a module that adjusts the audible and / or tactile feedback provided by the dynamic function row 104, a module that adjusts the settings of affordances and information (e.g., size, brightness, font, language, etc.) displayed by the dynamic function row 104, and a module that adjusts the current power mode (e.g., normal mode and low power mode) of the dynamic function row 104.

[0315] In some embodiments, the memory 402 includes an event sorter 470 (e.g., within the operating system 426). In some embodiments, the event sorter 470 performs the same functions as the event sorter 370 (FIG. 3B) and includes a subset or superset of the modules, procedures, and instructions of the event sorter 370 (FIG. 3B). Therefore, the event sorter 470 is not described for the sake of brevity.

[0316] The external electronic device 400 is merely an example. It should be understood that the external electronic device 400 may optionally have more or fewer components than those shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of those components. The various components shown in FIG. 4 are implemented in hardware, software, firmware, or a combination thereof, including one or more signal processing circuits and / or application-specific integrated circuits.

[0317] Each of the identified modules described above corresponds to a set of executable instructions for performing one or more of the functions described above and the methods described in this application (e.g., methods performed by a computer and other information processing methods described herein). These modules (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may optionally be combined or rearranged in various embodiments. In some embodiments, memory 402 optionally stores a subset of the modules and data structures identified above. Additionally, memory 402 optionally stores additional modules and data structures not described above.

[0318] As used herein, the term "focus selector" refers to an input element that indicates the current part of the user interface with which the user is interacting. In some implementations that include a cursor or other location marker, the cursor acts as the "focus selector" such that, while the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), if an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 355 in FIG. 3 or touch-sensitive surface 451 in FIG. 4B), the particular user interface element is adjusted according to the detected input. In some implementations that include a touch screen display that enables direct interaction with user interface elements on the touch screen display, a contact detected on the touch screen acts as the "focus selector" such that, if an input (e.g., a press input by the contact) is detected on the touch screen display at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations, the focus moves from one area of the user interface to another area of the user interface without a corresponding movement of the cursor or movement of the contact on the touch screen display (e.g., by using a tab key or arrow keys to move the focus from one button to another). In these implementations, the focus selector moves according to the movement of the focus between different areas of the user interface. Regardless of the specific form taken by the focus selector, the focus selector is generally a user interface element (or a contact on the touch screen display) that is controlled by the user to communicate the user's intended interaction with the user interface (e.g., by indicating to the device the element of the user interface with which the user is attempting to interact).For example, the position of a focus selector (e.g., a cursor, a contact, or a selection box) over each button while a press input is detected on a touch sensing surface (e.g., a touch pad or a touch screen) indicates that the user is attempting to activate each button (as contrasted with other user interface elements shown on the device's display).

[0319] As used in this specification and the claims, the term "intensity" of a contact on a touch sensing surface refers to the force or pressure (force per unit area) of a contact (e.g., finger contact or stylus contact) on the touch sensing surface, or an alternative (proxy) for the force or pressure of a contact on the touch sensing surface. The intensity of a contact has a range of values that includes at least four distinct values and more typically includes hundreds (e.g., at least 256) of distinct values. The intensity of a contact is optionally specified (or measured) using a variety of techniques and a variety of sensors or combinations of sensors. For example, one or more force sensors under or adjacent to the touch sensing surface are optionally used to measure the force at various points on the touch sensing surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted average or sum) to specify the estimated contact force. Similarly, a pressure-sensitive tip of a stylus is optionally used to specify the pressure of the stylus on the touch sensing surface. Alternatively, the size and / or change in size of the contact area detected on the touch sensing surface, the capacitance and / or change in capacitance of the touch sensing surface proximate to the contact, and / or the resistance and / or change in resistance of the touch sensing surface proximate to the contact are optionally used as an alternative for the force or pressure of a contact on the touch sensing surface. In some implementations, the alternative measurement for the force or pressure of a contact is used directly to determine whether it exceeds an intensity threshold (e.g., the intensity threshold is described in units corresponding to the alternative measurement). In some implementations, the alternative measurement for the force or pressure of a contact is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether it exceeds an intensity threshold (e.g., the intensity threshold is a pressure threshold measured in units of pressure).Using the intensity of contact as an attribute of user input enables a user to access additional device functions that may not otherwise be easily accessible to the user on a device of limited or reduced size where assets for displaying affordances (e.g., on a touch-sensitive display) and / or receiving user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls such as knobs or buttons) are restricted.

[0320] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by the user (e.g., to determine whether the user has "clicked" on an icon). In some embodiments, at least a subset of the intensity thresholds is determined according to software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of specific physical actuators and can be adjusted without changing the physical hardware of the system 100). For example, the mouse "click" threshold for a trackpad or touchscreen display can be set to any of a wide range of default thresholds without changing the hardware of the trackpad or touchscreen display. Further, in some implementations, a user of the device is provided with software settings for adjusting one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once with a system-level click "intensity" parameter).

[0321] As used herein and in the claims, the term "characteristic strength" of a contact refers to the characteristics of that contact based on one or more strengths of the contact. In some embodiments, the characteristic strength is based on a plurality of strength samples. The characteristic strength is optionally based on a set of strength samples collected during a predetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) associated with a predetermined number of strength samples, i.e., a predetermined event (e.g., after detecting a contact, before detecting a lift-off of the contact, before or after detecting a start of movement of the contact, before detecting an end of the contact, before or after detecting an increase in the strength of the contact, and / or before or after detecting a decrease in the strength of the contact). The characteristic strength of a contact is optionally based on one or more of a maximum value of the strength of the contact, a median value of the strength of the contact, an average value of the strength of the contact, a top 10 percent value of the strength of the contact, a value that is half of the maximum value of the strength of the contact, a value that is 90 percent of the maximum value of the strength of the contact, etc. In some embodiments, the duration of the contact is used when specifying the characteristic strength (e.g., when the characteristic strength is an average of the strength of the contact over time). In some embodiments, the characteristic strength is compared to a set of one or more strength thresholds to determine whether an operation has been performed by a user. For example, the set of one or more strength thresholds may include a first strength threshold and a second strength threshold. In this example, a contact having a characteristic strength that does not exceed the first threshold results in a first operation, a contact having a characteristic strength that exceeds the first strength threshold and does not exceed the second strength threshold results in a second operation, and a contact having a characteristic strength that exceeds the second strength threshold results in a third operation. In some embodiments, the comparison between the characteristic strength and the one or more strength thresholds is not used to determine whether to perform a first operation or a second operation, but rather is used to determine whether to perform one or more operations (e.g., whether to perform each option or refrain from performing each operation).

[0322] In some embodiments, for the purpose of identifying the characteristic intensity, a portion of the gesture is identified. For example, the touch sensing surface may receive a continuous swipe contact (e.g., a drag gesture) that transitions from a start position to reach an end position, and at that point the intensity of the contact increases. In this example, the characteristic intensity of the contact at the end position may be based on only a portion of the continuous swipe contact (e.g., only a portion of the swipe contact at the end position) rather than the entire swipe contact. In some embodiments, a smoothing algorithm may be applied to the intensity of the swipe contact before identifying the characteristic intensity of the contact. For example, the smoothing algorithm may optionally include one or more of a non - weighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some situations, these smoothing algorithms eliminate small increases or decreases in the intensity of the swipe contact for the purpose of identifying the characteristic intensity.

[0323] In some embodiments, one or more predetermined intensity thresholds are used to determine whether a particular input meets an intensity - based criterion. For example, the one or more predetermined intensity thresholds include (i) a contact detection intensity threshold IT0, (ii) a light press intensity threshold IT L , (iii) a deep press intensity threshold IT (e.g., at least initially higher than I L ), and / or (iv) one or more other intensity thresholds (e.g., an intensity threshold I lower than I D ), and / or (iv) one or more other intensity thresholds (e.g., an intensity threshold I lower than I L ), and / or (iv) one or more other intensity thresholds (e.g., an intensity threshold I lower than I H) can be mentioned. In some embodiments, the light pressing intensity threshold generally corresponds to the intensity at which the device performs an operation associated with clicking a button or trackpad of a physical mouse. In some embodiments, the deep pressing intensity threshold generally corresponds to the intensity at which the device performs an operation different from the operation associated with clicking a button or trackpad of a physical mouse. In some embodiments, when a contact having a characteristic intensity below the light pressing intensity threshold (for example, exceeding a nominal contact detection intensity threshold IT0 below which the contact is no longer detected) is detected, the device moves the focus selector according to the movement of the contact on the touch sensing surface without performing an operation associated with the light pressing intensity threshold or the deep pressing intensity threshold. Generally, unless otherwise specified, these intensity thresholds are consistent among various sets of user interface diagrams.

[0324] In some embodiments, the device's response to an input detected by the device depends on a criterion based on the contact intensity during the input. For example, in the case of some "light press" inputs, a contact intensity exceeding a first intensity threshold during the input triggers a first response. In some embodiments, the device's response to an input detected by the device depends on a criterion that includes both the contact intensity and a time-based criterion during the input. For example, in the case of some "deep press" inputs, a contact intensity exceeding a second intensity threshold during the input that is greater than the first intensity threshold for light presses triggers a second response only if a delay time has elapsed between satisfying the first intensity threshold and satisfying the second intensity threshold. This delay time is typically less than 200 ms in duration (e.g., 40 ms, 100 ms, or 120 ms depending on the magnitude of the second intensity threshold, and the delay time increases as the second intensity threshold increases). This delay time helps to avoid accidental deep press inputs. As another example, in the case of some "deep press" inputs, there is a time period of reduced sensitivity that occurs after the time when the first intensity threshold is satisfied. During the time period of reduced sensitivity, the second intensity threshold increases. This temporary increase in the second intensity threshold also helps to avoid accidental deep press inputs. In the case of other deep press inputs, the response to the detection of the deep press input does not depend on a time-based criterion.

[0325] In some embodiments, one or more of the input intensity thresholds and / or the corresponding outputs vary based on one or more factors such as user settings, movement of the contact, input timing, the application being executed, the rate at which the intensity is applied, the number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), the position of the focus selector, etc. Exemplary factors are described in U.S. Patent Application Publication Nos. 14 / 399,606 and 14 / 624,296, which are hereby incorporated by reference in their entirety.

[0326] For example, FIG. 3C shows a dynamic intensity threshold 480 that changes over time, based in part on the intensity of touch input 476 over time. The dynamic intensity threshold 480 is the sum of two components: a first component 474 that decays over time after a predetermined delay time p1 from when the touch input 476 was first detected, and a second component 478 that follows the trace of the intensity of the touch input 476 over time. The first high intensity threshold of the first component 474 reduces the accidental triggering of the "deep press" response while enabling an immediate "deep press" response when the touch input 476 provides sufficient intensity. The second component 478 reduces the unintentional triggering of the "deep press" response due to the gradual intensity variations of the touch input. In some embodiments, when the touch input 476 meets the dynamic intensity threshold 480 (e.g., at point 481 in FIG. 3C), a "deep press" response is triggered.

[0327] FIG. 3D shows another dynamic intensity threshold 486 (e.g., intensity threshold I D ). FIG. 3D also shows two other intensity thresholds: a first intensity threshold I H and a second intensity threshold I L . In FIG. 3D, the touch input 484 meets the first intensity threshold I H and the second intensity threshold I L before time p2, but no response is provided until the delay time p2 has elapsed at time 482. Also in FIG. 3D, the dynamic intensity threshold 486 decays over time, and the decay starts at time 488 after a predetermined delay time p1 has elapsed from time 482 (when the response associated with the second intensity threshold I L is triggered). This type of dynamic intensity threshold reduces the accidental triggering of the response associated with the dynamic intensity threshold I H or the second intensity threshold I L immediately after or simultaneously with triggering a response associated with a lower intensity threshold such as the first intensity threshold I D .

[0328] FIG. 3E shows yet another dynamic intensity threshold 492 (e.g., intensity threshold I D ). In FIG. 3E, the intensity threshold I LThe response associated with L is triggered after a delay time p2 has elapsed since the touch input 490 was first detected. At the same time, the dynamic intensity threshold 492 decays after a predetermined delay time p1 has elapsed since the touch input 490 was first detected. Therefore, without releasing the touch input 490, an increase in the intensity of the touch input 490 is followed by the intensity threshold I L even when the intensity of the touch input 490 falls below another intensity threshold, e.g., the intensity threshold I D associated with

[0329] The increase in the characteristic intensity of the contact from an intensity below the light press intensity threshold IT L to an intensity between the light press intensity threshold IT L and the deep press intensity threshold IT D may be referred to as a "light press" input. The increase in the characteristic intensity of the contact from an intensity below the deep press intensity threshold IT D to an intensity above the deep press intensity threshold IT D may be referred to as a "deep press" input. The increase in the characteristic intensity of the contact from an intensity below the contact detection intensity threshold IT0 to an intensity between the contact detection intensity threshold IT0 and the light press intensity threshold IT L may be referred to as the detection of contact on the touch surface. The decrease in the characteristic intensity of the contact from an intensity above the contact detection intensity threshold IT0 to an intensity below the contact detection intensity threshold IT0 may be referred to as the detection of lift-off of the contact from the touch surface. In some embodiments, IT0 is zero. In some embodiments, IT0 is greater than zero. In some examples, a shaded circle or ellipse is used to represent the intensity of the contact on the touch sensing surface. In some examples, an unshaded circle or ellipse is used to represent each contact on the touch sensing surface without specifying the intensity of each contact.

[0330] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a respective pressing input, or in response to detecting a respective pressing input that is performed at a respective contact (or contacts), where each pressing input is detected based at least in part on detecting an increase in the intensity of a contact (or contacts) that exceeds a pressing input intensity threshold. In some embodiments, each operation is performed in response to detecting an increase in the intensity of a respective contact that exceeds a pressing input intensity threshold (e.g., each operation is performed on the "downstroke" of a respective pressing input). In some embodiments, a pressing input includes an increase in the intensity of a respective contact that exceeds a pressing input intensity threshold and a subsequent decrease in the intensity of the contact that falls below the pressing input intensity threshold, and each operation is performed in response to detecting the subsequent decrease in the intensity of a respective contact that falls below a pressing input threshold (e.g., each operation is performed on the "upstroke" of a respective pressing input).

[0331] In some embodiments, the device employs intensity hysteresis to avoid spurious inputs sometimes referred to as "jitter", and the device defines or selects a hysteresis intensity threshold having a predefined relationship to the press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable percentage of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of each contact above the press input intensity threshold and a subsequent decrease in the intensity of the contact below the hysteresis intensity threshold corresponding to the press input intensity threshold, and each operation is performed in response to detecting a subsequent decrease in the intensity of each contact below the hysteresis intensity threshold (e.g., each operation is performed on the "upstroke" of each press input). Similarly, in some embodiments, a press input is detected only when the device detects an increase in the intensity of a contact from an intensity below the hysteresis intensity threshold to an intensity above the press input intensity threshold and optionally a subsequent decrease in the intensity of the contact to an intensity below the hysteresis intensity, and each operation is performed in response to detecting the press input (e.g., an increase in the intensity of the contact or a decrease in the intensity of the contact, depending on the situation).

[0332] For ease of explanation, descriptions of operations performed in response to a press input associated with a press input intensity threshold, or in response to a gesture that includ...

Claims

1. 1. A computing system comprising one or more processors, a memory, a first housing including a primary display, and a second housing including, at least in part, (i) a physical keyboard, and (ii) a touch-sensitive secondary display separate from the primary display, Receiving a request to open an application; In response to receiving the request, displaying on the primary display a plurality of user interface objects associated with the application, the plurality including a first user interface object displayed with content associated therewith and other user interface objects displayed without content associated therewith; displaying on the touch-sensitive secondary display a set of affordances, each affordance representing one of the plurality of user interface objects; detecting a swipe gesture via the touch-sensitive secondary display in a direction from a first affordance of the set of affordances to a second affordance of the set of affordances, the first affordance representing the first user interface object and the second affordance representing a second user interface object distinct from the first user interface object; updating the primary display to cease displaying content associated with the first user interface object and to display content associated with the second user interface object in response to detecting the swipe gesture; A method comprising:

2. detecting continued movement of the swipe gesture across the touch-sensitive secondary display, the swipe gesture including the swipe gesture contacting a third affordance representing a third user interface object; updating the primary display to display associated content of the third user interface object in response to detecting the swipe gesture contacting the third affordance; The method of claim 1 further comprising:

3. The method of claim 2 , wherein each affordance in the set of affordances includes a representation of a respective associated content for a respective user interface object of the plurality.

4. detecting an initial contact with the touch-sensitive secondary display over the first affordance prior to detecting the swipe gesture; In response to detecting the initial contact, increasing a magnification level of the first affordance; The method of claim 3 further comprising:

5. the application is a web browsing application; The method of claim 1 , wherein the plurality of user interface objects each correspond to a web browsing tab.

6. Detecting input in a uniform resource locator (URL) input portion of the web browsing application on the primary display; updating the touch-sensitive secondary display to include a representation of a favorite URL in response to detecting the input; The method of claim 5 further comprising:

7. the application is a photo browsing application; The method of claim 1 , wherein the plurality of user interface objects each correspond to a respective photograph.

8. the application is a video editing application; The method of claim 1 , wherein the plurality of user interface objects each correspond to an individual frame in a respective video.

9. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors of a computing system having a first housing including a primary display and a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, cause the computing system to perform the method of any one of claims 1 to 8.

10. one or more processors; a first housing including a primary display; a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display; A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 1 to 8; An electronic device comprising:

11. A graphical user interface on a computing system having one or more processors, a memory, a first housing including a primary display, and a second housing including, at least in part, (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, wherein the one or more processors execute one or more programs stored in the memory, and the graphical user interface comprises a user interface displayed in accordance with any one of the methods of claims 1 to 8.

12. 9. An electronic device having a first housing including a primary display and a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, the electronic device comprising means for performing the method of any one of claims 1 to 8.

13. 9. An information processing apparatus for use in an electronic device including a first housing including a primary display and a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, the electronic device comprising means for performing the method of any one of claims 1 to 8.

14. 1. A computing system comprising one or more processors, a memory, a first housing including a primary display, and a second housing including, at least in part, (i) a physical keyboard, and (ii) a touch-sensitive secondary display separate from the primary display, receiving a request to search within content displayed on the primary display of the computing device; In response to receiving the request, displaying on the primary display a plurality of search results responsive to the search, with focus on a first search result of the plurality of search results; displaying on the touch-sensitive secondary display a respective representation corresponding to each search result of the plurality of search results; detecting a touch input via the touch-sensitive secondary display selecting a representation of the respective representation, the representation corresponding to a second search result of the plurality of search results that is distinct from the first search result; and in response to detecting the input, changing focus on the primary display to the second search result; A method comprising:

15. The method of claim 14 , wherein changing focus includes modifying visual characteristics of the particular search result on the primary display.

16. detecting a gesture moving across at least two of the respective representations on the touch-sensitive secondary display; in response to detecting the gesture, changing focus on the primary display to respective search results corresponding to the at least two of the respective representations as the swipe gesture moves across the at least two of the respective representations; The method of claim 15 further comprising:

17. 17. The method of claim 16, further comprising: changing focus on the primary display to a respective search result in addition to one corresponding to the at least two of the respective representations in accordance with a determination that a velocity of the gesture exceeds a threshold velocity.

18. The method according to claim 16 or 17, wherein the gesture is a swipe gesture.

19. The method according to claim 16 or 17, wherein the gesture is a flick gesture.

20. The method of claim 14 , wherein the representations are tick marks respectively corresponding to respective ones of the search results.

21. The method of claim 20 , wherein the tick marks are displayed in rows on the touch-sensitive secondary display in an order that corresponds to an ordering of the search results on the primary display.

22. 18. The method of claim 14, wherein the request to search within the content is a request to locate a search string within the content, and the plurality of search results each include at least the search string.

23. 23. The method of claim 22, wherein displaying the plurality of search results includes highlighting the search string for each of the plurality of search results.

24. 24. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors of a computing system having a first housing including a primary display and a second housing including, at least in part, (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, cause the computing system to perform the method of any one of claims 14 to 23.

25. one or more processors; a first housing including a primary display; a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display; A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 14 to 23; and An electronic device comprising:

26. 24. A graphical user interface on a computing system having one or more processors, a memory, a first housing including a primary display, and a second housing including, at least in part, (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, wherein the one or more processors execute one or more programs stored in the memory, and the graphical user interface comprises a user interface displayed in accordance with any one of the methods of claims 14 to 23.

27. 24. An electronic device having a first housing including a primary display and a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, the electronic device comprising means for performing a method according to any one of claims 14 to 23.

28. 24. An information processing apparatus for use in an electronic device including a first housing including a primary display and a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, the electronic device comprising means for performing the method of any one of claims 14 to 23.

29. 1. A method included in a computing system comprising one or more processors, a memory, a first housing including a primary display, and a second housing including, at least in part, (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, the method comprising: displaying a calendar application on the primary display; receiving a request to display information about an event associated with the calendar application; In response to receiving the request, displaying event details for the first event on the primary display, the event details including a start time and an end time for the event; displaying an affordance on the touch-sensitive secondary display, the affordance indicating a range of time that includes at least the start time and the end time; A method comprising:

30. detecting, via the touch-sensitive secondary display, an input at the user interface control that modifies the range of time; In response to detecting the input, modifying at least one of the start time and the end time of the event according to the input; displaying a revised range of times for the event on the primary display according to the input; and 30. The method of claim 29, further comprising:

31. 31. The method of claim 30, further comprising saving the event along with the modified start and / or end times in the memory of the computing system.

32. 32. The method of claim 29, wherein the input that modifies the range of time is a press input that moves at least a portion of the affordance after maintaining contact with the affordance on the touch-sensitive secondary display for more than a threshold time.

33. 32. The method of claim 29, wherein the input modifying the range of time is a swipe gesture moved across the touch-sensitive secondary display to cause the computing system to select a new start time and a new end time for the event, the new start and end times corresponding to a time slot of the same duration as covered by the start and end times.

34. 34. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors of a computing system having a first housing including a primary display and a second housing including, at least in part, (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, cause the computing system to perform the method of any one of claims 29 to 33.

35. one or more processors; a first housing including a primary display; a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display; A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing a method according to any one of claims 29 to 33; An electronic device comprising:

36. 34. A graphical user interface on a computing system having one or more processors, a memory, a first housing including a primary display, and a second housing including, at least in part, (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, wherein the one or more processors execute one or more programs stored in the memory, and the graphical user interface comprises a user interface displayed in accordance with any one of the methods of claims 29 to 33.

37. 34. An electronic device having a first housing including a primary display and a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, the electronic device comprising means for performing a method according to any one of claims 29 to 33.

38. 34. An information processing apparatus for use in an electronic device including a first housing including a primary display and a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, the electronic device comprising means for performing a method according to any one of claims 29 to 33.

39. 1. A method included in a computing system comprising one or more processors, a memory, a first housing including a primary display, and a second housing including, at least in part, (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, the method comprising: Detecting a new connection between the computing system and an external device separate from the computing system; in response to detecting the new connection, displaying on the touch-sensitive secondary display a plurality of affordances corresponding to functionality available via the external device; A method comprising:

40. receiving via the touch-sensitive secondary display a selection of a first affordance corresponding to a first function available via the external device; initiating execution of the first function in response to receiving the selection; 40. The method of claim 39, further comprising:

41. 41. The method of claim 40, wherein the external device is an additional display separate from the primary display and the touch-sensitive display.

42. 42. The method of claim 41, wherein the plurality of affordances includes a first affordance that, when selected, causes the computing system to initiate performance of a display mirroring function via the additional display.

43. 43. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors of a computing system having a first housing including a primary display and a second housing including, at least in part, (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, cause the computing system to perform the method of any one of claims 39 to 42.

44. one or more processors; a first housing including a primary display; a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display; A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing a method according to any one of claims 39 to 42; and An electronic device comprising:

45. 43. A graphical user interface on a computing system having one or more processors, a memory, a first housing including a primary display, and a second housing including, at least in part, (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, wherein the one or more processors execute one or more programs stored in the memory, and the graphical user interface comprises a user interface displayed in accordance with any one of the methods of claims 39 to 42.

46. 43. An electronic device having a first housing including a primary display and a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, the electronic device comprising means for performing a method according to any one of claims 39 to 42.

47. 43. An information processing apparatus for use in an electronic device comprising a first housing including a primary display and a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, the electronic device comprising means for performing a method according to any one of claims 39 to 42.

48. 1. A method included in a computing system comprising one or more processors, a memory, a first housing including a primary display, and a second housing including, at least in part, (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, the method comprising: displaying on the primary display a user interface for an application executing on the computing system; Detecting a first input at a particular location within the user interface; displaying, on the touch-sensitive secondary display, a set of affordances each corresponding to a distinct character in response to detecting the first input; A method comprising:

49. detecting a second input via the touch-sensitive secondary display across a first affordance corresponding to a first one of the distinct characters; in response to detecting the second input, displaying a preview of the first character at the particular location on the primary display while the input remains in contact with the first affordance; 49. The method of claim 48, further comprising:

50. detecting a movement of the second input via the touch-sensitive secondary display from the first affordance to a second affordance corresponding to a second one of the distinct characters; in response to detecting the movement of the second input from the first affordance to the second affordance, replacing the preview of the first character with a preview of the second character; 50. The method of claim 49, further comprising:

51. 51. The method of any one of claims 48 to 50, further comprising determining affordances to include in the set of affordances based at least in part on textual content included in the user interface.

52. 52. The method of claim 51, wherein the determining is performed in response to detecting that a user has modified textual content included in the user interface.

53. Detecting lift-off of the second input while the second input is in contact with a second affordance; in response to detecting lift-off, updating the user interface to include the second user interface element; 53. The method of any one of claims 48 to 52, further comprising:

54. Detecting an additional input while the second input is in contact with the second affordance; and updating the user interface to include the second user interface element in response to detecting the additional input; 53. The method of any one of claims 48 to 52, further comprising:

55. 55. The method of claim 48, further comprising displaying a preview of each letter among the distinct letters such that a corresponding affordance in the set of affordances is contacted by the second input as the second input continues to move across the touch-sensitive secondary display.

56. 56. The method of any one of claims 48 to 55, wherein the preview of the second character remains displayed on the primary display while the second input remains in contact with the second affordance.

57. 57. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors of a computing system having a first housing including a primary display and a second housing including, at least in part, (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, cause the computing system to perform the method of any one of claims 48 to 56.

58. one or more processors; a first housing including a primary display; a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display; A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing a method according to any one of claims 48 to 56; and An electronic device comprising:

59. 57. A graphical user interface on a computing system having one or more processors, a memory, a first housing including a primary display, and a second housing including, at least in part, (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, wherein the one or more processors execute one or more programs stored in the memory, and the graphical user interface comprises a user interface displayed in accordance with any one of the methods of claims 48 to 56.

60. 57. An electronic device having a first housing including a primary display and a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, the electronic device comprising means for performing a method according to any one of claims 48 to 56.

61. 57. An information processing apparatus for use in an electronic device including a first housing including a primary display and a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, the electronic device comprising means for performing a method according to any one of claims 48 to 56.

62. 1. A method included in a computing system comprising one or more processors, a memory, a first housing including a primary display, and a second housing including, at least in part, (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, the method comprising: Receiving a request to open a content editing application; In response to receiving the request, displaying the content editing application on the primary display; displaying a user interface control on the touch-sensitive secondary display that modifies at least one visual characteristic used to render content within the content editing application; A method comprising:

63. detecting an input at the user interface control via the touch-sensitive secondary display selecting a first value of the at least one visual characteristic; after detecting the input, rendering content in the content editing application using the first value of the at least one visual characteristic; 63. The method of claim 62, further comprising:

64. 64. The method of claim 63, wherein the user interface controls include respective controls corresponding respectively to respective values ​​of the at least one visual characteristic along a sliding scale of values.

65. 65. The method of claim 64, wherein the sliding scale of values ​​represents distinct shades of color.

66. 66. The method of claim 65, wherein the first value corresponds to a first shade of a first color, the method further comprising modifying the user interface control on the touch-sensitive secondary display to include an option to select other shades of the first color, distinct from the first shade of the first color, pursuant to a determination that the input meets a predetermined criterion.

67. 67. The method of any one of claims 62 to 66, further comprising receiving a selection of the content prior to rendering the content, wherein rendering the content comprises presenting a preview of the content using the first value of the at least one visual characteristic.

68. 68. The method of claim 67, wherein the preview is presented while the input remains in contact with the touch-sensitive secondary display.

69. 70. The method of claim 68, further comprising: in response to detecting lift-off of the input, ceasing to display the preview.

70. 70. The method of claim 69, further comprising, in response to detecting lift-off of the input, displaying a portion of editable content with the modified value of the at least one visual characteristic.

71. 71. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors of a computing system having a first housing including a primary display and a second housing including, at least in part, (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, cause the computing system to perform the method of any one of claims 62 to 70.

72. one or more processors; a first housing including a primary display; a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display; A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing a method according to any one of claims 62 to 70; and An electronic device comprising:

73. 71. A graphical user interface on a computing system having one or more processors, a memory, a first housing including a primary display, and a second housing including, at least in part, (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, wherein the one or more processors execute one or more programs stored in the memory, and the graphical user interface comprises a user interface displayed in accordance with any one of the methods of claims 62 to 70.

74. 71. An electronic device having a first housing including a primary display and a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, the electronic device comprising means for performing a method according to any one of claims 62 to 70.

75. 71. An information processing apparatus for use in an electronic device including a first housing including a primary display and a second housing including at least in part (i) a physical keyboard and (ii) a touch-sensitive secondary display separate from the primary display, the electronic device comprising means for performing a method according to any one of claims 62 to 70.

76. In an electronic device having a display and a biosensor, displaying respective login user interfaces associated with logging into a plurality of user accounts, including a first user account and a second user account, while the device is in a locked state; receiving biometric information about a user while displaying the login user interface; In response to receiving the biometric information, pursuant to a determination that the biometric information matches biometric information of the first user account of the plurality of user accounts while the first user account does not have an active session on the device, displaying on the display a prompt to enter login credentials for the first user account; pursuant to a determination that the biometric information matches biometric information of the second user account of the plurality of user accounts, displaying on the display a prompt to enter login credentials for the second user account while the second user account does not have an active session on the device; A method comprising:

77. In response to receiving the biometric information, 77. The method of claim 76, comprising unlocking the device for the first user account pursuant to a determination that the biometric information matches biometric information of the first user account of the plurality of user accounts while the first user account has an active session on the device.

78. In response to receiving the biometric information, 80. The method of claim 77, comprising unlocking the device for the second user account pursuant to a determination that the biometric information matches biometric information of the second user account of the plurality of user accounts while the second user account has an active session on the device.

79. In response to receiving the biometric information, 80. The method of claim 78, comprising maintaining the device in the locked state pursuant to a determination that the biometric information does not match biometric information of any user account of the device.

80. 80. The method of any one of claims 76 to 79, wherein the login user interface includes a plurality of selectable affordances corresponding to the plurality of user accounts.

81. receiving entry of login credentials for the first user account while displaying the prompt to enter login credentials; In response to receiving the entry of the login credentials, unlocking the device for the first user account in accordance with a determination that the login credentials match login credentials for the first user account; maintaining the device in the locked state in accordance with a determination that the login credentials do not match the login credentials of the first user account; 80. The method of any one of claims 76 to 79, comprising:

82. receiving entry of login credentials for the second user account while displaying the prompt to enter login credentials for the second user account; In response to receiving the entry of the login credentials, unlocking the device for the first user account in accordance with a determination that the login credentials match login credentials for the second user account; maintaining the device in the locked state in accordance with a determination that the login credentials do not match the login credentials of the second user account; 80. The method of any one of claims 76 to 79, comprising:

83. 80. The method of any one of claims 76 to 79, wherein the login user interface includes instructions to provide biometric information.

84. the device includes a secondary display adjacent to the biosensor; 80. The method of any one of claims 76 to 79, the method including displaying on the secondary display instructions to provide biometric information via the biometric sensor while displaying the login user interface on the display of the device.

85. 80. The method of any one of claims 76 to 79, wherein the biometric sensor is a fingerprint sensor.

86. 80. The method of any one of claims 76 to 79, wherein the biometric sensor is a face detection sensor.

87. 80. The method of any one of claims 76 to 79, wherein the biosensor is a retinal scanner.

88. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors of a computing system having a display and a biometric sensor, cause the computing system to perform the method of any one of claims 76 to 87.

89. one or more processors; A display and A biosensor; a memory storing one or more programs configured to be executed by said one or more processors, said one or more programs including instructions for performing a method according to any one of claims 76 to 87; An electronic device comprising:

90. A graphical user interface on a computing system having one or more processors, a memory, a display, and a biosensor, wherein the one or more processors execute one or more programs stored in the memory, and the graphical user interface comprises a user interface displayed in accordance with any one of the methods of claims 76 to 87.

91. 88. An electronic device having a display and a biosensor, said electronic device comprising means for performing the method according to any one of claims 76 to 87.

92. 88. An information processing apparatus for use in an electronic device including a display and a biosensor, said electronic device comprising means for performing the method of any one of claims 76 to 87.

93. 1. An electronic device having a display and an input element with an integrated biosensor, comprising: displaying a user interface associated with a first user account while the device is logged into the first user account; the device is associated with a plurality of user accounts, including the first user account and a second user account; the second user account is associated with biometric information that enables logging into the second user account; And, receiving input via the input element having the integral biometric sensor while displaying the user interface associated with the first user account; in response to receiving the input via the input element having the integral biosensor; in accordance with a determination that the input satisfies second user switching criteria while the second user account has an active session on the device, the second user switching criteria including a requirement that biometric information detected during the input with the input element matches biometric information of the second user account of the plurality of user accounts; unlocking the device for the second user account; locking the device for the first user account; replacing a display of the user interface associated with the first account with a user interface associated with the second user account; A method comprising:

94. In response to receiving the biometric information, 94. The method of claim 93, comprising displaying a prompt on the display to enter login credentials for the second user account pursuant to a determination that the input satisfies second user switching criteria while the second user account does not have an active session on the device, the second user switching criteria including a requirement that biometric information detected during the input with the input element matches biometric information of the second user account of the plurality of user accounts.

95. In response to receiving the biometric information, in accordance with a determination that the input satisfies third user switching criteria while a third user account has an active session on the device, the third user switching criteria including a requirement that biometric information detected during the input with the input element matches biometric information of the third user account of the plurality of user accounts; unlocking the device for the third user account; and locking the device for the first user account; replacing a display of the user interface associated with the first account with a user interface associated with the third user account; 95. The method of claim 94, comprising:

96. In response to receiving the biometric information, 96. The method of claim 95, comprising displaying a prompt on the display to enter login credentials for the third user account pursuant to a determination that the input satisfies third user switching criteria while the third user account does not have an active session on the device, the third user switching criteria including a requirement that biometric information detected during the input with the input element matches biometric information of the third user account of the plurality of user accounts.

97. the input element is a button; 95. The method of claim 93 or 94, wherein the input via the input element with the integrated biosensor includes detecting a press input via the button.

98. the second user switching criterion includes a criterion that the press input lasts less than a first threshold time; 98. The method of claim 97, wherein the method includes, in response to receiving the press input via the input element with the integral biometric sensor, entering the device into a low power mode pursuant to a determination that the press input using the button lasts longer than the first threshold time.

99. the second user switching criterion includes a criterion that the button press is less than a first threshold time; The method includes, in response to receiving the input via the input element having the integral biometric sensor, entering the device into a low power mode in response to a determination that the press input using the button lasts longer than the first threshold time and shorter than a second threshold time; displaying a menu of options for changing a state of the device in response to a determination that the press input using the button lasts longer than the second threshold time; and 98. The method of claim 97, comprising:

100. in response to receiving the input via the input element having the integral biosensor; 100. The method of claim 99, comprising rebooting the device pursuant to a determination that the press input with the button lasts for longer than a third threshold time that is greater than the second threshold time.

101. replacing the display of the user interface associated with the first account with a user interface associated with the second user account; receiving a second input via the input element having the integral biometric sensor while displaying the user interface associated with the second user account; in response to receiving the second input via the input element having the integral biosensor; in accordance with a determination that the second input satisfies a first user switching criterion while the first user account has an active session on the device, the first user switching criterion including a requirement that biometric information detected during the input with the input element matches biometric information of the first user account of the plurality of user accounts; unlocking the device for the first user account; locking the device for the second user account; replacing a display of the user interface associated with the second account with a user interface associated with the first user account; 97. The method of any one of claims 93 to 96, comprising:

102. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors of a computing system having a display and a biometric sensor, cause the computing system to perform the method of any one of claims 93 to 101.

103. one or more processors; A display and A biosensor; A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing a method according to any one of claims 93 to 101; An electronic device comprising:

104. A graphical user interface on a computing system having one or more processors, a memory, a display, and a biosensor, wherein the one or more processors execute one or more programs stored in the memory, and the graphical user interface comprises a user interface displayed in accordance with any one of the methods of claims 93 to 101.

105. 102. An electronic device having a display and a biosensor, said electronic device comprising means for performing the method according to any one of claims 93 to 101.

106. 102. An information processing device for use in an electronic device including a display and a biosensor, said electronic device comprising means for performing the method of any one of claims 93 to 101.

Images